Terraform Configuration for AWS EKS Observability with Datadog and PagerDuty Alerting

In the dynamic world of cloud-native applications, maintaining robust observability for your Kubernetes clusters is paramount. This comprehensive guide will walk you through setting up a powerful observability stack for your AWS Elastic Kubernetes Service (EKS) cluster using Terraform, integrating Datadog for monitoring and PagerDuty for incident management. By codifying your infrastructure and monitoring setup, you achieve consistency, repeatability, and efficient management of your critical services.

Architecture Pro-Tip: Modular Observability

When designing your observability strategy for EKS, always strive for modularity. Separate your core EKS cluster configuration from your observability tooling. This allows you to independently manage and scale monitoring components, apply least-privilege principles to service accounts, and easily swap out tools if your requirements evolve without disrupting the underlying cluster. Utilize dedicated namespaces and service accounts for your observability agents.

Why Terraform for EKS Observability?

Terraform, as an Infrastructure as Code (IaC) tool, offers significant advantages for managing complex cloud environments like AWS EKS, especially when integrating third-party services for observability:

Automation & Repeatability: Define your entire monitoring stack in code, ensuring consistent deployments across environments.
Version Control: Track changes, review, and roll back your observability configurations like any other codebase.
Reduced Manual Error: Eliminate human error associated with manual configuration of dashboards, alerts, and integrations.
Scalability: Easily scale your monitoring footprint as your EKS clusters and applications grow.

Prerequisites

Before you begin, ensure you have the following:

An active AWS account with administrative access.
An existing AWS EKS cluster. This guide assumes you have one provisioned.
Terraform CLI installed (version 1.0+ recommended).
kubectl CLI configured to connect to your EKS cluster.
A Datadog account with an API Key and an Application Key.
A PagerDuty account with an API Token for service creation and an integration key (can be generated via Terraform).
Helm CLI installed (required by Terraform's Helm provider).

Core Components Overview

This setup will orchestrate the following:

Datadog Agent on EKS: Deployed as a DaemonSet using the Datadog Helm chart, collecting metrics, logs, and traces from your EKS nodes and pods.
Datadog Monitors: Configured to detect anomalous behavior (e.g., high CPU, low memory, pod restarts) within your EKS cluster.
PagerDuty Service and Integration: A dedicated PagerDuty service to receive alerts from Datadog, with an associated escalation policy to ensure timely notifications.
Terraform Providers: Leveraging the aws, datadog, and pagerduty providers to manage these resources declaratively.

Comprehensive Terraform Configuration

Let's dive into the Terraform code. Create a new directory for your project and populate it with the following files.

# main.tf - Provider and EKS Data Configuration
terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = "~> 5.0"
    }
    datadog = {
      source = "DataDog/datadog"
      version = "~> 3.0"
    }
    pagerduty = {
      source = "PagerDuty/pagerduty"
      version = "~> 2.0"
    }
    kubernetes = {
      source = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source = "hashicorp/helm"
      version = "~> 2.11"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

data "aws_eks_cluster" "this" {
  name = var.cluster_name
}

data "aws_eks_cluster_auth" "this" {
  name = var.cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.this.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.this.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.this.token
  }
}

# pagerduty.tf - PagerDuty Service and Escalation Policy
resource "pagerduty_escalation_policy" "eks_observability_policy" {
  name = "${var.cluster_name}-EKS-Observability-Policy"
  num_loops = 2

  rule {
    delay_in_minutes = 5
    target {
      type = "user"
      id   = var.pagerduty_user_id # Replace with a valid PagerDuty user ID
    }
  }
}

resource "pagerduty_service" "eks_observability_service" {
  name = "${var.cluster_name}-EKS-Observability-Service"
  description = "Service for AWS EKS ${var.cluster_name} observability alerts from Datadog."
  escalation_policy = pagerduty_escalation_policy.eks_observability_policy.id
}

resource "pagerduty_service_integration" "datadog_integration" {
  name = "Datadog Integration"
  service = pagerduty_service.eks_observability_service.id
  type = "datadog"
}

# datadog_agent.tf - Datadog Agent Deployment via Helm
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  version    = "2.34.0" # Use a recent stable version
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }

  set {
    name  = "kubeStateMetricsCore.enabled"
    value = true
  }

  set {
    name  = "clusterAgent.enabled"
    value = true
  }

  set {
    name  = "clusterChecksRunner.enabled"
    value = true
  }

  set {
    name  = "datadog.tags"
    value = "env:${var.environment},cluster:${var.cluster_name}"
  }

  set {
    name  = "datadog.site"
    value = "datadoghq.com" # or eu.datadoghq.com etc.
  }

  set {
    name  = "datadog.kubelet.tlsVerify"
    value = false
  }

  set {
    name  = "datadog.confd.kube_proxy.yaml"
    value = "init_config: {} instances: [ { url: http://localhost:10249/metrics } ]"
  }
}

# datadog_monitors.tf - Datadog Monitors
resource "datadog_monitor" "eks_node_cpu_utilization" {
  name    = "EKS Node CPU Utilization High (${var.cluster_name} - ${var.environment})"
  type    = "metric alert"
  query   = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.cluster_name}} by {host} > 80"
  message = "EKS node CPU utilization is high on {{host.name}} in cluster ${var.cluster_name}. @pagerduty-${pagerduty_service_integration.datadog_integration.integration_key}"
  tags    = ["env:${var.environment}", "service:eks", "severity:high"]

  escalation_message = "EKS Node CPU has remained high for 15 minutes. @pagerduty-${pagerduty_service_integration.datadog_integration.integration_key}"

  notify_no_data    = false
  no_data_timeframe = 20
  renotify_interval = 60 # re-notify every hour if unresolved
  require_full_window = false
  thresholds {
    warning  = "70.0"
    critical = "80.0"
  }
}

resource "datadog_monitor" "eks_pod_restarts" {
  name    = "EKS Pod Restarts Rate High (${var.cluster_name} - ${var.environment})"
  type    = "metric alert"
  query   = "sum(last_5m):kubernetes.pod.restarts.count{cluster_name:${var.cluster_name}} by {kube_namespace,kube_deployment} > 3"
  message = "High rate of pod restarts detected in {{kube_namespace}}/{{kube_deployment}} in cluster ${var.cluster_name}. @pagerduty-${pagerduty_service_integration.datadog_integration.integration_key}"
  tags    = ["env:${var.environment}", "service:eks", "severity:medium"]

  escalation_message = "Pod restarts persist in {{kube_namespace}}/{{kube_deployment}}. @pagerduty-${pagerduty_service_integration.datadog_integration.integration_key}"

  notify_no_data    = false
  no_data_timeframe = 20
  renotify_interval = 30
  require_full_window = false
  thresholds {
    warning  = "2.0"
    critical = "3.0"
  }
}

# variables.tf
variable "aws_region" {
  description = "AWS region for the EKS cluster"
  type        = string
}

variable "cluster_name" {
  description = "Name of the EKS cluster"
  type        = string
}

variable "environment" {
  description = "Environment tag (e.g., dev, staging, prod)"
  type        = string
  default     = "dev"
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

variable "pagerduty_user_id" {
  description = "The ID of a PagerDuty user to add to the escalation policy (e.g., PXXXXXX)"
  type        = string
}

# outputs.tf
output "datadog_agent_helm_release_status" {
  description = "Status of the Datadog Agent Helm release"
  value       = helm_release.datadog_agent.status
}

output "pagerduty_service_name" {
  description = "Name of the PagerDuty service created"
  value       = pagerduty_service.eks_observability_service.name
}

output "datadog_pagerduty_integration_key" {
  description = "The integration key for Datadog to PagerDuty"
  value       = pagerduty_service_integration.datadog_integration.integration_key
  sensitive   = true
}
    

Important Considerations for the Code:

EKS Authentication: The Kubernetes and Helm providers dynamically authenticate to EKS using data sources for the cluster endpoint and token, generated by the AWS provider. This avoids hardcoding credentials.
Datadog Agent Helm Chart: We are using the official Datadog Helm chart. The set blocks configure critical parameters like API/APP keys, enable Kube State Metrics, Cluster Agent, and Cluster Checks Runner for comprehensive EKS monitoring.
Datadog Monitors: Two example monitors are provided: one for high node CPU and another for frequent pod restarts. Notice the @pagerduty-${pagerduty_service_integration.datadog_integration.integration_key} syntax in the message. This is how Datadog knows to send alerts to the specific PagerDuty integration.
PagerDuty Escalation Policy: A basic escalation policy is created to ensure alerts are routed to a specified user. In a production environment, this would typically involve schedules and multiple layers of escalation.
Sensitive Variables: Datadog and PagerDuty keys are marked as sensitive = true to prevent them from being displayed in Terraform output. Pass them via environment variables or a .tfvars file.

Deployment Steps

Follow these steps to deploy your observability stack:

Save the Files: Save the code snippets above into main.tf, pagerduty.tf, datadog_agent.tf, datadog_monitors.tf, variables.tf, and outputs.tf in your project directory.
Configure Variables: Create a terraform.tfvars file with your specific values. For example:
aws_region = "us-east-1" cluster_name = "my-eks-cluster" environment = "production" datadog_api_key = "YOUR_DATADOG_API_KEY" datadog_app_key = "YOUR_DATADOG_APP_KEY" pagerduty_api_token = "YOUR_PAGERDUTY_API_TOKEN" pagerduty_user_id = "P123456" # Your PagerDuty User ID
Security Note: For production, consider using AWS Secrets Manager or HashiCorp Vault to securely inject sensitive variables instead of plain .tfvars files.
Initialize Terraform: Open your terminal in the project directory and run:
terraform init
Review the Plan: Examine the changes Terraform proposes:
terraform plan
Apply the Configuration: If the plan is satisfactory, apply the changes:
terraform apply
Type yes when prompted to confirm.

Verification

After a successful terraform apply:

Datadog Agent: Check your EKS cluster for the running Datadog Agent pods:
kubectl get pods -n datadog
You should see pods for datadog-agent, datadog-cluster-agent, and datadog-cluster-checks-runner in a Running state.
Datadog UI: Log into your Datadog account.
- Navigate to Monitors > Manage Monitors. You should see the "EKS Node CPU Utilization High" and "EKS Pod Restarts Rate High" monitors listed.
- Go to Infrastructure > Hosts. Your EKS nodes should appear and begin reporting metrics.
PagerDuty UI: Log into your PagerDuty account.
- Go to Services > Service Directory. Your "EKS-Observability-Service" should be listed.
- Check the integrations within that service; you should find the Datadog integration.

Troubleshooting and Best Practices

Common Issues:

EKS Authentication Errors: Ensure your AWS CLI is configured with credentials that have access to the EKS cluster. Run aws sts get-caller-identity to verify.
Datadog Agent Not Reporting: Double-check your datadog_api_key and datadog_app_key. Verify the Datadog site (datadoghq.com vs. eu.datadoghq.com) is correct. Check Datadog Agent logs for errors: kubectl logs -n datadog -l app=datadog --tail=100.
PagerDuty Alerts Not Firing: Ensure the pagerduty_api_token is correct. Verify the @pagerduty-${integration_key} syntax in your Datadog monitor message matches the outputted integration key exactly. Check the Datadog Event Explorer for events that should trigger alerts.
Helm Release Errors: Ensure Helm CLI is installed and configured. If you encounter timeout issues, increase the Helm provider's timeout values.

Best Practices:

Service Accounts & IAM Roles: For enhanced security, configure an IAM Role for Service Accounts (IRSA) for your Datadog Agent to allow it to collect metrics and logs from AWS services directly without storing AWS credentials in Kubernetes secrets.
Advanced Datadog Configuration: Explore Datadog's extensive integrations for specific AWS services, custom metrics, APM, and log management. Configure Autodiscovery for seamless monitoring of new services deployed on EKS.
Granular PagerDuty Escalation: Implement more sophisticated PagerDuty escalation policies, including multiple notification layers, on-call schedules, and services for different criticality levels.
Terraform Modules: For larger setups, consider encapsulating your Datadog and PagerDuty configurations into reusable Terraform modules.
State Management: Always use a remote backend (like AWS S3 with DynamoDB locking) for your Terraform state to enable collaboration and prevent state corruption.

Conclusion

By following this guide, you've successfully deployed a robust observability solution for your AWS EKS cluster using Terraform. You've integrated Datadog for comprehensive monitoring of your Kubernetes infrastructure and applications, and PagerDuty for reliable incident alerting and management. This IaC approach ensures that your observability stack is as resilient, scalable, and manageable as your cloud-native applications themselves, empowering your team with critical insights and rapid response capabilities. Continuously refine your monitors and alert thresholds to align with the evolving needs and performance characteristics of your applications.

CloudStack YAML