Terraform for AWS EKS Observability: Integrating Datadog and PagerDuty

In the dynamic landscape of cloud-native applications, maintaining robust observability for Kubernetes clusters is paramount. AWS EKS (Elastic Kubernetes Service) provides a powerful foundation, but effectively monitoring its health, performance, and operational incidents requires specialized tools. This comprehensive guide details how to leverage Terraform for automating the integration of Datadog for deep observability and PagerDuty for streamlined incident management within your EKS environment, ensuring your applications remain resilient and highly available.

Architecture Pro-Tip: Layered Observability Strategy

Always adopt a layered observability strategy for EKS. Combine in-cluster agents (Datadog Agent) for granular metrics, logs, and traces with external cloud-native services (AWS CloudWatch, Datadog) for comprehensive insights. Standardize your resource tagging across AWS, Kubernetes, and Datadog to enable powerful filtering, correlation, and cost attribution. This holistic approach ensures you capture data from every component, from the underlying EC2 instances to the individual application pods, making root cause analysis significantly faster.

Why Terraform, Datadog, and PagerDuty for EKS?

Managing a Kubernetes cluster, especially in production, demands visibility into its components and rapid response to any anomalies. Here’s why this trio is a winning combination:

Terraform: Infrastructure as Code (IaC)
Terraform allows you to define, provision, and manage your cloud infrastructure and services using a declarative configuration language. For EKS observability, this means deploying monitoring agents, configuring dashboards, setting up alerts, and defining incident response policies—all as code. This ensures consistency, repeatability, version control, and auditability across environments.
Datadog: Unified Observability Platform
Datadog provides end-to-end visibility across your entire technology stack. For EKS, it offers:
- Metrics: Real-time performance data for nodes, pods, containers, and applications.
- Logs: Centralized log aggregation and analysis from all Kubernetes components and applications.
- APM & Tracing: Distributed tracing for microservices running on EKS.
- Synthetics: Proactive monitoring of application uptime and performance.
- Security: Cloud Security Posture Management (CSPM) and Cloud Workload Security (CWS) for EKS.
- Alerting: Sophisticated anomaly detection and threshold-based alerts with rich context.
PagerDuty: Incident Management and Response
When incidents occur, PagerDuty ensures the right people are notified at the right time. It offers:
- On-call Management: Scheduling, rotations, and escalations.
- Alerting: Multi-channel notifications (SMS, phone, email, push).
- Incident Orchestration: Automated actions, status updates, and stakeholder communication.
- Analytics: Post-mortem analysis and continuous improvement.

Prerequisites

Before you begin, ensure you have the following:

An active AWS Account with administrative privileges.
An existing AWS EKS Cluster. For creating one with Terraform, refer to AWS EKS module documentation.
Terraform CLI installed (v1.0.0 or higher recommended).
An active Datadog Account with an API Key and Application Key.
An active PagerDuty Account with a PagerDuty API Key and a Service (or the ability to create one).
kubectl configured to connect to your EKS cluster.
Helm CLI installed (optional, but good for local testing).

Step-by-Step Integration with Terraform

1. Configure Terraform Providers

You'll need the AWS, Datadog, PagerDuty, and Kubernetes (or Helm) providers. The AWS provider is crucial for fetching EKS cluster details, while Kubernetes/Helm is for deploying the Datadog Agent.

provider "aws" {
  region = var.aws_region
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.eks_cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.eks_cluster_auth.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.eks_cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.eks_cluster_auth.token
  }
}

2. Deploy Datadog Agent to EKS using Helm and Terraform

The Datadog Agent is deployed as a DaemonSet across your EKS nodes and a Cluster Agent deployment. Using the Terraform Helm provider simplifies this. Ensure you provide your Datadog API key.

data "aws_eks_cluster" "eks_cluster" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "eks_cluster_auth" {
  name = var.eks_cluster_name
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  version    = "2.33.2" # Use a stable, recent version
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }
  set {
    name  = "datadog.site"
    value = var.datadog_site # e.g., "datadoghq.com"
  }
  set {
    name  = "datadog.kubelet.host"
    value = data.aws_eks_cluster.eks_cluster.endpoint
  }
  set {
    name  = "datadog.kubelet.tlsVerify"
    value = "false" # Set to true with proper cert setup for production
  }
  set {
    name  = "datadog.collectEvents"
    value = "true"
  }
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "apm.enabled"
    value = "true"
  }
  set {
    name  = "processAgent.enabled"
    value = "true"
  }
  set {
    name  = "clusterChecks.enabled"
    value = "true"
  }
  set {
    name  = "containerRuntime.criSocketPath"
    value = "/var/run/containerd/containerd.sock" # Adjust based on your EKS AMI container runtime
  }
  set {
    name  = "targetSystem"
    value = "linux"
  }
  set {
    name  = "agents.tolerations[0].operator"
    value = "Exists"
  }
  # Add other necessary configurations like RBAC, tags, etc.
}

3. Create PagerDuty Service and Integration with Terraform

Define a PagerDuty service that will receive incidents from Datadog. You can also define escalation policies and users, but for simplicity, we'll focus on a service and its integration key.

# Optional: Define an on-call team
resource "pagerduty_team" "devops_team" {
  name        = "DevOps Team"
  description = "Team responsible for EKS operations and incident response."
}

# Optional: Define an on-call user
resource "pagerduty_user" "oncall_engineer" {
  name  = "Alice Engineer"
  email = "alice.engineer@example.com"
  teams = [pagerduty_team.devops_team.id]
}

# Define an escalation policy (example for a single user/team)
resource "pagerduty_escalation_policy" "eks_escalation_policy" {
  name      = "EKS Primary Escalation"
  num_loops = 2
  teams     = [pagerduty_team.devops_team.id] # Associate with the team

  rule {
    escalation_delay_in_minutes = 15
    target {
      type = "user_reference"
      id   = pagerduty_user.oncall_engineer.id
    }
  }
  rule {
    escalation_delay_in_minutes = 30
    target {
      type = "team_reference"
      id   = pagerduty_team.devops_team.id
    }
  }
}

# Define the PagerDuty service for EKS alerts
resource "pagerduty_service" "eks_monitoring_service" {
  name                    = "${var.eks_cluster_name}-Observability"
  description             = "Service for EKS cluster observability alerts from Datadog."
  auto_resolve_timeout    = 60
  acknowledgement_timeout = 30
  escalation_policy       = pagerduty_escalation_policy.eks_escalation_policy.id
  teams                   = [pagerduty_team.devops_team.id]
}

# Create a Datadog integration for the PagerDuty service
resource "pagerduty_service_integration" "datadog_integration" {
  name    = "Datadog Integration"
  service = pagerduty_service.eks_monitoring_service.id
  type    = "generic_events_api_inbound_integration" # Datadog uses Generic Events API
}

output "pagerduty_integration_key" {
  description = "The integration key for Datadog to send events to PagerDuty."
  value       = pagerduty_service_integration.datadog_integration.integration_key
  sensitive   = true
}

4. Configure Datadog Monitors with PagerDuty Integration

Now, create Datadog monitors using the Terraform datadog_monitor resource. Crucially, you'll use the PagerDuty integration key generated in the previous step to route alerts.

resource "datadog_monitor" "eks_critical_cpu" {
  name                = "EKS Cluster CPU Utilization Critical - ${var.eks_cluster_name}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.eks_cluster_name}} by {host} > 85"
  message             = "EKS host CPU usage is critically high! Investigate host: {{host.name}}.\n\n@pagerduty-${pagerduty_service_integration.datadog_integration.integration_key}"
  monitor_threshold_window = "last_15m"
  evaluation_delay    = 90
  renotify_interval   = 0 # Do not renotify unless resolved and re-triggered
  timeout_h           = 0
  no_data_timeframe = 60 # Alert if no data for 60 minutes

  # Critical threshold
  threshold {
    critical = 85
  }

  # Warning threshold
  threshold_warning {
    warning = 75
  }

  notify_no_data = false
  new_group_delay = 300 # Wait 5 minutes before grouping new alerts

  tags = [
    "environment:${var.environment}",
    "service:eks-control-plane",
    "cluster:${var.eks_cluster_name}",
    "severity:critical"
  ]
}

resource "datadog_monitor" "eks_pod_restarts_alert" {
  name                = "EKS Pod Restarts High - ${var.eks_cluster_name}"
  type                = "metric alert"
  query               = "sum(last_5m):sum:kubernetes.pod.restarts{cluster_name:${var.eks_cluster_name}} by {pod_name} > 5"
  message             = "Pod {{pod_name.name}} on EKS cluster ${var.eks_cluster_name} is experiencing excessive restarts. Investigate underlying issues.\n\n@pagerduty-${pagerduty_service_integration.datadog_integration.integration_key}"
  monitor_threshold_window = "last_10m"
  evaluation_delay    = 60
  renotify_interval   = 0
  timeout_h           = 0
  no_data_timeframe = 30 # Alert if no data for 30 minutes

  threshold {
    critical = 5
  }

  notify_no_data = false
  new_group_delay = 300

  tags = [
    "environment:${var.environment}",
    "service:kubernetes",
    "cluster:${var.eks_cluster_name}",
    "severity:high"
  ]
}

Comprehensive Terraform Configuration (`main.tf`)

Here’s how you can structure your main.tf file with necessary variables for a complete setup.

# variables.tf
variable "aws_region" {
  description = "AWS region for the EKS cluster."
  type        = string
  default     = "us-east-1"
}

variable "eks_cluster_name" {
  description = "The name of the existing EKS cluster."
  type        = string
}

variable "datadog_api_key" {
  description = "Your Datadog API Key."
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Your Datadog Application Key."
  type        = string
  sensitive   = true
}

variable "datadog_site" {
  description = "The Datadog site to use (e.g., datadoghq.com, eu.datadoghq.com)."
  type        = string
  default     = "datadoghq.com"
}

variable "pagerduty_api_token" {
  description = "Your PagerDuty API Token (requires global API key, not integration key)."
  type        = string
  sensitive   = true
}

variable "environment" {
  description = "Environment tag for resources."
  type        = string
  default     = "production"
}

# main.tf

# --- Providers ---
provider "aws" {
  region = var.aws_region
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

data "aws_eks_cluster" "eks_cluster" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "eks_cluster_auth" {
  name = var.eks_cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.eks_cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.eks_cluster_auth.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.eks_cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.eks_cluster_auth.token
  }
}

# --- Datadog Agent Deployment ---
resource "kubernetes_namespace" "datadog_namespace" {
  metadata {
    name = "datadog"
  }
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = kubernetes_namespace.datadog_namespace.metadata[0].name
  version    = "2.33.2" # Always pin to a specific version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }
  set {
    name  = "datadog.site"
    value = var.datadog_site
  }
  set {
    name  = "datadog.kubelet.host"
    value = data.aws_eks_cluster.eks_cluster.endpoint
  }
  set {
    name  = "datadog.kubelet.tlsVerify"
    value = "false"
  }
  set {
    name  = "datadog.collectEvents"
    value = "true"
  }
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "apm.enabled"
    value = "true"
  }
  set {
    name  = "processAgent.enabled"
    value = "true"
  }
  set {
    name  = "clusterChecks.enabled"
    value = "true"
  }
  set {
    name  = "containerRuntime.criSocketPath"
    value = "/var/run/containerd/containerd.sock"
  }
  set {
    name  = "targetSystem"
    value = "linux"
  }
  set {
    name  = "agents.tolerations[0].operator"
    value = "Exists"
  }
  set {
    name  = "tags[0]"
    value = "environment:${var.environment}"
  }
  set {
    name  = "tags[1]"
    value = "cluster_name:${var.eks_cluster_name}"
  }
}

# --- PagerDuty Configuration ---
resource "pagerduty_team" "devops_team" {
  name        = "DevOps Team - ${var.eks_cluster_name}"
  description = "Team responsible for EKS operations and incident response for ${var.eks_cluster_name}."
}

resource "pagerduty_user" "oncall_engineer" {
  # In a real scenario, you would dynamically get this from an existing user or create multiple.
  # For simplicity, assuming a single placeholder user.
  name  = "EKS On-Call Engineer"
  email = "oncall-eks@example.com" # Replace with a valid email
  teams = [pagerduty_team.devops_team.id]
}

resource "pagerduty_escalation_policy" "eks_escalation_policy" {
  name      = "${var.eks_cluster_name} - Primary Escalation"
  num_loops = 2
  teams     = [pagerduty_team.devops_team.id]

  rule {
    escalation_delay_in_minutes = 15
    target {
      type = "user_reference"
      id   = pagerduty_user.oncall_engineer.id
    }
  }
  rule {
    escalation_delay_in_minutes = 30
    target {
      type = "team_reference"
      id   = pagerduty_team.devops_team.id
    }
  }
}

resource "pagerduty_service" "eks_monitoring_service" {
  name                    = "${var.eks_cluster_name}-Observability"
  description             = "Service for EKS cluster observability alerts from Datadog in ${var.environment}."
  auto_resolve_timeout    = 60
  acknowledgement_timeout = 30
  escalation_policy       = pagerduty_escalation_policy.eks_escalation_policy.id
  teams                   = [pagerduty_team.devops_team.id]
}

resource "pagerduty_service_integration" "datadog_integration" {
  name    = "Datadog Integration"
  service = pagerduty_service.eks_monitoring_service.id
  type    = "generic_events_api_inbound_integration"
}

# --- Datadog Monitors ---
resource "datadog_monitor" "eks_critical_cpu" {
  name                = "EKS Cluster CPU Utilization Critical - ${var.eks_cluster_name} (${var.environment})"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.eks_cluster_name}} by {host} > 85"
  message             = "EKS host CPU usage is critically high! Investigate host: {{host.name}}.\n\n@pagerduty-${pagerduty_service_integration.datadog_integration.integration_key}"
  monitor_threshold_window = "last_15m"
  evaluation_delay    = 90
  renotify_interval   = 0
  timeout_h           = 0
  no_data_timeframe = 60

  threshold {
    critical = 85
  }

  threshold_warning {
    warning = 75
  }

  notify_no_data = false
  new_group_delay = 300

  tags = [
    "environment:${var.environment}",
    "service:eks-control-plane",
    "cluster:${var.eks_cluster_name}",
    "severity:critical"
  ]
}

resource "datadog_monitor" "eks_pod_restarts_alert" {
  name                = "EKS Pod Restarts High - ${var.eks_cluster_name} (${var.environment})"
  type                = "metric alert"
  query               = "sum(last_5m):sum:kubernetes.pod.restarts{cluster_name:${var.eks_cluster_name}} by {pod_name} > 5"
  message             = "Pod {{pod_name.name}} on EKS cluster ${var.eks_cluster_name} is experiencing excessive restarts. Investigate underlying issues.\n\n@pagerduty-${pagerduty_service_integration.datadog_integration.integration_key}"
  monitor_threshold_window = "last_10m"
  evaluation_delay    = 60
  renotify_interval   = 0
  timeout_h           = 0
  no_data_timeframe = 30

  threshold {
    critical = 5
  }

  notify_no_data = false
  new_group_delay = 300

  tags = [
    "environment:${var.environment}",
    "service:kubernetes",
    "cluster:${var.eks_cluster_name}",
    "severity:high"
  ]
}

# --- Outputs ---
output "datadog_integration_key" {
  description = "PagerDuty integration key for Datadog."
  value       = pagerduty_service_integration.datadog_integration.integration_key
  sensitive   = true
}

output "pagerduty_service_url" {
  description = "URL to the configured PagerDuty service."
  value       = "https://${var.pagerduty_site}/services/${pagerduty_service.eks_monitoring_service.id}"
}

Deployment Steps:

Save the code above into main.tf and variables.tf files in a new directory.
Initialize Terraform: terraform init
Plan the changes: terraform plan -var="eks_cluster_name=your-eks-cluster-name" -var="datadog_api_key=..." -var="datadog_app_key=..." -var="pagerduty_api_token=..."
Apply the configuration: terraform apply -var="eks_cluster_name=your-eks-cluster-name" -var="datadog_api_key=..." -var="datadog_app_key=..." -var="pagerduty_api_token=..."

Verification and Testing

After applying your Terraform configuration:

Datadog Agent: Navigate to the Datadog Infrastructure list. You should see your EKS nodes appearing, sending metrics, logs, and traces. Verify the Datadog Agent pods are running in your EKS cluster: kubectl get pods -n datadog.
Datadog Monitors: Check the Datadog Monitor Management page. Your newly created monitors should be listed and evaluating.
PagerDuty Service: Log into PagerDuty and confirm that the service and integration were created.
Test Alert: To test the integration, you can manually trigger a test alert from Datadog for one of your monitors, or temporarily lower a monitor threshold to force an alert. Verify that an incident is created in PagerDuty and that the correct on-call team/user is notified.

Troubleshooting & Best Practices

Common Troubleshooting Steps:

API Keys/Tokens: Double-check that all Datadog and PagerDuty API keys/tokens are correct and have the necessary permissions. Refer to Datadog and PagerDuty documentation for required scopes.
Datadog Agent Logs: If EKS nodes aren't appearing in Datadog, check the Datadog Agent pod logs: kubectl logs -f <datadog-agent-pod-name> -n datadog.
EKS Permissions: Ensure the IAM role associated with your EKS worker nodes (or the Cluster Agent) has permissions to collect metrics and logs, if you customize the Datadog agent's roles.
Network Connectivity: Verify that your EKS cluster nodes can reach Datadog endpoints (e.g., app.datadoghq.com) and that Datadog can reach PagerDuty's API.
Helm Release Issues: If helm_release fails, use helm get values datadog -n datadog and helm get manifest datadog -n datadog to inspect the deployed configuration and Kubernetes resources.

Best Practices:

Secrets Management: Avoid hardcoding API keys. Use a secure secrets manager like AWS Secrets Manager or HashiCorp Vault with Terraform to pass sensitive variables.
Modularize Terraform: For complex setups, split your Terraform code into logical modules (e.g., eks-observability, pagerduty-incidents) to improve maintainability and reusability.
Tagging Strategy: Implement a consistent tagging strategy across AWS resources, Kubernetes objects, and Datadog to enable powerful filtering, correlation, and cost analysis.
Granular Monitors: While this guide shows basic monitors, create more granular alerts for specific applications, namespaces, and services crucial to your business. Leverage Datadog's anomaly detection and forecast monitors.
Test Regularly: Periodically test your entire observability and incident response pipeline, especially after major changes to your EKS cluster or monitoring configuration.
Runbooks: For each PagerDuty service, attach clear runbooks that guide your on-call team through initial triage, troubleshooting steps, and escalation paths.

Conclusion

Automating your AWS EKS observability and incident management with Terraform, Datadog, and PagerDuty provides a powerful, scalable, and reliable foundation for operating critical Kubernetes workloads. By treating your observability stack as code, you gain consistency, reduce human error, and accelerate your team's ability to detect, diagnose, and resolve issues. This integration ensures that your EKS clusters are not only highly observable but also backed by an efficient and proactive incident response mechanism, keeping your services performant and your customers happy.

CloudStack YAML