Datadog & PagerDuty for AWS EKS: Terraform-Automated Observability & Incident Response

In the dynamic world of cloud-native applications, maintaining robust observability and a streamlined incident response mechanism is paramount. AWS Elastic Kubernetes Service (EKS) provides a powerful platform for orchestrating containerized workloads, but it demands sophisticated monitoring and alerting to ensure high availability and performance. This guide delves into integrating Datadog for comprehensive observability and PagerDuty for efficient incident management, all automated through Terraform Infrastructure as Code (IaC).

Architecture Pro-Tip: Layered Observability Strategy

When designing your EKS observability, aim for a layered approach:

Infrastructure Layer: Monitor EKS control plane logs (CloudWatch), worker node metrics (EC2), and network performance.
Kubernetes Layer: Track Pod, Deployment, Service, and Node health; API server latency; scheduler and controller manager status.
Application Layer: Instrument your applications with custom metrics, logs, and traces (OpenTelemetry/APM).
Security Layer: Monitor EKS audit logs, network policies, and container image vulnerabilities.

Datadog excels at consolidating these layers, while PagerDuty ensures the right teams are alerted for critical issues.

Why Datadog & PagerDuty for AWS EKS?

The combination of Datadog and PagerDuty provides a powerful synergy for managing complex EKS environments:

Datadog: A unified observability platform offering end-to-end visibility into your EKS clusters, applications, and underlying AWS infrastructure. It collects metrics, logs, and traces, enabling advanced monitoring, alerting, and performance analysis. Its native Kubernetes integration simplifies agent deployment and data collection from Pods, Deployments, and Nodes.
PagerDuty: A leading incident management platform that transforms Datadog alerts into actionable incidents. It orchestrates on-call schedules, escalation policies, and automated workflows to ensure critical issues are routed to the right team members immediately, minimizing downtime and accelerating resolution.
Terraform: Automates the entire setup, from deploying the Datadog Agent on EKS to configuring monitors, PagerDuty services, and integration points. This ensures consistency, repeatability, and version control for your observability and incident response configurations.

Prerequisites

Before you begin, ensure you have the following:

An active AWS Account with administrative access.
An existing AWS EKS Cluster.
A Datadog Account with API and Application Keys.
A PagerDuty Account with a Global API Token.
Terraform CLI (v1.0+ recommended) installed and configured.
kubectl and helm CLIs installed and configured to interact with your EKS cluster.

Core Concepts and Integration Flow

The integration follows this general flow:

Datadog Agent Deployment: The Datadog Agent, typically deployed as a DaemonSet on EKS, collects metrics, logs, and traces from your cluster nodes, pods, and applications.
PagerDuty Service Configuration: In PagerDuty, you define services that represent components or applications, along with escalation policies to dictate who gets alerted and when.
Datadog PagerDuty Integration: A one-time setup in Datadog connects it to your PagerDuty account, allowing Datadog monitors to trigger PagerDuty incidents.
Datadog Monitor Creation: You define specific conditions (monitors) in Datadog that, when violated, generate alerts. These alerts can be configured to notify PagerDuty.
Terraform Automation: All of the above (Agent deployment, PagerDuty service, Datadog integration, and monitors) are provisioned and managed declaratively using Terraform.

Step-by-Step Terraform Automation

1. Configure Terraform Providers

First, set up your Terraform providers for AWS, Datadog, PagerDuty, and Helm.

Create a providers.tf file:

provider "aws" {
  region = "us-east-1" # Or your desired AWS region
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.this.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.this.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.this.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

data "aws_eks_cluster" "this" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "this" {
  name = var.eks_cluster_name
}
    

Define your variables in variables.tf:

variable "eks_cluster_name" {
  description = "The name of your EKS cluster"
  type        = string
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty Global API Token"
  type        = string
  sensitive   = true
}
    

2. Deploy Datadog Agent to EKS

Use the Helm provider to deploy the Datadog Agent as a DaemonSet to your EKS cluster. This will automatically collect metrics, logs, and traces.

Create a datadog_agent.tf file:

resource "helm_release" "datadog_agent" {
  name        = "datadog"
  namespace   = "datadog"
  repository  = "https://helm.datadoghq.com"
  chart       = "datadog"
  version     = "2.33.0" # Use a stable, recent version
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "datadog.site"
    value = "datadoghq.com" # Or your Datadog site (e.g., eu.datadoghq.com)
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
  }

  set {
    name  = "logAgent.enabled"
    value = "true"
  }

  set {
    name  = "processAgent.enabled"
    value = "true"
  }

  set {
    name  = "apm.enabled"
    value = "true"
  }

  # Add any EKS-specific configurations here, e.g., enabling Fargate, adding tags
  set {
    name = "tags[0]"
    value = "environment:production"
  }
  set {
    name = "tags[1]"
    value = "cluster:${var.eks_cluster_name}"
  }
}
    

3. Configure PagerDuty Service and Escalation Policy

Define a PagerDuty service that will receive incidents triggered by Datadog, along with an escalation policy.

Create a pagerduty.tf file:

resource "pagerduty_user" "devops_engineer" {
  name  = "DevOps Engineer"
  email = "devops@example.com"
  # Optional: time_zone, role, etc.
}

resource "pagerduty_team" "devops_team" {
  name        = "DevOps Team"
  description = "Team responsible for EKS infrastructure and applications."
}

resource "pagerduty_team_membership" "devops_member" {
  user_id = pagerduty_user.devops_engineer.id
  team_id = pagerduty_team.devops_team.id
  role    = "observer" # Can be 'observer', 'responder', or 'manager'
}

resource "pagerduty_schedule" "primary_on_call" {
  name      = "Primary On-Call Schedule"
  time_zone = "America/New_York"
  layer {
    name  = "Daily Rotation"
    start = "2023-01-01T09:00:00-05:00" # Start date for the rotation
    users = [pagerduty_user.devops_engineer.id]
    rotation_type          = "daily"
    rotation_virtual_start = "2023-01-01T09:00:00-05:00"
    id                     = "P012345" # Placeholder, PagerDuty generates this
  }
  team_id = pagerduty_team.devops_team.id
}

resource "pagerduty_escalation_policy" "eks_critical_policy" {
  name      = "EKS Critical Incident Policy"
  num_loops = 2
  team_id   = pagerduty_team.devops_team.id

  rule {
    escalation_delay_in_minutes = 5
    target {
      type = "user"
      id   = pagerduty_user.devops_engineer.id
    }
  }

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "schedule"
      id   = pagerduty_schedule.primary_on_call.id
    }
  }
}

resource "pagerduty_service" "eks_cluster_service" {
  name                       = "${var.eks_cluster_name}-Service"
  auto_resolve_timeout_minutes = 60
  acknowledgement_timeout_minutes = 30
  escalation_policy          = pagerduty_escalation_policy.eks_critical_policy.id
  description                = "Service for monitoring AWS EKS cluster: ${var.eks_cluster_name}"
  team                       = pagerduty_team.devops_team.id
}
    

4. Connect Datadog to PagerDuty

This step involves setting up the Datadog-PagerDuty integration within Datadog, which Terraform can also manage.

Create a datadog_pagerduty_integration.tf file:

resource "datadog_integration_pagerduty" "pagerduty_integration" {
  api_token = var.pagerduty_api_token
  # Optionally, you can specify `services` to link specific PagerDuty services by name,
  # but often it's configured directly in the monitor.
}
    

5. Create Datadog Monitors with PagerDuty Notifications

Now, define Datadog monitors that will trigger incidents in PagerDuty when alert conditions are met. We'll create an example monitor for EKS node readiness.

Create a datadog_monitors.tf file:

resource "datadog_monitor" "eks_node_not_ready" {
  name               = "EKS Node Not Ready - ${var.eks_cluster_name}"
  type               = "query alert"
  query              = "sum(last_5m):kubernetes.node.ready{cluster_name:${var.eks_cluster_name},kube_status:not_ready} > 0"
  message            = <

Ready-to-Use Configuration SummaryHere's a consolidated view of the essential Terraform configuration for quick deployment. Remember to replace placeholder values and sensitive data securely (e.g., using Terraform Cloud variables or AWS Secrets Manager).

# main.tf (or combine into various .tf files as shown above)

# --- Variables (variables.tf) ---
variable "eks_cluster_name" {
  description = "The name of your EKS cluster"
  type        = string
  default     = "my-prod-eks-cluster" # CHANGE ME
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty Global API Token"
  type        = string
  sensitive   = true
}

# --- Providers and Data Sources (providers.tf) ---
provider "aws" {
  region = "us-east-1"
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.this.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.this.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.this.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

data "aws_eks_cluster" "this" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "this" {
  name = var.eks_cluster_name
}

# --- Datadog Agent Deployment (datadog_agent.tf) ---
resource "helm_release" "datadog_agent" {
  name        = "datadog"
  namespace   = "datadog"
  repository  = "https://helm.datadoghq.com"
  chart       = "datadog"
  version     = "2.33.0" # Always use a specific version
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "datadog.site"
    value = "datadoghq.com"
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
  }
  set {
    name  = "logAgent.enabled"
    value = "true"
  }
  set {
    name  = "apm.enabled"
    value = "true"
  }
  set {
    name = "tags[0]"
    value = "environment:production"
  }
  set {
    name = "tags[1]"
    value = "cluster:${var.eks_cluster_name}"
  }
}

# --- PagerDuty Configuration (pagerduty.tf) ---
resource "pagerduty_user" "devops_engineer" {
  name  = "John Doe" # CHANGE ME
  email = "john.doe@example.com" # CHANGE ME
}

resource "pagerduty_team" "devops_team" {
  name        = "EKS Platform Team" # CHANGE ME
  description = "Team responsible for EKS cluster operations."
}

resource "pagerduty_team_membership" "devops_member" {
  user_id = pagerduty_user.devops_engineer.id
  team_id = pagerduty_team.devops_team.id
  role    = "responder"
}

resource "pagerduty_schedule" "primary_on_call" {
  name      = "${var.eks_cluster_name} On-Call Schedule"
  time_zone = "America/New_York" # CHANGE ME
  layer {
    name  = "Primary Shift"
    start = "2023-01-01T09:00:00-05:00"
    users = [pagerduty_user.devops_engineer.id]
    rotation_type          = "weekly" # or daily, monthly
    rotation_virtual_start = "2023-01-01T09:00:00-05:00"
    # id is auto-generated by PagerDuty
  }
  team_id = pagerduty_team.devops_team.id
}

resource "pagerduty_escalation_policy" "eks_critical_policy" {
  name      = "${var.eks_cluster_name} Critical Incident Policy"
  num_loops = 1
  team_id   = pagerduty_team.devops_team.id

  rule {
    escalation_delay_in_minutes = 5
    target {
      type = "schedule"
      id   = pagerduty_schedule.primary_on_call.id
    }
  }
}

resource "pagerduty_service" "eks_cluster_service" {
  name                       = "${var.eks_cluster_name}-EKS-Service"
  auto_resolve_timeout_minutes = 60
  acknowledgement_timeout_minutes = 30
  escalation_policy          = pagerduty_escalation_policy.eks_critical_policy.id
  description                = "Incident service for AWS EKS cluster: ${var.eks_cluster_name}"
  team                       = pagerduty_team.devops_team.id
}

# --- Datadog PagerDuty Integration (datadog_pagerduty_integration.tf) ---
resource "datadog_integration_pagerduty" "pagerduty_integration" {
  api_token = var.pagerduty_api_token
}

# --- Datadog Monitors (datadog_monitors.tf) ---
resource "datadog_monitor" "eks_node_not_ready" {
  name               = "EKS Node Not Ready - ${var.eks_cluster_name}"
  type               = "query alert"
  query              = "sum(last_5m):kubernetes.node.ready{cluster_name:${var.eks_cluster_name},kube_status:not_ready} > 0"
  message            = <Running TerraformOnce your Terraform files are set up:
Initialize Terraform: terraform init
Plan the changes: terraform plan -var="datadog_api_key=YOUR_DD_API_KEY" -var="datadog_app_key=YOUR_DD_APP_KEY" -var="pagerduty_api_token=YOUR_PD_API_TOKEN" (or use environment variables/tfvars files for sensitive data)
Apply the configuration: terraform apply -var="datadog_api_key=..." -var="datadog_app_key=..." -var="pagerduty_api_token=..."
Testing and ValidationAfter applying your Terraform configuration, it's crucial to validate the setup:
Datadog Agent: Check your Datadog UI under "Infrastructure" > "Container" to ensure EKS nodes and pods are reporting data. Verify logs and traces are flowing.
PagerDuty Service: Log into PagerDuty and confirm the new service, escalation policy, and team are created as expected.
Datadog Monitors: In Datadog, navigate to "Monitors" > "Manage Monitors" and verify your new monitors are listed and configured with PagerDuty as a notification channel.
Trigger a Test Alert: To thoroughly test the integration, you could temporarily cordon and drain an EKS node (e.g., kubectl cordon <node-name>) to simulate a "Not Ready" state and observe if a PagerDuty incident is created. Remember to uncordon/drain afterward.
Best Practices for Production EKS ObservabilityTagging Strategy: Implement a consistent tagging strategy across AWS resources, Kubernetes objects, and Datadog/PagerDuty entities (e.g., env:prod, service:frontend, team:devops). This enhances filtering, cost attribution, and incident routing.
Granular Monitors: Beyond basic health checks, create monitors for application-specific metrics, API latency, error rates, and resource saturation. Use Datadog's anomaly detection and forecast monitors.
Runbooks & Automation: Link PagerDuty services to detailed runbooks (e.g., in Confluence or a Git repo) that guide responders through diagnosis and remediation. Consider integrating PagerDuty with automation tools for self-healing.
SLOs & SLIs: Define Service Level Objectives (SLOs) and Service Level Indicators (SLIs) for your critical EKS services and use Datadog to track them, triggering PagerDuty alerts when SLOs are at risk.
Cost Management: Monitor Datadog agent resource consumption. Be mindful of data ingest costs; optimize log collection and metric cardinality.
Security: Follow least privilege for Datadog API keys and PagerDuty tokens. Use Kubernetes RBAC for the Datadog Agent.


Troubleshooting Common IssuesDatadog Agent not reporting:
            Check kubectl get pods -n datadog for running agents.
Examine agent logs: kubectl logs <datadog-agent-pod> -n datadog. Look for API key errors or connectivity issues.
Verify Network Policy: Ensure Datadog agents can reach the Datadog API endpoints (*.datadoghq.com).

        
PagerDuty incidents not triggering:
            Confirm the Datadog monitor's message content includes @pagerduty-<YOUR_PAGERDUTY_SERVICE_NAME>. The name must exactly match the service name in PagerDuty.
Check Datadog's "Integrations" > "PagerDuty" page for any error messages or misconfigurations.
Verify the PagerDuty API token used for the Datadog integration is valid and has the necessary permissions.
Ensure the monitor's alert conditions are actually being met and are not flapping too quickly.

        
Terraform provider errors:
            Double-check your API keys and tokens for typos or expiration.
Ensure your AWS credentials have permissions to describe EKS clusters and other necessary resources.
For Kubernetes/Helm providers, verify kubectl context is correct and you have access to the EKS cluster.

        
ConclusionAutomating the integration of Datadog and PagerDuty for AWS EKS using Terraform provides a robust, scalable, and auditable solution for maintaining high availability and rapid incident response. By embracing Infrastructure as Code for your observability and incident management, you empower your DevOps teams to operate with confidence, reduce manual errors, and focus on delivering value instead of fighting fires. Continuously refine your monitors, escalation policies, and runbooks to adapt to the evolving needs of your EKS workloads and keep your applications running smoothly.

CloudStack YAML