Terraform AWS EKS Observability: Datadog APM and PagerDuty Incident Response

In the dynamic landscape of cloud-native applications, maintaining robust observability and swift incident response for Kubernetes clusters is paramount. AWS Elastic Kubernetes Service (EKS) provides a managed control plane, but the responsibility for application monitoring, performance analytics, and operational alerting often falls to third-party solutions. This guide demonstrates how to architect a comprehensive observability stack for AWS EKS using Terraform for infrastructure as code (IaC), integrating Datadog for Application Performance Monitoring (APM) and full-stack visibility, and PagerDuty for streamlined incident management.

Architecture Pro-Tip: Always treat your observability stack as a first-class citizen in your infrastructure. Deploy monitoring agents, configure dashboards, and define alerting policies alongside your application deployment via IaC. This ensures consistency, repeatability, and immediate visibility upon deployment, preventing critical blind spots in production environments.

Why Terraform, Datadog, and PagerDuty for EKS Observability?

Terraform: Enables reproducible, version-controlled provisioning and management of EKS clusters, Datadog agents, and PagerDuty services. It enforces desired state configuration, reducing manual errors and accelerating deployments.
Datadog APM: Offers deep visibility into the performance of applications running on EKS. It provides distributed tracing, service maps, real user monitoring, and infrastructure metrics, all within a unified platform. Its Kubernetes integration is exceptional, automatically collecting metrics, logs, and traces from pods, nodes, and the control plane.
PagerDuty: Transforms monitoring alerts into actionable incidents, ensuring the right people are notified at the right time. Its on-call scheduling, escalation policies, and seamless integrations with monitoring tools like Datadog are critical for effective incident response and SRE practices.

Prerequisites

Before diving into the configuration, ensure you have the following:

An AWS account with appropriate permissions to create EKS clusters, EC2 instances, and IAM roles.
Terraform CLI installed (v1.0+ recommended).
AWS CLI configured with credentials.
A Datadog account with API and Application keys.
A PagerDuty account with an integration key or API token.
Basic understanding of AWS EKS, Kubernetes, Terraform, Datadog, and PagerDuty.

Terraform Setup for EKS Cluster

First, let's establish our EKS cluster using Terraform. This typically involves defining a VPC, subnets, security groups, and the EKS cluster itself, along with node groups.

A simplified `main.tf` for an EKS cluster might look like this:

resource "aws_vpc" "eks_vpc" {
  cidr_block = "10.0.0.0/16"
  tags = { Name = "eks-observability-vpc" }
}

resource "aws_subnet" "eks_public_subnets" {
  count             = 2
  vpc_id            = aws_vpc.eks_vpc.id
  cidr_block        = "10.0.${count.index}.0/24"
  availability_zone = data.aws_availability_zones.available.names[count.index]
  map_public_ip_on_launch = true
  tags = { Name = "eks-public-subnet-${count.index}" }
}

resource "aws_eks_cluster" "main" {
  name     = "observability-cluster"
  role_arn = aws_iam_role.eks_master.arn
  vpc_config {
    subnet_ids = aws_subnet.eks_public_subnets[*].id
  }
  depends_on = [
    aws_iam_role_policy_attachment.eks_cluster_policy_attachment,
    aws_iam_role_policy_attachment.eks_service_policy_attachment,
  ]
}

resource "aws_eks_node_group" "main" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "observability-ng"
  node_role_arn   = aws_iam_role.eks_nodes.arn
  subnet_ids      = aws_subnet.eks_public_subnets[*].id
  instance_types  = ["t3.medium"]
  scaling_config {
    desired_size = 2
    max_size     = 3
    min_size     = 1
  }
  depends_on = [
    aws_iam_role_policy_attachment.eks_worker_node_policy_attachment,
    aws_iam_role_policy_attachment.eks_cni_policy_attachment,
    aws_iam_role_policy_attachment.eks_ec2_container_registry_readonly_policy_attachment,
  ]
}

# ... IAM Roles and Policies definition ...
    

After applying this, you'll have a functional EKS cluster ready for application deployments and observability tooling.

Integrating Datadog APM with Terraform

Datadog's integration with Kubernetes is typically achieved by deploying the Datadog Agent as a DaemonSet. This ensures an agent runs on every node, collecting metrics, logs, and traces. We'll use the Terraform Kubernetes provider to deploy the Datadog Agent Helm chart.

Configure Kubernetes Provider

To interact with your EKS cluster, Terraform needs the Kubernetes and Helm providers configured.

provider "kubernetes" {
  host                   = aws_eks_cluster.main.endpoint
  cluster_ca_certificate = base64decode(aws_eks_cluster.main.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.main.token
}

provider "helm" {
  kubernetes {
    host                   = aws_eks_cluster.main.endpoint
    cluster_ca_certificate = base64decode(aws_eks_cluster.main.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.main.token
  }
}

data "aws_eks_cluster_auth" "main" {
  name = aws_eks_cluster.main.name
}
    

Deploy Datadog Agent via Helm Chart

The Datadog Agent Helm chart provides comprehensive configuration options. You'll need your Datadog API Key.

Terraform Code for Datadog Agent and Monitor

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "datadog.site"
    value = "datadoghq.com" # or eu.datadoghq.com, us3.datadoghq.com etc.
  }

  set {
    name  = "datadog.apm.enabled"
    value = "true"
  }

  set {
    name  = "datadog.apm.hostPortEnabled"
    value = "true"
  }

  set {
    name  = "datadog.logs.enabled"
    value = "true"
  }

  set {
    name  = "datadog.logs.containerCollectAll"
    value = "true"
  }

  set {
    name  = "datadog.processAgent.enabled"
    value = "true"
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "clusterAgent.metricsProvider.enabled"
    value = "true"
  }

  depends_on = [
    aws_eks_cluster.main # Ensure EKS is up before deploying agent
  ]
}

resource "datadog_monitor" "high_cpu_usage" {
  name                = "EKS Cluster High CPU Usage"
  type                = "metric alert"
  message             = "EKS node {{host.name}} CPU usage is high! @pagerduty-{{var.pagerduty_service_name}}"
  query               = "avg(last_5m):avg:system.cpu.idle{cluster_name:observability-cluster} by {host} < 10" # Alert if idle CPU is less than 10%
  monitor_threshold_windows {
    recovery_window = "10m"
    trigger_window  = "5m"
  }
  thresholds {
    warning          = 20
    warning_recovery = 15
    critical         = 10
    critical_recovery = 5
  }
  renotify_interval = 60
  tags = ["env:production", "service:eks", "team:devops"]
}
    

Remember to define `datadog_api_key` and `datadog_app_key` as Terraform variables and pass them securely.

For APM, your applications will also need to be instrumented with Datadog APM libraries (e.g., `dd-trace-java`, `dd-trace-python`). The agent deployed above will collect traces from these instrumented services.

PagerDuty Incident Response with Terraform

Integrating PagerDuty allows critical Datadog alerts to be escalated into actionable incidents. This section shows how to create a PagerDuty service and link it to Datadog using Terraform.

Configure PagerDuty Provider

You'll need your PagerDuty API Token to configure the provider.

provider "pagerduty" {
  token = var.pagerduty_api_token
}
    

Create PagerDuty Service and Integration

We'll create a PagerDuty service and an integration specific for Datadog. This integration will provide an integration key for Datadog.

resource "pagerduty_team" "devops" {
  name = "DevOps Team"
}

resource "pagerduty_user" "oncall_user" {
  name  = "Oncall Engineer"
  email = "oncall@example.com"
}

resource "pagerduty_schedule" "devops_schedule" {
  name      = "DevOps Oncall Schedule"
  time_zone = "America/New_York"
  layer {
    name      = "Primary Oncall"
    start     = "2023-01-01T09:00:00-05:00" # Example start time
    rotation_turn_length_seconds = 604800 # 1 week
    users     = [pagerduty_user.oncall_user.id]
  }
}

resource "pagerduty_escalation_policy" "devops_policy" {
  name      = "DevOps Escalation Policy"
  team      = pagerduty_team.devops.id
  num_loops = 2
  rule {
    delay_after_incident_minutes = 0
    target {
      type = "user_reference"
      id   = pagerduty_user.oncall_user.id
    }
  }
}

resource "pagerduty_service" "eks_observability_service" {
  name                    = var.pagerduty_service_name
  auto_resolve_timeout    = 14400 # 4 hours
  acknowledgement_timeout = 600   # 10 minutes
  escalation_policy       = pagerduty_escalation_policy.devops_policy.id
  description             = "Service for EKS Observability alerts from Datadog"
}

resource "pagerduty_service_integration" "datadog_integration" {
  name    = "Datadog Integration"
  type    = "datadog_inbound_integration" # Use datadog_inbound_integration for Datadog
  service = pagerduty_service.eks_observability_service.id
}
    

The key output here is `pagerduty_service_integration.datadog_integration.integration_key`. This key is used in Datadog monitor notifications to send alerts to this specific PagerDuty service.

Connecting Datadog to PagerDuty

With the PagerDuty integration key, you can update your Datadog monitors to notify PagerDuty. The message in the `datadog_monitor` resource should include `@pagerduty-{{SERVICE_NAME}}` where `SERVICE_NAME` is the name of your PagerDuty service (e.g., `eks-observability-service`). Datadog automatically discovers PagerDuty services configured with `datadog_inbound_integration` and allows you to reference them.

Testing and Validation

After applying your Terraform configuration:

Verify Datadog Agent: Check your Datadog account under "Infrastructure" -> "Hosts" to ensure EKS nodes are reporting. Go to "Integrations" -> "Kubernetes" and "EKS" dashboards for cluster-wide metrics.
Verify APM: Deploy a sample application with Datadog APM instrumentation (e.g., a simple web service). Send some requests to it and then check "APM" -> "Services" in Datadog to see traces and service maps.
Test Datadog Monitor: Manually trigger an alert by simulating high CPU usage (e.g., running a CPU-intensive process on an EKS node) or create a low-threshold test monitor.
Verify PagerDuty Incident: Upon Datadog alert, a new incident should appear in your designated PagerDuty service, triggering notifications based on your escalation policy.

Troubleshooting and Best Practices

Common Issues:

Datadog Agent not reporting: Check Kubernetes logs for the Datadog Agent pods (`kubectl logs -n datadog -l app=datadog`) for API key errors or connectivity issues. Ensure correct IAM permissions for the EKS nodes to pull container images.
PagerDuty incidents not firing: Verify the `@pagerduty-{{SERVICE_NAME}}` syntax in your Datadog monitor message. Double-check that the PagerDuty service integration key is correct and that the integration type in PagerDuty is set to Datadog.
Terraform plan/apply errors: Ensure all required variables are passed. Check AWS IAM roles and policies for EKS.

Best Practices:

Secrets Management: Use AWS Secrets Manager or HashiCorp Vault to store Datadog API/App keys and PagerDuty API tokens, referencing them securely in Terraform.
Granular Monitors: Create specific Datadog monitors for different aspects (CPU, memory, disk I/O, application errors, latency) and severity levels.
Dedicated PagerDuty Services: Consider creating separate PagerDuty services for different teams or critical application components, each with its own escalation policy.
Cost Optimization: Monitor Datadog usage and optimize agent configurations to collect only necessary metrics, logs, and traces to manage costs.
Infrastructure Testing: Incorporate automated tests for your Terraform configurations to validate resource creation and connectivity.

Conclusion

By leveraging Terraform, Datadog APM, and PagerDuty, you can build a robust, automated, and highly observable AWS EKS environment. This integrated approach ensures deep visibility into your applications and infrastructure, coupled with an efficient incident response mechanism, significantly reducing MTTR (Mean Time To Resolution) and enhancing overall system reliability. Embracing Infrastructure as Code for your entire observability stack is key to maintaining consistency and agility in modern cloud operations.

CloudStack YAML