Terraform for AWS EKS: Integrating Datadog Observability and PagerDuty Incident Management

In the dynamic landscape of cloud-native applications, maintaining robust observability and efficient incident management for your Kubernetes clusters is paramount. This comprehensive guide details how to leverage Terraform to provision and manage an AWS EKS cluster, seamlessly integrating Datadog for deep observability and PagerDuty for streamlined incident response. By codifying your infrastructure and monitoring setup, you achieve consistency, repeatability, and a faster mean time to resolution (MTTR).

Architecture Pro-Tip: Always treat your observability and incident management configurations as code. Storing Datadog monitors, PagerDuty services, and integration settings within your Terraform repository alongside your infrastructure ensures version control, peer review, and automated deployment, significantly reducing configuration drift and manual errors.

Prerequisites

Before we dive into the configurations, ensure you have the following:

An AWS account with appropriate IAM permissions.
Terraform CLI (v1.0.0+) installed.
AWS CLI installed and configured.
A Datadog account (with API and Application Keys).
A PagerDuty account (with an API Token and a Service API Key for integration).
Basic understanding of AWS EKS, Kubernetes, Terraform, Datadog, and PagerDuty.

Setting Up Your AWS EKS Cluster with Terraform

We'll use the official terraform-aws-modules/eks/aws module for simplicity and best practices. This module abstracts away much of the complexity of setting up an EKS cluster, including VPC, subnets, node groups, and associated IAM roles.

Core EKS Cluster Configuration (`main.tf`)

First, define your AWS provider and an EKS cluster resource. For brevity, we'll show a simplified setup. In a production environment, you'd customize VPC, subnets, and node group configurations extensively.

provider "aws" {
  region = "us-east-1"
}

resource "aws_vpc" "eks_vpc" {
  cidr_block = "10.0.0.0/16"
  tags = {
    Name = "eks-datadog-pagerduty-vpc"
  }
}

resource "aws_subnet" "public_subnets" {
  count             = 2
  vpc_id            = aws_vpc.eks_vpc.id
  cidr_block        = cidrsubnet(aws_vpc.eks_vpc.cidr_block, 8, count.index)
  availability_zone = data.aws_availability_zones.available.names[count.index]
  tags = {
    Name = "eks-public-subnet-${count.index}"
    "kubernetes.io/role/elb" = "1"
  }
}

resource "aws_subnet" "private_subnets" {
  count             = 2
  vpc_id            = aws_vpc.eks_vpc.id
  cidr_block        = cidrsubnet(aws_vpc.eks_vpc.cidr_block, 8, count.index + 2)
  availability_zone = data.aws_availability_zones.available.names[count.index]
  tags = {
    Name = "eks-private-subnet-${count.index}"
    "kubernetes.io/role/internal-elb" = "1"
    "kubernetes.io/cluster/${local.cluster_name}" = "owned"
  }
}

data "aws_availability_zones" "available" {}

locals {
  cluster_name = "datadog-pagerduty-eks"
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0"

  cluster_name    = local.cluster_name
  cluster_version = "1.28"

  vpc_id                   = aws_vpc.eks_vpc.id
  subnet_ids               = concat(aws_subnet.public_subnets[*].id, aws_subnet.private_subnets[*].id)
  control_plane_subnet_ids = aws_subnet.private_subnets[*].id # Place control plane in private subnets

  # EKS Managed Node Group
  eks_managed_node_groups = {
    default = {
      instance_types = ["t3.medium"]
      min_size       = 2
      max_size       = 4
      desired_size   = 2
      disk_size      = 20
    }
  }

  tags = {
    Project     = "EKS Observability"
    Environment = "Dev"
  }
}

output "kubeconfig" {
  description = "Kubernetes config file for EKS cluster"
  value       = module.eks.kubeconfig
  sensitive   = true
}

output "cluster_endpoint" {
  description = "Endpoint for EKS Control Plane"
  value       = module.eks.cluster_endpoint
}

Integrating Datadog Observability

Datadog provides comprehensive monitoring for your EKS clusters, including metrics, logs, traces, and events from Kubernetes components, applications, and AWS services.

Securely Storing Datadog API Keys

Never hardcode sensitive information like API keys. Use a secrets manager like AWS Secrets Manager or HashiCorp Vault. For this guide, we'll reference them as Terraform variables, assuming they are provided securely (e.g., via environment variables or a `terraform.tfvars` file that is `.gitignore`d).

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

Deploying the Datadog Agent to EKS

The Datadog Agent runs as a DaemonSet on your EKS worker nodes, collecting telemetry data. The easiest way to deploy it is via its official Helm chart using Terraform's Helm provider.

Terraform Configuration for Datadog Agent (Helm Provider)

First, configure the Kubernetes and Helm providers. The Kubernetes provider needs to know how to connect to your EKS cluster, which can be done using the output from our `eks` module.

# Configure Kubernetes Provider
provider "kubernetes" {
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  token                  = module.eks.cluster_id_token
}

# Configure Helm Provider
provider "helm" {
  kubernetes {
    host                   = module.eks.cluster_endpoint
    cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
    token                  = module.eks.cluster_id_token
  }
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  # Enable APM, Log Collection, and Network Performance Monitoring
  set {
    name  = "apm.enabled"
    value = "true"
  }
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }
  set {
    name  = "networkMonitoring.enabled"
    value = "true"
  }
  set {
    name  = "processAgent.enabled"
    value = "true"
  }
  set {
    name  = "kubeStateMetricsExternal.enabled"
    value = "true"
  }

  # Ensure Datadog Agent is deployed after EKS cluster is fully ready
  depends_on = [module.eks]
}

Configuring Datadog Monitors with Terraform

The Datadog provider for Terraform allows you to manage dashboards, monitors, and other Datadog resources as code. This ensures your alerting strategy is version-controlled and deployed consistently.

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

resource "datadog_monitor" "eks_node_cpu_utilization" {
  name               = "[EKS] High Node CPU Utilization on {{kube_cluster_name.name}}"
  type               = "query alert"
  query              = "avg(last_5m):avg:kubernetes.cpu.usage.total{kube_cluster_name:${local.cluster_name}} by {host} > 80"
  message            = "CPU utilization on host {{host.name}} is {{value}}% higher than 80%. Consider scaling or investigating processes."
  escalation_message = "CPU utilization remains high on {{host.name}}. Paging on-call."
  tags               = ["environment:dev", "team:devops", "kubernetes", "eks", "cpu"]

  # Alert if the threshold is breached for at least 5 minutes
  monitor_threshold_windows {
    recovery_window = "10m"
  }

  # Thresholds for alert, warning, and no data
  monitor_thresholds {
    critical = 80
    warning  = 70
  }

  # Notify PagerDuty service via integration (configured later)
  renotify_interval = 60
  notify_no_data    = false
  new_host_delay    = 300
  no_data_timeframe = 20
  include_tags      = true

  # Assuming a PagerDuty integration is set up in Datadog
  # You might reference an integration name directly here or use more advanced notification channels
  # This field will be updated once PagerDuty integration is setup.
  # For now, we use a placeholder or generic notification.
  # Example: notify_list = ["@pagerduty-eks-service"]
}

Integrating PagerDuty Incident Management

PagerDuty ensures that critical alerts from Datadog are routed to the right people at the right time, minimizing downtime and improving incident response efficiency.

Defining PagerDuty Service and Integration Keys

We'll use the PagerDuty Terraform provider to create a service and an integration directly. This allows Datadog to send events to a specific PagerDuty service.

variable "pagerduty_api_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

resource "pagerduty_user" "devops_user" {
  name  = "DevOps On-Call"
  email = "devops-oncall@example.com"
  # Add more fields like job_title, teams etc. as needed
}

resource "pagerduty_escalation_policy" "devops_policy" {
  name = "DevOps Primary Escalation Policy"
  rule {
    delay    = 0
    target_id = pagerduty_user.devops_user.id
    target_type = "user"
  }
}

resource "pagerduty_service" "eks_monitoring_service" {
  name                    = "${local.cluster_name}-Monitoring"
  auto_resolve_timeout    = "14400" # 4 hours
  acknowledgement_timeout = "600"   # 10 minutes
  escalation_policy       = pagerduty_escalation_policy.devops_policy.id
  description             = "Monitors EKS cluster: ${local.cluster_name} via Datadog"
}

resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog Integration"
  type        = "datadog_inbound_integration" # Use datadog_inbound_integration type
  service     = pagerduty_service.eks_monitoring_service.id
}

Connecting Datadog Monitors to PagerDuty

Now that you have a PagerDuty service and a Datadog integration, you can update your Datadog monitor to send alerts to this specific PagerDuty service. Datadog identifies PagerDuty integrations by name. You can use the `datadog_integration_pagerduty` resource to manage Datadog's PagerDuty integration details. Or, more simply, reference the service in the monitor's message.

In the `datadog_monitor` resource, modify the `message` to include PagerDuty notification:

resource "datadog_monitor" "eks_node_cpu_utilization" {
  name               = "[EKS] High Node CPU Utilization on {{kube_cluster_name.name}}"
  type               = "query alert"
  query              = "avg(last_5m):avg:kubernetes.cpu.usage.total{kube_cluster_name:${local.cluster_name}} by {host} > 80"
  message            = "@pagerduty-${pagerduty_service.eks_monitoring_service.name} - CPU utilization on host {{host.name}} is {{value}}% higher than 80%. Consider scaling or investigating processes."
  escalation_message = "@pagerduty-${pagerduty_service.eks_monitoring_service.name} - CPU utilization remains high on {{host.name}}. Paging on-call."
  tags               = ["environment:dev", "team:devops", "kubernetes", "eks", "cpu"]

  monitor_threshold_windows {
    recovery_window = "10m"
  }

  monitor_thresholds {
    critical = 80
    warning  = 70
  }

  renotify_interval = 60
  notify_no_data    = false
  new_host_delay    = 300
  no_data_timeframe = 20
  include_tags      = true

  # Ensure the monitor is created after the PagerDuty integration is available
  depends_on = [pagerduty_service_integration.datadog_integration]
}

End-to-End Workflow and Best Practices

A Unified Observability and Incident Response Workflow

With this setup, your workflow looks like this:

Terraform provisions the EKS cluster, deploys the Datadog Agent, and configures Datadog monitors and PagerDuty services.
Datadog Agent collects metrics, logs, and traces from your EKS cluster and applications.
Datadog monitors continuously evaluate these telemetry streams against predefined thresholds.
When a threshold is breached (e.g., high CPU utilization), the Datadog monitor triggers an alert.
The alert sends a notification to the PagerDuty integration, which then creates an incident in the designated PagerDuty service.
PagerDuty's escalation policies ensure the incident reaches the correct on-call team based on schedules and rotations.
On-call engineers can then use Datadog to investigate the root cause, leveraging its dashboards, logs, and trace explorers.

Best Practices for Production Environments

Secrets Management: Always use AWS Secrets Manager, HashiCorp Vault, or similar for API keys and sensitive data. Never commit them to version control.
Module Reusability: Create dedicated Terraform modules for Datadog agents, monitors, and PagerDuty services to promote reusability across multiple EKS clusters or environments.
Tagging: Implement a consistent tagging strategy across all AWS, Kubernetes, and Datadog resources for better cost allocation, filtering, and organization.
Alerting Thresholds: Continuously review and fine-tune your Datadog monitor thresholds to reduce alert fatigue and ensure actionable alerts.
IAM Least Privilege: Ensure the IAM roles used by Terraform, EKS, and the Datadog Agent adhere to the principle of least privilege.
Version Control: Store all your Terraform code in a Git repository, implement pull request reviews, and automate deployments via CI/CD pipelines.
Environments: Separate your infrastructure into distinct environments (dev, staging, prod) using Terraform workspaces or separate state files.

Troubleshooting and Common Issues

Datadog Agent Not Reporting Data

Check Helm Release Status: Use `helm status datadog -n datadog` to ensure the chart is deployed successfully.
Verify Pod Status: Use `kubectl get pods -n datadog` to confirm Datadog Agent pods are running. Check logs of any failing pods with `kubectl logs -n datadog`.
API Key Validation: Double-check that `datadog_api_key` and `datadog_app_key` are correct and have the necessary permissions in Datadog.
Network Connectivity: Ensure your EKS nodes have outbound internet access to Datadog's ingest endpoints.
RBAC Permissions: Confirm the Kubernetes service account used by the Datadog Agent has sufficient RBAC permissions (often managed by the Helm chart, but custom configurations might break it).

PagerDuty Incidents Not Triggering

Datadog Monitor Status: Verify in Datadog that your monitor is actually triggering an alert (e.g., check event stream or monitor history).
PagerDuty Integration Name: Ensure the name used in your Datadog monitor message (e.g., `@pagerduty-eks-monitoring`) exactly matches the name of the Datadog integration configured in PagerDuty, or the integration you've configured via Terraform.
Service Integration Key: For older Datadog integrations or direct API calls, ensure the PagerDuty integration key in Datadog matches the one created in PagerDuty. For the newer Datadog provider, the connection is typically managed by Datadog's internal integration mapping.
PagerDuty Service Configuration: Check the PagerDuty service's escalation policy and on-call schedules to ensure someone is available to be paged.
Datadog Event Stream: Look at the Datadog Event Stream for messages related to your monitor and PagerDuty integration to identify any errors in sending alerts.

Conclusion

By following this guide, you've established a robust, codified system for deploying AWS EKS with integrated Datadog observability and PagerDuty incident management. This approach not only streamlines your DevOps workflows but also significantly enhances your ability to proactively monitor your cloud-native applications and respond swiftly to critical incidents. Embracing Infrastructure as Code for every layer of your stack—from infrastructure to monitoring and incident response—is a cornerstone of modern, resilient cloud operations.

CloudStack YAML