Terraform AWS EKS with Datadog Observability and PagerDuty Incident Response

In the dynamic landscape of cloud-native applications, establishing a robust, scalable, and observable infrastructure is paramount. This comprehensive guide details how to provision an Amazon EKS (Elastic Kubernetes Service) cluster using Terraform Infrastructure as Code (IaC), integrate it with Datadog for end-to-end observability, and streamline incident response with PagerDuty. By the end of this guide, you'll have a fully automated, production-ready setup capable of proactive monitoring and efficient incident resolution.

Architecture Pro-Tip

For robust production EKS deployments, always decouple your VPC and core networking from the EKS cluster module. This approach provides greater reusability, clearer separation of concerns, and explicit control over network topology, security groups, and routing tables, preventing tightly coupled infrastructure dependencies.

Why This Stack? The Power of Integration

Integrating these leading tools offers a powerful synergy for modern DevOps teams:

Terraform: Enables declarative infrastructure provisioning, version control, and repeatable deployments for your AWS EKS cluster and its associated resources.
AWS EKS: A managed Kubernetes service that simplifies running Kubernetes on AWS without needing to install, operate, and maintain your own Kubernetes control plane.
Datadog: Provides unified observability across your entire stack – collecting metrics, logs, and traces from EKS, applications, and AWS infrastructure, offering real-time insights and intelligent alerting.
PagerDuty: Transforms Datadog alerts into actionable incidents, ensuring the right team members are notified promptly, facilitating faster incident resolution, and maintaining service reliability.

Prerequisites

Before you begin, ensure you have the following:

An active AWS Account with programmatic access keys configured.
Terraform CLI (v1.0.0+) installed.
AWS CLI installed and configured.
kubectl installed.
helm CLI installed.
A Datadog Account with API and Application keys.
A PagerDuty Account with an API key and a configured service to integrate with Datadog.
Basic understanding of AWS networking (VPC, subnets, security groups).

Step-by-Step Implementation

1. Initialize AWS and Kubernetes Providers

First, set up your Terraform providers. The Kubernetes provider will dynamically configure itself after the EKS cluster is created.

provider "aws" {
  region = "us-east-1"
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  token                  = data.aws_eks_cluster_auth.this.token
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
  # The 'depends_on' ensures Kubernetes provider waits for EKS creation
  depends_on = [aws_eks_cluster.this]
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.this.endpoint
    token                  = data.aws_eks_cluster_auth.this.token
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
  }
  depends_on = [aws_eks_cluster.this]
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

2. Provision AWS EKS Cluster and Node Group

We'll use the official terraform-aws-modules/eks/aws module for a streamlined EKS deployment. This example assumes you have an existing VPC and subnets.

# main.tf
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "eks-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]

  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true
  enable_dns_support   = true
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0"

  cluster_name    = "datadog-eks-cluster"
  cluster_version = "1.28"

  vpc_id                   = module.vpc.vpc_id
  subnet_ids               = module.vpc.private_subnets
  control_plane_subnet_ids = module.vpc.public_subnets # Public subnets for control plane for simplicity in this example

  # EKS Cluster logging
  cluster_enabled_log_types = ["api", "audit", "authenticator", "controllerManager", "scheduler"]

  # Node Groups
  eks_managed_node_groups = {
    datadog_workers = {
      min_size     = 2
      max_size     = 5
      desired_size = 3

      instance_types = ["t3.medium"]
      capacity_type  = "ON_DEMAND"
      # Additional tags for cost allocation and identification
      tags = {
        Name        = "datadog-eks-node-group"
        Environment = "DevOpsGuide"
      }
    }
  }

  # Add an IAM OpenID Connect (OIDC) provider for the EKS cluster
  enable_irsa = true

  tags = {
    Environment = "DevOpsGuide"
    Project     = "EKS-Datadog-PagerDuty"
  }
}

data "aws_eks_cluster" "this" {
  name = module.eks.cluster_name
}

data "aws_eks_cluster_auth" "this" {
  name = module.eks.cluster_name
}

3. Configure IAM for Service Accounts (IRSA)

To securely grant AWS permissions to Kubernetes service accounts, we use IAM Roles for Service Accounts (IRSA). This is crucial for the Datadog Agent to collect metrics directly from AWS services without exposing AWS credentials within the Pod.

# iam.tf
resource "aws_iam_policy" "datadog_agent_policy" {
  name        = "DatadogAgentPolicy-${module.eks.cluster_name}"
  description = "IAM policy for Datadog Agent to access AWS resources"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "ec2:DescribeInstances",
          "ec2:DescribeVolumes",
          "ec2:DescribeVpcs",
          "ec2:DescribeSubnets",
          "ec2:DescribeSecurityGroups",
          "autoscaling:DescribeAutoScalingGroups",
          "elasticloadbalancing:DescribeLoadBalancers",
          "elasticloadbalancing:DescribeTargetGroups",
          "route53:ListResourceRecordSets",
          "rds:DescribeDBInstances",
          "lambda:ListFunctions",
          "logs:DescribeLogGroups",
          "logs:FilterLogEvents"
        ]
        Effect   = "Allow"
        Resource = "*"
      },
      {
        Action = [
          "sqs:GetQueueAttributes",
          "sqs:ListQueues",
          "sqs:ReceiveMessage"
        ]
        Effect   = "Allow"
        Resource = "*"
      },
      {
        Action = [
          "sns:Publish"
        ]
        Effect   = "Allow"
        Resource = "*" # Restrict as needed in production
      }
    ]
  })
}

resource "aws_iam_role" "datadog_agent_irsa" {
  name               = "DatadogAgentIRSA-${module.eks.cluster_name}"
  assume_role_policy = module.eks.eks_oidc_issuer_url != null ? data.aws_iam_policy_document.datadog_agent_assume_role.json : null
}

data "aws_iam_policy_document" "datadog_agent_assume_role" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type        = "Federated"
      identifiers = [module.eks.oidc_provider_arn]
    }
    condition {
      test     = "StringEquals"
      variable = "${replace(module.eks.oidc_provider_extract_from_arn, "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/", "")}:sub"
      values   = ["system:serviceaccount:default:datadog-agent"] # Adjust namespace/service account if changed
    }
  }
}

resource "aws_iam_role_policy_attachment" "datadog_agent_policy_attach" {
  policy_arn = aws_iam_policy.datadog_agent_policy.arn
  role       = aws_iam_role.datadog_agent_irsa.name
}

4. Deploy Datadog Agent via Helm

Deploy the Datadog Agent to your EKS cluster using the Helm provider. This will automatically install the agent, APM, and log collection.

5. Ready-to-Use Configuration Example

Below is a consolidated example of how to deploy the Datadog Agent and configure a basic Datadog monitor that integrates with PagerDuty. This example assumes you have an existing PagerDuty integration named "PagerDuty" configured within Datadog.

# datadog_integration.tf

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Consider a dedicated 'datadog' namespace
  version    = "2.33.20" # Use a specific version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }

  set {
    name  = "clusterName"
    value = module.eks.cluster_name
  }

  set {
    name  = "datadog.site"
    value = var.datadog_site # e.g., "datadoghq.com" or "eu.datadoghq.com"
  }

  set {
    name  = "agents.image.tag"
    value = "7.52.0" # Specify agent image tag
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "kubeStateMetricsExternal.enabled"
    value = "true"
  }

  set {
    name  = "targetSystem"
    value = "linux"
  }

  # Enable APM and Log collection
  set {
    name  = "apm.enabled"
    value = "true"
  }
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }

  # Configure IRSA for Datadog Agent
  set {
    name  = "rbac.create"
    value = "true"
  }
  set {
    name  = "serviceAccount.create"
    value = "true"
  }
  set {
    name  = "serviceAccount.name"
    value = "datadog-agent"
  }
  set {
    name  = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
    value = aws_iam_role.datadog_agent_irsa.arn
  }

  # Add Kubernetes specific checks
  set {
    name  = "datadog.confd.kube_state_metrics_core.check_names[0]"
    value = "kube_state_metrics"
  }
  set {
    name  = "datadog.confd.kube_state_metrics_core.init_config"
    value = "{}"
  }
  set {
    name  = "datadog.confd.kube_state_metrics_core.instances[0].cluster_name"
    value = module.eks.cluster_name
  }

  depends_on = [
    module.eks,
    aws_iam_role_policy_attachment.datadog_agent_policy_attach
  ]
}

# datadog_monitor.tf
resource "datadog_monitor" "eks_node_cpu_utilization" {
  name                = "[EKS] High Node CPU Utilization on ${module.eks.cluster_name}"
  type                = "metric alert"
  message             = "EKS node CPU utilization is high. @webhook-PagerDuty-Datadog" # Assumes a PagerDuty integration named "PagerDuty-Datadog"
  escalation_message  = "CPU utilization remains critical after 5 minutes. Paging on-call team!"
  query               = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${module.eks.cluster_name}} by {host} > 80"
  monitor_thresholds {
    critical = 80
    warning  = 70
  }
  notify_no_data      = false
  new_host_delay      = 300
  no_data_timeframe   = 20
  include_tags        = true
  tags                = ["environment:production", "service:eks", "alert:cpu"]
}

# variables.tf
variable "datadog_api_key" {
  description = "Your Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Your Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "datadog_site" {
  description = "Datadog site (e.g., 'datadoghq.com' or 'eu.datadoghq.com')"
  type        = string
  default     = "datadoghq.com"
}

6. Apply the Configuration

With your Terraform configuration files (main.tf, iam.tf, datadog_integration.tf, variables.tf) in place, initialize and apply your infrastructure:

terraform init
terraform plan -var="datadog_api_key=YOUR_DD_API_KEY" -var="datadog_app_key=YOUR_DD_APP_KEY"
terraform apply -var="datadog_api_key=YOUR_DD_API_KEY" -var="datadog_app_key=YOUR_DD_APP_KEY"

Replace YOUR_DD_API_KEY and YOUR_DD_APP_KEY with your actual Datadog keys. For production, consider using environment variables or a secrets manager for these sensitive values.

Validation and Testing

After applying your Terraform configuration:

Verify EKS Cluster: Use kubectl get nodes and kubectl get pods -A to confirm your EKS cluster and Datadog Agent pods are running.
Check Datadog UI: Navigate to your Datadog dashboard. You should see host metrics, Kubernetes events, and logs appearing from your EKS cluster. Check the "Integrations" section for AWS and Kubernetes to ensure they are connected.
Test PagerDuty Integration: Manually trigger a test alert in Datadog or simulate a high CPU load on an EKS node to verify that the PagerDuty integration correctly creates an incident.

Best Practices for Production Environments

Dedicated Namespaces: Deploy infrastructure components like Datadog Agent into dedicated Kubernetes namespaces (e.g., datadog).
Least Privilege IAM: Refine IAM policies for the Datadog Agent to only include necessary permissions.
Secrets Management: Use AWS Secrets Manager or HashiCorp Vault to securely store Datadog API/App keys and other sensitive information, injecting them into Terraform via data sources.
Cost Optimization: Utilize AWS Spot Instances for non-critical workloads in your EKS node groups to reduce costs.
Advanced Datadog Monitoring: Beyond basic CPU, configure monitors for memory utilization, disk I/O, network latency, application-specific metrics, and Kubernetes events.
PagerDuty Escalation Policies: Design robust escalation policies in PagerDuty to ensure alerts reach the right on-call personnel based on urgency and time of day.
Network Security: Implement strict network policies (Kubernetes Network Policies, AWS Security Groups) to control traffic flow within your EKS cluster and to/from external services.

Troubleshooting Common Issues

Datadog Agent Pods Not Running

Check Pod Status: kubectl get pods -n default | grep datadog. Look for CrashLoopBackOff or Pending states.
Pod Logs: kubectl logs <datadog-agent-pod-name> -n default. Look for API key errors, permission issues, or configuration mistakes.
IRSA Configuration: Ensure the IAM role ARN is correctly annotated on the Datadog Agent service account and that the trust policy allows sts:AssumeRoleWithWebIdentity.

No Data in Datadog UI

API/App Keys: Double-check that your Datadog API and Application keys are correct and properly passed to the Helm chart.
Network Connectivity: Verify that your EKS nodes and Datadog Agent pods have outbound internet access to Datadog's ingest endpoints (*.datadoghq.com or your specific Datadog site).
IAM Permissions: Ensure the IAM role attached to the Datadog Agent service account has the necessary permissions to read AWS service metadata (e.g., EC2, RDS, ELB).

PagerDuty Incidents Not Triggering

Datadog-PagerDuty Integration: In Datadog, go to "Integrations" -> "PagerDuty" and ensure the integration is active and correctly configured with a PagerDuty service. The @webhook-PagerDuty-Datadog (or whatever you've named it) in the monitor message must match your Datadog integration name.
Monitor Thresholds: Verify that the Datadog monitor's query and thresholds are being met to trigger an alert.
Conclusion
By following this guide, you've successfully deployed a robust AWS EKS cluster with Terraform, integrated comprehensive observability using Datadog, and established an efficient incident response workflow with PagerDuty. This powerful combination empowers your DevOps team to build, monitor, and maintain highly available and reliable cloud-native applications with confidence, significantly reducing mean time to detection (MTTD) and mean time to resolution (MTTR). Continuously refine your monitoring and alerting strategies to adapt to evolving application needs and ensure peak operational excellence.

CloudStack YAML