Terraform AWS EKS: Automated Datadog Observability & PagerDuty Incident Response

In the dynamic landscape of cloud-native infrastructure, maintaining high availability and rapid incident resolution for Kubernetes clusters is paramount. This guide details a robust, automated approach using Terraform to provision and manage an AWS EKS cluster, integrate comprehensive Datadog observability, and establish a proactive PagerDuty incident response workflow. By codifying your infrastructure and operational tooling, you can achieve unparalleled consistency, scalability, and operational efficiency.

Architecture Pro-Tip: Embracing a fully declarative Infrastructure as Code (IaC) approach with Terraform for your EKS cluster, Datadog agents, and PagerDuty services minimizes configuration drift and ensures idempotent deployments. This vastly improves auditability, disaster recovery capabilities, and the overall reliability of your cloud-native platform. Always version control your Terraform configurations!

Why Automate Observability and Incident Response?

Operating mission-critical applications on Kubernetes demands more than just deployment. It requires deep visibility into cluster health, application performance, and timely alerts for anomalies. Automation through Terraform, Datadog, and PagerDuty delivers:

Rapid Detection: Datadog provides real-time metrics, logs, and traces, enabling quick identification of issues across EKS.
Proactive Alerting: Configurable monitors in Datadog trigger alerts based on defined thresholds, preventing minor issues from escalating.
Streamlined Incident Workflow: PagerDuty ensures critical alerts reach the right on-call personnel immediately, facilitating swift response and resolution.
Consistency & Reliability: Terraform guarantees that your observability and incident response configurations are consistently applied across all environments.
Reduced Manual Overhead: Automating setup frees up DevOps teams to focus on innovation rather than repetitive configuration tasks.

Prerequisites

Before you begin, ensure you have the following:

AWS Account: With necessary IAM permissions to create EKS clusters, VPCs, EC2 instances, and other related resources.
Terraform CLI: Installed and configured (version 1.0+ recommended).
AWS CLI: Configured for programmatic access.
Datadog Account: With API and Application Keys.
PagerDuty Account: With an API Token for integration.
Kubectl: Installed and configured to interact with your EKS cluster.
Helm CLI: Installed for deploying the Datadog Agent.

Core Components and Their Integration

This solution orchestrates several key technologies:

AWS EKS (Elastic Kubernetes Service): The managed Kubernetes service, provisioned and configured via Terraform.
Datadog: A comprehensive monitoring, logging, and tracing platform.
- Datadog Agent: Deployed as a DaemonSet on EKS, collecting cluster-wide metrics, events, and logs.
- Datadog Monitors: Automated alerts defined in Terraform, integrated with PagerDuty.
- Datadog Dashboards: Visualizations of EKS health and application performance.
PagerDuty: An incident management platform that routes alerts to on-call teams.
- PagerDuty Services: Representing logical components or applications, configured with escalation policies.
- PagerDuty Users & Teams: Defining on-call schedules and responsibilities.
Terraform: The Infrastructure as Code tool that provisions and manages all the above components declaratively.

Step-by-Step Implementation Guide

1. Terraform Setup & Providers

Start by defining your Terraform providers for AWS, Helm, Datadog, and PagerDuty.

provider "aws" {
  region = "us-east-1"
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.main.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.main.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.main.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.main.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

2. Provision AWS EKS Cluster with Terraform

Utilize the popular terraform-aws-modules/eks/aws module for a streamlined EKS cluster setup. This includes VPC, subnets, node groups, and IAM roles.

Example (simplified):

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0"

  cluster_name    = "my-eks-cluster"
  cluster_version = "1.28"

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  eks_managed_node_groups = {
    default = {
      instance_types = ["t3.medium"]
      desired_size   = 2
      max_size       = 3
      min_size       = 1
    }
  }

  tags = {
    Environment = "production"
    Project     = "my-app"
  }
}

data "aws_eks_cluster" "main" {
  name = module.eks.cluster_name
}

data "aws_eks_cluster_auth" "main" {
  name = module.eks.cluster_name
}

3. Deploy Datadog Agent to EKS via Helm/Terraform

The Datadog Agent collects all necessary metrics, logs, and traces from your EKS cluster and sends them to Datadog. Deploy it using the Helm provider in Terraform.

4. Configure PagerDuty Service with Terraform

Define your PagerDuty service, escalation policies, and users directly in Terraform. This ensures your incident response structure is version-controlled and consistently applied.

resource "pagerduty_user" "devops_engineer" {
  name  = "DevOps Engineer"
  email = "devops@example.com"
}

resource "pagerduty_escalation_policy" "high_priority_policy" {
  name      = "High Priority EKS Escalation"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 15
    target {
      type = "user_reference"
      id   = pagerduty_user.devops_engineer.id
    }
  }
  # Add more rules/targets as needed
}

resource "pagerduty_service" "eks_monitoring_service" {
  name                    = "EKS Cluster Monitoring"
  auto_resolve_timeout    = "14400" # 4 hours
  acknowledgement_timeout = "600"   # 10 minutes
  escalation_policy       = pagerduty_escalation_policy.high_priority_policy.id
}

5. Integrate Datadog Monitors with PagerDuty

Finally, create Datadog monitors that leverage the PagerDuty integration. When a monitor's alert condition is met, it will automatically trigger an incident in PagerDuty, notifying the appropriate on-call team.

Ready-to-Use Configuration Example (Combined)

This example brings together the core components. Remember to replace placeholder values and expand upon these basics for a production environment.

# main.tf

# --- AWS Provider (as defined previously) ---
provider "aws" { region = "us-east-1" }
# --- Kubernetes Provider (as defined previously) ---
provider "kubernetes" {
  host                   = data.aws_eks_cluster.main.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.main.token
}
# --- Helm Provider (as defined previously) ---
provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.main.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.main.token
  }
}
# --- Datadog Provider (as defined previously) ---
provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}
# --- PagerDuty Provider (as defined previously) ---
provider "pagerduty" {
  token = var.pagerduty_api_token
}

# --- Variables (for sensitive data) ---
variable "datadog_api_key" { description = "Datadog API Key" type = string sensitive = true }
variable "datadog_app_key" { description = "Datadog Application Key" type = string sensitive = true }
variable "pagerduty_api_token" { description = "PagerDuty API Token" type = string sensitive = true }
variable "pagerduty_service_name" { description = "PagerDuty service name to integrate with" type = string default = "EKS Cluster Monitoring" }

# --- EKS Cluster Module (simplified for example) ---
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "my-eks-vpc"
  cidr = "10.0.0.0/16"

  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24"]

  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0"

  cluster_name    = "my-eks-cluster"
  cluster_version = "1.28"

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  # Self-managed node group for simplicity, consider managed node groups for production
  self_managed_node_groups = {
    default = {
      instance_types = ["t3.medium"]
      desired_size   = 2
      max_size       = 3
      min_size       = 1
      disk_size      = 20
    }
  }

  tags = {
    Environment = "dev"
    Project     = "EKS-Datadog-PagerDuty"
  }
}

data "aws_eks_cluster" "main" { name = module.eks.cluster_name }
data "aws_eks_cluster_auth" "main" { name = module.eks.cluster_name }

# --- Datadog Agent Deployment (Helm) ---
resource "kubernetes_secret" "datadog_api_key" {
  metadata {
    name      = "datadog-api-key"
    namespace = "default"
  }
  data = {
    "api-key" = var.datadog_api_key
  }
  type = "Opaque"
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Or dedicated 'datadog' namespace
  version    = "2.33.0"  # Use a recent stable version

  set {
    name  = "datadog.site"
    value = "datadoghq.com"
  }
  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    # Alternatively, use existingSecret and reference kubernetes_secret.datadog_api_key
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "agents.tolerations[0].key"
    value = "node-role.kubernetes.io/master"
  }
  set {
    name  = "agents.tolerations[0].operator"
    value = "Exists"
  }
  set {
    name  = "agents.tolerations[0].effect"
    value = "NoSchedule"
  }
  # Enable APM, Log Collection, Live Processes as needed
  set {
    name  = "datadog.apm.enabled"
    value = "true"
  }
  set {
    name  = "datadog.logs.enabled"
    value = "true"
  }
  set {
    name  = "datadog.logs.containerCollectAll"
    value = "true"
  }
  set {
    name  = "datadog.processAgent.enabled"
    value = "true"
  }
  set {
    name  = "datadog.processAgent.processCollection"
    value = "true"
  }
  # EKS specific settings
  set {
    name  = "clusterName"
    value = module.eks.cluster_name
  }
}

# --- PagerDuty Service & Escalation Policy ---
resource "pagerduty_user" "on_call_devops" {
  name  = "Primary DevOps"
  email = "oncall@example.com"
}

resource "pagerduty_escalation_policy" "eks_critical_policy" {
  name      = "EKS Critical Escalation Policy"
  num_loops = 2 # Escalate twice

  rule {
    escalation_delay_in_minutes = 5
    target {
      type = "user_reference"
      id   = pagerduty_user.on_call_devops.id
    }
  }
}

resource "pagerduty_service" "eks_service" {
  name              = var.pagerduty_service_name
  escalation_policy = pagerduty_escalation_policy.eks_critical_policy.id
  auto_resolve_timeout = 14400 # 4 hours
  acknowledgement_timeout = 600 # 10 minutes
}

# --- Datadog Monitor for Node CPU Utilization ---
resource "datadog_monitor" "eks_node_cpu_high" {
  name                = "[EKS] High Node CPU Utilization on {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${module.eks.cluster_name},kube_container_name:kube-proxy} by {host} > 80"
  message             = <<EOF
EKS node {{host.name}} is experiencing high CPU utilization ({{value}}%). This might indicate resource contention or an overloaded workload.
@pagerduty-${pagerduty_service.eks_service.name}
EOF
  tags                = ["environment:dev", "project:eks", "severity:high"]
  priority            = 1
  restricted_roles    = []
  notify_no_data      = false
  new_group_delay     = 60
  no_data_timeframe   = 20
  renotify_interval   = 0
  escalation_message  = "CPU utilization remains high. Escalating to next level."
  include_tags        = true
  force_delete        = false
  notify_audit        = false
  locked              = false
  timeout_h           = 0
  require_full_window = false
  evaluation_delay    = 90
}

To deploy this configuration:

Save the code to main.tf in an empty directory.
Create a terraform.tfvars file and securely add your datadog_api_key, datadog_app_key, and pagerduty_api_token. Do not commit this file to version control.
Run terraform init
Run terraform plan to review the changes.
Run terraform apply to provision the infrastructure.

Benefits of this Automated Approach

Efficiency: Rapidly deploy EKS clusters with integrated observability and incident response in minutes.
Consistency: Eliminate manual errors and configuration drift across environments.
Scalability: Easily replicate and scale your infrastructure and monitoring stack.
Auditability: All changes are codified and version-controlled, providing a clear audit trail.
Faster MTTR: Proactive alerting and automated incident routing significantly reduce Mean Time To Resolution (MTTR).
DevOps Enablement: Empowers development and operations teams with self-service capabilities and standardized workflows.

Troubleshooting and Best Practices

IAM Permissions: Ensure your AWS credentials used by Terraform have sufficient permissions for EKS, EC2, IAM, and VPC. The EKS cluster also needs appropriate IAM roles for nodes.
Datadog API Keys: Double-check that your Datadog API and Application keys are correct and have the necessary permissions within Datadog.
PagerDuty Integration Key: Verify the PagerDuty API token has the correct scope to create services and interact with the API.
Kubernetes Context: After terraform apply for EKS, update your kubeconfig using aws eks update-kubeconfig --name <cluster-name> --region <region> to interact with the cluster via kubectl.
Helm Chart Versions: Always specify exact Helm chart versions in your Terraform configuration to avoid unexpected changes.
State Management: Use a remote backend (e.g., S3 with DynamoDB locking) for your Terraform state to enable collaboration and prevent state corruption.
Granular Monitoring: Beyond basic CPU/memory, consider creating monitors for specific application metrics, Kubernetes events, and custom log patterns.
Alert Fatigue: Carefully tune your Datadog monitors to avoid excessive alerting, which can lead to alert fatigue for your on-call teams.

Conclusion

Automating the deployment of AWS EKS with integrated Datadog observability and PagerDuty incident response through Terraform is a critical step towards building resilient, scalable, and operationally efficient cloud-native platforms. This comprehensive approach ensures that your infrastructure, monitoring, and incident management workflows are always in sync, reducing manual toil, improving reliability, and empowering your DevOps teams to focus on delivering value. By adopting these practices, organizations can achieve a mature and robust operational posture for their Kubernetes environments.

CloudStack YAML