Terraform Configuration for AWS EKS with Datadog and PagerDuty for Production Incident Management

In the fast-paced world of cloud-native applications, maintaining high availability and rapid incident response is paramount. This comprehensive guide details how to provision and configure an AWS Elastic Kubernetes Service (EKS) cluster using Terraform, integrate it with Datadog for robust monitoring and observability, and set up PagerDuty for streamlined incident management. By following this guide, DevOps teams and SREs can establish a resilient, observable, and automated production environment.

Architecture Pro-Tip:

Always segment your Terraform configurations into logical modules. For EKS, this typically means separating VPC, EKS cluster, node groups, and addon configurations. For integrations like Datadog and PagerDuty, dedicate specific modules or files to manage their resources. This modular approach enhances readability, reusability, and maintainability, crucial for large-scale production environments.

Why Terraform, AWS EKS, Datadog, and PagerDuty?

Terraform: An industry-standard Infrastructure as Code (IaC) tool for provisioning and managing cloud resources. It ensures declarative, repeatable, and version-controlled infrastructure deployments, critical for production consistency.
AWS EKS: A fully managed Kubernetes service by AWS, simplifying the deployment, management, and scaling of Kubernetes applications without needing to provision or maintain the Kubernetes control plane.
Datadog: A comprehensive monitoring, security, and analytics platform for cloud applications. It offers deep visibility into Kubernetes clusters, pods, services, and underlying infrastructure, enabling proactive issue detection.
PagerDuty: A leading incident management platform that aggregates alerts from various monitoring tools, intelligently routes them to the right on-call teams, and facilitates rapid incident resolution and post-mortems.

Prerequisites

Before you begin, ensure you have the following:

An AWS account with appropriate programmatic access (IAM user or role configured).
Terraform CLI installed (v1.0.0+ recommended).
AWS CLI installed and configured.
kubectl installed.
Helm CLI installed.
A Datadog account with an API key and Application key.
A PagerDuty account with an API token.
Basic understanding of AWS, Kubernetes, Terraform, Datadog, and PagerDuty concepts.

Terraform Configuration Breakdown

1. AWS EKS Cluster and Node Group

We'll define an EKS cluster, IAM roles, and a managed node group. For simplicity, we assume a VPC and subnets are already provisioned or will be created in a separate module.

2. Datadog Agent Deployment and Integration

Datadog provides deep observability into EKS by deploying its agent as a DaemonSet across your cluster. We'll use the Terraform Helm provider to deploy the Datadog Agent and configure a Kubernetes secret for the Datadog API key.

API Key Management: Never hardcode API keys. Use AWS Secrets Manager or Kubernetes Secrets, populated via Terraform, for secure storage.
Helm Chart: The Datadog Helm chart simplifies agent deployment and configuration.

3. PagerDuty Service and Integration

PagerDuty integration involves creating a service, an escalation policy, and an integration endpoint that Datadog will use to send alerts.

Service: Represents a system or application that needs monitoring and incident response.
Escalation Policy: Defines the sequence of users or teams to be notified when an incident occurs.
Integration Key: A unique key generated for the Datadog integration, allowing Datadog to trigger incidents in PagerDuty.

4. Alerting Configuration (Datadog to PagerDuty)

Once Datadog is collecting metrics and logs, you can define monitors. When a monitor's alert condition is met, it will trigger an incident in PagerDuty using the configured integration.

Ready-to-Use Terraform Configuration

This example provides a streamlined configuration to get you started. Remember to replace placeholder values with your specific details.

Project Structure:

.
├── main.tf
├── variables.tf
├── outputs.tf
├── providers.tf
└── datadog_pagerduty_integration.tf

# providers.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.0"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
    pagerduty = {
      source  = "PagerDuty/pagerduty"
      version = "~> 1.0"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

provider "kubernetes" {
  host                   = module.eks_cluster.eks_cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks_cluster.eks_cluster_certificate_authority_data)
  token                  = data.aws_eks_cluster_auth.this.token
}

provider "helm" {
  kubernetes {
    host                   = module.eks_cluster.eks_cluster_endpoint
    cluster_ca_certificate = base64decode(module.eks_cluster.eks_cluster_certificate_authority_data)
    token                  = data.aws_eks_cluster_auth.this.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

# main.tf (EKS Cluster & Node Group - simplified for brevity, using a community module)
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "eks-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["${var.aws_region}a", "${var.aws_region}b", "${var.aws_region}c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway = true
  single_nat_gateway = true
  enable_dns_hostnames = true
  enable_dns_support = true

  tags = {
    Environment = "Production"
    Project     = "EKS-Datadog-PagerDuty"
  }
}

module "eks_cluster" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0"

  cluster_name    = var.eks_cluster_name
  cluster_version = "1.28"
  vpc_id          = module.vpc.vpc_id
  subnet_ids      = module.vpc.private_subnets

  eks_managed_node_groups = {
    default = {
      min_size     = 2
      max_size     = 5
      desired_size = 2
      instance_types = ["t3.medium"]
      disk_size    = 20
    }
  }

  tags = {
    Environment = "Production"
    Project     = "EKS-Datadog-PagerDuty"
  }
}

data "aws_eks_cluster_auth" "this" {
  name = module.eks_cluster.eks_cluster_id
}

# datadog_pagerduty_integration.tf

# 1. Kubernetes Secret for Datadog API Key
resource "kubernetes_secret" "datadog_api_key" {
  metadata {
    name      = "datadog-api-key"
    namespace = "default" # Or a dedicated monitoring namespace
  }

  data = {
    api-key = base64encode(var.datadog_api_key)
  }

  type = "Opaque"
}

# 2. Deploy Datadog Agent using Helm
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Or a dedicated monitoring namespace, ensure it exists
  version    = "3.3.0" # Use a specific stable version

  set {
    name  = "datadog.site"
    value = var.datadog_site
  }

  set {
    name  = "datadog.apiKeyExistingSecret"
    value = kubernetes_secret.datadog_api_key.metadata[0].name
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key # App key for some integrations, can also use secret if needed
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "clusterChecksRunner.enabled"
    value = "true"
  }

  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
  }

  set {
    name  = "datadog.kubelet.host"
    value = "$${NODE_NAME}"
  }

  set {
    name  = "datadog.kubelet.tlsVerify"
    value = "false" # Set to true in production with proper cert setup
  }

  set {
    name  = "targetSystem"
    value = "linux"
  }

  values = [
    file("${path.module}/datadog-values.yaml") # For more complex configurations
  ]
}

# datadog-values.yaml (example content if you use it, otherwise remove `values` block from helm_release)
# This file would typically contain more specific Datadog agent configurations,
# like custom checks, logs configurations, APM settings etc.
# logs:
#   enabled: true
#   containerCollectAll: true
# apm:
#   enabled: true

# 3. PagerDuty Escalation Policy and Service
resource "pagerduty_escalation_policy" "devops_escalation_policy" {
  name    = "DevOps Team Primary Escalation"
  num_loops = 2 # How many times to loop through the policy
  teams   = [var.pagerduty_team_id] # Link to an existing PagerDuty team by ID

  rule {
    delay    = 5
    target {
      type = "user"
      id   = var.pagerduty_user_id # Example: Notify a specific user first
    }
  }

  rule {
    delay    = 10
    target {
      type = "schedule"
      id   = var.pagerduty_oncall_schedule_id # Example: Then notify the primary on-call schedule
    }
  }
}

resource "pagerduty_service" "eks_monitoring_service" {
  name                    = "EKS Cluster Monitoring"
  auto_resolve_timeout    = "14400" # 4 hours
  acknowledgement_timeout = "600"   # 10 minutes
  escalation_policy       = pagerduty_escalation_policy.devops_escalation_policy.id
  description             = "Monitors the health and performance of the AWS EKS cluster."
}

resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog Integration"
  service_id  = pagerduty_service.eks_monitoring_service.id
  type        = "generic_events_api_inbound_integration" # PagerDuty generic events API for Datadog
}

# 4. Datadog Monitor to trigger PagerDuty Incident
resource "datadog_monitor" "eks_high_cpu_monitor" {
  name  = "[EKS] High CPU Utilization Alert for {{cluster_name.name}}"
  type  = "metric alert"
  query = "avg(last_5m):avg:kubernetes.cpu.usage.total{kubernetes_cluster_name:\"${var.eks_cluster_name}\"} by {host} > 80"

  message = <<EOT
  @pagerduty-{{pagerduty_service.name}}
  EKS Node {{host.name}} is experiencing high CPU utilization ({{value}}%).
  Please investigate immediately.
  EOT

  tags = ["env:production", "service:eks", "severity:critical"]

  monitor_thresholds {
    critical = 80
    warning  = 70
  }

  notify_no_data    = false
  renotify_interval = 60
  escalation_message = "CPU still high after 60 minutes. Escalating."

  # The PagerDuty integration name is usually derived from the integration's name in PagerDuty,
  # but in Datadog, it refers to the integration configured in Datadog's UI.
  # For a PagerDuty integration, it's typically just `pagerduty`.
  # However, for multiple PagerDuty services, you define specific integration names within Datadog.
  # Assuming you have a PagerDuty integration set up in Datadog's UI and it's named "PagerDuty EKS".
  # The message structure `@pagerduty-{{pagerduty_service.name}}` or `@pagerduty-{{integration_name}}` is key.
  # For simplicity, we assume a single PagerDuty integration in Datadog, or you'd map to a specific one.
  # The specific service is often passed via a templated variable in the message field itself like `@pagerduty-EKS Cluster Monitoring`.
  # For this example, we'll use the service name.
}

# variables.tf
variable "aws_region" {
  description = "AWS region for deployments"
  type        = string
  default     = "us-east-1"
}

variable "eks_cluster_name" {
  description = "Name for the EKS cluster"
  type        = string
  default     = "production-eks-cluster"
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "datadog_site" {
  description = "Datadog site (e.g., datadoghq.com, eu.datadoghq.com)"
  type        = string
  default     = "datadoghq.com"
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

variable "pagerduty_team_id" {
  description = "ID of the PagerDuty team for the escalation policy"
  type        = string
  default     = "P123456" # Replace with your actual PagerDuty Team ID
}

variable "pagerduty_user_id" {
  description = "ID of the PagerDuty user for the escalation policy (e.g., initial contact)"
  type        = string
  default     = "PQRSTUV" # Replace with an actual PagerDuty User ID
}

variable "pagerduty_oncall_schedule_id" {
  description = "ID of the PagerDuty on-call schedule for the escalation policy"
  type        = string
  default     = "QWXNYZ" # Replace with an actual PagerDuty On-Call Schedule ID
}

# outputs.tf
output "eks_cluster_id" {
  description = "The ID of the EKS cluster."
  value       = module.eks_cluster.eks_cluster_id
}

output "datadog_agent_helm_release_status" {
  description = "The status of the Datadog Agent Helm release."
  value       = helm_release.datadog_agent.status
}

output "pagerduty_service_url" {
  description = "URL to the PagerDuty service."
  value       = pagerduty_service.eks_monitoring_service.html_url
}

output "pagerduty_integration_key" {
  description = "Integration key for the Datadog PagerDuty integration."
  value       = pagerduty_service_integration.datadog_integration.integration_key
  sensitive   = true
}
    

Deployment Steps

Follow these steps to deploy your EKS cluster with Datadog and PagerDuty integration:

Initialize Terraform: Navigate to your Terraform project directory and run terraform init to download providers and modules.
Plan the Deployment: Execute terraform plan to review the infrastructure changes Terraform will make. Carefully inspect the output.
Apply the Configuration: Run terraform apply and confirm with yes when prompted. This will provision your EKS cluster, deploy the Datadog Agent, and configure PagerDuty resources.
Configure kubectl: After EKS creation, update your kubectl configuration: aws eks update-kubeconfig --name ${var.eks_cluster_name} --region ${var.aws_region}.

Verification

Confirm that all components are correctly configured and communicating:

EKS Cluster Status: Use kubectl get nodes to verify your worker nodes are ready.
Datadog Agent: Run kubectl get pods -n default | grep datadog (or your chosen namespace) to ensure Datadog agent pods are running. Check Datadog's UI for EKS integration dashboards and host metrics.
PagerDuty Service: Log into your PagerDuty account and verify that the "EKS Cluster Monitoring" service and the "DevOps Team Primary Escalation" policy exist. The Datadog integration should also be present.
Datadog Monitor: In Datadog, confirm the "EKS High CPU Utilization Alert" monitor is active and configured to notify the PagerDuty service.
Test Incident: (Optional, but recommended in a controlled environment) Artificially increase CPU load on an EKS node or manually trigger the Datadog monitor to confirm an incident is created in PagerDuty and notifications are sent as expected.

Best Practices for Production Environments

Immutable Infrastructure: Treat your infrastructure as immutable. Any changes should be applied via Terraform, not manual interventions.
Secrets Management: Use AWS Secrets Manager or HashiCorp Vault for all sensitive data (API keys, tokens) instead of directly in `variables.tf` files or environment variables.
Separate Environments: Implement distinct Terraform configurations and AWS accounts for development, staging, and production environments.
GitOps Workflow: Integrate Terraform with a GitOps pipeline (e.g., Git, GitHub Actions/GitLab CI/CD, Atlantis) for automated, version-controlled deployments.
Cost Management: Monitor EKS cluster costs using AWS Cost Explorer and optimize node group sizes, instance types, and autoscaling settings.
Security Hardening: Implement EKS security best practices, including IAM roles for service accounts (IRSA), network policies, and regular security audits.
Granular Datadog Monitors: Create specific Datadog monitors for different components and criticality levels (e.g., pod restarts, OOMKills, network errors, application-specific metrics).
Effective PagerDuty Escalation Policies: Design escalation policies that ensure critical alerts reach the right people promptly, minimizing alert fatigue by tuning Datadog monitors.

Troubleshooting Common Issues

Terraform Apply Fails:
- Permissions: Ensure your AWS IAM user/role has sufficient permissions for EKS, IAM, VPC, and other AWS resources.
- State Lock: If using a remote backend (S3), check for state lock issues.
- Resource Limits: Verify you haven't hit AWS service limits for EKS clusters, NAT gateways, or EC2 instances.
Datadog Agent Not Reporting:
- Secret Errors: Double-check your datadog_api_key and datadog_app_key are correct and correctly mounted as Kubernetes secrets.
- Network Connectivity: Ensure EKS nodes can reach Datadog endpoints (e.g., app.datadoghq.com) on port 443. Check Security Groups and Network ACLs.
- Pod Logs: Inspect Datadog agent pod logs for errors: kubectl logs <datadog-agent-pod-name> -n default.
PagerDuty Incidents Not Triggering:
- Integration Key: Confirm the integration key used by Datadog matches the one generated by PagerDuty.
- Datadog Monitor Configuration: Verify the monitor's message template correctly references the PagerDuty integration (e.g., @pagerduty-{{service_name}}) and that the monitor is actually in an alert state.
- PagerDuty Escalation Policy: Check if the escalation policy is correctly linked to the service and has active on-call users or schedules.

Conclusion

Automating the deployment and integration of AWS EKS with robust monitoring from Datadog and efficient incident management via PagerDuty is a critical step towards building a resilient, observable, and highly available production environment. By leveraging Terraform as your Infrastructure as Code tool, you gain predictability, repeatability, and version control over your entire cloud-native stack. This setup empowers your DevOps and SRE teams to proactively identify issues, minimize downtime, and ensure a seamless experience for your users.

CloudStack YAML