Terraform for AWS EKS: Integrated Datadog Monitoring and PagerDuty Alerting

In the rapidly evolving landscape of cloud-native applications, maintaining robust observability and incident response for Kubernetes clusters is paramount. This guide provides a comprehensive, technical walkthrough on how to leverage Terraform to provision and manage an AWS EKS cluster, integrate it with Datadog Monitoring for deep insights, and establish a resilient alerting workflow through PagerDuty Alerting. By automating these integrations with Infrastructure as Code (IaC), organizations can achieve consistent, scalable, and reliable operational excellence.

Architecture Pro-Tip: Multi-Cloud Observability Strategy

When designing your observability stack, consider a platform that offers broad multi-cloud and hybrid-cloud support. Datadog excels here, allowing you to consolidate metrics, logs, and traces from diverse environments into a single pane of glass. For critical applications, always implement redundant alerting paths (e.g., email, Slack alongside PagerDuty) to minimize incident blind spots. Employ granular service ownership with PagerDuty to ensure alerts reach the correct team promptly, reducing Mean Time To Resolution (MTTR).

Why Terraform, EKS, Datadog, and PagerDuty?

Each component plays a critical role in a modern DevOps ecosystem:

Terraform: Enables declarative infrastructure management, ensuring consistency and version control for your AWS EKS cluster and its integrations.
AWS EKS: Provides a managed Kubernetes service, offloading the operational burden of managing the Kubernetes control plane.
Datadog Monitoring: Offers end-to-end observability, collecting metrics, logs, and traces from your EKS cluster, applications, and underlying AWS infrastructure.
PagerDuty Alerting: Acts as a robust incident management platform, ensuring critical alerts from Datadog are delivered to the right on-call teams, facilitating rapid response.

Prerequisites

Before diving into the configuration, ensure you have the following:

AWS Account: With programmatic access and necessary permissions to create EKS clusters, IAM roles, and other AWS resources.
Terraform CLI: Installed (version 1.0+ recommended).
AWS CLI: Configured with your credentials.
Kubectl: Installed and configured to interact with your EKS cluster.
Helm CLI: Installed (for deploying Datadog Agent).
Datadog Account: With an API Key and Application Key.
PagerDuty Account: With an API Key and knowledge of how to create services and escalation policies.

Step-by-Step Terraform Implementation

1. Setting up Terraform Providers

First, define the necessary providers for AWS, Kubernetes, Helm, Datadog, and PagerDuty.

provider "aws" {
  region = "us-east-1"
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.main.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.main.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.main.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.main.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_key
}

data "aws_eks_cluster" "main" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "main" {
  name = var.eks_cluster_name
}
    

Ensure your var.eks_cluster_name, var.datadog_api_key, var.datadog_app_key, and var.pagerduty_api_key are securely managed, preferably via environment variables or a secrets manager.

2. Deploying Datadog Agent to AWS EKS

The Datadog Agent is crucial for collecting metrics, logs, and traces from your Kubernetes cluster. We'll use the Helm provider to deploy it.

Ready-to-Use Configuration: Datadog Agent, Monitors, and PagerDuty Integration

# --- main.tf ---

# EKS Cluster Data (assuming EKS is already provisioned or in another module)
data "aws_eks_cluster" "this" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "this" {
  name = var.eks_cluster_name
}

# --- Providers ---
provider "aws" {
  region = var.aws_region
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.this.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.this.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.this.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_key
}

# --- 1. Datadog Agent Deployment via Helm ---
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }
  set {
    name  = "datadog.site"
    value = "datadoghq.com" # or eu.datadoghq.com, us3.datadoghq.com, etc.
  }
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "kubeStateMetricsCore.enabled"
    value = "true"
  }
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }
  set {
    name  = "processAgent.enabled"
    value = "true"
  }
  set {
    name  = "apm.enabled"
    value = "true"
  }
  set {
    name  = "networkMonitoring.enabled"
    value = "true"
  }
  # More configurations for specific needs (e.g., EKS Fargate, Admission Controller)
}

# --- 2. PagerDuty Service and Escalation Policy ---
resource "pagerduty_escalation_policy" "devops_ep" {
  name      = "DevOps Team Escalation Policy"
  num_loops = 2
  rule {
    delay    = 5
    target {
      type = "user_reference"
      id   = var.pagerduty_devops_lead_user_id # Replace with actual PagerDuty User ID
    }
  }
  rule {
    delay    = 10
    target {
      type = "team_reference"
      id   = var.pagerduty_devops_team_id # Replace with actual PagerDuty Team ID
    }
  }
}

resource "pagerduty_service" "eks_monitoring_service" {
  name                     = "EKS Monitoring Service"
  auto_resolve_timeout_days = 1
  acknowledgement_timeout_minutes = 10
  escalation_policy        = pagerduty_escalation_policy.devops_ep.id
  alert_creation           = "create_alerts_and_incidents"
}

# --- 3. Datadog Monitors Integrated with PagerDuty ---

# Datadog Integration for PagerDuty (required to connect monitors to PD service)
resource "datadog_integration_pagerduty" "pagerduty_integration" {
  api_token = var.pagerduty_api_key
  services = [
    {
      service_name = pagerduty_service.eks_monitoring_service.name
      service_key  = pagerduty_service.eks_monitoring_service.integration.0.integration_key # assuming default integration
    }
  ]
}

# Example 1: EKS Node CPU Utilization Monitor
resource "datadog_monitor" "eks_node_cpu_high" {
  name                = "[EKS] High Node CPU Utilization on {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:system.cpu.idle{cluster_name:${var.eks_cluster_name}} by {host} < 20"
  message             = "EKS node {{host.name}} CPU utilization is over 80%. Investigate for potential resource contention or rogue processes. @webhook-pagerduty-${pagerduty_service.eks_monitoring_service.name}"
  tags                = ["environment:${var.environment}", "service:eks", "alert-type:performance"]
  monitor_threshold_windows {
    recovery_window = "15m"
    trigger_window  = "5m"
  }
  thresholds {
    critical = 80
    warning  = 70
  }
  notify_no_data      = false
  renotify_interval   = 0
  no_data_timeframe   = 20
  include_tags        = true
  require_full_window = true
  force_delete        = false
  # Explicitly link to PagerDuty service via tag
  # Datadog automatically recognizes '@webhook-pagerduty-SERVICE_NAME' for custom integration
}

# Example 2: EKS Pod Restarts Monitor
resource "datadog_monitor" "eks_pod_restarts" {
  name                = "[EKS] Excessive Pod Restarts in {{kube_namespace.name}} - {{kube_container.name}}"
  type                = "metric alert"
  query               = "sum(last_5m):kube_container_restarts{cluster_name:${var.eks_cluster_name}} by {kube_namespace,kube_container} > 5"
  message             = "Pod {{kube_pod.name}} in namespace {{kube_namespace.name}} is experiencing excessive restarts. This often indicates application instability. @webhook-pagerduty-${pagerduty_service.eks_monitoring_service.name}"
  tags                = ["environment:${var.environment}", "service:eks", "alert-type:availability"]
  monitor_threshold_windows {} # Default
  thresholds {
    critical = 5
  }
  notify_no_data      = false
  renotify_interval   = 0
  no_data_timeframe   = 20
  include_tags        = true
  require_full_window = true
  force_delete        = false
}

# --- variables.tf ---
variable "aws_region" {
  description = "AWS region for the EKS cluster."
  type        = string
  default     = "us-east-1"
}

variable "eks_cluster_name" {
  description = "Name of the existing EKS cluster."
  type        = string
}

variable "datadog_api_key" {
  description = "Datadog API Key."
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key."
  type        = string
  sensitive   = true
}

variable "pagerduty_api_key" {
  description = "PagerDuty API Key."
  type        = string
  sensitive   = true
}

variable "pagerduty_devops_lead_user_id" {
  description = "PagerDuty User ID for the DevOps lead."
  type        = string
  sensitive   = true
}

variable "pagerduty_devops_team_id" {
  description = "PagerDuty Team ID for the DevOps team."
  type        = string
  sensitive   = true
}

variable "environment" {
  description = "Deployment environment (e.g., dev, staging, prod)."
  type        = string
  default     = "dev"
}
    

Explanation of the Configuration:

Helm Release: Deploys the Datadog Agent, configuring it with your API/App keys and enabling various features like log collection, process monitoring, and APM. Adjust datadog.site based on your Datadog region.
PagerDuty Resources: Defines an escalation_policy to dictate who gets alerted and in what order, and a service that incidents will be routed to.
Datadog Monitors: Creates two example monitors: one for high CPU utilization on EKS nodes and another for excessive pod restarts. These are critical for detecting performance issues and application instability.
PagerDuty Integration: The datadog_integration_pagerduty resource connects Datadog to your PagerDuty service. The @webhook-pagerduty-SERVICE_NAME syntax in the monitor message is crucial for routing alerts to the specific PagerDuty service configured via the Datadog integration.
Variables: All sensitive keys and cluster-specific details are parameterized for flexibility and security.

Deployment and Verification

Deployment Steps:

Save the code: Save the code blocks into main.tf and variables.tf files in your Terraform project directory.
Initialize Terraform: Open your terminal in the project directory and run terraform init to download providers.
Set Variables: Provide your variable values. You can do this via terraform.tfvars file, environment variables (TF_VAR_...), or command-line arguments. For sensitive data, environment variables are recommended.
export TF_VAR_eks_cluster_name="your-eks-cluster-name" export TF_VAR_datadog_api_key="your-datadog-api-key" export TF_VAR_datadog_app_key="your-datadog-app-key" export TF_VAR_pagerduty_api_key="your-pagerduty-api-key" export TF_VAR_pagerduty_devops_lead_user_id="PXXXXXXXXXXXXX" # e.g., P0123456789ABCDEF export TF_VAR_pagerduty_devops_team_id="TXXXXXXXXXXXXX" # e.g., T0123456789ABCDEF
Review Plan: Execute terraform plan to see the changes Terraform will apply.
Apply Changes: If the plan is satisfactory, run terraform apply and type yes to confirm.

Verification Steps:

Datadog Agent: Check your EKS cluster with kubectl get pods -n datadog. You should see Datadog Agent pods running.
Datadog Dashboard: Log in to your Datadog account. Navigate to "Infrastructure" -> "Containers" -> "Kubernetes" or "Hosts" to confirm metrics are flowing from your EKS cluster. Check "Monitors" -> "Manage Monitors" to see your newly created monitors.
PagerDuty: Log in to PagerDuty. Verify the "EKS Monitoring Service" and "DevOps Team Escalation Policy" are created under "Services" and "Configuration", respectively.
Trigger a Test Alert: While not covered directly by this Terraform, you could intentionally create a high CPU load on a node or force a pod restart to test the end-to-end alerting.

Advanced Considerations and Best Practices

Granular Permissions: Always apply the principle of least privilege. Ensure the IAM role associated with your EKS worker nodes has only the necessary permissions for Datadog to collect data.
Log Management: Beyond container logs, consider shipping AWS service logs (CloudTrail, VPC Flow Logs, ELB Access Logs) to Datadog for a complete picture.
Synthetics and RUM: Augment your EKS monitoring with Datadog Synthetics for proactive external checks and Real User Monitoring (RUM) for frontend visibility.
Costs Optimization: Monitor Datadog ingestion volumes to manage costs. Fine-tune what metrics and logs are collected, and configure retention policies.
Advanced PagerDuty Routing: Utilize PagerDuty's event orchestration capabilities to enrich, suppress, or route alerts based on specific criteria before they trigger an incident.
Terraform Modules: For larger deployments, encapsulate your Datadog and PagerDuty configurations into reusable Terraform modules.

Troubleshooting Common Issues

Terraform apply fails with Kubernetes error: Ensure your AWS CLI context and EKS cluster authentication are correctly set up. The Terraform Kubernetes provider relies on these. Run aws eks update-kubeconfig --name <your-eks-cluster-name> --region <your-region>.
Datadog Agent pods not running: Check kubectl describe pod <datadog-agent-pod-name> -n datadog for events and logs. Common issues include incorrect API keys or insufficient IAM permissions for the EKS worker nodes.
No data in Datadog: Verify the Datadog API and App keys. Ensure the datadog.site in the Helm chart is correct for your Datadog region. Check the Datadog Agent logs for connectivity issues to the Datadog ingest endpoints.
PagerDuty alerts not firing: Double-check the @webhook-pagerduty-SERVICE_NAME tag in your Datadog monitor message. The SERVICE_NAME must exactly match the name of the PagerDuty service created in Datadog's integration settings. Also, ensure the PagerDuty API key used by the Datadog integration has the necessary permissions.

Conclusion

Automating your AWS EKS observability and incident response with Terraform, Datadog Monitoring, and PagerDuty Alerting creates a powerful, resilient, and scalable operational framework. This guide has provided the foundational knowledge and a ready-to-use configuration to kickstart your journey towards integrated cloud-native operations. By embracing IaC for your monitoring and alerting infrastructure, you empower your DevOps teams to respond faster to incidents, ensure application reliability, and focus on delivering business value.

CloudStack YAML