Terraform AWS EKS with Datadog Observability and PagerDuty Incident Response: A Comprehensive Guide

In the rapidly evolving landscape of cloud-native infrastructure, robust deployment automation, deep observability, and efficient incident response are paramount. This guide provides a comprehensive, technical walkthrough on how to provision an Amazon Elastic Kubernetes Service (AWS EKS) cluster using Terraform, integrate Datadog for unparalleled observability, and connect PagerDuty for streamlined incident management. Embrace a future where your Kubernetes infrastructure is not only scalable and resilient but also proactively monitored and swiftly managed.

Architecture Pro-Tip: Always compartmentalize your Terraform configuration. Use modules for common resources like VPCs, EKS clusters, and IAM roles. This promotes reusability, reduces complexity, and significantly improves maintainability for your Infrastructure as Code (IaC). For production environments, consider dedicated AWS accounts or organizational units (OUs) for different stages (dev, staging, prod) to enforce stronger security boundaries and resource isolation.

1. The Modern DevOps Stack: An Overview

Building a resilient and observable Kubernetes platform requires a synergy of powerful tools. Here's why this specific stack is a game-changer:

Terraform: Infrastructure as Code (IaC)

Terraform, by HashiCorp, is the industry standard for defining, provisioning, and managing cloud infrastructure using a declarative configuration language. It allows you to reliably build, change, and version infrastructure safely and efficiently across any cloud provider, including AWS.

AWS EKS: Managed Kubernetes Service

Amazon EKS is a managed Kubernetes service that makes it easy to deploy, manage, and scale containerized applications using Kubernetes on AWS. EKS handles the heavy lifting of managing the Kubernetes control plane, offering high availability and robust security integrations with AWS services.

Datadog: Unified Observability Platform

Datadog provides end-to-end observability for your entire stack. It unifies metrics, traces, and logs from your applications, servers, and cloud infrastructure, offering real-time visibility and powerful analytics. For EKS, Datadog integrates deeply to monitor cluster health, node performance, pod metrics, and application logs.

PagerDuty: Intelligent Incident Response

PagerDuty is a leading digital operations management platform that empowers teams to prevent and resolve critical incidents quickly. By integrating with Datadog, PagerDuty automatically routes alerts to the right teams, ensuring timely notifications and structured incident workflows.

2. Prerequisites

Before you begin, ensure you have the following tools and accounts set up:

AWS Account: With necessary IAM permissions to create EKS clusters, VPCs, and related resources.
Terraform CLI: Installed locally (version 1.0+ recommended).
AWS CLI: Configured with credentials for your AWS account.
kubectl: Installed to interact with the Kubernetes cluster.
Datadog Account: With an API Key and Application Key.
PagerDuty Account: With administrator access to create services and integration keys.

3. Terraform AWS EKS Cluster Provisioning

We'll use the official terraform-aws-modules/eks/aws module for a streamlined EKS deployment. This module abstracts away much of the complexity, handling VPC, IAM, and EKS control plane creation.

Core Components: VPC, IAM, and EKS

A secure and functional EKS cluster requires a well-architected VPC, appropriate IAM roles for the EKS control plane and worker nodes, and the EKS cluster itself, including its node groups.

VPC Configuration: A dedicated VPC with public and private subnets is crucial for EKS. Worker nodes typically reside in private subnets, while load balancers can expose services in public subnets.
IAM Roles: EKS requires specific IAM roles for its control plane to interact with other AWS services (like EC2 for nodes, EBS for volumes) and for worker nodes to join the cluster.
EKS Cluster and Node Groups: Define the Kubernetes version, desired instance types for worker nodes, scaling configurations (min/max/desired), and networking settings.

4. Integrating Datadog Observability

Datadog's Kubernetes integration is achieved primarily through the Datadog Agent, deployed as a DaemonSet across your EKS worker nodes. This agent collects metrics, logs, and traces from your cluster, nodes, and running applications.

Datadog Agent Deployment

The Datadog Agent is usually deployed via a Helm chart or Kubernetes manifests. For EKS, using the Helm chart is the recommended and most flexible approach. You'll need your Datadog API Key and Application Key for authentication.

Key Configuration Points

API Key & Application Key: Essential for the agent to send data to your Datadog account. Store these securely, ideally using AWS Secrets Manager or Vault.
Cluster Agent: For larger clusters, deploy the Datadog Cluster Agent to centralize collection of cluster-level metrics and reduce resource consumption on individual nodes.
APM (Tracing) & Log Collection: Enable these features in the Helm chart values to get full-stack visibility.
RBAC: The Datadog Agent requires specific Kubernetes RBAC permissions to collect data. The Helm chart handles this automatically.

5. PagerDuty Incident Response Configuration

Integrating PagerDuty ensures that critical alerts from Datadog translate into actionable incidents, routed to the correct on-call teams. The most common pattern is to integrate Datadog directly with PagerDuty.

Creating a PagerDuty Service and Integration

In PagerDuty, you'll create a new service and add a "Datadog" integration. This will generate an Integration Key (also known as a Routing Key or Integration URL) that Datadog will use to send events.

Configuring Datadog to Trigger PagerDuty Incidents

Within Datadog, you'll set up your monitors to use the PagerDuty integration. When a monitor's alert condition is met, Datadog will send an event to PagerDuty, triggering an incident based on the configured routing policies.

Datadog Integration: Configure the PagerDuty integration in Datadog by navigating to Integrations -> PagerDuty.
Monitor Creation: Create Datadog monitors (e.g., high CPU utilization on EKS nodes, pod restarts, critical application errors).
Notification Configuration: In the monitor's notification section, select the PagerDuty integration and specify the service or integration key.

6. Ready-to-Use Terraform Configuration

Below is a simplified Terraform configuration demonstrating the core concepts. This example includes a basic EKS cluster and the necessary components for Datadog integration via Helm. Remember to replace placeholder values with your actual data and manage sensitive information securely.

main.tf

resource "aws_vpc" "eks_vpc" {
  cidr_block = "10.0.0.0/16"
  tags = { Name = "eks-datadog-pagerduty-vpc" }
}

resource "aws_subnet" "public" {
  count             = 2
  vpc_id            = aws_vpc.eks_vpc.id
  cidr_block        = "10.0.${10 + count.index}.0/24"
  availability_zone = data.aws_availability_zones.available.names[count.index]
  map_public_ip_on_launch = true
  tags = {
    Name = "eks-public-subnet-${count.index}"
    "kubernetes.io/cluster/eks-cluster" = "shared"
    "kubernetes.io/role/elb"            = "1"
  }
}

resource "aws_subnet" "private" {
  count             = 2
  vpc_id            = aws_vpc.eks_vpc.id
  cidr_block        = "10.0.${20 + count.index}.0/24"
  availability_zone = data.aws_availability_zones.available.names[count.index]
  tags = {
    Name = "eks-private-subnet-${count.index}"
    "kubernetes.io/cluster/eks-cluster" = "shared"
    "kubernetes.io/role/internal-elb"   = "1"
  }
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.1"

  cluster_name    = "eks-datadog-pagerduty"
  cluster_version = "1.28"

  vpc_id     = aws_vpc.eks_vpc.id
  subnet_ids = concat(aws_subnet.public.*.id, aws_subnet.private.*.id)

  eks_managed_node_groups = {
    default = {
      instance_types = ["t3.medium"]
      desired_size   = 2
      min_size       = 1
      max_size       = 3
      ami_type       = "AL2_x86_64" # Amazon Linux 2
    }
  }

  tags = {
    Environment = "Dev"
    Project     = "Observability"
  }
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # or a dedicated 'datadog' namespace
  version    = "2.35.0" # Use a recent stable version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "kubeStateMetrics.enabled"
    value = true
  }

  set {
    name  = "clusterAgent.enabled"
    value = true
  }

  set {
    name  = "clusterAgent.metricsProvider.enabled"
    value = true
  }

  # Enable APM and Log collection
  set {
    name  = "datadog.apm.enabled"
    value = true
  }
  set {
    name  = "datadog.logs.enabled"
    value = true
  }
  set {
    name  = "datadog.logs.containerCollectAll"
    value = true
  }

  depends_on = [module.eks.kubeconfig]
}

# --- Datadog PagerDuty Integration (managed outside Terraform or via Datadog Provider) ---
# For demonstration, we assume Datadog monitors are configured to alert PagerDuty manually
# or through the Datadog Terraform Provider if you manage Datadog resources via IaC.

# Example of configuring a PagerDuty service via Terraform
# You would need the PagerDuty Terraform Provider for this.
# resource "pagerduty_service" "eks_incidents" {
#   name        = "EKS Cluster Incidents"
#   description = "Critical alerts from EKS via Datadog"
#   escalation_policy = pagerduty_escalation_policy.primary.id
# }

# resource "pagerduty_service_integration" "datadog_integration" {
#   name        = "Datadog EKS Integration"
#   service     = pagerduty_service.eks_incidents.id
#   type        = "datadog_inbound_integration"
#   integration_key = "..." # This key would be used in Datadog monitors
# }
    

variables.tf

variable "region" {
  description = "AWS region"
  type        = string
  default     = "us-east-1"
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}
    

outputs.tf

output "eks_cluster_name" {
  description = "Name of the EKS cluster"
  value       = module.eks.cluster_name
}

output "kubeconfig" {
  description = "Kubeconfig for accessing the EKS cluster"
  value       = module.eks.kubeconfig
  sensitive   = true
}
    

versions.tf

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
  }
}

provider "aws" {
  region = var.region
}

data "aws_availability_zones" "available" {
  state = "available"
}
    

7. Deployment and Verification

With your Terraform configuration ready, deploying your EKS cluster and integrating observability is straightforward:

Terraform Workflow

Initialize: Run terraform init in your project directory to download necessary providers and modules.
Plan: Execute terraform plan -var="datadog_api_key=YOUR_DD_API_KEY" -var="datadog_app_key=YOUR_DD_APP_KEY" to review the changes Terraform will apply. Replace placeholders with your actual keys. Consider using environment variables or a terraform.tfvars file for sensitive data.
Apply: If the plan looks good, run terraform apply -var="datadog_api_key=YOUR_DD_API_KEY" -var="datadog_app_key=YOUR_DD_APP_KEY" and confirm with yes to provision the resources. This will take some time as EKS clusters provision.

Verifying the Deployment

EKS Cluster: Once terraform apply completes, retrieve the kubeconfig: aws eks update-kubeconfig --name eks-datadog-pagerduty --region us-east-1 (adjust region if needed). Then, verify cluster access: kubectl get nodes. You should see your worker nodes.
Datadog Agent: Check that the Datadog Agent pods are running: kubectl get pods -l app=datadog -n default (or your chosen namespace). Navigate to your Datadog dashboard -> Infrastructure -> Hosts to confirm your EKS nodes are reporting data. Check the Kubernetes overview dashboard for cluster metrics.
PagerDuty Integration: In Datadog, create a simple test monitor (e.g., alert if CPU usage is above 0% for 1 minute on a non-existent host) and configure it to notify PagerDuty. Trigger the alert (or wait for a real one) and verify an incident is created in PagerDuty.

8. Best Practices for Production EKS

For a production-grade EKS setup, consider these additional best practices:

Security: Implement Network Policies, use IAM Roles for Service Accounts (IRSA), restrict API server access, and regularly update Kubernetes versions.
Cost Optimization: Utilize Karpenter for intelligent cluster autoscaling, leverage EC2 Spot Instances where appropriate, and right-size your worker nodes.
High Availability: Distribute worker nodes across multiple Availability Zones, ensure your applications are designed for redundancy, and use highly available control planes (managed by EKS).
Continuous Integration/Continuous Deployment (CI/CD): Integrate your Terraform code into a CI/CD pipeline for automated, version-controlled deployments.
State Management: Always use a remote backend (like AWS S3 with DynamoDB locking) for your Terraform state to enable team collaboration and prevent state corruption.
Advanced Datadog Monitoring: Set up custom dashboards, sophisticated monitors with composite conditions, service-level objectives (SLOs), and anomaly detection.
PagerDuty Escalation Policies: Define granular escalation policies, on-call schedules, and incident playbooks to ensure critical issues are addressed efficiently.

9. Troubleshooting and FAQs

Common Terraform Issues

Permissions Errors: Ensure your AWS CLI credentials have sufficient IAM permissions for all resources Terraform attempts to create.
State Locking: If using S3 for state, ensure DynamoDB locking is configured to prevent concurrent modifications.
Module Versions: Always pin module versions to avoid unexpected breaking changes.

Datadog Agent Connectivity

API/APP Keys: Double-check that your Datadog API and Application keys are correct and securely passed to the Helm chart.
Network Connectivity: Ensure your EKS nodes (or specifically the Datadog Agent pods) have outbound access to Datadog's ingest endpoints (typically port 443).
RBAC Issues: Inspect Datadog Agent pod logs (kubectl logs -n default) for permission-related errors. The Helm chart usually sets this correctly, but custom configurations can introduce issues.

PagerDuty Incident Not Triggering

Datadog Monitor Condition: Verify the Datadog monitor's alert condition is actually met.
PagerDuty Integration Key: Confirm the PagerDuty integration key used in the Datadog monitor's notification settings is correct and belongs to the intended PagerDuty service.
Datadog-PagerDuty Connection: Check the Datadog Integrations page for PagerDuty to ensure it shows a healthy connection status.

10. Conclusion

By leveraging Terraform to provision AWS EKS, integrating Datadog for comprehensive observability, and connecting PagerDuty for intelligent incident response, you establish a robust, automated, and highly responsive cloud-native environment. This integrated approach not only accelerates your development cycles but also significantly improves the reliability and operational efficiency of your Kubernetes deployments, allowing your teams to focus on innovation rather than firefighting.

CloudStack YAML