Terraform AWS EKS: Production-Ready Clusters with Datadog Observability & PagerDuty Integration

In the dynamic landscape of cloud-native applications, Kubernetes has emerged as the de-facto standard for container orchestration. AWS EKS (Elastic Kubernetes Service) provides a robust, managed Kubernetes experience, but achieving true production readiness requires more than just launching a cluster. This comprehensive guide will walk you through building a resilient, scalable, and observable AWS EKS cluster using Terraform, integrated with Datadog for deep observability and PagerDuty for proactive incident management.

Architecture Pro-Tip:

Always design your EKS clusters with high availability and scalability in mind from day one. This means deploying across multiple Availability Zones, utilizing managed node groups or Fargate for compute, and establishing robust IAM roles for fine-grained access. Prioritize infrastructure as code (IaC) for consistent deployments and simplify disaster recovery. Observability and incident response are not afterthoughts; they are foundational pillars of a production-ready system.

Why Production-Ready AWS EKS?

A production-grade EKS cluster must meet stringent requirements beyond basic functionality:

High Availability & Reliability: Spanning multiple Availability Zones, redundant control plane, and robust node management.
Scalability: Ability to seamlessly scale compute resources (nodes) and application pods based on demand.
Security: Strong IAM policies, network segmentation, secure access to the API server, and regular vulnerability patching.
Observability: Comprehensive monitoring, logging, and tracing to understand system health and troubleshoot issues quickly.
Incident Management: Proactive alerting and streamlined on-call processes to respond to critical events.
Infrastructure as Code (IaC): Reproducible deployments, version control, and auditability.

The Terraform Advantage for EKS

Terraform, HashiCorp's open-source IaC tool, allows you to define and provision your entire cloud infrastructure using declarative configuration files. For EKS, Terraform offers:

Consistency: Deploy identical environments (dev, staging, production) with minimal effort.
Version Control: Treat your infrastructure like application code, enabling collaboration, peer reviews, and rollback capabilities.
Modularity: Break down complex infrastructure into reusable modules.
Reduced Manual Errors: Automate provisioning, reducing human error.
Dependency Management: Terraform intelligently handles resource dependencies, ensuring resources are created and updated in the correct order.

Datadog Observability for EKS

Datadog provides a unified platform for monitoring, logging, and tracing, essential for understanding the health and performance of your EKS clusters and the applications running within them.

Metrics: Collects host-level, container-level, and Kubernetes-specific metrics (pods, deployments, services, nodes) out-of-the-box.
Logs: Aggregates logs from all your containers, nodes, and EKS control plane components, enabling centralized analysis.
APM (Application Performance Monitoring): Distributed tracing helps visualize requests flowing through microservices, identifying bottlenecks.
Network Performance: Monitors network traffic within the cluster and to external services.
Security Monitoring: Detects threats and misconfigurations.
Custom Dashboards & Alerts: Build tailored views of your infrastructure and set up intelligent alerts.

PagerDuty Integration for Incident Management

While Datadog tells you what's happening, PagerDuty tells you who needs to know and ensures timely incident response. Integrating PagerDuty with Datadog allows for:

Automated Alerting: Critical alerts from Datadog automatically trigger incidents in PagerDuty.
On-Call Scheduling: PagerDuty manages on-call rotations, ensuring the right person is notified.
Escalation Policies: Incidents are escalated through predefined channels if not acknowledged promptly.
Incident Tracking & Post-Mortems: Centralized incident management for better insights and continuous improvement.

Prerequisites

Before you begin, ensure you have the following:

An AWS Account with administrative access.
Terraform CLI (v1.0+) installed.
AWS CLI configured with appropriate credentials.
kubectl installed.
Datadog Account with an API Key and Application Key.
PagerDuty Account with an API token and a service configured.

Terraform Project Structure

A recommended project structure for modularity and maintainability:

.
├── main.tf              # Main configuration: providers, modules
├── variables.tf         # Input variables
├── outputs.tf           # Output values
├── versions.tf          # Terraform & provider versions
├── modules/
│   ├── vpc/             # VPC, subnets, NAT Gateways, Internet Gateway
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── outputs.tf
│   ├── eks/             # EKS cluster, node groups, IAM roles
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── outputs.tf
│   ├── addons/          # Core EKS addons (CNI, CoreDNS, kube-proxy)
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── outputs.tf
│   └── datadog/         # Datadog agent deployment, monitors, PagerDuty integration
│       ├── main.tf
│       ├── variables.tf
│       └── outputs.tf
└── backend.tf           # S3 backend for Terraform state (optional but recommended)
    

Core AWS EKS Cluster Configuration with Terraform

1. VPC and Networking (`modules/vpc/main.tf`)

An EKS cluster requires a dedicated VPC with public and private subnets, NAT Gateways, and an Internet Gateway. Use the official AWS VPC module or create resources directly.

2. EKS Cluster (`modules/eks/main.tf`)

Define the EKS cluster, its control plane, and associated IAM roles. Ensure the cluster IAM role has the necessary permissions (e.g., arn:aws:iam::aws:policy/AmazonEKSClusterPolicy, arn:aws:iam::aws:policy/AmazonEKSVPCResourceController).

3. EKS Node Groups (Managed)

For worker nodes, AWS Managed Node Groups are highly recommended for ease of management, automated scaling, and patching. Define separate node groups for different workloads if necessary.

4. EKS Addons (`modules/addons/main.tf`)

Critical EKS addons like aws-vpc-cni, coredns, and kube-proxy should be managed and kept up-to-date.

Integrating Datadog and PagerDuty

1. Datadog Agent Deployment (`modules/datadog/main.tf`)

The Datadog Agent runs as a DaemonSet on each EKS worker node to collect metrics, logs, and traces. You'll typically deploy this using a Kubernetes manifest or Helm chart, which can be managed by Terraform's kubernetes_manifest or helm_release resources, respectively.

2. Datadog Monitors

Define Datadog monitors in Terraform (using the datadog_monitor resource) for key EKS health indicators:

Node CPU/Memory Utilization
Pod Status (CrashLoopBackOff, Pending)
EKS Control Plane Latency
Network Errors
Application-specific metrics (e.g., HTTP error rates, latency).

3. PagerDuty Integration with Datadog

Set up the Datadog-PagerDuty integration using Terraform's datadog_integration_pagerduty resource. This links your Datadog alerts to PagerDuty services.

Comprehensive Terraform Configuration Example

This example demonstrates a simplified structure of how you might combine these elements. Remember to replace placeholders and expand modules for a production environment.

`main.tf` (Root Configuration)

# main.tf

# Configure the AWS provider
provider "aws" {
  region = var.aws_region
}

# Configure the Datadog provider
provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

# Backend for Terraform state
terraform {
  backend "s3" {
    bucket         = "your-terraform-state-bucket"
    key            = "eks-production/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "your-terraform-lock-table"
  }
}

# Source the VPC module
module "vpc" {
  source = "./modules/vpc"

  project_name = var.project_name
  aws_region   = var.aws_region
  vpc_cidr     = "10.0.0.0/16"
  azs          = ["${var.aws_region}a", "${var.aws_region}b", "${var.aws_region}c"]
}

# Source the EKS cluster module
module "eks" {
  source = "./modules/eks"

  project_name    = var.project_name
  aws_region      = var.aws_region
  vpc_id          = module.vpc.vpc_id
  private_subnets = module.vpc.private_subnets
  public_subnets  = module.vpc.public_subnets
}

# Source the EKS Addons module
module "eks_addons" {
  source = "./modules/addons"

  cluster_name    = module.eks.cluster_name
  cluster_version = module.eks.cluster_version
  oidc_provider_arn = module.eks.oidc_provider_arn
}

# Source the Datadog integration module
module "datadog_integration" {
  source = "./modules/datadog"

  project_name         = var.project_name
  aws_region           = var.aws_region
  eks_cluster_name     = module.eks.cluster_name
  kubeconfig_filepath  = var.kubeconfig_filepath # Path to local kubeconfig
  datadog_api_key      = var.datadog_api_key
  datadog_app_key      = var.datadog_app_key
  pagerduty_api_token  = var.pagerduty_api_token
  pagerduty_service_id = var.pagerduty_service_id
}
    

`modules/vpc/main.tf` (Example)

# modules/vpc/main.tf
resource "aws_vpc" "main" {
  cidr_block = var.vpc_cidr
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name = "${var.project_name}-vpc"
  }
}

resource "aws_internet_gateway" "gw" {
  vpc_id = aws_vpc.main.id

  tags = {
    Name = "${var.project_name}-igw"
  }
}

# Public Subnets
resource "aws_subnet" "public" {
  count             = length(var.azs)
  vpc_id            = aws_vpc.main.id
  cidr_block        = cidrsubnet(var.vpc_cidr, 8, count.index) # Example: 10.0.0.0/24, 10.0.1.0/24...
  availability_zone = var.azs[count.index]
  map_public_ip_on_launch = true

  tags = {
    Name = "${var.project_name}-public-subnet-${count.index}"
    "kubernetes.io/cluster/${var.project_name}-eks" = "shared"
    "kubernetes.io/role/elb"                      = "1"
  }
}

# Private Subnets
resource "aws_subnet" "private" {
  count             = length(var.azs)
  vpc_id            = aws_vpc.main.id
  cidr_block        = cidrsubnet(var.vpc_cidr, 8, count.index + length(var.azs)) # Example: 10.0.3.0/24, 10.0.4.0/24...
  availability_zone = var.azs[count.index]

  tags = {
    Name = "${var.project_name}-private-subnet-${count.index}"
    "kubernetes.io/cluster/${var.project_name}-eks" = "shared"
    "kubernetes.io/role/internal-elb"             = "1"
  }
}

# NAT Gateways
resource "aws_eip" "nat" {
  count = length(var.azs)
  vpc   = true
  tags = {
    Name = "${var.project_name}-nat-eip-${count.index}"
  }
}

resource "aws_nat_gateway" "nat" {
  count         = length(var.azs)
  allocation_id = aws_eip.nat[count.index].id
  subnet_id     = aws_subnet.public[count.index].id

  tags = {
    Name = "${var.project_name}-nat-gw-${count.index}"
  }
  depends_on = [aws_internet_gateway.gw]
}

# Route Tables for private subnets
resource "aws_route_table" "private" {
  count  = length(var.azs)
  vpc_id = aws_vpc.main.id

  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.nat[count.index].id
  }

  tags = {
    Name = "${var.project_name}-private-rt-${count.index}"
  }
}

resource "aws_route_table_association" "private" {
  count          = length(var.azs)
  subnet_id      = aws_subnet.private[count.index].id
  route_table_id = aws_route_table.private[count.index].id
}
    

`modules/eks/main.tf` (Example)

# modules/eks/main.tf

# EKS Cluster IAM Role
resource "aws_iam_role" "eks_cluster" {
  name = "${var.project_name}-eks-cluster-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "eks.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eks_cluster.name
}

resource "aws_iam_role_policy_attachment" "eks_vpc_resource_controller_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSVPCResourceController"
  role       = aws_iam_role.eks_cluster.name
}

# EKS Cluster
resource "aws_eks_cluster" "main" {
  name     = "${var.project_name}-eks"
  role_arn = aws_iam_role.eks_cluster.arn
  version  = "1.28" # Or your desired version

  vpc_config {
    subnet_ids         = var.private_subnets
    endpoint_private_access = true
    endpoint_public_access  = true # Set to false in truly private environments
    public_access_cidrs     = ["0.0.0.0/0"]
  }

  tags = {
    Name = "${var.project_name}-eks-cluster"
  }

  depends_on = [
    aws_iam_role_policy_attachment.eks_cluster_policy,
    aws_iam_role_policy_attachment.eks_vpc_resource_controller_policy,
  ]
}

# EKS Node Group IAM Role
resource "aws_iam_role" "eks_node_group" {
  name = "${var.project_name}-eks-nodegroup-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_worker_node_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.eks_node_group.name
}

resource "aws_iam_role_policy_attachment" "eks_cni_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.eks_node_group.name
}

resource "aws_iam_role_policy_attachment" "ec2_container_registry_readonly_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.eks_node_group.name
}

# EKS Managed Node Group
resource "aws_eks_node_group" "primary" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "${var.project_name}-primary-nodegroup"
  node_role_arn   = aws_iam_role.eks_node_group.arn
  subnet_ids      = var.private_subnets
  instance_types  = ["t3.medium"] # Or t3.large, m5.large etc.
  disk_size       = 20
  capacity_type   = "ON_DEMAND"

  scaling_config {
    desired_size = 3
    max_size     = 5
    min_size     = 2
  }

  update_config {
    max_unavailable = 1
  }

  labels = {
    environment = "production"
    purpose     = "general"
  }

  tags = {
    Name = "${var.project_name}-primary-nodegroup"
  }

  depends_on = [
    aws_iam_role_policy_attachment.eks_worker_node_policy,
    aws_iam_role_policy_attachment.eks_cni_policy,
    aws_iam_role_policy_attachment.ec2_container_registry_readonly_policy,
  ]
}
    

`modules/datadog/main.tf` (Example)

This example shows how to deploy the Datadog Agent using the Helm provider and define a basic monitor linked to PagerDuty.

# modules/datadog/main.tf

# Configure the Kubernetes provider using the generated kubeconfig
resource "null_resource" "configure_kubectl" {
  triggers = {
    always_run = timestamp()
  }

  provisioner "local-exec" {
    command = "aws eks update-kubeconfig --name ${var.eks_cluster_name} --region ${var.aws_region} --file ${var.kubeconfig_filepath}"
  }
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
  config_path            = var.kubeconfig_filepath # Ensure this path is correct after update-kubeconfig
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.cluster.token
    config_path            = var.kubeconfig_filepath
  }
}

# Data source for EKS cluster details
data "aws_eks_cluster" "cluster" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = var.eks_cluster_name
}

# Deploy Datadog Agent using Helm
resource "helm_release" "datadog_agent" {
  name             = "datadog"
  repository       = "https://helm.datadoghq.com"
  chart            = "datadog"
  namespace        = "datadog"
  create_namespace = true
  version          = "2.33.0" # Or the latest stable version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    secret = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    secret = true
  }

  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "agents.enabled"
    value = "true"
  }

  set {
    name  = "logs.enabled"
    value = "true"
  }

  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }

  set {
    name  = "processAgent.enabled"
    value = "true"
  }

  set {
    name  = "apm.enabled"
    value = "true"
  }

  set {
    name = "clusterName"
    value = var.eks_cluster_name
  }

  depends_on = [null_resource.configure_kubectl]
}

# Datadog PagerDuty Integration (requires a PagerDuty API token and service_id)
resource "datadog_integration_pagerduty" "main" {
  api_token = var.pagerduty_api_token
}

# Example Datadog Monitor for CPU Utilization
resource "datadog_monitor" "high_cpu_alert" {
  name                = "[EKS - ${var.project_name}] High Node CPU Usage"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.eks_cluster_name}} by {host} > 80"
  message             = "Node {{host.name}} in EKS cluster ${var.eks_cluster_name} has high CPU usage. @pagerduty-your-service-name"
  escalation_message  = "CPU usage remains high. Paging on-call team."
  # Optionally, add specific Datadog integration for PagerDuty here, e.g. @pagerduty-EKS-Service
  tags                = ["environment:production", "service:eks", "alert:cpu"]
  new_group_delay     = 60
  new_host_delay      = 300
  notify_audit        = false
  notify_no_data      = false
  renotify_interval   = 0
  no_data_timeframe   = 20
  timeout_h           = 0
  include_tags        = true

  # Link to the PagerDuty service via Datadog's integration handle
  # Make sure "your-service-name" matches the integration name you set up in Datadog UI
  # or use `{{#is_match '{{integration_name}}' 'pagerduty'}}@pagerduty-{{integration_name}}-{{service_name}}{{/is_match}}` if using service key
  # For simplicity with direct integration, typically you would reference the integration.
  # If using a PagerDuty service key, ensure it's configured in Datadog.
  # For this example, we'll assume a direct integration setup where @pagerduty is sufficient with a configured service.
  # For specific service integration, you'd configure it in the Datadog UI and reference its handle.

  # In a production setup, you would use a dedicated notification channel for PagerDuty,
  # referencing the integration by its name configured in Datadog UI.
  # For example: `message = "... @pagerduty-EKS-Oncall"` assuming you set up an integration
  # named "EKS-Oncall" in Datadog's PagerDuty integration settings for a specific PagerDuty service.

  # The actual PagerDuty service integration in Datadog is typically done via the Datadog UI
  # where you connect a PagerDuty service to Datadog's integration.
  # Then in the monitor message, you use the @pagerduty-integration-name.
  # For the purpose of this guide, the `datadog_integration_pagerduty` resource
  # ensures the base integration is present.
}
    

Deployment Steps

Once your Terraform files are structured and configured, deploy your EKS cluster:

Initialize Terraform: Navigate to your root project directory and run terraform init. This downloads providers and sets up the backend.
Review the Plan: Run terraform plan to see what resources Terraform will create, modify, or destroy. Review this carefully.
Apply the Configuration: Execute terraform apply. Confirm the changes by typing yes when prompted. This process can take 15-30 minutes for an EKS cluster.
Verify Kubeconfig: The null_resource in the Datadog module should update your kubeconfig. Verify with kubectl get nodes.

Post-Deployment Validation

After successful deployment:

Kubernetes: Use kubectl get nodes, kubectl get pods -n datadog to confirm nodes are ready and Datadog agents are running.
Datadog Dashboards: Log into your Datadog account. You should see data flowing from your EKS cluster in the "Kubernetes" dashboard and your custom dashboards.
Datadog Monitors: Verify that your configured monitors are active and in the expected state.
PagerDuty Integration: Trigger a test alert from Datadog or manually adjust a threshold to confirm a PagerDuty incident is created.

Advanced Considerations

EKS Fargate: For serverless Kubernetes, consider using EKS Fargate profiles for certain workloads to eliminate node management overhead.
Cluster Autoscaler / Karpenter: Implement a cluster autoscaler (or Karpenter for more advanced dynamic provisioning) to automatically adjust the number of nodes based on pod demand.
IRSA (IAM Roles for Service Accounts): Use IRSA to grant AWS IAM permissions directly to Kubernetes service accounts, improving security by avoiding shared node IAM roles.
Secrets Management: Integrate with AWS Secrets Manager or HashiCorp Vault for secure management of sensitive application data.
GitOps with Argo CD / Flux CD: Implement GitOps principles for deploying and managing applications within your EKS cluster, leveraging Git as the single source of truth.
Security Hardening: Implement network policies, Pod Security Standards (PSS), and regular security audits.

Troubleshooting and Best Practices

Terraform State: Always use a remote backend (like S3 with DynamoDB locking) for Terraform state to enable collaboration and prevent corruption.
IAM Permissions: EKS relies heavily on correct IAM permissions. Double-check roles for the EKS cluster, node groups, and service accounts.
Networking Issues: Verify VPC CNI logs, security group rules, and network ACLs if pods can't communicate.
Datadog Agent Connectivity: Ensure network connectivity from nodes to Datadog endpoints and correct API/App keys. Check agent logs for errors.
Modularize: Break down your Terraform code into small, reusable modules.
Test Thoroughly: Use separate environments (dev, staging) for testing changes before deploying to production.
Automate Updates: Plan for regular updates of EKS versions, addons, and worker nodes.

Conclusion

Building a production-ready AWS EKS cluster is a multi-faceted endeavor that combines robust infrastructure provisioning with comprehensive observability and reliable incident management. By leveraging Terraform, Datadog, and PagerDuty, you can establish a powerful, automated, and resilient foundation for your cloud-native applications. This guide provides a solid starting point; remember to continuously adapt and optimize your setup to meet evolving application and business needs.

Ready to elevate your EKS deployments? Start implementing these practices today and build infrastructure that stands the test of production.

CloudStack YAML