Terraform AWS EKS Cluster with Datadog and PagerDuty Incident Management

In the modern cloud-native landscape, building scalable, resilient, and observable Kubernetes clusters is paramount. This guide provides a comprehensive, technical walkthrough for deploying an Amazon Elastic Kubernetes Service (EKS) cluster using Terraform, integrating it with Datadog for robust monitoring, and leveraging PagerDuty for critical incident management. By automating this entire stack, you ensure consistency, auditability, and rapid disaster response.

Architecture Pro-Tip:

Always design your EKS infrastructure with modularity and security in mind. Utilize private subnets for your nodes, restrict EKS API access, and employ distinct IAM roles for the EKS control plane and node groups. Integrate observability tools like Datadog from the outset, embedding their agents and integrations directly into your IaC to ensure comprehensive coverage from day one. This proactive approach minimizes operational overhead and enhances incident response capabilities.

Why Terraform, EKS, Datadog, and PagerDuty?

Combining these powerful tools creates a resilient, automated, and observable cloud infrastructure:

Terraform: Enables Infrastructure as Code (IaC) for consistent, repeatable, and version-controlled deployments of your entire AWS EKS environment, including networking, IAM, and the cluster itself.
AWS EKS: A fully managed Kubernetes service that simplifies the deployment, management, and scaling of Kubernetes applications in the cloud, offloading operational burdens.
Datadog: A leading monitoring and analytics platform offering comprehensive visibility into your EKS clusters, applications, logs, and infrastructure metrics, crucial for proactive issue detection.
PagerDuty: An incident management system that centralizes alerts, streamlines on-call scheduling, and automates incident escalation, ensuring critical issues are addressed promptly.

Prerequisites

Before you begin, ensure you have the following:

AWS Account: With necessary IAM permissions to create EKS clusters, VPCs, EC2 instances, and IAM roles.
Terraform CLI: Installed (version 1.0+ recommended).
AWS CLI: Configured with your credentials.
kubectl: Installed for interacting with the Kubernetes cluster.
Datadog Account: With an API key and Application key.
PagerDuty Account: With API access and an integration key (if setting up via Terraform).

Step-by-Step Terraform Implementation

1. Project Structure

Organize your Terraform code into logical modules for better management. A typical structure might look like this:

.
├── main.tf
├── variables.tf
├── outputs.tf
├── versions.tf
└── modules/
    ├── vpc/
    │   ├── main.tf
    │   ├── variables.tf
    │   └── outputs.tf
    ├── eks/
    │   ├── main.tf
    │   ├── variables.tf
    │   └── outputs.tf
    └── datadog/
        ├── main.tf
        └── variables.tf

2. AWS Provider and Backend Configuration (versions.tf)

Define your AWS provider and an S3 backend for remote state management.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.0"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
  }
  required_version = "~> 1.0"

  backend "s3" {
    bucket         = "your-terraform-state-bucket"
    key            = "eks-datadog-pagerduty/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "your-terraform-lock-table"
  }
}

provider "aws" {
  region = var.aws_region
}

provider "kubernetes" {
  host                   = module.eks.eks_cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.eks_cluster_certificate_authority_data)
  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args        = ["eks", "get-token", "--region", var.aws_region, "--cluster-name", module.eks.eks_cluster_id]
  }
}

provider "helm" {
  kubernetes {
    host                   = module.eks.eks_cluster_endpoint
    cluster_ca_certificate = base64decode(module.eks.eks_cluster_certificate_authority_data)
    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args        = ["eks", "get-token", "--region", var.aws_region, "--cluster-name", module.eks.eks_cluster_id]
    }
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

3. VPC and Networking (modules/vpc/main.tf)

A dedicated VPC for EKS is crucial for isolation and security. The terraform-aws-modules/vpc/aws module is highly recommended.

resource "aws_vpc" "main" {
  cidr_block           = var.vpc_cidr
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name = "${var.cluster_name}-vpc"
  }
}

resource "aws_subnet" "public" {
  count             = length(var.public_subnet_cidrs)
  vpc_id            = aws_vpc.main.id
  cidr_block        = var.public_subnet_cidrs[count.index]
  availability_zone = data.aws_availability_zones.available.names[count.index]
  map_public_ip_on_launch = true

  tags = {
    Name = "${var.cluster_name}-public-subnet-${count.index}"
  }
}

resource "aws_subnet" "private" {
  count             = length(var.private_subnet_cidrs)
  vpc_id            = aws_vpc.main.id
  cidr_block        = var.private_subnet_cidrs[count.index]
  availability_zone = data.aws_availability_zones.available.names[count.index]

  tags = {
    Name = "${var.cluster_name}-private-subnet-${count.index}"
  }
}

# ... NAT Gateways, Internet Gateway, Route Tables etc.
# For brevity, consider using the official AWS VPC module
# module "vpc" {
#   source  = "terraform-aws-modules/vpc/aws"
#   version = "~> 5.0"
#
#   name = var.cluster_name
#   cidr = var.vpc_cidr
#
#   azs             = data.aws_availability_zones.available.names
#   private_subnets = var.private_subnet_cidrs
#   public_subnets  = var.public_subnet_cidrs
#
#   enable_nat_gateway   = true
#   single_nat_gateway   = false
#   enable_dns_hostnames = true
#
#   tags = {
#     Environment = var.environment
#   }
# }

4. EKS Cluster (modules/eks/main.tf)

Deploy the EKS control plane and managed node groups. The terraform-aws-modules/eks/aws module is excellent here.

resource "aws_iam_role" "eks_cluster" {
  name = "${var.cluster_name}-eks-cluster-role"

  assume_role_policy = jsonencode({
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Principal = {
        Service = "eks.amazonaws.com"
      }
    }]
    Version = "2012-10-17"
  })
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eks_cluster.name
}

resource "aws_eks_cluster" "main" {
  name     = var.cluster_name
  role_arn = aws_iam_role.eks_cluster.arn
  vpc_config {
    subnet_ids         = var.private_subnet_ids # EKS nodes usually in private subnets
    endpoint_private_access = true # Restrict API endpoint to VPC
    endpoint_public_access  = false # Disable public access, only accessible via VPN/DirectConnect
  }

  version = var.kubernetes_version

  tags = {
    Name = var.cluster_name
  }

  # Ensure that compute capacity is available for cluster provisioning
  depends_on = [
    aws_iam_role_policy_attachment.eks_cluster_policy,
  ]
}

resource "aws_iam_role" "eks_node_group" {
  name = "${var.cluster_name}-eks-node-group-role"

  assume_role_policy = jsonencode({
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Principal = {
        Service = "ec2.amazonaws.com"
      }
    }]
    Version = "2012-10-17"
  })
}

resource "aws_iam_role_policy_attachment" "eks_node_group_policy_worker" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.eks_node_group.name
}

resource "aws_iam_role_policy_attachment" "eks_node_group_policy_cni" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.eks_node_group.name
}

resource "aws_iam_role_policy_attachment" "eks_node_group_policy_registry" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.eks_node_group.name
}

resource "aws_eks_node_group" "main" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "${var.cluster_name}-nodegroup"
  node_role_arn   = aws_iam_role.eks_node_group.arn
  subnet_ids      = var.private_subnet_ids # Nodes in private subnets
  instance_types  = var.node_instance_types
  disk_size       = var.node_disk_size
  ami_type        = "AL2_x86_64" # Amazon Linux 2
  capacity_type   = "ON_DEMAND"

  scaling_config {
    desired_size = var.node_group_desired_capacity
    min_size     = var.node_group_min_capacity
    max_size     = var.node_group_max_capacity
  }

  depends_on = [
    aws_iam_role_policy_attachment.eks_node_group_policy_worker,
    aws_iam_role_policy_attachment.eks_node_group_policy_cni,
    aws_iam_role_policy_attachment.eks_node_group_policy_registry,
  ]

  tags = {
    Name = "${var.cluster_name}-nodegroup"
  }
}

Datadog and PagerDuty Integration with Terraform

Integrating Datadog and PagerDuty involves two main steps: deploying the Datadog Agent to your EKS cluster and configuring Datadog monitors to alert PagerDuty. The Datadog provider for Terraform allows managing these integrations as code.

1. Deploy Datadog Agent to EKS (modules/datadog/main.tf)

The Datadog Agent collects metrics, logs, and traces from your EKS cluster. While it can be deployed with Kubernetes manifests, using the Helm chart via Terraform is often preferred for its flexibility.

resource "kubernetes_secret" "datadog_api_key" {
  metadata {
    name      = "datadog-secret"
    namespace = "default" # Or dedicated 'datadog' namespace
  }

  data = {
    "api-key" = var.datadog_api_key
    "app-key" = var.datadog_app_key
  }
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Or dedicated 'datadog' namespace

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    # Alternatively, refer to the secret if passing via values.yaml/env variables
    # value = kubernetes_secret.datadog_api_key.data["api-key"]
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    # value = kubernetes_secret.datadog_api_key.data["app-key"]
  }
  set {
    name  = "datadog.clusterName"
    value = var.cluster_name
  }
  set {
    name  = "datadog.site"
    value = "datadoghq.com" # or eu.datadoghq.com, etc.
  }
  set {
    name  = "kubeStateMetrics.enabled"
    value = true
  }
  set {
    name  = "clusterAgent.enabled"
    value = true
  }
  set {
    name  = "apm.enabled"
    value = true
  }
  set {
    name  = "logAgent.enabled"
    value = true
  }
  set {
    name  = "logAgent.containerCollectAll"
    value = true
  }
  set {
    name  = "processAgent.enabled"
    value = true
  }
  # Add other configurations as needed (e.g., tags, resource limits, RBAC)
  # For EKS, ensure appropriate IAM permissions for service accounts if using IRSA for Datadog Agent
}

2. Configure Datadog PagerDuty Integration (modules/datadog/main.tf)

First, set up the PagerDuty integration within Datadog using the datadog_integration_pagerduty resource. Then, define monitors that trigger alerts to this PagerDuty service.

# This assumes you have created a PagerDuty service manually or via PagerDuty's Terraform provider.
# For simplicity, we'll use an existing PagerDuty Service Integration Key.
# In a real-world scenario, you might use the PagerDuty Terraform provider to create services and escalation policies.

resource "datadog_integration_pagerduty" "pagerduty_integration" {
  # This resource only updates existing integrations, it cannot create new ones
  # You need to manually add the PagerDuty integration in Datadog UI first and get the service_key
  # Or use the PagerDuty provider to create the service and expose its integration key.
  # For the purpose of this example, we assume `pagerduty_integration_key` is a manually obtained value
  # from a PagerDuty service integration that uses the Datadog integration type.
  # More advanced usage would involve the pagerduty terraform provider.
  # e.g., service_key = pagerduty_service_integration.example.integration_key

  # Replace with your actual Datadog PagerDuty integration key
  # This is the 'Integration Key' from the Datadog integration in PagerDuty
  # or the 'Service Key' from a Datadog integration in a PagerDuty service.
  # Datadog integration setup in Datadog: Integrations -> PagerDuty -> Add Account
  # Or, PagerDuty service -> Integrations -> New Integration -> Datadog
  service_key = var.pagerduty_integration_key
}

# Example Datadog Monitor for high CPU utilization on EKS nodes
resource "datadog_monitor" "eks_high_cpu" {
  name               = "EKS Cluster: High CPU Usage on Node {{host.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:system.cpu.idle{cluster_name:${var.cluster_name}} by {host} < 20"
  message            = <

Deployment Workflow

Once your Terraform files are set up, follow these steps to deploy your EKS cluster with Datadog and PagerDuty integration:

Initialize Terraform: Navigate to your project root and run terraform init. This downloads necessary providers and sets up the S3 backend.
Review the Plan: Execute terraform plan to see exactly what infrastructure changes Terraform proposes. Carefully review the output to ensure it matches your expectations.
Apply Changes: If the plan looks correct, apply the changes with terraform apply. Type yes when prompted to confirm the deployment. This process can take 15-20 minutes for EKS cluster provisioning.
Verify EKS Connectivity: After terraform apply completes, configure kubectl to connect to your new EKS cluster:
aws eks update-kubeconfig --region <YOUR_AWS_REGION> --name <YOUR_CLUSTER_NAME> kubectl get nodes kubectl get pods -n default # Check Datadog agent pods
Verify Datadog Integration: Log into your Datadog account. You should see metrics, logs, and traces streaming from your EKS cluster. Check the "Integrations" -> "Kubernetes" and "EKS" dashboards.
Verify PagerDuty Integration: In Datadog, go to "Integrations" -> "PagerDuty" to ensure the integration is healthy. In PagerDuty, check your services for the new integration. You can test a monitor by artificially creating a critical condition or by temporarily lowering a threshold to trigger an alert.

Troubleshooting and Best Practices

Common Issues:

IAM Permissions: Ensure the AWS credentials used by Terraform and kubectl have sufficient permissions for EKS, EC2, IAM, VPC, etc.
Networking: Verify that your VPC, subnets, security groups, and route tables are correctly configured, especially for private EKS endpoints.
Datadog Agent Connectivity: If Datadog metrics aren't appearing, check the Datadog Agent pod logs (kubectl logs -f <datadog-agent-pod>) for API key issues or network connectivity problems.
PagerDuty Alerts: Confirm that the @pagerduty-{{datadog_integration_pagerduty.pagerduty_integration.service_name}} tag in your Datadog monitor message matches your configured PagerDuty service name/alias.

Best Practices:

Modularity: Break down your Terraform configuration into reusable modules (VPC, EKS, Datadog) for easier management and scaling.
State Management: Always use a remote backend (like S3 with DynamoDB locking) for Terraform state to enable team collaboration and prevent state corruption.
Secrets Management: Avoid hardcoding sensitive information. Use AWS Secrets Manager, HashiCorp Vault, or Kubernetes Secrets for Datadog API/App keys and PagerDuty integration keys.
IAM Roles for Service Accounts (IRSA): Configure EKS IRSA for your Datadog Agent service account to grant AWS permissions directly to Kubernetes pods, adhering to the principle of least privilege.
Monitoring as Code: Define all your Datadog monitors, dashboards, and integrations using Terraform to maintain version control and consistency across environments.
Testing: Implement automated tests for your Terraform configurations to catch errors early.

Conclusion

By following this guide, you've successfully provisioned an AWS EKS cluster with Terraform, integrated comprehensive monitoring through Datadog, and established robust incident response capabilities via PagerDuty. This setup provides a solid foundation for deploying and managing cloud-native applications with confidence, ensuring high availability, performance, and operational excellence. Embrace Infrastructure as Code for your entire observability and incident management stack to maintain agility and reliability.

CloudStack YAML