Terraform AWS EKS Cluster Deployment with Datadog APM and PagerDuty Incident Response

In the rapidly evolving landscape of cloud-native infrastructure, deploying and managing Kubernetes clusters efficiently is paramount. This guide provides a comprehensive, technical walkthrough on how to provision an Amazon Elastic Kubernetes Service (EKS) cluster using Terraform, integrate it with Datadog for robust Application Performance Monitoring (APM) and observability, and establish PagerDuty for streamlined incident response. By leveraging Infrastructure as Code (IaC), we ensure repeatable, scalable, and maintainable deployments.

Architecture Pro-Tip:

Always design your production EKS environments with a strong emphasis on automation, observability, and resiliency. Using Terraform for infrastructure provisioning ensures consistency and version control. Integrating Datadog from day one provides full-stack visibility, while PagerDuty closes the loop with automated, intelligent incident alerting, transforming reactive responses into proactive problem-solving. Prioritize least-privilege IAM roles and network segmentation.

Prerequisites

Before commencing the deployment, ensure you have the following tools and accounts configured:

AWS Account: With administrative access to create EKS clusters, IAM roles, VPCs, and EC2 instances.
AWS CLI: Configured with appropriate credentials and default region.
Terraform: Version 1.0 or newer, installed and configured.
kubectl: Kubernetes command-line tool, for interacting with the EKS cluster.
Helm: Kubernetes package manager, required for deploying the Datadog Agent.
Datadog Account: With API and Application Keys generated for integration.
PagerDuty Account: With a service created and an integration key ready for use.

Core Concepts Explained

Understanding the core technologies involved is crucial for a successful deployment.

Terraform for Infrastructure as Code (IaC)

Terraform is an open-source IaC tool that allows you to define and provision cloud and on-prem resources using a declarative configuration language. It enables managing infrastructure version control, collaboration, and automation.

AWS Elastic Kubernetes Service (EKS)

EKS is a managed Kubernetes service that makes it easy to deploy, manage, and scale containerized applications on AWS. EKS handles the Kubernetes control plane's availability, ensuring high reliability and reducing operational overhead.

Datadog APM and Observability

Datadog is a monitoring and analytics platform for cloud applications. It provides comprehensive visibility across infrastructure, applications, and logs, offering features like APM (Application Performance Monitoring) to trace requests, identify bottlenecks, and optimize performance within your EKS cluster.

PagerDuty Incident Response

PagerDuty is an incident management platform that provides reliable notifications, automatic escalations, and on-call scheduling to help teams resolve critical incidents faster. Integrating it with Datadog ensures that actionable alerts lead directly to an organized response.

Terraform Project Structure

A typical Terraform project for this deployment would involve several files to organize resources, variables, and outputs.

main.tf: Defines the primary resources like EKS cluster, VPC, and Datadog agent.
variables.tf: Declares input variables for customization (e.g., region, cluster name, API keys).
outputs.tf: Specifies values that will be outputted after deployment (e.g., EKS endpoint, Kubeconfig command).
providers.tf: Configures AWS, Kubernetes, and Helm providers.

Terraform Configuration for EKS, Datadog, and PagerDuty

Below is a comprehensive Terraform configuration that deploys an AWS EKS cluster, sets up necessary networking components (VPC, subnets), IAM roles, and then deploys the Datadog Agent to the EKS cluster via Helm. PagerDuty integration will primarily happen within the Datadog platform once the agent is reporting data.

Note: For production environments, consider using more granular security group rules, private subnets for nodes, and dedicated modules for better organization.

resource "aws_vpc" "eks_vpc" {
  cidr_block = "10.0.0.0/16"
  tags = {
    Name = "${var.cluster_name}-vpc"
    "kubernetes.io/cluster/${var.cluster_name}" = "owned"
  }
}

resource "aws_subnet" "eks_subnet_public" {
  count             = 2 # Deploy into 2 AZs for high availability
  vpc_id            = aws_vpc.eks_vpc.id
  cidr_block        = cidrsubnet(aws_vpc.eks_vpc.cidr_block, 8, count.index)
  availability_zone = data.aws_availability_zones.available.names[count.index]
  map_public_ip_on_launch = true # EKS Load Balancers in public subnets
  tags = {
    Name = "${var.cluster_name}-public-subnet-${count.index}"
    "kubernetes.io/cluster/${var.cluster_name}" = "owned"
    "kubernetes.io/role/elb"                  = "1"
  }
}

data "aws_availability_zones" "available" {
  state = "available"
}

# EKS Cluster IAM Role
resource "aws_iam_role" "eks_cluster_role" {
  name = "${var.cluster_name}-eks-cluster-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "eks.amazonaws.com"
        }
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eks_cluster_role.name
}

resource "aws_iam_role_policy_attachment" "eks_service_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
  role       = aws_iam_role.eks_cluster_role.name
}

# EKS Cluster
resource "aws_eks_cluster" "eks_cluster" {
  name     = var.cluster_name
  role_arn = aws_iam_role.eks_cluster_role.arn
  vpc_config {
    subnet_ids         = aws_subnet.eks_subnet_public[*].id
    security_group_ids = [aws_security_group.eks_master_sg.id]
  }
  version = "1.28" # Specify desired Kubernetes version

  tags = {
    Environment = "Dev"
    Project     = var.cluster_name
  }
  depends_on = [
    aws_iam_role_policy_attachment.eks_cluster_policy,
    aws_iam_role_policy_attachment.eks_service_policy,
  ]
}

# Security Group for EKS Master
resource "aws_security_group" "eks_master_sg" {
  name        = "${var.cluster_name}-eks-master-sg"
  description = "Security group for EKS control plane network interfaces"
  vpc_id      = aws_vpc.eks_vpc.id

  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"] # Restrict to specific IPs in production
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
  tags = {
    Name = "${var.cluster_name}-eks-master-sg"
  }
}

# EKS Node Group IAM Role
resource "aws_iam_role" "eks_node_group_role" {
  name = "${var.cluster_name}-eks-node-group-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_node_group_worker_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.eks_node_group_role.name
}

resource "aws_iam_role_policy_attachment" "eks_node_group_cni_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.eks_node_group_role.name
}

resource "aws_iam_role_policy_attachment" "eks_node_group_registry_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.eks_node_group_role.name
}

# EKS Managed Node Group
resource "aws_eks_node_group" "eks_node_group" {
  cluster_name    = aws_eks_cluster.eks_cluster.name
  node_group_name = "${var.cluster_name}-node-group"
  node_role_arn   = aws_iam_role.eks_node_group_role.arn
  subnet_ids      = aws_subnet.eks_subnet_public[*].id
  instance_types  = ["t3.medium"] # Choose appropriate instance type for your workload

  scaling_config {
    desired_size = 2
    max_size     = 3
    min_size     = 1
  }

  labels = {
    env = "dev"
  }

  tags = {
    Environment = "Dev"
    Project     = var.cluster_name
  }
  depends_on = [
    aws_iam_role_policy_attachment.eks_node_group_worker_policy,
    aws_iam_role_policy_attachment.eks_node_group_cni_policy,
    aws_iam_role_policy_attachment.eks_node_group_registry_policy,
  ]
}

# Kubernetes provider configuration (to deploy Datadog agent)
provider "kubernetes" {
  host                   = aws_eks_cluster.eks_cluster.endpoint
  cluster_ca_certificate = base64decode(aws_eks_cluster.eks_cluster.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.eks_auth.token
}

data "aws_eks_cluster_auth" "eks_auth" {
  name = aws_eks_cluster.eks_cluster.name
}

# Helm provider configuration
provider "helm" {
  kubernetes {
    host                   = aws_eks_cluster.eks_cluster.endpoint
    cluster_ca_certificate = base64decode(aws_eks_cluster.eks_cluster.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.eks_auth.token
  }
}

# Datadog Agent Deployment via Helm
resource "helm_release" "datadog_agent" {
  name       = "datadog-agent"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
  }

  set {
    name  = "apm.enabled"
    value = "true"
  }

  set {
    name  = "logs.enabled"
    value = "true"
  }

  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }

  set {
    name  = "processAgent.enabled"
    value = "true"
  }

  depends_on = [
    aws_eks_cluster.eks_cluster,
    aws_eks_node_group.eks_node_group # Ensure nodes are ready
  ]
}

# variables.tf
variable "aws_region" {
  description = "The AWS region to deploy resources into."
  type        = string
  default     = "us-east-1"
}

variable "cluster_name" {
  description = "Name for the EKS cluster."
  type        = string
  default     = "production-eks-cluster"
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

# outputs.tf
output "eks_cluster_name" {
  description = "The name of the EKS cluster"
  value       = aws_eks_cluster.eks_cluster.name
}

output "kubeconfig_command" {
  description = "Run this command to configure kubectl for your new EKS cluster."
  value       = "aws eks update-kubeconfig --name ${aws_eks_cluster.eks_cluster.name} --region ${var.aws_region}"
}
    

Deployment Steps

Follow these steps to deploy your EKS cluster with Datadog integration:

Initialize Terraform: Navigate to your project directory and run terraform init. This downloads the necessary providers.
Create a terraform.tfvars file: Populate it with your specific values.
aws_region = "us-east-1" cluster_name = "my-prod-eks" datadog_api_key = "your_datadog_api_key_here" datadog_app_key = "your_datadog_app_key_here"
Review the Plan: Execute terraform plan to see the infrastructure changes Terraform will make. Carefully review the output.
Apply the Configuration: If the plan looks correct, apply the changes with terraform apply. Type yes to confirm. This process can take 15-25 minutes as AWS provisions the EKS cluster and nodes.
Configure kubectl: Once terraform apply completes, use the command from the kubeconfig_command output to configure your kubectl context.
aws eks update-kubeconfig --name my-prod-eks --region us-east-1
Verify Datadog Agent: Check if the Datadog agents are running in your cluster:
kubectl get pods -n datadog
You should see Datadog Agent pods in a Running state.

Integrating Datadog with PagerDuty

Once the Datadog agents are reporting data from your EKS cluster, you can configure Datadog to send alerts to PagerDuty.

In PagerDuty: Create a new service or integrate into an existing one. For a new service, select "Datadog" as the integration type to get an Integration Key.
In Datadog:
- Navigate to Integrations > Integrations. Search for and install the "PagerDuty" integration.
- Add your PagerDuty Integration Key to the Datadog integration configuration.
- Once integrated, you can create monitors in Datadog (e.g., for high CPU utilization on a service, pod restarts, or EKS control plane health issues).
- When configuring a monitor, in the "Notify your team" section, select @pagerduty and specify the PagerDuty service you want to alert.

Validation and Verification

After deployment, ensure all components are functioning as expected:

EKS Cluster: Verify the cluster and node groups are active in the AWS EKS console.
Kubernetes Pods: Use kubectl get nodes and kubectl get pods -A to ensure all system and Datadog pods are running.
Datadog Metrics: Log in to your Datadog dashboard. You should start seeing metrics, logs, and traces from your EKS cluster, including Kubernetes components and your applications. Check the "Host Map" and "Container" views.
PagerDuty Alerts: Manually trigger a test alert in Datadog or create a low-threshold monitor to verify that alerts are correctly routed to PagerDuty and trigger an incident.

Troubleshooting and Best Practices

Common Troubleshooting Steps

IAM Permissions: Most EKS deployment issues stem from incorrect IAM roles or policies. Double-check that your EKS cluster role and node group roles have all required permissions.
VPC and Subnets: Ensure your subnets are tagged correctly for EKS and have sufficient IP addresses. Network connectivity issues can prevent nodes from joining the cluster.
Datadog Agent Pods Not Running: Check the logs of the Datadog Agent pods (kubectl logs -n datadog <pod-name>). Often, this is due to incorrect API/APP keys or network policies blocking communication with Datadog endpoints.
Kubeconfig Issues: If kubectl doesn't connect, ensure your AWS CLI is configured with the correct region and credentials, and you've run the aws eks update-kubeconfig command.

Best Practices for Production Environments

Module-Based Terraform: For complex setups, organize your Terraform code into reusable modules (e.g., separate modules for VPC, EKS cluster, node groups).
Private Node Groups: Deploy EKS nodes into private subnets for enhanced security, using NAT Gateways for outbound internet access.
Fine-grained IAM: Implement least-privilege IAM roles for all EKS components and service accounts.
Version Control: Always keep your Terraform configurations in a Git repository.
Terraform State Management: Use an S3 backend with DynamoDB locking for remote state management in collaborative environments.
Datadog Dashboards & Monitors: Create comprehensive dashboards for your EKS cluster and critical services. Implement proactive monitors with actionable alerts.
Security Groups: Restrict ingress traffic to the EKS control plane to only necessary sources.
Regular Updates: Keep EKS Kubernetes versions and node AMIs updated for security and feature benefits.

Conclusion

Deploying an AWS EKS cluster with Terraform, coupled with Datadog for APM and PagerDuty for incident response, establishes a robust and observable cloud-native environment. This setup provides the foundation for scalable application deployments, deep operational insights, and efficient incident resolution, crucial for maintaining high availability and performance in modern DevOps practices. By automating infrastructure provisioning and integrating best-in-class monitoring and alerting tools, teams can focus more on innovation and less on manual toil.

CloudStack YAML