Terraform for AWS EKS with Datadog Observability and PagerDuty Integration

Architecture Pro-Tip: Always design your Terraform configurations with modularity in mind. Use Terraform modules for common components like VPCs, EKS clusters, and Datadog integrations. This promotes reusability, reduces redundancy, and simplifies maintenance, especially as your cloud infrastructure scales. Explicitly define outputs for critical resources to enable seamless integration between modules and to easily retrieve important information like cluster endpoints or ARN identifiers.

Terraform for AWS EKS with Datadog Observability and PagerDuty Integration: A Comprehensive Guide

In the fast-paced world of cloud-native development, managing complex Kubernetes clusters like Amazon Elastic Kubernetes Service (EKS) requires robust automation, comprehensive observability, and efficient incident response. This guide provides a detailed walkthrough on how to provision and manage AWS EKS using Terraform Infrastructure as Code (IaC), integrate Datadog for full-stack observability, and establish an effective incident management workflow with PagerDuty.

Why This Stack Matters for Modern DevOps Teams

Terraform for EKS Automation: Declaratively define, provision, and manage your EKS clusters and associated AWS resources (VPC, IAM, security groups) in a reproducible and version-controlled manner. This eliminates manual errors and accelerates deployment cycles.
Datadog for EKS Observability: Gain deep insights into the health and performance of your EKS clusters, nodes, pods, and applications. Datadog unifies metrics, logs, traces (APM), and network data, providing a single pane of glass for monitoring, troubleshooting, and performance optimization.
PagerDuty for Incident Response: Automate incident escalation and on-call rotations based on alerts from Datadog. PagerDuty ensures that critical alerts reach the right team members promptly, reducing mean time to resolution (MTTR) and minimizing service disruptions.

Prerequisites

Before you begin, ensure you have the following:

An AWS Account with appropriate IAM permissions to create EKS clusters and related resources.
Terraform CLI installed (version 1.0 or higher recommended).
AWS CLI installed and configured with your AWS credentials.
A Datadog Account with an API Key and Application Key.
A PagerDuty Account with an Integration Key (or API token for advanced management).
kubectl CLI installed and configured for EKS cluster interaction.

Step 1: Core AWS EKS Infrastructure with Terraform

We'll start by defining the fundamental AWS resources required for an EKS cluster.

1. AWS Provider Configuration

Define the AWS provider and specify the desired region.

2. Network Infrastructure (VPC, Subnets)

EKS requires a robust network setup. It's best practice to create a dedicated VPC with public and private subnets across multiple Availability Zones for high availability.

VPC: A virtual private cloud to isolate your EKS resources.
Subnets: Private subnets for your worker nodes and public subnets for load balancers or bastion hosts.
Internet Gateway (IGW) & NAT Gateway: For outbound internet access from private subnets.
Route Tables: To control network traffic flow.

3. EKS Cluster Creation

The aws_eks_cluster resource defines your Kubernetes control plane.

IAM Role: An IAM role for the EKS control plane to interact with other AWS services.
Kubernetes Version: Specify your desired Kubernetes version.
VPC Configuration: Link to the subnets created earlier.

4. EKS Node Groups

Worker nodes are EC2 instances that run your applications. You can use managed node groups for easier management.

IAM Role: An IAM role for your worker nodes.
Instance Types: Choose appropriate EC2 instance types.
Scaling Configuration: Define desired, min, and max sizes for your node group.

Step 2: Integrating Datadog for Comprehensive Observability

Datadog provides deep visibility into EKS. We'll use Terraform to deploy the Datadog Agent and configure monitors.

1. Deploying the Datadog Agent on EKS

The Datadog Agent runs as a DaemonSet on your EKS cluster, collecting metrics, logs, and traces from all nodes and pods. You can deploy it using the official Datadog Helm chart via the Terraform Helm provider.

Helm Release: Use the helm_release resource to deploy the Datadog Agent.
API & App Keys: Pass your Datadog API and Application keys securely.
Configuration: Enable APM, log collection, network performance monitoring, and other features as needed.

2. Configuring Datadog Monitors and Dashboards with Terraform

Terraform can also manage Datadog resources like monitors, dashboards, and even integrations. This allows you to define your observability strategy as code.

Datadog Provider: Configure the Datadog Terraform provider with your API and Application keys.
datadog_monitor: Create monitors for critical EKS metrics (e.g., node CPU utilization, pod restarts, network errors).
datadog_dashboard: Define custom dashboards to visualize your EKS performance and health.

Step 3: Real-time Incident Management with PagerDuty

Integrate Datadog alerts with PagerDuty to ensure critical issues are addressed immediately by the right on-call team.

1. Datadog-PagerDuty Integration Setup

The primary method is to configure the integration within Datadog, then link your Datadog monitors to PagerDuty services.

PagerDuty Service: In PagerDuty, create a Service (e.g., "EKS Critical Alerts") and an associated Escalation Policy. Obtain the Integration Key for a "Datadog" or "Events API v2" integration.
Datadog Integration: In Datadog, navigate to Integrations -> PagerDuty. Add your PagerDuty Integration Key to establish the connection.
Link Monitors: When creating or updating a datadog_monitor resource in Terraform, use the PagerDuty integration name in the message field (e.g., @pagerduty-EKS_Critical_Alerts) to route alerts to the specific PagerDuty service.

2. (Optional) Managing PagerDuty Resources with Terraform

For more advanced scenarios, you can use the PagerDuty Terraform provider to manage services, users, and escalation policies directly.

PagerDuty Provider: Configure the PagerDuty Terraform provider with your PagerDuty API token.
pagerduty_service, pagerduty_escalation_policy: Define these resources in Terraform to create your incident management structure.

Terraform Configuration Example: Bringing It All Together

Below is a simplified, illustrative example of how these components can be defined in Terraform. For a production environment, you would typically use dedicated modules for VPC, EKS, etc., and manage sensitive data with a secrets manager.

Example: main.tf

resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
  tags = {
    Name = "eks-datadog-pd-vpc"
  }
}

resource "aws_subnet" "public" {
  count             = 2
  vpc_id            = aws_vpc.main.id
  cidr_block        = cidrsubnet(aws_vpc.main.cidr_block, 8, count.index)
  availability_zone = data.aws_availability_zones.available.names[count.index]
  map_public_ip_on_launch = true
  tags = {
    Name = "eks-public-subnet-${count.index}"
  }
}

resource "aws_subnet" "private" {
  count             = 2
  vpc_id            = aws_vpc.main.id
  cidr_block        = cidrsubnet(aws_vpc.main.cidr_block, 8, count.index + 2)
  availability_zone = data.aws_availability_zones.available.names[count.index]
  tags = {
    Name = "eks-private-subnet-${count.index}"
  }
}

# EKS Cluster IAM Role
resource "aws_iam_role" "eks_cluster_role" {
  name = "eks-cluster-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "eks.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}
resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eks_cluster_role.name
}
resource "aws_iam_role_policy_attachment" "eks_vpc_cni_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSVPCResourceControllerForPrivateLink" # Example for private link, consider AmazonEKS_CNI_Policy based on needs
  role       = aws_iam_role.eks_cluster_role.name
}

# EKS Cluster
resource "aws_eks_cluster" "main" {
  name     = "datadog-eks-cluster"
  role_arn = aws_iam_role.eks_cluster_role.arn
  vpc_config {
    subnet_ids = concat(aws_subnet.public.*.id, aws_subnet.private.*.id)
  }
  depends_on = [
    aws_iam_role_policy_attachment.eks_cluster_policy,
    aws_iam_role_policy_attachment.eks_vpc_cni_policy,
  ]
}

# EKS Node Group IAM Role
resource "aws_iam_role" "eks_node_role" {
  name = "eks-node-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}
resource "aws_iam_role_policy_attachment" "eks_worker_node_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.eks_node_role.name
}
resource "aws_iam_role_policy_attachment" "eks_cni_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.eks_node_role.name
}
resource "aws_iam_role_policy_attachment" "ec2_container_registry_readonly" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.eks_node_role.name
}

# EKS Managed Node Group
resource "aws_eks_node_group" "main" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "datadog-node-group"
  node_role_arn   = aws_iam_role.eks_node_role.arn
  subnet_ids      = aws_subnet.private.*.id
  instance_types  = ["t3.medium"]
  disk_size       = 20

  scaling_config {
    desired_size = 2
    max_size     = 3
    min_size     = 1
  }

  depends_on = [
    aws_iam_role_policy_attachment.eks_worker_node_policy,
    aws_iam_role_policy_attachment.eks_cni_policy,
    aws_iam_role_policy_attachment.ec2_container_registry_readonly,
  ]
}

# Datadog Provider Configuration
provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

# Datadog Monitor for High CPU Utilization
resource "datadog_monitor" "high_cpu_alert" {
  name               = "[EKS] High Node CPU Utilization - {{cluster_name.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:kubernetes.cpu.usage.total{kubernetes_cluster_name:${aws_eks_cluster.main.name}} by {host} > 80"
  message            = "High CPU usage detected on an EKS node in cluster ${aws_eks_cluster.main.name}. Please investigate. @slack-devops @pagerduty-EKS_Critical_Alerts"
  tags               = ["environment:production", "service:eks"]
  escalation_message = "CPU usage remains high, escalating to on-call."
  no_data_timeframe  = 20
  notify_no_data     = true
  renotify_interval  = 60
  thresholds {
    critical = 80
    warning  = 70
  }
}

# Helm Provider Configuration for Datadog Agent (Requires kubectl configured to target the EKS cluster)
provider "helm" {
  kubernetes {
    host                   = aws_eks_cluster.main.endpoint
    cluster_ca_certificate = base64decode(aws_eks_cluster.main.certificate_authority.0.data)
    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args        = ["eks", "get-token", "--cluster-name", aws_eks_cluster.main.name]
    }
  }
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default"
  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }
  set {
    name  = "clusterName"
    value = aws_eks_cluster.main.name
  }
  set {
    name  = "kubeStateMetrics.enabled"
    value = true
  }
  set {
    name  = "logs.enabled"
    value = true
  }
  set {
    name  = "logs.containerCollectAll"
    value = true
  }
  set {
    name  = "apm.enabled"
    value = true
  }
  set {
    name  = "processAgent.enabled"
    value = true
  }
  set {
    name  = "networkHost.enabled"
    value = true
  }
}

Best Practices and Troubleshooting

Best Practices

Terraform State Management: Always use a remote backend (e.g., S3 with DynamoDB locking) for your Terraform state to enable collaboration and prevent state corruption.
Least Privilege IAM: Grant only the necessary IAM permissions to EKS roles, node roles, and your CI/CD pipelines.
Modular Design: Break down your Terraform configuration into reusable modules (VPC, EKS, Datadog setup) to enhance maintainability and scalability.
Cost Optimization: Regularly review your EKS instance types, scaling policies, and Datadog usage to optimize costs. Utilize Karpenter for intelligent autoscaling.
Security Updates: Keep your EKS control plane and worker nodes updated to the latest stable Kubernetes versions and patch levels.
Secrets Management: Do not hardcode API keys or sensitive information directly in Terraform files. Use AWS Secrets Manager, Datadog Secrets, or Vault for secure storage and retrieval.

Common Troubleshooting Tips

EKS Cluster Creation Issues: Check IAM roles for missing permissions, ensure subnets are correctly tagged for EKS, and verify network ACLs/security groups. Review CloudFormation stack events if Terraform errors are vague.
Worker Node Join Failures: Ensure the EKS node IAM role has proper policies. Check security group rules to allow communication between control plane and nodes. Verify AMI compatibility with the EKS version.
Datadog Agent Connectivity: Confirm your DD_API_KEY and DD_APP_KEY are correct. Check Datadog Agent logs (kubectl logs -f datadog-agent-...) for errors. Ensure network connectivity from your EKS nodes to Datadog endpoints.
PagerDuty Alerts Not Firing: Verify the Datadog-PagerDuty integration is active and correctly configured in Datadog. Double-check the @pagerduty- syntax in your Datadog monitor messages. Test the monitor threshold with simulated data.

Conclusion

By leveraging Terraform for AWS EKS provisioning, integrating Datadog for comprehensive observability, and streamlining incident response with PagerDuty, your DevOps teams can build a highly automated, resilient, and observable cloud-native infrastructure. This powerful combination empowers you to deploy applications faster, troubleshoot issues proactively, and maintain high service availability, ensuring a superior experience for your users.

CloudStack YAML