Seamless Operations: Terraform-Managed AWS EKS with Datadog Monitoring and PagerDuty Alerting

In the fast-evolving landscape of modern cloud infrastructure, managing Kubernetes clusters efficiently and reliably is paramount. This technical guide delves into building a robust, observable, and resilient AWS EKS (Elastic Kubernetes Service) environment using Terraform for Infrastructure as Code (IaC), Datadog for comprehensive monitoring, and PagerDuty for intelligent incident management and alerting. This integrated approach empowers DevOps teams to deploy, monitor, and respond to issues with unparalleled speed and confidence, ensuring optimal application performance and reduced Mean Time To Resolution (MTTR).

Architecture Pro-Tip

Always design your EKS infrastructure with high availability and scalability in mind from day one. Leverage separate VPC subnets for different tiers, apply least-privilege IAM roles, and integrate your observability stack (monitoring, logging, tracing) as part of your core IaC deployment. This proactive approach minimizes operational overhead and strengthens your platform's resilience.

Why This Stack? The Power of Integration

Terraform: The Foundation of Repeatable Infrastructure

Terraform, HashiCorp's open-source IaC tool, allows you to define and provision an entire cloud infrastructure using a declarative configuration language. For EKS, this means managing your VPC, subnets, security groups, IAM roles, EKS cluster, and node groups consistently and reproducibly. It streamlines deployments, reduces human error, and facilitates version control of your infrastructure.

AWS EKS: Managed Kubernetes at Scale

EKS provides a highly available and scalable Kubernetes control plane, removing the operational burden of managing master nodes. It seamlessly integrates with other AWS services like IAM for authentication, VPC for networking, and EC2 for worker nodes, making it a robust platform for containerized applications.

Datadog: Unified Observability for Kubernetes

Datadog offers a comprehensive monitoring solution, providing deep visibility into your EKS clusters and the applications running within them. It collects metrics, logs, and traces from all components—nodes, pods, containers, and applications—and consolidates them into a single pane of glass. Its powerful dashboards, anomaly detection, and machine learning capabilities enable proactive issue identification.

PagerDuty: Intelligent Incident Management

When issues arise, prompt and intelligent alerting is critical. PagerDuty integrates seamlessly with Datadog (and hundreds of other tools) to transform monitoring alerts into actionable incidents. It ensures the right people are notified at the right time through flexible on-call schedules, escalation policies, and various communication channels, drastically reducing incident response times.

Prerequisites

Before you begin, ensure you have the following:

An AWS Account with programmatic access keys configured.
Terraform CLI (v1.0+) installed.
AWS CLI installed and configured.
kubectl CLI installed.
Helm CLI (v3+) installed.
A Datadog Account with an API Key and Application Key.
A PagerDuty Account with a Service and Integration Key configured for Datadog.

Step-by-Step Implementation Guide

1. Setting up AWS EKS with Terraform

We'll use Terraform to provision the foundational AWS infrastructure required for EKS, including a VPC, subnets, IAM roles, and the EKS cluster itself. For simplicity, we'll use an existing, well-maintained Terraform EKS module.

2. Deploying Datadog Agent to EKS via Helm

Once your EKS cluster is up, the next step is to deploy the Datadog Agent as a DaemonSet to collect metrics, logs, and traces from all nodes and pods. We'll use the official Datadog Helm chart for this.

Add the Datadog Helm repository: helm repo add datadog https://helm.datadoghq.com
Update Helm repositories: helm repo update

3. Configuring PagerDuty Integration and Datadog Monitors

With Datadog collecting data, you can set up monitors that trigger alerts. PagerDuty will serve as the incident management system, receiving these alerts and escalating them according to your predefined schedules.

In Datadog: Go to Integrations -> Integrations -> PagerDuty and follow the steps to connect your PagerDuty service. You'll need the PagerDuty Integration Key.
Create Monitors: Define monitors in Datadog (e.g., high CPU utilization on nodes, pod failures, API server latency). In the notification section of your monitor, select the PagerDuty integration to send alerts.

Ready-to-Use Configuration: Terraform EKS with Datadog and PagerDuty

Here's a consolidated Terraform configuration demonstrating the setup. Remember to replace placeholder values with your actual credentials and desired settings. This example uses a simplified EKS module and demonstrates the Datadog Agent deployment via the helm_release resource and a basic Datadog monitor for illustration.

resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name = "eks-vpc"
  }
}

resource "aws_subnet" "public" {
  count             = 2
  vpc_id            = aws_vpc.main.id
  cidr_block        = cidrsubnet(aws_vpc.main.cidr_block, 8, count.index)
  availability_zone = data.aws_availability_zones.available.names[count.index]
  map_public_ip_on_launch = true

  tags = {
    Name                                       = "eks-public-subnet-${count.index}"
    "kubernetes.io/cluster/my-eks-cluster"     = "owned"
    "kubernetes.io/role/elb"                   = "1"
  }
}

resource "aws_subnet" "private" {
  count             = 2
  vpc_id            = aws_vpc.main.id
  cidr_block        = cidrsubnet(aws_vpc.main.cidr_block, 8, count.index + 2)
  availability_zone = data.aws_availability_zones.available.names[count.index]

  tags = {
    Name                                       = "eks-private-subnet-${count.index}"
    "kubernetes.io/cluster/my-eks-cluster"     = "owned"
    "kubernetes.io/role/internal-elb"          = "1"
  }
}

data "aws_availability_zones" "available" {}

resource "aws_iam_role" "eks_cluster" {
  name = "eks-cluster-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action = "sts:AssumeRole",
        Effect = "Allow",
        Principal = {
          Service = "eks.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eks_cluster.name
}

resource "aws_iam_role_policy_attachment" "eks_service_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
  role       = aws_iam_role.eks_cluster.name
}

resource "aws_eks_cluster" "main" {
  name     = "my-eks-cluster"
  role_arn = aws_iam_role.eks_cluster.arn
  version  = "1.28" # Or your desired EKS version

  vpc_config {
    subnet_ids = concat(aws_subnet.public.*.id, aws_subnet.private.*.id)
  }

  depends_on = [
    aws_iam_role_policy_attachment.eks_cluster_policy,
    aws_iam_role_policy_attachment.eks_service_policy,
  ]
}

resource "aws_iam_role" "eks_node_group" {
  name = "eks-node-group-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action = "sts:AssumeRole",
        Effect = "Allow",
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_worker_node_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.eks_node_group.name
}

resource "aws_iam_role_policy_attachment" "eks_cni_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.eks_node_group.name
}

resource "aws_iam_role_policy_attachment" "ec2_container_registry_readonly" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.eks_node_group.name
}

resource "aws_eks_node_group" "main" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "default-node-group"
  node_role_arn   = aws_iam_role.eks_node_group.arn
  subnet_ids      = aws_subnet.private.*.id
  instance_types  = ["t3.medium"] # Choose appropriate instance types
  desired_size    = 2
  max_size        = 3
  min_size        = 1

  scaling_config {
    desired_size = 2
    max_size     = 3
    min_size     = 1
  }

  update_config {
    max_unavailable = 1
  }

  depends_on = [
    aws_iam_role_policy_attachment.eks_worker_node_policy,
    aws_iam_role_policy_attachment.eks_cni_policy,
    aws_iam_role_policy_attachment.ec2_container_registry_readonly,
  ]
}

# --- Datadog Agent Deployment via Helm ---
resource "kubernetes_secret" "datadog_api_key" {
  metadata {
    name      = "datadog-api-key"
    namespace = "default" # Or your desired namespace for Datadog Agent
  }

  data = {
    "api-key" = var.datadog_api_key
  }
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Or your desired namespace
  version    = "2.33.0" # Use a specific version

  set {
    name  = "datadog.apiKeyExistingSecret"
    value = kubernetes_secret.datadog_api_key.metadata[0].name
  }

  set {
    name  = "clusterAgent.enabled"
    value = true
  }

  set {
    name  = "clusterAgent.metricsProvider.enabled"
    value = true
  }

  set {
    name  = "kubeStateMetrics.enabled"
    value = true
  }

  set {
    name  = "targetSystem"
    value = "linux"
  }

  # Enable APM, Logs, Process collection
  set {
    name  = "agent.apm.enabled"
    value = true
  }
  set {
    name  = "agent.logs.enabled"
    value = true
  }
  set {
    name  = "agent.logs.containerCollectAll"
    value = true
  }
  set {
    name  = "agent.processAgent.enabled"
    value = true
  }

  # Set Datadog site if not US
  # set {
  #   name  = "datadog.site"
  #   value = "eu.datadoghq.com"
  # }

  depends_on = [
    aws_eks_node_group.main,
    kubernetes_secret.datadog_api_key,
  ]
}

# --- Datadog Monitor for Node CPU Utilization (integrating with PagerDuty) ---
resource "datadog_monitor" "high_cpu_alert" {
  name                = "[EKS] High Node CPU Utilization on my-eks-cluster"
  type                = "metric alert"
  query               = "avg(last_5m):avg:system.cpu.idle{cluster_name:my-eks-cluster} by {host} < 10" # Alert if avg idle CPU < 10%
  message             = <<EOF
@pagerduty
Node {{host.name}} in EKS cluster my-eks-cluster is experiencing high CPU utilization.
Current idle CPU: {{value}}%. Please investigate immediately.
[Dashboard Link](https://app.datadoghq.com/dashboard/xxxx/your-kubernetes-dashboard)
EOF
  tags                = ["environment:production", "service:eks", "severity:critical"]
  priority            = 1
  escalation_message  = "CPU utilization remains high. Escalating to engineering lead."
  no_data_timeframe   = 20
  new_host_delay      = 300

  # This assumes you've set up the PagerDuty integration in Datadog and referenced it in the message above.
  # For direct PagerDuty service key via Datadog, you might use:
  # notifier = "pagerduty"
  # service_key = "YOUR_PAGERDUTY_SERVICE_KEY_HERE" # Not recommended to hardcode, use variables or secrets

  monitor_thresholds {
    critical = 10
    warning  = 20
  }

  notify_no_data = false
}

# Terraform Variables
variable "aws_region" {
  description = "AWS region"
  type        = string
  default     = "us-east-1"
}

variable "datadog_api_key" {
  description = "Your Datadog API Key"
  type        = string
  sensitive   = true
}

output "eks_cluster_endpoint" {
  description = "The endpoint for your EKS Kubernetes API."
  value       = aws_eks_cluster.main.endpoint
}

output "kubeconfig_certificate_authority_data" {
  description = "The base64 encoded certificate data required to communicate with your cluster."
  value       = aws_eks_cluster.main.certificate_authority[0].data
}

output "kubeconfig_token" {
  description = "Kubeconfig token for EKS cluster access."
  value       = data.aws_eks_cluster_auth.main.token
  sensitive   = true
}

data "aws_eks_cluster_auth" "main" {
  name = aws_eks_cluster.main.name
}

provider "kubernetes" {
  host                   = aws_eks_cluster.main.endpoint
  token                  = data.aws_eks_cluster_auth.main.token
  cluster_ca_certificate = base64decode(aws_eks_cluster.main.certificate_authority[0].data)
}

provider "helm" {
  kubernetes {
    host                   = aws_eks_cluster.main.endpoint
    token                  = data.aws_eks_cluster_auth.main.token
    cluster_ca_certificate = base64decode(aws_eks_cluster.main.certificate_authority[0].data)
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key # Define this variable for your Datadog App Key
}

    

To apply this configuration:

Save the code above into main.tf.
Create a terraform.tfvars file with your Datadog API and App keys:
datadog_api_key = "YOUR_DATADOG_API_KEY" datadog_app_key = "YOUR_DATADOG_APP_KEY"
Run terraform init.
Run terraform plan to review changes.
Run terraform apply to provision the resources.

Benefits of this Integrated Approach

Automation & Consistency: Terraform ensures your EKS cluster and monitoring agents are deployed uniformly across environments.
Comprehensive Visibility: Datadog provides a 360-degree view of your Kubernetes health, performance, and resource utilization.
Faster Incident Response: PagerDuty's intelligent alerting and escalation policies reduce MTTR significantly.
Scalability & Resilience: EKS handles the control plane, while robust monitoring and alerting ensure operational stability even under load.
Reduced Operational Burden: Automating setup and having integrated tools frees up your team to focus on development, not firefighting.

Advanced Considerations & Best Practices

Security: Implement strict IAM roles for EKS, enable Kubernetes network policies, and integrate with AWS Security Hub. Use OIDC provider for fine-grained IAM roles for service accounts.
Cost Optimization: Explore using EC2 Spot Instances for non-critical workloads, managed node groups with mixed instance types, and tools like Karpenter for intelligent autoscaling.
CI/CD Integration: Automate Terraform deployments and Helm chart updates through your CI/CD pipelines (e.g., Jenkins, GitLab CI, GitHub Actions) for true GitOps.
Logging & Tracing: Beyond basic logs, implement structured logging and distributed tracing (e.g., OpenTelemetry integrated with Datadog APM) for deep application insights.
Environment Management: Use Terraform workspaces or separate state files to manage multiple environments (dev, staging, prod) effectively.

Troubleshooting & Common Issues

Terraform Apply Failures: Check AWS account limits, IAM permissions, and VPC CIDR blocks. Ensure all required providers are initialized (terraform init).
EKS Node Group Not Ready: Verify node IAM role permissions (e.g., AmazonEKSWorkerNodePolicy, AmazonEC2ContainerRegistryReadOnly). Check security groups and network ACLs.
Datadog Agent Not Reporting: Confirm DD_API_KEY is correctly set in the Helm values or Kubernetes Secret. Check Datadog Agent pod logs for errors (kubectl logs -f datadog-agent-xxxx). Ensure network connectivity to Datadog endpoints.
PagerDuty Alerts Not Firing: Double-check the Datadog-PagerDuty integration in Datadog. Ensure the monitor's notification message explicitly includes @pagerduty or the correct integration ID. Verify PagerDuty service is healthy and on-call schedules are configured.

Conclusion

Building a modern cloud-native platform demands a holistic approach to infrastructure, observability, and incident management. By meticulously integrating Terraform for EKS provisioning, Datadog for comprehensive monitoring, and PagerDuty for effective alerting, organizations can achieve a highly automated, resilient, and transparent Kubernetes environment on AWS. This powerful combination minimizes manual intervention, provides deep insights into system health, and ensures rapid response to any operational challenges, paving the way for sustained innovation and business continuity.

CloudStack YAML

Terraform-Managed AWS EKS with Datadog Monitoring and PagerDuty Alerting