Terraform AWS EKS: Integrated Datadog Monitoring and PagerDuty Incident Response

In the dynamic world of cloud-native applications, maintaining robust observability and rapid incident response is paramount. This guide provides a comprehensive, technical walkthrough on deploying an AWS Elastic Kubernetes Service (EKS) cluster using Terraform, integrating it with Datadog for advanced monitoring, and configuring PagerDuty for automated incident management. By the end of this article, you will have a fully automated, observable, and resilient Kubernetes environment.

Architecture Pro-Tip

Always design your cloud infrastructure with modularity and automation in mind. Leverage Terraform modules for reusable components (EKS, VPC, IAM), employ a strong GitOps methodology for configuration management, and ensure a clear separation of concerns in your monitoring and alerting strategies. Prioritize security from the outset by implementing least privilege IAM roles and network policies.

Understanding the Core Components

This solution integrates four powerful platforms to create a resilient and observable Kubernetes ecosystem:

AWS EKS: Amazon's managed Kubernetes service, providing a highly available and scalable control plane for your containerized applications.
Terraform: HashiCorp's Infrastructure as Code (IaC) tool, enabling declarative definition and provisioning of cloud resources across various providers, including AWS.
Datadog: A leading monitoring and analytics platform that provides end-to-end visibility across your applications, infrastructure, and logs. It's crucial for Kubernetes observability, offering detailed metrics, traces, and logs.
PagerDuty: An incident management platform that aggregates alerts from various monitoring tools, intelligently routes them to the right teams, and facilitates rapid incident resolution.

Prerequisites

Before you begin, ensure you have the following:

An AWS Account with programmatic access configured via AWS CLI and appropriate IAM permissions.
Terraform CLI installed (version 1.0+ recommended).
kubectl CLI installed and configured.
Helm CLI installed (version 3+ recommended).
A Datadog Account with an API Key and Application Key.
A PagerDuty Account with an API Key and a service integrated for Datadog.

Step 1: Terraform Base Configuration for AWS EKS

We'll start by defining our AWS provider and setting up the basic EKS cluster and its required network infrastructure (VPC, subnets, security groups).

Project Structure Example

A typical Terraform project structure:

main.tf: Defines resources and modules.
variables.tf: Declares input variables.
outputs.tf: Defines output values.
versions.tf: Specifies Terraform and provider versions.

Step 2: Integrating Datadog for EKS Monitoring

Datadog provides deep visibility into your Kubernetes clusters through the Datadog Agent, deployed as a DaemonSet. We will use Terraform's Helm provider to manage the Datadog Agent deployment.

Datadog Agent Deployment via Helm

The Datadog Agent collects metrics, logs, and traces from your EKS cluster nodes, pods, and applications. Crucially, it needs to be configured with your Datadog API key and application key.

Step 3: Configuring PagerDuty for Incident Response

PagerDuty serves as the central hub for incident management. We'll integrate Datadog with PagerDuty so that critical alerts generated by Datadog automatically trigger incidents in PagerDuty, notifying the on-call team.

Datadog-PagerDuty Integration Setup

While PagerDuty itself can be managed with Terraform, the most common approach for Datadog-triggered incidents is to configure the integration directly within Datadog. Datadog provides a native integration with PagerDuty where you simply link a PagerDuty service integration key to a Datadog monitor's notification settings.

Example: Datadog Monitor Alerting PagerDuty

A Datadog monitor for high CPU utilization on an EKS node could be configured to notify your PagerDuty service via an @pagerduty tag in the alert message, or by specifying the integration directly in the monitor definition.

Terraform Configuration for Integrated Monitoring & Response

Below is a consolidated Terraform configuration demonstrating the setup of an EKS cluster, the deployment of the Datadog Agent via Helm, and an example of a Datadog monitor configured to alert PagerDuty. This assumes you have an existing VPC or will create one as part of a larger setup.

resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
  tags = {
    Name = "eks-datadog-pagerduty-vpc"
  }
}

resource "aws_subnet" "public" {
  count = 2
  vpc_id     = aws_vpc.main.id
  cidr_block = cidrsubnet(aws_vpc.main.cidr_block, 8, count.index)
  availability_zone = data.aws_availability_zones.available.names[count.index]
  tags = {
    Name = "eks-public-subnet-${count.index}"
    "kubernetes.io/cluster/eks-datadog-cluster" = "shared"
    "kubernetes.io/role/elb"                    = "1"
  }
}

resource "aws_internet_gateway" "gw" {
  vpc_id = aws_vpc.main.id
  tags = {
    Name = "eks-datadog-gw"
  }
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.gw.id
  }
  tags = {
    Name = "eks-public-route-table"
  }
}

resource "aws_route_table_association" "public" {
  count          = 2
  subnet_id      = aws_subnet.public[count.index].id
  route_table_id = aws_route_table.public.id
}

resource "aws_iam_role" "eks_cluster_role" {
  name = "eks-cluster-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "eks.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eks_cluster_role.name
}

resource "aws_iam_role_policy_attachment" "eks_service_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
  role       = aws_iam_role.eks_cluster_role.name
}

resource "aws_iam_role" "eks_node_group_role" {
  name = "eks-node-group-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_worker_node_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.eks_node_group_role.name
}

resource "aws_iam_role_policy_attachment" "eks_cni_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.eks_node_group_role.name
}

resource "aws_iam_role_policy_attachment" "ec2_container_registry_readonly" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.eks_node_group_role.name
}

resource "aws_eks_cluster" "main" {
  name     = "eks-datadog-cluster"
  role_arn = aws_iam_role.eks_cluster_role.arn
  vpc_config {
    subnet_ids         = aws_subnet.public[*].id
    endpoint_private_access = false
    endpoint_public_access  = true
  }
  depends_on = [
    aws_iam_role_policy_attachment.eks_cluster_policy,
    aws_iam_role_policy_attachment.eks_service_policy,
  ]
}

resource "aws_eks_node_group" "main" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "eks-datadog-node-group"
  node_role_arn   = aws_iam_role.eks_node_group_role.arn
  subnet_ids      = aws_subnet.public[*].id
  instance_types  = ["t3.medium"]
  scaling_config {
    desired_size = 2
    max_size     = 3
    min_size     = 1
  }
  depends_on = [
    aws_iam_role_policy_attachment.eks_worker_node_policy,
    aws_iam_role_policy_attachment.eks_cni_policy,
    aws_iam_role_policy_attachment.ec2_container_registry_readonly,
  ]
}

data "aws_eks_cluster_auth" "main" {
  name = aws_eks_cluster.main.name
}

provider "kubernetes" {
  host                   = aws_eks_cluster.main.endpoint
  cluster_ca_certificate = base64decode(aws_eks_cluster.main.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.main.token
}

provider "helm" {
  kubernetes {
    host                   = aws_eks_cluster.main.endpoint
    cluster_ca_certificate = base64decode(aws_eks_cluster.main.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.main.token
  }
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default"
  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }
  set {
    name  = "targetSystem"
    value = "linux"
  }
  # Enable APM, Log, and Process Collection
  set {
    name  = "apm.enabled"
    value = true
  }
  set {
    name  = "logs.enabled"
    value = true
  }
  set {
    name  = "logs.containerCollectAll"
    value = true
  }
  set {
    name  = "processAgent.enabled"
    value = true
  }
  set {
    name  = "clusterAgent.enabled"
    value = true
  }
  set {
    name  = "kubeStateMetricsExternal"
    value = true
  }
  set {
    name  = "datadog.site"
    value = "datadoghq.com" # or eu.datadoghq.com etc.
  }
  depends_on = [aws_eks_node_group.main]
}

# Configure Datadog Provider
provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

# Example: Datadog Monitor for high EKS node CPU utilization
resource "datadog_monitor" "high_cpu_eks_node" {
  name               = "[EKS] High Node CPU Utilization on {{host.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:system.cpu.idle{cluster_name:eks-datadog-cluster} by {host} < 10" # Alert if CPU idle is less than 10% (i.e., usage is > 90%)
  message            = "High CPU utilization detected on EKS node {{host.name}} ({{value}}% idle). Please investigate. @webhook-pagerduty-eks-alerts"
  tags               = ["environment:dev", "team:devops", "service:eks"]
  require_full_window = false
  notify_no_data      = false
  new_group_delay     = 60
  no_data_timeframe   = 20
  renotify_interval   = 0
  escalation_message  = "CPU usage remains high. PagerDuty incident escalated."
  # The @webhook-pagerduty-eks-alerts tag implies a Datadog integration webhook
  # configured to send to PagerDuty. For direct PagerDuty notifications,
  # you'd configure the PagerDuty integration in Datadog and reference it here.
}

# In a production setup, you would have a datadog_integration_pagerduty resource
# or a datadog_integration_webhook to integrate directly.
# For simplicity, we assume the PagerDuty integration is manually configured in Datadog
# and referenced by its name in the monitor message (e.g., @pagerduty-service-name).
#
# If using `datadog_integration_pagerduty`:
# resource "datadog_integration_pagerduty" "pagerduty_integration" {
#   api_key = var.agerduty_api_key
#   services {
#     service_name = "EKS Critical Alerts"
#     service_key  = var.pagerduty_service_integration_key
#   }
# }
#
# Then the monitor message would be:
# message = "High CPU utilization detected on EKS node {{host.name}} ({{value}}% idle). Please investigate. @pagerduty-eks-critical-alerts"


output "eks_cluster_name" {
  description = "Name of the EKS cluster"
  value       = aws_eks_cluster.main.name
}

output "kubeconfig_command" {
  description = "Command to configure kubectl"
  value       = "aws eks update-kubeconfig --name ${aws_eks_cluster.main.name} --region ${data.aws_region.current.name}"
}

output "datadog_agent_helm_release_status" {
  description = "Status of the Datadog Agent Helm release"
  value       = helm_release.datadog_agent.status
}
    

Variables (`variables.tf`)

variable "region" {
  description = "AWS region"
  type        = string
  default     = "us-east-1"
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

# variable "pagerduty_api_key" {
#   description = "PagerDuty API Key (for direct PagerDuty provider integration, if used)"
#   type        = string
#   sensitive   = true
# }

# variable "pagerduty_service_integration_key" {
#   description = "PagerDuty Service Integration Key (for Datadog integration)"
#   type        = string
#   sensitive   = true
# }
    

Providers and Data Sources (`versions.tf`)

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
  }
}

provider "aws" {
  region = var.region
}

data "aws_region" "current" {}
data "aws_availability_zones" "available" {}
    

Deployment Steps

Follow these steps to deploy your EKS cluster with integrated monitoring and incident response:

Save the Configuration: Place the Terraform code into .tf files as described (e.g., main.tf, variables.tf, versions.tf).
Initialize Terraform: Open your terminal in the project directory and run:
terraform init
Set Environment Variables: Export your Datadog API and Application keys:
export TF_VAR_datadog_api_key="<YOUR_DATADOG_API_KEY>" export TF_VAR_datadog_app_key="<YOUR_DATADOG_APP_KEY>"
(Alternatively, you can provide these via a terraform.tfvars file or directly on the command line).
Review the Plan: Examine the changes Terraform will make:
terraform plan
Apply the Configuration: Execute the deployment:
terraform apply --auto-approve
This process can take 10-15 minutes as EKS cluster creation is time-consuming.
Configure Kubeconfig: After deployment, update your kubeconfig using the output provided by Terraform:
aws eks update-kubeconfig --name eks-datadog-cluster --region us-east-1
(Replace us-east-1 with your region).

Verification

Once deployed, verify the integration:

EKS Cluster: Check its status:
kubectl get nodes
Datadog Agent: Verify the DaemonSet and its pods:
kubectl get daemonset datadog-agent -n default kubectl get pods -l app=datadog-agent -n default
Datadog Dashboards: Log into your Datadog account. You should see your EKS cluster, nodes, and running pods under "Infrastructure" -> "Hosts" and "Kubernetes" -> "Clusters." The deployed monitor should also appear under "Monitors" -> "Manage Monitors."
PagerDuty: Ensure your Datadog integration with PagerDuty is set up correctly in Datadog. When the CPU utilization monitor triggers (you can simulate this by putting a load on an EKS node), an incident should be created in your configured PagerDuty service.

Troubleshooting Common Issues

EKS Cluster Creation Failure: Often related to IAM permissions or VPC/subnet misconfigurations. Review the Terraform apply logs carefully. Ensure your IAM roles have the necessary policies (AmazonEKSClusterPolicy, AmazonEKSServicePolicy for the cluster role; AmazonEKSWorkerNodePolicy, AmazonEC2ContainerRegistryReadOnly, AmazonEKS_CNI_Policy for the node group role).
Datadog Agent Pods Not Running:
- Check logs: kubectl logs <datadog-agent-pod-name> -n default
- Verify API/App keys: Ensure TF_VAR_datadog_api_key and TF_VAR_datadog_app_key are correctly set and accessible.
- Resource constraints: Ensure your EKS nodes have enough resources (CPU/memory) for the Datadog Agent.
Datadog Not Receiving Metrics:
- Check Datadog Agent status on the cluster.
- Verify network connectivity from EKS nodes to Datadog endpoints (e.g., app.datadoghq.com).
- Ensure security groups allow outbound traffic.
PagerDuty Incidents Not Triggering:
- Confirm the Datadog monitor is indeed triggering (check Datadog events).
- Verify the PagerDuty integration in Datadog (under "Integrations" -> "PagerDuty"). Make sure the correct service integration key is used.
- Ensure the monitor's message references the PagerDuty integration correctly (e.g., @pagerduty-service-name or via a configured webhook).

Conclusion

By following this guide, you have successfully leveraged Terraform to deploy a highly observable AWS EKS cluster, integrated with Datadog for comprehensive monitoring and PagerDuty for streamlined incident response. This architecture forms a robust foundation for running critical applications in a cloud-native environment, ensuring visibility into performance and rapid action during outages. Continuously refine your Datadog monitors and PagerDuty escalation policies to adapt to your evolving application needs and operational best practices.

CloudStack YAML