Terraform AWS EKS Observability with Datadog APM and PagerDuty Incident Management

In the dynamic world of cloud-native applications, maintaining robust observability and efficient incident management for your Kubernetes clusters is not just a best practice—it's a necessity. This guide provides a comprehensive, technical deep-dive into how to establish a resilient observability framework for AWS EKS using Terraform, integrated with Datadog for APM, metrics, and logs, and PagerDuty for streamlined incident response.

Architecture Pro-Tip:

Always design your observability stack for scale and resiliency from day one. Leverage dedicated IAM roles for integrations (Datadog, PagerDuty) to adhere to the principle of least privilege. For EKS, consider separate node groups for critical workloads versus monitoring agents to prevent resource contention. Furthermore, centralize your Terraform state management (e.g., S3 backend with DynamoDB locking) for collaborative and secure infrastructure deployments.

Why Terraform, EKS, Datadog, and PagerDuty?

Each component plays a crucial role in building a robust, automated, and observable cloud-native environment:

Terraform: As an Infrastructure as Code (IaC) tool, Terraform allows you to define and provision your entire infrastructure—from AWS EKS clusters to Datadog monitors and PagerDuty services—in a declarative manner. This ensures consistency, repeatability, and version control.
AWS EKS: Amazon Elastic Kubernetes Service offers a highly available and scalable managed Kubernetes control plane, simplifying the deployment and management of containerized applications on AWS.
Datadog APM: Datadog provides end-to-end observability, aggregating metrics, logs, and traces from your EKS clusters, applications, and AWS infrastructure. Its Application Performance Monitoring (APM) capabilities are essential for understanding application health and performance bottlenecks.
PagerDuty: PagerDuty streamlines incident response by alerting the right team members at the right time based on signals from monitoring tools like Datadog. It centralizes incident communications, automates escalation policies, and helps resolve issues faster.

Prerequisites

Before you begin, ensure you have the following:

An AWS account with appropriate permissions to create EKS clusters, IAM roles, and other AWS resources.
A Datadog account with an API key and Application key.
A PagerDuty account with an API key.
Terraform CLI installed (v1.0.0+ recommended).
kubectl installed.
Helm CLI installed.

Step-by-Step Implementation with Terraform

1. Configure AWS Provider and EKS Cluster

We'll start by defining our AWS provider and creating the EKS cluster and its node group. This involves setting up the VPC, subnets, IAM roles for EKS, and the EKS cluster itself.

2. Integrate Datadog with AWS

For Datadog to collect metrics from AWS services (EC2, Load Balancers, CloudWatch), you need to establish an AWS integration. This is typically done by creating an IAM role in your AWS account that Datadog can assume.

3. Deploy Datadog Agent to EKS

The Datadog Agent runs as a DaemonSet on your EKS worker nodes, collecting host metrics, process data, logs, and traces from your Kubernetes environment. We'll deploy it using the official Datadog Helm chart.

4. Configure Datadog Monitors and Dashboards

Leverage Terraform to define Datadog monitors for critical EKS health indicators (e.g., node CPU utilization, pod restarts, EKS control plane health) and dashboards for visual insights.

5. Set up PagerDuty Service and Integration

Create a PagerDuty service that will receive incidents from Datadog. The integration between Datadog and PagerDuty can be configured directly in Datadog, often via a webhook or dedicated integration type, which is then referenced in your Datadog monitors.

Below is a comprehensive Terraform configuration combining these steps. This example assumes you have a pre-existing VPC and subnets. For production environments, consider dedicated modules for VPC and EKS.

Comprehensive Terraform Configuration

resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
  tags = { Name = "eks-observability-vpc" }
}

resource "aws_subnet" "public" {
  count = 2
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.0.${count.index}.0/24"
  availability_zone = element(["us-east-1a", "us-east-1b"], count.index)
  map_public_ip_on_launch = true
  tags = { Name = "eks-observability-public-subnet-${count.index}" }
}

resource "aws_internet_gateway" "gw" {
  vpc_id = aws_vpc.main.id
  tags = { Name = "eks-observability-igw" }
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.gw.id
  }
  tags = { Name = "eks-observability-public-rt" }
}

resource "aws_route_table_association" "public" {
  count          = length(aws_subnet.public)
  subnet_id      = aws_subnet.public[count.index].id
  route_table_id = aws_route_table.public.id
}

# EKS Cluster IAM Role
resource "aws_iam_role" "eks_cluster_role" {
  name = "eks-observability-cluster-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "eks.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eks_cluster_role.name
}

resource "aws_iam_role_policy_attachment" "eks_service_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
  role       = aws_iam_role.eks_cluster_role.name
}

# EKS Node Group IAM Role
resource "aws_iam_role" "eks_node_role" {
  name = "eks-observability-node-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_worker_node_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.eks_node_role.name
}

resource "aws_iam_role_policy_attachment" "eks_cni_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.eks_node_role.name
}

resource "aws_iam_role_policy_attachment" "ec2_container_registry_readonly" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.eks_node_role.name
}

# EKS Cluster
resource "aws_eks_cluster" "main" {
  name     = "eks-observability-cluster"
  role_arn = aws_iam_role.eks_cluster_role.arn
  vpc_config {
    subnet_ids = aws_subnet.public[*].id
  }
  version = "1.28" # Use a recent, stable EKS version

  depends_on = [
    aws_iam_role_policy_attachment.eks_cluster_policy,
    aws_iam_role_policy_attachment.eks_service_policy,
  ]
}

# EKS Node Group
resource "aws_eks_node_group" "main" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "eks-observability-node-group"
  node_role_arn   = aws_iam_role.eks_node_role.arn
  subnet_ids      = aws_subnet.public[*].id
  instance_types  = ["t3.medium"] # Choose appropriate instance types
  desired_size    = 2
  min_size        = 1
  max_size        = 3

  depends_on = [
    aws_iam_role_policy_attachment.eks_worker_node_policy,
    aws_iam_role_policy_attachment.eks_cni_policy,
    aws_iam_role_policy_attachment.ec2_container_registry_readonly,
  ]
}

# Datadog AWS Integration IAM Role
resource "aws_iam_role" "datadog_integration_role" {
  name = "datadog-integration-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          AWS = "arn:aws:iam:::root" # Replace with Datadog's AWS account ID
        }
        Action = "sts:AssumeRole"
        Condition = {
          StringEquals = {
            "sts:ExternalId" = "" # Replace with your Datadog External ID
          }
        }
      }
    ]
  })
}

resource "aws_iam_policy" "datadog_read_only_policy" {
  name = "datadog-read-only-policy"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "ec2:Describe*",
          "rds:Describe*",
          "s3:GetBucketLocation",
          "s3:ListAllMyBuckets",
          "cloudwatch:ListMetrics",
          "cloudwatch:GetMetricStatistics",
          "tag:GetResources",
          "eks:DescribeCluster",
          # Add other necessary read-only permissions for AWS services you use
        ]
        Effect   = "Allow"
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "datadog_read_only_attach" {
  role       = aws_iam_role.datadog_integration_role.name
  policy_arn = aws_iam_policy.datadog_read_only_policy.arn
}

# Datadog Provider Configuration
terraform {
  required_providers {
    datadog = {
      source = "DataDog/datadog"
      version = "~> 3.0"
    }
    kubernetes = {
      source = "hashicorp/kubernetes"
      version = "~> 2.0"
    }
    helm = {
      source = "hashicorp/helm"
      version = "~> 2.0"
    }
    pagerduty = {
      source = "PagerDuty/pagerduty"
      version = "~> 2.0"
    }
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

# PagerDuty Provider Configuration
provider "pagerduty" {
  token = var.pagerduty_api_key
}

# Datadog AWS Integration
resource "datadog_integration_aws" "main" {
  account_id  = data.aws_caller_identity.current.account_id
  role_name   = aws_iam_role.datadog_integration_role.name
  filter_tags = ["environment:production"] # Optional: Filter EC2 instances by tag
}

# Kubernetes Provider Configuration (for Helm chart deployment)
data "aws_eks_cluster_auth" "main" {
  name = aws_eks_cluster.main.name
}

provider "kubernetes" {
  host                   = aws_eks_cluster.main.endpoint
  cluster_ca_certificate = base64decode(aws_eks_cluster.main.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.main.token
}

# Datadog Agent Helm Chart
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Consider creating a dedicated namespace for monitoring
  version    = "2.33.0" # Use a stable version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }
  set {
    name  = "datadog.clusterName"
    value = aws_eks_cluster.main.name
  }
  set {
    name  = "targetSystem"
    value = "linux"
  }
  set {
    name  = "agents.tolerations[0].effect"
    value = "NoSchedule"
  }
  set {
    name  = "agents.tolerations[0].key"
    value = "node-role.kubernetes.io/master"
  }
  set {
    name  = "agents.tolerations[0].operator"
    value = "Exists"
  }
  # Enable APM for trace collection
  set {
    name  = "datadog.apmEnabled"
    value = "true"
  }
  set {
    name  = "datadog.apmConfig.apmNonLocalTraffic"
    value = "true"
  }
  # Enable Log collection
  set {
    name  = "datadog.logs.enabled"
    value = "true"
  }
  set {
    name  = "datadog.logs.containerCollectAll"
    value = "true"
  }
  # Enable Process agent for process-level metrics
  set {
    name  = "datadog.processAgent.enabled"
    value = "true"
  }
  # Enable EKS specific metrics collection
  set {
    name  = "datadog.kubeStateMetricsCore.enabled"
    value = "true"
  }
  set {
    name  = "datadog.networkHostPort"
    value = "false" # Set to false if not exposing host ports
  }
  set {
    name  = "datadog.admissionController.enabled"
    value = "true"
  }
  set {
    name  = "datadog.hostPort"
    value = "false"
  }
  set {
    name  = "datadog.prometheusScrape.enabled"
    value = "true"
  }
}

# PagerDuty Service
resource "pagerduty_service" "eks_observability_service" {
  name        = "EKS Observability Service"
  auto_resolve_timeout = "14400" # 4 hours
  acknowledgement_timeout = "600" # 10 minutes
  escalation_policy = pagerduty_escalation_policy.devops_ep.id # Assume you have an EP defined
  alert_creation = "create_incidents_and_alert_supressed_notifications"
}

# PagerDuty Service Integration for Datadog
resource "pagerduty_service_integration" "datadog_integration" {
  name    = "Datadog Integration"
  type    = "datadog_inbound_integration" # Use the appropriate integration type
  service = pagerduty_service.eks_observability_service.id
}

# Datadog Monitor for EKS Node CPU Utilization
resource "datadog_monitor" "eks_node_cpu_utilization" {
  name = "[EKS] High Node CPU Utilization on {{host.name}}"
  type = "metric alert"
  query = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster_name:${aws_eks_cluster.main.name}} by {host} < 10" # Alert if idle CPU is < 10% (i.e., usage > 90%)
  message = "Node {{host.name}} is experiencing high CPU utilization. Please investigate. @webhook-pagerduty" # Use the webhook integration name
  tags = ["environment:production", "service:eks", "severity:critical"]
  priority = 1
  options {
    thresholds = {
      warning  = 20
      critical = 10
    }
    notify_audit = false
    locked       = false
    timeout_h    = 0
    require_full_window = true
    renotify_interval = 0
    evaluation_delay = 300
    new_host_delay = 300
    notify_no_data = false
    escalation_message = "Escalating: Node CPU remains high."
    include_tags = true
  }
}

# Datadog Monitor for EKS Pod Restarts
resource "datadog_monitor" "eks_pod_restarts" {
  name = "[EKS] High Pod Restarts in {{kube_container_name}} ({{kubernetes_pod_name}})"
  type = "log alert"
  query = "logs(\"status:error source:kubernetes.container restarts_total:[1 TO *] kubernetes_cluster_name:${aws_eks_cluster.main.name}\").index(\"main\").rollup(\"count\").last(\"5m\") > 3"
  message = "Pod {{kubernetes_pod_name}} in container {{kube_container_name}} is restarting frequently. Investigate application health. @webhook-pagerduty"
  tags = ["environment:production", "service:eks", "severity:high"]
  priority = 2
  options {
    thresholds = {
      critical = 3
    }
    notify_audit = false
    locked       = false
    timeout_h    = 0
    require_full_window = true
    renotify_interval = 0
    evaluation_delay = 300
    new_host_delay = 300
    notify_no_data = false
    escalation_message = "Escalating: Pod restarts continue."
    include_tags = true
  }
}

# Variables
variable "aws_region" {
  description = "AWS region"
  type        = string
  default     = "us-east-1"
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_key" {
  description = "PagerDuty API Key"
  type        = string
  sensitive   = true
}

# Data source for current AWS account ID
data "aws_caller_identity" "current" {}

# Example Escalation Policy (replace with your actual policy if it exists)
resource "pagerduty_escalation_policy" "devops_ep" {
  name      = "DevOps Team Escalation Policy"
  num_loops = 2
  rule {
    delay = 5
    target {
      id   = pagerduty_user.devops_engineer.id # Assume a PagerDuty user exists
      type = "user"
    }
  }
  rule {
    delay = 10
    target {
      id   = pagerduty_team.devops_team.id # Assume a PagerDuty team exists
      type = "team"
    }
  }
}

# Example PagerDuty User and Team (replace with your actual users/teams)
resource "pagerduty_user" "devops_engineer" {
  name  = "DevOps Engineer"
  email = "devops-engineer@example.com"
}

resource "pagerduty_team" "devops_team" {
  name = "DevOps Team"
}
    

Deployment Steps

Save the above configuration in a `main.tf` file.
Create a `terraform.tfvars` file (or use environment variables) to provide your sensitive API keys:
datadog_api_key = "YOUR_DATADOG_API_KEY" datadog_app_key = "YOUR_DATADOG_APP_KEY" pagerduty_api_key = "YOUR_PAGERDUTY_API_KEY"
Run `terraform init` to initialize the working directory and download providers.
Run `terraform plan` to review the changes Terraform will make.
Run `terraform apply` to provision the infrastructure. Confirm with `yes`.
After the EKS cluster is provisioned, configure `kubectl` to connect to it:
aws eks update-kubeconfig --name ${aws_eks_cluster.main.name} --region ${var.aws_region}
Verify Datadog Agent deployment: `kubectl get pods -l app=datadog --namespace default`. You should see Datadog Agent pods running.

Validation and Advanced Observability

Once deployed, verify the integration:

Datadog Dashboards: Navigate to your Datadog account. You should see data flowing from your EKS cluster and AWS services on the out-of-the-box AWS and Kubernetes dashboards. Your custom dashboards and monitors should also be visible.
APM Traces: If you have applications deployed with Datadog APM tracing libraries, you should start seeing traces and service maps in Datadog.
Incident Management: Trigger a test incident (e.g., by intentionally causing high CPU load on a node) and verify that PagerDuty receives the alert from Datadog and escalates according to your policy.

Troubleshooting and Best Practices

Common Issues:

IAM Permissions: Most integration issues stem from incorrect IAM roles or policies. Double-check that Datadog's AWS integration role has all necessary read-only permissions and the correct external ID.
Datadog Agent Connectivity: Ensure your EKS nodes can reach the Datadog ingest endpoints. Check security groups, network ACLs, and VPC routing. Look at the Datadog Agent pod logs (`kubectl logs -f `).
PagerDuty Webhook: Confirm the webhook URL in Datadog is correct and the PagerDuty integration service is active.
Terraform State Locking: Always use a remote backend (like S3 with DynamoDB locking) for Terraform state to prevent corruption, especially in team environments.

Best Practices:

Granular IAM: Refine IAM policies to only grant the minimum necessary permissions for each integration.
Dedicated Monitoring Namespace: Deploy Datadog agents and other monitoring tools into a dedicated Kubernetes namespace (e.g., `datadog`, `monitoring`) for better organization and resource isolation.
Version Control: Keep all Terraform configurations in a Git repository for version control, collaboration, and auditability.
Cost Optimization: Monitor Datadog ingestion volumes to manage costs. Use tag filtering for AWS integration and fine-tune log collection rules.
Alert Fatigue: Regularly review and fine-tune Datadog monitors and PagerDuty escalation policies to reduce alert fatigue and ensure actionable alerts.
Secret Management: Use AWS Secrets Manager or other secure secret management solutions for API keys instead of direct `tfvars` files in production.

Conclusion

By leveraging Terraform, AWS EKS, Datadog APM, and PagerDuty, you can build a highly observable and resilient cloud-native platform. This automated approach ensures that your infrastructure and applications are continuously monitored, performance issues are quickly identified, and incidents are managed efficiently, leading to improved reliability and faster recovery times. Embrace Infrastructure as Code to bring consistency, scalability, and peace of mind to your DevOps operations.

CloudStack YAML