Automating Production-Ready AWS EKS Observability with Terraform, Datadog, and PagerDuty Incident Management

In the dynamic world of cloud-native applications, maintaining robust observability for Kubernetes clusters is paramount. AWS Elastic Kubernetes Service (EKS) offers a powerful platform, but achieving production-grade monitoring, logging, and tracing—coupled with efficient incident response—requires sophisticated tooling and automation. This guide provides a comprehensive technical walkthrough on how to seamlessly automate EKS observability using Terraform for Infrastructure as Code (IaC), Datadog for a unified observability platform, and PagerDuty for streamlined incident management. By the end, you’ll have a resilient, automated, and proactive system for your EKS environments.

Architecture Pro-Tip:

Always design your observability stack for scale and resiliency from day one. Leverage managed services where possible (like AWS EKS itself) and use IaC tools like Terraform to ensure reproducibility, version control, and auditability across all environments. Centralize observability data for easier analysis and integrate incident management platforms to ensure critical alerts translate into actionable incidents swiftly.

Why Automate EKS Observability?

Manual configuration of monitoring tools, alert thresholds, and incident escalation policies is prone to errors, inconsistency, and significant time investment, especially in fast-evolving Kubernetes environments. Automation through IaC addresses these challenges head-on:

Consistency and Reproducibility: Ensure identical observability setups across development, staging, and production environments.
Version Control and Auditability: Track changes, roll back configurations, and maintain a clear history of your observability stack.
Speed and Efficiency: Rapidly deploy and update observability components without manual intervention.
Reduced Human Error: Eliminate misconfigurations that can lead to missed alerts or false positives.
Scalability: Easily extend observability to new clusters or services as your infrastructure grows.

Key Components of Our Stack

AWS EKS: The Foundation

AWS EKS provides a managed Kubernetes control plane, simplifying the deployment, management, and scaling of containerized applications. It integrates deeply with other AWS services, making it a robust platform for production workloads.

Terraform: Infrastructure as Code for Automation

Terraform, by HashiCorp, is an open-source IaC tool that allows you to define and provision infrastructure using a declarative configuration language. We'll use Terraform to provision our EKS cluster, deploy the Datadog Agent, and configure Datadog monitors and PagerDuty services.

Datadog: Unified Observability Platform

Datadog offers a comprehensive, SaaS-based monitoring and analytics platform for cloud applications. It aggregates metrics, logs, traces, and user experience data, providing end-to-end visibility into your EKS clusters, applications, and underlying AWS infrastructure.

PagerDuty: Incident Management and On-Call Automation

PagerDuty is a leading digital operations management platform that provides on-call scheduling, alerting, and incident response automation. By integrating Datadog with PagerDuty, critical alerts from EKS are automatically routed to the right teams, ensuring rapid acknowledgment and resolution.

Prerequisites

Before you begin, ensure you have the following:

An AWS Account with administrative access.
A Datadog Account (with API and Application keys).
A PagerDuty Account (with an API token).
Terraform CLI installed (v1.0+ recommended).
kubectl CLI installed and configured.
AWS CLI installed and configured.
Helm CLI installed (for Datadog Agent deployment).

Step-by-Step Implementation Guide

Step 1: Setting up Your Terraform Project

Create a new directory for your Terraform project. Initialize your Terraform providers for AWS, Datadog, and PagerDuty.

provider "aws" {
  region = "us-east-1"
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
    pagerduty = {
      source  = "PagerDuty/pagerduty"
      version = "~> 3.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.0"
    }
  }
}
    

Ensure you set your datadog_api_key, datadog_app_key, and pagerduty_api_token as sensitive Terraform variables or environment variables.

Step 2: Provisioning an AWS EKS Cluster with Terraform

While a full EKS cluster setup is extensive, here's a simplified example using the popular terraform-aws-modules/eks/aws module. This module provides a robust, opinionated way to deploy EKS.

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0" # Use a stable version

  cluster_name    = "production-eks-cluster"
  cluster_version = "1.27"
  vpc_id          = var.vpc_id
  subnet_ids      = var.private_subnet_ids

  eks_managed_node_groups = {
    default = {
      min_size     = 2
      max_size     = 5
      desired_size = 3
      instance_types = ["t3.medium"]
    }
  }

  tags = {
    Environment = "Production"
    ManagedBy   = "Terraform"
  }
}
    

Remember to define vpc_id and private_subnet_ids in your variables.tf and provide values (e.g., from an existing VPC). Output the EKS cluster details for kubectl configuration.

Step 3: Integrating Datadog with EKS using Terraform

To get Datadog collecting data from your EKS cluster, you need to deploy the Datadog Agent. The most common and recommended way is using the Datadog Helm chart.

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }

  set {
    name  = "datadog.site"
    value = "datadoghq.com" # or "eu.datadoghq.com" etc.
  }

  set {
    name  = "datadog.kubeStateMetricsCore.enabled"
    value = "true"
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "clusterChecksRunner.enabled"
    value = "true"
  }

  set {
    name  = "logs.enabled"
    value = "true"
  }

  set {
    name  = "apm.enabled"
    value = "true"
  }

  set {
    name  = "processAgent.enabled"
    value = "true"
  }

  # Ensure the Helm provider is configured to use the EKS cluster
  # This typically involves setting KUBECONFIG or using the `kubernetes` provider data source
  # For simplicity, assuming kubectl context is set correctly or using a `kubernetes_config` block
  # in a `kubernetes` provider.
}
    

This Helm release deploys the Datadog Agent, Cluster Agent, and enables various features like logs, APM, and process monitoring. After applying, you should see your EKS metrics, logs, and traces appear in your Datadog account.

Step 4: Connecting Datadog to PagerDuty

First, define a PagerDuty service that Datadog will integrate with. This service represents the component or team responsible for handling incidents generated by Datadog.

resource "pagerduty_user" "devops_engineer" {
  name  = "DevOps Engineer"
  email = "devops@example.com"
}

resource "pagerduty_team" "sre_team" {
  name        = "SRE Team"
  description = "Site Reliability Engineering Team"
}

resource "pagerduty_team_membership" "devops_member" {
  user_id = pagerduty_user.devops_engineer.id
  team_id = pagerduty_team.sre_team.id
  role    = "admin" # or "member", "observer"
}

resource "pagerduty_escalation_policy" "eks_escalation_policy" {
  name      = "EKS Critical Incident Policy"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "user"
      id   = pagerduty_user.devops_engineer.id
    }
  }

  rule {
    escalation_delay_in_minutes = 20
    target {
      type = "team"
      id   = pagerduty_team.sre_team.id
    }
  }
}

resource "pagerduty_service" "eks_observability_service" {
  name                      = "EKS Observability"
  auto_resolve_timeout_days = 1
  acknowledgement_timeout_minutes = 10
  escalation_policy         = pagerduty_escalation_policy.eks_escalation_policy.id
}
    

Next, you'll need the PagerDuty integration key for this service, which Datadog uses to send alerts. This is automatically generated when the service is created.

Step 5: Defining Datadog Monitors with PagerDuty Integration

Now, create Datadog monitors that leverage the PagerDuty service for incident notification. This example sets up a monitor for high CPU utilization on EKS nodes.

Ready-to-Use Configuration Example: Datadog CPU Monitor for EKS with PagerDuty

This Terraform snippet demonstrates how to create a Datadog metric monitor that checks EKS node CPU utilization. If it exceeds 80% for 5 minutes, it triggers a warning, and if it exceeds 95% for 2 minutes, it triggers a critical alert, notifying the PagerDuty service we created earlier.

resource "datadog_monitor" "eks_node_cpu_utilization" {
  name                = "EKS Node CPU Utilization High on {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster_name:production-eks-cluster} by {host} < 20"
  message             = <<EOT
EKS node {{host.name}} is experiencing high CPU utilization.
Current idle CPU: {{value}}%
Please investigate.
@pagerduty-${pagerduty_service.eks_observability_service.name}
EOT
  tags                = ["environment:production", "service:eks", "severity:high"]
  priority            = 1
  restricted_roles    = []

  monitor_thresholds {
    warning          = 20 # 100-20 = 80% utilization
    critical         = 5  # 100-5  = 95% utilization
  }

  notify_no_data = false
  new_group_delay = 60
  notify_audit = false
  timeout_h = 0
  escalation_message = "CPU utilization remains high. Escalating to SRE team."
  # Optional: Automatically resolve incidents if CPU drops below thresholds
  renotify_interval = 0
  no_data_timeframe = 120 # Warn if no data for 2 hours

  # This value represents the PagerDuty service name configured in PagerDuty,
  # which Datadog's integration uses to find the correct service key.
  # The `@pagerduty-` syntax is crucial for Datadog to map to the PagerDuty integration.
}
    

Apply these Terraform configurations. This will:

Provision or update your EKS cluster.
Deploy the Datadog Agent to collect data.
Create the PagerDuty service, user, team, and escalation policy.
Configure the Datadog monitor to trigger PagerDuty incidents when thresholds are breached.

Best Practices for Production Observability

Tag Everything: Use consistent tagging (e.g., env:production, service:api, team:sre) across AWS resources, Kubernetes objects, and Datadog monitors for powerful filtering and analytics.
Define Clear SLOs/SLAs: Establish Service Level Objectives (SLOs) and Service Level Agreements (SLAs) for your applications. Configure Datadog monitors to alert when these are at risk or breached.
Granular Alerting: Avoid alert fatigue. Create monitors for symptoms (e.g., high latency, error rates) rather than causes (e.g., high CPU), and use different alert severities (warning, critical) with distinct PagerDuty escalation policies.
Dashboards for Overview and Deep Dive: Build high-level dashboards for quick health checks and detailed dashboards for specific services or components to aid in troubleshooting. Automate their creation with Terraform's datadog_dashboard resource.
Cost Optimization: Regularly review your Datadog ingestion volumes for metrics, logs, and traces. Use Datadog's retention and sampling features to manage costs without sacrificing critical data.
Security & Compliance: Ensure sensitive data is not logged or traced unnecessarily. Leverage IAM roles for service accounts (IRSA) for the Datadog Agent for secure access to AWS resources.

Troubleshooting and Common Pitfalls

Datadog Agent Not Reporting Data

Check API/App Keys: Verify that datadog.apiKey and datadog.appKey in your Helm release are correct.
Agent Pod Status: Use kubectl get pods -n datadog to ensure Datadog Agent pods are running. Check logs with kubectl logs <datadog-agent-pod> -n datadog for errors.
Network Connectivity: Ensure your EKS nodes can reach api.datadoghq.com (or your specific Datadog site).
RBAC Permissions: The Datadog Agent requires specific RBAC permissions within Kubernetes. The Helm chart typically handles this, but custom EKS security policies might interfere.

PagerDuty Incidents Not Firing

Monitor Thresholds: Double-check that your Datadog monitor's query and thresholds are actually being breached.
PagerDuty Integration String: Ensure the @pagerduty-${pagerduty_service.eks_observability_service.name} syntax in the Datadog monitor message is correct and matches the service name in PagerDuty. The pagerduty_service resource in Terraform automatically creates the necessary integration keys for Datadog to use.
Datadog & PagerDuty Setup: In Datadog, go to Integrations -> PagerDuty and ensure the integration is set up and healthy. The Terraform pagerduty_service creates the PagerDuty side, Datadog's side is typically manual or can be configured via Datadog API directly, though often the @pagerduty- syntax is enough.
Escalation Policy & On-Call Schedule: Verify that the PagerDuty escalation policy is correctly configured with users/teams on an active on-call schedule.

Conclusion

Automating your production-ready AWS EKS observability with Terraform, Datadog, and PagerDuty transforms a complex, manual process into a resilient, scalable, and proactive system. By defining your entire observability stack as code, you gain consistency, reproducibility, and faster iteration cycles. This not only enhances your ability to detect and resolve issues quickly but also frees your engineering teams to focus on innovation rather than firefighting. Embrace these practices to build truly robust and observable cloud-native applications on AWS EKS.

CloudStack YAML