Terraform-Driven AWS EKS Observability: Datadog APM, Prometheus Metrics, and PagerDuty Incident Management

In the dynamic landscape of cloud-native applications, maintaining robust observability for Kubernetes clusters is paramount. AWS EKS provides a powerful, managed Kubernetes service, but understanding its internal workings and the health of your deployed applications requires a comprehensive observability strategy. This guide explores how to build a fully automated, Terraform-driven observability stack for AWS EKS, integrating Datadog for APM and unified metrics (including Prometheus), and PagerDuty for streamlined incident management. By leveraging Infrastructure as Code (IaC), you can ensure consistent, scalable, and reliable monitoring from day one.

Architecture Pro-Tip: Unified Observability Strategy

When designing your EKS observability strategy, prioritize a unified platform. While Prometheus is excellent for scraping metrics, integrating its data into a comprehensive platform like Datadog provides a single pane of glass for metrics, logs, traces (APM), and synthetic checks. This reduces context switching for engineers and accelerates incident resolution. Use Terraform to codify this integration from the outset for consistency across environments.

Why This Stack? The Benefits of Integrated Observability

Operating highly available applications on AWS EKS demands proactive monitoring and efficient incident response. This integrated stack delivers:

Automated Deployment with Terraform: Provision and configure all observability components reliably and repeatably, eliminating manual errors and accelerating setup.
End-to-End Visibility (Datadog APM): Gain deep insights into application performance, identify bottlenecks, and trace requests across microservices.
Comprehensive Metrics (Prometheus & Datadog): Collect granular infrastructure, Kubernetes, and application metrics, including those exposed via Prometheus endpoints, all centralized in Datadog for correlation.
Proactive Incident Management (PagerDuty): Automate alert routing, on-call scheduling, and incident escalation, ensuring critical issues are addressed promptly.
Reduced Mean Time To Resolution (MTTR): Faster identification of root causes through correlated data and efficient incident handoffs.

Core Components Overview

AWS EKS: The Foundation

Amazon Elastic Kubernetes Service (EKS) provides a managed control plane for Kubernetes, simplifying the deployment, management, and scaling of containerized applications. Our observability stack will run within and monitor the EKS cluster.

Terraform: Infrastructure as Code (IaC)

Terraform is used to define and provision the entire infrastructure stack, from the EKS cluster itself to the Datadog agents and PagerDuty integrations. This ensures consistency and version control.

AWS Provider: Manages EKS cluster, VPC, IAM, etc.
Kubernetes Provider: Interacts with the EKS cluster to deploy resources like Service Accounts, RBAC.
Helm Provider: Deploys Helm charts for Datadog Agent, Prometheus (if separate).
Datadog Provider: Manages Datadog monitors, dashboards, and integrations.
PagerDuty Provider: Manages PagerDuty services, escalation policies, and users.

Datadog APM & Metrics

Datadog is a unified monitoring and analytics platform. For EKS, we'll leverage it for:

Application Performance Monitoring (APM): Distributed tracing across microservices to pinpoint latency issues and errors.
Infrastructure Monitoring: CPU, memory, network, disk metrics for EKS nodes and pods.
Log Management: Centralized collection and analysis of container and application logs.
Prometheus Metric Collection: Datadog can automatically scrape and ingest metrics from applications exposing Prometheus endpoints, centralizing them with other Datadog metrics.
Custom Metrics & Events: Monitor specific application-level data.

PagerDuty Incident Management

PagerDuty streamlines incident response by consolidating alerts, orchestrating on-call schedules, and providing real-time notification capabilities. We will integrate Datadog alerts directly into PagerDuty.

Prerequisites

Before you begin, ensure you have:

An existing AWS EKS cluster, or Terraform code to provision one.
Terraform CLI installed and configured with AWS credentials.
kubectl configured to connect to your EKS cluster.
A Datadog account with an API key and Application key.
A PagerDuty account with an API key (for Terraform provider) and a service integration key (for Datadog).

Terraform Implementation: Putting It All Together

Let's break down the Terraform configuration for each component. We'll assume your EKS cluster is already provisioned or is managed by a separate Terraform module.

1. Configure Terraform Providers

Initialize your Terraform project and configure the necessary providers.

provider "aws" {
  region = "us-east-1" # Replace with your AWS region
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.this.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.this.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.this.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_token
}

data "aws_eks_cluster" "this" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "this" {
  name = var.eks_cluster_name
}
    

2. Deploy Datadog Agent with Helm

The Datadog Agent will be deployed as a DaemonSet in your EKS cluster to collect metrics, logs, and traces. We'll enable APM, log collection, and Prometheus metric scraping via its Helm chart values.

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  values = [
    yamlencode({
      # Global settings for Datadog Agent
      clusterName = var.eks_cluster_name
      site        = "datadoghq.com" # Or eu.datadoghq.com, us3.datadoghq.com, etc.

      # APM Configuration
      apm = {
        enabled = true
        hostPort = 8126
      }

      # Log Collection Configuration
      logs = {
        enabled = true
        containerCollectAll = true
      }

      # Kubernetes & Infrastructure Monitoring
      kubeStateMetricsCore = {
        enabled = true # Deploy Kube-state-metrics if not already present
      }
      kubelet = {
        tlsVerify = false # Adjust based on your Kubelet setup
      }

      # Prometheus Scraper Configuration (Datadog Agent can scrape Prometheus endpoints)
      # This enables the Datadog Agent to discover and scrape Prometheus metrics from your applications.
      datadog = {
        apm_enabled = true
        logs_enabled = true
        containerCollectAll = true
        process_agent = {
          enabled = true
        }
        dogstatsd = {
          nonLocalTraffic = true
        }
        confd = {
          "prometheus.yaml" = <Note on Prometheus Scraper: The example prometheus.yaml in the confd section demonstrates how to configure the Datadog Agent to scrape Prometheus endpoints. Datadog's Auto-Discovery mechanisms can simplify this further, automatically finding services annotated for Prometheus exposition. For services exposing Prometheus metrics, ensure your Kubernetes service/pod manifest includes the correct annotations for Datadog to pick them up, e.g., prometheus.io/scrape: "true", prometheus.io/port: "8000".
3. Configure PagerDuty Service with TerraformWe'll define a PagerDuty service, escalation policy, and a Datadog integration using the PagerDuty Terraform provider.

# Define an Escalation Policy for EKS incidents
resource "pagerduty_escalation_policy" "eks_observability_policy" {
  name      = "EKS Observability Incidents"
  num_loops = 2
  rule {
    delay = 10
    target {
      type = "user"
      id   = var.pagerduty_oncall_user_id # Replace with a PagerDuty user ID
    }
  }
}

# Define a PagerDuty service for EKS Observability
resource "pagerduty_service" "eks_observability_service" {
  name                    = "${var.eks_cluster_name}-Observability"
  auto_resolve_timeout_s  = 1800 # 30 minutes
  acknowledgement_timeout_s = 600 # 10 minutes
  escalation_policy       = pagerduty_escalation_policy.eks_observability_policy.id
}

# Create a Datadog integration for the PagerDuty service
resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog"
  service_id  = pagerduty_service.eks_observability_service.id
  type        = "datadog_inbound_integration"
}

# Output the integration key to use in Datadog monitors
output "pagerduty_datadog_integration_key" {
  value = pagerduty_service_integration.datadog_integration.integration_key
  sensitive = true
}
    
4. Create a Datadog Monitor and Integrate with PagerDutyNow, let's create a sample Datadog monitor (e.g., for high EKS node CPU utilization) and configure it to send alerts to the PagerDuty service created above.

resource "datadog_monitor" "eks_high_cpu" {
  name                = "EKS Cluster: High Node CPU Utilization - ${var.eks_cluster_name}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.eks_cluster_name}} by {host} > 80"
  message             = <Important: The @pagerduty-${pagerduty_service.eks_observability_service.name} syntax in the message block is crucial. Datadog automatically detects this PagerDuty integration by name and sends alerts to it. Ensure the PagerDuty integration is set up in Datadog (this is done automatically if you follow the guide and use the integration key from the pagerduty_service_integration resource to set up the Datadog integration in PagerDuty, or if Datadog is directly configured with PagerDuty credentials which Terraform can also manage). The service name should match what Datadog expects for its PagerDuty integration.
5. Define VariablesCreate a variables.tf file for sensitive and environment-specific inputs.

variable "eks_cluster_name" {
  description = "Name of the EKS cluster to monitor"
  type        = string
}

variable "datadog_api_key" {
  description = "Your Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Your Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_token" {
  description = "PagerDuty API Token (requires v2 API token for Terraform)"
  type        = string
  sensitive   = true
}

variable "pagerduty_oncall_user_id" {
  description = "PagerDuty User ID to assign to the escalation policy"
  type        = string
}

variable "environment" {
  description = "Environment tag (e.g., dev, staging, prod)"
  type        = string
  default     = "dev"
}
    
Deployment WorkflowFollow these steps to deploy your observability stack:
Initialize Terraform: Navigate to your project directory and run terraform init.
Plan Changes: Run terraform plan -var-file="your-vars.tfvars" to preview the changes. Ensure you supply your sensitive variables via a .tfvars file or environment variables.
Apply Configuration: Execute terraform apply -var-file="your-vars.tfvars" to provision the resources.
Verify Datadog Integration: Check your Datadog dashboard for EKS metrics, logs, and APM traces. Ensure the Datadog Agent pods are running in your EKS cluster.
Verify PagerDuty Setup: Confirm the new service and escalation policy exist in your PagerDuty account.


Best Practices and Advanced ConsiderationsGitOps for Observability: Treat your observability configuration as code and manage it via Git. Integrate Terraform into your CI/CD pipeline for automated deployments on every change.
Resource Limits & Requests: Configure appropriate resource limits and requests for your Datadog Agent pods to prevent them from consuming too many resources or being evicted.
Security Best Practices:
            Use AWS Secrets Manager or HashiCorp Vault to store sensitive API keys and tokens, integrating them securely with Terraform.
Implement fine-grained IAM roles for service accounts (IRSA) for the Datadog Agent, granting only necessary permissions.

        
Custom Prometheus Metrics: For application-specific metrics, ensure your applications expose a /metrics endpoint and configure Datadog's Auto-Discovery or custom checks to scrape them.
Datadog Dashboards: Complement your monitors with comprehensive Datadog dashboards for real-time visualization and troubleshooting. These can also be managed via the Datadog Terraform provider.
Sophisticated PagerDuty Escalations: Leverage PagerDuty's advanced features like schedules, dependencies, and suppression rules to fine-tune your incident response.
Troubleshooting Common IssuesDatadog Agent Pods Not Ready: Check logs of Datadog Agent pods (kubectl logs -n datadog <datadog-agent-pod>). Verify API and APP keys are correct and network policies allow outbound traffic to Datadog endpoints.
No Metrics in Datadog:
            Ensure the Datadog Agent has the correct permissions to access Kubelet and API server.
For Prometheus metrics, verify the application pods are annotated correctly and the Datadog Agent's Prometheus check is configured to discover them.

        
PagerDuty Alerts Not Triggering:
            Double-check the monitor's message content for the correct PagerDuty integration syntax (e.g., @pagerduty-<YOUR_SERVICE_NAME>).
Verify the Datadog monitor's query is actually triggering and that the integration key for PagerDuty in Datadog is valid.

        
ConclusionImplementing robust observability for AWS EKS is a non-negotiable for reliable operations. By unifying Datadog APM, Prometheus metrics collection, and PagerDuty incident management, all driven by Terraform, you create a powerful, automated, and scalable system for monitoring your cloud-native applications. This approach not only provides deep visibility but also streamlines incident response, ultimately leading to higher application availability and a more efficient engineering team.

CloudStack YAML