Terraform-Driven Unified Observability for AWS EKS with Datadog, Prometheus, and PagerDuty

In the rapidly evolving landscape of cloud-native applications, maintaining robust observability for Kubernetes clusters is paramount. AWS EKS (Elastic Kubernetes Service) provides a powerful platform for orchestrating containers, but its dynamic nature demands a sophisticated, unified observability strategy. This technical guide explores how to leverage Terraform for Infrastructure as Code (IaC) to seamlessly integrate Datadog for comprehensive monitoring, Prometheus for open-source metrics collection, and PagerDuty for intelligent incident management, establishing a truly unified observability stack for your EKS environments.

Architecture Pro-Tip: Layered Observability as Code

Design your observability stack with a layered approach, managed entirely via Terraform. Separate your core EKS infrastructure from your observability tooling deployments (e.g., Datadog Agent, Prometheus operator), and then from your alert definitions and incident management configurations. This modularity ensures maintainability, reusability, and clear separation of concerns, crucial for scalable and resilient cloud-native operations.

The Need for Unified Observability in AWS EKS

AWS EKS environments are complex, consisting of numerous microservices, pods, nodes, and intricate network interactions. Traditional monitoring tools often fall short in providing a holistic view. A unified observability strategy brings together metrics, logs, traces, and events into a single pane of glass, enabling teams to:

Rapidly Detect Issues: Identify performance bottlenecks and errors before they impact users.
Accelerate Root Cause Analysis: Correlate data across different layers of the stack.
Improve System Reliability: Proactively address potential failures and optimize resource utilization.
Enhance Developer Productivity: Provide developers with the insights needed to build resilient applications.

Key Components of the Unified Stack

AWS EKS: The Foundation

EKS provides a managed Kubernetes control plane, simplifying the deployment, management, and scaling of containerized applications. While EKS offers basic integration with AWS CloudWatch, a deeper level of insight requires specialized tools.

Datadog: The Unified Monitoring Platform

Datadog is a leading SaaS monitoring and analytics platform for cloud-scale applications. It offers:

Metrics & APM: Real-time performance metrics, distributed tracing for microservices.
Log Management: Centralized log aggregation, processing, and analysis.
Network Performance Monitoring: Visibility into network traffic and dependencies.
Security Monitoring: Detection of threats and vulnerabilities.
Out-of-the-box Integrations: Seamlessly collects data from EKS, AWS services, and other popular technologies.

Prometheus: Open-Source Metric Collection

Prometheus is a powerful open-source monitoring system with a flexible query language (PromQL) and a robust ecosystem. While Datadog can act as a primary monitoring solution, integrating Prometheus brings several benefits:

Vendor Agnosticism: Prometheus metrics are a de facto standard, providing flexibility.
Rich Ecosystem: Many applications natively expose Prometheus endpoints.
Local Debugging: Operators can query metrics directly from the cluster.

Datadog offers excellent integration with Prometheus, allowing the Datadog Agent to scrape Prometheus endpoints and forward metrics to Datadog, providing the best of both worlds.

PagerDuty: Intelligent Incident Management

PagerDuty transforms alerts into actionable incidents, ensuring the right people are notified at the right time. Its capabilities include:

On-Call Management: Dynamic scheduling and escalation policies.
Intelligent Alerting: Deduplication, enrichment, and suppression of noisy alerts.
Automated Incident Response: Triggering runbooks and integrations with collaboration tools.

By integrating Datadog with PagerDuty, critical alerts detected by Datadog are immediately routed to the appropriate on-call teams for swift resolution.

Terraform: Infrastructure as Code (IaC)

Terraform enables you to define and provision your entire observability stack using declarative configuration files. This includes:

Deploying the Datadog Agent to EKS.
Configuring Datadog monitors, dashboards, and integrations.
Setting up PagerDuty services, escalation policies, and users.
Managing Kubernetes resources for Prometheus (if a full Prometheus deployment is desired, though Datadog can scrape directly).

Managing your observability infrastructure with Terraform ensures consistency, repeatability, version control, and auditability.

Prerequisites

Before you begin, ensure you have the following:

An AWS Account with administrative privileges.
Terraform CLI installed (v1.0+ recommended).
AWS CLI configured with appropriate credentials.
An existing AWS EKS cluster. (This guide assumes EKS is already provisioned. If not, Terraform can also provision EKS itself.)
Datadog API and Application Keys.
PagerDuty API Token.
kubectl configured to access your EKS cluster.
Helm CLI installed (for Datadog Agent deployment via Helm).

Step-by-Step Implementation with Terraform

1. Setting up Terraform Providers

First, define the necessary Terraform providers in your versions.tf or main.tf file.

provider "aws" {
  region = "us-east-1" # Or your desired AWS region
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.main.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.main.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.main.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.main.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

# Data sources to get EKS cluster details
data "aws_eks_cluster" "main" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "main" {
  name = var.eks_cluster_name
}

variable "eks_cluster_name" {
  description = "The name of your EKS cluster."
  type        = string
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}
    

2. Deploying the Datadog Agent to EKS

The Datadog Agent is deployed as a DaemonSet on EKS, ensuring it runs on every node to collect metrics, logs, and traces. We'll use the Helm provider for this.

Create a file named datadog-agent.tf:

Terraform Configuration for Datadog Agent

resource "kubernetes_namespace" "datadog" {
  metadata {
    name = "datadog"
  }
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = kubernetes_namespace.datadog.metadata[0].name

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "clusterChecksRunner.enabled"
    value = "true"
  }

  set {
    name  = "agents.log.level"
    value = "INFO"
  }

  set {
    name  = "kubeStateMetricsExternal.enabled"
    value = "true" # Enable Kube-state-metrics collection
  }

  # Enable Prometheus scraping by Datadog Agent
  set {
    name  = "datadog.prometheusScrape.enabled"
    value = "true"
  }
  set {
    name  = "datadog.prometheusScrape.serviceEndpoints"
    value = "true" # Scrape services annotated for Prometheus
  }
  set {
    name  = "datadog.prometheusScrape.kubelet.enabled"
    value = "true"
  }

  # Enable APM
  set {
    name  = "apm.enabled"
    value = "true"
  }

  # Enable Log Collection
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }
}
    

This configuration deploys the Datadog Agent, Cluster Agent, and Cluster Checks Runner, enabling collection for metrics, logs, APM, and crucial Kubernetes state metrics. Critically, datadog.prometheusScrape.enabled is set to true to allow the Datadog Agent to scrape Prometheus endpoints.

3. Integrating Prometheus Metrics via Datadog

Instead of deploying a full Prometheus server, we can leverage the Datadog Agent's ability to scrape Prometheus-formatted metrics directly from your application pods. You only need to annotate your Kubernetes services or pods.

Here's an example of how to annotate a Kubernetes Deployment to expose Prometheus metrics for the Datadog Agent to scrape. This would typically be part of your application's Kubernetes manifests, not directly in Terraform, but it's crucial for the integration.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-prometheus-app
spec:
  selector:
    matchLabels:
      app: my-prometheus-app
  replicas: 1
  template:
    metadata:
      labels:
        app: my-prometheus-app
      annotations:
        # Datadog annotation for Prometheus scraping
        # This tells the Datadog agent to scrape metrics from port 8080 at /metrics path
        ad.datadoghq.com/my-prometheus-app.check_names: '["prometheus"]'
        ad.datadoghq.com/my-prometheus-app.init_configs: '[{}]'
        ad.datadoghq.com/my-prometheus-app.instances: |
          [
            {
              "prometheus_url": "http://%%host%%:8080/metrics",
              "namespace": "my_app",
              "metrics": ["*"]
            }
          ]
    spec:
      containers:
      - name: my-prometheus-app
        image: my-company/my-prometheus-app:latest
        ports:
        - name: web
          containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: my-prometheus-app
spec:
  selector:
    app: my-prometheus-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
    

The Datadog Agent automatically discovers pods with these annotations and begins scraping metrics. These metrics are then forwarded to Datadog, where they can be visualized and used for alerting alongside other Datadog-collected data.

4. Configuring Datadog Monitors and Dashboards (Terraform)

Once metrics are flowing into Datadog, you can define monitors and create dashboards using Terraform. Here's an example of a simple CPU utilization monitor and an EKS dashboard.

Create a file named datadog-monitors.tf:

resource "datadog_monitor" "eks_node_cpu_high" {
  name               = "EKS Node CPU Utilization High (Terraform)"
  type               = "metric alert"
  query              = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster_name:${var.eks_cluster_name}} by {host} < 10"
  message            = "CPU utilization for {{host.name}} is critically high! @pagerduty-devops"
  tags               = ["environment:production", "service:eks", "severity:critical"]
  priority           = 1
  restricted_roles   = ["{{ YOUR_DD_ROLE_ID }}"] # Optional: restrict who can manage this monitor

  # Thresholds for alerting
  monitor_threshold_windows {
    recovery_window = "10m"
  }

  thresholds {
    critical = 90 # If idle CPU is less than 10%, means usage is > 90%
    warning  = 70
  }

  # Options for monitor behavior
  include_tags = true
  require_full_window = true
  new_group_delay     = 60
  new_host_delay      = 300
  notify_no_data      = false
  notify_audit        = false
  timeout_h           = 0
  no_data_timeframe   = 20
  renotify_interval   = 0
}

resource "datadog_dashboard" "eks_overview" {
  title        = "EKS Cluster Overview (Terraform)"
  description  = "High-level overview of EKS cluster health."
  layout_type  = "free"
  is_read_only = false
  tags         = ["environment:production", "service:eks"]

  widget {
    definition {
      title     = "Cluster CPU Utilization"
      type      = "timeseries"
      requests {
        q = "avg:system.cpu.idle{kubernetes_cluster_name:${var.eks_cluster_name}} by {host}"
        display_type = "line"
      }
    }
  }

  widget {
    definition {
      title     = "Cluster Memory Utilization"
      type      = "timeseries"
      requests {
        q = "avg:system.mem.used{kubernetes_cluster_name:${var.eks_cluster_name}} by {host}"
        display_type = "line"
      }
    }
  }

  widget {
    definition {
      title     = "Pods Running"
      type      = "timeseries"
      requests {
        q = "avg:kubernetes.pods.running{kubernetes_cluster_name:${var.eks_cluster_name}}"
        display_type = "line"
      }
    }
  }

  widget {
    definition {
      title     = "Kubernetes Events"
      type      = "event_stream"
      query     = "tags:kubernetes_cluster_name:${var.eks_cluster_name}"
      limit     = 10
      no_group  = false
    }
  }
}
    

Note the @pagerduty-devops in the monitor message. This relies on a Datadog integration with PagerDuty, which we'll set up next. The kubernetes_cluster_name tag is automatically applied by the Datadog Agent, making it easy to filter metrics specific to your EKS cluster.

5. Configuring PagerDuty for Incident Management

Now, let's configure PagerDuty services and escalation policies using Terraform. This example creates a team, an escalation policy, and a service that integrates with Datadog.

Create a file named pagerduty.tf:

Terraform Configuration for PagerDuty

# Example PagerDuty User (replace with your actual users or fetch existing)
resource "pagerduty_user" "devops_engineer_1" {
  name  = "DevOps Engineer One"
  email = "devops1@example.com"
  teams = [pagerduty_team.devops_team.id]
}

resource "pagerduty_user" "devops_engineer_2" {
  name  = "DevOps Engineer Two"
  email = "devops2@example.com"
  teams = [pagerduty_team.devops_team.id]
}

resource "pagerduty_team" "devops_team" {
  name        = "DevOps Team"
  description = "Team responsible for EKS and infrastructure operations"
}

# Example On-Call Schedule
resource "pagerduty_schedule" "devops_primary" {
  name      = "DevOps Primary Schedule"
  time_zone = "America/New_York"
  layer {
    name                 = "Daily Rotation"
    start                = "2023-01-01T09:00:00-05:00" # Adjust start time
    rotation_type        = "daily"
    rotation_virtual_start = "2023-01-01T09:00:00-05:00"
    users                = [pagerduty_user.devops_engineer_1.id, pagerduty_user.devops_engineer_2.id]
  }
}

# Escalation Policy
resource "pagerduty_escalation_policy" "eks_critical_policy" {
  name      = "EKS Critical Escalation Policy"
  num_loops = 2
  team      = pagerduty_team.devops_team.id

  rule {
    delay_in_minutes = 5
    target {
      type = "user"
      id   = pagerduty_user.devops_engineer_1.id
    }
    target {
      type = "schedule"
      id   = pagerduty_schedule.devops_primary.id
    }
  }

  rule {
    delay_in_minutes = 15
    target {
      type = "user"
      id   = pagerduty_user.devops_engineer_2.id
    }
    target {
      type = "team"
      id   = pagerduty_team.devops_team.id
    }
  }
}

# PagerDuty Service for EKS alerts
resource "pagerduty_service" "eks_observability_service" {
  name                    = "EKS Observability Alerts"
  auto_resolve_timeout    = "14400" # 4 hours
  acknowledgement_timeout = "600"   # 10 minutes
  escalation_policy       = pagerduty_escalation_policy.eks_critical_policy.id
  team                    = pagerduty_team.devops_team.id
}

# PagerDuty integration for Datadog
resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog Integration"
  service     = pagerduty_service.eks_observability_service.id
  type        = "datadog_inbound_integration"
}

output "pagerduty_datadog_integration_key" {
  description = "The integration key for Datadog-PagerDuty integration."
  value       = pagerduty_service_integration.datadog_integration.integration_key
  sensitive   = true
}
    

After running this Terraform code, the output "pagerduty_datadog_integration_key" will provide the integration key. You need to manually add this key to your Datadog PagerDuty integration settings (Integrations -> PagerDuty -> Add Account or Edit Account -> Add API Key). This allows Datadog monitors to send alerts to the specified PagerDuty service using the @pagerduty-devops (or whatever name you configure) notification syntax.

Execution Steps

Save the Terraform configurations into .tf files in a dedicated directory.
Populate the variable values (e.g., datadog_api_key, pagerduty_api_token, eks_cluster_name) in a terraform.tfvars file or as environment variables.
Run terraform init to initialize the providers.
Run terraform plan to review the changes that will be applied.
Run terraform apply to provision your observability stack.
Once applied, retrieve the pagerduty_datadog_integration_key from the output and configure it in your Datadog UI under Integrations > PagerDuty.

Verification and Testing

Datadog UI: Navigate to your Datadog account.
- Check the Host Map and Infrastructure List to ensure your EKS nodes and pods are reporting.
- Verify logs are streaming in the Log Explorer.
- Browse the "EKS Cluster Overview" dashboard you created.
- Confirm the Prometheus metrics from your annotated applications are visible.
PagerDuty UI:
- Verify the "EKS Observability Alerts" service, escalation policy, and users exist.
- To test the alert, you can either intentionally create a high CPU load on an EKS node or manually trigger a test alert from Datadog that targets the PagerDuty integration.

Best Practices for Production Environments

Secret Management: Never hardcode API keys. Use tools like AWS Secrets Manager or HashiCorp Vault to store sensitive credentials and retrieve them dynamically in Terraform.
Modularity: Break down your Terraform code into logical modules (e.g., EKS core, Datadog agents, Datadog monitors, PagerDuty setup) for better organization and reusability.
GitOps Workflow: Store your Terraform configurations in a Git repository and use a CI/CD pipeline (e.g., Atlantis, Terraform Cloud, GitHub Actions) to automate `terraform plan` and `terraform apply`.
Tagging Strategy: Implement a consistent tagging strategy across all AWS resources, EKS components, and Datadog/PagerDuty configurations. This improves filtering, cost analysis, and correlation.
Fine-tune Alerts: Start with basic alerts and refine them over time to reduce alert fatigue. Leverage Datadog's anomaly detection and forecast monitors.
Role-Based Access Control (RBAC): Ensure least-privilege access for the Datadog Agent and any other observability components within your EKS cluster.

Troubleshooting Common Issues

Datadog Agent not reporting:
- Check logs of Datadog Agent pods: kubectl logs -n datadog -l app=datadog-agent
- Verify API and APP keys are correct.
- Ensure network connectivity from EKS nodes to Datadog endpoints.
Prometheus metrics not appearing in Datadog:
- Double-check pod/service annotations for correctness.
- Confirm datadog.prometheusScrape.enabled is set to true in the Helm release.
- Validate that your application is actually exposing metrics at the specified prometheus_url.
PagerDuty alerts not triggering:
- Ensure the PagerDuty integration key is correctly configured in Datadog.
- Verify the notification syntax (e.g., @pagerduty-devops) in your Datadog monitor message matches your PagerDuty integration name.
- Check PagerDuty's incident log for any errors.

Conclusion

Implementing a unified observability strategy for AWS EKS is no longer optional; it's a critical component for maintaining high-performing, resilient, and secure cloud-native applications. By harnessing the power of Terraform for declarative infrastructure management, integrating Datadog for comprehensive monitoring, leveraging Prometheus for flexible metric collection, and orchestrating incident response with PagerDuty, DevOps teams can gain unparalleled visibility and control over their EKS environments. This approach ensures that issues are detected, diagnosed, and resolved efficiently, ultimately leading to better system reliability and a superior user experience.

CloudStack YAML