Terraform for AWS EKS Observability: Integrating Prometheus and Datadog with PagerDuty Alerting

Architecture Pro-Tip: When designing your EKS observability stack, prioritize a layered approach. Separate metrics collection (Prometheus) from log/trace aggregation (Datadog) and incident response (PagerDuty). This modularity, managed by Terraform, ensures resilience, reduces vendor lock-in risk for specific components, and allows for specialized tooling to excel at its core function. Always define clear data retention policies and alert thresholds from the outset to manage costs and reduce alert fatigue.

In today's dynamic cloud-native landscape, ensuring the robust health and performance of your Amazon EKS clusters is paramount. Observability isn't just about collecting data; it's about understanding the internal state of a system from its external outputs, enabling proactive problem-solving and rapid incident response. This comprehensive guide will walk you through leveraging Terraform to establish a powerful observability stack for AWS EKS, integrating Prometheus for metrics, Datadog for comprehensive monitoring and logging, and PagerDuty for efficient incident management and alerting.

Why Terraform for EKS Observability?

Infrastructure as Code (IaC) is the cornerstone of modern DevOps practices. Terraform brings consistency, repeatability, and version control to your observability infrastructure. By defining your monitoring agents, dashboards, alerts, and integrations in code, you:

Ensure Consistency: Deploy identical observability setups across development, staging, and production environments.
Automate Deployments: Reduce manual errors and accelerate the provisioning of monitoring tools.
Enable Version Control: Track changes, revert to previous configurations, and collaborate effectively.
Improve Auditability: Maintain a clear history of your observability infrastructure changes.
Scale Efficiently: Easily replicate and scale your monitoring solutions as your EKS footprint grows.

The Observability Stack: Components and Roles

1. Prometheus: The Metrics King

Prometheus is an open-source monitoring system and time-series database. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts. For EKS, Prometheus typically scrapes metrics from:

Kubernetes API Server: Cluster health.
kube-state-metrics: Kubernetes object states (e.g., deployment, pod, node status).
Node Exporter: Host-level metrics (CPU, memory, disk I/O) from EKS worker nodes.
Application-specific Exporters: Custom application metrics.

Often deployed with Grafana for visualization and Alertmanager for flexible alerting.

2. Datadog: Unified Monitoring and Analytics

Datadog is a SaaS-based monitoring and analytics platform that brings together metrics, logs, and traces from your entire stack. It offers:

Comprehensive Dashboards: Visualize data from various sources in a unified view.
Log Management: Collect, process, and analyze logs from EKS, applications, and AWS services.
APM & Distributed Tracing: Gain deep visibility into application performance.
Synthetics & RUM: Monitor user experience and availability from an external perspective.
Powerful Alerting: Create sophisticated alerts based on any metric, log, or trace.

Datadog complements Prometheus by providing a centralized platform for all observability data, advanced analytics, and cross-platform correlation.

3. PagerDuty: Incident Management and On-Call Automation

PagerDuty is an incident management platform that provides reliable alerts, on-call scheduling, and automated escalation policies. It acts as the final destination for critical alerts from Prometheus Alertmanager and Datadog, ensuring:

Timely Notifications: Via SMS, phone call, email, or push notifications.
On-Call Management: Flexible schedules and rotations.
Escalation Policies: Automatically escalate incidents if not acknowledged.
Incident Orchestration: Streamline response workflows.

Prerequisites

Before diving into Terraform configurations, ensure you have the following:

An active AWS Account with administrative access.
An existing AWS EKS Cluster. If not, consider using Terraform to provision your EKS cluster first.
Terraform CLI installed (v1.0+ recommended).
Kubectl installed and configured to connect to your EKS cluster.
Helm CLI installed (v3+ recommended).
A Datadog Account with an API Key and Application Key.
A PagerDuty Account with an API Key and a service integration key.

Deploying Prometheus on EKS with Terraform

The easiest way to deploy Prometheus and its ecosystem (Grafana, Alertmanager, kube-state-metrics, node-exporter) on EKS is by using the kube-prometheus-stack Helm chart. Terraform can manage this Helm chart deployment.

Terraform Configuration for Prometheus (via Helm)

First, set up your AWS and Kubernetes providers. Ensure your Kubernetes provider is configured to use your EKS cluster's kubeconfig.

resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = "monitoring"
  }
}

resource "helm_release" "prometheus_stack" {
  name       = "prometheus-stack"
  namespace  = kubernetes_namespace.monitoring.metadata[0].name
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  version    = "55.0.0" # Use the latest stable version
  timeout    = 600

  values = [file("./prometheus-values.yaml")]
}

And an example prometheus-values.yaml (truncated for brevity, focus on Alertmanager and PagerDuty):

alertmanager:
  enabled: true
  config:
    global:
      resolve_timeout: 5m
    route:
      group_by: ['alertname', 'cluster', 'service']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 1h
      receiver: 'pagerduty'
    receivers:
      - name: 'pagerduty'
        pagerduty_configs:
          - service_key: "YOUR_PAGERDUTY_PROMETHEUS_INTEGRATION_KEY"
            send_resolved: true
    inhibit_rules:
      - source_match:
          severity: "critical"
        target_match:
          severity: "warning"
        equal: ['alertname', 'cluster']

Replace YOUR_PAGERDUTY_PROMETHEUS_INTEGRATION_KEY with the actual key from your PagerDuty service for Prometheus.

Integrating Datadog with EKS using Terraform

The Datadog Agent is typically deployed as a DaemonSet across your EKS nodes to collect metrics, logs, and traces. Terraform will manage the deployment of the Datadog Agent Helm chart.

Terraform Configuration for Datadog Agent

resource "kubernetes_namespace" "datadog" {
  metadata {
    name = "datadog"
  }
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  namespace  = kubernetes_namespace.datadog.metadata[0].name
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  version    = "2.42.0" # Use the latest stable version
  timeout    = 600

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "datadog.site"
    value = "datadoghq.com" # or eu.datadoghq.com etc.
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "kubeStateMetricsCore.enabled"
    value = "true"
  }

  set {
    name  = "processAgent.enabled"
    value = "true"
  }

  set {
    name  = "logs.enabled"
    value = "true"
  }

  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }

  set {
    name  = "apm.enabled"
    value = "true"
  }

  # For EKS Fargate profiles, additional configurations might be needed
  # E.g., datadog.fargate.enabled = true
}

# Define these in your variables.tf
variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

Ensure you provide your Datadog API and Application keys securely, for example, via environment variables or a secrets manager when running Terraform.

Setting up PagerDuty for Alerting with Terraform

PagerDuty will receive alerts from both Prometheus (via Alertmanager) and Datadog. Terraform allows you to define PagerDuty services, escalation policies, and service integrations.

Terraform Configuration for PagerDuty

First, configure the PagerDuty provider:

provider "pagerduty" {
  token = var.pagerduty_api_token
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

Next, define an escalation policy and a service:

resource "pagerduty_escalation_policy" "eks_critical_policy" {
  name      = "EKS Critical Alerts"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 15
    target {
      type = "user"
      id   = "YOUR_PAGERDUTY_USER_ID" # Replace with an actual PagerDuty User ID
    }
  }

  rule {
    escalation_delay_in_minutes = 30
    target {
      type = "team"
      id   = "YOUR_PAGERDUTY_TEAM_ID" # Replace with an actual PagerDuty Team ID
    }
  }
}

resource "pagerduty_service" "eks_observability_service" {
  name                    = "EKS Observability Service"
  auto_resolve_timeout    = "14400" # 4 hours
  acknowledgement_timeout = "600"   # 10 minutes
  escalation_policy       = pagerduty_escalation_policy.eks_critical_policy.id
}

# Integration for Prometheus (Generic Events API v2)
resource "pagerduty_service_integration" "prometheus_integration" {
  name        = "Prometheus Alertmanager"
  service     = pagerduty_service.eks_observability_service.id
  type        = "generic_events_api_v2"
}

# Integration for Datadog (Datadog Events API)
resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog"
  service     = pagerduty_service.eks_observability_service.id
  type        = "datadog_events_api"
}

output "prometheus_pagerduty_integration_key" {
  description = "PagerDuty integration key for Prometheus Alertmanager."
  value       = pagerduty_service_integration.prometheus_integration.integration_key
  sensitive   = true
}

You'll need to obtain your PagerDuty User and Team IDs (from the PagerDuty UI or API). The output block will provide the integration key for Prometheus, which you'll use in the prometheus-values.yaml.

Datadog Monitor with PagerDuty Integration (via Terraform)

With the Datadog Agent deployed and PagerDuty configured, you can now define Datadog monitors that send alerts to PagerDuty.

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

resource "datadog_monitor" "eks_high_cpu" {
  name          = "EKS Cluster Node High CPU Usage"
  type          = "metric alert"
  message       = "@pagerduty-{{ pagerduty_service_name.eks_observability_service }} EKS node CPU is at {{ value }}% for {{ time_ago }}. Check for runaway pods."
  query         = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster_name:your-eks-cluster-name} by {host} < 20"
  
  thresholds {
    critical = 20
    warning  = 30
  }

  tags = ["eks", "cpu", "critical"]
  renotify_interval = 60 # minutes
  notify_no_data    = false
  new_group_delay   = 300 # seconds
  include_tags      = true
  require_full_window = false
}

In the message field, @pagerduty-{{ pagerduty_service_name.eks_observability_service }} refers to the PagerDuty integration name you set up in Datadog (which will automatically be created when the Datadog PagerDuty integration is established via Terraform or UI). Ensure your-eks-cluster-name matches your actual EKS cluster name tag in Datadog.

Ready-to-Use Terraform Configuration Example (Condensed)

This example brings together the core components. Remember to replace placeholders like YOUR_EKS_CLUSTER_NAME, YOUR_PAGERDUTY_USER_ID, and sensitive API/App keys with your actual values. Define your provider configurations (AWS, Kubernetes) and secrets in appropriate variables.tf and terraform.tfvars or environment variables.

# main.tf

# --- AWS & Kubernetes Providers (example, adjust to your setup) ---
# provider "aws" {
#   region = "us-east-1"
# }
#
# data "aws_eks_cluster" "eks_cluster" {
#   name = "YOUR_EKS_CLUSTER_NAME"
# }
#
# data "aws_eks_cluster_auth" "eks_auth" {
#   name = "YOUR_EKS_CLUSTER_NAME"
# }
#
# provider "kubernetes" {
#   host                   = data.aws_eks_cluster.eks_cluster.endpoint
#   cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)
#   token                  = data.aws_eks_cluster_auth.eks_auth.token
# }
# --- End of Provider Example ---

# --- PagerDuty Provider & Resources ---
provider "pagerduty" {
  token = var.pagerduty_api_token
}

resource "pagerduty_escalation_policy" "eks_policy" {
  name      = "EKS Cluster Escalation"
  num_loops = 1
  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "user"
      id   = "YOUR_PAGERDUTY_USER_ID" # e.g., "P012345"
    }
  }
}

resource "pagerduty_service" "eks_critical_service" {
  name                    = "EKS Critical Alerts"
  escalation_policy       = pagerduty_escalation_policy.eks_policy.id
  auto_resolve_timeout    = "120"
  acknowledgement_timeout = "30"
}

resource "pagerduty_service_integration" "prometheus_pd_integration" {
  name    = "Prometheus Alertmanager"
  service = pagerduty_service.eks_critical_service.id
  type    = "generic_events_api_v2"
}

resource "pagerduty_service_integration" "datadog_pd_integration" {
  name    = "Datadog Integration"
  service = pagerduty_service.eks_critical_service.id
  type    = "datadog_events_api"
}

# --- Datadog Provider & Resources ---
provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

resource "datadog_monitor" "eks_node_cpu_critical" {
  name          = "EKS Node Critical CPU Usage"
  type          = "metric alert"
  query         = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster_name:YOUR_EKS_CLUSTER_NAME} by {host} < 10"
  message       = "@pagerduty-${pagerduty_service.eks_critical_service.name} CRITICAL: EKS node CPU high on {{host.name}} ({{value}}%)."
  thresholds {
    critical = 10
  }
  tags = ["eks", "cpu", "critical"]
}

# --- Kubernetes Namespace for Monitoring ---
resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = "monitoring"
  }
}

# --- Helm Release for Datadog Agent ---
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  namespace  = kubernetes_namespace.monitoring.metadata[0].name
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  version    = "2.42.0" # Always verify latest stable
  
  set { name = "datadog.apiKey" ; value = var.datadog_api_key ; sensitive = true }
  set { name = "datadog.appKey" ; value = var.datadog_app_key ; sensitive = true }
  set { name = "datadog.site"   ; value = "datadoghq.com" }
  set { name = "kubeStateMetricsCore.enabled" ; value = "true" }
  set { name = "logs.enabled"             ; value = "true" }
  set { name = "logs.containerCollectAll" ; value = "true" }
  set { name = "apm.enabled"              ; value = "true" }
}

# --- Helm Release for Kube-Prometheus-Stack ---
resource "helm_release" "prometheus_stack" {
  name       = "prometheus-stack"
  namespace  = kubernetes_namespace.monitoring.metadata[0].name
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  version    = "55.0.0" # Always verify latest stable
  timeout    = 600

  # Values for Alertmanager to integrate with PagerDuty
  # Note: PagerDuty integration key is dynamic from the pagerduty_service_integration resource
  # For this to work seamlessly, ensure your `prometheus-values.yaml` includes:
  # alertmanager:
  #   config:
  #     receivers:
  #       - name: 'pagerduty'
  #         pagerduty_configs:
  #           - service_key: "${prometheus_pd_integration.integration_key}" # Inject this from Terraform output
  #             send_resolved: true
  #     route:
  #       receiver: 'pagerduty'
  #       # ... other routing rules
  values = [
    yamlencode({
      alertmanager = {
        enabled = true
        config = {
          global = {
            resolve_timeout = "5m"
          }
          route = {
            group_by        = ["alertname", "cluster", "service"]
            group_wait      = "30s"
            group_interval  = "5m"
            repeat_interval = "1h"
            receiver        = "pagerduty-receiver"
          }
          receivers = [
            {
              name = "pagerduty-receiver"
              pagerduty_configs = [
                {
                  service_key = pagerduty_service_integration.prometheus_pd_integration.integration_key
                  send_resolved = true
                }
              ]
            }
          ]
        }
      }
    })
  ]
}

# variables.tf
variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

Note: For the kube-prometheus-stack Helm chart, we are dynamically injecting the PagerDuty integration key into the Alertmanager configuration using yamlencode directly within the Terraform values block. This ensures the Prometheus Alertmanager seamlessly integrates with the PagerDuty service created by Terraform.

Deployment Workflow

Once your Terraform files are prepared:

Initialize Terraform: terraform init
Review Plan: terraform plan (carefully examine the proposed changes).
Apply Changes: terraform apply (confirm with yes).

This will provision the PagerDuty resources, deploy the Datadog Agent, and the Prometheus stack on your EKS cluster, all configured to send alerts to PagerDuty.

Verifying Observability

Prometheus & Grafana: Access Grafana (usually exposed via an ingress or port-forwarding) to view EKS cluster metrics and dashboards. Verify Alertmanager UI to see active alerts.
Datadog: Log into your Datadog account. You should see your EKS cluster, nodes, pods, and container metrics/logs appearing in the Infrastructure List, Log Explorer, and APM sections. Check your configured dashboards.
PagerDuty: Ensure your services and escalation policies are active. You can manually trigger a test alert in Datadog or Prometheus (if configured) to verify PagerDuty notifications.

Troubleshooting Common Issues

Helm Release Stuck/Failed: Check kubectl get events -n <namespace> and kubectl logs -n <namespace> <pod-name> for relevant pods (e.g., Prometheus operator, Datadog agent).
Missing Datadog Metrics/Logs: Verify datadog.apiKey and datadog.appKey are correct. Check Datadog Agent pod logs for errors (kubectl logs -n datadog -l app=datadog). Ensure necessary IAM permissions for EKS nodes if using IRSA (IAM Roles for Service Accounts) for Datadog.
Prometheus Not Scraping: Check Prometheus target status in the Prometheus UI. Ensure correct service monitors or pod annotations are applied.
Alerts Not Firing to PagerDuty:
- From Prometheus: Check Alertmanager logs. Verify the service_key in Alertmanager configuration matches the PagerDuty integration key. Ensure alert rules are correctly defined and firing.
- From Datadog: Check Datadog monitor history. Ensure the @pagerduty-<service_name> notification syntax is correct and the PagerDuty integration is active in Datadog.
Terraform Apply Issues: Read the error messages carefully. Often, it's a syntax error, a missing variable, or permission issue with the AWS/Kubernetes/PagerDuty/Datadog provider.

Conclusion

Establishing robust observability is non-negotiable for operating critical applications on AWS EKS. By using Terraform, you gain the power of Infrastructure as Code to deploy and manage a sophisticated monitoring stack featuring Prometheus for deep metrics, Datadog for unified visibility, and PagerDuty for streamlined incident response. This integrated approach ensures your EKS clusters are not only performant but also resilient, allowing your team to proactively identify and resolve issues before they impact your users.

Continuously refine your observability strategy by adding more custom metrics, specific log parsing rules, and dynamic alerts as your EKS environment evolves. The power of IaC makes this iterative improvement process efficient and reliable.

CloudStack YAML