Architecture Pro-Tip: When designing your EKS observability stack, prioritize a unified data platform. While Prometheus excels at Kubernetes-native metrics, integrating it with a comprehensive platform like Datadog provides a single pane of glass for metrics, logs, traces, and incident management (via PagerDuty), significantly reducing context switching and accelerating Mean Time To Resolution (MTTR).

Terraform-Managed AWS EKS Observability: Datadog, Prometheus, and PagerDuty

In the dynamic landscape of cloud-native applications, managing and monitoring an AWS Elastic Kubernetes Service (EKS) cluster effectively is paramount. Modern DevOps teams demand deep visibility into their distributed systems to ensure reliability, performance, and security. This comprehensive guide details how to establish a robust observability framework for your EKS clusters using a powerful trifecta: Datadog for unified monitoring, Prometheus for rich Kubernetes-native metrics, and PagerDuty for incident management, all orchestrated and managed seamlessly with Terraform.

Why Terraform for EKS Observability?

Infrastructure as Code (IaC) is the cornerstone of modern cloud operations. Terraform, as the industry-leading IaC tool, allows you to define, provision, and manage your EKS clusters and their associated observability tooling in a declarative and reproducible manner. Key benefits include:

Reproducibility: Deploy identical observability stacks across development, staging, and production environments.
Version Control: Track changes to your monitoring infrastructure, facilitating rollbacks and audits.
Automation: Eliminate manual configuration errors and accelerate deployment times.
Scalability: Easily scale your observability components as your EKS cluster grows.

The Observability Stack: Datadog, Prometheus, and PagerDuty

Our chosen stack combines specialized tools to provide comprehensive visibility and incident response capabilities:

Datadog: Unified Monitoring and Analytics

Datadog offers a full-stack monitoring solution, collecting metrics, logs, and traces from your entire infrastructure and applications. For EKS, it provides:

Real-time performance metrics for nodes, pods, and containers.
Centralized log management and analytics.
Distributed tracing for microservices.
Unified dashboards and robust alerting capabilities.
Seamless integration with Prometheus metrics and PagerDuty for incident response.

Prometheus: Kubernetes-Native Metrics Collection

Prometheus is an open-source monitoring system with a powerful data model and query language (PromQL). Its strength lies in its native integration with Kubernetes, allowing it to:

Scrape metrics directly from Kubernetes components and applications.
Provide granular, high-cardinality metrics.
Power custom dashboards (e.g., with Grafana, or directly imported into Datadog).
Serve as a primary source for specific operational metrics that can then be forwarded or integrated with Datadog.

PagerDuty: Incident Management and On-Call Automation

PagerDuty bridges the gap between monitoring and response. It transforms monitoring signals into actionable incidents, ensuring the right teams are notified at the right time. Key features include:

On-call scheduling and escalation policies.
Automatic incident routing based on service and severity.
Comprehensive integrations with monitoring tools like Datadog.
Post-incident analysis and reporting.

Prerequisites

Before you begin, ensure you have the following:

An AWS account with appropriate permissions to create EKS clusters and related resources.
Terraform (v1.0+) installed and configured.
kubectl configured to interact with your EKS cluster.
A Datadog account with API and Application keys.
A PagerDuty account with an API key and a service set up.
AWS CLI installed and configured.
Helm CLI installed (optional, but useful for debugging).

Terraform Configuration for EKS Observability

We'll use Terraform to deploy the Datadog Agent and the Prometheus stack (via kube-prometheus-stack Helm chart) onto an existing or newly created EKS cluster. We'll also cover how to integrate PagerDuty.

1. EKS Cluster Setup (Assumed or Minimal)

For brevity, we assume you have an EKS cluster configured. If not, you can create one using Terraform's EKS module (terraform-aws-modules/eks/aws).

2. Datadog Agent Deployment

The Datadog Agent is deployed as a DaemonSet to collect metrics, logs, and traces from your EKS nodes and pods. We use the helm_release resource to manage its deployment.

3. Prometheus (kube-prometheus-stack) Deployment

The kube-prometheus-stack is a collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with Prometheus Operator, Alertmanager, and Node Exporter. It's the standard way to deploy Prometheus on Kubernetes.

4. Datadog-Prometheus Integration

Datadog can ingest metrics directly from Prometheus exporters. The Datadog Agent can be configured to scrape Prometheus endpoints, allowing you to centralize all metrics in Datadog while leveraging Prometheus's rich scraping capabilities.

5. PagerDuty Integration with Datadog

Integrate PagerDuty directly with Datadog. This typically involves configuring an integration in Datadog that points to your PagerDuty service. Datadog monitors can then trigger incidents in PagerDuty based on thresholds or anomalies.

Ready-to-Use Configuration Example

Below is a simplified Terraform configuration demonstrating how to deploy Datadog Agent and the kube-prometheus-stack to an existing EKS cluster. Remember to replace placeholder values with your actual cluster details and API keys.

// main.tf for EKS Observability
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
  }
}
 
provider "aws" {
  region = var.aws_region
}
 
data "aws_eks_cluster" "cluster" {
  name = var.cluster_name
}
 
data "aws_eks_cluster_auth" "cluster" {
  name = var.cluster_name
}
 
provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  token                  = data.aws_eks_cluster_auth.cluster.token
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
}
 
provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    token                  = data.aws_eks_cluster_auth.cluster.token
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
  }
}
 
// Datadog Agent Deployment
resource "kubernetes_namespace" "datadog" {
  metadata {
    name = "datadog"
  }
}
 
resource "helm_release" "datadog_agent" {
  name      = "datadog"
  repository = "https://helm.datadoghq.com"
  chart     = "datadog"
  namespace = kubernetes_namespace.datadog.metadata[0].name
  version   = "2.33.0" # Use a stable version
 
  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }
  set {
    name  = "datadog.site"
    value = var.datadog_site
  }
  set {
    name  = "kubeStateMetricsV2.enabled"
    value = "true"
  }
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "processAgent.enabled"
    value = "true"
  }
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }
  set {
    name  = "apm.enabled"
    value = "true"
  }
  // Enable Prometheus integration if you want Datadog to scrape Prometheus endpoints
  set {
    name  = "prometheusScrape.enabled"
    value = "true"
  }
  set {
    name  = "datadog.confd.prometheus.yaml"
    value = <
    init_config:
    instances:
      - prometheus_url: http://kube-prometheus-stack-prometheus.prometheus.svc.cluster.local:9090/metrics
        namespace: "prometheus"
        metrics:
          - prometheus.up
          - prometheus.tsdb.head_chunks_created_total
    EOT
  }
}
 
// Prometheus Stack Deployment
resource "kubernetes_namespace" "prometheus" {
  metadata {
    name = "prometheus"
  }
}
 
resource "helm_release" "kube_prometheus_stack" {
  name      = "kube-prometheus-stack"
  repository = "https://prometheus-community.github.io/helm-charts"
  chart     = "kube-prometheus-stack"
  namespace = kubernetes_namespace.prometheus.metadata[0].name
  version   = "48.3.0" # Use a stable version
 
  set {
    name  = "prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues"
    value = "false"
  }
  set {
    name  = "prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues"
    value = "false"
  }
  set {
    name  = "grafana.enabled"
    value = "false" # Datadog will be our primary dashboard
  }
  // Configure Alertmanager to forward alerts to PagerDuty
  set {
    name  = "alertmanager.alertmanagerSpec.route.receiver"
    value = "pagerduty"
  }
  set {
    name  = "alertmanager.alertmanagerSpec.receivers[0].name"
    value = "pagerduty"
  }
  set {
    name  = "alertmanager.alertmanagerSpec.receivers[0].pagerduty_configs[0].service_key"
    value = var.pagerduty_service_key
    sensitive = true
  }
  set {
    name  = "alertmanager.alertmanagerSpec.receivers[0].pagerduty_configs[0].routing_key"
    value = var.pagerduty_routing_key // PagerDuty Events API v2 routing key
    sensitive = true
  }
  set {
    name  = "alertmanager.alertmanagerSpec.receivers[0].pagerduty_configs[0].client"
    value = "Prometheus Alertmanager"
  }
  set {
    name  = "alertmanager.alertmanagerSpec.receivers[0].pagerduty_configs[0].client_url"
    value = "http://<your-alertmanager-url>" // Replace with actual Alertmanager URL
  }
  // Alternatively, configure PagerDuty integration in Datadog directly.
  // This example shows both options for demonstration.
}
 
// variables.tf
variable "aws_region" {
  description = "AWS region for the EKS cluster"
  type        = string
  default     = "us-east-1"
}
 
variable "cluster_name" {
  description = "Name of the existing EKS cluster"
  type        = string
}
 
variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}
 
variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}
 
variable "datadog_site" {
  description = "Datadog site (e.g., datadoghq.com, eu.datadoghq.com)"
  type        = string
  default     = "datadoghq.com"
}
 
variable "pagerduty_service_key" {
  description = "PagerDuty service integration key (legacy v1 API)"
  type        = string
  sensitive   = true
  default     = "" // Consider using v2 routing_key if possible
}
 
variable "pagerduty_routing_key" {
  description = "PagerDuty Events API v2 routing key"
  type        = string
  sensitive   = true
  default     = ""
}

How to Apply:

Save the code above into main.tf and variables.tf files.
Create a terraform.tfvars file with your sensitive variables:

cluster_name = "your-eks-cluster-name"

datadog_api_key = "your_datadog_api_key"

datadog_app_key = "your_datadog_app_key"

pagerduty_service_key = "your_pagerduty_service_integration_key"

pagerduty_routing_key = "your_pagerduty_events_api_v2_routing_key"
Run terraform init to initialize the providers.
Run terraform plan to review the changes.
Run terraform apply to deploy the observability components.

Post-Deployment & Best Practices

Verify Datadog Integration

After deployment, log into your Datadog account. You should see metrics, logs, and traces from your EKS cluster flowing in. Explore the out-of-the-box Kubernetes dashboards and verify host map, container map, and live processes. Check the "Integrations" section to confirm Prometheus and PagerDuty are active.

Datadog Monitors for PagerDuty

Create Datadog monitors that trigger PagerDuty incidents. For example:

EKS Node CPU Utilization above 90% for 5 minutes.
Pod Restarts exceeding a threshold.
Application Latency Spikes (via APM traces).

In the monitor notification section, select your PagerDuty integration to route critical alerts.

Prometheus Alerts and Alertmanager

Leverage Prometheus Alertmanager for granular, domain-specific alerts. The example configuration above routes Alertmanager alerts to PagerDuty. Define custom PrometheusRule resources in Kubernetes using Terraform or kubectl to create alerts based on PromQL queries.

Example PrometheusRule (apply via kubectl or Terraform kubernetes_manifest):

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: kubernetes-apps
  namespace: prometheus
spec:
  groups:
  - name: kubernetes-apps
    rules:
    - alert: KubePodCrashLooping
      expr: sum(increase(kube_pod_container_status_restarts_total{job="kube-state-metrics",container=~".+"}[1m])) by (pod, namespace) > 2
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash looping
        description: Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) has restarted {{ $value }} times in the last 5 minutes.

Unified Alerting Strategy

Decide whether Datadog or Prometheus Alertmanager will be the primary source for PagerDuty incidents. For simplicity and comprehensive visibility, leveraging Datadog as the central alerting hub (feeding into PagerDuty) is often preferred, as it consolidates all types of signals (metrics, logs, traces) for alerting.

Troubleshooting and FAQ

Datadog Agent Pods Not Running

Check logs of the Datadog Agent pods (kubectl logs -f -n datadog <datadog-agent-pod>). Ensure your datadog_api_key and datadog_app_key are correct and have appropriate permissions. Verify network connectivity from EKS nodes to Datadog endpoints.

Prometheus Not Scarping Metrics

Access the Prometheus UI (usually via port-forward: kubectl port-forward -n prometheus svc/kube-prometheus-stack-prometheus 9090:9090) and check the "Status > Targets" page. Ensure your ServiceMonitors and PodMonitors are correctly configured and match your application labels.

PagerDuty Incidents Not Triggering

For Datadog-triggered incidents, verify the monitor is configured to send notifications to your PagerDuty integration. Check Datadog's event stream for monitor alerts. For Alertmanager, check Alertmanager logs and its UI (Status > Alerts). Ensure the pagerduty_service_key or pagerduty_routing_key is correct and the PagerDuty service is configured to receive events.

Conclusion

Building a robust observability pipeline for AWS EKS is essential for maintaining application health and performance. By leveraging Terraform to manage the deployment of Datadog, Prometheus, and PagerDuty, you create a scalable, repeatable, and resilient system that provides deep insights into your Kubernetes workloads and automates incident response. This integrated approach empowers your DevOps teams to proactively identify issues, troubleshoot effectively, and minimize downtime, ensuring your cloud-native applications run smoothly.

CloudStack YAML