Terraform for Centralized AWS EKS Observability and PagerDuty Incident Response with Datadog

Architecture Pro-Tip:

Always enforce a strict GitOps workflow for your EKS observability stack. Treat your Datadog monitors, dashboards, and PagerDuty integrations as code alongside your EKS cluster definition. This ensures version control, peer review, and automated deployment, significantly reducing configuration drift and improving incident response consistency across environments.

Introduction

In today's dynamic cloud-native landscape, managing Kubernetes clusters can be a complex endeavor, especially when operating at scale. AWS EKS provides a robust foundation for running containerized applications, but effective observability and swift incident response are paramount for maintaining high availability and performance. This guide explores how to leverage Terraform for Centralized AWS EKS Observability, integrating Datadog for comprehensive monitoring and PagerDuty for streamlined incident management. By codifying your observability and alerting infrastructure, you can achieve greater consistency, reliability, and faster Mean Time To Resolution (MTTR).

Why Centralized Observability for AWS EKS?

Operating multiple EKS clusters across various AWS accounts or regions introduces significant challenges. Centralized observability addresses these by providing a single pane of glass for all your Kubernetes resources.

Unified View: Consolidate metrics, logs, and traces from all EKS clusters and underlying AWS infrastructure into one platform.
Faster Troubleshooting: Quickly identify root causes by correlating data across different services and clusters.
Proactive Alerting: Implement consistent monitoring thresholds and alerts across your entire EKS fleet.
Reduced Operational Burden: Automate the deployment and management of observability agents and configurations using Infrastructure as Code (IaC) with Terraform.
Improved Compliance: Maintain a clear audit trail of observability configurations and changes.

The Power Trio: Terraform, Datadog, and PagerDuty

This integration creates a robust framework for managing your Kubernetes environments.

Terraform: Infrastructure as Code (IaC) for Everything

Terraform enables you to define and provision your entire infrastructure using a declarative configuration language. For EKS, this means not only managing the cluster itself but also deploying monitoring agents, configuring Datadog monitors, and integrating with PagerDuty. This IaC approach ensures consistency, reproducibility, and version control for your DevOps IaC Automation.

Datadog: Comprehensive Cloud Monitoring

Datadog offers an end-to-end observability platform that collects metrics, logs, and traces from your AWS EKS clusters, applications, and underlying AWS services. With its specialized Kubernetes integration, Datadog provides out-of-the-box dashboards, intelligent alerting, and anomaly detection for containers, pods, nodes, and more. This makes it an ideal solution for Centralized Cloud Monitoring.

PagerDuty: Incident Response Automation

PagerDuty acts as the critical bridge between your monitoring system and your on-call teams. When Datadog detects an issue, PagerDuty automatically triggers incidents, notifies the right people through various channels, and facilitates structured incident response workflows. This integration is crucial for effective Kubernetes Incident Response.

Architecture Overview

The proposed architecture involves deploying Datadog Agents as a DaemonSet within each AWS EKS cluster. These agents collect metrics, logs, and traces and forward them to the centralized Datadog platform. Terraform manages the deployment of these agents, configures Datadog monitors (e.g., for EKS node health, pod restarts, deployment failures), and establishes the integration with PagerDuty. When a monitor's condition is met, Datadog triggers an event in PagerDuty, initiating the incident response process.

AWS EKS: Hosts containerized applications.
Datadog Agent (DaemonSet): Deployed on each EKS node to collect observability data.
Datadog Platform: Ingests, processes, visualizes data, and triggers alerts.
PagerDuty: Receives alerts from Datadog and manages incident workflows.
Terraform: Deploys EKS, Datadog Agent, Datadog monitors, and configures Datadog PagerDuty Integration.

Prerequisites

Before you begin, ensure you have the following in place:

An AWS account with appropriate IAM permissions to create/manage EKS clusters and associated resources.
Terraform CLI (v1.0+) installed.
Kubectl CLI installed and configured to interact with your EKS cluster.
A Datadog account with API and Application Keys.
A PagerDuty account with a Service Integration Key.
An existing AWS EKS cluster, or the ability to provision one using Terraform (recommended).

Terraform Configuration for EKS Observability

This section outlines the key Terraform components required to set up your Terraform EKS Observability stack. We'll use the Datadog and Kubernetes Terraform providers.

1. Datadog Provider Configuration

First, configure the Datadog provider using your API and Application keys. These should ideally be managed via a secrets manager (e.g., AWS Secrets Manager or Vault) and injected as environment variables.

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}
    

2. Deploying the Datadog Agent to EKS

The Datadog Agent is typically deployed as a Kubernetes DaemonSet. While you can use the Helm provider with Terraform, a direct Kubernetes manifest application via the kubernetes_manifest resource or a Helm release for the Datadog Agent is also very common. Here, we illustrate with a conceptual kubernetes_manifest. You would typically use the official Datadog Agent manifest or Helm chart values.

Ensure your EKS IAM role for service accounts (IRSA) has permissions if you plan to use Datadog for IAM roles.

3. Ready-to-Use Terraform Configuration Example

This example demonstrates how to deploy the Datadog Agent, define a basic Datadog monitor for EKS node CPU, and configure a PagerDuty service integration. This is a simplified example; for production, consider dedicated modules and more comprehensive monitoring.

# main.tf

# --- Providers ---
provider "aws" {
  region = "us-east-1"
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.example.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.example.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.example.token
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

# --- Data Sources (assuming EKS cluster already exists) ---
data "aws_eks_cluster" "example" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "example" {
  name = var.eks_cluster_name
}

# --- Datadog Agent Deployment ---
# For a real-world scenario, you would use the official Datadog Helm Chart
# or a full manifest. This is a simplified representation.
resource "kubernetes_manifest" "datadog_agent_daemonset" {
  depends_on = [kubernetes_namespace.datadog] # Ensure namespace exists

  manifest = {
    apiVersion = "apps/v1"
    kind       = "DaemonSet"
    metadata = {
      name      = "datadog-agent"
      namespace = "datadog"
      labels = {
        app = "datadog-agent"
      }
    }
    spec = {
      selector = {
        matchLabels = {
          app = "datadog-agent"
        }
      }
      template = {
        metadata = {
          labels = {
            app = "datadog-agent"
          }
        }
        spec = {
          serviceAccountName = "datadog-agent" # Ensure this SA is created and has permissions
          containers = [
            {
              name  = "agent"
              image = "datadog/agent:latest"
              env = [
                { name = "DD_API_KEY", value = var.datadog_api_key },
                { name = "DD_KUBERNETES_HOST_SUFFIX", value = "yes" },
                { name = "DD_COLLECT_KUBERNETES_EVENTS", value = "true" },
                { name = "DD_LOGS_ENABLED", value = "true" },
                { name = "DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL", value = "true" },
              ]
              resources = {
                requests = {
                  memory = "256Mi"
                  cpu    = "200m"
                }
                limits = {
                  memory = "512Mi"
                  cpu    = "500m"
                }
              }
              volumeMounts = [
                { name = "dockersocket", mountPath = "/var/run/docker.sock" },
                { name = "procdir", mountPath = "/host/proc", readOnly = true },
                { name = "cgroups", mountPath = "/host/sys/fs/cgroup", readOnly = true },
              ]
            },
          ]
          volumes = [
            { name = "dockersocket", hostPath = { path = "/var/run/docker.sock" } },
            { name = "procdir", hostPath = { path = "/proc" } },
            { name = "cgroups", hostPath = { path = "/sys/fs/cgroup" } },
          ]
        }
      }
    }
  }
}

resource "kubernetes_namespace" "datadog" {
  metadata {
    name = "datadog"
  }
}

resource "kubernetes_service_account_v1" "datadog_agent" {
  metadata {
    name      = "datadog-agent"
    namespace = "datadog"
  }
}

# --- Datadog Monitor for EKS Node CPU ---
resource "datadog_monitor" "eks_node_cpu_alert" {
  name                = "[EKS] Node CPU Utilization High on {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:system.cpu.idle{kube_cluster_name:{{var.eks_cluster_name}}} by {host} < 10" # Adjust threshold as needed (e.g., < 10 means > 90% utilization)
  message             = "EKS Node {{host.name}} CPU utilization is high ({{value}}%). Please investigate."
  tags                = ["environment:${var.environment}", "service:eks", "alert-type:performance"]
  notify_no_data      = false
  no_data_timeframe   = 20
  renotify_interval   = 60
  escalation_message  = "CPU usage remains high after 1 hour, escalating to management."
  require_full_window = true
  timeout_h           = 0
  priority            = 2 # Higher priority for critical alerts

  thresholds = {
    critical = 10
    warning  = 20
  }

  # Integration with PagerDuty (see next step for service key setup)
  notify_audit = false
  include_tags = true
  restricted_roles = [] # Or specify role IDs for restricted access
}

# --- PagerDuty Service Integration (via Datadog) ---
# Datadog handles the integration with PagerDuty. You create a PagerDuty service
# and get an integration key (e.g., "Datadog Events API" integration).
# You then specify this integration in Datadog.

# First, ensure the PagerDuty integration is configured in Datadog UI
# (Integrations -> PagerDuty).
# Then, reference the PagerDuty service by its name in Datadog monitor's message,
# or set up an integration through the datadog_integration_pagerduty resource.
# For simplicity, we assume the integration is already setup in Datadog UI
# and we reference it by service name in the monitor message.
# To configure the Datadog PagerDuty integration itself via Terraform:
resource "datadog_integration_pagerduty" "main_pagerduty_integration" {
  services {
    service_name = var.pagerduty_service_name
    service_key  = var.pagerduty_service_key
  }
}

# Then, modify the monitor message to include PagerDuty notification:
# message             = "@pagerduty-${var.pagerduty_service_name} EKS Node {{host.name}} CPU utilization is high ({{value}}%). Please investigate."


# --- Variables ---
variable "eks_cluster_name" {
  description = "The name of the EKS cluster."
  type        = string
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_service_name" {
  description = "The name of the PagerDuty service configured in Datadog."
  type        = string
}

variable "pagerduty_service_key" {
  description = "The PagerDuty integration key for the service."
  type        = string
  sensitive   = true
}

variable "environment" {
  description = "The deployment environment (e.g., 'dev', 'prod')."
  type        = string
}
    

Deployment Steps

Follow these steps to deploy your centralized observability stack:

Prepare your EKS Cluster: Ensure your EKS cluster is running and your `kubectl` is configured. If using IRSA for Datadog, ensure the necessary IAM roles and service accounts are created and configured for the Datadog agent.
Create Terraform Files: Save the code above into `main.tf`, `variables.tf`, etc., in a dedicated Terraform project directory.
Set Variables: Provide values for `eks_cluster_name`, `datadog_api_key`, `datadog_app_key`, `pagerduty_service_name`, and `pagerduty_service_key`. Use environment variables or a `terraform.tfvars` file (with caution for sensitive data).
Initialize Terraform: Run `terraform init` in your project directory.
Review Plan: Execute `terraform plan` to see the resources that will be created or modified.
Apply Changes: Apply the configuration with `terraform apply`. Confirm with `yes`.

Testing and Validation

After applying your Terraform configuration, validate that the observability and incident response pipeline is functional.

Datadog Agent Status: Verify that the Datadog Agents are running correctly in your EKS cluster by running `kubectl get pods -n datadog`. Check Datadog's Infrastructure page for agent health.
Metrics and Logs: Navigate to Datadog dashboards or Metric Explorer to confirm that EKS metrics (e.g., `kubernetes.cpu.usage`, `system.cpu.idle`) and logs are being ingested.
Monitor Status: Check the Datadog Monitors page to ensure your `eks_node_cpu_alert` monitor is active and correctly evaluating.
PagerDuty Integration: Artificially trigger an alert (e.g., by stressing an EKS node's CPU, or temporarily adjusting the monitor threshold) and verify that an incident is created in PagerDuty and the on-call team is notified.

Advanced Considerations

IAM Roles for Service Accounts (IRSA): For enhanced security, configure your Datadog Agent's Kubernetes Service Account to assume an IAM role with the necessary permissions instead of embedding AWS credentials.
Custom Metrics and APM: Extend your Datadog setup to collect custom application metrics and enable Application Performance Monitoring (APM) for deeper insights.
Log Management: Configure Datadog Log Collection for all your EKS workloads, ensuring proper parsing and indexing of logs.
Cost Optimization: Monitor Datadog usage closely and adjust sampling rates or data retention policies as needed to manage costs.
Terraform Modules: For large-scale deployments, consider creating reusable Terraform modules for Datadog Agent deployment, EKS monitors, and PagerDuty integrations to enforce standards and reduce duplication.

Troubleshooting and FAQ

Q: Datadog Agent pods are not running.

A: Check `kubectl describe pod -n datadog` for events and error messages. Common issues include incorrect API/APP keys, insufficient resource limits, or missing Kubernetes permissions (RBAC). Ensure the `serviceAccountName` exists and has the necessary ClusterRole/ClusterRoleBinding.

Q: Metrics are not appearing in Datadog.

A: Verify network connectivity from your EKS nodes to Datadog's ingestion endpoints. Check Datadog Agent logs (`kubectl logs -n datadog`) for any submission errors or configuration issues. Ensure the correct tags (e.g., `kube_cluster_name`) are being sent.

Q: PagerDuty incidents are not being triggered.

A: Double-check the PagerDuty integration setup in Datadog. Ensure the monitor's message explicitly includes the PagerDuty service name (e.g., `@pagerduty-your-service-name`). Verify the `datadog_integration_pagerduty` resource is correctly applied and the `service_key` is valid.

Conclusion

Implementing Terraform for Centralized AWS EKS Observability with Datadog and PagerDuty is a critical step towards building resilient, scalable, and manageable cloud-native infrastructure. By codifying your entire observability and incident response stack, you gain unprecedented control, consistency, and automation. This approach not only streamlines operations but also empowers your teams to proactively address issues, reduce downtime, and focus on innovation rather than firefighting. Embrace Infrastructure as Code to master your EKS environments and elevate your operational excellence.

CloudStack YAML