Terraform-Managed AWS EKS Observability with Datadog and PagerDuty

In today's dynamic cloud-native landscape, managing Amazon Elastic Kubernetes Service (EKS) clusters requires robust observability and an efficient incident response strategy. This guide provides a comprehensive technical walkthrough on establishing a fully automated, Terraform-managed observability stack for AWS EKS using Datadog for monitoring and PagerDuty for incident management. By leveraging Infrastructure as Code (IaC), teams can ensure consistent, scalable, and auditable deployment of their monitoring and alerting infrastructure.

Architecture Pro-Tip

Always treat your observability configuration as code. Integrate your Terraform modules for Datadog agents, monitors, and PagerDuty services into your GitOps workflows. This ensures version control, peer review, and automated deployment, significantly reducing configuration drift and enabling faster recovery from incidents. Prioritize proactive alerting and dashboarding during your initial setup to establish a strong baseline for system health.

The Pillars of EKS Observability

Achieving comprehensive observability for AWS EKS involves collecting, analyzing, and acting upon vast amounts of data. This solution integrates three critical components:

AWS EKS: The managed Kubernetes service providing the core container orchestration platform.
Terraform: An open-source IaC tool used to provision and manage the EKS cluster itself, as well as the Datadog and PagerDuty integrations.
Datadog: A leading monitoring and analytics platform that provides end-to-end visibility into EKS clusters, including metrics, logs, traces, and events.
PagerDuty: An incident management platform that orchestrates timely and effective responses to operational incidents by routing alerts to the right teams.

Prerequisites

Before diving into the configuration, ensure you have the following in place:

An active AWS account with necessary permissions to create EKS clusters and associated resources.
Terraform CLI installed (v1.0.0 or higher recommended).
AWS CLI installed and configured with appropriate credentials.
Kubectl installed to interact with your EKS cluster.
A Datadog account with API and Application keys.
A PagerDuty account with an API key for creating services and integrations.
Helm CLI installed (v3.0.0 or higher) for deploying Datadog agents.

Step-by-Step Implementation

1. Provisioning AWS EKS with Terraform

While this guide focuses on observability, a functional EKS cluster is essential. We recommend using the official Terraform AWS EKS module for robust and production-ready cluster creation. For simplicity, we'll assume an EKS cluster is already provisioned or will be provisioned alongside the observability stack.

2. Integrating Datadog for EKS Observability

Datadog's strength lies in its comprehensive agent and powerful integrations. We'll deploy the Datadog Agent as a DaemonSet across your EKS cluster and define initial monitors and dashboards using the Datadog Terraform provider.

Deploying the Datadog Agent via Helm and Terraform

The Datadog Agent is best deployed on Kubernetes via its official Helm chart. Terraform's Helm provider can manage this deployment.

Key considerations for the Datadog Agent:

API Key: Required for the agent to send data to Datadog.
Cluster Agent: Recommended for large clusters to reduce API calls and manage metrics efficiently.
Features: Enable host metrics, Kubernetes metrics, APM (Application Performance Monitoring), logs collection, and network performance monitoring (NPM).

Defining Datadog Monitors and Dashboards with Terraform

Once data flows into Datadog, you can define monitors to alert on anomalies and dashboards to visualize your EKS health. The Datadog Terraform provider allows you to manage these resources as code.

Monitors: Create alerts for critical EKS metrics like node CPU/memory pressure, pod restarts, deployment failures, or high network utilization.
Dashboards: Build custom dashboards to provide at-a-glance visibility into cluster health, resource utilization, and application performance.

3. Configuring PagerDuty for Incident Response

PagerDuty streamlines incident response. We'll use the PagerDuty Terraform provider to define services, escalation policies, and integration endpoints.

Creating PagerDuty Services and Escalation Policies

A PagerDuty service represents a component or application that your team is responsible for. Each service is linked to an escalation policy that defines who gets notified and in what order.

Setting up PagerDuty Integrations

To receive alerts from Datadog, a PagerDuty service needs an integration. For Datadog, this is typically a "Datadog" integration type within PagerDuty, which provides an integration key. The Terraform provider can manage this, or you can retrieve it manually for use in Datadog monitor configurations.

4. Connecting Datadog to PagerDuty

The final step is to tell Datadog which PagerDuty service to alert when a monitor triggers. This is configured within the Datadog monitor definition using the PagerDuty integration key.

Terraform Configuration Example

Below is a consolidated Terraform configuration demonstrating how to set up the providers, deploy the Datadog Agent, create a sample Datadog monitor, and configure a PagerDuty service and integration.

provider "aws" {
  region = "us-east-1"
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.example.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.example.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.example.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.example.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.example.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.example.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_key
}

# --- Data Sources for EKS Cluster (assuming pre-existing) ---
data "aws_eks_cluster" "example" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "example" {
  name = var.eks_cluster_name
}

# --- Deploy Datadog Agent with Helm ---
resource "helm_release" "datadog_agent" {
  name       = "datadog-agent"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Or dedicated 'datadog' namespace
  version    = "2.33.0" # Use a specific version
  
  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }
  
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
  }

  set {
    name  = "datadog.logs.enabled"
    value = "true"
  }

  set {
    name  = "datadog.apm.enabled"
    value = "true"
  }
  
  # Ensure the agent collects EKS specific metrics and logs
  set {
    name  = "datadog.confd.kube_state_metrics_core.yaml"
    value = file("datadog_kube_state_metrics_core.yaml") # Custom config if needed
    type  = "string"
  }
}

# --- PagerDuty Service Configuration ---
resource "pagerduty_user" "on_call_engineer" {
  name  = "On-Call Engineer"
  email = "oncall@example.com"
}

resource "pagerduty_escalation_policy" "devops_escalation" {
  name = "DevOps Primary Escalation"
  rule {
    delay_minutes = 0
    target {
      id   = pagerduty_user.on_call_engineer.id
      type = "user_reference"
    }
  }
}

resource "pagerduty_service" "eks_observability" {
  name                     = "EKS Observability"
  escalation_policy        = pagerduty_escalation_policy.devops_escalation.id
  auto_resolve_timeout     = 14400 # 4 hours
  acknowledgement_timeout = 600   # 10 minutes
}

resource "pagerduty_service_integration" "datadog_integration" {
  name    = "Datadog Integration"
  service = pagerduty_service.eks_observability.id
  type    = "datadog_inbound_integration"
}

# --- Datadog Monitor Configuration ---
resource "datadog_monitor" "eks_high_cpu" {
  name                = "EKS Node High CPU Usage ({{host.name}})"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.cpu.usage.total{*} by {host} > 80"
  message             = "EKS Node {{host.name}} is experiencing high CPU utilization ({{value}}%). Please investigate. @pagerduty-EKS-Observability"
  
  tags = [
    "environment:production",
    "service:eks-cluster",
    "alert-type:critical",
  ]

  options {
    thresholds {
      critical = 80
      warning  = 70
    }

    notify_no_data    = false
    new_host_delay    = 300
    no_data_timeframe = 10
    renotify_interval = 60
    silenced          = {}
  }
  
  # Reference PagerDuty integration by its name or ID. 
  # For PagerDuty, typically you mention @pagerduty- in the message.
  # The datadog_monitor resource itself doesn't directly link to PagerDuty service via ID.
  # Instead, Datadog uses the integration name setup in its UI, referenced in the monitor message.
  # The `pagerduty_service_integration` resource above gives you the `integration_key`.
  # You would configure the actual PagerDuty integration in Datadog UI under integrations, 
  # then reference it in the monitor message as '@pagerduty-'.
  # For full automation, you might need to use Datadog API directly to configure the PagerDuty integration itself.
  # The example above assumes the PagerDuty integration is named "EKS-Observability" in Datadog.
}

# --- Variables ---
variable "eks_cluster_name" {
  description = "Name of the existing EKS cluster"
  type        = string
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_key" {
  description = "PagerDuty API Key"
  type        = string
  sensitive   = true
}

Note: The direct programmatic linking of a Datadog monitor to a PagerDuty service via Terraform can be nuanced. Datadog integrates with PagerDuty by referencing a configured PagerDuty integration (set up once in Datadog's integrations section) within the monitor's message using an @pagerduty-<IntegrationName> tag. The pagerduty_service_integration resource in Terraform generates an integration key that you would use to configure the Datadog side of the integration (typically manually or via Datadog's API if not available in the Terraform provider directly).

Verification and Testing

After applying your Terraform configuration, it's crucial to verify that everything is working as expected:

Datadog Agent: Check your Datadog account under "Infrastructure" -> "Container" to ensure your EKS nodes and pods are reporting metrics, logs, and traces.
Datadog Dashboards: Verify that your custom dashboards are populated with EKS data.
PagerDuty Service: Confirm that the "EKS Observability" service (or whatever you named it) exists in PagerDuty and has the correct escalation policy.
Alerting Flow: To test the full flow, you can simulate a critical event. For example, manually exhaust CPU on an EKS node or create a deployment that fails repeatedly to trigger the Datadog monitor. Verify that an incident is created in PagerDuty and the on-call team is notified.

Advanced Considerations

RBAC for Datadog Agent: Ensure your Datadog Agent has appropriate Kubernetes RBAC permissions to collect all necessary metrics and logs.
Custom Metrics and Logs: Extend your Datadog integration to collect custom application metrics and structured logs for deeper insights.
Distributed Tracing (APM): Instrument your applications to leverage Datadog APM for end-to-end distributed tracing across your microservices.
Cost Optimization: Regularly review your Datadog usage, especially for logs and custom metrics, to optimize costs. Implement intelligent filtering and sampling where appropriate.
Automated Remediation: Integrate PagerDuty webhooks with custom scripts or serverless functions to trigger automated remediation steps for common, well-understood incidents.
Security Best Practices: Store API keys and other sensitive information securely using AWS Secrets Manager or HashiCorp Vault, and reference them in Terraform.

Troubleshooting and FAQ

Q: Datadog Agent pods are not running or reporting.

A: Check the pod logs for errors (kubectl logs <datadog-agent-pod>). Verify the datadog.apiKey is correctly set. Ensure the agent has sufficient RBAC permissions (check associated ClusterRole and ClusterRoleBinding). Network policies might be blocking egress to Datadog endpoints.

Q: Datadog monitors are not triggering PagerDuty incidents.

A: First, ensure the Datadog monitor itself is triggering an alert within Datadog. If it is, verify that the @pagerduty-<IntegrationName> tag in the monitor's message exactly matches the name of your PagerDuty integration configured in Datadog. Check the PagerDuty integration's event log for incoming events.

Q: Terraform apply fails with PagerDuty API errors.

A: Double-check your pagerduty_api_key. Ensure it has the necessary permissions to create services, escalation policies, and integrations. API keys can sometimes be scope-limited.

Q: How do I manage multiple EKS clusters with this setup?

A: You can modularize your Terraform code. Create a root module for each EKS cluster, calling a shared observability module that deploys the Datadog Agent, monitors, and PagerDuty services, possibly using naming conventions to differentiate resources per cluster.

Conclusion

By strategically combining Terraform, AWS EKS, Datadog, and PagerDuty, organizations can build a robust, automated, and highly effective observability and incident response framework. This approach not only provides deep insights into the health and performance of your EKS clusters but also ensures that critical issues are identified and addressed rapidly, significantly reducing downtime and improving operational efficiency. Embracing Infrastructure as Code for your observability stack is a pivotal step towards true cloud-native operational excellence.

CloudStack YAML