Terraform for AWS EKS Observability with Datadog and PagerDuty Alerting

Architecture Pro-Tip:

For large-scale or multi-account AWS EKS deployments, centralize your observability configuration. Utilize dedicated IAM roles with minimal necessary permissions for Datadog agents and integrations. Structure your Terraform modules to be reusable and environment-agnostic, separating EKS cluster deployment from observability tooling. This modular approach significantly reduces complexity, improves maintainability, and enhances security posture across your Kubernetes fleet.

In today's dynamic cloud-native landscape, ensuring the health, performance, and reliability of your Kubernetes workloads on AWS EKS is paramount. Observability is no longer an optional add-on but a fundamental requirement for proactive incident response and maintaining optimal application performance. This guide provides a comprehensive, technical walkthrough on leveraging Terraform to establish robust observability for AWS EKS using Datadog for monitoring and PagerDuty for incident alerting.

Why Terraform, AWS EKS, Datadog, and PagerDuty?

Terraform: Infrastructure as Code (IaC) for Consistency

Terraform empowers you to define and provision your infrastructure using a declarative configuration language. For EKS observability, this means managing Datadog integrations, monitors, and PagerDuty services as code, ensuring:

Repeatability: Deploy identical observability stacks across development, staging, and production environments.
Version Control: Track changes, review configurations, and roll back if necessary.
Automation: Eliminate manual errors and accelerate deployment cycles.
Drift Detection: Identify and rectify discrepancies between desired and actual state.

AWS EKS: Managed Kubernetes Service

Amazon Elastic Kubernetes Service (EKS) simplifies running Kubernetes on AWS without needing to install, operate, and maintain your own Kubernetes control plane. However, the complexity of distributed systems within Kubernetes still demands comprehensive visibility into metrics, logs, and traces from pods, nodes, and the control plane itself.

Datadog: Unified Observability Platform

Datadog provides an end-to-end observability solution, consolidating metrics, logs, traces, and synthetic monitoring into a single pane of glass. For EKS, Datadog offers:

Kubernetes Integration: Automatically collects data from EKS clusters, nodes, pods, and containers.
Rich Dashboards: Pre-built and customizable dashboards for EKS health and performance.
APM and Tracing: Detailed visibility into application performance bottlenecks.
Log Management: Centralized log collection, analysis, and alerting.
Anomaly Detection: Proactive identification of unusual behavior using machine learning.

PagerDuty: Incident Management and On-Call Automation

When issues arise, prompt and efficient incident response is critical. PagerDuty integrates seamlessly with Datadog to transform alerts into actionable incidents, ensuring the right team member is notified at the right time. Key benefits include:

On-Call Scheduling: Manages complex on-call rotations and escalation policies.
Automated Notifications: Delivers alerts via multiple channels (SMS, phone call, email, push).
Incident Tracking: Provides a centralized platform for managing incident lifecycle.
Reduced Alert Fatigue: Intelligent grouping and deduplication of alerts.

Prerequisites

Before you begin, ensure you have the following:

AWS Account: With administrative access to create EKS clusters and associated resources.
Terraform CLI: Installed (version 1.0+ recommended).
AWS CLI: Configured with appropriate credentials.
kubectl: Installed and configured to interact with your EKS cluster.
Helm CLI: Installed (version 3+ recommended) for deploying Datadog Agent.
Datadog Account: With API and Application keys.
PagerDuty Account: With an API key and a service already created or to be created.
Existing EKS Cluster: This guide assumes you have an EKS cluster already running. If not, you can use Terraform to provision one.

Terraform Setup: Providers and Authentication

First, define your Terraform providers for AWS, Kubernetes, Helm, and Datadog. Ensure your AWS provider is configured for the correct region and that your Kubernetes provider can authenticate to your EKS cluster.

provider "aws" {
  region = "us-east-1"
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.example.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.example.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.example.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.example.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.example.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.example.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

data "aws_eks_cluster" "example" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "example" {
  name = var.eks_cluster_name
}

variable "eks_cluster_name" {
  description = "The name of your EKS cluster"
  type        = string
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}
    

Integrating Datadog with EKS

The Datadog Agent is deployed as a DaemonSet across your EKS cluster nodes. This agent collects metrics, logs, and traces, sending them to your Datadog account. We'll use the Helm provider to deploy the official Datadog Agent chart.

Deploying the Datadog Agent with Helm

The `helm_release` resource allows you to manage Helm chart deployments directly via Terraform. We'll configure it to deploy the Datadog Agent, providing the necessary API keys and enabling specific integrations.

Configuring Datadog Monitors with Terraform

Once the agent is collecting data, you can define monitors (alerts) using the `datadog_monitor` resource. These monitors check for specific conditions (e.g., high CPU utilization, low available memory, pod restarts) and trigger alerts if thresholds are breached.

Integrating PagerDuty for Alerting

To route Datadog alerts to PagerDuty, you first need to set up the integration in Datadog, then reference this integration within your Datadog monitors.

Setting up the Datadog-PagerDuty Integration

The `datadog_integration_pagerduty` resource configures the connection between Datadog and PagerDuty. You'll need a PagerDuty API key and the service keys for the services you want to integrate.

Ready-to-Use Terraform Configuration

Below is a comprehensive Terraform configuration block that demonstrates how to:

Define Datadog and PagerDuty API keys securely (via variables).
Deploy the Datadog Agent Helm chart to your EKS cluster.
Configure the Datadog-PagerDuty integration.
Create a sample Datadog monitor for EKS node CPU utilization, sending alerts to PagerDuty.

# main.tf

# --- Providers (as shown above) ---
# provider "aws" { ... }
# provider "kubernetes" { ... }
# provider "helm" { ... }
# provider "datadog" { ... }
# data "aws_eks_cluster" "example" { ... }
# data "aws_eks_cluster_auth" "example" { ... }

# --- Variables ---
# variable "eks_cluster_name" { ... }
# variable "datadog_api_key" { ... }
# variable "datadog_app_key" { ... }

variable "pagerduty_api_key" {
  description = "PagerDuty API Key for Datadog integration"
  type        = string
  sensitive   = true
}

variable "pagerduty_service_key_eks_critical" {
  description = "PagerDuty service key for critical EKS alerts"
  type        = string
  sensitive   = true
}

# --- Deploy Datadog Agent to EKS ---
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }
  set {
    name  = "datadog.site"
    value = "datadoghq.com" # or eu.datadoghq.com, us3.datadoghq.com, etc.
  }
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
  }
  set {
    name  = "targetSystem"
    value = "linux"
  }
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }
  set {
    name  = "apm.enabled"
    value = "true"
  }
  set {
    name  = "processAgent.enabled"
    value = "true"
  }
  set {
    name  = "processAgent.processCollection"
    value = "true"
  }
}

# --- Configure Datadog PagerDuty Integration ---
resource "datadog_integration_pagerduty" "eks_critical_pd_integration" {
  api_key     = var.pagerduty_api_key
  services = [
    {
      service_name = "EKS Critical Alerts"
      service_key  = var.pagerduty_service_key_eks_critical
    }
  ]
}

# --- Datadog Monitor for EKS Node CPU Utilization ---
resource "datadog_monitor" "eks_node_cpu_critical" {
  name               = "EKS Node CPU Utilization Critical - {{host.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster_name:${var.eks_cluster_name}} by {host} < 10" # less than 10% idle (i.e., > 90% utilized)
  message            = <Note: For the PagerDuty notification within the Datadog monitor message, the exact syntax (e.g., `@webhook-pagerduty-eks-critical-alerts` or `@pagerduty-service-name`) depends on how you've named and configured your PagerDuty integration within Datadog. Verify this name in your Datadog account under Integrations -> PagerDuty. The `datadog_integration_pagerduty` resource primarily sets up the backend connection.
Deployment StepsTo deploy this configuration:
Save the code in a `main.tf` file.
Create a `terraform.tfvars` file (or use environment variables) to provide your sensitive API keys and cluster name:
            
eks_cluster_name                 = "your-eks-cluster-name"
datadog_api_key                  = "YOUR_DATADOG_API_KEY"
datadog_app_key                  = "YOUR_DATADOG_APP_KEY"
pagerduty_api_key                = "YOUR_PAGERDUTY_API_KEY"
pagerduty_service_key_eks_critical = "YOUR_PAGERDUTY_EKS_CRITICAL_SERVICE_KEY"
            
Run `terraform init` to initialize the working directory and download providers.
Run `terraform plan` to preview the changes.
Run `terraform apply` to apply the configuration.
Best Practices for EKS ObservabilityTagging: Consistently tag all your AWS resources and Kubernetes objects. Datadog leverages these tags for filtering, grouping, and context.
Custom Metrics: Beyond standard metrics, expose application-specific custom metrics to Datadog for deeper insights into business logic.
Distributed Tracing (APM): Instrument your applications to send traces to Datadog, providing end-to-end visibility across microservices.
Service Level Objectives (SLOs): Define SLOs in Datadog to measure the reliability of your EKS applications and ensure you meet user expectations.
Cost Optimization: Monitor Datadog agent resource consumption. Be mindful of log volume and custom metric count to manage costs effectively.
Security: Regularly review IAM roles and Kubernetes RBAC for the Datadog Agent, adhering to the principle of least privilege.


Troubleshooting Common IssuesDatadog Agent Pods Not Running: Check `kubectl get pods -n datadog` and `kubectl describe pod  -n datadog` for events. Ensure sufficient resources (CPU/Memory) are available on nodes.
No Data in Datadog:
            Verify `datadog.apiKey` and `datadog.appKey` in your Helm release are correct.
Check agent logs: `kubectl logs -f  -n datadog`. Look for errors sending data.
Ensure security groups and network ACLs allow outbound traffic from your EKS nodes to Datadog endpoints.

        
PagerDuty Alerts Not Triggering:
            Confirm the `pagerduty_api_key` and `pagerduty_service_key` are valid in your Terraform configuration and Datadog integration.
Double-check the `@webhook-pagerduty-` or `@pagerduty-service-name` syntax in your Datadog monitor message matches your Datadog PagerDuty integration alias exactly.
Test the Datadog monitor by temporarily lowering thresholds to trigger an alert.

        
Terraform `kubernetes` Provider Authentication: If you're having issues, try running `aws eks update-kubeconfig --name ${var.eks_cluster_name}` manually and verify `kubectl` works before running Terraform. Ensure your AWS credentials have permission to describe EKS clusters and generate tokens.
ConclusionBy adopting Terraform for managing your EKS observability stack with Datadog and PagerDuty, you gain unparalleled control, consistency, and automation. This approach not only streamlines deployment but also enforces best practices, allowing your team to focus on innovation rather than manual configuration and firefighting. A well-implemented observability strategy is the cornerstone of reliable and high-performing cloud-native applications, ensuring you're always aware of your EKS environment's health and can respond effectively to incidents.

CloudStack YAML