Terraform-Managed PagerDuty Integration for AWS EKS Datadog Alerts

In modern cloud-native environments, robust monitoring and incident response are non-negotiable. Managing alerts from complex systems like AWS EKS, collecting metrics with Datadog, and ensuring critical incidents reach the right on-call personnel via PagerDuty requires a scalable, automated approach. This guide provides a comprehensive, technical walkthrough on integrating Datadog alerts for AWS EKS with PagerDuty, all managed as Infrastructure as Code (IaC) using Terraform.

Architecture Pro-Tip

Standardizing your monitoring and incident response integrations with Terraform provides unparalleled benefits: version control, auditability, disaster recovery, and consistent deployments across environments. Treat your alerting configuration not just as settings, but as critical infrastructure. Automating this layer reduces human error, speeds up onboarding, and ensures your incident response posture evolves with your infrastructure through code reviews and CI/CD pipelines.

Why Terraform, Datadog, PagerDuty, and AWS EKS?

AWS EKS (Elastic Kubernetes Service): A fully managed Kubernetes service that simplifies the deployment, management, and scaling of containerized applications on AWS. It forms the backbone of highly available, scalable microservices architectures.
Datadog: A leading monitoring and analytics platform that brings together data from servers, containers, databases, and third-party services to make your stack observable. Its powerful alerting engine can detect anomalies and critical states within your EKS clusters.
PagerDuty: An incident management platform that routes alerts to the right teams, manages on-call schedules, and streamlines the incident resolution process. It transforms raw alerts into actionable incidents.
Terraform: An open-source Infrastructure as Code tool from HashiCorp. It allows you to define and provision infrastructure in a declarative configuration language. By managing Datadog monitors and PagerDuty services with Terraform, you ensure consistency, version control, and auditability.

Prerequisites

Before you begin, ensure you have the following:

An active AWS Account with administrative access.
An existing AWS EKS cluster. Datadog agents should ideally be deployed and configured to collect metrics from your cluster.
An active Datadog Account with API and Application keys.
An active PagerDuty Account with a Global API Key.
Terraform CLI installed locally (version 1.0+ recommended).
Familiarity with HCL (HashiCorp Configuration Language).

Step-by-Step Integration Guide

Step 1: PagerDuty Setup with Terraform

First, we'll define the necessary PagerDuty components using Terraform. This includes an Escalation Policy, an on-call Service, and the integration key required by Datadog.

1.1 Configure PagerDuty Provider

provider "pagerduty" {
  token = var.pagerduty_api_token
}

variable "pagerduty_api_token" {
  description = "PagerDuty Global API Token"
  type        = string
  sensitive   = true
}
    

1.2 Create PagerDuty Escalation Policy

This policy defines who gets notified and when. For simplicity, we'll create a basic policy that notifies a specific user immediately.

resource "pagerduty_user" "devops_oncall" {
  name  = "DevOps Engineer"
  email = "devops@example.com"
  # Optional: time_zone, job_title
}

resource "pagerduty_escalation_policy" "eks_alerts_policy" {
  name      = "EKS Datadog Alert Escalation Policy"
  num_loops = 2 # Number of times to repeat the policy

  rule {
    escalation_delay_in_minutes = 0
    target {
      type = "user"
      id   = pagerduty_user.devops_oncall.id
    }
  }

  # Add more rules for additional escalation tiers if needed
  # rule {
  #   escalation_delay_in_minutes = 30
  #   target {
  #     type = "team"
  #     id   = pagerduty_team.my_team.id # Requires defining a pagerduty_team resource
  #   }
  # }
}
    

1.3 Create PagerDuty Service for EKS Datadog Alerts

This service acts as the integration point for Datadog alerts.

resource "pagerduty_service" "eks_datadog_service" {
  name                    = "EKS Datadog Monitoring"
  auto_resolve_timeout    = "14400" # 4 hours
  acknowledgement_timeout = "600"   # 10 minutes
  escalation_policy       = pagerduty_escalation_policy.eks_alerts_policy.id
  description             = "Service for receiving alerts from Datadog about EKS cluster health."
}

resource "pagerduty_service_integration" "datadog_integration" {
  name    = "Datadog Integration"
  type    = "datadog" # This specific type creates a Datadog-specific integration
  service = pagerduty_service.eks_datadog_service.id
}

output "pagerduty_datadog_integration_key" {
  description = "The PagerDuty integration key for Datadog."
  value       = pagerduty_service_integration.datadog_integration.integration_key
  sensitive   = true
}
    

Step 2: Datadog Setup with Terraform

Next, we'll configure Datadog to use the PagerDuty integration and define monitors for our EKS cluster.

2.1 Configure Datadog Provider

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}
    

2.2 Configure Datadog PagerDuty Integration

This step establishes the connection between Datadog and PagerDuty using the integration key obtained from PagerDuty.

resource "datadog_integration_pagerduty" "pagerduty_default" {
  api_key = output.pagerduty_datadog_integration_key # Referencing the output from PagerDuty setup
}
    

2.3 Create Datadog Monitors for EKS

Here, we define a couple of example monitors relevant to EKS. Replace `your_eks_cluster_name` with your actual EKS cluster name.

resource "datadog_monitor" "eks_high_cpu" {
  name  = "[EKS - ${var.eks_cluster_name}] High CPU Usage on Nodes"
  type  = "metric alert"
  query = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.eks_cluster_name}} by {host} > 80"

  message = <Step 3: AWS EKS Context (Datadog Agent)While the direct integration between PagerDuty and Datadog is managed by Terraform, it's crucial to ensure your AWS EKS cluster is properly sending metrics to Datadog. This typically involves deploying the Datadog Agent as a DaemonSet within your EKS cluster.
Ensure the Datadog Agent is deployed on your EKS cluster, preferably using Helm or the Datadog Operator.
Configure the agent with your Datadog API key and enable Kubernetes metrics collection.
Verify that EKS metrics (CPU, memory, network, pod status, etc.) are visible in your Datadog dashboard. The monitors defined above rely on these metrics being present.


Consolidated Terraform ConfigurationHere's a complete `main.tf` file combining all the pieces. Remember to create a `terraform.tfvars` file or use environment variables for sensitive data.

# main.tf

# PagerDuty Provider
provider "pagerduty" {
  token = var.pagerduty_api_token
}

# Datadog Provider
provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

# PagerDuty User (On-Call Engineer)
resource "pagerduty_user" "devops_oncall" {
  name  = "DevOps Engineer"
  email = "devops@example.com"
}

# PagerDuty Escalation Policy
resource "pagerduty_escalation_policy" "eks_alerts_policy" {
  name      = "EKS Datadog Alert Escalation Policy"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 0
    target {
      type = "user"
      id   = pagerduty_user.devops_oncall.id
    }
  }
}

# PagerDuty Service for Datadog EKS Alerts
resource "pagerduty_service" "eks_datadog_service" {
  name                    = "EKS Datadog Monitoring"
  auto_resolve_timeout    = "14400"
  acknowledgement_timeout = "600"
  escalation_policy       = pagerduty_escalation_policy.eks_alerts_policy.id
  description             = "Service for receiving alerts from Datadog about EKS cluster health."
}

# PagerDuty Service Integration for Datadog
resource "pagerduty_service_integration" "datadog_integration" {
  name    = "Datadog Integration"
  type    = "datadog"
  service = pagerduty_service.eks_datadog_service.id
}

# Datadog PagerDuty Integration
resource "datadog_integration_pagerduty" "pagerduty_default" {
  api_key = pagerduty_service_integration.datadog_integration.integration_key
}

# Datadog Monitor: High CPU Usage on EKS Nodes
resource "datadog_monitor" "eks_high_cpu" {
  name  = "[EKS - ${var.eks_cluster_name}] High CPU Usage on Nodes"
  type  = "metric alert"
  query = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.eks_cluster_name}} by {host} > 80"

  message = <Deployment and VerificationTo deploy this configuration:
Save the code above as `main.tf` in an empty directory.
Create a `terraform.tfvars` file for your sensitive variables:
            
pagerduty_api_token = "YOUR_PAGERDUTY_GLOBAL_API_TOKEN"
datadog_api_key     = "YOUR_DATADOG_API_KEY"
datadog_app_key     = "YOUR_DATADOG_APP_KEY"
eks_cluster_name    = "your-prod-eks-cluster"
            
Run `terraform init` to initialize the providers.
Run `terraform plan` to review the changes Terraform will apply.
Run `terraform apply` and type `yes` when prompted to create the resources.
Verification:
Check your PagerDuty account: A new service "EKS Datadog Monitoring" should appear, linked to the "EKS Datadog Alert Escalation Policy".
Check your Datadog account: Two new monitors should be visible under Monitors -> Manage Monitors. The PagerDuty integration should also show as configured.
To test an alert, you could intentionally cause a high CPU load on an EKS node (e.g., by deploying a busybox pod with an infinite loop) or simulate pod restarts. This should trigger a Datadog alert, which in turn should create an incident in PagerDuty.
Advanced ConsiderationsTerraform State Management: For production environments, always configure a remote backend (e.g., AWS S3 with DynamoDB locking) for your Terraform state.
Terraform Workspaces: Use Terraform workspaces or separate directories to manage configurations for different environments (dev, staging, prod).
Custom Alerting Logic: Expand on the Datadog monitors with more sophisticated queries, composite monitors, or anomaly detection for advanced EKS health checks.
Runbook Automation: Enhance your PagerDuty service with custom incident actions or integrate with runbook automation tools to streamline response.
Team Integration: Instead of individual users, integrate PagerDuty teams with escalation policies for better on-call rotation management.


Troubleshooting and Best PracticesCommon IssuesAPI Key/Token Errors: Double-check that your PagerDuty Global API Key, Datadog API Key, and Datadog Application Key are correct and have the necessary permissions. These are common sources of `401 Unauthorized` or `403 Forbidden` errors.
Datadog Agent Connectivity: Ensure your Datadog Agent on EKS nodes can successfully connect to the Datadog ingest endpoints and that metrics are flowing. Without metrics, monitors won't trigger.
Monitor Query Issues: Verify the Datadog monitor queries match the actual metric names and tags being reported by your EKS cluster. Use the Datadog Metric Explorer to confirm metric availability and correct tag syntax.
PagerDuty Integration Key Propagation: Ensure the `output` from the PagerDuty resource correctly feeds into the `datadog_integration_pagerduty` resource. Terraform's dependency graph should handle this automatically.
Best PracticesSecrets Management: Never hardcode API keys or tokens. Use `terraform.tfvars` (excluded from VCS), environment variables, or ideally, a secrets manager like AWS Secrets Manager or HashiCorp Vault.
Idempotency: Terraform's nature makes it idempotent. Repeated `apply` operations should not cause unintended side effects, only reconcile the state.
Modularization: For larger setups, break down your Terraform configuration into modules (e.g., `pagerduty-module`, `datadog-monitors-module`) for better organization and reusability.
Documentation: Keep your Terraform code well-documented, explaining the purpose of each resource and variable.
Alert Fatigue: Design your monitors carefully. Too many alerts can lead to alert fatigue. Focus on actionable alerts that indicate a genuine problem requiring human intervention.
ConclusionBy leveraging Terraform to manage your PagerDuty and Datadog integrations for AWS EKS alerts, you establish a robust, automated, and auditable incident management workflow. This Infrastructure as Code approach ensures consistency, reduces manual errors, and empowers your DevOps teams to respond to critical events efficiently, ultimately improving system reliability and reducing MTTR (Mean Time To Resolution). Embrace the power of automation to build resilient cloud-native operations.

CloudStack YAML