Terraform-Automated PagerDuty Incident Management for AWS EKS Prometheus Alerts via Datadog

In the fast-paced world of modern cloud infrastructure, ensuring high availability and rapid response to incidents is paramount. For organizations running critical applications on AWS EKS with Prometheus for monitoring, integrating an automated incident management system like PagerDuty is crucial. This guide will walk you through setting up a robust, automated incident response pipeline, leveraging Terraform for Infrastructure as Code (IaC) to manage PagerDuty services and Datadog monitors, which ingest Prometheus alerts from your EKS clusters.

Architecture Pro-Tip: Federated Monitoring for Scalability

For large-scale EKS deployments, consider a federated Prometheus setup or a centralized Prometheus-compatible metrics store like Amazon Managed Service for Prometheus (AMP). Datadog can then ingest metrics from these centralized sources, reducing the monitoring load on individual EKS clusters and simplifying alert configuration across your entire estate. Ensure consistent naming conventions for metrics and labels to streamline Datadog queries and PagerDuty routing.

The Incident Response Ecosystem Explained

Our integrated solution brings together several powerful tools:

• AWS EKS: The managed Kubernetes service, providing the backbone for containerized applications.
• Prometheus: The de facto open-source monitoring system, deployed within EKS to scrape metrics from applications and Kubernetes components.
• Datadog: A comprehensive monitoring and analytics platform that can ingest Prometheus metrics, provide advanced visualization, and act as an alert manager.
• PagerDuty: A leading incident management platform that orchestrates on-call rotations, escalates incidents, and ensures timely responses.
• Terraform: An IaC tool that allows us to define and provision our cloud and SaaS infrastructure, including PagerDuty services and Datadog monitors, in a repeatable and version-controlled manner.

Prerequisites

Before you begin, ensure you have the following:

• An active AWS account with an EKS cluster running.
• Prometheus deployed and collecting metrics within your EKS cluster (e.g., via kube-prometheus-stack).
• A Datadog account with API and Application keys.
• A PagerDuty account with API keys (a REST API key and optionally an Admin API key for full Terraform control).
• Terraform CLI installed and configured with AWS, Datadog, and PagerDuty providers.
• Basic understanding of Kubernetes, Prometheus, Datadog, PagerDuty, and Terraform concepts.

Step 1: Datadog Integration with EKS Prometheus

Datadog offers robust integrations for Kubernetes and Prometheus. You'll typically deploy the Datadog Agent to your EKS cluster as a DaemonSet. This agent can automatically discover and scrape Prometheus metrics endpoints within your cluster.

Ensure your Datadog Agent configuration (e.g., via Helm chart values) includes Prometheus metric collection:

datadog: apiKey: <YOUR_DATADOG_API_KEY> appKey: <YOUR_DATADOG_APP_KEY> kubeStateMetricsEnabled: true prometheusEnabled: true # For custom Prometheus endpoints, configure in annotations or agent config # For example, to scrape Prometheus service discovery: confd: prometheus.yaml: | init_config: instances: - prometheus_url: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090/metrics metrics: - '*' # Collect all metrics, or specify a list

Once the agent is deployed and configured, you should see your Prometheus metrics appearing in Datadog's Metrics Explorer.

Step 2: PagerDuty Service Setup

Before automating with Terraform, understand the core PagerDuty components:

• Escalation Policy: Defines the order in which users or teams are notified when an incident occurs.
• Service: Represents a component or application that PagerDuty monitors. Each service has an integration endpoint.
• Integration: The mechanism through which an external system (like Datadog) sends alerts to a PagerDuty service. We'll use a Datadog integration type.

Step 3: Terraform for PagerDuty and Datadog Automation

This is where Terraform shines. We'll define our PagerDuty services, escalation policies, and Datadog monitors as code, ensuring consistency and enabling version control.

Terraform Providers Configuration

First, set up your Terraform providers for PagerDuty and Datadog:

terraform { required_providers { pagerduty = { source = "pagerduty/pagerduty" version = "~> 1.14" } datadog = { source = "DataDog/datadog" version = "~> 3.0" } } } provider "pagerduty" { token = var.pagerduty_api_token } provider "datadog" { api_key = var.datadog_api_key app_key = var.datadog_app_key } variable "pagerduty_api_token" { description = "PagerDuty API token" type = string sensitive = true } variable "datadog_api_key" { description = "Datadog API Key" type = string sensitive = true } variable "datadog_app_key" { description = "Datadog Application Key" type = string sensitive = true }

Terraform for PagerDuty Escalation Policy and Service

Define an escalation policy and a service. For simplicity, we'll create a basic policy and one EKS service.

# Define an on-call schedule (ensure this exists in PagerDuty or create one with Terraform) # For this example, we'll assume a schedule ID exists. # data "pagerduty_schedule" "devops_schedule" { # name = "DevOps On-Call Schedule" # } # Define an escalation policy resource "pagerduty_escalation_policy" "eks_critical_ep" { name = "EKS Critical Alerts Escalation Policy" num_loops = 2 rule { delay_before_free = 15 target { type = "user" id = var.pagerduty_user_id # Replace with a user ID or use data "pagerduty_user" to fetch } # Or use a schedule: # target { # type = "schedule" # id = data.pagerduty_schedule.devops_schedule.id # } } } # Define the PagerDuty service for EKS alerts resource "pagerduty_service" "eks_prometheus_service" { name = "EKS Prometheus Alerts" auto_resolve_timeout = "14400" # 4 hours acknowledgement_timeout = "600" # 10 minutes escalation_policy = pagerduty_escalation_policy.eks_critical_ep.id alert_creation = "create_alerts_and_incidents" # Or "create_incidents" } # Add a Datadog integration to the PagerDuty service resource "pagerduty_service_integration" "datadog_integration" { name = "Datadog Integration" type = "datadog_reference_v2" service = pagerduty_service.eks_prometheus_service.id } output "pagerduty_integration_key" { description = "The PagerDuty integration key for Datadog" value = pagerduty_service_integration.datadog_integration.integration_key sensitive = true } variable "pagerduty_user_id" { description = "ID of a PagerDuty user to assign to the escalation policy" type = string sensitive = true # User ID can be sensitive if it directly identifies someone }

Terraform Configuration for Datadog Monitors

Now, let's create a Datadog monitor that checks for a critical Prometheus metric, and if triggered, sends an alert to our PagerDuty service using the integration key.

We'll define a monitor for a common EKS/Prometheus alert scenario: high CPU utilization for a specific deployment.

resource "datadog_monitor" "eks_high_cpu" { name = "EKS High CPU: {{kube_deployment}} in {{kube_namespace}}" type = "metric alert" query = "avg(last_5m):avg:kubernetes.cpu.usage.total{kube_deployment:my-critical-app} by {kube_deployment,kube_namespace} > 80" message = "EKS CPU utilization for {{kube_deployment}} in namespace {{kube_namespace}} is above 80% for 5 minutes. @pagerduty-EKS-Prometheus-Alerts" tags = ["environment:production", "team:devops", "service:my-critical-app"] monitor_threshold_windows { trigger_window = "last_5m" } new_group_delay = 60 # seconds no_data_timeframe = 20 # minutes notify_no_data = false renotify_interval = 0 # No re-notification escalation_message = "CPU remains high for {{kube_deployment}}. Escalating!" require_full_window = true evaluation_delay = 30 # seconds # PagerDuty integration via the message attribute # The '@pagerduty-SERVICE_NAME' syntax is how Datadog identifies the PagerDuty service. # The SERVICE_NAME corresponds to the PagerDuty service you created. # Alternatively, you can use the integration_id if your PagerDuty integration supports it. # Ensure 'EKS-Prometheus-Alerts' matches the PagerDuty service name configured earlier. } resource "datadog_monitor" "eks_high_memory" { name = "EKS High Memory: {{kube_deployment}} in {{kube_namespace}}" type = "metric alert" query = "avg(last_5m):avg:kubernetes.memory.usage.total{kube_deployment:my-critical-app} by {kube_deployment,kube_namespace} > 90" message = "EKS Memory utilization for {{kube_deployment}} in namespace {{kube_namespace}} is above 90% for 5 minutes. @pagerduty-EKS-Prometheus-Alerts" tags = ["environment:production", "team:devops", "service:my-critical-app"] monitor_threshold_windows { trigger_window = "last_5m" } new_group_delay = 60 no_data_timeframe = 20 notify_no_data = false renotify_interval = 0 escalation_message = "Memory remains high for {{kube_deployment}}. Escalating!" require_full_window = true evaluation_delay = 30 }

Deployment Steps

With your Terraform configuration ready:

1. Save: Save the `.tf` files (e.g., `main.tf`, `variables.tf`).
2. Initialize: Run `terraform init` in your terminal to download providers.
3. Plan: Run `terraform plan` to see what resources Terraform will create, update, or destroy. Review this carefully.
4. Apply: Run `terraform apply` to provision the resources. You'll be prompted to confirm.

Ensure you provide the sensitive variables (PagerDuty API token, Datadog API/App keys, PagerDuty user ID) via environment variables or a `terraform.tfvars` file (with caution for sensitive data).

Testing and Validation

To validate your setup:

• Verify PagerDuty: Log into PagerDuty. You should see the new "EKS Prometheus Alerts" service and the "EKS Critical Alerts Escalation Policy."
• Verify Datadog: Log into Datadog. Navigate to Monitors -> Monitor Management. You should see the "EKS High CPU" and "EKS High Memory" monitors.
• Trigger an Alert: Manually simulate high CPU/memory usage on your EKS `my-critical-app` deployment (e.g., by running a stress test or configuring a pod with artificial resource limits that are quickly exceeded).
• Observe Incident: Within a few minutes, you should see an incident triggered in Datadog, which then creates an incident in PagerDuty, notifying the specified on-call personnel.
• Resolve: Once the issue is mitigated, confirm that the Datadog monitor resolves, and the PagerDuty incident is automatically resolved or can be manually resolved.

Troubleshooting Common Issues

• No metrics in Datadog: Verify Datadog Agent deployment, API/APP keys, and Prometheus scraping configuration. Check agent logs for errors.
• Datadog monitor not triggering: Double-check the monitor query syntax and thresholds. Ensure the metric name is correct in Datadog.
• PagerDuty incident not created: Verify the @pagerduty-SERVICE_NAME in the Datadog monitor message matches your PagerDuty service name exactly. Check Datadog event logs for PagerDuty integration errors.
• Terraform authentication issues: Ensure your API tokens for Datadog and PagerDuty are correct and have the necessary permissions. Use environment variables for sensitive data.
• Escalation Policy not working: Confirm the users/schedules in your PagerDuty escalation policy are valid and have contact methods configured.

Conclusion

By leveraging Terraform, Datadog, Prometheus, and PagerDuty, you've established a powerful, automated incident management system for your AWS EKS clusters. This IaC approach ensures consistency, reduces manual errors, and accelerates your team's ability to respond to critical alerts. As your EKS environment scales, this foundation will prove invaluable for maintaining operational excellence and minimizing downtime.