Automated Incident Response for AWS EKS with Datadog and PagerDuty using Terraform

Automated Incident Response for AWS EKS with Datadog and PagerDuty using Terraform

In the fast-paced world of cloud-native applications, managing incidents efficiently is paramount. For environments running on AWS Elastic Kubernetes Service (EKS), manual incident response can be slow, error-prone, and costly. This comprehensive guide details how to establish a robust, automated incident response system for AWS EKS by integrating Datadog for monitoring and alerting, PagerDuty for incident management and on-call routing, all orchestrated with Terraform for Infrastructure as Code (IaC).

Architecture Pro-Tip

For mission-critical EKS clusters, always design your incident response architecture with a 'shift-left' mindset. Automate detection as close to the source as possible (Datadog agents on EKS nodes/pods), and integrate with a centralized incident management platform (PagerDuty) that can trigger automated remediation runbooks or escalation paths. Leveraging Terraform ensures that this entire setup is repeatable, auditable, and version-controlled, minimizing configuration drift and enabling rapid disaster recovery of your observability stack.

Why Automate Incident Response for EKS?

The complexity of Kubernetes clusters demands proactive and automated incident management. Manual processes lead to:

• Delayed Response: Slower detection and notification result in longer mean time to resolution (MTTR).
• Human Error: Manual configuration and notification can lead to missed alerts or incorrect escalations.
• Operational Burnout: On-call teams become overwhelmed by noisy, unactionable alerts.
• Inconsistent Workflows: Lack of standardized procedures across incidents.

By automating, you gain:

• Faster MTTR: Rapid detection and automated notification to the right team.
• Reduced Toil: Less manual intervention for alert routing and escalation.
• Improved Reliability: Consistent and predictable incident handling.
• Better On-Call Experience: More actionable alerts and fewer false positives.

Core Components Overview

AWS EKS: The Foundation

Amazon Elastic Kubernetes Service (EKS) provides a managed Kubernetes control plane. It's where your containerized applications run, making it the primary source of operational metrics, logs, and events that need monitoring.

Datadog: Unified Observability and Alerting

Datadog offers a comprehensive monitoring platform that aggregates metrics, logs, and traces from your EKS clusters. It excels at:

• EKS-specific Monitoring: Collecting data from Kubernetes API server, Kubelet, control plane, pods, and nodes.
• Advanced Alerting: Creating sophisticated monitors based on various data types with robust notification capabilities.
• Contextual Dashboards: Visualizing EKS health and performance.

PagerDuty: Intelligent Incident Management

PagerDuty is a leading incident management platform that streamlines on-call scheduling, alerting, and escalation. Key features include:

• On-Call Management: Dynamic scheduling and automated escalation policies.
• Incident Routing: Directing alerts to the correct team or individual based on pre-defined rules.
• Communication: Facilitating rapid collaboration during incidents via multiple channels.
• Post-Mortem Analysis: Tools for reviewing and improving incident response.

Terraform: Infrastructure as Code (IaC)

Terraform by HashiCorp allows you to define and provision infrastructure using a declarative configuration language. For this setup, Terraform will be used to:

• Manage PagerDuty Services: Define services, escalation policies, and integration keys.
• Configure Datadog Monitors: Create and manage alert conditions, thresholds, and notification settings.
• Integrate Services: Programmatically connect Datadog alerts to PagerDuty services.

The Automated Incident Response Flow

1. EKS Monitoring: Datadog Agents deployed within your EKS cluster collect metrics, logs, and events from pods, nodes, and the control plane.
2. Alert Generation: Datadog Monitors are configured to detect anomalies or breaches of predefined thresholds (e.g., high CPU utilization, pod crashes, OOMKilled events).
3. Incident Creation: When a Datadog monitor triggers, it sends an event to a pre-configured PagerDuty integration key.
4. On-Call Notification: PagerDuty receives the event, creates an incident, and notifies the appropriate on-call team members via their preferred communication channels (phone, SMS, email, push notification) according to escalation policies.
5. Resolution/Escalation: The on-call team acknowledges and resolves the incident. If not acknowledged within a specified time, PagerDuty automatically escalates to the next tier.

Prerequisites

• An AWS Account with an existing EKS Cluster.
• A Datadog Account with API and Application Keys.
• A PagerDuty Account with API Keys.
• Terraform CLI installed (v1.0+ recommended).
• Basic understanding of Terraform, EKS, Datadog, and PagerDuty concepts.

Step-by-Step Implementation Guide with Terraform

1. Setting up PagerDuty Services and Integration

First, we'll define a PagerDuty service and an integration for Datadog using Terraform. This service will be the target for alerts from Datadog.

2. Configuring Datadog for EKS Monitoring and Alerting

Ensure your Datadog Agent is deployed on your EKS cluster. The official Datadog Kubernetes documentation provides Helm charts for easy deployment. For our automation, we'll focus on creating Datadog monitors.

3. Integrating Datadog with PagerDuty

The integration between Datadog and PagerDuty is established by linking a Datadog monitor's notification channel to the PagerDuty integration key generated in step 1. When a monitor triggers, Datadog sends a POST request with incident details to PagerDuty's API endpoint associated with that key.

Terraform Configuration Example

Here's a complete Terraform configuration that defines a PagerDuty service, an escalation policy, a Datadog-PagerDuty integration, and a Datadog monitor to alert on high EKS node CPU usage.

provider "pagerduty" { token = var.pagerduty_api_token } provider "datadog" { api_key = var.datadog_api_key app_key = var.datadog_app_key } # Define a PagerDuty User resource "pagerduty_user" "devops_engineer" { name = "DevOps Engineer" email = var.pagerduty_engineer_email role = "user" } # Define a PagerDuty Team (optional, but good practice) resource "pagerduty_team" "devops_team" { name = "EKS Ops Team" description = "Team responsible for EKS cluster operations" } # Add user to the team resource "pagerduty_team_membership" "devops_team_member" { team_id = pagerduty_team.devops_team.id user_id = pagerduty_user.devops_engineer.id } # Define a PagerDuty Escalation Policy resource "pagerduty_escalation_policy" "eks_critical_policy" { name = "EKS Critical Policy" num_loops = 2 team_id = pagerduty_team.devops_team.id rule { escalation_delay_in_minutes = 15 target { type = "user" id = pagerduty_user.devops_engineer.id } } } # Define a PagerDuty Service resource "pagerduty_service" "eks_monitoring_service" { name = "EKS Cluster Monitoring" auto_resolve_timeout = "14400" # 4 hours acknowledgement_timeout = "600" # 10 minutes escalation_policy = pagerduty_escalation_policy.eks_critical_policy.id description = "Service for critical EKS cluster alerts from Datadog." team = pagerduty_team.devops_team.id } # Define the PagerDuty Integration for Datadog resource "pagerduty_service_integration" "datadog_integration" { name = "Datadog Integration" service_id = pagerduty_service.eks_monitoring_service.id type = "generic_events_api_inbound_integration" } # Create a Datadog Integration for PagerDuty # This is a global integration in Datadog, not service-specific for Terraform setup, # but can be referenced by monitors. # The PagerDuty provider integration key is used directly in the Datadog monitor. # Define a Datadog Monitor for EKS Node CPU usage resource "datadog_monitor" "eks_high_cpu_node_alert" { name = "[EKS] High Node CPU Usage Alert for {{host.name}}" type = "metric alert" query = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster_name:your-eks-cluster-name} by {host} < 20" # Less than 20% idle means > 80% usage message = <<EOT EKS node {{host.name}} in cluster `your-eks-cluster-name` has high CPU utilization (over 80%) for the last 5 minutes. @pagerduty-${pagerduty_service.eks_monitoring_service.name} @slack-your-ops-channel **Impact:** Potential performance degradation or unresponsiveness of applications on this node. **Action:** Investigate running pods, resource limits, and consider scaling. EOT tags = ["environment:production", "service:eks", "severity:high"] monitor_threshold_windows { recovery_window = "15m" } renotify_interval = 60 no_data_timeframe = 20 include_tags = true require_full_window = true force_delete = true restricted_roles = [] priority = 1 # PagerDuty uses this as severity if mapped # PagerDuty integration is specified directly in the message field using @pagerduty-service-name # The service_name comes from the PagerDuty service created above. } output "pagerduty_service_url" { description = "URL of the PagerDuty service" value = pagerduty_service.eks_monitoring_service.html_url } output "pagerduty_integration_key" { description = "Integration key for Datadog to send events to PagerDuty" value = pagerduty_service_integration.datadog_integration.integration_key sensitive = true } output "datadog_monitor_url" { description = "URL of the Datadog monitor" value = datadog_monitor.eks_high_cpu_node_alert.url }

Variables (`variables.tf`)

variable "pagerduty_api_token" { description = "Your PagerDuty API Token" type = string sensitive = true } variable "datadog_api_key" { description = "Your Datadog API Key" type = string sensitive = true } variable "datadog_app_key" { description = "Your Datadog Application Key" type = string sensitive = true } variable "pagerduty_engineer_email" { description = "Email of the DevOps Engineer to add to PagerDuty" type = string } # Ensure to replace 'your-eks-cluster-name' in the datadog_monitor query with your actual cluster name. # You might also want to set your PagerDuty service name.

Deployment Steps:

1. Save the above code into `main.tf` and `variables.tf` files in an empty directory.
2. Initialize Terraform: terraform init
3. Review the plan: terraform plan -var="pagerduty_api_token=PD_TOKEN" -var="datadog_api_key=DD_API_KEY" -var="datadog_app_key=DD_APP_KEY" -var="pagerduty_engineer_email=your-email@example.com" (Replace placeholders with your actual keys and email).
4. Apply the configuration: terraform apply -var="pagerduty_api_token=PD_TOKEN" -var="datadog_api_key=DD_API_KEY" -var="datadog_app_key=DD_APP_KEY" -var="pagerduty_engineer_email=your-email@example.com"

Testing and Validation

After applying the Terraform configuration:

1. Verify PagerDuty Setup: Log into your PagerDuty account. You should see the new user, team, escalation policy, and service (`EKS Cluster Monitoring`) created. The Datadog integration should be listed under the service's integrations.
2. Verify Datadog Monitor: Log into your Datadog account. Navigate to Monitors -> Monitor Status. You should find the `[EKS] High Node CPU Usage Alert` monitor.
3. Trigger an Alert: To test the full flow, you'll need to simulate high CPU usage on an EKS node in the specified cluster. You can do this by running a CPU-intensive workload on a pod or by temporarily adjusting resource limits to force contention.
4. Check PagerDuty Incident: Once the CPU threshold is breached, Datadog should trigger the monitor, and a new incident should appear in PagerDuty, notifying the `DevOps Engineer` based on the escalation policy.

Troubleshooting Common Issues

• API Key Issues: Double-check that your Datadog API/Application keys and PagerDuty API token are correct and have the necessary permissions. Store them securely, preferably using a secrets manager like AWS Secrets Manager or HashiCorp Vault, and pass them as environment variables or via `terraform.tfvars`.
• Monitor Query Incorrect: Ensure your Datadog monitor query is correctly targeting your EKS cluster and metrics. Check if `kubernetes_cluster_name:your-eks-cluster-name` matches your actual cluster tag in Datadog.
• PagerDuty Service Name Mismatch: The Datadog monitor message references the PagerDuty service name using `@pagerduty-service-name`. Make sure the service name in the message matches the actual name of the PagerDuty service created.
• Datadog Agent Health: Verify the Datadog Agent is running correctly on your EKS nodes and successfully sending metrics to Datadog. Check agent logs for errors.
• Timeframes and Thresholds: Adjust Datadog monitor thresholds and timeframes to reduce flapping or false positives.

Conclusion

Automating incident response for AWS EKS using Datadog, PagerDuty, and Terraform significantly enhances your operational resilience and reduces the burden on your on-call teams. By treating your observability and incident management configuration as code, you gain consistency, repeatability, and the ability to quickly adapt to changing infrastructure needs. This setup provides a powerful foundation for proactive problem-solving, enabling your teams to focus on innovation rather than constantly reacting to unforeseen issues.