Terraform-driven PagerDuty Incident Management for AWS EKS with Datadog Anomaly Detection

In the dynamic world of cloud-native applications, maintaining robust uptime and quick incident resolution is paramount. For organizations leveraging AWS EKS (Elastic Kubernetes Service), managing incidents effectively often involves a complex interplay of monitoring tools, alerting systems, and on-call schedules. This comprehensive guide details how to establish a resilient, automated incident management workflow for your AWS EKS clusters using Terraform Infrastructure as Code (IaC), integrated with Datadog for advanced anomaly detection and PagerDuty for streamlined incident response.

By codifying your incident management setup, you gain version control, auditability, and the ability to replicate environments with ease, significantly reducing human error and accelerating deployment. This approach transforms reactive problem-solving into proactive, automated resilience.

Architecture Pro-Tip:

Always define your incident management tooling (PagerDuty services, escalation policies, Datadog monitors) as code. This ensures consistency across environments, facilitates disaster recovery, and allows for rapid iteration. Treat your monitoring and alerting infrastructure with the same rigor as your application code to achieve true GitOps maturity.

The Power of Integrated Incident Management

Modern cloud infrastructure demands a tightly integrated ecosystem for observability and incident response. This solution brings together best-in-class tools to provide a seamless experience:

• AWS EKS: The foundation for running highly available and scalable Kubernetes clusters.
• Datadog: A unified monitoring platform providing deep visibility into EKS performance, metrics, logs, traces, and critical anomaly detection capabilities to identify issues before they become outages.
• PagerDuty: The industry standard for intelligent incident management, on-call scheduling, and automated alerting, ensuring the right person is notified at the right time.
• Terraform: The IaC tool that orchestrates the entire setup, defining PagerDuty services, escalation policies, and Datadog monitors in a declarative manner.

Why Automate with Terraform?

Automating your incident management setup with Terraform offers significant advantages:

• Consistency: Ensure uniform configuration across multiple EKS clusters or environments.
• Version Control: Track changes, roll back configurations, and collaborate on incident response definitions using Git.
• Scalability: Easily scale your incident management capabilities as your EKS footprint grows.
• Reduced Manual Error: Eliminate the risk of misconfigurations common with manual setup.
• Auditability: Maintain a clear audit trail of all changes to your alerting infrastructure.

Prerequisites

Before diving into the configuration, ensure you have the following in place:

• An active AWS account with an existing EKS cluster or the ability to create one.
• A Datadog account with an API key and Application key. Ensure the Datadog Agent is deployed on your EKS cluster to collect metrics, logs, and traces.
• A PagerDuty account with API access enabled. You'll need an API key.
• Terraform CLI installed (version 1.0+ recommended).
• AWS CLI configured with appropriate credentials.
• Basic understanding of Terraform, AWS EKS, Datadog, and PagerDuty concepts.

Step-by-Step Implementation Guide

1. Configure Datadog for EKS Monitoring

Ensure your Datadog Agent is collecting comprehensive data from your EKS cluster. This typically involves deploying the Datadog Agent as a DaemonSet within your Kubernetes cluster. For anomaly detection, Datadog needs a history of metrics.

Refer to the official Datadog Kubernetes Integration documentation for detailed installation instructions. Verify that metrics like `kubernetes.cpu.usage.total`, `kubernetes.memory.usage`, `kubernetes.pod.status`, and `kubernetes.network.bytes_transmitted` are flowing into Datadog.

2. Integrate Datadog with PagerDuty

Before using Terraform, set up the initial integration in PagerDuty. This involves creating a PagerDuty service that Datadog will send alerts to.

1. In PagerDuty, navigate to Services > Service Directory and create a new service (e.g., "EKS Critical Alerts").
2. Add an Integration of type "Datadog". This will generate an Integration Key. Keep this key handy.
3. In Datadog, go to Integrations > Integrations, search for "PagerDuty", and click "Install".
4. Add a new configuration, providing the PagerDuty Integration Key obtained in step 2. Give it a descriptive name.

This establishes the communication channel. Terraform will then manage the PagerDuty services and Datadog monitors.

3. Define PagerDuty Resources with Terraform

We'll use Terraform to define PagerDuty users, escalation policies, and services. This ensures that your on-call rotations and alerting rules are version-controlled.

First, set up your Terraform providers for PagerDuty and Datadog:

provider "pagerduty" { api_token = var.pagerduty_api_token } provider "datadog" { api_key = var.datadog_api_key app_key = var.datadog_app_key }

Next, define users and an escalation policy:

resource "pagerduty_user" "devops_engineer_one" { name = "DevOps Engineer One" email = "devops.one@example.com" role = "user" } resource "pagerduty_user" "devops_engineer_two" { name = "DevOps Engineer Two" email = "devops.two@example.com" role = "user" } resource "pagerduty_escalation_policy" "eks_critical_policy" { name = "EKS Critical Escalation Policy" num_loops = 2 rule { escalation_delay_in_minutes = 10 target { type = "user" id = pagerduty_user.devops_engineer_one.id } } rule { escalation_delay_in_minutes = 20 target { type = "user" id = pagerduty_user.devops_engineer_two.id } } } resource "pagerduty_service" "eks_monitoring_service" { name = "EKS Monitoring Service" auto_resolve_timeout_minutes = 60 acknowledgement_timeout_minutes = 30 escalation_policy = pagerduty_escalation_policy.eks_critical_policy.id }

4. Create Datadog Monitors with Anomaly Detection via Terraform

Now, let's define Datadog monitors. We'll focus on a critical EKS metric like CPU usage and leverage Datadog's anomaly detection. When an anomaly is detected, it will trigger an incident in the PagerDuty service we just created.

Ready-to-Use Terraform Configuration Example

This example sets up a Datadog monitor for EKS CPU usage anomaly detection, linking it to the PagerDuty service created earlier. It includes variables for sensitive API keys.

# main.tf variable "pagerduty_api_token" { description = "PagerDuty API Token" type = string sensitive = true } variable "datadog_api_key" { description = "Datadog API Key" type = string sensitive = true } variable "datadog_app_key" { description = "Datadog Application Key" type = string sensitive = true } variable "datadog_pagerduty_integration_name" { description = "Name of the PagerDuty integration in Datadog (e.g., 'PagerDuty-EKS')" type = string } provider "pagerduty" { api_token = var.pagerduty_api_token } provider "datadog" { api_key = var.datadog_api_key app_key = var.datadog_app_key } # PagerDuty Resources resource "pagerduty_user" "devops_engineer_one" { name = "DevOps Engineer One" email = "devops.one@example.com" role = "user" } resource "pagerduty_user" "devops_engineer_two" { name = "DevOps Engineer Two" email = "devops.two@example.com" role = "user" } resource "pagerduty_escalation_policy" "eks_critical_policy" { name = "EKS Critical Escalation Policy" num_loops = 2 rule { escalation_delay_in_minutes = 10 target { type = "user" id = pagerduty_user.devops_engineer_one.id } } rule { escalation_delay_in_minutes = 20 target { type = "user" id = pagerduty_user.devops_engineer_two.id } } } resource "pagerduty_service" "eks_monitoring_service" { name = "EKS Monitoring Service" auto_resolve_timeout_minutes = 60 acknowledgement_timeout_minutes = 30 escalation_policy = pagerduty_escalation_policy.eks_critical_policy.id description = "Service for critical EKS alerts from Datadog." } # Datadog Monitor for EKS CPU Anomaly Detection resource "datadog_monitor" "eks_cpu_anomaly" { name = "[EKS] CPU Usage Anomaly Detected for {{host.name}}" type = "metric alert" query = "anomalies(avg:kubernetes.cpu.usage.total{*} by {kube_cluster_name, host}, 'aggressiveness':'high', 2, queries('avg:kubernetes.cpu.usage.total{*} by {kube_cluster_name, host}').last(5m)) >= 1" message = <<EOT EKS CPU Usage Anomaly Detected! Cluster: {{kube_cluster_name.name}} Host: {{host.name}} Current Usage: {{value}}% Please investigate immediately. @pagerduty-${var.datadog_pagerduty_integration_name} EOT tags = ["environment:production", "eks", "cpu-anomaly", "incident-management"] monitor_threshold_windows { recovery_window = "last_5m" } notify_no_data = false new_group_delay = 60 new_host_delay = 300 no_data_timeframe = 20 renotify_interval = 0 escalation_message = "Critical CPU anomaly persists on EKS host. Escalating to next level." include_tags = true require_full_window = true }

To deploy this configuration:

1. Save the code as `main.tf` in an empty directory.
2. Create a `terraform.tfvars` file (or use environment variables) to provide the sensitive API keys and integration name:

# terraform.tfvars pagerduty_api_token = "YOUR_PAGERDUTY_API_TOKEN" datadog_api_key = "YOUR_DATADOG_API_KEY" datadog_app_key = "YOUR_DATADOG_APP_KEY" datadog_pagerduty_integration_name = "PagerDuty-EKS" # Use the name configured in Datadog for PagerDuty integration

3. Run `terraform init` to initialize the providers.
4. Run `terraform plan` to review the changes.
5. Run `terraform apply` to create the PagerDuty services and Datadog monitors.

Testing and Validation

After applying the Terraform configuration:

1. Verify PagerDuty: Log into your PagerDuty account. Confirm that the `EKS Monitoring Service`, `EKS Critical Escalation Policy`, and the defined users exist.
2. Verify Datadog: Log into your Datadog account. Navigate to Monitors and confirm that the `[EKS] CPU Usage Anomaly Detected for {{host.name}}` monitor is present and configured correctly.
3. Simulate an Anomaly: This can be challenging with anomaly detection. You might need to artificially generate high CPU load on an EKS node or pod for a sustained period to trigger the monitor. Observe if a Datadog alert is generated and subsequently if a PagerDuty incident is created and escalated according to your policy.

Advanced Scenarios and Best Practices

Centralized Incident Management

For multiple EKS clusters, consider a centralized Terraform repository to manage all PagerDuty and Datadog resources. Use Terraform workspaces or a structured folder approach to separate configurations for different environments (e.g., `prod`, `dev`).

Runbook Automation

Enhance your PagerDuty service by linking detailed runbooks to common incidents. In the `message` field of your Datadog monitor, include direct links to internal documentation or troubleshooting guides. This empowers your on-call team to resolve issues faster.

Granular Alerting

Beyond CPU anomalies, consider other critical EKS metrics and logs for anomaly detection:

• Memory Usage: Detect unusual spikes or sustained high memory consumption.
• Network Traffic: Identify sudden drops or increases in network I/O.
• Pod Restarts/Failures: Monitor `kubernetes.pod.status` and `kubernetes.containers.state.waiting` for unusual patterns.
• Application Logs: Leverage Datadog Log Management with anomaly detection for error rates or specific log patterns.

Tagging Strategy

Implement a consistent tagging strategy across EKS, Datadog, and PagerDuty. Tags allow for better filtering, organization, and correlation of incidents, making troubleshooting more efficient.

Troubleshooting and FAQ

Q: My Terraform `apply` failed for PagerDuty/Datadog resources. What should I check?

A: Double-check your API tokens and keys. Ensure they have the necessary permissions. For PagerDuty, the API token needs full access for creating services and policies. For Datadog, both the API key and Application key are required, and they need permissions to create and manage monitors. Also, ensure the `datadog_pagerduty_integration_name` variable exactly matches the name you configured in Datadog's PagerDuty integration settings.

Q: Datadog isn't reporting any EKS metrics.

A: Verify the Datadog Agent is correctly deployed as a DaemonSet on your EKS cluster and that it has the correct API key configured. Check the Agent's logs for errors. Ensure the necessary Kubernetes roles and service accounts are configured to allow the Agent to access Kubernetes API resources.

Q: My Datadog monitor isn't triggering PagerDuty incidents.

A:

1. Ensure the Datadog monitor query is correct and that the underlying metric data is being collected.
2. Check the notification section of the Datadog monitor. The `@pagerduty-${var.datadog_pagerduty_integration_name}` syntax is crucial for directing alerts to PagerDuty.
3. Verify the Datadog-PagerDuty integration is healthy in both platforms. Sometimes re-generating the PagerDuty integration key and updating it in Datadog can resolve connectivity issues.
4. Ensure your PagerDuty service has an active on-call schedule and escalation policy that includes contact methods.

Q: How can I manage on-call rotations with Terraform?

A: While this guide focuses on services and policies, the PagerDuty Terraform provider also supports managing `pagerduty_team`, `pagerduty_schedule`, and `pagerduty_rotation` resources. You can define your on-call schedules, teams, and rotations as code to complete the IaC approach for incident management.

Conclusion

By adopting a Terraform-driven approach for integrating PagerDuty and Datadog with your AWS EKS environment, you move beyond manual configuration to a robust, scalable, and auditable incident management system. Datadog's anomaly detection significantly reduces alert fatigue by identifying genuine deviations, while PagerDuty ensures critical incidents are acted upon swiftly by the right team members.

Embrace Infrastructure as Code for your incident response framework to build more resilient EKS operations and empower your DevOps teams to focus on innovation rather than reactive firefighting.