Terraform Configuration for Datadog-PagerDuty Incident Management on AWS EKS

In the fast-paced world of cloud-native operations, ensuring high availability and rapid incident response for applications running on AWS EKS is paramount. This comprehensive guide details how to leverage Terraform to seamlessly integrate Datadog for robust monitoring and PagerDuty for efficient incident management, creating a resilient and automated incident response pipeline for your Kubernetes clusters.

Architecture Pro-Tip: Modular Design for Scalability

When building your incident management infrastructure with Terraform, consider a modular design. Separate your Datadog monitors, PagerDuty services, and AWS EKS resources into distinct Terraform modules. This approach enhances reusability, improves readability, and makes it significantly easier to manage configurations across multiple environments (development, staging, production) or different EKS clusters. Always prioritize least privilege principles for API keys and service accounts used by Datadog and PagerDuty to interact with AWS.

Introduction to Automated Incident Management on EKS

Managing incidents in a dynamic Kubernetes environment like AWS EKS requires not only keen observability but also a robust system to notify the right teams at the right time. Manual configuration of monitoring alerts and on-call schedules can be error-prone and time-consuming. By codifying these configurations with Terraform, you gain:

• Consistency: Ensure all alerts and incident rules are applied uniformly across your infrastructure.
• Version Control: Track changes to your incident response logic, enabling rollbacks and auditing.
• Speed: Quickly deploy or modify incident response configurations as your EKS environment evolves.
• Reliability: Reduce human error in critical alerting pathways.

This guide will walk you through setting up Datadog monitors for EKS health, creating PagerDuty services and escalation policies, and connecting them all using Terraform, ensuring that critical alerts from your EKS clusters are promptly escalated to your on-call teams.

Prerequisites

Before you begin, ensure you have the following:

• An AWS Account with administrative access and an existing AWS EKS cluster.
• A Datadog Account with an API Key and Application Key. The Datadog Agent should be deployed to your EKS cluster to collect metrics and logs.
• A PagerDuty Account with an API Key.
• Terraform CLI installed (v1.0.0 or higher recommended).
• AWS CLI installed and configured.
• Basic understanding of Terraform, AWS EKS, Datadog, and PagerDuty concepts.

Step 1: Configure Terraform Providers

First, define the necessary Terraform providers in your versions.tf or main.tf file. This includes AWS, Datadog, and PagerDuty. You'll typically store API keys as environment variables or use a secure secret management solution like AWS Secrets Manager.

Provider Configuration Example

Create a file named providers.tf:

Ready-to-Use Terraform Configuration

terraform { required_providers { aws = { source = "hashicorp/aws" version = "~> 5.0" } datadog = { source = "DataDog/datadog" version = "~> 3.0" } pagerduty = { source = "PagerDuty/pagerduty" version = "~> 2.0" } } } # AWS Provider Configuration provider "aws" { region = "us-east-1" # Replace with your AWS region } # Datadog Provider Configuration # Ensure DATADOG_API_KEY and DATADOG_APP_KEY are set as environment variables provider "datadog" {} # PagerDuty Provider Configuration # Ensure PAGERDUTY_TOKEN is set as an environment variable provider "pagerduty" {} # --- PagerDuty Resources --- # 1. PagerDuty Team (optional, but good for organization) resource "pagerduty_team" "devops_team" { name = "DevOps Incident Response" description = "Team responsible for EKS incident management." } # 2. PagerDuty Escalation Policy resource "pagerduty_escalation_policy" "eks_escalation_policy" { name = "EKS Critical Alert Escalation" team = pagerduty_team.devops_team.id rule { delay_after_incident_minutes = 0 target { type = "user" id = "P012345" # Replace with a valid PagerDuty User ID } } rule { delay_after_incident_minutes = 30 target { type = "schedule" id = "PABCDEF" # Replace with a valid PagerDuty Schedule ID } } description = "Escalates EKS critical incidents from Datadog." } # 3. PagerDuty Service resource "pagerduty_service" "eks_monitoring_service" { name = "EKS Cluster Monitoring" description = "Receives critical alerts from Datadog for EKS cluster health." escalation_policy = pagerduty_escalation_policy.eks_escalation_policy.id auto_resolve_timeout_minutes = 1440 # Auto-resolve after 24 hours if not manually resolved acknowledgement_timeout_minutes = 15 # Acknowledge within 15 minutes team = pagerduty_team.devops_team.id incident_urgency_rule { type = "constant" urgency = "high" } } # 4. PagerDuty Integration (for Datadog) resource "pagerduty_extension" "datadog_integration" { name = "Datadog to EKS Monitoring Service" endpoint_url = "https://datadog.pagerduty.com/integration/create_incident.json" # PagerDuty Datadog V2 Integration URL extension_type = "generic_events" service = pagerduty_service.eks_monitoring_service.id } # --- Datadog Resources --- # 1. Datadog Monitor for EKS Node CPU Utilization resource "datadog_monitor" "eks_node_cpu_monitor" { name = "[EKS] High Node CPU Usage on {{host.name}}" type = "metric alert" query = "avg(last_5m):avg:kubernetes.cpu.usage.total{kubernetes_cluster_name:your-eks-cluster-name} by {host} > 85" # Replace 'your-eks-cluster-name' message = "EKS node {{host.name}} CPU usage is above 85% for 5 minutes. @pagerduty-EKS Cluster Monitoring" # The PagerDuty integration name must match. tags = ["environment:production", "service:eks", "alert-type:cpu"] new_group_delay = 60 new_host_delay = 300 no_data_timeframe = 20 notify_no_data = false renotify_interval = 0 # No re-notification for this example escalation_message = "CPU usage remains high. Investigate node {{host.name}} immediately." # The PagerDuty integration is referenced by its name configured in Datadog. # Ensure you've set up the PagerDuty integration in Datadog UI and named it "EKS Cluster Monitoring" or similar. # Alternatively, you can use the @webhook-PagerDuty URL if you manually configured it in Datadog. # For direct integration through Datadog's built-in PagerDuty integration, # the recipient format is typically @pagerduty- # where SERVICE_NAME_IN_PAGERDUTY_UI is the name of the Datadog integration in PagerDuty (e.g., "Datadog"). # If you create the PagerDuty service via Terraform, you will typically link Datadog to it via the PagerDuty UI # or ensure your Datadog integration within PagerDuty is named correctly. # For this example, we assume a Datadog PagerDuty integration exists in Datadog's integrations list, # and the PagerDuty service created by Terraform is linked to it. # A more robust way might involve PagerDuty's Events API V2. # For the purpose of this guide, the `@pagerduty-EKS Cluster Monitoring` # assumes a PagerDuty integration named "EKS Cluster Monitoring" is configured in Datadog. # The PagerDuty service created by Terraform (eks_monitoring_service) needs to be mapped in Datadog's PagerDuty integration settings. } # 2. Datadog Monitor for EKS Pod Restarts resource "datadog_monitor" "eks_pod_restart_monitor" { name = "[EKS] High Pod Restart Rate on {{kubernetes.pod_name}}" type = "metric alert" query = "sum(last_5m):sum:kubernetes.pod.restarts{kubernetes_cluster_name:your-eks-cluster-name} by {kubernetes_pod_name} > 3" # Replace 'your-eks-cluster-name' message = "Pod {{kubernetes.pod_name}} in namespace {{kubernetes.namespace}} has restarted more than 3 times in 5 minutes. @pagerduty-EKS Cluster Monitoring" tags = ["environment:production", "service:eks", "alert-type:pod-restart"] new_group_delay = 60 new_host_delay = 300 no_data_timeframe = 20 notify_no_data = false renotify_interval = 0 escalation_message = "Pod restart rate remains high. Investigate pod {{kubernetes.pod_name}} immediately." }

Understanding the Configuration

The Terraform configuration above sets up a complete incident management workflow:

• PagerDuty Team: An optional but recommended resource to group related services and escalation policies.
• PagerDuty Escalation Policy: Defines the order and timing of notifications for an incident. In this example, it first notifies a specific user, then escalates to a schedule after 30 minutes. Remember to replace placeholder IDs with your actual PagerDuty User and Schedule IDs.
• PagerDuty Service: This service acts as the endpoint for Datadog alerts. When an alert fires, it creates an incident in this service, triggering the associated escalation policy.
• Datadog Monitors: Two example monitors are provided:
  • A CPU Utilization Monitor for EKS nodes, triggering if average CPU usage exceeds 85% for 5 minutes.
  • A Pod Restart Rate Monitor, alerting if any pod restarts more than 3 times in 5 minutes.
• Integration Point: The key to linking Datadog and PagerDuty is the message field in the Datadog monitor, specifically @pagerduty-EKS Cluster Monitoring. This tag tells Datadog to send the alert to the PagerDuty integration named "EKS Cluster Monitoring" which is configured in your Datadog account. You must ensure that in your Datadog account, under Integrations -> PagerDuty, you have an integration configured that links to the PagerDuty service created by Terraform.

Deployment Steps

Follow these steps to deploy your incident management configuration:

1. Save the Configuration: Create a directory (e.g., terraform-eks-incidents) and save the code above into main.tf, providers.tf, or split as you prefer.
2. Set Environment Variables: Export your API keys as environment variables:
   export DATADOG_API_KEY="your_datadog_api_key" export DATADOG_APP_KEY="your_datadog_app_key" export PAGERDUTY_TOKEN="your_pagerduty_api_token"

   For production environments, consider using Terraform's sensitive data handling with a secrets manager like AWS Secrets Manager or HashiCorp Vault.

3. Initialize Terraform:
   terraform init
4. Review the Plan:
   terraform plan

   Carefully examine the plan to ensure Terraform will create the expected resources without unintended changes.

5. Apply the Configuration:
   terraform apply

   Type yes when prompted to confirm the application.

Verification and Testing

After applying the Terraform configuration:

• Check Datadog: Log in to your Datadog account. Navigate to Monitors -> Manage Monitors. You should see the "EKS Node CPU Usage" and "EKS Pod Restart Rate" monitors listed.
• Check PagerDuty: Log in to your PagerDuty account. Navigate to Services. You should find the "EKS Cluster Monitoring" service. Verify its linked escalation policy and team.
• Trigger a Test Alert: To test the full flow, you can manually trigger one of the Datadog monitors (if possible, by artificially increasing CPU load on an EKS node, or forcing a pod restart). Alternatively, Datadog allows you to test notification channels.
• Verify Incident Creation: Upon a triggered alert, an incident should be created in the "EKS Cluster Monitoring" service in PagerDuty, and your on-call team should be notified according to the escalation policy.

Troubleshooting and Best Practices

Common Issues

• API Key/Token Errors: Double-check your environment variables for typos or incorrect keys. Ensure the keys have the necessary permissions.
• Datadog Monitor Query: Ensure the kubernetes_cluster_name in the Datadog monitor query matches your actual EKS cluster name. Verify that the Datadog Agent is correctly collecting metrics from your EKS cluster.
• PagerDuty IDs: Make sure the PagerDuty User ID and Schedule ID in the escalation policy are correct and belong to your PagerDuty account.
• Datadog-PagerDuty Integration Name: The @pagerduty-EKS Cluster Monitoring tag in the Datadog message must precisely match the name of the Datadog integration configured within PagerDuty or the PagerDuty integration within Datadog that points to your service. It's crucial to map the Datadog monitor to the correct PagerDuty service via Datadog's built-in PagerDuty integration settings.

Best Practices

• GitOps Workflow: Store your Terraform configuration in a Git repository and use a CI/CD pipeline (e.g., GitHub Actions, GitLab CI, Jenkins) to automate terraform plan and terraform apply. This ensures all changes are reviewed and version-controlled.
• State Management: Use a remote backend for your Terraform state (e.g., AWS S3 with DynamoDB locking) to enable collaboration and prevent state corruption.
• Environment Variables for Secrets: Never hardcode API keys or tokens directly in your Terraform files. Use environment variables, a .tfvars file (with caution and `.gitignore`), or integrate with a dedicated secret management service.
• Granular Monitoring: Expand your Datadog monitors beyond basic CPU and restarts. Consider memory usage, network latency, application-specific metrics, and custom metrics for critical services running on EKS.
• Runbook Automation: Enhance your PagerDuty services with runbooks that provide clear, actionable steps for resolving common incidents, reducing resolution time.
• Testing in Staging: Always deploy and test incident management configurations in a non-production environment first.

Conclusion

By adopting Terraform for managing your Datadog and PagerDuty integrations on AWS EKS, you establish a robust, scalable, and automated incident management framework. This not only streamlines your DevOps operations but also significantly improves your team's ability to respond to and resolve critical issues, minimizing downtime and safeguarding the reliability of your cloud-native applications. Embrace Infrastructure as Code for every aspect of your observability and incident response to build a truly resilient system.