Terraform-Managed PagerDuty Incident Response from Datadog Alerts in AWS EKS

In the fast-paced world of cloud-native applications, maintaining system reliability and ensuring rapid incident response is paramount. For organizations leveraging AWS EKS for Kubernetes deployments and Datadog for comprehensive monitoring, integrating PagerDuty for incident management becomes a critical piece of the operational puzzle. This guide details how to establish a robust, automated incident response pipeline, entirely managed by Terraform, ensuring consistency, scalability, and auditability.

Architecture Pro-Tip

For mission-critical systems, always design your incident response architecture with redundancy and immutable infrastructure principles. Store your Terraform state securely in a remote backend like AWS S3 with state locking (e.g., DynamoDB) to prevent concurrent modifications and ensure disaster recovery. Furthermore, implement version control for all your Terraform configurations to track changes and facilitate rollbacks, adhering to GitOps best practices for your operational infrastructure.

The Need for Automated Incident Response

Manual intervention in incident management is prone to errors, delays, and inconsistencies. As environments scale, especially with dynamic orchestrators like Kubernetes in AWS EKS, a reactive manual approach becomes unsustainable. Automating the entire process – from alert detection in Datadog to incident creation and routing in PagerDuty – using Infrastructure as Code (IaC) with Terraform offers significant advantages:

Consistency: Ensure all alerts follow predefined escalation paths.
Speed: Reduce mean time to detect (MTTD) and mean time to resolve (MTTR).
Auditability: Track all changes to incident policies and integrations via version control.
Scalability: Easily apply the same configurations across multiple services or environments.
Reduced Human Error: Eliminate manual configuration mistakes.

Core Components of Our Solution

This guide integrates three powerful platforms to create a seamless incident response workflow:

AWS EKS (Amazon Elastic Kubernetes Service)

Our target environment where containerized applications run. Datadog agents will collect metrics and logs from this Kubernetes cluster.

Datadog

A unified monitoring and analytics platform that provides observability into your EKS clusters, applications, and infrastructure. Datadog will be responsible for detecting anomalies and generating alerts based on predefined monitors.

PagerDuty

An incident management platform that ingests alerts, notifies on-call teams, manages escalations, and facilitates resolution. Datadog alerts will trigger incidents in PagerDuty.

Terraform

Our IaC tool of choice. Terraform will manage the configuration of PagerDuty services and escalation policies, as well as Datadog's integrations and monitors, ensuring that our incident response setup is fully declaratively defined.

Prerequisites

Before you begin, ensure you have the following:

An active AWS account with an existing EKS cluster.
A Datadog account with API and Application keys.
A PagerDuty account with a User Token.
Terraform CLI installed (v1.0.0 or higher).
Basic understanding of Terraform, AWS EKS, Datadog, and PagerDuty concepts.
Datadog Agent deployed on your EKS cluster to collect metrics and logs.

Implementing the Solution with Terraform

This section walks you through the Terraform configurations required to set up the integration. We'll define PagerDuty services and escalation policies, configure the Datadog-PagerDuty integration, and create a sample Datadog monitor for EKS.

1. PagerDuty Provider Configuration

First, configure the PagerDuty provider with your API token. This typically goes into a versions.tf or provider.tf file.

provider "pagerduty" {
  token = var.pagerduty_token
}

terraform {
  required_providers {
    pagerduty = {
      source  = "pagerduty/pagerduty"
      version = "~> 2.0"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
  }
  backend "s3" {
    bucket         = "your-terraform-state-bucket"
    key            = "pagerduty-datadog-eks.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "your-terraform-lock-table"
  }
}
    

2. Datadog Provider Configuration

Similarly, configure the Datadog provider using your API and Application keys.

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}
    

3. PagerDuty Users, Teams, Escalation Policies, and Service

Define your PagerDuty users, teams, escalation policies, and the service that Datadog will trigger incidents against. This example assumes you have existing users. If not, you'd create them with pagerduty_user.

# PagerDuty Team
resource "pagerduty_team" "devops_team" {
  name = "DevOps EKS Operations"
}

# Example PagerDuty User (if not existing)
# resource "pagerduty_user" "oncall_engineer" {
#   name  = "Oncall Engineer"
#   email = "oncall@example.com"
# }

# Escalation Policy
resource "pagerduty_escalation_policy" "eks_critical_policy" {
  name      = "EKS Critical Incident Policy"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 5
    target {
      type = "user"
      id   = "P2M3X5Y" # Replace with actual user ID for Oncall Engineer
    }
  }

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "team"
      id   = pagerduty_team.devops_team.id
    }
  }
}

# PagerDuty Service for EKS alerts
resource "pagerduty_service" "eks_application_service" {
  name                    = "EKS Application Criticals"
  auto_resolve_timeout    = "14400" # 4 hours
  acknowledgement_timeout = "600"   # 10 minutes
  escalation_policy       = pagerduty_escalation_policy.eks_critical_policy.id
}
    

4. Datadog PagerDuty Integration and Monitor

Now, configure the Datadog PagerDuty integration and create a sample monitor that targets your EKS cluster and uses the PagerDuty service you just created.

# Datadog PagerDuty Integration
resource "datadog_integration_pagerduty" "pagerduty_integration" {
  services {
    key = pagerduty_service.eks_application_service.name
    value = pagerduty_service.eks_application_service.id
  }
}

# Datadog Monitor for EKS high CPU utilization
resource "datadog_monitor" "eks_high_cpu" {
  name               = "EKS Cluster High CPU Utilization"
  type               = "metric alert"
  query              = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:your-eks-cluster-name} by {kubernetes_state} > 80"
  message            = "High CPU usage detected in EKS cluster: {{cluster_name.name}}! Please investigate. @pagerduty-EKS Application Criticals"
  tags               = ["environment:production", "service:eks", "severity:critical"]
  priority           = 1
  no_data_timeframe  = 20
  renotify_interval  = 60
  notify_audit       = false
  timeout_h          = 0
  escalation_message = "Still high CPU! Escalating to next level. @pagerduty-EKS Application Criticals"
  require_full_window = false
  new_group_delay    = 300 # 5 minutes

  # PagerDuty integration through the service_name in the message
  # Ensure the service name matches the key used in datadog_integration_pagerduty
  # The `@pagerduty-SERVICE_NAME` syntax directly maps to the PagerDuty service
}
    

5. Variables File (`variables.tf`)

Define your sensitive API keys and other dynamic values as variables.

variable "pagerduty_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "eks_cluster_name" {
  description = "Name of your AWS EKS cluster"
  type        = string
}

# Example of how to pass the PagerDuty user ID if not hardcoded
# variable "oncall_engineer_user_id" {
#   description = "PagerDuty User ID for the Oncall Engineer"
#   type        = string
# }
    

6. Execution

Initialize Terraform, plan your changes, and apply them.

Save the above configurations into .tf files in a directory.
Create a terraform.tfvars file (add to .gitignore!) or use environment variables for sensitive data:
pagerduty_token = "your_pagerduty_api_token" datadog_api_key = "your_datadog_api_key" datadog_app_key = "your_datadog_app_key" eks_cluster_name = "your-eks-cluster-name" # oncall_engineer_user_id = "P2M3X5Y"
Run: terraform init
Run: terraform plan (review the proposed changes)
Run: terraform apply (confirm with 'yes')

This will create the PagerDuty resources and the Datadog integration and monitor. You can verify their creation in the respective Datadog and PagerDuty web UIs.

Testing and Validation

After applying the Terraform configuration, it's crucial to test the entire workflow:

Manual Datadog Alert: In Datadog, manually trigger the EKS high CPU monitor (if possible, or simulate the condition in your EKS cluster).
PagerDuty Incident: Verify that a new incident is created in PagerDuty for the "EKS Application Criticals" service.
Escalation Policy: Ensure the incident follows the defined escalation policy, notifying the correct users and teams.
Resolution: Resolve the incident in PagerDuty and observe if the corresponding Datadog alert state changes (if configured for auto-resolution).

Best Practices for Scalability and Maintenance

Modularize Terraform: As your environment grows, split your Terraform configuration into logical modules (e.g., PagerDuty modules, Datadog monitors module, EKS cluster module).
Naming Conventions: Implement clear and consistent naming conventions for all PagerDuty services, escalation policies, and Datadog monitors.
Tagging: Use tags extensively in Datadog monitors to categorize alerts by environment, service, team, and severity. This aids in filtering and analysis.
Version Control: Keep all your Terraform code in a Git repository. Implement PR reviews for changes to incident response configurations.
Secrets Management: Use a dedicated secrets manager (e.g., AWS Secrets Manager, HashiCorp Vault) for your API tokens instead of .tfvars files, especially in production.
Synthetic Monitoring: Complement metric-based EKS alerts with Datadog Synthetic Monitoring to proactively test user-facing endpoints and services.
Automated Cleanup: For temporary environments, ensure you have Terraform destroy capabilities or scripts to clean up incident response resources when environments are deprovisioned.

Conclusion

By leveraging Terraform to manage your PagerDuty incident response from Datadog alerts in AWS EKS, you're not just automating a process; you're building a resilient, auditable, and scalable operational backbone. This Infrastructure as Code approach ensures that your critical systems are always under vigilant watch, and any issues are promptly addressed by the right teams, minimizing downtime and safeguarding your service level objectives. Embrace automation to elevate your cloud-native incident response to the next level.

CloudStack YAML