Terraform for Datadog Monitors and PagerDuty Incident Response on AWS EKS Workloads

In the dynamic world of cloud-native applications, maintaining robust observability and rapid incident response is paramount, especially for complex environments like AWS EKS (Elastic Kubernetes Service). This guide details how to leverage Terraform, the industry-leading Infrastructure as Code (IaC) tool, to programmatically define, deploy, and manage Datadog monitors and integrate them seamlessly with PagerDuty for streamlined incident response workflows on your EKS clusters.

Architecture Pro-Tip:

Always encapsulate your monitoring and alerting configurations in reusable Terraform modules. This promotes consistency across multiple EKS clusters or services, reduces configuration drift, and significantly accelerates new service onboarding. Centralize API keys securely using a secret management solution like AWS Secrets Manager or HashiCorp Vault, referenced by your Terraform configurations, rather than hardcoding them.

Why Infrastructure as Code for Observability?

Managing monitoring and alerting configurations manually, especially at scale, is prone to errors, inconsistencies, and becomes a significant operational overhead. Terraform addresses these challenges by providing:

Version Control: Treat your monitoring configurations like application code, tracking changes, enabling rollbacks, and facilitating collaborative development through Git.
Consistency: Ensure every EKS cluster or service adheres to predefined monitoring standards, preventing alert blind spots.
Auditability: Every change to a monitor or alert policy is logged and reviewable, enhancing compliance and security.
Scalability: Easily replicate monitoring setups across new environments or services with minimal effort.
Reduced Toil: Automate the creation and modification of monitors, freeing up valuable DevOps time.

Prerequisites

Before you begin, ensure you have the following:

An active AWS Account with an existing EKS Cluster.
A Datadog Account with appropriate API and Application Keys.
A PagerDuty Account with an existing service or the ability to create new ones.
Terraform CLI (v1.0+) installed on your local machine.
AWS CLI configured with credentials to manage EKS.

Setting up Datadog and PagerDuty Integration

Datadog API and Application Keys

You'll need your Datadog API Key and Application Key for Terraform to authenticate with the Datadog API. Find these under Organization Settings -> API Keys in the Datadog UI.

Integrating PagerDuty with Datadog

Within Datadog, navigate to Integrations -> PagerDuty. Here you'll set up the global integration. You can connect it to an existing PagerDuty service by providing its API key, or use Datadog to create a new service. For full IaC, we'll define PagerDuty services directly in Terraform.

Terraform Provider Configuration

First, define the necessary providers in your `versions.tf` or `main.tf` file. You'll need the `datadog` and `pagerduty` providers.

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_token
}
    

It's crucial to manage your API keys and tokens securely, typically through environment variables or a secrets management service, rather than hardcoding them in your Terraform files.

Defining PagerDuty Services and Escalation Policies with Terraform

Before creating Datadog monitors that alert PagerDuty, let's define the PagerDuty components using Terraform. This ensures your incident response structure is also version-controlled.

PagerDuty Escalation Policy

An escalation policy defines who gets alerted and in what sequence.

resource "pagerduty_escalation_policy" "eks_critical_policy" {
  name      = "EKS Critical Incident Policy"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "user"
      id   = "PAG000000000000000000000000000000" # Replace with your PagerDuty User ID
    }
  }

  rule {
    escalation_delay_in_minutes = 15
    target {
      type = "schedule"
      id   = "PAS000000000000000000000000000000" # Replace with your PagerDuty Schedule ID
    }
  }

  # Add more rules for additional escalation steps if needed
}
    

Note: You'll need the PagerDuty User IDs and Schedule IDs. These can be found in the PagerDuty UI or retrieved using the `pagerduty_user` and `pagerduty_schedule` data sources in Terraform if they are managed outside this configuration.

PagerDuty Service

A service represents a component of your infrastructure or application that PagerDuty monitors. Incidents are routed to these services.

resource "pagerduty_service" "eks_critical_service" {
  name                    = "EKS Critical Workloads"
  auto_resolve_timeout_s  = 14400 # 4 hours
  acknowledgement_timeout_s = 600   # 10 minutes
  escalation_policy       = pagerduty_escalation_policy.eks_critical_policy.id

  incident_urgency_rule {
    type = "constant"
    urgency = "high"
  }

  alert_creation_limited_to_integrations = true
}

resource "pagerduty_service_integration" "datadog_integration" {
  name    = "Datadog"
  service = pagerduty_service.eks_critical_service.id
  type    = "datadog_inbound_integration"
}
    

The `pagerduty_service_integration` resource creates the necessary integration endpoint that Datadog will use to send alerts.

Defining Datadog Monitors with Terraform for EKS

Now we can define Datadog monitors that specifically target EKS workloads. We'll use the `datadog_monitor` resource.

Example 1: EKS Node CPU Utilization Monitor

This monitor will trigger if any EKS node's CPU utilization exceeds 90% for 5 minutes, alerting the PagerDuty service we just created.

Ready-to-Use Terraform Configuration

# main.tf
resource "datadog_monitor" "eks_node_cpu_high" {
  name                = "[EKS] High Node CPU Utilization on {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:system.cpu.idle{eks_cluster_name:{{var.eks_cluster_name}}} by {host} < 10"
  message             = "EKS node {{host.name}} CPU utilization is above 90%. {{viz.url}} @pagerduty-{{pagerduty_service.eks_critical_service.name}}"
  tags                = ["environment:production", "service:eks", "alert_type:critical", "cluster:${var.eks_cluster_name}"]
  group_by            = ["host"]
  thresholds {
    critical = 90
    warning  = 80
  }
  
  renotify_interval         = 60
  notify_no_data            = false
  no_data_timeframe         = 20
  include_tags              = true
  require_full_window       = true
  new_group_delay           = 300 # seconds
  new_host_delay            = 300 # seconds
  notify_audit              = false
  timeout_h                 = 0
  escalation_message        = "EKS node CPU remains high after initial alert."
}

# Example 2: EKS Pod Restarts Monitor
resource "datadog_monitor" "eks_pod_restarts" {
  name                = "[EKS] Excessive Pod Restarts in {{kube_namespace}} on {{kube_container_name}}"
  type                = "metric alert"
  query               = "sum(last_5m):kubernetes.pod.restarts{eks_cluster_name:{{var.eks_cluster_name}}} by {kube_namespace,kube_container_name} > 5"
  message             = "Pod {{kube_container_name}} in namespace {{kube_namespace}} is restarting excessively. {{viz.url}} @pagerduty-{{pagerduty_service.eks_critical_service.name}}"
  tags                = ["environment:production", "service:eks", "alert_type:warning", "cluster:${var.eks_cluster_name}"]
  group_by            = ["kube_namespace", "kube_container_name"]
  thresholds {
    critical = 5
  }

  renotify_interval         = 30
  notify_no_data            = false
  no_data_timeframe         = 20
  include_tags              = true
  require_full_window       = false
  notify_audit              = false
  timeout_h                 = 0
}

# variables.tf
variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

variable "eks_cluster_name" {
  description = "Name of the AWS EKS cluster to monitor"
  type        = string
}

# outputs.tf
output "datadog_monitor_cpu_id" {
  description = "The ID of the Datadog EKS Node CPU monitor"
  value       = datadog_monitor.eks_node_cpu_high.id
}

output "pagerduty_service_name" {
  description = "The name of the PagerDuty service created"
  value       = pagerduty_service.eks_critical_service.name
}
    

Understanding the Datadog Monitor Attributes

`name`: A descriptive name for your monitor. Template variables (like `{{host.name}}`) can make names dynamic.
`type`: The type of monitor (e.g., "metric alert", "query alert", "log alert").
`query`: The Datadog query that defines the condition for the alert. This is where you specify your EKS metrics.
`message`: The notification message. Crucially, use `@pagerduty-servicename` to route alerts to PagerDuty. The `pagerduty_service.eks_critical_service.name` will output the exact service name required.
`tags`: Metadata tags for filtering and organization within Datadog. Highly recommended for EKS to include cluster name, namespace, service, etc.
`thresholds`: Defines the critical and optional warning thresholds for your monitor.
`renotify_interval`: How often Datadog should re-notify if the alert condition persists.
`notify_no_data`: Whether to alert if data stops flowing.
`no_data_timeframe`: How long to wait before alerting on no data.

Deploying the Configuration

Once your Terraform files are set up, deploy them using the standard Terraform workflow:

Initialize your Terraform directory:
terraform init
Review the planned changes:
terraform plan -var="datadog_api_key=$DATADOG_API_KEY" -var="datadog_app_key=$DATADOG_APP_KEY" -var="pagerduty_token=$PAGERDUTY_TOKEN" -var="eks_cluster_name=my-eks-cluster"

(Replace `$DATADOG_API_KEY`, `$DATADOG_APP_KEY`, `$PAGERDUTY_TOKEN` with your actual keys/tokens, ideally set as environment variables, and `my-eks-cluster` with your cluster name).
Apply the changes to create the Datadog monitors and PagerDuty resources:
terraform apply -var="datadog_api_key=$DATADOG_API_KEY" -var="datadog_app_key=$DATADOG_APP_KEY" -var="pagerduty_token=$PAGERDUTY_TOKEN" -var="eks_cluster_name=my-eks-cluster"

Testing and Validation

After deployment, it's crucial to validate your setup:

Datadog UI: Navigate to Monitors -> Manage Monitors. You should see your newly created monitors listed.
PagerDuty UI: Verify the new service and escalation policy exist.
Trigger an Alert: Intentionally generate a condition that should trigger one of your monitors (e.g., spike CPU on an EKS node, force a pod restart).
Verify Incident: Confirm that an alert is generated in Datadog and a corresponding incident is created in PagerDuty, routed to the correct service and escalation policy.

Advanced Concepts and Best Practices

Terraform Modules: Create reusable modules for common EKS monitoring patterns (e.g., a "basic-eks-pod-monitor" module) to promote consistency and reduce code duplication across different services or teams.
Contextual Alerting: Enhance your Datadog monitor messages with more context, such as links to runbooks, relevant dashboards, or specific kubectl commands, to accelerate incident resolution.
Event-Based Monitors: For specific EKS events (e.g., failed scheduled jobs, OOMKilled pods), consider using `log alert` types in Datadog that parse Kubernetes event logs.
Composite Monitors: Combine multiple simple monitors into a single, more intelligent composite monitor in Datadog to reduce noise and focus on higher-level service health.
GitOps Workflow: Integrate your Terraform configurations into a GitOps pipeline (e.g., using Atlantis, GitLab CI/CD, GitHub Actions) to automate `terraform plan` and `terraform apply` on every pull request merge to your main branch, ensuring monitoring changes are as rigorously reviewed as application code.

Troubleshooting Common Issues

API Key/Token Errors

Double-check that your `datadog_api_key`, `datadog_app_key`, and `pagerduty_token` are correct and have the necessary permissions. The Datadog API Key requires "API Key Write" and "Monitors Write" permissions. PagerDuty tokens need "Manage" permissions for services and escalation policies.

Monitor Query Syntax

Datadog queries can be complex. Test your query directly in the Datadog Metric Explorer or Log Explorer before embedding it into Terraform. Pay close attention to tags, metric names, and aggregation functions. Ensure the Datadog Agent is correctly deployed on your EKS cluster and collecting the necessary metrics and logs.

PagerDuty Service Name Mismatch

The service name used in the Datadog monitor message (`@pagerduty-servicename`) must exactly match the name of the PagerDuty service created or integrated with Datadog. Using the Terraform output `pagerduty_service.eks_critical_service.name` helps prevent this.

Terraform State Management

Ensure your Terraform state is properly configured for remote storage (e.g., S3 backend with DynamoDB locking) to prevent state corruption and enable team collaboration.

Conclusion

Leveraging Terraform for managing Datadog monitors and PagerDuty incident response on AWS EKS workloads transforms your observability strategy from reactive to proactive and standardized. By treating your monitoring and alerting as code, you gain unparalleled consistency, auditability, and scalability, allowing your DevOps teams to respond faster and more effectively to incidents, ultimately enhancing the reliability and performance of your critical EKS applications. This IaC approach is not just an operational improvement; it's a fundamental shift towards a more resilient and efficient cloud-native ecosystem.

CloudStack YAML