Terraform for AWS EKS Observability: Datadog Monitors and PagerDuty Incident Automation

In the dynamic landscape of cloud-native applications, maintaining robust observability for Kubernetes clusters is paramount. AWS EKS, a popular managed Kubernetes service, requires sophisticated monitoring and incident response mechanisms to ensure high availability and performance. This comprehensive guide delves into leveraging Terraform to codify Datadog monitors and integrate them with PagerDuty for automated incident management, bringing Infrastructure as Code (IaC) principles to your observability stack.

Architecture Pro-Tip:

Always define your observability resources (monitors, dashboards, alerts) as close to your infrastructure code as possible. Storing them in the same Terraform repository as your EKS cluster definition or application deployments ensures version control, simplifies rollbacks, and fosters a "single source of truth" for your operational state. This prevents drift and accelerates recovery during incidents.

The Pillars of EKS Observability with IaC

Effective observability on AWS EKS requires a trifecta of metrics, logs, and traces. Datadog excels at unifying these data types into a single pane of glass, providing deep insights into cluster health, application performance, and user experience. Integrating this monitoring capability with an automated incident response system like PagerDuty ensures that critical issues are addressed promptly, minimizing downtime.

Why Terraform for Observability?

Terraform brings the benefits of Infrastructure as Code to your monitoring and alerting configurations. By defining Datadog monitors and PagerDuty services in declarative code, you achieve:

Version Control: Track changes, review, and audit all modifications to your alerting policies.
Consistency: Apply standardized monitoring across multiple EKS clusters or environments.
Automation: Eliminate manual configuration, reducing human error and accelerating deployment.
Repeatability: Easily replicate your entire observability setup for new projects or disaster recovery scenarios.

Prerequisites and Setup

Before diving into the code, ensure you have the following:

An AWS Account with an existing EKS cluster.
A Datadog Account with API and Application keys. Ensure the Datadog Agent is deployed to your EKS cluster to collect metrics, logs, and traces.
A PagerDuty Account with an API token.
Terraform CLI (v1.0.0+) installed locally.

Datadog and PagerDuty Provider Configuration

First, set up your Terraform providers. You'll need `datadog` and `pagerduty` providers configured with appropriate API keys and tokens. It's best practice to manage these credentials securely, for example, using environment variables or a secrets manager.

provider "aws" {
  region = "us-east-1" # Or your desired region
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_token
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}
    

Defining PagerDuty Services with Terraform

Before Datadog can send alerts to PagerDuty, you need a PagerDuty service to receive them. Terraform allows you to define these services, including their escalation policies, directly in code.

# Define a PagerDuty Escalation Policy (e.g., notifying primary on-call, then secondary)
resource "pagerduty_escalation_policy" "eks_critical_policy" {
  name      = "EKS Critical Alerts Policy"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 5
    target {
      type = "user"
      id   = var.pagerduty_primary_oncall_user_id # ID of your primary on-call user
    }
  }

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "user"
      id   = var.pagerduty_secondary_oncall_user_id # ID of your secondary on-call user
    }
  }
}

# Define a PagerDuty Service
resource "pagerduty_service" "eks_observability_service" {
  name                    = "EKS Observability"
  auto_resolve_timeout_in_minutes = 1440 # Auto-resolve after 24 hours
  acknowledgement_timeout_in_minutes = 30
  escalation_policy       = pagerduty_escalation_policy.eks_critical_policy.id
  alert_creation          = "create_alerts_and_incidents"
}

variable "pagerduty_primary_oncall_user_id" {
  description = "ID of the primary PagerDuty on-call user"
  type        = string
}

variable "pagerduty_secondary_oncall_user_id" {
  description = "ID of the secondary PagerDuty on-call user"
  type        = string
}
    

Terraform for Datadog Monitors

The core of Datadog's alerting system lies in its monitors. Terraform's `datadog_monitor` resource allows you to define various types of monitors, from metric-based to log-based or anomaly detection. Each monitor can be configured with specific alert thresholds, recovery thresholds, and notification messages.

Integrating PagerDuty with Datadog

To send alerts from Datadog to PagerDuty, you first need to establish the integration. The `datadog_integration_pagerduty` resource facilitates this, though in modern Datadog, the integration is often configured manually in the UI and then referenced by name or ID. However, the connection within the monitor's message is key.

Example: EKS Node CPU Utilization Monitor with PagerDuty

Let's create a Datadog monitor that triggers an alert when EKS node CPU utilization exceeds a certain threshold and routes this incident to the PagerDuty service we defined earlier.

# Create a Datadog monitor for EKS Node CPU Utilization
resource "datadog_monitor" "eks_node_cpu_utilization" {
  name                = "[EKS Node Alert] High CPU Utilization - {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:system.cpu.idle{eks_cluster_name:your-eks-cluster-name} by {host} < 20"
  message             = <<EOF
@pagerduty-{{pagerduty_service.name}}
EKS Node CPU utilization is high!
Host: {{host.name}} ({{host.ip}})
Current CPU Idle: {{value}}% (Threshold: <20%)
Impact: Potential performance degradation or unresponsiveness for pods on this node.
Runbook: [Link to your Runbook for high CPU](https://your-company.runbooks.com/eks-cpu-issue)
EOF
  tags                = ["eks", "observability", "cpu", "critical"]
  priority            = 1
  enabled             = true
  no_data_timeframe   = 20 # Alert if no data for 20 minutes

  # Alert threshold: CPU idle drops below 20% (i.e., utilization goes above 80%)
  monitor_thresholds {
    critical = 20
    warning  = 30
  }

  # Recovery threshold: CPU idle recovers above 40% (i.e., utilization drops below 60%)
  monitor_thresholds {
    ok = 40
  }

  # Configure PagerDuty as a notification channel
  # The name here should match the PagerDuty service created or existing in Datadog UI
  # Datadog often automatically creates a service integration name based on the PagerDuty service name
  # In this example, we directly reference the PagerDuty service ID for robust integration
  # Ensure the PagerDuty integration is configured in Datadog UI, or use datadog_integration_pagerduty if needed.
  # For existing PagerDuty integration in Datadog, the notification format is `@pagerduty-[integration_name]`
  # A more robust way to link a specific PagerDuty service to a Datadog monitor is via the monitor message itself,
  # referencing the PagerDuty service name.
}

# Ensure to replace 'your-eks-cluster-name' with the actual name of your EKS cluster.
# For PagerDuty integration, the `@pagerduty-{{pagerduty_service.name}}` syntax
# dynamically picks up the integration corresponding to the service named in PagerDuty.
# You might need to manually ensure the PagerDuty integration is configured in Datadog
# to recognize services created via Terraform.
    

More EKS-Specific Monitors

Here are additional examples of critical EKS observability monitors you can define with Terraform:

Pod CrashloopBackoff: Alerts when pods are repeatedly failing to start.
EKS Control Plane Latency: Monitors the health of the EKS control plane components.
Node Disk Utilization: Prevents nodes from running out of disk space.
Kubernetes API Server Errors: Tracks errors from the API server.
High HPA Scale Events: Indicates applications under unexpected load.

Deployment and Validation

Once your Terraform configuration is ready:

Initialize Terraform: Navigate to your Terraform directory and run terraform init.
Plan Changes: Execute terraform plan to review the resources that will be created or modified.
Apply Configuration: Run terraform apply and confirm with yes to provision the Datadog monitors and PagerDuty services.

Validation:

Verify in the Datadog UI that your monitors are listed and correctly configured.
Check the PagerDuty UI to ensure the service and escalation policy are present.
For critical monitors, consider simulating an incident (e.g., by intentionally overloading a test node) to confirm PagerDuty alerts are triggered and routed correctly.

Advanced Considerations

Modularization: For large-scale environments, organize your Terraform code into modules (e.g., a `datadog-monitors` module, a `pagerduty-services` module) to improve reusability and maintainability.
Contextual Information: Enhance your monitor messages with more context, such as links to runbooks, relevant dashboards, or specific team contacts. Use Datadog's template variables (`{{variable_name}}`) extensively.
Tags and Filters: Leverage Datadog tags (`tags = ["env:prod", "team:sre"]`) to categorize monitors and apply them selectively to different EKS clusters or applications.
No Data Alerts: Configure `no_data_timeframe` in Datadog monitors to alert if expected metrics stop arriving, indicating potential agent issues or data pipeline failures.
Log-Based Monitors: Extend your observability with log-based alerts using Datadog's log management capabilities, for example, alerting on specific error patterns in application logs.

Troubleshooting and FAQ

Q: My PagerDuty alerts aren't triggering from Datadog. What should I check?

Datadog-PagerDuty Integration: Ensure the integration is correctly configured in the Datadog UI (Integrations -> PagerDuty). The `integration_name` you use in your Datadog monitor message (`@pagerduty-[integration_name]`) must match the name given in Datadog's integration settings.
Monitor Thresholds: Double-check that your monitor's query and thresholds are actually being met by the data.
Notification Scope: Verify that the monitor is not silenced or suppressed by any notification rules in Datadog.
PagerDuty Service Key: If you're using a generic PagerDuty integration, ensure the routing key in the Datadog alert message matches the integration key for the PagerDuty service.

Q: How do I manage sensitive API keys and tokens securely with Terraform?

A: Avoid hardcoding credentials. Use Terraform variables and provide their values via environment variables (e.g., `TF_VAR_datadog_api_key`), integrate with a secrets manager like AWS Secrets Manager or HashiCorp Vault, or use Terraform Cloud/Enterprise's variable management.

Q: Can I manage PagerDuty on-call schedules with Terraform?

A: Yes, the PagerDuty Terraform provider supports managing schedules, users, teams, and more. This allows for complete IaC control over your incident management setup.

Conclusion

Adopting Infrastructure as Code for your AWS EKS observability stack with Terraform, Datadog, and PagerDuty is a strategic move towards building resilient, scalable, and maintainable cloud-native environments. By codifying your monitoring and incident response, you empower your DevOps and SRE teams with automation, consistency, and rapid iteration capabilities, ensuring that your EKS clusters remain performant and available around the clock. Start by implementing critical monitors and gradually expand your codified observability footprint to achieve unparalleled operational excellence.

CloudStack YAML