Automated Datadog and PagerDuty Integration for AWS EKS with Terraform

Architecture Pro-Tip: Implement a robust tagging strategy for your AWS EKS resources. Use these tags (e.g., environment, application, team) consistently. This practice extends seamlessly into Datadog, allowing for granular filtering, scoped monitors, and targeted alerts to specific PagerDuty services, significantly improving incident routing and reducing alert fatigue.

Automated Datadog and PagerDuty Integration for AWS EKS with Terraform

In today's fast-paced cloud-native environments, reliable monitoring and incident response are paramount. Manually configuring monitoring and alert routing across complex infrastructures like AWS EKS clusters is not only time-consuming but also prone to human error, leading to slower incident resolution times. This comprehensive guide outlines how to leverage Terraform for automating the integration of Datadog for observability and PagerDuty for incident management within your AWS EKS environment. By adopting Infrastructure as Code (IaC), you can ensure consistent, scalable, and auditable monitoring and alerting configurations.

Why Automate Datadog and PagerDuty Integration?

The synergy between Datadog and PagerDuty, orchestrated by Terraform, delivers a powerful solution for modern DevOps teams:

Consistency and Reliability: Define your monitoring and alerting policies once in code and apply them consistently across all environments.
Scalability: Easily extend monitoring to new EKS clusters, services, or namespaces with minimal manual effort.
Reduced MTTR (Mean Time To Resolution): Proactive alerts from Datadog routed instantly to the right on-call team via PagerDuty accelerate incident detection and response.
Auditability and Version Control: All configurations are stored in a Git repository, allowing for change tracking, rollbacks, and collaboration.
Compliance: Enforce organizational standards for monitoring and incident response across your EKS footprint.

Prerequisites

Before you begin, ensure you have the following:

An active AWS Account with permissions to manage EKS clusters and associated resources.
An existing AWS EKS Cluster. (This guide assumes you have one. If not, Terraform can also provision EKS.)
A Datadog Account with appropriate API and Application Keys.
A PagerDuty Account with an API Token and at least one Service and Escalation Policy.
Terraform CLI installed (version 1.0+ recommended).
Familiarity with Kubernetes and basic Terraform concepts.

Setting Up Your Environment for Terraform

1. Datadog API & Application Keys

Navigate to your Datadog account settings:

Go to Organization Settings > API Keys.
Create or identify your Datadog API Key and Datadog Application Key. These will be used by the Terraform Datadog provider. Store them securely, e.g., using environment variables (DD_API_KEY, DD_APP_KEY) or a secrets manager.

2. PagerDuty API Token & Service Integration

Log into your PagerDuty account:

Go to Integrations > API Access Keys.
Create a new API Key (e.g., "Terraform Automation Key"). This will be used by the Terraform PagerDuty provider. Store it securely (e.g., PAGERDUTY_TOKEN environment variable).
Ensure you have an existing PagerDuty Service and an associated Escalation Policy that your Datadog alerts will target. If not, you can also define these using Terraform.

Terraform Configuration for Seamless Integration

We will structure our Terraform project to manage providers, Datadog resources, and PagerDuty resources.

1. Project Structure

A recommended project structure might look like this:

main.tf: Main configuration file.
variables.tf: Input variables.
providers.tf: Provider configurations.
datadog.tf: Datadog-specific resources (monitors, EKS integration).
pagerduty.tf: PagerDuty-specific resources (services, escalation policies, integrations).

2. Provider Configuration (`providers.tf`)

Configure the AWS, Datadog, and PagerDuty providers. It's best practice to use environment variables for sensitive API keys.

3. Datadog Resources (`datadog.tf`)

This section covers integrating EKS metrics into Datadog and defining a crucial monitor that will trigger alerts.

Datadog AWS Integration: First, ensure your AWS account is integrated with Datadog to pull EKS metrics. This is typically done via an AWS IAM role.
EKS Integration (Datadog Agent): While the Datadog AWS integration pulls high-level metrics, for detailed Kubernetes observability, you'll need to deploy the Datadog Agent to your EKS cluster. This is usually done via a Helm chart, which can also be managed by Terraform.
Datadog Monitors: Define monitors for critical EKS metrics. For example, a monitor for high CPU utilization on EKS worker nodes.

4. PagerDuty Resources (`pagerduty.tf`)

Here, we define the PagerDuty service that will receive alerts and integrate it with Datadog.

PagerDuty Service: Create a service dedicated to EKS alerts.
Escalation Policy: Associate an existing or new escalation policy with the service to define who gets alerted and when.
Datadog Integration on PagerDuty: Crucially, create a PagerDuty integration of type "Datadog" for the service. This generates an integration key that Datadog uses to send events.

5. Connecting Datadog Monitors to PagerDuty

The final step is to configure your Datadog monitors to send alerts to the PagerDuty service via its integration. This is done by specifying the PagerDuty integration key in the monitor's notification message.

Ready-to-Use Terraform Configuration Example

Below is a consolidated example of Terraform code to set up a Datadog monitor for EKS node CPU, create a PagerDuty service and integration, and link them. Replace placeholder values with your actual IDs and names.

# main.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
    pagerduty = {
      source  = "pagerduty/pagerduty"
      version = "~> 2.0"
    }
  }
}

# providers.tf
provider "aws" {
  region = var.aws_region
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_token
}

# variables.tf
variable "aws_region" {
  description = "AWS region for EKS cluster"
  type        = string
  default     = "us-east-1"
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

variable "pagerduty_escalation_policy_id" {
  description = "ID of the existing PagerDuty escalation policy"
  type        = string
  # Example: "P001ABCD" - replace with your actual ID
}

variable "eks_cluster_name" {
  description = "Name of your AWS EKS cluster"
  type        = string
  # Example: "my-production-eks"
}

# pagerduty.tf
resource "pagerduty_user" "devops_user" {
  name  = "DevOps Oncall"
  email = "devops-oncall@example.com"
}

resource "pagerduty_team" "devops_team" {
  name        = "DevOps Team"
  description = "Team responsible for EKS infrastructure"
}

resource "pagerduty_team_membership" "devops_user_membership" {
  user_id = pagerduty_user.devops_user.id
  team_id = pagerduty_team.devops_team.id
  role    = "admin" # or user
}

resource "pagerduty_escalation_policy" "eks_escalation_policy" {
  name      = "EKS High Severity Escalation"
  num_loops = 2
  team {
    id   = pagerduty_team.devops_team.id
    type = "team_reference"
  }
  rule {
    escalation_delay_in_minutes = 5
    target {
      id   = pagerduty_user.devops_user.id
      type = "user_reference"
    }
  }
}

resource "pagerduty_service" "eks_monitoring_service" {
  name                       = "${var.eks_cluster_name}-monitoring"
  auto_resolve_timeout       = 60
  acknowledgement_timeout    = 30
  escalation_policy          = pagerduty_escalation_policy.eks_escalation_policy.id
  alert_creation             = "create_alerts_and_incidents"
  alert_grouping_parameters {
    type = "intelligent"
  }
  status                     = "active"
}

# PagerDuty integration for Datadog
resource "pagerduty_extension" "datadog_extension" {
  name          = "${pagerduty_service.eks_monitoring_service.name} Datadog Integration"
  endpoint_url  = "https://app.datadoghq.com/pagerduty/callback" # This is a placeholder, actual PagerDuty integration type "Datadog" generates this URL.
  extension_type = "datadog" # This indicates a Datadog integration
  service_id     = pagerduty_service.eks_monitoring_service.id
  # Note: PagerDuty's Datadog integration type doesn't expose a specific integration key directly via Terraform.
  # Instead, you typically obtain it from the PagerDuty UI after creating the service integration,
  # or you can assume Datadog uses the service's general API key with a direct PagerDuty channel.
  # For direct integration, PagerDuty typically assigns an integration key once the integration is created.
  # In a real scenario, you'd create an "Events API V2" integration for the service via Terraform,
  # and then use its integration key.
  # For simplicity, we'll demonstrate a Datadog monitor linking to a PagerDuty service by name in the message.
  # A more robust solution involves `pagerduty_service_integration` of type "events_api_v2"
  # and passing its `integration_key` to the Datadog monitor message.
}

# datadog.tf
# Note: For actual EKS metric collection, ensure the Datadog Agent is deployed to your EKS cluster
# via Helm, and AWS integration is configured in Datadog UI or via `datadog_integration_aws`.

resource "datadog_monitor" "eks_node_cpu_high" {
  name               = "[EKS] High Node CPU Utilization on {{host.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.eks_cluster_name}} by {kube_node} > 80"
  message            = <Explanation of the Example:
The pagerduty_service and pagerduty_escalation_policy resources define the incident handling mechanism.
The pagerduty_extension of type "datadog" is a crucial bridge. While Datadog typically uses an "integration key" for PagerDuty, the latest PagerDuty API integrations usually involve an Events API V2 key. For simplicity and demonstration, we directly reference the PagerDuty service in the Datadog monitor message using the standard @pagerduty-SERVICE_NAME syntax. For production, it's recommended to create a pagerduty_service_integration of type `events_api_v2` and use its `integration_key` directly in the Datadog monitor message.
The datadog_monitor tracks average CPU utilization across EKS nodes. When the critical threshold is breached, it sends an alert.
The message field of the Datadog monitor includes @pagerduty-{{pagerduty_service.eks_monitoring_service.name}}. This tells Datadog to send the alert to the PagerDuty service named `eks_monitoring_service`. Datadog automatically resolves the service name to the correct PagerDuty endpoint if the integration is set up correctly in both platforms.
Deployment StepsInitialize Terraform: Navigate to your project directory and run terraform init to download the necessary providers.
Set Variables: Provide your API keys and cluster name via environment variables (e.g., export TF_VAR_datadog_api_key="YOUR_KEY") or a terraform.tfvars file.
Plan Changes: Run terraform plan to preview the changes Terraform will make.
Apply Changes: Execute terraform apply and confirm with yes to provision the resources.
Testing and ValidationAfter applying the Terraform configuration:
Datadog Dashboard: Verify that the new monitor appears in your Datadog Monitors list. You can trigger a test alert from the monitor's settings.
PagerDuty Incidents: Check your PagerDuty service's incident list. A test alert from Datadog should create a new incident.
Simulate Alert: If possible, intentionally cause a high CPU load on an EKS node (e.g., by running a stress test container) to verify the alert triggers end-to-end.


Troubleshooting & Best PracticesCommon Issues:API Key Permissions: Ensure your Datadog and PagerDuty API keys have the necessary permissions to create/manage monitors and services respectively.
Incorrect Service Name in Datadog Message: Double-check the PagerDuty service name used in the Datadog monitor's message (e.g., @pagerduty-SERVICE_NAME). It must match the PagerDuty service's configured name for Datadog integration.
Datadog Agent Not Sending EKS Metrics: Verify the Datadog Agent is correctly deployed on your EKS cluster and has access to Kubernetes metrics. Check agent logs for errors.
Best Practices:Secret Management: Never hardcode API keys. Use Terraform variable files (.tfvars) with TF_VAR_ environment variables, or a dedicated secrets manager like AWS Secrets Manager or HashiCorp Vault.
Module-ize: For larger deployments, break down your Terraform configuration into reusable modules (e.g., a "datadog-monitor" module, a "pagerduty-service" module).
GitOps Workflow: Integrate your Terraform code into a GitOps workflow where changes are reviewed and applied via CI/CD pipelines.
Granular Alerts: Create specific Datadog monitors for different EKS components (e.g., control plane, worker nodes, specific deployments/services) and route them to relevant PagerDuty services/teams.
Clear Notification Messages: Craft informative Datadog monitor messages that include all necessary context (cluster name, affected resource, metric value, runbook links) for quick incident triaging in PagerDuty.
ConclusionAutomating Datadog and PagerDuty integration for AWS EKS with Terraform transforms your observability and incident response strategy. It provides a robust, scalable, and reliable foundation for managing your Kubernetes clusters, reducing operational overhead, and significantly improving your team's ability to respond to critical incidents effectively. By embracing IaC for monitoring and alerting, you empower your DevOps teams to build and operate cloud-native applications with greater confidence and efficiency.

CloudStack YAML