Terraform-Managed AWS EKS Observability with Datadog and PagerDuty Incident Management

Architecture Pro-Tip: Always treat your observability stack as infrastructure. Automating its deployment and configuration with Terraform ensures consistency, repeatability, and version control, crucial for maintaining high reliability in dynamic EKS environments. Integrate it into your CI/CD pipelines from day one for a truly resilient and scalable monitoring solution.

In the fast-evolving landscape of cloud-native applications, maintaining robust observability for Kubernetes clusters is paramount. AWS Elastic Kubernetes Service (EKS) provides a managed Kubernetes control plane, but ensuring your applications running on it are healthy, performant, and resilient requires a comprehensive monitoring and incident management strategy. This guide details how to establish a fully automated, Terraform-managed observability stack for AWS EKS, leveraging Datadog for deep monitoring and PagerDuty for streamlined incident response.

Why Terraform, Datadog, and PagerDuty?

Integrating these three powerful platforms provides a holistic approach to managing the health and incidents of your EKS clusters:

Terraform for Infrastructure as Code (IaC): Terraform allows you to define and provision your entire observability infrastructure – from Datadog agents and monitors to PagerDuty services and escalation policies – as code. This ensures consistency, repeatability, version control, and seamless integration into CI/CD pipelines, eliminating manual configuration errors.
Datadog for Unified Observability: Datadog offers a comprehensive platform for collecting, correlating, and visualizing metrics, logs, and traces from your EKS clusters and the applications running within them. Its native Kubernetes integration provides deep visibility into node health, pod performance, container logs, and more, all within a single pane of glass.
PagerDuty for Proactive Incident Management: PagerDuty transforms observability data into actionable insights by routing critical alerts to the right on-call teams at the right time. Its robust incident response capabilities, including on-call scheduling, escalation policies, and post-incident analysis, minimize downtime and improve overall operational efficiency.

Prerequisites

Before diving into the configuration, ensure you have the following in place:

An AWS account with administrative access.
An existing AWS EKS cluster. (This guide assumes you have one. If not, Terraform can provision that too).
Terraform CLI installed (v1.0+ recommended).
AWS CLI configured with credentials to interact with your AWS account.
A Datadog account with API and Application keys.
A PagerDuty account with a personal API token (for the Terraform provider) and a service integration key (to connect Datadog).
kubectl and helm CLIs installed and configured to connect to your EKS cluster.

Step-by-Step Implementation Guide

1. Setting up Terraform Providers

Your Terraform configuration will need providers for AWS, Kubernetes, Helm, Datadog, and PagerDuty. Ensure your AWS credentials are configured (e.g., via environment variables or AWS CLI configuration). The Kubernetes and Helm providers will use data from your EKS cluster to connect.

2. Deploying Datadog Agent to EKS via Terraform and Helm

To gather metrics, logs, and traces from your EKS cluster, you need to deploy the Datadog Agent. We'll use the Terraform Helm provider to manage the official Datadog Helm chart, ensuring the agent is deployed consistently across your cluster.

Datadog API/APP Keys: These are essential for the Datadog Agent to send data to your Datadog account. Keep them secure, ideally managed as Terraform sensitive variables or via a secrets manager.
Helm Release Configuration: The Helm release will deploy the Datadog Agent and its components (e.g., Cluster Agent, Node Agents, Kube-State-Metrics integration) to your EKS cluster.

3. Configuring PagerDuty Integration with Terraform

Automate the creation of PagerDuty services, escalation policies, and users using the Terraform PagerDuty provider. This ensures your incident response framework is version-controlled and auditable.

PagerDuty Provider Setup: Configure the PagerDuty provider with your API token.
Create Escalation Policy (Optional but Recommended): Define who gets alerted and in what sequence. If you have an existing policy, you can reference its ID.
Create PagerDuty Service: Services represent the components you want to monitor. Alerts from Datadog will be routed to this service.
Create PagerDuty Service Integration: This resource provides the integration key that Datadog will use to send events to this specific PagerDuty service.

4. Creating Datadog Monitors and Dashboards with Terraform

Once the Datadog Agent is collecting data, define monitors to detect anomalies and dashboards for visualization. Terraform allows you to manage these configurations as code, making them repeatable and easy to update.

Datadog Provider Setup: Configure the Datadog provider with your API and Application keys.
Example EKS Node CPU Monitor: Create a monitor that triggers an alert when EKS node CPU utilization exceeds a threshold.
Integrating with PagerDuty: The key to linking Datadog alerts to PagerDuty is to include an @pagerduty-<service_name> tag in your Datadog monitor message. This tells Datadog's built-in PagerDuty integration where to route the incident. Ensure the service name matches exactly the name of your PagerDuty service.
Example EKS Dashboard (Optional): While not included in the primary code block for brevity, Terraform can also manage Datadog dashboards using the datadog_dashboard resource.

Ready-to-Use Terraform Configuration Example

Below is a comprehensive Terraform configuration snippet demonstrating the deployment of the Datadog Agent, creation of a PagerDuty service, and a basic Datadog monitor linked to PagerDuty. Replace placeholder variable values with your actual configuration details.

# main.tf

# Configure AWS Provider
provider "aws" {
  region = var.aws_region
}

# Configure Kubernetes Provider to interact with EKS
data "aws_eks_cluster" "eks_cluster" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "eks_auth" {
  name = var.eks_cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.eks_cluster.endpoint
  token                  = data.aws_eks_cluster_auth.eks_auth.token
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)
}

# Configure Helm Provider
provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.eks_cluster.endpoint
    token                  = data.aws_eks_cluster_auth.eks_auth.token
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)
  }
}

# Configure Datadog Provider
provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

# Configure PagerDuty Provider
provider "pagerduty" {
  token = var.pagerduty_api_key
}

# Deploy Datadog Agent using Helm
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Consider a dedicated namespace like 'datadog'
  version    = "2.33.0" # Use a specific stable version
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "clusterChecksRunner.enabled"
    value = "true"
  }
  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
  }
  set {
    name  = "collectEvents"
    value = "true"
  }
  set {
    name  = "agents.tolerations[0].operator"
    value = "Exists"
  }
  set {
    name  = "agents.logLevel"
    value = "INFO"
  }
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "apm.enabled"
    value = "true"
  }
  set {
    name  = "processAgent.enabled"
    value = "true"
  }
  set {
    name  = "orchestratorExplorer.enabled"
    value = "true"
  }
}

# Create a PagerDuty Escalation Policy (optional, if you want to manage it via Terraform)
resource "pagerduty_escalation_policy" "devops_oncall_policy" {
  name      = "DevOps On-Call Policy"
  # Example team_id; replace with your team's ID if applicable
  # team_id   = var.pagerduty_team_id
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 30
    target {
      type = "user"
      id   = var.pagerduty_user_id # ID of an existing PagerDuty user
    }
  }
  # Add more rules or targets as needed, e.g., to escalate to another user or schedule
}

# Create a PagerDuty Service for EKS Observability
resource "pagerduty_service" "eks_observability_service" {
  name        = "EKS Cluster Observability"
  description = "Monitors AWS EKS health and critical metrics. Integrated with Datadog."
  escalation_policy = pagerduty_escalation_policy.devops_oncall_policy.id
  auto_resolve_timeout = 14400 # 4 hours
  acknowledgement_timeout = 300 # 5 minutes
}

# Create a PagerDuty Service Integration for Datadog
# This resource creates an integration key that you would typically configure within Datadog
# as a PagerDuty integration. However, Datadog also has a native integration where you specify
# PagerDuty services by name in the monitor message directly (e.g., @pagerduty-YourServiceName).
# This resource's 'integration_key' might be needed for older Datadog PagerDuty integrations,
# or for more custom webhook setups. For most modern native Datadog integrations,
# specifying the service name in the message is sufficient after the initial Datadog-PagerDuty API key connection.
resource "pagerduty_service_integration" "datadog_integration" {
  name    = "Datadog Integration"
  type    = "datadog_integration"
  service = pagerduty_service.eks_observability_service.id
}

# Create a Datadog Monitor for EKS Node CPU Utilization
resource "datadog_monitor" "eks_node_cpu_utilization" {
  name                = "EKS Node CPU Utilization High on {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.cpu.usage.total{kubernetes_cluster_name:${var.eks_cluster_name}} by {host} > 80"
  message             = "EKS Node CPU usage is high on {{host.name}} ({{value}}%). Investigate potential bottlenecks or scaling issues. @pagerduty-${pagerduty_service.eks_observability_service.name}"
  tags                = ["environment:${var.environment}", "service:eks", "alert-type:cpu"]
  escalation_message  = "CPU usage remains critical on {{host.name}}. Escalating to next level."
  alert_priority      = 2
  notify_no_data      = false
  no_data_timeframe   = 20
  renotify_interval   = 30
  require_full_window = false
  timeout_h           = 0
  enable_logs_sample  = false
}

# output.tf
output "pagerduty_service_id" {
  description = "The ID of the PagerDuty service created."
  value       = pagerduty_service.eks_observability_service.id
}

output "pagerduty_service_name" {
  description = "The name of the PagerDuty service created, used for Datadog integration."
  value       = pagerduty_service.eks_observability_service.name
}

# variables.tf
variable "aws_region" {
  description = "The AWS region where your EKS cluster is located."
  type        = string
  default     = "us-east-1"
}

variable "eks_cluster_name" {
  description = "The name of your existing EKS cluster."
  type        = string
  # Example: default = "my-eks-cluster"
}

variable "datadog_api_key" {
  description = "Your Datadog API Key."
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Your Datadog Application Key."
  type        = string
  sensitive   = true
}

variable "pagerduty_api_key" {
  description = "Your PagerDuty API Token (for Terraform provider)."
  type        = string
  sensitive   = true
}

variable "pagerduty_user_id" {
  description = "ID of a PagerDuty user for the escalation policy (e.g., 'PXXXXXX')."
  type        = string
  # Example: default = "PXXXXXX"
}

variable "environment" {
  description = "The deployment environment (e.g., dev, staging, prod) for tagging."
  type        = string
  default     = "dev"
}
        

Deployment Steps

To deploy this configuration and establish your EKS observability:

Save the code above into main.tf, variables.tf, and outputs.tf files within a new, dedicated Terraform directory.
Populate variables.tf with your actual values, or set them as environment variables (e.g., export TF_VAR_datadog_api_key="your_key"). Remember to keep sensitive keys secure.
Initialize Terraform in your directory:
terraform init
Review the planned changes to ensure they align with your expectations:
terraform plan
Apply the configuration to deploy your Datadog agents, PagerDuty service, and Datadog monitors:
terraform apply

Best Practices for EKS Observability

Granular Monitoring: Beyond basic CPU/memory, monitor application-specific metrics (custom metrics), business KPIs, network performance, and Kubernetes control plane health.
Alerting Strategy: Implement a tiered alerting system. Minor issues might trigger informational alerts, while critical issues should immediately page on-call teams through PagerDuty. Focus on actionable alerts to avoid alert fatigue.
Continuous Improvement: Regularly review your monitors and dashboards. Are they still relevant? Are there false positives or missed critical events? Update your Terraform configurations accordingly.
Security First: Ensure your Datadog API/APP keys and PagerDuty API tokens are stored securely (e.g., AWS Secrets Manager, HashiCorp Vault) and accessed by Terraform with least privilege.
Logging and Tracing: While metrics are covered, fully integrate distributed tracing (APM) and centralized logging for comprehensive troubleshooting. Datadog agents can be configured to collect these as well.

Troubleshooting and FAQ

Datadog Agent Not Reporting Data:

Check API/APP Keys: Verify that your datadog_api_key and datadog_app_key are correct and have the necessary permissions.
Verify Helm Deployment: Run kubectl get pods -n default -l app.kubernetes.io/name=datadog to ensure Datadog agent pods are running. Check logs of the Datadog pods for errors: kubectl logs <datadog-pod-name> -n default.
EKS Network/Security Group: Ensure your EKS nodes' security groups allow outbound HTTPS traffic to Datadog endpoints (e.g., api.datadoghq.com).

PagerDuty Alerts Not Firing:

Datadog Monitor Configuration: Double-check the message field in your datadog_monitor resource. The @pagerduty-<service_name> part must exactly match the name of your PagerDuty service created by pagerduty_service.eks_observability_service.name.
PagerDuty Service Integration: Confirm that the PagerDuty service has an active integration configured to receive alerts from Datadog. While Terraform creates the service, the initial connection between Datadog and PagerDuty often requires setting up a global PagerDuty integration in the Datadog UI using a PagerDuty API key (not the service integration key).
Datadog Event Log: In Datadog, go to "Monitors" -> "Monitor Status" to see if your monitor is triggering. Then check the "Event Explorer" for PagerDuty events and any associated errors.

Conclusion

By implementing a Terraform-managed observability stack for AWS EKS with Datadog and PagerDuty, organizations can achieve unparalleled visibility into their cloud-native applications and significantly improve their incident response capabilities. This infrastructure-as-code approach ensures that your monitoring and alerting infrastructure scales seamlessly with your EKS clusters, providing a reliable, automated, and auditable foundation for operational excellence. Embrace IaC for your observability, and empower your teams to build, deploy, and operate highly resilient applications with confidence.

CloudStack YAML