Terraform for AWS EKS Monitoring with Datadog and PagerDuty Incident Response

Architecture Pro-Tip:

For highly available and resilient EKS monitoring, always deploy the Datadog Agent as a DaemonSet across all worker nodes. Couple this with Datadog's Cluster Agent for centralized metric collection and API proxying, reducing direct API calls from individual agents. Integrate PagerDuty for critical alerts, ensuring on-call rotations and escalation policies are clearly defined and tested to minimize mean time to recovery (MTTR).

In the dynamic world of cloud-native applications, maintaining robust observability and a rapid incident response mechanism for your Kubernetes clusters is paramount. AWS EKS (Elastic Kubernetes Service) provides a powerful foundation, but its effective operation demands sophisticated monitoring and alerting. This guide demonstrates how to leverage Terraform Infrastructure as Code (IaC) to seamlessly integrate Datadog for comprehensive monitoring and PagerDuty for streamlined incident response, creating a resilient and automated observability pipeline for your AWS EKS environment.

Why Terraform, Datadog, and PagerDuty?

Terraform: The IaC Backbone

Terraform allows you to define and provision your entire infrastructure, including monitoring and alerting configurations, in human-readable configuration files. This brings numerous benefits:

Version Control: Track changes, roll back to previous states, and collaborate effectively.
Automation: Eliminate manual errors and accelerate deployments.
Consistency: Ensure identical environments across development, staging, and production.
Scalability: Easily scale your monitoring infrastructure alongside your EKS clusters.

Datadog: Comprehensive Cloud-Native Monitoring

Datadog offers an unparalleled platform for observing the health and performance of your EKS clusters and the applications running within them. Key capabilities include:

Unified Metrics & Logs: Collects and correlates metrics, logs, and traces from EKS, pods, containers, and underlying AWS infrastructure.
APM & Distributed Tracing: Gain deep insights into application performance across microservices.
Synthetic Monitoring: Proactively test application availability and performance from various global locations.
Rich Dashboards & Alerts: Visualize data with customizable dashboards and set up intelligent alerts.

PagerDuty: Intelligent Incident Response

When Datadog detects an anomaly or a critical threshold breach, PagerDuty takes over to ensure the right people are notified immediately. PagerDuty facilitates:

Automated On-Call Scheduling: Manages complex schedules and escalations.
Multi-Channel Notifications: Alerts via SMS, phone calls, email, and push notifications.
Incident Management: Centralizes incident information, enabling quicker resolution.
Runbook Automation: Integrate automated actions to resolve common issues.

Prerequisites

Before diving into the configuration, ensure you have the following set up:

AWS Account: With necessary permissions to manage EKS and IAM.
AWS EKS Cluster: An existing EKS cluster. This guide assumes you have one; provisioning an EKS cluster with Terraform is a separate, extensive topic.
Terraform CLI: Installed and configured.
Kubectl: Configured to connect to your EKS cluster.
Datadog Account: With an API Key and Application Key.
PagerDuty Account: With an API Key and an existing PagerDuty Service (or permission to create one).

Setting Up Terraform for EKS Monitoring and Incident Response

1. Provider Configuration

First, configure the necessary Terraform providers: AWS, Kubernetes, Datadog, and PagerDuty. Store sensitive API keys securely, ideally using environment variables or a secret manager.

2. Deploying the Datadog Agent to EKS

The Datadog Agent collects metrics, logs, and traces from your EKS cluster. We'll use the helm_release resource to deploy the official Datadog Helm chart.

3. Configuring Datadog Monitors with Terraform

Define Datadog monitors using the datadog_monitor resource. These monitors will detect issues like high CPU utilization, low memory, or pod restarts.

4. Integrating PagerDuty with Datadog via Terraform

To connect Datadog alerts to PagerDuty, you'll need a PagerDuty service integration. While Datadog provides a direct PagerDuty integration that can be configured in their UI, you can also define PagerDuty services and integrations using Terraform. For simplicity and best practice, we'll configure Datadog monitors to send alerts to PagerDuty using an integration that's typically set up once in Datadog and referenced by its name in monitor notifications.

Ensure you have a PagerDuty integration configured in Datadog (e.g., named "PagerDuty-EKS-Critical"). You can find this under "Integrations" -> "PagerDuty" in the Datadog UI. This integration provides a service key that Datadog uses to send alerts to PagerDuty.

5. Linking Datadog Monitors to PagerDuty for Incident Response

Within your datadog_monitor resources, specify the PagerDuty integration name in the message field using the @pagerduty- syntax.

Ready-to-Use Configuration

Below is a comprehensive Terraform configuration demonstrating how to:

Configure AWS, Kubernetes, and Datadog providers.
Retrieve EKS cluster details.
Deploy the Datadog Agent using the Helm provider.
Create example Datadog monitors for EKS health, configured to alert a PagerDuty service.

main.tf (Example)

resource "kubernetes_namespace" "datadog_agent" {
  metadata {
    name = "datadog"
  }
}

data "aws_eks_cluster" "eks_cluster" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "eks_cluster" {
  name = var.eks_cluster_name
}

provider "aws" {
  region = var.aws_region
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.eks_cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.eks_cluster.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.eks_cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.eks_cluster.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

# Deploy Datadog Agent using Helm
resource "helm_release" "datadog_agent" {
  name       = "datadog-agent"
  namespace  = kubernetes_namespace.datadog_agent.metadata[0].name
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  version    = "2.37.0" # Use a stable and recent version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }
  set {
    name  = "datadog.site"
    value = "datadoghq.com" # or eu.datadoghq.com etc.
  }
  set {
    name  = "agents.enabled"
    value = "true"
  }
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "clusterChecksRunner.enabled"
    value = "true"
  }
  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
  }
  set {
    name  = "targetSystem"
    value = "linux"
  }
  set {
    name = "providers.aws.region"
    value = var.aws_region
  }
  set {
    name = "datadog.logLevel"
    value = "INFO"
  }
  set {
    name = "logs.enabled"
    value = "true"
  }
  set {
    name = "logs.containerCollectAll"
    value = "true"
  }
  set {
    name = "apm.enabled"
    value = "true"
  }
  set {
    name = "processAgent.enabled"
    value = "true"
  }
  set {
    name = "networkMonitoring.enabled"
    value = "true"
  }
}

# Example Datadog Monitors
# Make sure "PagerDuty-EKS-Critical" is the exact name of your PagerDuty integration in Datadog.

resource "datadog_monitor" "high_node_cpu_utilization" {
  name               = "[EKS] High Node CPU Utilization on {{host.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster_name:${var.eks_cluster_name},kube_namespace:default} by {host} < 10"
  message            = "High CPU utilization detected on EKS node {{host.name}} ({{value}}% idle). Investigate potential resource contention or runaway processes. @pagerduty-EKS-Critical @slack-devops"
  threshold_warnings = { "45" = "50" }
  threshold_critical = { "30" = "70" } # < 30% idle means > 70% used
  include_tags       = true
  notify_no_data     = true
  no_data_timeframe  = 10
  renotify_interval  = 60
  tags               = ["environment:${var.environment}", "team:devops", "eks-monitoring"]
  priority           = 1
}

resource "datadog_monitor" "low_node_memory_available" {
  name               = "[EKS] Low Node Memory Available on {{host.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:system.mem.total{kubernetes_cluster_name:${var.eks_cluster_name},kube_namespace:default} by {host} - avg(last_5m):avg:system.mem.free{kubernetes_cluster_name:${var.eks_cluster_name},kube_namespace:default} > (avg(last_5m):avg:system.mem.total{kubernetes_cluster_name:${var.eks_cluster_name},kube_namespace:default} * 0.9)"
  message            = "Memory available is critically low on EKS node {{host.name}}. Current usage is {{value}}% of total. This could impact application stability. @pagerduty-EKS-Critical @slack-devops"
  threshold_warnings = { "0.8" = "" } # 80% used
  threshold_critical = { "0.9" = "" } # 90% used
  include_tags       = true
  notify_no_data     = true
  no_data_timeframe  = 10
  renotify_interval  = 60
  tags               = ["environment:${var.environment}", "team:devops", "eks-monitoring"]
  priority           = 1
}

resource "datadog_monitor" "pod_restarts" {
  name               = "[EKS] High Pod Restart Rate for {{kube_container_name}} in {{kube_namespace}}"
  type               = "metric alert"
  query              = "sum(last_5m):sum:kubernetes.containers.restarts{kubernetes_cluster_name:${var.eks_cluster_name},kube_namespace:!kube-system} by {kube_container_name,kube_namespace} > 5"
  message            = "Multiple restarts detected for pod {{kube_container_name}} in namespace {{kube_namespace}}. Investigate pod logs for errors. @pagerduty-EKS-Critical @slack-devops"
  threshold_warnings = { "3" = "" }
  threshold_critical = { "5" = "" }
  include_tags       = true
  notify_no_data     = true
  no_data_timeframe  = 10
  renotify_interval  = 60
  tags               = ["environment:${var.environment}", "team:devops", "eks-monitoring"]
  priority           = 2
}

variables.tf

variable "aws_region" {
  description = "The AWS region where your EKS cluster is located."
  type        = string
  default     = "us-east-1"
}

variable "eks_cluster_name" {
  description = "The name of your existing EKS cluster."
  type        = string
}

variable "datadog_api_key" {
  description = "Datadog API Key."
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key."
  type        = string
  sensitive   = true
}

variable "environment" {
  description = "The environment name (e.g., dev, staging, prod)."
  type        = string
  default     = "dev"
}

terraform.tfvars (Example)

eks_cluster_name = "my-production-eks-cluster"
datadog_api_key  = "YOUR_DATADOG_API_KEY"
datadog_app_key  = "YOUR_DATADOG_APP_KEY"
environment      = "production"

Note on PagerDuty Integration: The example uses @pagerduty-EKS-Critical in the Datadog monitor message. This assumes you have already set up a PagerDuty integration in your Datadog account and named it "EKS-Critical". Datadog's built-in PagerDuty integration supports various service keys and routing rules. For Terraform-managed PagerDuty services, you would use pagerduty_service and pagerduty_service_integration resources, then feed the integration key into your Datadog monitors, or simply rely on the Datadog-managed integration and refer to it by name.

Deployment Steps

1. Initialize Terraform

Navigate to your Terraform project directory and run:

terraform init

2. Review the Plan

Examine the changes Terraform plans to make:

terraform plan

3. Apply the Configuration

Apply the changes. Type yes when prompted:

terraform apply

Upon successful application, the Datadog Agent will be deployed to your EKS cluster, and the specified Datadog monitors will be created.

Verifying the Setup

Datadog Agent Status

Check if the Datadog agent pods are running in your EKS cluster:

kubectl get pods -n datadog

You should see datadog-agent-* pods in a Running state.

Datadog UI Validation

Log into your Datadog account:

Navigate to Infrastructure > Hosts: Your EKS nodes should appear.
Navigate to Monitors > Monitor Status: Your newly created monitors should be listed.
Simulate an alert (e.g., intentionally stress a node) and check if a PagerDuty incident is created.

Incident Response Workflow with PagerDuty

Once a Datadog monitor triggers an alert (e.g., node CPU goes critical), Datadog sends an event to PagerDuty via the configured integration. PagerDuty then:

Creates an Incident: Based on the alert details.
Notifies On-Call Personnel: Following predefined schedules and escalation policies (via SMS, phone, email, push).
Tracks Resolution: Provides a centralized platform for teams to collaborate and resolve the incident.

Optimizing Your Monitoring Strategy and Incident Response

Best Practices for EKS Monitoring

Comprehensive Coverage: Monitor not just EKS, but also underlying AWS services (EC2, EBS, RDS, Load Balancers) and your application logs/traces.
SLOs and SLO-based Alerting: Define Service Level Objectives and create monitors based on these, focusing on user experience rather than just infrastructure health.
Custom Metrics: Instrument your applications to emit custom metrics that provide business-specific insights.
Contextual Alerting: Include relevant links (Datadog dashboards, runbooks, logs) in alert messages to expedite troubleshooting.
Cost Optimization: Regularly review your Datadog usage and adjust data retention or sampling rates if necessary.

Enhancing PagerDuty Incident Response

Detailed Runbooks: Attach specific runbooks to PagerDuty services or even individual incidents to guide responders through common issues.
Automated Remediation: Integrate PagerDuty with automation tools (e.g., AWS Lambda, Ansible, StackStorm) to automatically resolve non-critical issues.
Post-Mortems: Conduct regular post-incident reviews to identify root causes and implement preventive measures.
Testing: Periodically test your on-call rotations and escalation policies to ensure they function as expected.

Troubleshooting Common Issues

Datadog Agent Not Reporting:
- Check kubectl logs <datadog-agent-pod> -n datadog for errors.
- Verify DATADOG_API_KEY and DATADOG_APP_KEY are correctly passed.
- Ensure firewall rules allow outbound traffic from EKS nodes to Datadog endpoints.
Monitors Not Triggering / PagerDuty Not Receiving Incidents:
- Confirm the Datadog monitor query is correct and data is being collected.
- Double-check the @pagerduty-<integration-name> syntax in the monitor message matches the exact name configured in Datadog.
- Verify the PagerDuty integration in Datadog is healthy (Datadog UI -> Integrations -> PagerDuty).
- Check PagerDuty's incident log for incoming events.
Terraform Provider Issues:
- Ensure AWS IAM roles/users have permissions for EKS, Datadog, and PagerDuty resources.
- Verify environment variables for API keys are set correctly if not using .tfvars.

Conclusion

By embracing Terraform for AWS EKS monitoring with Datadog and PagerDuty, you establish a robust, automated, and scalable observability and incident response framework. This IaC approach ensures consistency, reduces manual errors, and empowers your DevOps teams to proactively manage the health and performance of your critical cloud-native applications, ultimately leading to higher availability and quicker resolution of operational issues. Implement these strategies to elevate your EKS operational excellence.

CloudStack YAML