Automating AWS EKS Observability and Incident Response with Terraform, Datadog, and PagerDuty

In the dynamic world of cloud-native applications, managing Kubernetes clusters can be a complex endeavor. AWS EKS (Elastic Kubernetes Service) simplifies cluster operations, but ensuring robust observability and a streamlined incident response remains crucial for maintaining application reliability and performance. This guide provides a comprehensive technical walkthrough on how to leverage Terraform for infrastructure as code, Datadog for deep monitoring and alerting, and PagerDuty for efficient incident management, transforming your EKS operations from reactive firefighting to proactive SRE.

Architecture Pro-Tip

Always design your observability stack with a 'shift-left' mindset. Integrate monitoring and alerting configurations directly into your Infrastructure as Code (IaC) pipelines. This ensures that as new EKS services or applications are deployed, their corresponding observability measures and incident response protocols are automatically provisioned, reducing configuration drift and manual errors. Prioritize unified dashboards and cross-platform correlation for faster root cause analysis.

Why Automate EKS Observability and Incident Response?

Operating mission-critical applications on AWS EKS requires a sophisticated approach to understanding system health and responding to disruptions. Manual configuration of monitoring agents, alert thresholds, and incident escalation rules is not only error-prone but also scales poorly with growing infrastructure. Automation through Terraform, Datadog, and PagerDuty offers:

Consistency: Standardized configurations across all EKS clusters and environments.
Scalability: Easily extend observability and response capabilities as your EKS footprint grows.
Speed: Rapid deployment of new monitors and incident rules.
Auditability: All changes are version-controlled and traceable within your Git repository.
Reliability: Reduced human error in critical configurations.

Core Components Explained

Terraform: Infrastructure as Code for Your Observability Stack

Terraform, by HashiCorp, allows you to define and provision infrastructure using a declarative configuration language. For EKS observability, Terraform excels at:

Deploying the Datadog Agent to your EKS clusters via Helm or Kubernetes manifests.
Configuring Datadog monitors, dashboards, and integration settings.
Managing PagerDuty services, escalation policies, and user teams.
Establishing the crucial link between Datadog alerts and PagerDuty incidents.

Datadog: Unified Observability for EKS

Datadog provides an end-to-end observability platform, collecting metrics, logs, and traces from your entire EKS environment. Key Datadog features for EKS include:

Container Monitoring: Detailed metrics for pods, nodes, deployments, and services.
Log Management: Centralized collection and analysis of EKS control plane and application logs.
APM (Application Performance Monitoring): Distributed tracing for microservices running on EKS.
Network Performance Monitoring: Visibility into network traffic between Kubernetes services.
Alerting & Detection: Sophisticated anomaly detection and threshold-based alerts.
Dashboarding: Customizable visualizations for real-time system health.

PagerDuty: Intelligent Incident Management

PagerDuty transforms monitoring signals into actionable incidents, ensuring the right people are notified at the right time. For EKS incident response, PagerDuty offers:

On-Call Management: Dynamic scheduling and escalation policies.
Automated Incident Creation: Triggering incidents from Datadog alerts.
Communication: Facilitating immediate communication via SMS, phone, email, and push notifications.
Post-Mortem Tools: Helping teams learn from incidents and prevent recurrence.

Step-by-Step Implementation Guide

Prerequisites

Before you begin, ensure you have the following:

An active AWS Account with an existing EKS cluster.
Terraform CLI installed and configured with AWS credentials.
A Datadog Account with an API Key and Application Key.
A PagerDuty Account with an API Key.
kubectl and Helm CLI installed and configured to connect to your EKS cluster.

1. Configure Terraform Providers

Start by defining the required Terraform providers in your main.tf:

provider "aws" {
  region = "us-east-1"
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.my_cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.my_cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.my_cluster.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.my_cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.my_cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.my_cluster.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

data "aws_eks_cluster" "my_cluster" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "my_cluster" {
  name = var.eks_cluster_name
}

Ensure you have variables defined for eks_cluster_name, datadog_api_key, datadog_app_key, and pagerduty_api_token, ideally managed via a secure secrets manager like AWS Secrets Manager or Vault.

2. Deploy Datadog Agent to EKS

The Datadog Agent can be deployed as a DaemonSet on your EKS cluster using the Helm provider. This collects metrics, logs, and traces from your nodes and pods.

resource "kubernetes_secret" "datadog_api_key" {
  metadata {
    name      = "datadog-api-key"
    namespace = "datadog" # Ensure this namespace exists or is created
  }
  data = {
    "api-key" = var.datadog_api_key
  }
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  version    = "2.33.2" # Use a specific version
  
  set {
    name  = "datadog.site"
    value = "datadoghq.com" # Or eu.datadoghq.com, etc.
  }
  set {
    name  = "datadog.apiKeyExistingSecret"
    value = kubernetes_secret.datadog_api_key.metadata.0.name
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }
  set {
    name  = "kubeStateMetricsExternal.enabled"
    value = "true"
  }
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "clusterAgent.metricsProvider.enabled"
    value = "true"
  }
  set {
    name  = "datadog.logs.enabled"
    value = "true"
  }
  set {
    name  = "datadog.logs.containerCollectAll"
    value = "true"
  }
  set {
    name  = "datadog.apm.enabled"
    value = "true"
  }
  set {
    name  = "datadog.processAgent.enabled"
    value = "true"
  }
  set {
    name  = "datadog.processAgent.containerCollection.enabled"
    value = "true"
  }
}

3. Configure PagerDuty Service and Escalation Policies

Before Datadog can send alerts, PagerDuty needs a service to receive them. You'll define a service, potentially an escalation policy, and integrate with Datadog.

resource "pagerduty_user" "devops_engineer" {
  name  = "DevOps Engineer"
  email = "devops@example.com"
}

resource "pagerduty_team" "sre_team" {
  name        = "SRE Team"
  description = "Site Reliability Engineering Team"
}

resource "pagerduty_team_membership" "sre_membership" {
  user_id = pagerduty_user.devops_engineer.id
  team_id = pagerduty_team.sre_team.id
  role    = "admin" # or "member", "observer"
}

resource "pagerduty_escalation_policy" "eks_critical_policy" {
  name      = "EKS Critical Policy"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "user"
      id   = pagerduty_user.devops_engineer.id
    }
  }

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "team"
      id   = pagerduty_team.sre_team.id
    }
  }
}

resource "pagerduty_service" "eks_observability_service" {
  name                    = "EKS Observability"
  auto_resolve_timeout    = "14400" # 4 hours
  acknowledgement_timeout = "600"   # 10 minutes
  escalation_policy       = pagerduty_escalation_policy.eks_critical_policy.id
  description             = "Monitors critical EKS infrastructure health"
}

resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog Integration"
  service_id  = pagerduty_service.eks_observability_service.id
  type        = "generic_events_api_inbound_integration"
  vendor      = "Datadog" # Optional, for better display in PagerDuty UI
}

4. Integrate Datadog with PagerDuty

Now, you need to tell Datadog how to send alerts to PagerDuty. This is done via the datadog_integration_pagerduty resource.

resource "datadog_integration_pagerduty" "main_integration" {
  api_key = pagerduty_service_integration.datadog_integration.integration_key
  name    = pagerduty_service.eks_observability_service.name
  # The service_name here refers to the PagerDuty service name, NOT the Datadog integration name
  # service_name = pagerduty_service.eks_observability_service.name 
  # This field is deprecated and not directly needed for generic events API
  # The integration_key is what links them.
}

Important: The api_key for the Datadog PagerDuty integration is actually the integration_key from the PagerDuty service integration you created. This is a common point of confusion.

5. Create Datadog Monitors for EKS

With the agent deployed and the PagerDuty integration set up, you can now define Datadog monitors to detect issues and trigger incidents.

Example: EKS Node CPU Utilization Monitor

This monitor will alert if any EKS node's CPU utilization exceeds 80% for more than 5 minutes.

resource "datadog_monitor" "eks_node_cpu_utilization" {
  name               = "[EKS] High Node CPU Utilization on {{host.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster_name:${var.eks_cluster_name}} by {host} < 20"
  message            = "CPU utilization on node {{host.name}} is above 80% for 5 minutes. @pagerduty-${pagerduty_service.eks_observability_service.name}"
  
  tags = [
    "environment:${var.environment}",
    "service:eks-control-plane",
    "severity:high"
  ]

  monitor_thresholds {
    critical = 80
    warning  = 70
  }

  include_tags = true
  renotify_interval = 30 # Renotify every 30 minutes if unresolved
  notify_no_data    = false
  require_full_window = false
}

Notice the @pagerduty-${pagerduty_service.eks_observability_service.name} in the message. This tells Datadog to send the alert to the PagerDuty service named "EKS Observability" (or whatever you named your PagerDuty service).

Example: EKS Pod Restarts Monitor

Monitors for excessive pod restarts, indicating application instability.

resource "datadog_monitor" "eks_pod_restarts" {
  name               = "[EKS] High Pod Restarts for {{kube_container_name.name}}"
  type               = "metric alert"
  query              = "sum(last_5m):sum:kubernetes.container.restarts{kubernetes_cluster_name:${var.eks_cluster_name}} by {kube_container_name} > 3"
  message            = "Container {{kube_container_name.name}} has restarted more than 3 times in 5 minutes. Investigate application stability. @pagerduty-${pagerduty_service.eks_observability_service.name}"
  
  tags = [
    "environment:${var.environment}",
    "service:application",
    "severity:high"
  ]

  monitor_thresholds {
    critical = 3
  }

  include_tags = true
  renotify_interval = 60
}

6. Apply Terraform Configuration

Once all your .tf files are configured:

Run terraform init to initialize the working directory and download providers.
Run terraform plan to review the changes Terraform will make.
Run terraform apply to provision your resources. Confirm with yes.

Testing and Validation

After applying the Terraform configuration:

Verify Datadog Agent: Check your Datadog account under "Infrastructure" -> "Container Map" or "Hosts" to ensure EKS nodes and pods are reporting data.
Verify Datadog Monitors: Navigate to "Monitors" in Datadog to confirm your newly created monitors are present and evaluating. You can manually trigger a test alert if needed.
Verify PagerDuty Integration: In PagerDuty, check your "EKS Observability" service. You should see a Datadog integration listed. Simulate an incident (e.g., by intentionally overloading a test EKS node) and confirm an incident is created in PagerDuty and notifications are sent as per your escalation policy.

Advanced Strategies and Best Practices

Tagging Strategy: Implement a consistent tagging strategy across AWS resources, EKS objects, and Datadog monitors. This allows for powerful filtering, aggregation, and cost allocation.
Custom Metrics & Checks: Extend Datadog's capabilities with custom metrics from your applications and custom health checks for specific services.
Log Management: Fully integrate EKS control plane logs (API server, controller manager, scheduler, authenticator) and application logs into Datadog for comprehensive troubleshooting.
APM and Tracing: Instrument your EKS applications with Datadog APM for distributed tracing, enabling deep visibility into request flows across microservices.
Synthetic Monitoring: Use Datadog Synthetics to proactively test your EKS application's endpoints and user journeys from various global locations.
Automated Remediation: Explore integrating PagerDuty with automation tools (like Rundeck or AWS Systems Manager Automation) to trigger self-healing runbooks for common EKS issues.
GitOps Workflow: Store all your Terraform configurations in Git and use a CI/CD pipeline (e.g., GitLab CI/CD, GitHub Actions, Atlantis) to automate terraform plan and apply operations, enforcing a true GitOps model for your observability stack.

Troubleshooting Common Issues

Here are some common issues and their solutions:

Datadog Agent Not Reporting:
- Ensure the Datadog API key is correct and accessible to the agent.
- Check EKS node security groups and Network ACLs to allow outbound traffic to Datadog endpoints.
- Inspect Datadog Agent pod logs (kubectl logs -n datadog -l app=datadog --all-containers) for errors.
- Verify the datadog.site parameter in the Helm chart matches your Datadog region.
PagerDuty Incidents Not Triggering:
- Confirm the @pagerduty-... tag in your Datadog monitor message exactly matches the PagerDuty service name.
- Ensure the datadog_integration_pagerduty resource in Terraform is correctly configured with the PagerDuty integration key.
- Check the PagerDuty service for any global event rules or throttling.
Terraform EKS Provider Authentication Errors:
- Ensure your AWS CLI is configured with credentials that have permission to describe EKS clusters and create authentication tokens.
- Verify aws-iam-authenticator is installed and in your PATH.

Conclusion

Automating AWS EKS observability and incident response with Terraform, Datadog, and PagerDuty is a strategic investment in the reliability and stability of your cloud-native infrastructure. By treating your monitoring and incident management as code, you gain unparalleled consistency, scalability, and auditability. This holistic approach empowers your DevOps and SRE teams to move beyond reactive incident handling, fostering a proactive culture that drives continuous improvement and delivers a superior user experience.

Start implementing these practices today to build a more resilient and observable EKS environment.

CloudStack YAML