Terraform-Managed AWS EKS Observability with Datadog and PagerDuty Incident Response

Architecture Pro-Tip

For robust, scalable EKS observability, always prioritize an Infrastructure as Code (IaC) approach. Separate your observability tooling configuration (Datadog agents, monitors, PagerDuty services) into dedicated Terraform modules. This promotes reusability, version control, and auditability, ensuring your monitoring and incident response evolve consistently with your EKS clusters. Integrate monitoring from day one, not as an afterthought.

In the dynamic world of cloud-native applications, managing Kubernetes clusters on AWS EKS demands proactive and comprehensive observability. This guide provides a technical deep-dive into leveraging Terraform for automating the deployment and configuration of an end-to-end observability stack for AWS EKS, integrating Datadog for monitoring, logging, and tracing, and PagerDuty for streamlined incident response.

Introduction to Cloud-Native Observability Challenges

AWS EKS offers a powerful platform for running containerized workloads, but its distributed nature introduces significant challenges for understanding system health and performance. Traditional monitoring approaches often fall short in the face of ephemeral pods, dynamic scaling, and complex service meshes. A modern observability strategy for EKS must encompass:

Metrics: Real-time performance data from nodes, pods, containers, and applications.
Logs: Centralized collection and analysis of events across the cluster.
Traces: Distributed transaction monitoring to pinpoint latency and errors across microservices.
Incident Response: Automated alerting and escalation to minimize downtime.

By managing this entire stack with Terraform, we ensure consistency, repeatability, and version control for our critical operational infrastructure.

Core Components and Their Roles

Terraform: Infrastructure as Code for Observability

Terraform, HashiCorp's open-source IaC tool, allows you to define and provision infrastructure using declarative configuration files. For EKS observability, Terraform enables us to:

Deploy and manage the EKS cluster itself.
Install and configure the Datadog Agent via Helm charts.
Define Datadog monitors, dashboards, and integrations.
Set up PagerDuty services, escalation policies, and integrations.

Datadog: Unified Monitoring and Analytics

Datadog provides a comprehensive SaaS platform for monitoring cloud-scale applications. Its capabilities crucial for EKS include:

Infrastructure Monitoring: Collects metrics from EKS nodes, pods, and containers.
Log Management: Aggregates, processes, and analyzes logs from all cluster components and applications.
APM (Application Performance Monitoring): Distributed tracing for microservices running on EKS.
Network Performance Monitoring (NPM): Visibility into network traffic between pods and services.
Synthetic Monitoring: Proactive testing of application availability and performance.

PagerDuty: Intelligent Incident Response

PagerDuty is a leading incident management platform that transforms signals into incidents and orchestrates the appropriate response. Key features for EKS operations include:

Automated On-Call Schedules: Ensures the right person is notified at the right time.
Escalation Policies: Defines rules for escalating incidents if primary responders are unavailable.
Service Directory: Organizes and defines services with clear ownership.
Integrations: Seamlessly connects with monitoring tools like Datadog to trigger incidents.

Prerequisites

Before you begin, ensure you have the following:

AWS Account: With necessary permissions to create EKS clusters and associated resources.
Terraform CLI: Version 1.0 or newer installed.
AWS CLI: Configured with your AWS credentials.
Kubectl: Configured to interact with your EKS cluster.
Helm CLI: Version 3 or newer installed.
Datadog Account: With an API key and Application key.
PagerDuty Account: With an API key and service integration keys.

Step-by-Step Implementation with Terraform

1. Set up AWS EKS Cluster with Terraform

First, you'll need an EKS cluster. While this guide focuses on observability, here's a conceptual overview of a Terraform EKS setup. We'll assume you have an EKS cluster and its Kubeconfig available for Helm deployments.

A typical EKS Terraform module would create the VPC, subnets, EKS control plane, node groups, and associated IAM roles. The output would include the EKS cluster endpoint and certificate authority data, which are used by the Kubernetes and Helm providers.

2. Configure Datadog Integration via Terraform

We will use the Helm provider in Terraform to deploy the Datadog Agent to our EKS cluster and the Datadog provider to define monitors.

2.1. Deploy Datadog Agent with Helm

The Datadog Agent collects metrics, logs, and traces from your EKS cluster. It's deployed as a DaemonSet to ensure an agent runs on every node.

2.2. Terraform Configuration for Datadog Agent & Monitor

Create a `main.tf` file (or separate files for better organization) that will handle the Datadog Agent deployment and a sample monitor. Remember to replace placeholder values like `DD_API_KEY` and `DD_APP_KEY` with your actual Datadog keys.

resource "kubernetes_secret" "datadog_api_key" {
  metadata {
    name      = "datadog-api-key"
    namespace = "datadog" # Ensure this namespace exists or create it
  }

  data = {
    api-key = var.datadog_api_key
  }
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true # Creates the namespace if it doesn't exist

  set {
    name  = "datadog.apiKeyExistingSecret"
    value = kubernetes_secret.datadog_api_key.metadata[0].name
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "datadog.site"
    value = "datadoghq.com" # Or your specific Datadog site, e.g., "eu.datadoghq.com"
  }

  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "clusterAgent.metricsProvider.enabled"
    value = "true"
  }

  set {
    name  = "clusterAgent.kubeStateMetrics.enabled"
    value = "true"
  }

  set {
    name  = "apm.enabled"
    value = "true"
  }

  set {
    name  = "logs.enabled"
    value = "true"
  }

  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }

  set {
    name  = "processAgent.enabled"
    value = "true"
  }

  # Ensure the Kubernetes provider is configured to connect to your EKS cluster
  # Example:
  # provider "kubernetes" {
  #   host                   = aws_eks_cluster.main.endpoint
  #   cluster_ca_certificate = base64decode(aws_eks_cluster.main.certificate_authority[0].data)
  #   token                  = data.aws_eks_cluster_auth.main.token
  # }
}

resource "datadog_monitor" "high_cpu_usage_eks" {
  name                = "[EKS] High CPU Usage on Node {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.cpu.usage.total{kubernetes_cluster_name:your-eks-cluster-name} by {host} > 80"
  message             = "EKS node CPU usage is high! Check {{host.name}}."
  escalation_message  = "CPU usage remains high for more than 15 minutes. Escalating to PagerDuty."
  tags                = ["environment:production", "service:eks", "severity:critical"]
  priority            = 1
  new_group_delay     = 60
  new_host_delay      = 300
  notify_no_data      = false
  renotify_interval   = 0
  no_data_timeframe   = 20

  monitor_thresholds {
    critical = 80
    warning  = 70
  }

  # Assuming a PagerDuty integration is set up in Datadog with a specific name
  # The @pagerduty-integration-name part is crucial for routing to PagerDuty
  # Make sure "pagerduty" is configured as an integration in your Datadog account,
  # and the service you want to alert is linked.
  # For specific PagerDuty service routing, you might need to use specific PagerDuty-Datadog integration keys
  # or tags on the Datadog monitor.
  notify_list = ["@pagerduty-integration-name-for-critical-alerts"] 
}

3. Integrate PagerDuty for Incident Response via Terraform

To fully automate incident response, you can define PagerDuty services and escalation policies using the PagerDuty Terraform provider. This ensures that when a Datadog monitor triggers an alert, PagerDuty knows exactly which team to notify and how.

3.1. Define PagerDuty Service and Escalation Policy

Here's how you might set up a PagerDuty service and a basic escalation policy. You'll need to create an `auth.tf` or similar for your PagerDuty provider configuration with your API token.

# main.tf for PagerDuty resources

# PagerDuty Provider Configuration (e.g., in providers.tf)
# provider "pagerduty" {
#   token = var.pagerduty_api_token
# }

# Example PagerDuty User (needed for escalation policy)
resource "pagerduty_user" "devops_engineer" {
  name  = "DevOps Engineer"
  email = "devops@example.com"
  teams = [pagerduty_team.devops_team.id]
}

# Example PagerDuty Team
resource "pagerduty_team" "devops_team" {
  name        = "DevOps Team"
  description = "Manages EKS infrastructure and applications"
}

# Example On-Call Schedule
resource "pagerduty_schedule" "primary_devops_schedule" {
  name      = "Primary DevOps Schedule"
  time_zone = "America/New_York"
  
  layer {
    name       = "Daily On-Call"
    start      = "2023-01-01T09:00:00-05:00" # ISO 8601 format
    users      = [pagerduty_user.devops_engineer.id]
    rotation_turn_length_seconds = 86400 # 24 hours
    rotation_virtual_start       = "2023-01-01T09:00:00-05:00"
    rotation_type                = "daily"
  }
}

# Example Escalation Policy
resource "pagerduty_escalation_policy" "eks_critical_policy" {
  name      = "EKS Critical Alerts Policy"
  num_loops = 2

  rule {
    delay    = 5
    target {
      type = "schedule"
      id   = pagerduty_schedule.primary_devops_schedule.id
    }
  }

  rule {
    delay    = 10
    target {
      type = "user"
      id   = pagerduty_user.devops_engineer.id
    }
  }
}

# Example PagerDuty Service
resource "pagerduty_service" "eks_observability_service" {
  name                       = "EKS Observability Service"
  auto_resolve_timeout       = "14400" # 4 hours
  acknowledgement_timeout    = "600"   # 10 minutes
  escalation_policy          = pagerduty_escalation_policy.eks_critical_policy.id
}

# Create a Datadog integration for this PagerDuty service
# This is where Datadog sends alerts to PagerDuty
resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog"
  service_id  = pagerduty_service.eks_observability_service.id
  type        = "datadog_v2" # Use datadog_v2 for modern Datadog integrations
}

output "pagerduty_datadog_integration_key" {
  description = "The integration key for Datadog to send alerts to PagerDuty"
  value       = pagerduty_service_integration.datadog_integration.integration_key
  sensitive   = true
}

After applying this Terraform, you'll get an `integration_key` output. You need to configure this key within Datadog's PagerDuty integration settings, or directly within the Datadog monitor's `notify_list` (e.g., `@pagerduty-${pagerduty_service_integration.datadog_integration.integration_key}`). Note that directly embedding the integration key in the monitor requires careful handling of secrets and might be better managed by linking Datadog integrations via names in the `notify_list` as shown in the Datadog monitor example earlier, after setting up the integration in Datadog UI or via the Datadog provider itself for `datadog_integration_pagerduty`.

4. Apply Terraform Configurations

Once your Terraform files are ready:

Initialize Terraform: terraform init
Review the plan: terraform plan
Apply the changes: terraform apply

This will deploy the Datadog Agent, create the Datadog monitor, and configure the PagerDuty service and integration.

Testing and Validation

After applying your Terraform, it's crucial to validate the setup:

Datadog Agent: Check your Datadog UI under Infrastructure > Hosts. You should see your EKS nodes and pods reporting data. Verify metrics, logs, and traces are being ingested.
Datadog Monitor: In Datadog, navigate to Monitors > Manage Monitors and confirm your high_cpu_usage_eks monitor is active.
PagerDuty Service: In PagerDuty, confirm the new service, schedule, and escalation policy are correctly configured.
Incident Flow: To test the full flow, you can intentionally trigger the CPU alert (e.g., by running a busybox pod with a CPU-intensive command). Verify that Datadog triggers an alert and PagerDuty creates an incident and notifies the on-call team.

Best Practices and Advanced Considerations

Modularize Terraform: Break down your Terraform configuration into logical modules (e.g., `eks`, `datadog-agent`, `datadog-monitors`, `pagerduty-services`) for better organization and reusability.
Secrets Management: Do not hardcode API keys. Use a secure method like AWS Secrets Manager, HashiCorp Vault, or environment variables in your CI/CD pipeline for sensitive credentials.
Custom Metrics & Logs: For application-specific metrics or logs, configure Datadog custom integrations or enrich logs with processors via Terraform.
Runbooks: For each PagerDuty service or critical Datadog monitor, link to a comprehensive runbook. This guides responders on how to troubleshoot and resolve issues efficiently.
Automated Remediation: Explore integrating PagerDuty with automation tools (e.g., AWS Lambda, Ansible) to trigger automated remediation steps for well-understood incidents.
Cost Optimization: Monitor Datadog ingestion volumes to manage costs. Use features like log exclusion filters and metric cardinality management.

Troubleshooting and FAQ

Q: Datadog Agent isn't reporting any data. What should I check?

A: Verify the `datadog` namespace exists and the Helm chart deployed successfully (`kubectl get pods -n datadog`). Check the Datadog Agent pods for errors (`kubectl logs -f datadog-agent-xxxx -n datadog`). Ensure your `DD_API_KEY` is correctly set and has permissions to send data to Datadog.

Q: PagerDuty incidents are not being triggered from Datadog.

A: Confirm the Datadog-PagerDuty integration is correctly set up in both platforms. In Datadog, go to Integrations > PagerDuty and ensure it's enabled and configured with the correct PagerDuty service integration key. Verify the `notify_list` in your Datadog monitor explicitly references the PagerDuty integration name or key.

Q: My Terraform `helm_release` for Datadog Agent fails.

A: Ensure your Kubernetes provider in Terraform is correctly configured to connect to your EKS cluster. This typically involves passing the cluster endpoint, CA certificate, and a valid token. Check the `helm_release` logs for specific errors from Helm.

Conclusion

Establishing robust observability for AWS EKS is non-negotiable for maintaining healthy and performant cloud-native applications. By harnessing the power of Terraform, Datadog, and PagerDuty, organizations can create a resilient, automated, and scalable system for monitoring, alerting, and incident response. This IaC approach not only streamlines deployment but also fosters collaboration, reduces manual errors, and ensures that your EKS clusters are always under vigilant watch, allowing your teams to focus on innovation rather than firefighting.

CloudStack YAML