Terraform-Managed End-to-End Observability for AWS EKS with Datadog and PagerDuty

In the dynamic world of cloud-native applications, maintaining robust observability for Kubernetes clusters is paramount. AWS Elastic Kubernetes Service (EKS) provides a powerful foundation, but effectively monitoring its health, performance, and application behavior requires a comprehensive strategy. This guide details how to implement an end-to-end observability solution for AWS EKS using Terraform for infrastructure as code (IaC), Datadog for comprehensive monitoring and logging, and PagerDuty for incident response and alerting. By integrating these tools, you'll establish a proactive system that ensures the stability and reliability of your EKS workloads.

Architecture Pro-Tip

Always design your observability stack with a "shift-left" mindset. Integrate monitoring and alerting definitions directly into your Infrastructure as Code (IaC) alongside your application and infrastructure deployments. This ensures that every new service or component deployed via Terraform automatically comes with its respective observability configuration, preventing blind spots and maintaining consistency across environments. Leverage Terraform modules to encapsulate common observability patterns for reusability.

Why Terraform for Observability?

Managing complex cloud environments like AWS EKS necessitates a declarative approach. Terraform, as the leading IaC tool, brings significant advantages to observability:

Consistency: Define Datadog monitors, dashboards, and PagerDuty services alongside your EKS cluster and applications, ensuring consistent monitoring across environments.
Version Control: Treat your observability configuration like application code. Store it in Git, enabling versioning, peer review, and audit trails.
Automation: Automate the deployment and updates of monitoring agents, dashboards, and alerting rules, reducing manual effort and potential errors.
Scalability: Easily replicate your observability setup for new clusters or microservices as your architecture grows.
Drift Detection: Terraform helps identify and rectify configuration drift in your observability stack, ensuring desired states are maintained.

Core Components Overview

AWS EKS: The Foundation

AWS EKS provides a managed Kubernetes control plane, simplifying the deployment, management, and scaling of containerized applications. Our observability solution will focus on collecting metrics, logs, and traces from the EKS cluster, its nodes, and the applications running within it.

Datadog: The Observability Platform

Datadog offers a unified platform for monitoring, logging, and tracing across your entire stack. For EKS, it provides:

Datadog Agent: A lightweight agent deployed as a DaemonSet on EKS nodes, collecting infrastructure metrics, logs, and APM traces from pods and containers.
EKS Integration: Specific integrations for EKS control plane metrics, Fargate, and other AWS services.
Monitors & Alerts: Configurable alerts based on collected data, with various notification channels.
Dashboards: Customizable visualizations for quick insights into cluster and application health.
Log Management: Centralized log aggregation and analysis.
APM & Distributed Tracing: End-to-end visibility into application performance.

PagerDuty: Incident Management and On-Call

PagerDuty acts as the central hub for incident response, transforming Datadog alerts into actionable incidents. Key features include:

On-Call Management: Automated scheduling and escalation policies.
Incident Routing: Directing alerts to the right teams and individuals based on context.
Notification Channels: SMS, phone calls, email, and push notifications for critical alerts.
Reporting & Analytics: Post-incident analysis to improve response times.

Prerequisites

Before diving into the Terraform configuration, ensure you have the following:

AWS Account: With necessary permissions to create/manage EKS, IAM roles, and other resources.
AWS EKS Cluster: An existing EKS cluster (or you can create one using Terraform within your project).
Datadog Account: With API and Application keys.
PagerDuty Account: With API key and a service already set up (or we can create one with Terraform).
Terraform CLI: Installed and configured on your local machine.
Kubectl: Configured to connect to your EKS cluster.
Helm CLI: Used by the Terraform Helm provider to deploy the Datadog agent.

Step-by-Step Implementation with Terraform

We will structure our Terraform project to manage the Datadog Agent deployment, Datadog monitors, PagerDuty service, and its integration.

1. Configure Terraform Providers

Define the AWS, Datadog, PagerDuty, and Kubernetes/Helm providers.

2. Deploy Datadog Agent to EKS

The Datadog Agent is typically deployed via a Helm chart. Terraform's Helm provider facilitates this.

3. Configure Datadog Monitors and Dashboards

Define critical monitors for EKS health, node utilization, pod restarts, etc., using the Datadog Terraform provider.

4. Set up PagerDuty Service and Integration

Create a PagerDuty service and an integration that will receive alerts from Datadog.

5. Connect Datadog Alerts to PagerDuty

Establish the notification channel in Datadog that points to your PagerDuty integration.

Ready-to-Use Terraform Configuration Example

Here’s a consolidated example of a main.tf that orchestrates the described observability setup. Remember to replace placeholder values with your actual credentials and cluster details.

provider "aws" {
  region = var.aws_region
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  token                  = data.aws_eks_cluster_auth.cluster.token
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    token                  = data.aws_eks_cluster_auth.cluster.token
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

# --- AWS EKS Cluster Data (assuming existing cluster) ---
data "aws_eks_cluster" "cluster" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = var.eks_cluster_name
}

# --- PagerDuty Configuration ---
resource "pagerduty_user" "devops_team_lead" {
  name  = "DevOps Team Lead"
  email = "devops_lead@example.com"
  teams = [pagerduty_team.devops_team.id]
}

resource "pagerduty_team" "devops_team" {
  name        = "DevOps Engineering"
  description = "Manages infrastructure and platform reliability"
}

resource "pagerduty_schedule" "devops_oncall_schedule" {
  name      = "DevOps Primary On-Call"
  time_zone = "America/Los_Angeles"

  layer {
    name       = "Daily Layer"
    start      = "2023-01-01T09:00:00-08:00"
    rotation_turn_length_seconds = 86400 # 24 hours
    users      = [pagerduty_user.devops_team_lead.id] # Example, add more users for a real schedule
  }
}

resource "pagerduty_escalation_policy" "devops_escalation_policy" {
  name      = "DevOps Critical Alert Escalation"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "user"
      id   = pagerduty_user.devops_team_lead.id
    }
  }
  rule {
    escalation_delay_in_minutes = 20
    target {
      type = "schedule"
      id   = pagerduty_schedule.devops_oncall_schedule.id
    }
  }
}

resource "pagerduty_service" "eks_monitoring_service" {
  name                    = "EKS Core Monitoring"
  auto_resolve_timeout_s  = 14400 # 4 hours
  acknowledgement_timeout_s = 600   # 10 minutes
  escalation_policy       = pagerduty_escalation_policy.devops_escalation_policy.id
}

resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog"
  service_id  = pagerduty_service.eks_monitoring_service.id
  type        = "datadog_inbound_integration"
}

# --- Datadog Agent Deployment (Helm) ---
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
  }
  set {
    name  = "datadog.kubelet.host"
    value = data.aws_eks_cluster.cluster.endpoint # For EKS control plane metrics
  }
  set {
    name  = "datadog.logs.enabled"
    value = "true"
  }
  set {
    name  = "datadog.logs.containerCollectAll"
    value = "true"
  }
  set {
    name  = "datadog.apm.enabled"
    value = "true"
  }
  set {
    name  = "datadog.processAgent.enabled"
    value = "true"
  }
  # Enable PagerDuty notification channel in Datadog for this setup
  set {
    name  = "datadog.site"
    value = "datadoghq.com" # or eu.datadoghq.com, etc.
  }
}

# --- Datadog Monitors ---
# PagerDuty integration ID (needs to be configured manually or pulled dynamically)
# For simplicity, we assume an existing PagerDuty integration in Datadog named "PagerDuty EKS Alerts"
# A robust solution might involve using `datadog_integration_pagerduty` if you manage PagerDuty integrations via Datadog's API.
# For now, we assume this is pre-configured or manually set up in Datadog UI
# You'd typically find the ID by making an API call or configuring it in Datadog UI and referencing its name.

resource "datadog_monitor" "eks_node_cpu_high" {
  name                = "[EKS] High Node CPU Usage on {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.eks_cluster_name}} by {host} > 80"
  message             = "EKS node {{host.name}} CPU usage is above 80% for 5 minutes. @pagerduty-eks-alerts"
  tags                = ["environment:${var.environment}", "service:eks", "alert-type:critical"]
  require_full_window = false
  notify_no_data      = false
  new_group_delay     = 60
  no_data_timeframe   = 20
  escalation_message  = "CPU usage remains high. Paging on-call engineer."

  # Reference the PagerDuty integration in Datadog.
  # This requires the PagerDuty integration to be configured in Datadog beforehand.
  # The `@pagerduty-eks-alerts` in the message directly maps to a PagerDuty integration configured in Datadog with that name.
}

resource "datadog_monitor" "eks_node_memory_high" {
  name                = "[EKS] High Node Memory Usage on {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.mem.usage.pct{cluster_name:${var.eks_cluster_name}} by {host} > 85"
  message             = "EKS node {{host.name}} memory usage is above 85% for 5 minutes. @pagerduty-eks-alerts"
  tags                = ["environment:${var.environment}", "service:eks", "alert-type:critical"]
  require_full_window = false
  notify_no_data      = false
  new_group_delay     = 60
  no_data_timeframe   = 20
}

resource "datadog_monitor" "kubernetes_pod_restarts" {
  name                = "[EKS] Frequent Pod Restarts in {{kube_namespace.name}}/{{kube_app.name}}"
  type                = "query alert"
  query               = "sum(last_5m):kube_container_restarts{cluster_name:${var.eks_cluster_name}} by {kube_namespace,kube_app} > 3"
  message             = "Pod {{kube_app.name}} in namespace {{kube_namespace.name}} is restarting frequently. @pagerduty-eks-alerts"
  tags                = ["environment:${var.environment}", "service:eks", "alert-type:warning"]
  require_full_window = false
  notify_no_data      = false
  new_group_delay     = 60
  no_data_timeframe   = 20
}
    

Variables (`variables.tf`)

variable "aws_region" {
  description = "AWS region for EKS cluster."
  type        = string
  default     = "us-east-1"
}

variable "eks_cluster_name" {
  description = "Name of the existing EKS cluster."
  type        = string
}

variable "datadog_api_key" {
  description = "Datadog API Key."
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key."
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token."
  type        = string
  sensitive   = true
}

variable "environment" {
  description = "Environment tag for resources (e.g., dev, staging, prod)."
  type        = string
  default     = "dev"
}
    

Outputs (`outputs.tf`)

output "pagerduty_service_url" {
  description = "URL to the PagerDuty EKS Monitoring Service."
  value       = pagerduty_service.eks_monitoring_service.html_url
}

output "pagerduty_datadog_integration_url" {
  description = "URL to the Datadog integration within PagerDuty."
  value       = pagerduty_service_integration.datadog_integration.html_url
}

output "datadog_node_cpu_monitor_id" {
  description = "ID of the Datadog Node CPU monitor."
  value       = datadog_monitor.eks_node_cpu_high.id
}
    

Deployment Steps:

Save the above code into files named main.tf, variables.tf, and outputs.tf in a directory.
Create a terraform.tfvars file with your sensitive variables:
eks_cluster_name = "my-eks-cluster" datadog_api_key = "YOUR_DATADOG_API_KEY" datadog_app_key = "YOUR_DATADOG_APP_KEY" pagerduty_api_token = "YOUR_PAGERDUTY_API_TOKEN"
Initialize Terraform: terraform init
Review the plan: terraform plan
Apply the configuration: terraform apply

Testing and Validation

After applying the Terraform configuration:

Datadog Agent: Verify the Datadog Agent pods are running in your datadog namespace: kubectl get pods -n datadog. Check the Datadog UI for EKS integration data, host metrics, and logs flowing in.
Datadog Monitors: Navigate to Datadog's Monitors section to confirm your new monitors are listed and in an OK state (if no issues).
PagerDuty Service: Check your PagerDuty account to ensure the new service, escalation policy, and integration are created.
Trigger an Alert: For testing, you might temporarily lower a monitor threshold (e.g., CPU to 1%) or induce a load on a node to trigger an alert and verify PagerDuty receives it.

Advanced Considerations

This guide provides a foundational setup. Consider these enhancements for a production-grade solution:

Custom Metrics: Instrument your applications to send custom metrics to Datadog.
APM and Tracing: Integrate Datadog APM into your application code for distributed tracing.
Synthetics Monitoring: Use Datadog Synthetics to proactively test application endpoints and user journeys.
Security Monitoring: Leverage Datadog Cloud SIEM for threat detection and compliance.
Log Management: Fine-tune log collection, processing, and retention policies in Datadog.
Automated Remediation: Explore integrating PagerDuty with automation tools (e.g., AWS Lambda, Ansible) to initiate automatic remediation steps for certain alerts.
Terraform Modules: Create reusable Terraform modules for your Datadog monitors, PagerDuty services, and agent deployments to maintain consistency across multiple EKS clusters or teams.

Troubleshooting and Best Practices

Common Troubleshooting Steps:

Datadog Agent Pods Not Running: Check pod logs (kubectl logs -f <datadog-agent-pod> -n datadog) for configuration errors, missing API/APP keys, or insufficient permissions.
No Data in Datadog: Verify network connectivity from EKS nodes to Datadog endpoints. Ensure the correct Datadog site (datadoghq.com vs. eu.datadoghq.com) is configured in the Helm chart.
PagerDuty Not Receiving Alerts: Double-check the Datadog monitor message for the correct @pagerduty-integration-name syntax. Ensure the PagerDuty integration in Datadog is correctly configured and active. Verify your PagerDuty API token.
Terraform Authentication Issues: Ensure your AWS CLI and Kubernetes contexts are correctly configured for Terraform to interact with your EKS cluster.

Best Practices:

Secrets Management: Use AWS Secrets Manager or HashiCorp Vault to store your Datadog and PagerDuty API keys, integrating them securely with Terraform.
Granular Permissions: Apply the principle of least privilege to IAM roles used by the Datadog Agent and for Terraform deployments.
Tagging: Consistently tag all your AWS, Datadog, and PagerDuty resources for better organization, cost allocation, and filtering.
Review and Refine Alerts: Regularly review and fine-tune your Datadog monitors to minimize alert fatigue and ensure they are actionable.
Dedicated Observability Team: For larger organizations, consider a dedicated team or individual responsible for maintaining and optimizing the observability stack.

Conclusion

Establishing robust, end-to-end observability for AWS EKS is critical for operational excellence. By leveraging Terraform for IaC, Datadog for comprehensive monitoring, and PagerDuty for intelligent incident management, you can build a resilient system that provides deep insights into your Kubernetes workloads and ensures rapid response to critical issues. This declarative approach streamlines deployment, enhances consistency, and ultimately leads to more stable and reliable cloud-native applications.

Embrace this powerful combination to elevate your EKS operations and deliver exceptional service to your users.

CloudStack YAML