Terraform for Production-Ready AWS EKS Observability with Datadog and PagerDuty Integration

In today's dynamic cloud-native landscape, ensuring the reliability and performance of Kubernetes clusters is paramount for any production environment. AWS Elastic Kubernetes Service (EKS) provides a robust platform, but true production readiness demands comprehensive observability. This guide delves into leveraging Terraform to provision and manage a state-of-the-art observability stack for EKS, integrating Datadog for holistic monitoring and alerting, and PagerDuty for streamlined incident response.

Architecture Pro-Tip

Always design your observability stack alongside your application infrastructure. Embed monitoring agents, trace collectors, and log forwarders directly into your Infrastructure as Code (IaC) templates. This ensures consistency, reduces configuration drift, and accelerates time-to-insight for issues, making observability a first-class citizen in your production deployment pipeline.

The Pillars of Production Observability for EKS

A truly observable EKS cluster means having deep insights into its health, performance, and behavior across several dimensions:

Metrics: Quantitative measurements reflecting the state of your cluster, nodes, pods, and applications (CPU utilization, memory usage, network I/O, request latency, error rates).
Logs: Structured or unstructured textual records of events occurring within your cluster, essential for debugging and understanding specific incidents.
Traces: End-to-end views of requests as they flow through distributed microservices, crucial for identifying bottlenecks and latency issues in complex applications.
Events: Significant occurrences within Kubernetes (e.g., pod scheduling failures, image pull errors, service scaling).

Datadog excels at unifying these data types, providing a single pane of glass for monitoring, while PagerDuty ensures critical alerts lead to prompt human intervention.

Why Terraform, Datadog, and PagerDuty for EKS?

Terraform: Infrastructure as Code (IaC)

Terraform allows you to define and provision your entire cloud infrastructure, including EKS, its associated resources, and your observability tooling, using declarative configuration files. This brings numerous benefits:

Automation: Eliminate manual configuration errors and accelerate deployments.
Consistency: Ensure identical environments across development, staging, and production.
Version Control: Track changes, roll back configurations, and collaborate effectively.
Repeatability: Spin up and tear down environments with ease.

Datadog: Unified Monitoring and Analytics

Datadog offers a powerful cloud-native monitoring platform that aggregates metrics, logs, and traces from your EKS clusters, applications, and underlying AWS infrastructure. Key features include:

EKS Integration: Deep visibility into Kubernetes components (kube-state-metrics, cAdvisor, control plane metrics).
APM & Distributed Tracing: Understand application performance across microservices.
Log Management: Collect, process, and analyze logs from all sources.
Dashboards & Alerts: Create custom visualizations and define intelligent alerts with anomaly detection.
Integrations: Seamlessly connect with AWS services, other third-party tools, and incident management platforms like PagerDuty.

PagerDuty: Incident Management and On-Call Automation

When incidents strike, PagerDuty ensures the right people are notified at the right time. Its capabilities include:

On-Call Schedules: Manage complex rotations and escalations.
Incident Routing: Automatically direct alerts to the appropriate team or individual.
Automated Notifications: Deliver alerts via SMS, phone, email, and push notifications.
Rich Integrations: Connects with monitoring tools (like Datadog) to trigger incidents automatically.
Incident Response Playbooks: Streamline resolution processes.

Prerequisites

Before you begin, ensure you have the following:

An AWS Account with administrative access.
Terraform CLI (v1.0+) installed.
AWS CLI configured with appropriate credentials.
Kubectl installed and configured (to verify EKS cluster).
A Datadog Account with an API key and Application key.
A PagerDuty Account with an API key.
An existing AWS EKS Cluster or the ability to create one via Terraform. (This guide focuses on observability setup *after* EKS cluster creation, though Terraform can create the EKS cluster itself).

Terraform Implementation: Orchestrating EKS Observability

We'll structure our Terraform project to provision the necessary Datadog agents, configure monitors, and integrate with PagerDuty.

1. Project Structure

A typical Terraform project structure:

.
├── main.tf
├── variables.tf
├── outputs.tf
└── providers.tf
    

2. Configure Terraform Providers

Define the AWS, Kubernetes, Helm, Datadog, and PagerDuty providers. The Kubernetes and Helm providers will need to authenticate against your EKS cluster.

# providers.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
    pagerduty = {
      source  = "PagerDuty/pagerduty"
      version = "~> 1.0"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

# Configure Kubernetes provider to connect to EKS
data "aws_eks_cluster" "eks" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "eks" {
  name = var.eks_cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.eks.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.eks.token
}

# Configure Helm provider to connect to Kubernetes
provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.eks.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.eks.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}
    

3. Deploy Datadog Agent to EKS

The Datadog Agent is typically deployed as a DaemonSet to collect metrics, logs, and traces from all nodes and pods in your EKS cluster. We'll use the Helm provider for this.

# main.tf - Datadog Agent Deployment
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    type  = "string"
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    type  = "string"
  }
  set {
    name  = "clusterAgent.enabled"
    value = "true"
    type  = "bool"
  }
  set {
    name  = "clusterChecksRunner.enabled"
    value = "true"
    type  = "bool"
  }
  set {
    name  = "agents.tolerations[0].key"
    value = "node-role.kubernetes.io/master"
  }
  set {
    name  = "agents.tolerations[0].operator"
    value = "Exists"
  }
  set {
    name  = "agents.tolerations[0].effect"
    value = "NoSchedule"
  }
  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
    type  = "bool"
  }
  set {
    name  = "datadog.logLevel"
    value = "INFO"
  }
  set {
    name  = "logs.enabled"
    value = "true"
    type  = "bool"
  }
  set {
    name  = "logs.containerCollectAll"
    value = "true"
    type  = "bool"
  }
  set {
    name  = "processAgent.enabled"
    value = "true"
    type  = "bool"
  }
  set {
    name  = "apm.enabled"
    value = "true"
    type  = "bool"
  }
  set {
    name  = "datadog.site"
    value = "datadoghq.com" # Or eu.datadoghq.com, us3.datadoghq.com etc.
  }

  # Add any custom Datadog Agent configurations here
  # E.g., for custom metrics, Autodiscovery, etc.
}
    

4. Configure PagerDuty Service and Escalation Policy

Define an escalation policy and a service in PagerDuty that Datadog will integrate with to trigger incidents.

# main.tf - PagerDuty Configuration
resource "pagerduty_escalation_policy" "eks_observability_policy" {
  name      = "EKS Observability On-Call Policy"
  teams     = [var.pagerduty_team_id] # Replace with your PagerDuty Team ID
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 5
    target {
      type = "user"
      id   = var.pagerduty_oncall_user_id # Replace with a user ID or schedule ID
    }
  }

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "schedule"
      id   = var.pagerduty_oncall_schedule_id # Replace with your PagerDuty Schedule ID
    }
  }
}

resource "pagerduty_service" "eks_observability_service" {
  name                     = "EKS Cluster Observability"
  auto_resolve_timeout     = 14400 # 4 hours
  acknowledgement_timeout  = 600   # 10 minutes
  escalation_policy        = pagerduty_escalation_policy.eks_observability_policy.id
  incident_urgency_rule {
    type    = "constant"
    urgency = "high"
  }
}

resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog Integration"
  service     = pagerduty_service.eks_observability_service.id
  type        = "generic_events_api_inbound_integration"
}
    

5. Integrate Datadog with PagerDuty

Configure the Datadog PagerDuty integration and then define a Datadog monitor that uses this integration.

# main.tf - Datadog Integration
resource "datadog_integration_pagerduty" "pagerduty_integration" {
  api_token = var.pagerduty_api_token
  # Add specific PagerDuty services to integrate if needed
  # services {
  #   service_name = pagerduty_service.eks_observability_service.name
  #   service_key  = pagerduty_service_integration.datadog_integration.integration_key
  # }
}

# Example: Datadog Monitor for EKS Node CPU Utilization
resource "datadog_monitor" "eks_node_cpu_high" {
  name            = "[EKS] High Node CPU Utilization on {{host.name}}"
  type            = "metric alert"
  query           = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster_name:${var.eks_cluster_name}} by {host} < 10" # Alert if idle CPU is less than 10% for 5 minutes
  message         = "EKS node {{host.name}} is experiencing high CPU utilization. Review application workloads. @pagerduty-EKS Cluster Observability" # This targets the PagerDuty integration by its service name.
  
  # Ensure the service name matches the PagerDuty service created
  # The @pagerduty-SERVICE_NAME syntax is crucial for Datadog to send to the right PagerDuty service.
  # This relies on the Datadog PagerDuty integration being configured and aware of your PagerDuty service.
  
  tags = ["env:${var.environment}", "eks_cluster:${var.eks_cluster_name}", "team:devops"]

  # Alerting Conditions
  critical {
    threshold = 10
    time_aggregator = "avg"
    # trigger_window = 300 # 5 minutes
  }

  # Optional: Warning condition
  warning {
    threshold = 20
    time_aggregator = "avg"
  }

  renotify_interval = 60 # Renotify every 60 minutes if alert persists
  notify_no_data    = false
  notify_audit      = false
  require_full_window = true
  timeout_h           = 0 # Alert immediately
  include_tags        = true
  # Force deletion of the monitor from Datadog when the resource is removed from Terraform
  force_delete        = true
}

# Example: Datadog Monitor for EKS Node Not Ready
resource "datadog_monitor" "eks_node_not_ready" {
  name            = "[EKS] Node Not Ready in {{kubernetes_cluster_name}}"
  type            = "metric alert"
  query           = "sum(last_5m):kubernetes.node.ready{kubernetes_cluster_name:${var.eks_cluster_name}} by {kubernetes_node_name} == 0"
  message         = "EKS node {{kubernetes_node_name}} is not ready. Investigate node health or underlying AWS infrastructure. @pagerduty-EKS Cluster Observability"
  
  tags = ["env:${var.environment}", "eks_cluster:${var.eks_cluster_name}", "team:devops"]

  critical {
    threshold = 0 # If the count of ready nodes drops to 0 for a specific node
  }

  renotify_interval = 60
  notify_no_data    = false
  notify_audit      = false
  require_full_window = true
  timeout_h           = 0
  include_tags        = true
  force_delete        = true
}
    

6. Variables and Outputs

Define variables for sensitive information and environment-specific settings, and outputs for useful information.

# variables.tf
variable "aws_region" {
  description = "AWS region for the EKS cluster"
  type        = string
  default     = "us-east-1"
}

variable "eks_cluster_name" {
  description = "Name of the existing EKS cluster"
  type        = string
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

variable "pagerduty_team_id" {
  description = "PagerDuty Team ID (for escalation policy)"
  type        = string
}

variable "pagerduty_oncall_user_id" {
  description = "PagerDuty User ID for initial alert target"
  type        = string
}

variable "pagerduty_oncall_schedule_id" {
  description = "PagerDuty Schedule ID for escalation"
  type        = string
}

variable "environment" {
  description = "Environment tag for resources"
  type        = string
  default     = "production"
}

# outputs.tf
output "datadog_agent_helm_release_status" {
  description = "Status of the Datadog Agent Helm release"
  value       = helm_release.datadog_agent.status
}

output "pagerduty_service_url" {
  description = "URL to the created PagerDuty service"
  value       = "https://your-domain.pagerduty.com/services/${pagerduty_service.eks_observability_service.id}"
}

output "datadog_monitor_url" {
  description = "URL to the created Datadog CPU Monitor"
  value       = "https://app.datadoghq.com/monitors/${datadog_monitor.eks_node_cpu_high.id}"
}
    

Deployment Steps

Save the files: Place the `providers.tf`, `main.tf`, `variables.tf`, and `outputs.tf` in a directory (e.g., `eks-observability`).
Initialize Terraform: Navigate to your directory and run `terraform init`.
Review the plan: Run `terraform plan -var="eks_cluster_name=your-eks-cluster-name" -var="datadog_api_key=..." -var="datadog_app_key=..." -var="pagerduty_api_token=..." -var="pagerduty_team_id=..." -var="pagerduty_oncall_user_id=..." -var="pagerduty_oncall_schedule_id=..."` to see what resources Terraform will create.
Apply the configuration: If the plan looks correct, execute `terraform apply -var="eks_cluster_name=your-eks-cluster-name" ...` (using the same variables) and type `yes` to confirm.

Verification and Testing

After applying the Terraform configuration:

Verify Datadog Agent: Run `kubectl get pods -n datadog` in your EKS cluster. You should see Datadog agent pods running.
Check Datadog UI: Log into your Datadog account. You should start seeing metrics, logs, and traces from your EKS cluster under Infrastructure -> Kubernetes, and your custom dashboards.
Verify PagerDuty Setup: Log into PagerDuty. Confirm that the "EKS Observability On-Call Policy" and "EKS Cluster Observability" service are created.
Test Monitor: For a quick test, you might temporarily lower the CPU threshold on the Datadog monitor or intentionally create a high CPU load on a node to trigger an alert and verify PagerDuty integration.

Advanced Observability Considerations

Custom Application Metrics and Tracing

Beyond infrastructure monitoring, instrument your applications with Datadog's client libraries to send custom metrics and distributed traces. Use annotations in your Kubernetes deployments for Datadog Autodiscovery to automatically configure checks for your services.

Cost Optimization

Datadog pricing can scale with data volume. Implement thoughtful log filtering, metric aggregation, and intelligent sampling for traces to manage costs without sacrificing critical visibility.

Security Best Practices

Ensure the Datadog Agent runs with the least privilege necessary. Regularly rotate API keys and store them securely, ideally using AWS Secrets Manager or HashiCorp Vault, and reference them in Terraform.

GitOps Integration

For true production readiness, integrate your Terraform configurations into a GitOps workflow (e.g., using ArgoCD or FluxCD). This ensures that desired state, including observability, is consistently applied and maintained.

Troubleshooting Common Issues

Datadog Agent Pods Not Running

Check `kubectl describe pod -n datadog` for events and `kubectl logs -n datadog` for errors. Common issues include incorrect API/APP keys, insufficient resource requests/limits, or network connectivity problems to Datadog endpoints.

No Data in Datadog

Verify the `datadog.site` parameter in your Helm release matches your Datadog account's region (e.g., `datadoghq.com` for US1, `eu.datadoghq.com` for EU). Ensure the correct API and APP keys are used. Check agent logs for errors related to data ingestion.

PagerDuty Alerts Not Triggering

Ensure the PagerDuty integration in Datadog is correctly configured with the PagerDuty API token. Double-check the `@pagerduty-SERVICE_NAME` syntax in your Datadog monitor message matches the actual PagerDuty service name exactly. Verify that the monitor itself is in an alerting state in Datadog.

Conclusion

Achieving production-ready observability for AWS EKS is a critical step in maintaining highly available and performant applications. By leveraging Terraform for declarative infrastructure, Datadog for comprehensive monitoring, and PagerDuty for efficient incident response, you can establish a robust, automated, and scalable observability solution. This guide provides a solid foundation, empowering your DevOps teams to proactively identify, diagnose, and resolve issues, ensuring the smooth operation of your cloud-native workloads.

CloudStack YAML