Terraform for AWS EKS: Automated Datadog APM, Prometheus, and PagerDuty Alerting

In the dynamic world of cloud-native applications, maintaining robust observability and proactive incident response for Kubernetes clusters is paramount. AWS Elastic Kubernetes Service (EKS) offers a powerful platform for orchestrating containers, but its true potential is unlocked when integrated with comprehensive monitoring and alerting systems. This guide delves into automating the setup of Datadog for Application Performance Monitoring (APM), Prometheus for infrastructure metrics, and PagerDuty for incident management on AWS EKS, all orchestrated seamlessly with Terraform.

Architecture Pro-Tip:

When designing your observability stack for EKS, always prioritize modularity and least privilege. Separate your Terraform configurations for EKS cluster infrastructure, monitoring agents, and alert definitions. Utilize dedicated IAM roles with minimal necessary permissions for each service (e.g., Datadog Agent, Prometheus scraping). This enhances security, maintainability, and allows for independent upgrades of components.

Why Automate Observability with Terraform on EKS?

Manual configuration of monitoring and alerting tools across multiple EKS environments (dev, staging, production) is error-prone, time-consuming, and lacks consistency. Terraform, as an Infrastructure as Code (IaC) tool, provides the definitive solution:

Consistency: Ensure identical monitoring setups across all environments.
Repeatability: Spin up new EKS clusters with pre-configured observability in minutes.
Version Control: Track changes to your monitoring configurations, enabling rollbacks and audits.
Efficiency: Reduce operational overhead and human error.

Key Components of Our Observability Stack

1. AWS EKS: The Foundation

Our target environment is a running AWS EKS cluster. While the full EKS setup with Terraform is outside the scope of this monitoring-focused guide, we'll assume you have an EKS cluster already provisioned or are using a standard Terraform EKS module.

2. Datadog: APM, Logs, and Infrastructure Monitoring

Datadog provides a unified platform for monitoring, offering comprehensive capabilities:

APM: End-to-end visibility into application performance, tracing requests across microservices.
Logs: Centralized log management for all EKS pods and services.
Infrastructure: Deep metrics and events from EKS nodes, pods, deployments, and AWS services.
Network Performance: Visibility into network traffic between services.

The Datadog Agent runs as a DaemonSet on EKS nodes to collect host-level metrics, events, and logs, while its APM libraries are integrated within your application code.

3. Prometheus: Open-Source Metrics Collection

Prometheus is a powerful, open-source monitoring system and time-series database. It's excellent for collecting metrics from Kubernetes components and custom application endpoints. We'll leverage the kube-prometheus-stack Helm chart for a batteries-included setup that includes:

Prometheus Server: Scrapes and stores metrics.
Alertmanager: Handles alerts sent by Prometheus.
Grafana: For powerful data visualization and dashboards.
Node Exporter: For host-level metrics.
Kube-state-metrics: For Kubernetes object metrics.

4. PagerDuty: Incident Management

PagerDuty acts as our central nervous system for incident response. It integrates with monitoring tools like Datadog and Prometheus's Alertmanager to route alerts to the right teams, at the right time, using on-call schedules, escalation policies, and various notification channels. This ensures that critical issues are never missed and are addressed promptly.

Prerequisites

Before you begin, ensure you have the following:

An active AWS account with appropriate permissions to manage EKS and IAM resources.
Terraform CLI installed (v1.0+ recommended).
AWS CLI installed and configured.
kubectl configured to interact with your EKS cluster.
A Datadog account with API and Application keys.
A PagerDuty account with an API token.
Helm CLI installed (for local testing, though Terraform manages Helm releases).

Terraform Setup: Providers and Variables

We'll start by defining the necessary Terraform providers and input variables for sensitive credentials and configuration details.

provider "aws" {
  region = var.aws_region
}

provider "kubernetes" {
  host                   = var.eks_cluster_endpoint
  cluster_ca_certificate = base64decode(var.eks_cluster_certificate_authority_data)
  token                  = data.aws_eks_cluster_auth.this.token
}

data "aws_eks_cluster_auth" "this" {
  name = var.eks_cluster_name
}

provider "helm" {
  kubernetes {
    host                   = var.eks_cluster_endpoint
    cluster_ca_certificate = base64decode(var.eks_cluster_certificate_authority_data)
    token                  = data.aws_eks_cluster_auth.this.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_token
}
    

And for `variables.tf`:

variable "aws_region" {
  description = "AWS region for EKS cluster"
  type        = string
  default     = "us-east-1"
}

variable "eks_cluster_name" {
  description = "Name of the existing EKS cluster"
  type        = string
}

variable "eks_cluster_endpoint" {
  description = "Endpoint of the existing EKS cluster"
  type        = string
}

variable "eks_cluster_certificate_authority_data" {
  description = "Base64 encoded certificate data for the EKS cluster"
  type        = string
  sensitive   = true
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_token" {
  description = "PagerDuty API token"
  type        = string
  sensitive   = true
}

variable "datadog_pagerduty_integration_name" {
  description = "Name for the Datadog PagerDuty integration"
  type        = string
  default     = "EKS-Incidents"
}

variable "pagerduty_service_name" {
  description = "Name of the PagerDuty service to create or link"
  type        = string
  default     = "EKS Monitoring"
}
    

Step-by-Step Implementation

1. Deploying Datadog Agent on EKS with Terraform Helm Provider

The Datadog Agent is deployed via its official Helm chart. We'll use the Terraform `helm_release` resource to manage this deployment. Ensure you have your `datadog_api_key` and `datadog_app_key` set as environment variables or passed securely.

We will also create an IAM policy and role for the Datadog Agent to allow it to collect metadata from AWS services, enhancing the observability context within Datadog.

Ready-to-Use Configuration: Datadog, Prometheus, and PagerDuty

Below is a comprehensive Terraform configuration (`main.tf`) that orchestrates the deployment of Datadog and kube-prometheus-stack via Helm, sets up the Datadog-PagerDuty integration, and defines a sample Datadog monitor linked to PagerDuty. Replace placeholder values (like EKS cluster details) with your actual environment specifics.

# main.tf

# ----------------------------------------------------
# PagerDuty Service Setup
# (Optional, you can link to an existing PagerDuty service)
# ----------------------------------------------------
resource "pagerduty_user" "example_user" {
  # This is a placeholder for a PagerDuty user to be linked to an escalation policy.
  # In a real scenario, you'd fetch an existing user or create one with proper email.
  # For simplicity, we'll just define a dummy for the service to be created.
  name  = "Terraform Admin"
  email = "terraform-admin@example.com" # Replace with a real email
  role  = "admin"
}

resource "pagerduty_escalation_policy" "eks_monitoring_policy" {
  name      = "EKS Critical Alerts Policy"
  num_loops = 2
  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "user"
      id   = pagerduty_user.example_user.id
    }
  }
}

resource "pagerduty_service" "eks_monitoring_service" {
  name                    = var.pagerduty_service_name
  auto_resolve_timeout    = "14400" # 4 hours
  acknowledgement_timeout = "600"   # 10 minutes
  escalation_policy       = pagerduty_escalation_policy.eks_monitoring_policy.id
}

# ----------------------------------------------------
# Datadog Agent Deployment via Helm
# ----------------------------------------------------
resource "kubernetes_secret" "datadog_api_key" {
  metadata {
    name      = "datadog-api-key"
    namespace = "default" # Or your desired namespace for Datadog
  }
  data = {
    api_key = var.datadog_api_key
  }
  type = "Opaque"
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Or your desired namespace
  version    = "2.33.0"  # Use the latest stable version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key # Sensitive, consider Kubernetes Secret or AWS Secrets Manager
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key # Sensitive
  }
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "kubeStateMetricsCore.enabled"
    value = "true"
  }
  set {
    name  = "networkMonitoring.enabled"
    value = "true"
  }
  set {
    name  = "apm.enabled"
    value = "true"
  }
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }
  set {
    name  = "tags"
    value = "environment:production,cluster_name:${var.eks_cluster_name}"
  }
  set {
    name  = "clusterName"
    value = var.eks_cluster_name
  }
  # Enable PagerDuty integration through Datadog for easy alerting
  set {
    name  = "integrations.pagerduty.enabled"
    value = "true"
  }
}

# ----------------------------------------------------
# Kube-Prometheus-Stack Deployment via Helm
# ----------------------------------------------------
resource "helm_release" "kube_prometheus_stack" {
  name       = "prometheus"
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  namespace  = "monitoring" # Recommended namespace
  create_namespace = true
  version    = "58.0.0" # Use the latest stable version

  values = [
    "${file("prometheus-values.yaml")}"
  ]
}

# prometheus-values.yaml (create this file in the same directory)
# This snippet provides basic settings, customize as needed.
# alertmanager:
#   enabled: true
#   config:
#     global:
#       resolve_timeout: 5m
#     route:
#       receiver: 'pagerduty'
#       group_by: ['alertname', 'cluster', 'service']
#       group_wait: 30s
#       group_interval: 5m
#       repeat_interval: 1h
#     receivers:
#       - name: 'pagerduty'
#         pagerduty_configs:
#           - service_key: "{{ .Values.alertmanager.pagerdutyServiceKey }}" # Placeholder, inject securely or use Datadog for PagerDuty
#     # For simplicity, we recommend Datadog for PagerDuty integration from Prometheus alerts
#     # You can configure Alertmanager to send alerts to Datadog's webhook, then Datadog routes to PagerDuty.
#     # Alternatively, direct PagerDuty integration here if preferred.
#
# grafana:
#   enabled: true
#   adminPassword: "admin" # Change this in production! Use secrets.
#   service:
#     type: LoadBalancer # Expose Grafana externally (for testing)
#
# prometheus:
#   prometheusSpec:
#     serviceMonitorSelectorNilUsesPods: false
#     podMonitorSelectorNilUsesPods: false

# ----------------------------------------------------
# Datadog PagerDuty Integration and Monitor
# ----------------------------------------------------
resource "datadog_integration_pagerduty" "eks_pd_integration" {
  # Note: The `datadog_integration_pagerduty` resource can only manage
  # one PagerDuty integration in Datadog.
  # If you already have one, this resource might conflict.
  # Ensure the PagerDuty API Key is configured in the Datadog UI for this integration.
  # This resource essentially links Datadog to PagerDuty,
  # allowing monitors to reference PagerDuty services.
  # The API Key here refers to a PagerDuty API key (not service key).
}

resource "datadog_monitor" "high_cpu_alert" {
  name                = "EKS Cluster High CPU Utilization (${var.eks_cluster_name})"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.eks_cluster_name}} by {host} > 80"
  message             = <<EOF
@pagerduty-${var.datadog_pagerduty_integration_name}-service-${pagerduty_service.eks_monitoring_service.name}
EKS cluster CPU utilization on host {{host.name}} is {{value}}% which is above 80%.
This indicates potential resource exhaustion or misbehaving workloads.
@webhook-your-slack-channel (optional)
EOF
  tags                = ["environment:production", "eks", "cpu-alert", "pagerduty"]
  require_full_window = false
  notify_no_data      = false
  new_group_delay     = 60
  no_data_timeframe   = 20
  renotify_interval   = 0
  escalation_message  = "CPU utilization remains high. Escalating to next level."
  include_tags        = true

  # Ensure the Datadog PagerDuty integration is already set up in the Datadog UI
  # or managed by `datadog_integration_pagerduty` resource.
  # The @pagerduty-{{INTEGRATION_NAME}}-service-{{SERVICE_NAME}} syntax is crucial.
  # {{INTEGRATION_NAME}} is the name you gave the integration in Datadog.
  # {{SERVICE_NAME}} is the PagerDuty service name.
}

output "datadog_agent_status" {
  description = "Status of the Datadog Agent Helm release"
  value       = helm_release.datadog_agent.status
}

output "prometheus_stack_status" {
  description = "Status of the Prometheus Stack Helm release"
  value       = helm_release.kube_prometheus_stack.status
}

output "pagerduty_service_url" {
  description = "URL of the created PagerDuty service"
  value       = pagerduty_service.eks_monitoring_service.html_url
}
    

To run this configuration:

Save the main code block as main.tf.
Save the variables block as variables.tf.
Create a file named prometheus-values.yaml in the same directory, starting with an empty content or basic settings as commented in the `main.tf` file.
Provide values for your EKS cluster and API keys. You can use a terraform.tfvars file or environment variables.
Run terraform init to initialize the providers.
Run terraform plan to review the changes.
Run terraform apply to deploy the resources.

Verification and Testing

After applying the Terraform configuration:

Datadog:
- Log into your Datadog account.
- Navigate to Infrastructure List to see your EKS nodes appearing.
- Check APM Services for any instrumented applications.
- Go to Log Explorer to verify logs are being ingested.
- Check Monitors to ensure your EKS CPU alert is present.
Prometheus/Grafana:
- If you exposed Grafana via a LoadBalancer, access its public IP. Log in with the configured `adminPassword`.
- Explore the pre-built Kubernetes dashboards to see cluster metrics.
- Verify Prometheus targets are healthy.
PagerDuty:
- Log into your PagerDuty account.
- Go to Services and confirm the 'EKS Monitoring' service exists.
- Manually trigger a test alert from Datadog that targets this PagerDuty service to ensure end-to-end functionality.

Troubleshooting and Best Practices

Common Issues:

`kubectl` Authentication: Ensure your `kubectl` context is correctly set for the target EKS cluster. Terraform providers for Kubernetes and Helm rely on this.
Missing API Keys: Double-check that all Datadog and PagerDuty API/App keys are correctly provided and not expired.
IAM Permissions: Verify that the IAM role associated with your EKS worker nodes (or the service account for Datadog Agent if using IRSA) has permissions to publish metrics and logs to Datadog if necessary, and for the Datadog Agent to query AWS APIs.
Helm Chart Versions: Always specify exact Helm chart versions (`version` attribute) to ensure repeatable deployments. Periodically review and update to leverage new features and fixes.
Resource Limits: Ensure your EKS cluster has sufficient resources (CPU, Memory) for the Datadog Agent, Prometheus, and Grafana pods.

Best Practices:

Secrets Management: Avoid hardcoding API keys directly in Terraform files. Use a secure secrets management solution like AWS Secrets Manager or HashiCorp Vault, and fetch them using Terraform data sources.
Modularity: Break down your Terraform configuration into logical modules (e.g., `eks-cluster`, `datadog-monitoring`, `prometheus-stack`, `pagerduty-alerts`).
Observability as Code: Beyond deploying agents, define your Datadog dashboards, monitors, and PagerDuty escalation policies directly in Terraform to maintain a single source of truth.
Custom Metrics: For Prometheus, define `ServiceMonitor` or `PodMonitor` resources (managed by the Prometheus Operator) to scrape custom metrics from your applications.
Cost Optimization: Monitor Datadog ingestion volumes and Prometheus storage to optimize costs. Tailor what metrics, logs, and traces are collected.

Conclusion

Automating your observability stack for AWS EKS with Terraform is a critical step towards building resilient, scalable, and manageable cloud-native applications. By integrating Datadog for comprehensive APM and infrastructure insights, Prometheus for granular metrics collection, and PagerDuty for reliable incident alerting, you empower your DevOps teams with the tools needed to detect, diagnose, and resolve issues proactively. This IaC approach ensures consistency, reduces manual overhead, and provides a robust foundation for operational excellence.

CloudStack YAML