Terraform for AWS EKS Observability with Datadog, Prometheus, and PagerDuty Alerting

Architecture Pro-Tip: Always treat your observability stack as mission-critical infrastructure. Automate its deployment and configuration using Terraform alongside your core EKS cluster. This ensures consistency, simplifies recovery, and integrates seamlessly into your GitOps workflows, making your monitoring setup as robust and auditable as the applications it observes.

Terraform for AWS EKS Observability with Datadog, Prometheus, and PagerDuty Alerting

In the dynamic world of cloud-native applications, maintaining robust observability for Kubernetes clusters is paramount. AWS EKS (Elastic Kubernetes Service) provides a powerful foundation, but its true potential is unlocked when coupled with comprehensive monitoring, logging, and alerting solutions. This guide delves into leveraging Terraform, the industry-standard Infrastructure as Code (IaC) tool, to provision and manage a world-class observability stack on EKS, integrating Datadog for unified visibility, Prometheus for deep metric collection, and PagerDuty for critical incident management and alerting.

By the end of this technical guide, you'll understand how to use Terraform to automate the deployment of these tools, ensuring your EKS environments are not just running, but are also transparent, resilient, and ready for any operational challenge.

Why Observability is Critical for AWS EKS

Kubernetes, while incredibly powerful, introduces significant complexity. Microservices, dynamic scaling, and ephemeral containers make traditional monitoring approaches insufficient. A comprehensive observability strategy is essential for:

Proactive Issue Detection: Identify anomalies and potential problems before they impact users.
Faster Root Cause Analysis: Quickly pinpoint the source of issues across distributed systems using correlated data.
Performance Optimization: Understand resource utilization, bottlenecks, and application behavior to optimize performance and cost.
Security and Compliance: Monitor cluster health and activity for security breaches and compliance adherence.
Improved Developer Productivity: Provide developers with the insights they need to build and deploy reliable applications.

The Observability Stack: Datadog, Prometheus, and PagerDuty

This guide focuses on a powerful combination of tools, each excelling in its specific domain:

Datadog: Comprehensive Monitoring & APM

Datadog is a unified monitoring and analytics platform that provides end-to-end visibility across infrastructure, applications, and logs. For EKS, Datadog offers:

Unified Dashboarding: Correlate metrics, traces, and logs from your EKS cluster, applications, and underlying AWS infrastructure.
Kubernetes Integration: Out-of-the-box dashboards and monitors for EKS health, pod states, deployments, and resource utilization.
Application Performance Monitoring (APM): Distributed tracing to visualize and debug requests across microservices.
Log Management: Centralized log collection, processing, and analysis from all EKS components and applications.
Synthetic Monitoring: Proactive checks for external endpoint availability and performance.

Prometheus: Open-Source Metrics & Querying

Prometheus is an open-source monitoring system and time-series database. It is a fundamental component for collecting highly granular metrics directly from your Kubernetes workloads and nodes. Key advantages include:

Pull-Based Model: Scrapes metrics from configured targets (exporters) at specified intervals.
Powerful Query Language (PromQL): A flexible language for querying, aggregating, and transforming time-series data.
Ecosystem of Exporters: Integrates with virtually any application or service via dedicated exporters (e.g., `kube-state-metrics`, `node_exporter`).
Local Storage & Alerting: Stores metrics locally and includes Alertmanager for sophisticated alert routing.

In this setup, Prometheus can act as a primary metric collection layer, with Datadog potentially ingesting these metrics for a unified view, or standing alongside it for specific use cases.

PagerDuty: Incident Management & Alerting

PagerDuty is a leading incident management platform that transforms any signal into an actionable incident. It is crucial for ensuring that critical alerts from Datadog (and potentially Prometheus) reach the right on-call teams immediately. Its features include:

Intelligent Alert Routing: Based on on-call schedules, escalation policies, and services.
Multi-Channel Notifications: Phone calls, SMS, email, push notifications, and chat tools.
Incident Collaboration: Tools for managing and resolving incidents efficiently.
Reporting and Analytics: Insights into incident frequency, duration, and team performance.

Terraform: Infrastructure as Code for Observability

Using Terraform to deploy and manage this observability stack offers significant advantages:

Automation: Eliminate manual configuration, reducing human error and accelerating deployment.
Consistency: Ensure identical configurations across development, staging, and production environments.
Version Control: Treat your observability setup as code, enabling tracking, auditing, and easy rollbacks.
Dependency Management: Terraform understands and manages the order of resource creation and updates.
Cost Management: Clearly define and manage cloud resources for monitoring, preventing unexpected costs.

Prerequisites

Before you begin, ensure you have the following:

An AWS Account with appropriate IAM permissions.
An existing AWS EKS Cluster. This guide assumes your EKS cluster is already provisioned, ideally with Terraform, and `kubectl` is configured to interact with it.
Terraform CLI (v1.0.0+) installed.
Datadog API and Application Keys: Obtain these from your Datadog account settings.
PagerDuty Service Integration Key: Create a new PagerDuty service and an integration (e.g., "Datadog Integration") to obtain this key.
`kubectl` installed and configured for your EKS cluster.
Helm CLI installed (Terraform will interact with Helm charts).

Step-by-Step Implementation with Terraform

We'll walk through deploying the Datadog Agent, Prometheus, and setting up PagerDuty alerting via Datadog, all managed by Terraform.

1. Project Structure and Providers

Organize your Terraform code into a logical structure. Start with `main.tf`, `variables.tf`, and `providers.tf`.

        # providers.tf
        terraform {
          required_providers {
            aws = {
              source  = "hashicorp/aws"
              version = "~> 5.0"
            }
            kubernetes = {
              source  = "hashicorp/kubernetes"
              version = "~> 2.23"
            }
            helm = {
              source  = "hashicorp/helm"
              version = "~> 2.11"
            }
            datadog = {
              source  = "DataDog/datadog"
              version = "~> 3.0"
            }
          }
        }

        provider "aws" {
          region = var.aws_region
        }

        provider "kubernetes" {
          host                   = data.aws_eks_cluster.eks_cluster.endpoint
          cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority.0.data)
          token                  = data.aws_eks_cluster_auth.eks_cluster_auth.token
        }

        provider "helm" {
          kubernetes {
            host                   = data.aws_eks_cluster.eks_cluster.endpoint
            cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority.0.data)
            token                  = data.aws_eks_cluster_auth.eks_cluster_auth.token
          }
        }

        provider "datadog" {
          api_key = var.datadog_api_key
          app_key = var.datadog_app_key
        }

        # main.tf (EKS data source to configure providers)
        data "aws_eks_cluster" "eks_cluster" {
          name = var.eks_cluster_name
        }

        data "aws_eks_cluster_auth" "eks_cluster_auth" {
          name = var.eks_cluster_name
        }

        # variables.tf
        variable "aws_region" {
          description = "AWS region"
          type        = string
          default     = "us-east-1"
        }

        variable "eks_cluster_name" {
          description = "Name of the EKS cluster"
          type        = string
        }

        variable "datadog_api_key" {
          description = "Datadog API Key"
          type        = string
          sensitive   = true
        }

        variable "datadog_app_key" {
          description = "Datadog Application Key"
          type        = string
          sensitive   = true
        }

        variable "pagerduty_service_integration_key" {
          description = "PagerDuty service integration key for Datadog"
          type        = string
          sensitive   = true
        }
    

2. Integrating Datadog with EKS

The Datadog Agent is deployed as a DaemonSet to collect metrics and logs from each node, and a Cluster Agent is deployed as a Deployment for cluster-wide data collection (e.g., Kubernetes API server metrics). We use the Helm provider for this.

Terraform Configuration for Datadog

Below is a basic Terraform configuration to deploy the Datadog Agent and set up a simple monitor. Remember to create the `datadog` namespace beforehand or via Terraform if it doesn't exist.

        # datadog.tf

        resource "kubernetes_namespace" "datadog_ns" {
          metadata {
            name = "datadog"
          }
        }

        resource "helm_release" "datadog_agent" {
          name       = "datadog"
          repository = "https://helm.datadoghq.com"
          chart      = "datadog"
          namespace  = kubernetes_namespace.datadog_ns.metadata.0.name

          set {
            name  = "datadog.apiKey"
            value = var.datadog_api_key
            sensitive = true
          }
          set {
            name  = "datadog.appKey"
            value = var.datadog_app_key
            sensitive = true
          }
          set {
            name  = "clusterAgent.enabled"
            value = "true"
          }
          set {
            name  = "kubeStateMetrics.enabled"
            value = "true"
          }
          set {
            name  = "clusterChecksRunner.enabled"
            value = "true"
          }
          set {
            name  = "logs.enabled"
            value = "true"
          }
          set {
            name  = "logs.containerCollectAll"
            value = "true"
          }
          set {
            name  = "processAgent.enabled"
            value = "true"
          }
          set {
            name  = "containerd.socketPath"
            value = "/var/run/containerd/containerd.sock" # Adjust for your container runtime
          }
          # Enable APM if needed
          set {
            name  = "apm.enabled"
            value = "true"
          }
          # Enable Datadog-Prometheus integration (see later section)
          set {
            name  = "datadog.prometheusScrape.enabled"
            value = "true"
          }
          # Ensure proper service account and permissions if using IAM Roles for Service Accounts (IRSA)
          # set { name = "datadog.createServiceAccount", value = "true" }
          # set { name = "datadog.serviceAccount.create", value = "true" }
          # set { name = "datadog.serviceAccount.annotations.eks\.amazonaws\.com/role-arn", value = "arn:aws:iam::ACCOUNT_ID:role/YOUR_DATADOG_IAM_ROLE" }
          
          # Wait for the release to be healthy before marking as complete
          wait       = true
          timeout    = 600
        }

        resource "datadog_monitor" "high_cpu_usage" {
          name         = "EKS Cluster {{kube_cluster_name.name}} Node CPU Usage High"
          type         = "metric alert"
          query        = "avg(last_5m):avg:kubernetes.cpu.usage.total{kube_cluster_name:${var.eks_cluster_name}} by {host} > 80"
          message      = "Node {{host.name}} CPU usage is over 80%. Consider scaling or investigating. @pagerduty-Datadog"
          tags         = ["env:production", "service:eks", "alert-type:cpu"]
          priority     = 2
          # Optional: Automatically resolve after a period of recovery
          # no_data_timeframe = 10 
          # require_full_window = false 
          
          # This assumes a PagerDuty integration named "Datadog" exists in PagerDuty
          # and the Datadog integration with PagerDuty is configured.
          # We'll set up the integration using Terraform next.
        }
    

3. Deploying Prometheus on EKS

To deploy Prometheus, we typically use the Prometheus Operator, which simplifies deployment and management of Prometheus instances and related components like Alertmanager, `kube-state-metrics`, and `node_exporter`. We will deploy this via a Helm chart.

Terraform Configuration for Prometheus

This example deploys the kube-prometheus-stack Helm chart, which includes Prometheus, Grafana, Alertmanager, `kube-state-metrics`, and `node_exporter`.

        # prometheus.tf

        resource "kubernetes_namespace" "monitoring_ns" {
          metadata {
            name = "monitoring"
          }
        }

        resource "helm_release" "kube_prometheus_stack" {
          name       = "kube-prometheus-stack"
          repository = "https://prometheus-community.github.io/helm-charts"
          chart      = "kube-prometheus-stack"
          namespace  = kubernetes_namespace.monitoring_ns.metadata.0.name

          version = "50.0.0" # Use a specific version to ensure consistency

          set {
            name  = "grafana.enabled"
            value = "true" # Enable Grafana for dashboards
          }
          set {
            name  = "grafana.adminPassword"
            value = "prom-admin-password" # IMPORTANT: Use a secret manager for production
            sensitive = true
          }
          set {
            name  = "prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues"
            value = "false"
          }
          set {
            name  = "prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues"
            value = "false"
          }
          # Configure retention for Prometheus (e.g., 30 days)
          set {
            name  = "prometheus.prometheusSpec.retention"
            value = "30d"
          }
          # Optional: Increase resource limits for Prometheus server in larger clusters
          # set { name = "prometheus.prometheusSpec.resources.requests.cpu", value = "500m" }
          # set { name = "prometheus.prometheusSpec.resources.requests.memory", value = "1Gi" }

          # Configure Alertmanager (if not using Datadog for all alerts)
          set {
            name  = "alertmanager.enabled"
            value = "true"
          }
          # Example Alertmanager configuration for PagerDuty (basic)
          # This would typically be a more complex YAML structure in a values file or a dedicated resource.
          # For simplicity here, we assume Datadog drives PagerDuty. If using Prometheus Alertmanager directly,
          # refer to Prometheus documentation for PagerDuty configuration.

          wait    = true
          timeout = 900
          depends_on = [
            kubernetes_namespace.monitoring_ns
          ]
        }
    

4. Connecting Datadog to Prometheus Metrics

Datadog can integrate with Prometheus in several ways:

Datadog Agent Scraping: The Datadog Agent can be configured to scrape Prometheus metrics from endpoints exposed by your applications or even from the Prometheus server itself. This provides a unified view in Datadog.
OpenMetrics Integration: Datadog supports OpenMetrics, allowing it to collect metrics from any service exposing them in the Prometheus format.

In the `helm_release` for the Datadog Agent above, we included `datadog.prometheusScrape.enabled: "true"`, which allows the Datadog Agent to discover and scrape Prometheus endpoints. You might need to add specific annotations to your Kubernetes services/pods to indicate which ports and paths expose Prometheus metrics for Datadog to discover.

        # Example Kubernetes deployment annotation for Datadog to scrape Prometheus metrics
        # This would be part of your application's Kubernetes deployment manifest (not directly Terraform)
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: my-app
        spec:
          template:
            metadata:
              annotations:
                ad.datadoghq.com/my-app.check_names: '["prometheus"]'
                ad.datadoghq.com/my-app.init_configs: '[{}]'
                ad.datadoghq.com/my-app.instances: '[{"prometheus_url": "http://%%host%%:8080/metrics"}]' # Adjust port/path
            spec:
              containers:
              - name: my-app
                image: my-registry/my-app:latest
                ports:
                - containerPort: 8080
                  name: metrics
    

5. Setting Up PagerDuty Alerting

Datadog integrates seamlessly with PagerDuty. You can configure this integration directly within Datadog, and then reference it in your monitors. Terraform can manage both the integration and the monitors.

Terraform Configuration for PagerDuty Integration (via Datadog)

First, establish the integration in Datadog using your PagerDuty service integration key. Then, update your Datadog monitors to route alerts to PagerDuty.

        # pagerduty.tf

        resource "datadog_integration_pagerduty" "main" {
          services {
            service_name = "Datadog" # Name of the PagerDuty service in Datadog
            service_key  = var.pagerduty_service_integration_key
          }
          # Add more services as needed
          # services {
          #   service_name = "AnotherService"
          #   service_key  = "YOUR_ANOTHER_PAGERDUTY_KEY"
          # }
        }

        # Update datadog_monitor to use PagerDuty (re-using the earlier CPU monitor example)
        resource "datadog_monitor" "high_cpu_usage" {
          name         = "EKS Cluster {{kube_cluster_name.name}} Node CPU Usage High"
          type         = "metric alert"
          query        = "avg(last_5m):avg:kubernetes.cpu.usage.total{kube_cluster_name:${var.eks_cluster_name}} by {host} > 80"
          message      = "Node {{host.name}} CPU usage is over 80%. Consider scaling or investigating. @pagerduty-${datadog_integration_pagerduty.main.services.0.service_name}"
          tags         = ["env:production", "service:eks", "alert-type:cpu"]
          priority     = 2
          # Depends on the integration being set up
          depends_on = [datadog_integration_pagerduty.main]
        }

        # Example: Alert for high EKS control plane latency
        resource "datadog_monitor" "eks_api_latency_high" {
          name         = "EKS Cluster {{kube_cluster_name.name}} API Server Latency High"
          type         = "metric alert"
          query        = "avg(last_5m):aws.eks.api_server.latency.average{kube_cluster_name:${var.eks_cluster_name}} > 0.5"
          message      = "EKS API Server latency is above 0.5s. Investigate control plane health. @pagerduty-${datadog_integration_pagerduty.main.services.0.service_name}"
          tags         = ["env:production", "service:eks", "alert-type:api-latency"]
          priority     = 1 # High priority alert
          depends_on = [datadog_integration_pagerduty.main]
        }
    

Comprehensive Terraform Module Structure (Example)

For larger setups, it's best to organize your Terraform code into modules. A typical structure might look like this:

        .
        ├── main.tf
        ├── variables.tf
        ├── outputs.tf
        ├── providers.tf
        └── modules/
            ├── datadog/
            │   ├── main.tf
            │   ├── variables.tf
            │   └── outputs.tf
            ├── prometheus/
            │   ├── main.tf
            │   ├── variables.tf
            │   └── outputs.tf
            └── eks-cluster/ # If you also manage EKS with Terraform
                ├── main.tf
                ├── variables.tf
                └── outputs.tf
    

In your root `main.tf`, you would then call these modules:

        # main.tf (root)

        # ... (provider configuration and EKS data sources as before) ...

        module "datadog_observability" {
          source               = "./modules/datadog"
          eks_cluster_name     = var.eks_cluster_name
          datadog_api_key      = var.datadog_api_key
          datadog_app_key      = var.datadog_app_key
          pagerduty_service_key = var.pagerduty_service_integration_key # Pass this down
        }

        module "prometheus_monitoring" {
          source           = "./modules/prometheus"
          eks_cluster_name = var.eks_cluster_name
          # ... other Prometheus-specific variables ...
        }
    

Best Practices and Advanced Considerations

GitOps for Configuration: Integrate your Terraform setup with a GitOps workflow (e.g., Argo CD, Flux CD) to manage both your EKS cluster and observability stack from a single source of truth in Git.
IAM Roles for Service Accounts (IRSA): For enhanced security, use IRSA to grant necessary AWS permissions (e.g., for Datadog to pull CloudWatch metrics) to Kubernetes service accounts rather than directly managing AWS credentials.
Secrets Management: Never hardcode API keys or sensitive credentials. Use AWS Secrets Manager, HashiCorp Vault, or Kubernetes Secrets (with proper encryption like Sealed Secrets or CSI Secrets Store driver) for managing sensitive data.
Custom Metrics and Logs: Extend your observability by pushing custom application metrics to Datadog or Prometheus, and routing application logs to Datadog for centralized analysis.
Cost Optimization: Monitor your Datadog ingestion volumes and Prometheus retention policies. Configure filters to collect only essential metrics and logs to control costs.
SLOs and SLO/SLA Monitoring: Define Service Level Objectives (SLOs) and use Datadog to monitor against them, providing clear indicators of service health and customer impact.
Alert Fatigue Management: Carefully tune your Datadog and PagerDuty alerts. Use alert conditions, composite alerts, and incident suppression rules to reduce noise and ensure only actionable alerts reach on-call teams.

Troubleshooting Common Issues

Datadog Agent Not Reporting:
- Verify `datadog.apiKey` and `datadog.appKey` are correct in your Helm values.
- Check Datadog Agent pod logs for errors: `kubectl logs -n datadog -l app=datadog --tail 100`.
- Ensure network policies are not blocking communication to Datadog endpoints.
- Confirm the Datadog Agent DaemonSet pods are running on all nodes: `kubectl get ds -n datadog`.
Prometheus Metrics Missing:
- Access the Prometheus UI (port-forward if needed) and check "Targets" status to see if exporters are being scraped.
- Verify `ServiceMonitor` or `PodMonitor` configurations are correctly selecting your application services/pods.
- Check application logs to ensure Prometheus metrics endpoints are correctly exposed and accessible.
PagerDuty Alerts Not Firing:
- Ensure the `datadog_integration_pagerduty` resource was applied successfully.
- Check the PagerDuty service integration key for accuracy.
- Verify the Datadog monitor's message template correctly references the PagerDuty integration (e.g., `@pagerduty-Datadog`).
- Test the Datadog monitor by temporarily lowering its threshold to trigger an alert.
- Review PagerDuty's incident logs for incoming events.

Conclusion

Establishing robust observability for AWS EKS is a critical step towards maintaining high availability, performance, and operational efficiency. By leveraging Terraform, you can automate the deployment and management of a powerful observability stack composed of Datadog for unified visibility, Prometheus for deep metric collection, and PagerDuty for reliable incident response. This IaC approach ensures consistency, reduces manual effort, and provides an auditable, version-controlled foundation for your cloud-native monitoring strategy. Embrace automation to gain unparalleled insight into your EKS environments and empower your teams to resolve issues faster than ever before.

CloudStack YAML