Building a Robust Observability Stack for AWS EKS with Terraform, Datadog, Prometheus, and PagerDuty

In the dynamic world of cloud-native applications, maintaining high availability and performance for Kubernetes clusters is paramount. AWS EKS (Elastic Kubernetes Service) provides a managed control plane, but effective monitoring, logging, tracing, and incident response for your applications and infrastructure remain critical responsibilities. This guide outlines how to leverage the power of Terraform for Infrastructure as Code (IaC) to deploy a comprehensive observability stack on EKS, integrating industry-leading tools like Datadog, Prometheus, and PagerDuty.

Architecture Pro-Tip: Unified Observability vs. Best-of-Breed

While this guide deploys a best-of-breed approach with Datadog, Prometheus, and PagerDuty, consider your organizational needs. Datadog excels in unified observability (metrics, logs, traces, security). Prometheus, paired with Grafana, offers powerful open-source metric collection. For optimal results, configure Datadog to ingest Prometheus metrics where applicable, consolidating your dashboards and alerts while retaining the granular control Prometheus offers for specific workloads or custom exporters. This strategy minimizes tool switching for engineers and streamlines incident correlation.

Why a Holistic Observability Stack for EKS?

Running applications on Kubernetes introduces significant complexity. A robust observability stack provides the visibility needed to understand your system's health, troubleshoot issues efficiently, and ensure a seamless user experience. Here's why these tools, combined, are essential:

Datadog: A unified platform for metrics, logs, traces, and synthetic monitoring. It offers out-of-the-box integrations for AWS EKS, Kubernetes, and hundreds of other technologies, providing a single pane of glass for end-to-end visibility.
Prometheus: A powerful open-source monitoring system, particularly strong for collecting time-series metrics from Kubernetes components and applications. Its flexible query language (PromQL) and robust alerting capabilities make it a favorite for in-depth metric analysis. Datadog can be configured to scrape and ingest Prometheus metrics, consolidating data.
PagerDuty: An incident management platform that streamlines on-call rotations, automates incident routing, and facilitates quick response to critical alerts. Integrating it with your monitoring tools ensures that important alerts translate into actionable incidents for your DevOps teams.
Terraform: Enables you to define and provision your entire EKS cluster and the observability components as code. This ensures consistency, repeatability, version control, and auditability across environments.

Core Architecture Overview

Our observability architecture on AWS EKS will look something like this:

AWS EKS Cluster: The foundation, provisioned via Terraform.
Datadog Agent: Deployed as a DaemonSet on EKS (via Helm/Terraform). It collects metrics, logs, and traces from nodes, pods, and services, including direct integration with EKS and AWS services. It can also scrape Prometheus endpoints.
Prometheus (Kube-Prometheus-Stack): Deployed via Helm/Terraform. This includes Prometheus server, Alertmanager, Grafana, Kube-state-metrics, and Node-exporter. It scrapes cluster components and applications, and its metrics can be forwarded to Datadog for unified dashboards.
Datadog Integrations: Configured to pull additional metrics and logs from AWS services (CloudWatch, S3, RDS, etc.) and to send alerts to PagerDuty.
PagerDuty: Receives critical alerts from Datadog (or directly from Prometheus Alertmanager) and manages the incident response workflow.

Prerequisites

Before you begin, ensure you have the following:

An active AWS Account with administrative privileges.
Terraform CLI (v1.0+) installed.
AWS CLI configured with appropriate credentials.
kubectl CLI installed and configured.
Helm CLI (v3.0+) installed.
A Datadog account with an API Key and Application Key.
A PagerDuty account with a Service Integration Key (e.g., from a Datadog integration).

Step-by-Step Terraform Deployment Guide

1. Initialize Your Terraform Project

Create a new directory for your Terraform project and set up your providers:

        # main.tf
        terraform {
          required_providers {
            aws = {
              source  = "hashicorp/aws"
              version = "~> 5.0"
            }
            kubernetes = {
              source  = "hashicorp/kubernetes"
              version = "~> 2.23"
            }
            helm = {
              source  = "hashicorp/helm"
              version = "~> 2.11"
            }
            datadog = {
              source  = "DataDog/datadog"
              version = "~> 3.23"
            }
          }
        }

        provider "aws" {
          region = "us-east-1" # Or your preferred AWS region
        }

        resource "aws_eks_cluster" "main" {
          # ... (EKS cluster configuration details) ...
        }

        data "aws_eks_cluster" "main" {
          name = aws_eks_cluster.main.name
        }

        data "aws_eks_cluster_auth" "main" {
          name = aws_eks_cluster.main.name
        }

        provider "kubernetes" {
          host                   = data.aws_eks_cluster.main.endpoint
          cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority[0].data)
          token                  = data.aws_eks_cluster_auth.main.token
        }

        provider "helm" {
          kubernetes {
            host                   = data.aws_eks_cluster.main.endpoint
            cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority[0].data)
            token                  = data.aws_eks_cluster_auth.main.token
          }
        }

        provider "datadog" {
          api_key = var.datadog_api_key
          app_key = var.datadog_app_key
          # site = "datadoghq.com" for US, "datadoghq.eu" for EU, etc.
        }

        variable "datadog_api_key" {
          description = "Datadog API Key"
          type        = string
          sensitive   = true
        }

        variable "datadog_app_key" {
          description = "Datadog Application Key"
          type        = string
          sensitive   = true
        }
    

2. Deploy AWS EKS Cluster (using a module for brevity)

It's highly recommended to use a well-maintained Terraform EKS module, such as the official HashiCorp EKS module, to provision your cluster. This example assumes you have an EKS cluster running, or you're adding it here.

        # eks.tf (Example using community module)
        module "eks_cluster" {
          source  = "terraform-aws-modules/eks/aws"
          version = "~> 19.0"

          cluster_name    = "my-observability-eks"
          cluster_version = "1.27"

          vpc_id     = "vpc-0xxxxxxxxxxxxxxx" # Replace with your VPC ID
          subnet_ids = ["subnet-0xxxxxxxxxxxxxxx", "subnet-0yyyyyyyyyyyyyyy"] # Replace with your subnet IDs

          eks_managed_node_groups = {
            default = {
              instance_types = ["t3.medium"]
              min_size       = 1
              max_size       = 3
              desired_size   = 1
            }
          }

          # Enable IRSA for service accounts to interact with AWS services
          enable_irsa = true
        }
    

3. Deploy Datadog Agent with Terraform Helm Provider

The Datadog Agent is deployed as a DaemonSet to ensure it runs on every node, collecting metrics, logs, and traces. We'll use the Helm provider in Terraform to manage its deployment.

First, create an IAM Policy and Role for the Datadog Agent for IRSA (IAM Roles for Service Accounts). This grants the agent necessary permissions to collect data from AWS resources without using long-lived credentials.

        # datadog_irsa.tf
        resource "aws_iam_policy" "datadog_agent" {
          name        = "${module.eks_cluster.cluster_name}-datadog-agent-policy"
          description = "IAM policy for Datadog Agent on EKS"
          policy      = data.aws_iam_policy_document.datadog_agent_policy.json
        }

        data "aws_iam_policy_document" "datadog_agent_policy" {
          statement {
            effect = "Allow"
            actions = [
              "ec2:DescribeInstances",
              "ec2:DescribeVolumes",
              "ec2:DescribeTags",
              "cloudwatch:ListMetrics",
              "cloudwatch:GetMetricData",
              "logs:DescribeLogGroups",
              "logs:FilterLogEvents",
              "s3:GetBucketLocation",
              "s3:ListAllMyBuckets",
              "eks:DescribeCluster",
            ]
            resources = ["*"]
          }
        }

        resource "aws_iam_role" "datadog_agent" {
          name               = "${module.eks_cluster.cluster_name}-datadog-agent-role"
          assume_role_policy = data.aws_iam_policy_document.datadog_agent_assume_role_policy.json
        }

        data "aws_iam_policy_document" "datadog_agent_assume_role_policy" {
          statement {
            effect  = "Allow"
            actions = ["sts:AssumeRoleWithWebIdentity"]
            principals {
              type        = "Federated"
              identifiers = [module.eks_cluster.oidc_provider_arn]
            }
            condition {
              test     = "StringEquals"
              variable = "${replace(module.eks_cluster.oidc_provider, "https://", "")}:sub"
              values   = ["system:serviceaccount:datadog:datadog-agent"]
            }
          }
        }

        resource "aws_iam_role_policy_attachment" "datadog_agent" {
          role       = aws_iam_role.datadog_agent.name
          policy_arn = aws_iam_policy.datadog_agent.arn
        }
    

Now, the Helm chart for Datadog Agent:

Datadog Agent Helm Chart Configuration

        # datadog_helm.tf
        resource "kubernetes_namespace" "datadog" {
          metadata {
            name = "datadog"
          }
        }

        resource "helm_release" "datadog_agent" {
          name       = "datadog"
          repository = "https://helm.datadoghq.com"
          chart      = "datadog"
          namespace  = kubernetes_namespace.datadog.metadata[0].name
          version    = "2.34.0" # Use the latest stable version

          set {
            name  = "datadog.apiKey"
            value = var.datadog_api_key
            sensitive = true
          }
          set {
            name  = "datadog.appKey"
            value = var.datadog_app_key
            sensitive = true
          }
          set {
            name  = "datadog.site"
            value = "datadoghq.com" # Or your Datadog site (e.g., datadoghq.eu)
          }

          values = [
            <<-EOT
              clusterAgent:
                enabled: true
              agents:
                tagEndpoints: true
                kubelet:
                  enabled: true
                  host:
                    tlsVerify: false # For self-signed certificates, consider true with proper config
              logs:
                enabled: true
                containerCollectAll: true
              apm:
                enabled: true
              processAgent:
                enabled: true
              kubernetes:
                kubeStateMetrics:
                  enabled: true # Datadog can collect KSM metrics directly
              aws:
                installDeps: true # Install boto3 for AWS integrations
                // For IRSA, configure serviceAccount.create to false and provide existing role ARN
                hostTags:
                  // Example of custom tags
                  env: production
                  project: my-app
              clusterAgent:
                rbac:
                  create: true
                metricsProvider:
                  enabled: true # Enable Datadog as a custom metrics provider for HPA
              serviceAccount:
                create: false # We'll manage the SA role via IRSA
                name: datadog-agent # Must match the SA name in the assume role policy
                annotations:
                  eks.amazonaws.com/role-arn: ${aws_iam_role.datadog_agent.arn}
              providers:
                aws:
                  region: ${data.aws_region.current.name} # Ensure the correct region is used
            EOT
          ]
          depends_on = [
            kubernetes_namespace.datadog,
            aws_iam_role_policy_attachment.datadog_agent
          ]
        }
    

4. Deploy Prometheus (Kube-Prometheus-Stack) with Terraform Helm Provider

The kube-prometheus-stack Helm chart includes Prometheus, Alertmanager, Grafana, and various exporters. We'll deploy this for comprehensive Kubernetes native monitoring. For simplicity, we'll configure Datadog to scrape metrics from Prometheus endpoints rather than running a fully separate Grafana/Alertmanager setup if Datadog is your primary tool.

        # prometheus_helm.tf
        resource "kubernetes_namespace" "monitoring" {
          metadata {
            name = "monitoring"
          }
        }

        resource "helm_release" "kube_prometheus_stack" {
          name       = "kube-prometheus-stack"
          repository = "https://prometheus-community.github.io/helm-charts"
          chart      = "kube-prometheus-stack"
          namespace  = kubernetes_namespace.monitoring.metadata[0].name
          version    = "48.2.1" # Use the latest stable version

          values = [
            <<-EOT
              // Disable Grafana if Datadog is your primary dashboarding tool
              grafana:
                enabled: false
              // Optionally disable Alertmanager if PagerDuty integration is purely via Datadog
              alertmanager:
                enabled: true # Keep Alertmanager for local alerts, or disable for Datadog-only alerts
                config:
                  global:
                    resolve_timeout: 5m
                  route:
                    group_by: ['alertname']
                    group_wait: 30s
                    group_interval: 5m
                    repeat_interval: 12h
                    receiver: 'webhook-datadog' # Send alerts to Datadog if Alertmanager is enabled
                  receivers:
                    - name: 'webhook-datadog'
                      webhook_configs:
                        - url: "https://api.datadoghq.com/api/v1/integration/prometheus" # Datadog Prometheus integration endpoint
                          send_resolved: true
                          headers:
                            "DD-API-KEY": ${var.datadog_api_key}
                            "DD-APPLICATION-KEY": ${var.datadog_app_key}
              prometheus:
                prometheusSpec:
                  serviceMonitorSelectorNilUsesHelmValues: false
                  podMonitorSelectorNilUsesHelmValues: false
            EOT
          ]
          depends_on = [
            kubernetes_namespace.monitoring
          ]
        }
    

Note on Prometheus & Datadog Integration: The above Prometheus configuration shows how to configure Alertmanager to send alerts to Datadog's Prometheus integration endpoint. Additionally, the Datadog Agent, if configured correctly, can scrape metrics directly from Prometheus exporters (like kube-state-metrics and node-exporter which are part of kube-prometheus-stack) and unify them in Datadog dashboards.

5. Configure PagerDuty Integration with Datadog

We'll use the Datadog Terraform provider to configure the PagerDuty integration. This ensures that alerts triggered in Datadog can be automatically routed to PagerDuty as incidents.

        # pagerduty_integration.tf
        resource "datadog_integration_pagerduty" "main" {
          api_token = var.pagerduty_api_token
        }

        variable "pagerduty_api_token" {
          description = "PagerDuty API token (for Datadog integration)"
          type        = string
          sensitive   = true
        }

        # You can also define PagerDuty services and service integrations here
        # For example, creating a specific PagerDuty service via Terraform:
        // resource "datadog_integration_pagerduty_service" "eks_alerts" {
        //   service_name = "EKS Cluster Critical Alerts"
        //   service_key = var.pagerduty_eks_service_key # This should be a PagerDuty Integration Key for the service
        // }
    

Ensure your pagerduty_api_token is a Datadog API token generated in PagerDuty specifically for the Datadog integration, or a general PagerDuty API token with appropriate permissions. You would then reference this integration within your Datadog monitors to send alerts to PagerDuty.

6. Apply the Terraform Configuration

Once all your .tf files are set up, run the following commands:

terraform init: Initializes the working directory.
terraform plan -var="datadog_api_key=YOUR_DD_API_KEY" -var="datadog_app_key=YOUR_DD_APP_KEY" -var="pagerduty_api_token=YOUR_PD_API_TOKEN": Review the changes Terraform will make. Replace placeholders with your actual keys. For production, use secure methods for sensitive variables (e.g., AWS Secrets Manager, Vault, or environment variables).
terraform apply -var="datadog_api_key=YOUR_DD_API_KEY" -var="datadog_app_key=YOUR_DD_APP_KEY" -var="pagerduty_api_token=YOUR_PD_API_TOKEN": Apply the configuration. Confirm with 'yes' when prompted.

Validation and Post-Deployment Checks

After applying your Terraform configuration, it's crucial to validate that all components are functioning as expected:

Kubernetes Pods: Run kubectl get pods -n datadog and kubectl get pods -n monitoring to ensure all Datadog Agent and Prometheus stack pods are running and healthy.
Datadog Dashboard: Log in to your Datadog account. You should start seeing metrics, logs, and traces from your EKS cluster, nodes, and applications. Check the Infrastructure List, Kubernetes Dashboard, and Log Explorer.
Prometheus UI (Optional): If you enabled the Prometheus UI (by exposing its service), access it to confirm metrics scraping from EKS components.
PagerDuty Integration: In Datadog, go to Integrations -> PagerDuty and ensure the integration is active. Create a test monitor in Datadog that triggers an alert to PagerDuty to verify the end-to-end incident flow.

Best Practices and Advanced Configurations

Custom Metrics: Leverage Datadog's Custom Metrics or Prometheus exporters to monitor specific application-level metrics crucial for your business logic.
Cost Optimization: Monitor your Datadog usage closely, especially for custom metrics and log ingestion. Optimize log retention and filter unnecessary logs. Consider a hybrid approach where less critical metrics stay in Prometheus, while key metrics are forwarded to Datadog.
Security: Implement robust network policies in EKS to restrict communication between namespaces. Regularly review IAM roles and policies for least privilege. Use secrets management for API keys.
Alerting Strategy: Define clear alert thresholds, severity levels, and on-call rotations in PagerDuty. Leverage Datadog's advanced anomaly detection and forecasting capabilities to reduce alert fatigue.
Distributed Tracing: Instrument your applications with Datadog APM (or OpenTelemetry agents that can export to Datadog) to gain deep insights into request flows and performance bottlenecks.

Troubleshooting Common Issues

Datadog Agent Pods Not Running: Check logs of the Datadog Agent pods (kubectl logs -n datadog <pod-name>). Verify API and App keys are correct. Ensure the service account has the correct IAM role ARN annotation for IRSA.
No Metrics in Datadog: Confirm that the Datadog Agent is running. Check agent status (kubectl exec -it -n datadog <datadog-agent-pod> -- agent status). Review Datadog configuration in the Helm values for enabled integrations (e.g., Kubernetes, logs, APM).
PagerDuty Alerts Not Triggering: Verify the PagerDuty integration in Datadog is configured with a valid API token. Check your Datadog monitor's notification settings to ensure PagerDuty is selected as a recipient. Perform a test alert.
Prometheus Not Scraping: Access the Prometheus UI (if enabled) and check its targets status. Ensure ServiceMonitors/PodMonitors are correctly defined and that their labels match the Prometheus instance's selector.

Conclusion

Building a robust observability stack for AWS EKS is fundamental for operating resilient and high-performing cloud-native applications. By leveraging Terraform, you can automate the deployment of Datadog, Prometheus, and PagerDuty, establishing a powerful framework for monitoring, logging, tracing, and incident response. This integrated approach provides your DevOps and SRE teams with the tools needed to gain deep insights into your EKS clusters, proactively identify issues, and respond effectively to ensure optimal application health and user satisfaction. Continuously refine your observability strategy as your EKS environment evolves, ensuring you maintain full visibility into your critical workloads.

CloudStack YAML

Terraform AWS EKS Observability Stack with Datadog, Prometheus, and PagerDuty