Production-Ready AWS EKS Observability with Prometheus, Datadog, and PagerDuty via Terraform

In the dynamic world of cloud-native applications, maintaining robust observability for Kubernetes clusters is not just a best practice—it's a critical necessity. For organizations leveraging AWS EKS, establishing a comprehensive monitoring, logging, and alerting strategy ensures application reliability, performance, and operational efficiency. This guide will walk you through building a production-ready AWS EKS observability stack using a powerful combination of Prometheus for metrics, Datadog for unified visibility, and PagerDuty for incident management, all orchestrated predictably and scalably with Terraform.

Architecture Pro-Tip:

Always aim for a multi-layered observability strategy. While Prometheus excels at granular metric collection within the cluster, Datadog provides a holistic, cross-platform view encompassing metrics, logs, traces, and APM. Integrating these with PagerDuty ensures critical issues are never missed, transforming raw data into actionable alerts and streamlined incident response. Leverage Infrastructure as Code (IaC) with Terraform from day one to enforce consistency, version control your observability setup, and enable rapid disaster recovery.

Why This Observability Stack for AWS EKS?

A modern EKS environment generates an immense volume of data from various sources: pods, nodes, services, and the EKS control plane itself. Effectively managing this data requires specialized tools that work in synergy:

Prometheus: The de-facto standard for Kubernetes native monitoring. It collects metrics from applications and infrastructure, offering powerful querying (PromQL) and local alerting capabilities. When deployed via the kube-prometheus-stack Helm chart, it includes Grafana for visualization and Alertmanager for routing alerts.
Datadog: A unified observability platform that aggregates metrics (including those from Prometheus via integrations), logs, and traces. It provides powerful dashboards, anomaly detection, APM, and network monitoring, giving a single pane of glass for all your AWS EKS and broader cloud infrastructure. Datadog's extensive integrations simplify data correlation.
PagerDuty: The industry leader in incident response management. PagerDuty takes alerts from Datadog (or other sources) and orchestrates notifications, on-call schedules, escalation policies, and post-incident analysis, ensuring critical issues are addressed promptly by the right team.
Terraform: An Infrastructure as Code (IaC) tool that allows you to define and provision your entire EKS cluster, its IAM roles, networking, and crucially, all your observability components in a repeatable, version-controlled manner. This eliminates manual configuration drift and accelerates deployment.

Prerequisites

Before diving into the Terraform configurations, ensure you have the following:

An AWS Account with administrative access.
Terraform CLI installed (v1.0+ recommended).
AWS CLI configured with appropriate credentials.
kubectl installed and configured to interact with your EKS cluster.
Helm CLI installed (v3+ recommended).
A Datadog Account with API and Application keys.
A PagerDuty Account with an API token and a Service Integration Key.
An existing AWS EKS Cluster or the ability to create one using Terraform (this guide assumes an existing one for brevity in the core examples but mentions its creation).

Setting Up IAM Roles for Service Accounts (IRSA)

AWS EKS supports IAM Roles for Service Accounts (IRSA), allowing Kubernetes service accounts to assume IAM roles. This is crucial for securely granting AWS permissions to your Prometheus and Datadog components without distributing AWS credentials.

For Datadog:

The Datadog Cluster Agent and Datadog Agent will need permissions to fetch EKS metadata, logs from CloudWatch, and potentially other AWS services depending on your configuration. Terraform can provision an IAM role and associate it with a Kubernetes Service Account.

For Prometheus (`kube-prometheus-stack`):

While Prometheus itself typically doesn't need direct AWS IAM access for metric collection, Alertmanager or other components within the stack might, for example, to send notifications via SNS or manage autoscaling groups. For this guide, we'll focus on the Datadog IRSA as it's more commonly required.

Deploying Prometheus (kube-prometheus-stack) with Terraform

The kube-prometheus-stack Helm chart is a comprehensive solution that includes Prometheus, Alertmanager, Grafana, and various exporters. We'll use Terraform's helm_release resource to manage its deployment.

Key configurations typically involve:

Specifying values to tailor Prometheus and Alertmanager to your EKS cluster.
Enabling or disabling specific components like Prometheus node-exporter, kube-state-metrics, etc.
Configuring persistent storage for Prometheus data.
Setting up ingress rules for Grafana if public access is desired (though often Grafana is accessed privately or via VPN).

Integrating Datadog with Terraform

Datadog integration involves deploying the Datadog Agent and Datadog Cluster Agent, typically via a Helm chart, and then configuring monitors and alerts using the Datadog Terraform provider.

Deploying Datadog Agent via Helm:

The Datadog Agent will run on each EKS node to collect host-level metrics, logs, and trace data. The Cluster Agent runs as a deployment to collect cluster-level metrics and manage API calls to Datadog. IRSA is crucial here for secure AWS integration.

Provide your Datadog API and APP keys securely.
Enable necessary features like APM, Log Collection, Network Performance Monitoring (NPM).
Configure the IAM Role ARN for the Datadog Service Account to leverage IRSA.
Enable the Prometheus integration for Datadog to scrape metrics from existing Prometheus exporters or the Prometheus server itself.

Configuring Datadog Monitors with Terraform:

Once data flows into Datadog, you can define monitors to track key metrics and log patterns. The datadog_monitor resource in Terraform allows you to codify these alerts, ensuring consistency and version control.

Connecting PagerDuty for Incident Management

PagerDuty acts as the final destination for critical alerts that require human intervention. You'll typically configure Datadog monitors to send notifications to PagerDuty services.

Terraform for PagerDuty Services and Integrations:

The pagerduty_service and pagerduty_integration resources allow you to create and manage PagerDuty services, escalation policies, and integration points (e.g., for Datadog) directly through Terraform.

Full Terraform Configuration Example

Illustrative Terraform Code for EKS Observability Stack

This example demonstrates how to set up the core components. Remember to replace placeholders like <YOUR_AWS_ACCOUNT_ID>, <YOUR_EKS_CLUSTER_NAME>, <YOUR_AWS_REGION>, <DATADOG_API_KEY>, <DATADOG_APP_KEY>, <PAGERDUTY_SERVICE_NAME>, and <PAGERDUTY_INTEGRATION_KEY> with your actual values. This assumes your EKS cluster is already provisioned and its kubeconfig is accessible.

resource "aws_iam_openid_connect_provider" "eks" {
  url             = data.aws_eks_cluster.eks_cluster.identity[0].oidc[0].issuer
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.tls_certificate.eks.certificates[0].sha1_fingerprint]
}

resource "tls_certificate" "eks" {
  url = data.aws_eks_cluster.eks_cluster.identity[0].oidc[0].issuer
}

data "aws_eks_cluster" "eks_cluster" {
  name = "<YOUR_EKS_CLUSTER_NAME>"
}

data "aws_eks_cluster_auth" "eks_cluster_auth" {
  name = "<YOUR_EKS_CLUSTER_NAME>"
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.eks_cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.eks_cluster_auth.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.eks_cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.eks_cluster_auth.token
  }
}

provider "datadog" {
  api_key = "<DATADOG_API_KEY>"
  app_key = "<DATADOG_APP_KEY>"
}

provider "pagerduty" {
  token = var.pagerduty_api_token # Use a sensitive variable for the PagerDuty API token
}

# --- IAM Role for Datadog Agent (IRSA) ---
resource "aws_iam_policy" "datadog_agent_policy" {
  name_prefix = "datadog-agent-policy-"
  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action = [
          "ec2:DescribeInstances",
          "ec2:DescribeVolumes",
          "ec2:DescribeTags",
          "logs:DescribeLogGroups",
          "logs:DescribeLogStreams",
          "logs:GetLogEvents",
          "cloudwatch:ListMetrics",
          "cloudwatch:GetMetricData",
          "s3:ListAllMyBuckets",
          "s3:GetBucketLocation",
          "s3:GetBucketTagging"
        ],
        Effect   = "Allow",
        Resource = "*"
      },
    ],
  })
}

resource "aws_iam_role" "datadog_agent_irsa" {
  name_prefix = "datadog-agent-irsa-"
  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect = "Allow",
        Principal = {
          Federated = "arn:aws:iam::<YOUR_AWS_ACCOUNT_ID>:oidc-provider/${replace(data.aws_eks_cluster.eks_cluster.identity[0].oidc[0].issuer, "https://", "")}"
        },
        Action = "sts:AssumeRoleWithWebIdentity",
        Condition = {
          StringEquals = {
            "${replace(data.aws_eks_cluster.eks_cluster.identity[0].oidc[0].issuer, "https://", "")}:sub" = "system:serviceaccount:datadog:datadog-agent"
          }
        }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "datadog_agent_attach" {
  role       = aws_iam_role.datadog_agent_irsa.name
  policy_arn = aws_iam_policy.datadog_agent_policy.arn
}

# --- Helm Release for kube-prometheus-stack ---
resource "helm_release" "kube_prometheus_stack" {
  name       = "kube-prometheus-stack"
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  namespace  = "monitoring"
  create_namespace = true
  version    = "58.1.0" # Use a specific chart version
  timeout    = 600

  values = [
    yamlencode({
      grafana = {
        enabled = true
        adminPassword = "prom-admin-password" # production use secrets manager
        service = {
          type = "ClusterIP"
        }
      }
      prometheus = {
        prometheusSpec = {
          retention = "7d" # Adjust retention as needed
          # Add service account annotation if Prometheus needs IRSA for custom tasks
        }
      }
      alertmanager = {
        enabled = true
        config = {
          global = {
            resolve_timeout = "5m"
          }
          route = {
            group_by        = ["alertname", "cluster", "service"]
            group_wait      = "30s"
            group_interval  = "5m"
            repeat_interval = "4h"
            receiver        = "null" # Placeholder, configure receivers if not using Datadog for final alerts
          }
          receivers = [
            {
              name = "null"
            }
          ]
        }
      }
    })
  ]
}

# --- Helm Release for Datadog Agent ---
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true
  version    = "2.38.1" # Use a specific chart version
  timeout    = 600

  set {
    name  = "datadog.apiKey"
    value = "<DATADOG_API_KEY>"
  }
  set {
    name  = "datadog.appKey"
    value = "<DATADOG_APP_KEY>"
  }

  values = [
    yamlencode({
      datadog = {
        kubeStateMetricsCore = {
          enabled = true
        }
        # Enable IRSA for Datadog Agent
        serviceAccount = {
          create = true
          name   = "datadog-agent"
          annotations = {
            "eks.amazonaws.com/role-arn" = aws_iam_role.datadog_agent_irsa.arn
          }
        }
        clusterAgent = {
          enabled = true
          serviceAccount = {
            create = true
            name   = "datadog-cluster-agent"
            annotations = {
              "eks.amazonaws.com/role-arn" = aws_iam_role.datadog_agent_irsa.arn # Re-use or create specific for Cluster Agent
            }
          }
          conf = {
            clusterName = data.aws_eks_cluster.eks_cluster.name
          }
        }
        # Enable features
        apm = {
          enabled = true
        }
        logs = {
          enabled = true
          containerCollectAll = true
          # For collecting EKS logs from CloudWatch
          # podLogs = {
          #   enabled = true
          # }
          # For DaemonSet
          # filepaths = [
          #   { path = "/var/log/containers/*.log" }
          # ]
        }
        networkMonitoring = {
          enabled = true
        }
        processAgent = {
          enabled = true
        }
        # Prometheus integration for Datadog to scrape metrics
        # For auto-discovery of Prometheus endpoints
        prometheusScrape = {
          enabled = true
        }
        # For external_labels in Prometheus to map to Datadog tags
        tags = [
          "cluster_name:${data.aws_eks_cluster.eks_cluster.name}",
          "env:production"
        ]
      }
    })
  ]
}

# --- PagerDuty Service and Integration ---
resource "pagerduty_escalation_policy" "example_policy" {
  name      = "EKS Critical Alerts Policy"
  num_loops = 2
  rule {
    delay = 5
    target {
      type = "user"
      id   = "Pxxxxxxxxxxxxxxxxx" # Replace with a valid PagerDuty User ID
    }
  }
}

resource "pagerduty_service" "eks_critical_service" {
  name                    = "<PAGERDUTY_SERVICE_NAME>"
  description             = "EKS Critical Service for Production Alerts"
  escalation_policy       = pagerduty_escalation_policy.example_policy.id
  auto_resolve_timeout    = "14400" # 4 hours
  acknowledgement_timeout = "600"   # 10 minutes
}

resource "pagerduty_integration" "datadog_integration" {
  name    = "Datadog Integration for EKS"
  type    = "datadog_inbound_integration"
  service = pagerduty_service.eks_critical_service.id
}

# --- Example Datadog Monitor for EKS Node CPU Utilization ---
resource "datadog_monitor" "high_node_cpu" {
  name               = "[EKS] High Node CPU Utilization on {{host.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:system.cpu.idle{cluster_name:${data.aws_eks_cluster.eks_cluster.name}} by {host} < 20"
  message            = "CPU utilization is too high on host {{host.name}} in EKS cluster {{cluster_name}}. @slack-devops @pagerduty-{{pagerduty_integration.datadog_integration.service_name}}"
  tags               = ["eks", "cpu", "critical"]
  priority           = 1
  restricted_roles   = [] # Empty means visible to all
  notify_no_data     = false
  new_group_delay    = 60
  no_data_timeframe  = 20
  include_tags       = true

  monitor_thresholds {
    critical = 20
    warning  = 30
  }

  escalation_message = "Still high CPU! Please investigate immediately. @pagerduty-{{pagerduty_integration.datadog_integration.service_name}}"
}

Configuring Datadog Monitors and PagerDuty Alerts

With your agents deployed and data flowing, the next step is to define meaningful alerts. Datadog's flexible monitoring capabilities allow for a wide range of alert types. When creating monitors in Datadog, specify the PagerDuty integration as a notification channel for critical alerts.

Key Metrics to Monitor in EKS:

Node Health: CPU, memory, disk utilization, network I/O, node status (ready/not ready).
Pod Health: Restarts, container failures, pending pods, resource limits/requests.
Deployment Status: Unavailable replicas, rollout failures.
Networking: Latency, error rates, DNS resolution issues.
API Server: Latency, error rates, request queueing.
Workload Specific Metrics: Application-specific KPIs collected via Prometheus exporters or custom metrics.

When a Datadog monitor's threshold is breached and configured to notify PagerDuty, it will trigger an incident on the associated PagerDuty service, initiating your predefined escalation policies.

Best Practices for Production Observability

Granular Alerting: Avoid alert fatigue. Start with critical alerts for immediate impact, then build out warning and informational alerts as you understand your system's baseline.
Cost Optimization: Monitor Datadog usage closely. Optimize log ingestion by filtering non-essential logs and adjust Prometheus retention policies.
Security: Regularly review IAM policies and ensure least privilege for all components. Use secrets management for API keys.
Dashboards & Visualization: Leverage both Grafana (for Prometheus-specific data) and Datadog dashboards for clear, actionable insights into your EKS cluster's health and performance.
Testing Alerts: Periodically test your PagerDuty integrations and Datadog monitors to ensure they function as expected and reach the correct on-call teams.
Automated Remediation: For certain predictable alerts, explore integrating automated remediation actions via tools like AWS Lambda or Kubernetes operators.

Troubleshooting Common Issues

Datadog Agent/Prometheus Not Reporting Data:

Check Pod Status: Use kubectl get pods -n <namespace> to ensure agents are running.
Review Logs: kubectl logs <pod-name> -n <namespace> for any errors (e.g., API key issues, permission denied).
Verify IRSA: Ensure the IAM Role ARN in your Helm values matches the role created by Terraform, and the trust policy correctly references the EKS OIDC provider and service account.
Network Connectivity: Confirm EKS nodes can reach Datadog endpoints and Prometheus endpoints.

PagerDuty Alerts Not Triggering:

Datadog Monitor Status: Check the Datadog monitor's history to see if it's actually triggering an alert.
Integration Key: Ensure the PagerDuty integration key used in Datadog (or within the monitor message, e.g., @pagerduty-<INTEGRATION_KEY>) is correct and corresponds to the service created by Terraform.
PagerDuty Service Configuration: Verify the escalation policy and on-call schedules are correctly set up in PagerDuty.

Conclusion

Achieving production-ready AWS EKS observability is a continuous journey, but with Prometheus, Datadog, and PagerDuty orchestrated by Terraform, you establish a robust and scalable foundation. This powerful combination provides deep insights into your Kubernetes workloads, proactive alerting, and efficient incident response, ensuring the reliability and performance of your critical applications. By embracing Infrastructure as Code, your observability stack becomes as resilient and manageable as the infrastructure it monitors.

Start implementing this stack today to elevate your EKS operations from reactive firefighting to proactive, intelligent monitoring.

CloudStack YAML