Terraform AWS EKS Observability Stack with Datadog, Prometheus, and PagerDuty Alerting

In the dynamic world of cloud-native applications, maintaining robust observability for Kubernetes clusters is not just a best practice, it's a necessity. This comprehensive guide outlines how to deploy a powerful observability stack on AWS EKS using Terraform for infrastructure as code (IaC), integrating Datadog for unified monitoring, Prometheus for deep metrics collection, and PagerDuty for critical incident management and alerting. This stack provides a holistic view of your EKS environment, ensuring high availability and operational excellence.

Architecture Pro-Tip:

Always design your observability stack with a "single pane of glass" philosophy. While Prometheus excels at collecting raw, high-cardinality metrics directly from Kubernetes, Datadog acts as the aggregation and visualization layer, consolidating metrics, logs, and traces. Integrating PagerDuty then ensures that critical anomalies detected by Datadog translate into actionable alerts, minimizing mean time to resolution (MTTR). Leverage Terraform's modularity to manage each component independently, ensuring reusability and maintainability across multiple EKS environments.

Why This Observability Stack?

Modern applications running on AWS EKS demand sophisticated monitoring capabilities. Each component in this stack plays a vital role:

Terraform: Enables declarative, repeatable, and version-controlled deployment of your entire EKS cluster and its observability agents. This eliminates configuration drift and streamlines environment provisioning.
AWS EKS: Provides a managed Kubernetes service, offloading the operational overhead of managing the Kubernetes control plane.
Datadog: A comprehensive monitoring platform offering end-to-end visibility. It collects metrics, logs, and traces, provides powerful dashboards, anomaly detection, and synthetic monitoring. Its native integrations simplify Kubernetes monitoring.
Prometheus: An open-source monitoring system, particularly strong for collecting metrics from Kubernetes and its workloads. It uses a pull model, scraping HTTP endpoints for time-series data.
PagerDuty: The industry standard for incident management. It ingests alerts from Datadog, intelligently routes them to on-call teams, and provides robust escalation policies and notification channels.

Core Components and Their Synergy

Terraform for Infrastructure as Code (IaC)

Terraform manages the full lifecycle of your EKS cluster, IAM roles, and the deployment of monitoring agents via Helm charts. This ensures consistency and reproducibility.

AWS EKS Cluster Provisioning

We'll provision a robust EKS cluster, ensuring proper networking (VPC, subnets, security groups) and node group configurations for our observability agents.

Datadog Agent Deployment

The Datadog Agent, deployed as a DaemonSet across EKS worker nodes, collects system metrics, application logs, and traces. It supports Kubernetes-specific integrations to gather metrics from the API server, Kubelet, and other core components. Crucially, it can also act as a Prometheus scraper, forwarding those metrics to Datadog for unified visibility.

Prometheus and Kube-Prometheus-Stack

While Datadog provides extensive Kubernetes monitoring, Prometheus (often deployed via the Kube-Prometheus-Stack Helm chart) offers specialized, granular control over Kubernetes-native metrics, including kube-state-metrics and Node Exporter. This allows for deep dives and specific custom metric collection that can then be scraped and forwarded by Datadog.

PagerDuty for Actionable Alerts

Datadog's powerful monitoring capabilities are integrated directly with PagerDuty. When a Datadog monitor triggers an alert based on defined thresholds or anomaly detection, it automatically creates an incident in PagerDuty, notifying the appropriate on-call team based on escalation policies.

Prerequisites

AWS Account with appropriate IAM permissions.
Terraform CLI (v1.0+).
AWS CLI configured.
kubectl CLI configured.
Datadog Account with an API Key and Application Key.
PagerDuty Account with a Service Integration Key.
Helm CLI (for local testing/chart inspection, though Terraform manages Helm releases).

Implementing the Stack with Terraform

Step 1: EKS Cluster Setup

Provisioning the EKS cluster is the foundation. We'll use the eks module from the Terraform registry for a streamlined setup, ensuring correct VPC, IAM roles, and node group configurations.

Step 2: IAM Roles for Service Accounts (IRSA)

For secure access to AWS services (e.g., CloudWatch, S3 for logs), Datadog and Prometheus agents should leverage IRSA. This involves creating IAM roles and associating them with Kubernetes service accounts.

Step 3: Datadog Agent Deployment via Helm

The Datadog Agent is deployed using its official Helm chart. Key configurations include setting your Datadog API and Application keys, enabling EKS integration, Prometheus scraping, and log collection.

Step 4: Prometheus (Kube-Prometheus-Stack) Deployment via Helm

The Kube-Prometheus-Stack bundles Prometheus, Grafana, Alertmanager, and various exporters. While Datadog will be our primary dashboard, Kube-Prometheus-Stack provides a robust Prometheus instance for raw metric collection and local troubleshooting if needed. We'll configure it to expose metrics that Datadog can then scrape.

Step 5: Integrating Datadog and PagerDuty

Once Datadog is collecting data, you'll configure its integration with PagerDuty. This involves setting up a PagerDuty service integration in Datadog and defining monitors that trigger incidents when thresholds are breached. Terraform can also manage Datadog monitors and PagerDuty services using their respective providers.

Example Terraform Configuration Snippets

Below is a simplified example of how you might structure your Terraform files. This assumes you have an existing VPC and are using the terraform-aws-modules/eks/aws module for EKS cluster provisioning.

main.tf - Core EKS, Datadog, and Prometheus Setup

resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true
  tags = { Name = "eks-observability-vpc" }
}

resource "aws_subnet" "private" {
  for_each = toset(["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"])
  vpc_id     = aws_vpc.main.id
  cidr_block = each.key
  availability_zone = element(data.aws_availability_zones.available.names, index(toset(["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]), each.key))
  tags = { Name = "eks-private-subnet-${replace(each.key, "/\\./", "-")}" }
}

resource "aws_internet_gateway" "gw" {
  vpc_id = aws_vpc.main.id
  tags = { Name = "eks-observability-igw" }
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.gw.id
  }
}

resource "aws_subnet" "public" {
  for_each = toset(["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"])
  vpc_id     = aws_vpc.main.id
  cidr_block = each.key
  availability_zone = element(data.aws_availability_zones.available.names, index(toset(["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]), each.key))
  map_public_ip_on_launch = true
  tags = { Name = "eks-public-subnet-${replace(each.key, "/\\./", "-")}" }
}

resource "aws_route_table_association" "public" {
  for_each = aws_subnet.public
  subnet_id      = each.value.id
  route_table_id = aws_route_table.public.id
}

data "aws_availability_zones" "available" {
  state = "available"
}

module "eks_cluster" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0"

  cluster_name    = "eks-observability-cluster"
  cluster_version = "1.28"
  vpc_id          = aws_vpc.main.id
  subnet_ids      = [for s in aws_subnet.private : s.id]

  eks_managed_node_groups = {
    default = {
      instance_types = ["t3.medium"]
      desired_size   = 2
      min_size       = 1
      max_size       = 3
      disk_size      = 20
    }
  }

  tags = {
    Environment = "Dev"
    Project     = "Observability"
  }
}

resource "kubernetes_namespace" "datadog" {
  metadata {
    name = "datadog"
  }
}

resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = "monitoring"
  }
}

resource "helm_release" "datadog_agent" {
  name       = "datadog-agent"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = kubernetes_namespace.datadog.metadata[0].name
  version    = "2.37.0" # Use a stable and recent version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "datadog.site"
    value = "datadoghq.com" # Or eu.datadoghq.com, etc.
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "kubeStateMetricsNetworkPolicy.enabled"
    value = "false" # Adjust based on your network policies
  }

  set {
    name  = "logs.enabled"
    value = "true"
  }

  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }

  set {
    name  = "processAgent.enabled"
    value = "true"
  }

  set {
    name  = "apm.enabled"
    value = "true"
  }

  set {
    name  = "systemProbe.enabled"
    value = "true"
  }

  set {
    name  = "prometheusScrape.enabled"
    value = "true"
  }

  set {
    name  = "datadog.env[0].name"
    value = "DD_EKS_FARGATE"
  }
  set {
    name  = "datadog.env[0].value"
    value = "true"
  }
}

resource "helm_release" "kube_prometheus_stack" {
  name       = "kube-prometheus-stack"
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  namespace  = kubernetes_namespace.monitoring.metadata[0].name
  version    = "51.0.0" # Use a stable and recent version

  values = [
    "${file("values/prometheus-values.yaml")}"
  ]
}
    

values/prometheus-values.yaml (Example)

alertmanager:
  enabled: false # Datadog will handle alerting
grafana:
  enabled: false # Datadog will be our primary visualization
prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    # Configure storage, retention, etc.
    retention: 7d
kubeStateMetrics:
  # Expose metrics for Datadog to scrape
  service:
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "8080"
nodeExporter:
  # Expose metrics for Datadog to scrape
  service:
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "9100"
    

Remember to replace placeholder values like var.datadog_api_key and var.datadog_app_key with your actual Datadog credentials, preferably managed securely via Terraform variables or environment variables.

Configuring Datadog for PagerDuty Alerting

After deploying the Datadog Agent, ensure you configure the PagerDuty integration within Datadog. This typically involves:

Navigating to Integrations -> PagerDuty in your Datadog console.
Adding an integration by providing your PagerDuty Service API Key (or Global API Key for older integrations).
Once integrated, when creating new monitors in Datadog (e.g., for high CPU utilization, low available memory, or critical pod restarts), you can select PagerDuty as an alert recipient in the notification section (@pagerduty-servicename).

For advanced setups, consider using the Datadog Terraform provider to manage monitors, dashboards, and PagerDuty integrations as code, further enhancing your IaC approach.

Troubleshooting and Best Practices

Verifying Datadog Agent Status

Check the Datadog Agent status in your EKS cluster:

kubectl get pods -n datadog
kubectl logs -f  -n datadog
    

Also, review the Datadog Agent status page within the Datadog UI to ensure data is being collected from your EKS cluster.

Prometheus Metrics Verification

You can port-forward to the Prometheus server deployed by Kube-Prometheus-Stack to ensure metrics are being scraped correctly:

kubectl get svc -n monitoring
kubectl port-forward svc/kube-prometheus-stack-prometheus 9090:9090 -n monitoring
# Then access http://localhost:9090 in your browser
    

Security Considerations

Use IRSA for all Kubernetes workloads requiring AWS access.
Store sensitive credentials (Datadog API/App keys, PagerDuty keys) securely using AWS Secrets Manager or Vault, referencing them in Terraform.
Implement strict network policies (NetworkPolicy) in EKS to control traffic between namespaces and pods.

Cost Optimization

Monitor Datadog ingestion volumes to optimize metric and log collection, filtering out unnecessary data.
Right-size your EKS node groups to avoid over-provisioning resources.
Consider selective deployment of monitoring agents based on application criticality.

Conclusion

Building a robust observability stack for AWS EKS is paramount for managing complex, cloud-native environments. By leveraging Terraform for declarative infrastructure, Datadog for unified monitoring, Prometheus for granular metrics, and PagerDuty for incident management, organizations can achieve unparalleled visibility and rapid response capabilities. This guide provides a solid foundation for deploying such a stack, ensuring your applications remain performant, reliable, and secure.

CloudStack YAML