Terraform-Managed AWS EKS Observability with Datadog, Prometheus, and PagerDuty Alerting

In today's dynamic cloud-native landscape, ensuring robust observability for Kubernetes clusters is paramount. AWS Elastic Kubernetes Service (EKS) offers a powerful platform for deploying containerized applications, but without a comprehensive monitoring and alerting strategy, managing its complexity can become a significant challenge. This technical guide will walk you through establishing a state-of-the-art observability stack for your Terraform-managed AWS EKS cluster, leveraging the strengths of Datadog for unified monitoring, Prometheus for deep metric collection, and PagerDuty for reliable incident management.

Architecture Pro-Tip: Unified Observability Platform

While Prometheus is excellent for scraping metrics, a unified platform like Datadog excels at consolidating metrics, logs, traces, and events across your entire EKS environment and beyond. Use Datadog as your single pane of glass, integrating Prometheus-native metrics into it, rather than maintaining disparate monitoring systems. This simplifies dashboards, alerting, and cross-correlation, drastically reducing MTTR (Mean Time To Resolution).

Understanding the Core Components of Your Observability Stack

Before diving into implementation, let's briefly review the role of each technology in our integrated observability solution.

AWS EKS: The Foundation of Your Cloud-Native Applications

AWS EKS provides a managed Kubernetes service, abstracting away the operational complexities of running a Kubernetes control plane. It's a highly scalable and reliable platform, making it a popular choice for deploying microservices and containerized workloads. Monitoring its health, performance, and resource utilization is crucial for maintaining application stability and optimizing costs.

Datadog: Your Unified Monitoring & Analytics Platform

Datadog is a comprehensive SaaS-based monitoring and analytics platform that brings together metrics, logs, and traces from your entire infrastructure and applications. For EKS, Datadog offers deep integration, providing visibility into cluster health, node performance, pod resource usage, application performance, and more. Its powerful dashboards, anomaly detection, and machine learning-driven insights are invaluable for proactive operations.

Prometheus: The Standard for Kubernetes Metric Collection

Prometheus is an open-source monitoring system with a dimensional data model, flexible query language (PromQL), and powerful alerting capabilities. It's become the de facto standard for collecting metrics in Kubernetes environments. While Datadog has its own agent, it can also seamlessly scrape and ingest Prometheus-formatted metrics, allowing you to leverage existing Prometheus instrumentation within your EKS applications and services.

PagerDuty: Incident Response & Management

PagerDuty is a leading incident management platform that integrates with monitoring tools like Datadog to ensure critical alerts are routed to the right teams, at the right time. It facilitates on-call scheduling, escalations, and streamlined incident response workflows, transforming raw alerts into actionable incidents that minimize downtime and operational impact.

Prerequisites for Implementation

Before you begin, ensure you have the following in place:

An active AWS account with appropriate IAM permissions to create EKS clusters, VPCs, and related resources.
A Datadog account with an API Key and Application Key.
A PagerDuty account with an integration key for Datadog.
Terraform CLI (v1.0+) installed.
AWS CLI configured with credentials.
kubectl CLI installed and configured.
Helm CLI (v3+) installed.

Step-by-Step Implementation with Terraform

1. Provisioning Your AWS EKS Cluster with Terraform

We'll use the popular terraform-aws-modules/eks/aws module, which simplifies EKS cluster deployment considerably. This module handles the VPC, subnets, IAM roles, security groups, and the EKS control plane itself.

First, set up your AWS provider and define variables:

provider "aws" {
  region = "us-east-1"
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.20"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
  }
}

variable "cluster_name" {
  description = "Name of the EKS cluster"
  type        = string
  default     = "my-observability-eks"
}
variable "vpc_cidr" {
  description = "CIDR block for the VPC"
  type        = string
  default     = "10.0.0.0/16"
}

Next, define your VPC and EKS cluster using the module:

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "${var.cluster_name}-vpc"
  cidr = var.vpc_cidr

  azs              = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnets  = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets   = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]
  database_subnets = ["10.0.7.0/24", "10.0.8.0/24", "10.0.9.0/24"]

  enable_nat_gateway = true
  single_nat_gateway = true
  enable_dns_hostnames = true

  tags = {
    Environment = "Dev"
    Project     = "Observability"
  }
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0"

  cluster_name    = var.cluster_name
  cluster_version = "1.28"

  vpc_id                   = module.vpc.vpc_id
  subnet_ids               = module.vpc.private_subnets
  control_plane_subnet_ids = module.vpc.public_subnets # Use public for simplicity, private is recommended for prod

  enable_irsa = true

  eks_managed_node_groups = {
    default = {
      name           = "default-ng"
      instance_types = ["t3.medium"]
      min_size       = 2
      max_size       = 4
      desired_size   = 2
      vpc_security_group_ids = [module.vpc.default_security_group_id]
      block_device_mappings = {
        xvda = {
          device_name = "/dev/xvda"
          ebs = {
            volume_size           = 20
            volume_type           = "gp3"
            delete_on_termination = true
          }
        }
      }
    }
  }

  tags = {
    Environment = "Dev"
    Project     = "Observability"
  }

  # For Kubernetes provider configuration
  cluster_endpoint         = module.eks.cluster_endpoint
  cluster_certificate_authority_data = module.eks.cluster_certificate_authority_data
}

Don't forget to configure the Kubernetes and Helm providers to interact with your newly created EKS cluster:

data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}

2. Integrating Datadog for Comprehensive EKS Monitoring

The Datadog Agent is deployed as a DaemonSet on your EKS cluster to collect metrics, logs, and traces. We'll deploy it using the Helm provider in Terraform.

You'll need your Datadog API and Application keys. It's best practice to store these securely, for example, using AWS Secrets Manager or environment variables.

Create a main.tf or separate file for Datadog integration:

resource "kubernetes_secret" "datadog_api_key" {
  metadata {
    name = "datadog-secret"
    namespace = "default" # Consider a dedicated namespace like 'monitoring'
  }

  data = {
    "api-key" = var.datadog_api_key
    "app-key" = var.datadog_app_key
  }
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Consider a dedicated namespace like 'monitoring'

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "clusterName"
    value = var.cluster_name
  }

  set {
    name  = "kubeStateMetricsExternal.enabled"
    value = "true"
  }

  set {
    name  = "targetSystem"
    value = "linux"
  }

  # Enable APM and Log collection
  set {
    name  = "apm.enabled"
    value = "true"
  }
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }

  # Enable Prometheus scrape for metrics
  set {
    name  = "datadog.prometheusScrape.enabled"
    value = "true"
  }
  set {
    name  = "datadog.confd.prometheus_example.yaml"
    value = <<EOF
ad_identifiers:
  - my-app
init_config:
instances:
  - prometheus_url: http://%%host%%:8080/metrics
    namespace: my-app
    metrics:
      - my_app_requests_total
      - my_app_errors_total
EOF
  }

  values = [
    "${file("datadog-values.yaml")}" # For more extensive configurations
  ]
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

The datadog-values.yaml file (optional, but good for complex configs) would contain additional Helm chart values.

3. Leveraging Prometheus Metrics within Datadog

As shown in the Datadog Agent Helm configuration above, the agent can be configured to scrape Prometheus endpoints directly. By setting datadog.prometheusScrape.enabled=true, the agent will discover and collect metrics from pods exposing Prometheus metrics.

For applications that expose Prometheus metrics, you can annotate your Kubernetes pods or services to instruct the Datadog Agent on where to scrape:

annotations:
  ad.datadoghq.com/my-app.check_names: '["prometheus"]'
  ad.datadoghq.com/my-app.init_configs: '[{}]'
  ad.datadoghq.com/my-app.instances: |
    [
      {
        "prometheus_url": "http://%%host%%:8080/metrics",
        "namespace": "my_app",
        "metrics": ["my_app_requests_total", "my_app_errors_total"]
      }
    ]

This method is more dynamic and scalable. After applying your Terraform, the Datadog Agent will automatically start collecting these Prometheus metrics, making them available in your Datadog dashboards and for alerting.

4. Configuring PagerDuty for Incident Alerting

Datadog integrates directly with PagerDuty to send alerts for critical incidents. First, you need to set up the integration in Datadog's UI: navigate to Integrations -> Integrations -> PagerDuty and follow the steps to connect your Datadog account to PagerDuty using your PagerDuty integration key.

Once integrated, you can create Datadog monitors that trigger PagerDuty incidents. While the integration itself is typically configured via the UI, you can manage Datadog monitors using the Datadog Terraform provider:

resource "datadog_monitor" "eks_node_cpu_utilization" {
  name               = "[EKS] Node CPU Utilization High on ${var.cluster_name}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.cluster_name}} by {host} > 80"
  message            = "CPU usage on EKS node {{host.name}} is above 80%. @pagerduty-your-service-name"
  escalation_message = "CPU usage remains high, escalating to on-call."
  tags               = ["environment:dev", "team:devops", "eks"]
  
  # Notification options
  notify_no_data    = false
  new_group_delay   = 60
  no_data_timeframe = 20

  # PagerDuty specific settings (configured in Datadog UI, referenced by name)
  # Ensure "your-service-name" matches the PagerDuty service name configured in Datadog integration
  # The @pagerduty-your-service-name in the message tells Datadog to use that integration.
}

resource "datadog_monitor" "eks_node_memory_utilization" {
  name               = "[EKS] Node Memory Utilization High on ${var.cluster_name}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:kubernetes.memory.usage.total{cluster_name:${var.cluster_name}} by {host} > 75"
  message            = "Memory usage on EKS node {{host.name}} is above 75%. @pagerduty-your-service-name"
  tags               = ["environment:dev", "team:devops", "eks"]
  
  notify_no_data    = false
  new_group_delay   = 60
  no_data_timeframe = 20
}

Replace @pagerduty-your-service-name with the actual service name you configured in the Datadog PagerDuty integration. This simple annotation in the message field is how Datadog knows to send the alert to PagerDuty.

Ready-to-Use Terraform Configuration Examples

Here's a consolidated example of the Terraform files for deploying your EKS cluster and integrating Datadog with basic Prometheus scraping and PagerDuty alerting. Remember to replace placeholder values for Datadog keys and PagerDuty service names.

# main.tf
# Configure AWS and Terraform providers
provider "aws" {
  region = "us-east-1"
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.20"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
  }
}

# Variables for EKS and Datadog
variable "cluster_name" {
  description = "Name of the EKS cluster"
  type        = string
  default     = "my-observability-eks"
}
variable "vpc_cidr" {
  description = "CIDR block for the VPC"
  type        = string
  default     = "10.0.0.0/16"
}
variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}
variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}
variable "pagerduty_service_name" {
  description = "The name of the PagerDuty service configured in Datadog"
  type        = string
  default     = "your-pagerduty-service-name" # IMPORTANT: Update this
}

# AWS VPC Module
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "${var.cluster_name}-vpc"
  cidr = var.vpc_cidr

  azs              = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnets  = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets   = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]

  enable_nat_gateway = true
  single_nat_gateway = true
  enable_dns_hostnames = true

  tags = {
    Environment = "Dev"
    Project     = "Observability"
  }
}

# AWS EKS Cluster Module
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0"

  cluster_name    = var.cluster_name
  cluster_version = "1.28"

  vpc_id                   = module.vpc.vpc_id
  subnet_ids               = module.vpc.private_subnets
  control_plane_subnet_ids = module.vpc.public_subnets 

  enable_irsa = true

  eks_managed_node_groups = {
    default = {
      name           = "default-ng"
      instance_types = ["t3.medium"]
      min_size       = 2
      max_size       = 4
      desired_size   = 2
    }
  }

  tags = {
    Environment = "Dev"
    Project     = "Observability"
  }
}

# Configure Kubernetes and Helm providers using EKS cluster outputs
data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}

# Configure Datadog provider
provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

# Deploy Datadog Agent via Helm
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Or dedicated 'monitoring' namespace

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "clusterName"
    value = var.cluster_name
  }

  set {
    name  = "kubeStateMetricsExternal.enabled"
    value = "true"
  }

  set {
    name  = "targetSystem"
    value = "linux"
  }

  set {
    name  = "apm.enabled"
    value = "true"
  }
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }
  set {
    name  = "datadog.prometheusScrape.enabled"
    value = "true"
  }
  # Example custom Prometheus scrape configuration (optional, can be done via annotations)
  # set {
  #   name  = "datadog.confd.prometheus_my_app.yaml"
  #   value = <<EOT
  # ad_identifiers:
  #   - my-app
  # init_config:
  # instances:
  #   - prometheus_url: http://%%host%%:8080/metrics
  #     namespace: my_app
  #     metrics:
  #       - my_app_requests_total
  #       - my_app_errors_total
  # EOT
  # }
}

# Datadog Monitors for EKS (integrated with PagerDuty)
resource "datadog_monitor" "eks_node_cpu_utilization" {
  name               = "[EKS] Node CPU Utilization High on ${var.cluster_name}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.cluster_name}} by {host} > 80"
  message            = "CPU usage on EKS node {{host.name}} is above 80%. @pagerduty-${var.pagerduty_service_name}"
  escalation_message = "CPU usage remains high, escalating to on-call."
  tags               = ["environment:dev", "team:devops", "eks", "observability"]
  
  notify_no_data    = false
  new_group_delay   = 60
  no_data_timeframe = 20
}

resource "datadog_monitor" "eks_node_memory_utilization" {
  name               = "[EKS] Node Memory Utilization High on ${var.cluster_name}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:kubernetes.memory.usage.total{cluster_name:${var.cluster_name}} by {host} > 75"
  message            = "Memory usage on EKS node {{host.name}} is above 75%. @pagerduty-${var.pagerduty_service_name}"
  tags               = ["environment:dev", "team:devops", "eks", "observability"]
  
  notify_no_data    = false
  new_group_delay   = 60
  no_data_timeframe = 20
}

# Output EKS Kubeconfig (for manual access if needed)
output "kubeconfig" {
  description = "Kubeconfig for the EKS cluster"
  value       = module.eks.kubeconfig
  sensitive   = true
}

To apply this configuration:

Save the code as main.tf.
Initialize Terraform: terraform init
Plan the deployment: terraform plan
Apply the changes: terraform apply

You will be prompted for your Datadog API and App keys. Once applied, your EKS cluster will be up, the Datadog Agent deployed, and basic monitoring with PagerDuty alerting configured.

Best Practices for EKS Observability

Centralized Logging: Ensure all application and system logs are collected and sent to Datadog. Leverage Datadog's log processing pipelines for parsing and enrichment.
Application Performance Monitoring (APM): Instrument your applications with Datadog APM libraries to collect traces and visualize service dependencies. This is crucial for microservices architectures.
Custom Dashboards: Build tailored Datadog dashboards for different teams (Dev, Ops, SRE) focusing on the metrics and logs most relevant to their responsibilities.
Resource Tagging: Consistently tag your AWS resources and Kubernetes objects. Datadog automatically ingests these tags, enabling powerful filtering and aggregation in your monitoring.
Granular Alerting: Create specific alerts for different services and components. Avoid alert fatigue by setting meaningful thresholds and leveraging Datadog's anomaly detection.
Cost Monitoring: Utilize Datadog's cloud cost management features or integrate with AWS Cost Explorer to keep an eye on your EKS expenditures alongside performance.

Troubleshooting Common Observability Issues

Datadog Agent Pods Not Running: Check kubectl get pods -n default (or your namespace) and kubectl describe pod <datadog-agent-pod-name> -n default for events and error messages. Ensure nodes have sufficient resources.
Missing Metrics/Logs in Datadog:
- Verify the Datadog API key and Application key are correct and active.
- Check the Datadog Agent logs: kubectl logs <datadog-agent-pod-name> -n default.
- Ensure proper Prometheus annotations are applied to your application pods if scraping custom metrics.
PagerDuty Alerts Not Firing:
- Confirm the Datadog-PagerDuty integration is correctly configured in the Datadog UI.
- Double-check that the @pagerduty-your-service-name tag in your Datadog monitor message exactly matches the service name configured in Datadog.
- Test the Datadog monitor manually to see if it triggers an alert within Datadog.

Conclusion

Building a robust observability pipeline for AWS EKS is no longer optional; it's a critical component of successful cloud-native operations. By leveraging Terraform for infrastructure as code, Datadog for unified monitoring, Prometheus for detailed metric collection, and PagerDuty for efficient incident response, you can gain unparalleled visibility and control over your EKS environments.

This guide provides a solid foundation for your Terraform-managed AWS EKS observability stack. Remember to adapt and expand upon these configurations to meet the unique needs and complexities of your applications and organizational structure. Continuous improvement in your observability practices will lead to faster problem resolution, improved system reliability, and ultimately, better user experiences.

CloudStack YAML