Terraform for AWS EKS: Production Datadog & Prometheus Observability with PagerDuty Integration

Architecture Pro-Tip: Unified Observability Strategy

When designing your production observability stack for AWS EKS, prioritize a unified approach. While Prometheus excels in metric collection within Kubernetes, integrating it with a comprehensive platform like Datadog provides advanced dashboarding, anomaly detection, log management, and APM capabilities. Complement this with an incident management system like PagerDuty to ensure critical alerts are actionable and routed to the right teams promptly, minimizing Mean Time To Resolution (MTTR). Leverage Infrastructure as Code (IaC) with Terraform to manage the entire observability setup, ensuring consistency, version control, and auditability across environments.

Terraform for AWS EKS: Production Datadog & Prometheus Observability with PagerDuty Integration

In the dynamic world of cloud-native applications, maintaining robust observability for your Kubernetes clusters is not merely a best practice; it's a critical requirement for production stability and performance. AWS EKS provides a highly scalable and reliable foundation for running containerized workloads, but without proper monitoring, logging, and alerting, diagnosing issues in a distributed environment can become a daunting task. This guide will walk you through setting up a comprehensive observability stack for your AWS EKS cluster using Terraform, integrating Datadog for unified monitoring, leveraging Prometheus for metric collection, and ensuring critical incident response with PagerDuty.

Why a Unified Observability Stack is Essential for EKS

Proactive Issue Detection: Identify anomalies and potential problems before they impact users.
Faster Root Cause Analysis: Correlate metrics, logs, and traces across your EKS cluster and applications.
Improved Application Performance: Optimize resource utilization and identify performance bottlenecks.
Reliable Incident Response: Automate alerting and escalation to the right teams.
Operational Efficiency: Streamline monitoring setup and management through Infrastructure as Code (IaC).

Prerequisites

Before you begin, ensure you have the following:

An active AWS Account with necessary permissions to manage EKS, IAM, and other AWS resources.
Terraform installed (v1.0.0 or higher).
kubectl installed and configured to interact with your EKS cluster.
A provisioned AWS EKS cluster. This guide assumes you have an existing cluster or know how to create one using Terraform.
A Datadog Account with your API and Application keys.
A PagerDuty Account with an Integration Key for Datadog.
Helm installed, as we will use the Terraform Helm provider to deploy the Datadog Agent.

Core Components Explained

Datadog for EKS Observability

Datadog offers a unified platform for monitoring, logging, and tracing. For EKS, the Datadog Agent runs as a DaemonSet, collecting metrics, logs, and traces from your cluster nodes, pods, and applications. It can also integrate with AWS services directly, providing a complete view of your cloud infrastructure. Datadog's ability to ingest Prometheus metrics makes it an excellent choice for consolidating your monitoring efforts.

Prometheus for EKS Metrics

Prometheus is the de facto standard for open-source monitoring in Kubernetes. It scrapes metrics from configured targets and stores them. While powerful, managing Prometheus at scale, especially for long-term storage and cross-cluster visibility, can be complex. Datadog's integration with Prometheus allows you to leverage Prometheus's robust metric collection capabilities within your EKS cluster while offloading storage, aggregation, and advanced analytics to Datadog.

PagerDuty for Incident Management

PagerDuty is a leading incident management platform that integrates with monitoring tools like Datadog to ensure critical alerts are escalated to the right on-call teams immediately. It provides sophisticated alerting policies, on-call schedules, and incident response automation, crucial for maintaining high availability in production environments.

Terraform Setup for Datadog Agent, Prometheus & PagerDuty Integration

1. AWS and Kubernetes Provider Configuration

First, define your AWS and Kubernetes providers. The Kubernetes provider needs to be configured to connect to your EKS cluster.

provider "aws" {
  region = "us-east-1"
}

data "aws_eks_cluster" "eks_cluster" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "eks_cluster_auth" {
  name = var.eks_cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.eks_cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.eks_cluster_auth.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.eks_cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.eks_cluster_auth.token
  }
}

variable "eks_cluster_name" {
  description = "The name of your EKS cluster"
  type        = string
}

2. Datadog Provider Configuration

Configure the Datadog provider using your API and Application keys. These should ideally be stored securely, e.g., using AWS Secrets Manager or environment variables.

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

3. Deploying Datadog Agent with Helm (via Terraform)

We'll use the Terraform helm_release resource to deploy the Datadog Agent. This approach ensures Infrastructure as Code for your monitoring agent. The values block is crucial for configuring the agent, including enabling Prometheus metric collection, APM, and log collection.

Important: For production EKS clusters, it is highly recommended to use IAM Roles for Service Accounts (IRSA) for the Datadog Agent to provide granular permissions without distributing AWS credentials directly. This setup involves creating an IAM role and associating it with the Kubernetes Service Account used by the Datadog Agent.

Below is an example snippet showing the Helm release and a simplified IAM policy for context. You would typically create the IAM Role and Service Account separately and reference them here.

# Create an IAM Policy for the Datadog Agent (example, refine permissions)
resource "aws_iam_policy" "datadog_agent_policy" {
  name        = "${var.eks_cluster_name}-datadog-agent-policy"
  description = "IAM policy for Datadog Agent on EKS"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "ec2:DescribeInstances",
          "ec2:DescribeVolumes",
          "tag:GetResources",
          "logs:DescribeLogGroups",
          "logs:DescribeLogStreams",
          "logs:GetLogEvents",
          "sts:AssumeRole" # Needed for IRSA
        ]
        Effect   = "Allow"
        Resource = "*"
      },
    ]
  })
}

# Create IAM Role for Service Account (IRSA)
resource "aws_iam_role" "datadog_agent_irsa" {
  name = "${var.eks_cluster_name}-datadog-agent-irsa"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Federated = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${replace(data.aws_eks_cluster.eks_cluster.identity.0.oidc.0.issuer, "https://", "")}"
        }
        Action = "sts:AssumeRoleWithWebIdentity"
        Condition = {
          StringEquals = {
            "${replace(data.aws_eks_cluster.eks_cluster.identity.0.oidc.0.issuer, "https://", "")}:sub" : "system:serviceaccount:default:datadog-agent" # Adjust namespace if needed
          }
        }
      },
    ]
  })
}

# Attach policy to the role
resource "aws_iam_role_policy_attachment" "datadog_agent_attachment" {
  role       = aws_iam_role.datadog_agent_irsa.name
  policy_arn = aws_iam_policy.datadog_agent_policy.arn
}

data "aws_caller_identity" "current" {}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Consider a dedicated 'datadog' namespace
  version    = "2.33.0"  # Use a specific chart version for production

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }

  # Enable APM
  set {
    name  = "datadog.apm.enabled"
    value = "true"
  }

  # Enable Log Collection
  set {
    name  = "datadog.logs.enabled"
    value = "true"
  }
  set {
    name  = "datadog.logs.containerCollectAll"
    value = "true"
  }

  # Enable Process Monitoring
  set {
    name  = "datadog.processAgent.enabled"
    value = "true"
  }

  # Enable Host-level network performance monitoring
  set {
    name  = "datadog.networkMonitoring.enabled"
    value = "true"
  }

  # Enable admission controller for APM/logging injection
  set {
    name  = "datadog.admissionController.enabled"
    value = "true"
  }

  # Kubernetes Metrics Collection (via kube-state-metrics and cAdvisor)
  set {
    name  = "datadog.kubeStateMetrics.enabled"
    value = "true"
  }

  # Configure Prometheus Scraper (via Datadog Agent)
  set {
    name  = "datadog.containerExclusions"
    value = "image:datadog/agent" # Exclude Datadog's own metrics
  }

  # Example: Enable Prometheus scraping for a specific application
  # You would typically annotate your application pods for auto-discovery
  # or configure specific jobs in datadog.yaml if using raw Prometheus targets.
  # The Datadog Agent auto-discovers Prometheus endpoints based on annotations.
  # For detailed Prometheus integration, refer to Datadog's official docs.

  # IRSA Configuration for Datadog Agent (replace with your EKS Service Account if needed)
  set {
    name  = "datadog.serviceAccount.create"
    value = "true" # Let Helm create the service account for the agent
  }
  set {
    name  = "datadog.serviceAccount.name"
    value = "datadog-agent"
  }
  set {
    name  = "datadog.serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
    value = aws_iam_role.datadog_agent_irsa.arn
  }

  # Environment variables for the agent if needed (e.g., custom tags)
  set {
    name  = "datadog.env[0].name"
    value = "DD_TAGS"
  }
  set {
    name  = "datadog.env[0].value"
    value = "env:production,cluster:${var.eks_cluster_name}"
  }
}

4. Integrating PagerDuty with Datadog

To integrate PagerDuty, we'll use the datadog_integration_pagerduty resource. This registers your PagerDuty service with Datadog, allowing you to select it as a notification option for your monitors.

resource "datadog_integration_pagerduty" "pagerduty_integration" {
  services {
    service_name  = "EKS Production Incidents"
    service_key   = var.pagerduty_service_key
  }
  # Add more services here if needed
}

variable "pagerduty_service_key" {
  description = "PagerDuty Integration Key for the Datadog service"
  type        = string
  sensitive   = true
}

5. Creating Datadog Monitors with PagerDuty Integration

Now, let's create a sample Datadog monitor for CPU utilization on your EKS nodes and configure it to send alerts to PagerDuty. You can define various types of monitors (metric, anomaly, log, APM, etc.).

resource "datadog_monitor" "eks_node_cpu_alert" {
  name               = "[EKS-${var.eks_cluster_name}] High Node CPU Utilization"
  type               = "metric alert"
  query              = "avg(last_5m):avg:system.cpu.idle{kube_cluster_name:${var.eks_cluster_name}} by {host} < 20"
  message            = <

Ready-to-Use Configuration: Full Terraform ExampleHere's a consolidated example of the Terraform configuration. Remember to replace placeholder values and refine permissions for your specific production needs. Store sensitive variables securely.

# main.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

data "aws_caller_identity" "current" {}

data "aws_eks_cluster" "eks_cluster" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "eks_cluster_auth" {
  name = var.eks_cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.eks_cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.eks_cluster_auth.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.eks_cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.eks_cluster_auth.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

# IAM Policy for Datadog Agent (minimal example, refine for production)
resource "aws_iam_policy" "datadog_agent_policy" {
  name        = "${var.eks_cluster_name}-datadog-agent-policy"
  description = "IAM policy for Datadog Agent on EKS with IRSA"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "ec2:DescribeInstances",
          "ec2:DescribeVolumes",
          "tag:GetResources",
          "logs:DescribeLogGroups",
          "logs:DescribeLogStreams",
          "logs:GetLogEvents",
          "sts:AssumeRole", # Required for IRSA
        ]
        Effect   = "Allow"
        Resource = "*"
      },
      {
        Action   = ["s3:GetObject"]
        Effect   = "Allow"
        Resource = "arn:aws:s3:::datadog-agent-helm-charts/*" # If fetching from private S3
      }
    ]
  })
}

# IAM Role for Service Account (IRSA) for Datadog Agent
resource "aws_iam_role" "datadog_agent_irsa" {
  name = "${var.eks_cluster_name}-datadog-agent-irsa"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Federated = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${replace(data.aws_eks_cluster.eks_cluster.identity.0.oidc.0.issuer, "https://", "")}"
        }
        Action = "sts:AssumeRoleWithWebIdentity"
        Condition = {
          StringEquals = {
            "${replace(data.aws_eks_cluster.eks_cluster.identity.0.oidc.0.issuer, "https://", "")}:sub" : "system:serviceaccount:default:datadog-agent", # Adjust namespace and service account name if different
            "${replace(data.aws_eks_cluster.eks_cluster.identity.0.oidc.0.issuer, "https://", "")}:aud" : "sts.amazonaws.com"
          }
        }
      },
    ]
  })
  tags = {
    Cluster = var.eks_cluster_name
  }
}

# Attach policy to the role
resource "aws_iam_role_policy_attachment" "datadog_agent_attachment" {
  role       = aws_iam_role.datadog_agent_irsa.name
  policy_arn = aws_iam_policy.datadog_agent_policy.arn
}

# Datadog Agent Helm Release
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Consider a dedicated 'datadog' namespace
  version    = "2.33.0"  # Pin to a specific, tested version

  values = [
    yamlencode({
      datadog = {
        apiKey = var.datadog_api_key
        appKey = var.datadog_app_key
        site   = "datadoghq.com" # or eu.datadoghq.com etc.

        # Enable core features
        apm = {
          enabled = true
        }
        logs = {
          enabled          = true
          containerCollectAll = true
        }
        processAgent = {
          enabled = true
        }
        networkMonitoring = {
          enabled = true
        }
        admissionController = {
          enabled = true
        }
        kubeStateMetrics = {
          enabled = true
        }
        # Enable Prometheus Scraper
        # The agent auto-discovers Prometheus endpoints based on pod annotations.
        # For example, annotate your application pods with:
        # prometheus.io/scrape: "true"
        # prometheus.io/path: "/metrics"
        # prometheus.io/port: "8080"
        #
        # For specific Prometheus config or JMX checks, you might need to
        # use datadog.confd or extend the values block.
        # Example: datadog.confd."my-app-prometheus.yaml" = "..."

        # IRSA Configuration
        serviceAccount = {
          create      = true
          name        = "datadog-agent" # Must match serviceaccount name in IRSA assume role policy
          annotations = {
            "eks.amazonaws.com/role-arn" = aws_iam_role.datadog_agent_irsa.arn
          }
        }

        # Global tags for all metrics, logs, traces
        env = [
          { name = "DD_TAGS", value = "env:production,cluster:${var.eks_cluster_name}" }
        ]
      }
      # Set resources limits for production environment
      agents = {
        resources = {
          limits = {
            cpu    = "500m"
            memory = "512Mi"
          }
          requests = {
            cpu    = "200m"
            memory = "256Mi"
          }
        }
      }
      clusterAgent = {
        resources = {
          limits = {
            cpu    = "200m"
            memory = "256Mi"
          }
          requests = {
            cpu    = "100m"
            memory = "128Mi"
          }
        }
      }
    })
  ]
}

# Datadog PagerDuty Integration
resource "datadog_integration_pagerduty" "pagerduty_integration" {
  services {
    service_name  = "EKS Production Incidents"
    service_key   = var.pagerduty_service_key
  }
}

# Datadog Monitor for EKS Node CPU Utilization
resource "datadog_monitor" "eks_node_cpu_alert" {
  name               = "[EKS-${var.eks_cluster_name}] High Node CPU Utilization"
  type               = "metric alert"
  query              = "avg(last_5m):avg:system.cpu.idle{kube_cluster_name:${var.eks_cluster_name}} by {host} < 20"
  message            = <Verification and TestingAfter applying the Terraform configuration:
Verify Datadog Agent pods: Run kubectl get pods -n default -l app=datadog. You should see Datadog Agent pods (one per node) and a Datadog Cluster Agent pod running.
Check Datadog UI: Log into your Datadog account.
            Navigate to "Infrastructure" -> "Containers" to see your EKS cluster, nodes, and pods reporting.
Go to "Integrations" -> "PagerDuty" to confirm the integration is active.
Check "Monitors" -> "Manage Monitors" to see your newly created monitors.

        
Trigger an Alert (optional, for testing): Artificially increase CPU usage on an EKS node or simulate pod restarts to ensure the Datadog monitor triggers and a PagerDuty incident is created.
Best Practices for Production ObservabilityGranular IAM Permissions: Always follow the principle of least privilege for the Datadog Agent's IRSA role.
Tagging Strategy: Implement a consistent tagging strategy (e.g., env:production, service:my-app, team:devops) for all your AWS resources and Kubernetes objects. Datadog leverages these tags for filtering, aggregation, and context.
Custom Metrics and Application APM: Instrument your applications to emit custom metrics (Prometheus format for easy ingestion) and traces (Datadog APM).
Log Management Best Practices: Standardize log formats (e.g., JSON), enrich logs with relevant metadata, and configure Datadog to parse and index them effectively.
Prevent Alert Fatigue: Carefully tune your monitors and use composite alerts, anomaly detection, and suppression rules to ensure only actionable alerts are sent to PagerDuty.
Dashboards for Context: Create comprehensive Datadog dashboards for your EKS clusters and applications to provide quick overviews and aid in troubleshooting.
Version Control: Keep your entire observability configuration (Terraform, monitor definitions) in version control.


Troubleshooting Common IssuesDatadog Agent Pods Not Running: Check kubectl describe pod <datadog-agent-pod> for errors. Common issues include insufficient resources, incorrect API/APP keys, or IAM role misconfigurations.
No Data in Datadog: Ensure the Datadog Agent has network connectivity to Datadog endpoints. Verify the API/APP keys. Check agent logs for errors related to metric collection. For Prometheus metrics, ensure your application pods are correctly annotated for Datadog's autodiscovery.
PagerDuty Incidents Not Triggering: Verify the @pagerduty-<service_name> syntax in your Datadog monitor message matches the PagerDuty service name configured in Datadog. Check the Datadog event stream for monitor alerts.
IRSA Permissions: If metrics related to AWS services (e.g., EBS volumes) are missing, double-check the IAM policy attached to the Datadog Agent's service account and ensure the OIDC provider URL and subject match the EKS cluster and service account.
ConclusionBy leveraging Terraform, Datadog, Prometheus, and PagerDuty, you can build a robust, scalable, and automated observability solution for your AWS EKS production workloads. This comprehensive setup ensures that your infrastructure and applications are continuously monitored, critical issues are identified proactively, and your on-call teams are effectively alerted and equipped to respond. Embracing Infrastructure as Code for your observability stack brings consistency, reliability, and agility, allowing your teams to focus on delivering value rather than firefighting.

CloudStack YAML