Terraform for AWS EKS Production Observability with Datadog and PagerDuty

Architecture Pro-Tip: Layered Observability Strategy

For production AWS EKS environments, implement a layered observability strategy. This means not only monitoring the Kubernetes cluster itself (nodes, pods, deployments) but also the underlying AWS infrastructure (EC2 instances, EBS volumes, VPC flow logs) and application performance (APM, distributed tracing). Terraform facilitates this holistic approach by enabling consistent, repeatable deployments of all observability components, ensuring no critical blind spots exist from infrastructure to application layer.

Terraform for AWS EKS Production Observability with Datadog and PagerDuty

In today's dynamic cloud-native landscape, ensuring the reliability and performance of production workloads running on AWS Elastic Kubernetes Service (EKS) is paramount. Terraform, as the leading Infrastructure as Code (IaC) tool, provides the declarative power to provision and manage not just your EKS clusters, but also the critical observability tools required to keep them healthy. This guide delves into deploying a robust observability stack for EKS using Terraform, integrating Datadog for comprehensive monitoring and PagerDuty for effective incident response.

Why Terraform for EKS Observability?

Leveraging Terraform for your observability setup offers several key advantages:

Consistency and Repeatability: Define your monitoring agents, dashboards, alerts, and incident response workflows as code, ensuring identical configurations across development, staging, and production environments.
Version Control: Store your observability configurations in Git, allowing for change tracking, collaboration, and easy rollback to previous states.
Automation: Automate the deployment and updates of Datadog agents, monitors, and PagerDuty services, reducing manual effort and potential for human error.
Scalability: Easily scale your observability infrastructure as your EKS clusters grow, applying consistent monitoring standards across new services and deployments.

The Power Duo: Datadog and PagerDuty

Datadog and PagerDuty form a formidable combination for production observability:

Datadog: Comprehensive Monitoring: Datadog provides end-to-end visibility across your EKS environment. It collects metrics, logs, and traces from your Kubernetes clusters, applications, and underlying AWS infrastructure. With capabilities like APM, network performance monitoring, security monitoring, and synthetic monitoring, Datadog offers a unified view of your system's health.
PagerDuty: Intelligent Incident Response: PagerDuty takes the actionable insights from Datadog and converts them into structured incidents, routing them to the right on-call teams based on escalation policies. It streamlines the incident management lifecycle, reducing mean time to acknowledge (MTTA) and mean time to resolution (MTTR).

Prerequisites

Before you begin, ensure you have the following:

An active AWS account with appropriate permissions to manage EKS, IAM, and other AWS resources.
Terraform CLI installed (v1.0+ recommended).
AWS CLI installed and configured.
A Datadog account with API and Application keys.
A PagerDuty account with a personal API token.
An existing AWS EKS cluster. This guide assumes your EKS cluster is already provisioned and you have kubeconfig access.
Helm CLI installed locally, as we'll use the Terraform Helm provider.

Terraform Configuration for Datadog on EKS

1. Configure AWS IAM for Datadog Agent

The Datadog Agent deployed to your EKS cluster will need IAM permissions to collect metadata and potentially metrics from AWS services. We'll create an IAM policy and role, which can then be associated with the Kubernetes service account used by the Datadog Agent via IAM Roles for Service Accounts (IRSA).

2. Deploy Datadog Agent via Helm and Terraform

The recommended way to deploy the Datadog Agent to Kubernetes is via its Helm chart. Terraform's Helm provider allows you to manage Helm releases declaratively.

3. Define Datadog Monitors and Dashboards

Once the agent is collecting data, you can use Terraform to define specific monitors (alerts) and dashboards within Datadog. These monitors will be the triggers for PagerDuty incidents.

Integrating PagerDuty with Terraform

1. Configure PagerDuty Services and Escalation Policies

Before linking Datadog alerts, define your PagerDuty services and escalation policies using Terraform. A service represents a component or application that PagerDuty monitors, and an escalation policy dictates who gets notified and when.

2. Link Datadog Monitors to PagerDuty Services

The connection between Datadog and PagerDuty is typically made by configuring the notification channel within a Datadog monitor to point to a specific PagerDuty service integration key. Terraform allows you to specify this directly in the datadog_monitor resource.

Ready-to-Use Terraform Configuration

Below is a consolidated Terraform configuration that demonstrates the setup for EKS observability with Datadog and PagerDuty. Remember to replace placeholder values with your actual API keys, cluster names, and desired configurations.

variable "aws_region" {
  description = "AWS region for EKS cluster"
  type        = string
  default     = "us-east-1"
}

variable "eks_cluster_name" {
  description = "Name of the existing EKS cluster"
  type        = string
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_token" {
  description = "PagerDuty personal API token"
  type        = string
  sensitive   = true
}

# AWS Provider
provider "aws" {
  region = var.aws_region
}

# Kubernetes Provider
data "aws_eks_cluster" "main" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "main" {
  name = var.eks_cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.main.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.main.token
}

# Helm Provider for Datadog Agent
provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.main.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.main.token
  }
}

# Datadog Provider
provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

# PagerDuty Provider
provider "pagerduty" {
  token = var.pagerduty_token
}

# --- AWS IAM for Datadog Agent (IRSA) ---

resource "aws_iam_policy" "datadog_agent_policy" {
  name        = "${var.eks_cluster_name}-datadog-agent-policy"
  description = "IAM policy for Datadog Agent to collect AWS EKS metrics"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "ec2:Describe*",
          "logs:Describe*",
          "logs:Get*",
          "logs:List*",
          "logs:FilterLogEvents",
          "autoscaling:Describe*",
          "tag:GetResources",
          # Add more specific permissions as needed for other integrations (e.g., S3, RDS)
        ]
        Effect   = "Allow"
        Resource = "*"
      },
    ]
  })
}

resource "aws_iam_role" "datadog_agent_role" {
  name               = "${var.eks_cluster_name}-datadog-agent-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Federated = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${replace(data.aws_eks_cluster.main.identity[0].oidc[0].issuer, "https://", "")}"
        }
        Action = "sts:AssumeRoleWithWebIdentity"
        Condition = {
          StringEquals = {
            "${replace(data.aws_eks_cluster.main.identity[0].oidc[0].issuer, "https://", "")}:sub" : "system:serviceaccount:default:datadog-agent", # Adjust namespace/serviceaccount if changed
            "${replace(data.aws_eks_cluster.main.identity[0].oidc[0].issuer, "https://", "")}:aud" : "sts.amazonaws.com"
          }
        }
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "datadog_agent_attach" {
  role       = aws_iam_role.datadog_agent_role.name
  policy_arn = aws_iam_policy.datadog_agent_policy.arn
}

data "aws_caller_identity" "current" {}

# --- Datadog Agent Helm Chart Deployment ---

resource "kubernetes_service_account_v1" "datadog_agent" {
  metadata {
    name      = "datadog-agent"
    namespace = "default" # Or dedicated 'datadog' namespace
    annotations = {
      "eks.amazonaws.com/role-arn" = aws_iam_role.datadog_agent_role.arn
    }
  }
}

resource "helm_release" "datadog" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = kubernetes_service_account_v1.datadog_agent.metadata[0].namespace
  version    = "2.33.0" # Use a stable, recent version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "clusterAgent.metricsProvider.enabled"
    value = "true"
  }

  set {
    name  = "datadog.kubelet.host"
    value = "$${NODE_IP}"
  }
  set {
    name  = "datadog.kubelet.tlsVerify"
    value = "false" # Set to true with proper cert setup
  }

  set {
    name  = "datadog.apm.enabled"
    value = "true"
  }

  set {
    name  = "datadog.logs.enabled"
    value = "true"
  }
  set {
    name  = "datadog.logs.containerCollectAll"
    value = "true"
  }

  set {
    name  = "datadog.processAgent.enabled"
    value = "true"
  }

  set {
    name  = "agents.podAnnotations.eks\\.amazonaws\\.com/role-arn"
    value = aws_iam_role.datadog_agent_role.arn
    type  = "string"
  }

  set {
    name  = "clusterAgent.rbac.create"
    value = "true"
  }
  set {
    name  = "agents.rbac.create"
    value = "true"
  }

  set {
    name  = "agents.serviceAccount.create"
    value = "false" # We create it manually above for IRSA
  }
  set {
    name  = "agents.serviceAccount.name"
    value = kubernetes_service_account_v1.datadog_agent.metadata[0].name
  }
  set {
    name  = "clusterAgent.serviceAccount.create"
    value = "false" # We create it manually above for IRSA
  }
  set {
    name  = "clusterAgent.serviceAccount.name"
    value = kubernetes_service_account_v1.datadog_agent.metadata[0].name
  }

  # EKS-specific settings for host networking for full visibility
  set {
    name  = "datadog.hostPortEnabled"
    value = "true"
  }
  set {
    name  = "agents.containerRuntime.criSocketPath"
    value = "/var/run/containerd/containerd.sock" # For EKS optimized AMIs
  }
  # ... other Datadog agent configurations as needed (APM, Logs, etc.)
}

# --- PagerDuty Configuration ---

resource "pagerduty_user" "oncall_engineer" {
  name  = "On-Call Engineer"
  email = "oncall-engineer@example.com" # Replace with actual email
}

resource "pagerduty_escalation_policy" "eks_critical_policy" {
  name      = "EKS Critical Service Escalation"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "user"
      id   = pagerduty_user.oncall_engineer.id
    }
  }
  # Add more rules for additional users/teams if needed
}

resource "pagerduty_service" "eks_critical_service" {
  name                    = "EKS Critical Cluster Monitoring"
  auto_resolve_timeout_s  = 3600 # Auto-resolve after 1 hour if not acknowledged
  acknowledgement_timeout_s = 600  # Acknowledge within 10 minutes
  escalation_policy       = pagerduty_escalation_policy.eks_critical_policy.id
  alert_creation          = "create_alerts_and_incidents" # Or "create_incidents_and_alerts"

  incident_urgency_rule {
    type    = "constant"
    urgency = "high"
  }

  # Create a Datadog integration for this PagerDuty service
  integration {
    name = "Datadog Integration"
    type = "datadog_api_inbound_integration"
  }
}

# --- Datadog Monitor Linked to PagerDuty ---

resource "datadog_monitor" "eks_high_cpu_monitor" {
  name  = "${var.eks_cluster_name} EKS Cluster Node High CPU"
  type  = "metric alert"
  query = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.eks_cluster_name}} by {host} > 80"

  message = <<EOM
@webhook-${pagerduty_service.eks_critical_service.integration[0].id}
EKS Node {{host.name}} CPU utilization is over 80%.
Please investigate the cause immediately.
EOM

  tags = ["env:production", "service:kubernetes", "team:devops", "eks_cluster:${var.eks_cluster_name}"]
  notify_no_data = false
  renotify_interval = 60 # Renotify every 60 minutes if the alert persists

  # The PagerDuty integration is specified in the message using the webhook syntax
  # The integration ID is derived from the pagerduty_service resource.
}

output "datadog_agent_helm_status" {
  description = "Status of the Datadog Agent Helm release"
  value       = helm_release.datadog.status
}

output "pagerduty_service_url" {
  description = "URL to the PagerDuty service"
  value       = "https://app.pagerduty.com/services/${pagerduty_service.eks_critical_service.id}"
}

output "pagerduty_integration_key" {
  description = "PagerDuty integration key for Datadog"
  value       = pagerduty_service.eks_critical_service.integration[0].id
  sensitive   = true
}

Deployment Steps

Save the code above as main.tf in an empty directory.
Create a terraform.tfvars file with your sensitive variables:
eks_cluster_name = "your-eks-cluster-name" datadog_api_key = "YOUR_DATADOG_API_KEY" datadog_app_key = "YOUR_DATADOG_APP_KEY" pagerduty_token = "YOUR_PAGERDUTY_API_TOKEN"
Initialize Terraform: terraform init
Review the planned changes: terraform plan
Apply the configuration: terraform apply (type 'yes' when prompted)

Validation and Testing

After applying the Terraform configuration:

Verify Datadog Agent: Check Kubernetes pods: kubectl get pods -n default | grep datadog. You should see Datadog Agent pods running.
Check Datadog UI: Log into your Datadog account.
- Navigate to Infrastructure -> Host Map to see your EKS nodes.
- Go to Monitors -> Manage Monitors to find the "EKS Cluster Node High CPU" monitor.
Check PagerDuty UI: Log into your PagerDuty account.
- Verify the "EKS Critical Service Monitoring" service and the "EKS Critical Service Escalation" policy exist.
- Trigger a test alert (e.g., by artificially increasing CPU usage on an EKS node or manually triggering the Datadog monitor) to ensure PagerDuty receives the incident.

Advanced Observability Patterns

This foundational setup can be extended with more advanced capabilities:

Custom Metrics: Instrument your applications to emit custom metrics that Datadog can collect, providing deeper insights into business-specific KPIs.
Distributed Tracing (APM): Configure Datadog APM to trace requests across microservices running on EKS, identifying latency bottlenecks.
Synthetic Monitoring: Use Datadog Synthetics to proactively test your application's availability and performance from various global locations.
Security Monitoring: Integrate Datadog Security Monitoring to detect threats and vulnerabilities within your EKS cluster and workloads.
Advanced Alerting: Utilize Datadog's machine learning-driven anomaly detection for more intelligent alerting that adapts to baseline behavior.

Troubleshooting and FAQ

Datadog Agent Pods Not Running

Check pod logs (kubectl logs <pod-name> -n default) for errors. Ensure the correct Datadog API and APP keys are provided. Verify the IAM role for the service account has been correctly applied and the OIDC provider is configured for your EKS cluster.

Metrics/Logs Not Appearing in Datadog

Confirm the Datadog Agent has the necessary AWS IAM permissions. Check agent logs for errors related to collection. Ensure the Helm chart values for APM, Logs, and process collection are enabled if you expect to see that data.

PagerDuty Incidents Not Triggering

Verify the Datadog monitor's message content correctly references the PagerDuty integration key (e.g., @webhook-<INTEGRATION_KEY>). Ensure the PagerDuty service is correctly configured with an associated escalation policy and on-call users. Test the Datadog monitor directly from the Datadog UI to see if it triggers an alert there.

Conclusion

Building a robust observability pipeline for AWS EKS in production is not merely a best practice; it's a critical requirement for maintaining service reliability and ensuring rapid incident response. By leveraging Terraform, you can declaratively provision and manage Datadog for comprehensive monitoring and PagerDuty for intelligent incident management. This approach ensures consistency, reduces operational overhead, and empowers your DevOps teams with the insights and tools needed to keep your cloud-native applications performing optimally.

Embrace Infrastructure as Code for your observability stack, and transform reactive troubleshooting into proactive, data-driven operations.

CloudStack YAML