Production-Ready AWS EKS Observability with Terraform, Datadog, and PagerDuty

In the dynamic world of cloud-native applications, maintaining robust visibility into your infrastructure and applications is paramount. For teams leveraging AWS Elastic Kubernetes Service (EKS), achieving true production-readiness demands a comprehensive observability strategy. This guide will walk you through building a resilient and automated observability stack for EKS using the power of Terraform for Infrastructure as Code (IaC), Datadog for unified monitoring, and PagerDuty for effective incident response.

Architecture Pro-Tip:

Design your observability strategy proactively, not reactively. Integrate monitoring and alerting definitions directly into your IaC from day one. This ensures consistency, reduces manual errors, and provides a traceable history of your observability configurations, making it an integral part of your application's lifecycle, not an afterthought.

Why a Unified Observability Stack for EKS?

Kubernetes, while incredibly powerful, introduces significant complexity. Distributed systems, ephemeral containers, and dynamic scaling create a need for deep insights into metrics, logs, and traces. A unified observability stack addresses these challenges by:

Reducing Mean Time To Resolution (MTTR): Quickly pinpoint issues by correlating data across your entire EKS environment.
Proactive Issue Detection: Identify anomalies and potential problems before they impact users.
Optimizing Performance & Cost: Gain insights into resource utilization to fine-tune your EKS clusters and applications.
Improving Developer Experience: Empower developers with self-service dashboards and application-level insights.

The Core Components Explained

1. AWS EKS: The Foundation

AWS EKS provides a highly available and scalable Kubernetes control plane. While EKS manages the underlying infrastructure for the control plane, the responsibility for monitoring worker nodes, pods, and application performance falls to you.

2. Terraform: Infrastructure as Code (IaC)

Terraform allows you to define and provision your entire cloud infrastructure, including EKS clusters, IAM roles, and even observability configurations, using declarative configuration files. This ensures consistency, repeatability, and version control for your infrastructure.

3. Datadog: Unified Monitoring & Analytics

Datadog is a comprehensive monitoring, logging, and APM platform. For EKS, it provides:

Infrastructure Monitoring: Deep visibility into EKS nodes, pods, containers, and services.
APM & Distributed Tracing: End-to-end visibility into application performance.
Log Management: Centralized collection, processing, and analysis of logs from all EKS components and applications.
Network Performance Monitoring (NPM): Insight into network traffic between pods and services.
Security Monitoring: Detection of security threats and misconfigurations.

4. PagerDuty: Incident Management & On-Call Automation

PagerDuty acts as your central nervous system for incident response. It integrates with monitoring tools like Datadog to ingest alerts, route them to the correct on-call teams based on escalation policies, and ensure incidents are acknowledged and resolved efficiently.

Implementing the Stack with Terraform

Prerequisites:

An existing AWS EKS cluster.
Terraform installed and configured with AWS provider.
Datadog API and APP keys.
PagerDuty API token.
Helm installed (for Datadog Agent).

Step 1: Configure Terraform Providers

Ensure your Terraform configuration includes the AWS, Datadog, and PagerDuty providers.

provider "aws" {
  region = "us-east-1"
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.example.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.example.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.example.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.example.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.example.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.example.token
  }
}

Step 2: Deploy Datadog Agent to EKS with Helm (via Terraform)

The Datadog Agent is critical for collecting metrics, logs, and traces. We'll use the Terraform Helm provider to deploy the Datadog Agent chart.

First, ensure your EKS cluster can grant necessary IAM permissions to the Datadog Agent for collecting AWS service metrics (CloudWatch, S3, RDS, etc.). This typically involves creating an IAM role and associating it with a Kubernetes Service Account.

Example: Deploying Datadog Agent, Configuring Monitors, and PagerDuty Service

This comprehensive example demonstrates how to set up the Datadog Agent, a basic CPU utilization monitor in Datadog, and a corresponding PagerDuty service, all managed by Terraform. Remember to replace placeholder values with your actual cluster details and API keys.

# main.tf

# Data sources for your existing EKS cluster
data "aws_eks_cluster" "example" {
  name = var.cluster_name
}

data "aws_eks_cluster_auth" "example" {
  name = var.cluster_name
}

# IAM Role and Service Account for Datadog Agent (recommended for AWS integration)
resource "aws_iam_policy" "datadog_agent_policy" {
  name        = "DatadogAgentPolicy-${var.cluster_name}"
  description = "Allows Datadog Agent to access AWS services for metrics"
  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action = [
          "ec2:Describe*",
          "logs:Describe*",
          "logs:Get*",
          "logs:FilterLogEvents",
          "tag:GetResources",
          "sts:AssumeRole",
          # Add more AWS service permissions as needed (e.g., CloudWatch, S3, RDS)
          "cloudwatch:GetMetricData",
          "cloudwatch:ListMetrics",
          "cloudwatch:DescribeAlarms"
        ],
        Effect   = "Allow",
        Resource = "*"
      },
    ]
  })
}

resource "aws_iam_role" "datadog_agent_role" {
  name               = "DatadogAgentRole-${var.cluster_name}"
  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect = "Allow",
        Principal = {
          Federated = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${replace(data.aws_eks_cluster.example.identity.0.oidc.0.issuer, "https://", "")}"
        },
        Action = "sts:AssumeRoleWithWebIdentity",
        Condition = {
          StringEquals = {
            "${replace(data.aws_eks_cluster.example.identity.0.oidc.0.issuer, "https://", "")}:sub" : "system:serviceaccount:datadog:datadog-agent"
          }
        }
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "datadog_agent_attach" {
  role       = aws_iam_role.datadog_agent_role.name
  policy_arn = aws_iam_policy.datadog_agent_policy.arn
}

resource "kubernetes_service_account_v1" "datadog_agent" {
  metadata {
    name      = "datadog-agent"
    namespace = "datadog" # Ensure this namespace exists or create it
    annotations = {
      "eks.amazonaws.com/role-arn" = aws_iam_role.datadog_agent_role.arn
    }
  }
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true # Creates the namespace if it doesn't exist

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }
  set {
    name  = "datadog.clusterName"
    value = var.cluster_name
  }
  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
  }
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "clusterAgent.metricsProvider.enabled"
    value = "true"
  }
  set {
    name  = "datadog.site"
    value = "datadoghq.com" # or eu.datadoghq.com etc.
  }
  # Enable log collection
  set {
    name  = "datadog.logs.enabled"
    value = "true"
  }
  set {
    name  = "datadog.logs.containerCollectAll"
    value = "true"
  }
  # Use the IAM Role for service account
  set {
    name  = "serviceAccount.create"
    value = "false" # We create it above
  }
  set {
    name  = "serviceAccount.name"
    value = kubernetes_service_account_v1.datadog_agent.metadata[0].name
  }
  set {
    name  = "clusterAgent.rbac.create"
    value = "true"
  }
  set {
    name  = "agents.rbac.create"
    value = "true"
  }
  # APM and Tracing (enable if needed)
  set {
    name  = "datadog.apm.enabled"
    value = "true"
  }
  set {
    name  = "datadog.apm.hostPort"
    value = "8126"
  }
  # ... other Datadog agent configurations ...
}

# Datadog Monitor for EKS Node CPU Utilization
resource "datadog_monitor" "eks_node_cpu_critical" {
  name               = "[EKS] ${var.cluster_name} Node CPU Utilization Critical"
  type               = "metric alert"
  query              = "avg(last_5m):avg:kubernetes.cpu.usage.total{kubernetes_cluster_name:${var.cluster_name}} by {host} > 80"
  message            = "CPU utilization on host {{host.name}} in EKS cluster ${var.cluster_name} is above 80% for 5 minutes. @pagerduty-EKS_CPU_Critical"
  tags               = ["environment:${var.environment}", "service:kubernetes", "team:devops"]
  priority           = 1
  notify_no_data     = false
  new_group_delay    = 600 # 10 minutes
  notify_audit       = false
  renotify_interval  = 0
  no_data_timeframe  = 20
  escalation_message = "Still high CPU! Please investigate immediately. @slack-devops"

  thresholds {
    critical = 80
    warning  = 70
  }
}

# Datadog Monitor for EKS Node CPU Utilization (Warning)
resource "datadog_monitor" "eks_node_cpu_warning" {
  name               = "[EKS] ${var.cluster_name} Node CPU Utilization Warning"
  type               = "metric alert"
  query              = "avg(last_5m):avg:kubernetes.cpu.usage.total{kubernetes_cluster_name:${var.cluster_name}} by {host} > 70"
  message            = "CPU utilization on host {{host.name}} in EKS cluster ${var.cluster_name} is above 70% for 5 minutes. This is a warning. @slack-devops"
  tags               = ["environment:${var.environment}", "service:kubernetes", "team:devops"]
  priority           = 2
  notify_no_data     = false
  new_group_delay    = 300
  notify_audit       = false
  renotify_interval  = 0
  no_data_timeframe  = 20
  escalation_message = "CPU utilization remains high. Escalating to critical soon."

  thresholds {
    critical = 80
    warning  = 70
  }
}

# PagerDuty User (example, typically pre-created or managed by another system)
resource "pagerduty_user" "devops_engineer" {
  name  = "DevOps Engineer"
  email = "devops-engineer@example.com"
  role  = "user" # or "admin", "owner"
}

# PagerDuty Escalation Policy
resource "pagerduty_escalation_policy" "devops_escalation" {
  name      = "DevOps Primary Escalation"
  num_loops = 2

  rule {
    id = "first_level"
    target {
      type = "user"
      id   = pagerduty_user.devops_engineer.id
    }
    delay_in_minutes = 5
  }
  rule {
    id = "second_level"
    target {
      type = "user"
      id   = pagerduty_user.devops_engineer.id # Could be a different user or team
    }
    delay_in_minutes = 10
  }
}

# PagerDuty Service integrated with Datadog
resource "pagerduty_service" "eks_monitoring_service" {
  name                    = "EKS Monitoring Service (${var.cluster_name})"
  auto_resolve_timeout    = "14400" # 4 hours
  acknowledgement_timeout = "600"   # 10 minutes
  escalation_policy       = pagerduty_escalation_policy.devops_escalation.id
  alert_creation          = "create_alerts_and_incidents"

  # Integration with Datadog
  integration {
    name = "Datadog Integration"
    type = "datadog_api_inbound_integration" # Use datadog_events_api_inbound_integration for new
  }
}

# Datadog Integration with PagerDuty (via terraform datadog provider)
resource "datadog_integration_pagerduty" "pagerduty_integration" {
  # This resource links Datadog to PagerDuty globally,
  # the service-specific routing is done via @pagerduty-SERVICE_NAME in monitors
  services = [
    {
      service_name = pagerduty_service.eks_monitoring_service.name
      service_key  = pagerduty_service.eks_monitoring_service.integration[0].integration_key
    }
    # Add more services as needed
  ]
}

# variables.tf
variable "cluster_name" {
  description = "Name of the EKS cluster"
  type        = string
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog APP Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

variable "environment" {
  description = "Environment tag for resources"
  type        = string
  default     = "production"
}

Step 3: Apply Terraform Configuration

Initialize your Terraform workspace, plan, and apply the changes:

terraform init
terraform plan -var="cluster_name=my-eks-cluster" -var="datadog_api_key=..." -var="datadog_app_key=..." -var="pagerduty_api_token=..."
terraform apply -var="cluster_name=my-eks-cluster" -var="datadog_api_key=..." -var="datadog_app_key=..." -var="pagerduty_api_token=..."

This will deploy the Datadog Agent, create the specified Datadog monitors, and set up the PagerDuty service and escalation policies.

Datadog Post-Deployment Configuration

Once the agent is deployed, explore Datadog's capabilities:

EKS Integration Dashboard: Datadog automatically populates an EKS integration dashboard with key metrics.
Custom Dashboards: Create dashboards tailored to your applications, combining metrics, logs, and traces.
Log Explorer: Use the Log Explorer to search, filter, and analyze EKS logs. Set up log patterns and metrics.
APM: Instrument your applications with Datadog APM libraries to get full trace context.
Monitors: Beyond basic CPU, set up monitors for memory, network, pod restarts, application errors (from logs/traces), and custom application metrics.

PagerDuty Incident Response Flow

With Datadog integrated, critical alerts will automatically trigger incidents in PagerDuty:

Alert Ingestion: Datadog monitors configured with @pagerduty-SERVICE_NAME will send alerts to the specified PagerDuty service.
Escalation: PagerDuty's escalation policies ensure the right people are notified via multiple channels (SMS, phone call, email, push notification).
On-Call Management: Teams manage on-call schedules, ensuring 24/7 coverage.
Incident Resolution: PagerDuty facilitates communication, runbook execution, and post-incident analysis.

Best Practices for Production-Ready Observability

Observability as Code: Continue to define all Datadog monitors, dashboards, and PagerDuty services in Terraform.
Granular IAM: Ensure the Datadog Agent's IAM role has only the necessary permissions (least privilege).
Tagging Strategy: Implement a consistent tagging strategy across AWS, EKS, and Datadog to enable powerful filtering and analysis.
Alert Fatigue Mitigation: Tune your monitors to minimize false positives. Use composite monitors, anomaly detection, and correlation.
Regular Review: Periodically review your dashboards, monitors, and escalation policies to ensure they remain relevant.
Cost Management: Monitor Datadog ingestion volumes (logs, metrics) to manage costs effectively.

Troubleshooting Common Issues

Datadog Agent Not Reporting Data:

Check Pod Status: kubectl get pods -n datadog. Ensure agents are running.
View Agent Logs: kubectl logs <datadog-agent-pod> -n datadog. Look for API key errors, connectivity issues, or permission errors.
Verify API/APP Keys: Double-check the datadog.apiKey and datadog.appKey values in your Helm release.
IAM Permissions: Ensure the IAM role attached to the Datadog Service Account has adequate permissions.

Datadog Alerts Not Firing or PagerDuty Not Receiving Incidents:

Monitor Query: Verify the Datadog monitor's query is correct and actually triggering in Datadog's UI.
Notification Syntax: Ensure the @pagerduty-SERVICE_NAME syntax in the monitor message is correct and matches the PagerDuty service name configured in Datadog.
Datadog-PagerDuty Integration: Check the Datadog "Integrations > PagerDuty" page to ensure the integration is active and correctly configured with your PagerDuty services.
PagerDuty API Key: Confirm your PagerDuty API token used by Terraform is valid.

Conclusion

Establishing production-ready observability for AWS EKS is a critical step towards maintaining application reliability and operational efficiency. By leveraging Terraform for automated deployment, Datadog for comprehensive monitoring, and PagerDuty for streamlined incident response, you empower your DevOps teams with the tools needed to confidently manage complex cloud-native environments. This integrated approach not only reduces MTTR but also fosters a culture of proactive problem-solving, making your EKS deployments truly resilient and future-proof.

CloudStack YAML