Automating AWS EKS Observability with Terraform, Datadog, and PagerDuty

Architecture Pro-Tip: Always design your observability stack for scale and resiliency from day one. Decouple data collection from processing and alerting. Utilize Infrastructure as Code (IaC) for every component, ensuring repeatable deployments and easy version control. For EKS, treat your observability agents like any other critical application within your cluster, managing their lifecycle with Kubernetes-native tools and IaC.

In the dynamic world of cloud-native applications, maintaining robust observability for Kubernetes clusters is paramount. AWS Elastic Kubernetes Service (EKS) offers a powerful platform, but understanding its intricate workings and the applications deployed on it requires a sophisticated monitoring and alerting strategy. This guide details how to seamlessly automate EKS observability using a trifecta of industry-leading tools: Terraform for infrastructure as code, Datadog for comprehensive monitoring, and PagerDuty for efficient incident response.

The Pillars of EKS Observability Automation

Achieving true observability means gathering metrics, logs, and traces, then correlating them to understand system behavior. Automating this entire pipeline with IaC ensures consistency, reduces manual errors, and accelerates deployment cycles. Here’s how each tool contributes:

AWS EKS: The Foundation

Amazon EKS provides a managed Kubernetes service, simplifying the deployment and management of Kubernetes clusters on AWS. While EKS handles the control plane, monitoring the data plane (worker nodes) and the applications running within the cluster remains your responsibility.

Terraform: Infrastructure as Code (IaC)

Terraform, by HashiCorp, allows you to define and provision an entire infrastructure using a declarative configuration language. For our purpose, Terraform will not only provision AWS resources but also configure Datadog and PagerDuty, unifying our entire observability setup under a single IaC umbrella.

Datadog: Comprehensive Monitoring and Analytics

Datadog is a leading monitoring and analytics platform that collects, aggregates, and visualizes data from across your entire stack. For EKS, Datadog offers deep integrations for:

Metrics: Host metrics, Kubernetes control plane metrics, pod metrics, custom application metrics.
Logs: Centralized log collection, parsing, and analysis from all EKS components and applications.
APM (Tracing): Distributed tracing for microservices running on EKS.
Network Performance: Visibility into network traffic between services.
Security Monitoring: Detection of security threats and misconfigurations.

PagerDuty: Incident Management and On-Call Automation

PagerDuty streamlines incident response by alerting the right people at the right time. Integrating Datadog with PagerDuty ensures that critical alerts from your EKS cluster and applications are immediately routed to your on-call teams, facilitating faster diagnosis and resolution.

Step-by-Step Implementation with Terraform

We'll walk through setting up the necessary components using Terraform, starting from the Datadog agent deployment on EKS, to configuring monitors, and finally integrating with PagerDuty.

1. Preparing Your Environment

Ensure you have the AWS CLI, kubectl, and Terraform installed and configured. You'll also need Datadog API and application keys, and a PagerDuty API token. Store these securely, ideally using a secrets manager like AWS Secrets Manager or HashiCorp Vault, and reference them in your Terraform configuration.

2. Terraform Provider Configuration

Set up your Terraform providers for AWS, Datadog, and PagerDuty.

provider "aws" {
  region = "us-east-1" # Or your desired region
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.example.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.example.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.example.token
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

data "aws_eks_cluster" "example" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "example" {
  name = var.eks_cluster_name
}

3. Deploying the Datadog Agent on EKS with Terraform

The Datadog Agent is deployed as a DaemonSet across your EKS worker nodes to collect metrics, logs, and traces. We'll use Terraform to manage the Kubernetes manifests.

a. Create an IAM Policy and Role for the Datadog Agent (Optional, but Recommended)

Grant the Datadog Agent appropriate permissions, especially if using Fargate or advanced integrations requiring AWS API access.

b. Kubernetes Secret for Datadog API Key

Store your Datadog API key as a Kubernetes Secret for the agent to use.

c. Deploy Datadog Agent DaemonSet

The core of the Datadog integration is the agent. You can either use the official Helm chart via Terraform's Helm provider or inline Kubernetes manifests.

4. Terraform Configuration for Datadog Agent Deployment and Observability Setup

This comprehensive example combines the IAM role, Kubernetes secret, and the Datadog agent DaemonSet, along with a basic Datadog monitor and PagerDuty service.

# main.tf

# Variables for sensitive data and cluster name
variable "datadog_api_key" {
  description = "Your Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Your Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "Your PagerDuty API Token"
  type        = string
  sensitive   = true
}

variable "eks_cluster_name" {
  description = "Name of your EKS cluster"
  type        = string
}

variable "aws_region" {
  description = "AWS region for resources"
  type        = string
  default     = "us-east-1"
}

# AWS Provider
provider "aws" {
  region = var.aws_region
}

# Kubernetes Provider (requires kubeconfig context or explicit config)
provider "kubernetes" {
  host                   = data.aws_eks_cluster.example.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.example.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.example.token
}

# Datadog Provider
provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

# PagerDuty Provider
provider "pagerduty" {
  token = var.pagerduty_api_token
}

# Data source to fetch EKS cluster details
data "aws_eks_cluster" "example" {
  name = var.eks_cluster_name
}

# Data source to fetch EKS cluster authentication token
data "aws_eks_cluster_auth" "example" {
  name = var.eks_cluster_name
}

# --- AWS IAM Role for Datadog Agent (if needing AWS permissions) ---
resource "aws_iam_policy" "datadog_agent_policy" {
  name        = "${var.eks_cluster_name}-datadog-agent-policy"
  description = "IAM policy for Datadog Agent to access AWS services"

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action = [
          "ec2:DescribeTags",
          "ec2:DescribeInstances",
          "ec2:DescribeVolumes",
          "autoscaling:DescribeAutoScalingGroups",
          "lambda:ListFunctions",
          "ecs:ListClusters",
          "ecs:DescribeClusters",
          "ecs:ListContainerInstances",
          "ecs:DescribeContainerInstances",
          "ecs:ListTasks",
          "ecs:DescribeTasks"
        ],
        Effect   = "Allow",
        Resource = "*"
      },
    ],
  })
}

resource "aws_iam_role" "datadog_agent_role" {
  name_prefix        = "${var.eks_cluster_name}-datadog-agent-role-"
  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action = "sts:AssumeRole",
        Effect = "Allow",
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      },
      # For IRSA (IAM Roles for Service Accounts) on EKS, use below instead of Service Principal
      # {
      #   Action = "sts:AssumeRoleWithWebIdentity"
      #   Effect = "Allow"
      #   Principal = {
      #     Federated = "arn:aws:iam::ACCOUNT_ID:oidc-provider/OIDC_PROVIDER_URL" # Replace with your OIDC provider
      #   }
      #   Condition = {
      #     StringEquals = {
      #       "OIDC_PROVIDER_URL:sub" = "system:serviceaccount:datadog:datadog-agent" # Replace with your namespace/serviceaccount
      #     }
      #   }
      # }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "datadog_agent_attach" {
  role       = aws_iam_role.datadog_agent_role.name
  policy_arn = aws_iam_policy.datadog_agent_policy.arn
}

resource "aws_iam_instance_profile" "datadog_agent_profile" {
  name = "${var.eks_cluster_name}-datadog-agent-profile"
  role = aws_iam_role.datadog_agent_role.name
}


# --- Kubernetes Secret for Datadog API Key ---
resource "kubernetes_secret" "datadog_api_key_secret" {
  metadata {
    name      = "datadog-api-key"
    namespace = "datadog" # Ensure this namespace exists or create it
  }

  data = {
    "api-key" = var.datadog_api_key
  }

  type = "Opaque"
}

# --- Kubernetes Namespace for Datadog (if not existing) ---
resource "kubernetes_namespace" "datadog" {
  metadata {
    name = "datadog"
  }
}

# --- Datadog Agent DaemonSet Kubernetes Manifest ---
# This is a simplified example. For full production setup, consider Helm chart.
resource "kubernetes_daemonset" "datadog_agent" {
  metadata {
    name      = "datadog-agent"
    namespace = kubernetes_namespace.datadog.metadata[0].name
    labels = {
      app = "datadog-agent"
    }
  }

  spec {
    selector {
      match_labels = {
        app = "datadog-agent"
      }
    }

    template {
      metadata {
        labels = {
          app = "datadog-agent"
        }
      }

      spec {
        service_account_name = "datadog-agent" # Ensure this service account exists or create it
        # For IRSA:
        # annotations = {
        #   "eks.amazonaws.com/role-arn" = aws_iam_role.datadog_agent_role.arn
        # }

        container {
          name  = "agent"
          image = "gcr.io/datadoghq/agent:latest" # Use specific version in production
          env {
            name  = "DD_KUBERNETES_KUBELET_HOST"
            value = "$${DD_HOST_IP}"
          }
          env {
            name = "DD_API_KEY"
            value_from {
              secret_key_ref {
                name = kubernetes_secret.datadog_api_key_secret.metadata[0].name
                key  = "api-key"
              }
            }
          }
          env {
            name  = "DD_EKS_FARGATE" # Set to true if using Fargate
            value = "false"
          }
          env {
            name  = "DD_LOGS_ENABLED"
            value = "true"
          }
          env {
            name  = "DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL"
            value = "true"
          }
          env {
            name  = "DD_APM_ENABLED"
            value = "true"
          }
          env {
            name  = "DD_PROCESS_AGENT_ENABLED"
            value = "true"
          }
          env {
            name = "DD_SITE"
            value = "datadoghq.com" # or eu.datadoghq.com etc.
          }
          env {
            name  = "DD_TAGS"
            value = "env:${var.eks_cluster_name},cluster_name:${var.eks_cluster_name}"
          }
          resources {
            limits = {
              cpu    = "200m"
              memory = "256Mi"
            }
            requests = {
              cpu    = "100m"
              memory = "128Mi"
            }
          }
          security_context {
            capabilities {
              add = ["NET_ADMIN", "SYS_ADMIN"]
            }
          }
          volume_mount {
            name       = "procfs"
            mount_path = "/host/proc"
            read_only  = true
          }
          volume_mount {
            name       = "cgroup"
            mount_path = "/host/sys/fs/cgroup"
            read_only  = true
          }
          volume_mount {
            name       = "docker-socket"
            mount_path = "/var/run/docker.sock"
            read_only  = true
          }
          volume_mount {
            name       = "log-path"
            mount_path = "/var/log"
            read_only  = true
          }
          volume_mount {
            name       = "run-path"
            mount_path = "/var/run"
            read_only  = true
          }
          volume_mount {
            name       = "confd"
            mount_path = "/conf.d"
            read_only  = true
          }
          volume_mount {
            name       = "passwd"
            mount_path = "/etc/passwd"
            read_only  = true
          }
        }

        host_network = true # Required for host-level monitoring
        dns_policy   = "ClusterFirstWithHostNet"

        volume {
          name = "procfs"
          host_path {
            path = "/proc"
          }
        }
        volume {
          name = "cgroup"
          host_path {
            path = "/sys/fs/cgroup"
          }
        }
        volume {
          name = "docker-socket"
          host_path {
            path = "/var/run/docker.sock"
          }
        }
        volume {
          name = "log-path"
          host_path {
            path = "/var/log"
          }
        }
        volume {
          name = "run-path"
          host_path {
            path = "/var/run"
          }
        }
        volume {
          name = "confd"
          host_path {
            path = "/etc/datadog-agent/conf.d"
          }
        }
        volume {
          name = "passwd"
          host_path {
            path = "/etc/passwd"
          }
        }
      }
    }
  }
}

# --- Datadog Monitor Configuration with PagerDuty Integration ---

# PagerDuty Service
resource "pagerduty_service" "eks_critical_alerts_service" {
  name        = "${var.eks_cluster_name}-Critical-EKS-Alerts"
  description = "Service for critical AWS EKS alerts from Datadog"
  escalation_policy = pagerduty_escalation_policy.default_policy.id # Assuming you have a default policy
}

# Example PagerDuty Escalation Policy (simplified)
resource "pagerduty_escalation_policy" "default_policy" {
  name      = "${var.eks_cluster_name}-EKS-Escalation-Policy"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 30
    target {
      type = "user"
      id   = "PAGERDUTY_USER_ID" # Replace with actual PagerDuty User ID
    }
  }
  rule {
    escalation_delay_in_minutes = 30
    target {
      type = "team"
      id   = "PAGERDUTY_TEAM_ID" # Replace with actual PagerDuty Team ID
    }
  }
}

# Datadog Integration with PagerDuty (via PagerDuty service integration)
# A Datadog integration resource can reference a PagerDuty service.
# For PagerDuty integration specifically, you usually set it up in Datadog UI
# and then link it via a notification in the monitor.
# The 'id' here points to a PagerDuty service integration ID created in Datadog,
# which Terraform doesn't directly manage at this level without a specific resource.
# Instead, monitors refer to the PagerDuty service by name/alias configured in Datadog UI.

# Datadog Monitor for high CPU utilization on EKS nodes
resource "datadog_monitor" "eks_node_cpu_utilization" {
  name                = "[EKS - ${var.eks_cluster_name}] High Node CPU Utilization"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.eks_cluster_name}} by {host} > 80"
  message             = "EKS node {{host.name}} CPU utilization is over 80%! @pagerduty-${pagerduty_service.eks_critical_alerts_service.name}"
  tags                = ["environment:${var.eks_cluster_name}", "team:devops", "severity:critical"]
  renotify_interval   = 60
  notify_no_data      = false
  new_group_delay     = 60
  timeout_h           = 0
  no_data_timeframe   = 30 # Consider node down if no data for 30 minutes
  evaluation_delay    = 90
  include_tags        = true
  require_full_window = false
  notify_audit        = false
  force_delete        = false # Set to true to allow Terraform to delete monitors. Be cautious.

  # This is how you reference the PagerDuty service in Datadog.
  # The 'name' of the pagerduty_service resource needs to match an integration
  # or alert group configured in your Datadog PagerDuty integration settings.
  # For direct integration, use @pagerduty-
  # For specific PagerDuty service via a Datadog integration that points to it,
  # the @pagerduty-{{service.name}} might require custom mapping in Datadog.
  # Simpler: configure Datadog's PagerDuty integration to map certain tags/monitors to services.
  # Or, just use a generic @pagerduty handle if the integration is configured to use one service.
}

# To apply this:
# terraform init
# terraform plan -var="datadog_api_key=YOUR_DD_API_KEY" -var="datadog_app_key=YOUR_DD_APP_KEY" -var="pagerduty_api_token=YOUR_PD_API_TOKEN" -var="eks_cluster_name=YOUR_EKS_CLUSTER_NAME"
# terraform apply -var="datadog_api_key=YOUR_DD_API_KEY" -var="datadog_app_key=YOUR_DD_APP_KEY" -var="pagerduty_api_token=YOUR_PD_API_TOKEN" -var="eks_cluster_name=YOUR_EKS_CLUSTER_NAME"

Post-Deployment Verification and Enhancement

Datadog Dashboards and Alerts

Once the Datadog Agent is deployed, navigate to your Datadog account. You should start seeing metrics, logs, and traces populate. Explore the out-of-the-box EKS dashboards. You can further refine monitors using Datadog's query language to cover specific application behaviors, custom metrics, or log patterns. Consider creating specific monitors for:

Pod Restarts and Failures
Deployment Rollout Failures
Service Latency and Error Rates
Resource Quota Violations
Kubernetes API Server Health

PagerDuty Escalation Policies and Services

Verify that the PagerDuty service and escalation policy are created in your PagerDuty account. Test the Datadog monitor by intentionally triggering an alert (e.g., by creating a high load) to ensure an incident is created in PagerDuty and the correct on-call team is notified.

Troubleshooting and Best Practices

Common Issues

Datadog Agent not reporting: Check Kubernetes logs for the Datadog Agent pods (kubectl logs -n datadog -l app=datadog-agent). Verify the API key secret is correctly mounted. Ensure necessary IAM roles/policies are attached if using IRSA.
Missing metrics/logs: Confirm agent permissions to access Kubelet and Docker sockets. Check Datadog Agent configuration for enabled integrations (e.g., APM, logs).
PagerDuty alerts not triggering: Double-check the @pagerduty-<integration_name> syntax in your Datadog monitor message. Ensure the Datadog-PagerDuty integration is correctly configured in Datadog UI and linked to your PagerDuty service.

Best Practices

Version Control Everything: Treat your observability configuration as critical code. Store all Terraform files in a Git repository.
Use Helm for Datadog Agent: For production-grade deployments, consider using the official Datadog Helm chart through Terraform's Helm provider for a more robust and feature-rich agent deployment. This simplifies managing configurations for various integrations.
Least Privilege IAM: Grant the Datadog Agent only the permissions it absolutely needs. Leverage AWS IAM Roles for Service Accounts (IRSA) for fine-grained permissions instead of instance profiles where possible.
Tagging Strategy: Implement a consistent tagging strategy across AWS resources, EKS objects, and Datadog entities. This allows for powerful filtering, correlation, and cost allocation.
Regular Review: Periodically review your Datadog monitors and PagerDuty escalation policies to ensure they remain relevant and effective as your applications and infrastructure evolve.

Conclusion

Automating AWS EKS observability with Terraform, Datadog, and PagerDuty creates a robust, scalable, and maintainable monitoring and incident response framework. By defining your entire observability stack as code, you gain consistency, reduce operational overhead, and empower your teams to quickly identify and resolve issues, ensuring the reliability and performance of your cloud-native applications on EKS.

CloudStack YAML