Automating AWS EKS Observability with Terraform, Datadog, and PagerDuty Integration

Architecture Pro-Tip: Embrace Shift-Left Observability

Integrate observability configuration directly into your Infrastructure as Code (IaC) from day one. By defining Datadog monitors, dashboards, and PagerDuty integrations alongside your AWS EKS cluster in Terraform, you ensure consistency, auditability, and immediate visibility upon deployment. This "shift-left" approach prevents observability gaps and accelerates incident response.

Automating AWS EKS Observability with Terraform, Datadog, and PagerDuty Integration

In today's dynamic cloud-native landscape, managing and monitoring Kubernetes clusters, especially AWS Elastic Kubernetes Service (EKS), can be a complex endeavor. Ensuring high availability, performance, and security requires a robust observability strategy. This technical guide outlines a comprehensive approach to automating AWS EKS observability using a powerful triumvirate: Terraform for Infrastructure as Code (IaC), Datadog for unified monitoring, and PagerDuty for intelligent incident management. By the end, you'll have a clear understanding of how to build an automated, resilient, and highly visible EKS environment.

The Challenge of EKS Observability

AWS EKS abstracts away much of the Kubernetes control plane's operational burden, but the responsibility for monitoring applications, worker nodes, and the Kubernetes components themselves still lies with the user. Traditional monitoring tools often struggle with the ephemeral nature of containers and the distributed architecture of microservices. Furthermore, converting alerts into actionable incidents that reach the right people promptly is crucial for maintaining service level objectives (SLOs).

Our Solution Stack: Terraform, Datadog, PagerDuty

This guide leverages a best-of-breed toolchain to establish a fully automated and integrated observability pipeline for AWS EKS:

Terraform: For defining, provisioning, and managing your AWS EKS cluster, associated IAM roles, and crucially, deploying the Datadog Agent and configuring Datadog monitors and PagerDuty integrations as code.
Datadog: A comprehensive monitoring and analytics platform that provides end-to-end visibility across your EKS environment. It collects metrics, logs, and traces from your cluster, containers, applications, and AWS infrastructure, presenting them in unified dashboards.
PagerDuty: An industry-leading incident management platform that transforms Datadog alerts into actionable incidents. It enables on-call scheduling, escalation policies, and seamless communication to ensure critical issues are addressed rapidly.

Architecture Overview

The proposed architecture establishes a closed-loop system for observability and incident response:

Infrastructure Provisioning: Terraform provisions the AWS EKS cluster, worker node groups, and necessary IAM roles.
Agent Deployment: Terraform, utilizing the Helm provider, deploys the Datadog Agent onto the EKS cluster.
Data Collection: The Datadog Agent collects metrics, logs, and traces from EKS control plane, nodes, pods, and applications.
Monitoring & Alerting: Datadog processes this data, visualizes it in dashboards, and evaluates it against Terraform-defined monitors.
Incident Creation: When a monitor's threshold is breached, Datadog triggers an alert and sends it to PagerDuty via a pre-configured integration.
Incident Response: PagerDuty then routes the incident to the appropriate on-call team based on escalation policies, facilitating rapid response and resolution.

Step-by-Step Implementation Guide

Prerequisites

Before you begin, ensure you have the following:

An AWS Account with administrative access.
Terraform CLI (v1.0+) installed.
AWS CLI configured with appropriate credentials.
A Datadog Account with an API Key and Application Key.
A PagerDuty Account with an Integration Key (or Admin API Key for full automation).
`kubectl` and `helm` CLIs installed.

Step 1: Terraform Setup for AWS EKS

First, set up your basic Terraform configuration for AWS and EKS. We'll use the `eks` module for simplicity, but you can tailor this to your needs.

Create a file named `main.tf`:

resource "aws_vpc" "eks_vpc" {
  cidr_block = "10.0.0.0/16"
  tags = { Name = "eks-observability-vpc" }
}

resource "aws_subnet" "public_subnets" {
  count             = 2
  vpc_id            = aws_vpc.eks_vpc.id
  cidr_block        = cidrsubnet(aws_vpc.eks_vpc.cidr_block, 8, count.index)
  availability_zone = data.aws_availability_zones.available.names[count.index]
  tags = {
    Name                                    = "eks-observability-public-${count.index}"
    "kubernetes.io/cluster/eks-observability" = "shared"
    "kubernetes.io/role/elb"                  = "1"
  }
}

resource "aws_internet_gateway" "eks_gw" {
  vpc_id = aws_vpc.eks_vpc.id
  tags   = { Name = "eks-observability-gw" }
}

resource "aws_route_table" "eks_public_rt" {
  vpc_id = aws_vpc.eks_vpc.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.eks_gw.id
  }
}

resource "aws_route_table_association" "public_rt_association" {
  count          = length(aws_subnet.public_subnets)
  subnet_id      = aws_subnet.public_subnets[count.index].id
  route_table_id = aws_route_table.eks_public_rt.id
}

module "eks_cluster" {
  source  = "terraform-aws-modules/eks/aws"
  version = "19.15.0" # Use a stable version

  cluster_name    = "eks-observability"
  cluster_version = "1.27"
  vpc_id          = aws_vpc.eks_vpc.id
  subnet_ids      = aws_subnet.public_subnets[*].id

  eks_managed_node_groups = {
    observability_nodes = {
      disk_size      = 20
      instance_types = ["t3.medium"]
      min_size       = 1
      max_size       = 3
      desired_size   = 2
    }
  }

  tags = {
    Environment = "Dev"
    Project     = "EKS-Observability"
  }
}

resource "kubernetes_config_map_v1" "aws_auth" {
  metadata {
    name      = "aws-auth"
    namespace = "kube-system"
  }
  data = {
    mapUsers = yamlencode([
      {
        userarn  = aws_iam_user.example.arn
        username = aws_iam_user.example.name
        groups   = ["system:masters"]
      }
    ])
  }
  depends_on = [module.eks_cluster]
}

resource "aws_iam_user" "example" {
  name = "terraform-admin"
  tags = {
    Name = "terraform-admin"
  }
}

resource "aws_iam_user_policy_attachment" "example_admin" {
  user       = aws_iam_user.example.name
  policy_arn = "arn:aws:iam::aws:policy/AdministratorAccess"
}
    

And `versions.tf` for providers:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

provider "kubernetes" {
  host                   = module.eks_cluster.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks_cluster.cluster_certificate_authority_data)
  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args        = ["eks", "get-token", "--cluster-name", module.eks_cluster.cluster_name]
  }
}

provider "helm" {
  kubernetes {
    host                   = module.eks_cluster.cluster_endpoint
    cluster_ca_certificate = base64decode(module.eks_cluster.cluster_certificate_authority_data)
    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args        = ["eks", "get-token", "--cluster-name", module.eks_cluster.cluster_name]
    }
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}
    

And `variables.tf`:

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_service_key" {
  description = "PagerDuty integration key for the Datadog service"
  type        = string
  sensitive   = true
}
    

Initialize and apply your Terraform configuration:

terraform init
terraform apply --var='datadog_api_key=YOUR_DATADOG_API_KEY' --var='datadog_app_key=YOUR_DATADOG_APP_KEY' --var='pagerduty_service_key=YOUR_PAGERDUTY_SERVICE_KEY'
    

Step 2: Deploying Datadog Agent to EKS with Terraform & Helm

Now, let's deploy the Datadog Agent using the `helm_release` resource within your Terraform configuration (`main.tf`). This will ensure the agent is deployed consistently across your EKS cluster.

Add the following to your `main.tf`:

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Or a dedicated monitoring namespace
  version    = "2.38.6"  # Use a recent stable version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }
  set {
    name  = "datadog.kubelet.host"
    value = module.eks_cluster.node_groups.observability_nodes.id
  }
  set {
    name  = "datadog.clusterName"
    value = module.eks_cluster.cluster_name
  }
  set {
    name  = "agents.image.tag"
    value = "7.48.0" # Specify agent version
  }
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "apm.enabled"
    value = "true"
  }
  set {
    name  = "processAgent.enabled"
    value = "true"
  }
  set {
    name  = "systemProbe.enabled"
    value = "true"
  }
  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
  }
  set {
    name  = "networkMonitoring.enabled"
    value = "true"
  }
  set {
    name  = "datadog.env[0].name"
    value = "DD_ADDTIONAL_TAGS"
  }
  set {
    name  = "datadog.env[0].value"
    value = "env:dev,project:eks-observability"
  }
  set {
    name  = "datadog.collectKubeAPIEvents"
    value = "true"
  }

  depends_on = [module.eks_cluster]
}
    

Apply the changes to deploy the Datadog Agent:

terraform apply --var='datadog_api_key=YOUR_DATADOG_API_KEY' --var='datadog_app_key=YOUR_DATADOG_APP_KEY' --var='pagerduty_service_key=YOUR_PAGERDUTY_SERVICE_KEY'
    

Verify the Datadog Agent pods are running in your EKS cluster:

kubectl get pods -n default -l app=datadog
    

Step 3: Configuring Datadog Monitors via Terraform

Now that Datadog is collecting data, let's define some critical monitors using Terraform to detect issues automatically.

Add the following to your `main.tf`:

resource "datadog_monitor" "high_node_cpu" {
  name                = "[EKS] High Node CPU Utilization on {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster_name:eks-observability} by {host} < 10" # less than 10% idle means >90% usage
  message             = "EKS node {{host.name}} CPU utilization is over 90%.\n@webhook-pagerduty-eks-observability"
  escalation_message  = "CPU is still high after 15 minutes. Escalating to on-call."
  tags                = ["environment:dev", "team:devops", "service:eks"]
  priority            = 1
  notify_no_data      = false
  new_group_delay     = 60
  no_data_timeframe   = 20
  renotify_interval   = 0
  include_tags        = true
  require_full_window = true
  timeout_h           = 0
  thresholds {
    critical = 10
    warning  = 20
  }
}

resource "datadog_monitor" "node_memory_pressure" {
  name                = "[EKS] Node Memory Pressure on {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:system.mem.used{kubernetes_cluster_name:eks-observability} by {host} > 90"
  message             = "EKS node {{host.name}} is experiencing high memory usage (>90%).\n@webhook-pagerduty-eks-observability"
  tags                = ["environment:dev", "team:devops", "service:eks"]
  priority            = 1
  notify_no_data      = false
  new_group_delay     = 60
  renotify_interval   = 0
  include_tags        = true
  require_full_window = true
  thresholds {
    critical = 90
    warning  = 80
  }
}

resource "datadog_monitor" "pod_restarts" {
  name                = "[EKS] High Pod Restarts in {{kube_namespace}}"
  type                = "metric alert"
  query               = "sum(last_5m):sum:kubernetes.containers.restarts{kubernetes_cluster_name:eks-observability} by {kube_namespace} > 5"
  message             = "Multiple pod restarts detected in namespace {{kube_namespace}}. Investigate pod logs.\n@webhook-pagerduty-eks-observability"
  tags                = ["environment:dev", "team:devops", "service:eks"]
  priority            = 2
  notify_no_data      = false
  new_group_delay     = 60
  renotify_interval   = 0
  include_tags        = true
  require_full_window = true
  thresholds {
    critical = 5
    warning  = 3
  }
}
    

Re-apply Terraform to create these monitors in Datadog:

terraform apply --var='datadog_api_key=YOUR_DATADOG_API_KEY' --var='datadog_app_key=YOUR_DATADOG_APP_KEY' --var='pagerduty_service_key=YOUR_PAGERDUTY_SERVICE_KEY'
    

Step 4: Integrating Datadog with PagerDuty

The final piece of the puzzle is connecting Datadog alerts to PagerDuty. You'll create a PagerDuty service and then configure Datadog to send alerts to it.

First, within PagerDuty, navigate to Integrations > API Integrations and create a new service. Select Datadog as the integration type. This will give you a PagerDuty Integration Key. If you already have a service, you can obtain its Integration Key from its "Integrations" tab.

Now, let's configure the Datadog PagerDuty integration via Terraform. Add this to `main.tf`:

resource "datadog_integration_pagerduty" "eks_observability_pd" {
  api_key = var.pagerduty_service_key
  name    = "eks-observability-pagerduty-service"
}

resource "datadog_integration_webhook" "pagerduty_webhook" {
  name = "pagerduty-eks-observability"
  url  = "https://events.pagerduty.com/integration/${var.pagerduty_service_key}/enqueue"
  encode_as = "json"
}
    

Notice the `pagerduty_webhook` resource. Datadog's monitor messages (`@webhook-pagerduty-eks-observability`) will automatically trigger this webhook, sending the alert payload to PagerDuty.

Apply the changes:

terraform apply --var='datadog_api_key=YOUR_DATADOG_API_KEY' --var='datadog_app_key=YOUR_DATADOG_APP_KEY' --var='pagerduty_service_key=YOUR_PAGERDUTY_SERVICE_KEY'
    

Step 5: Testing and Validation

After applying all Terraform configurations, perform these validation steps:

Datadog UI: Log into your Datadog account.
- Navigate to Infrastructure > Hosts and verify your EKS nodes are reporting.
- Go to Metrics > Explorer and search for Kubernetes/EKS metrics (e.g., `kubernetes.cpu.usage`, `system.mem.used`).
- Check Logs > Live Tail for EKS logs streaming in.
- Confirm your new monitors are listed under Monitors > Manage Monitors.
Trigger an Alert: To test the PagerDuty integration, you can deliberately cause a monitor to trigger. For example, if you have a CPU monitor, you might temporarily run a CPU-intensive workload on an EKS node.
PagerDuty UI: Verify that an incident is created in PagerDuty once the Datadog monitor state changes to "Alert." Check for correct routing and escalation.

Benefits of This Automated Approach

Automating AWS EKS observability with Terraform, Datadog, and PagerDuty delivers significant advantages:

Reduced MTTR (Mean Time To Resolution): Proactive monitoring and automated incident routing mean issues are detected and addressed faster.
Consistency and Reliability: IaC ensures that your observability stack is deployed uniformly across all environments, reducing configuration drift.
Scalability: As your EKS clusters grow, Terraform easily scales the deployment of Datadog agents and monitor configurations.
Unified Visibility: Datadog provides a single pane of glass for all your EKS metrics, logs, and traces, simplifying troubleshooting.
Optimized On-Call: PagerDuty's intelligent routing minimizes alert fatigue and ensures the right team is notified at the right time.

Best Practices and Advanced Considerations

Granular IAM Permissions: Ensure your EKS nodes and Datadog agents have only the necessary IAM permissions.
Resource Tagging: Extensively tag your AWS resources and Kubernetes objects. Datadog automatically ingests these tags, enabling powerful filtering and segmentation in dashboards and monitors.
Custom Dashboards: While monitors catch issues, build custom Datadog dashboards for operational visibility, capacity planning, and post-mortem analysis.
Synthetic Monitoring: Implement Datadog Synthetics to proactively test your EKS-hosted applications' availability and performance from an end-user perspective.
Runbook Automation: Integrate runbook links or automated remediation steps into your PagerDuty incidents to empower on-call teams for faster resolution.
Cost Optimization: Monitor Datadog ingestion volumes and optimize logging/metric collection to manage costs, especially in large EKS environments.

Troubleshooting / FAQ

Datadog Agent not reporting?

Check Kubernetes logs for the Datadog Agent pods (`kubectl logs -f `). Ensure correct API/APP keys are set in the Helm release. Verify network connectivity from EKS nodes to Datadog endpoints.

PagerDuty incidents not triggering?

Double-check the PagerDuty Integration Key in your Datadog Webhook configuration (`datadog_integration_webhook` resource). Ensure the `@webhook-pagerduty-eks-observability` notification is correctly included in your Datadog monitor messages. Test the webhook manually within Datadog if necessary.

Terraform apply issues with Kubernetes/Helm?

Ensure your `kubectl` context is correctly configured and can access the EKS cluster. The Terraform Kubernetes and Helm providers rely on `kubectl` for authentication via `aws eks get-token`.

Conclusion

Automating AWS EKS observability with Terraform, Datadog, and PagerDuty provides a robust, scalable, and highly efficient solution for maintaining the health and performance of your cloud-native applications. By codifying your infrastructure and monitoring, you empower your DevOps teams to operate with greater confidence, significantly reduce incident response times, and ultimately deliver a more reliable service to your users. Embrace this integrated approach to take your EKS operations to the next level of maturity.

CloudStack YAML