Automating AWS EKS Observability with Terraform, Datadog, and PagerDuty

In the dynamic world of cloud-native applications, maintaining robust observability for your Kubernetes clusters is paramount. This guide provides a comprehensive, technical walkthrough on how to fully automate the setup of AWS EKS observability using a powerful triumvirate: Terraform for Infrastructure as Code (IaC), Datadog for unified monitoring and alerting, and PagerDuty for incident management and on-call automation. By the end of this guide, you'll have a fully automated, scalable, and resilient observability pipeline for your EKS environment.

Architecture Pro-Tip: Always treat your observability stack as a first-class citizen in your infrastructure. Automating its deployment and configuration alongside your core services ensures consistency, reduces manual errors, and accelerates recovery times during incidents. Leverage IaC tools like Terraform to manage Datadog monitors, dashboards, and PagerDuty services, treating them as code artifacts within your version control system.

The Observability Challenge in EKS

Kubernetes, while incredibly powerful, introduces significant complexity in monitoring. Traditional monitoring tools often fall short in dynamic, ephemeral container environments. AWS EKS further abstracts the underlying infrastructure, making a holistic view of performance, health, and security challenging without dedicated solutions. Key challenges include:

Dynamic Workloads: Pods, nodes, and services are constantly scaling up and down.
Distributed Nature: Applications span multiple microservices, requiring distributed tracing.
Resource Utilization: Accurately tracking CPU, memory, and network usage across hundreds of containers.
Alerting Fatigue: Differentiating critical alerts from noisy warnings.
Incident Response: Ensuring timely notification and escalation to the right teams.

Introducing the Stack: Terraform, Datadog, and PagerDuty

Terraform: Infrastructure as Code for Observability

Terraform, by HashiCorp, allows you to define and provision infrastructure using a declarative configuration language. In this context, it extends beyond just AWS EKS infrastructure to manage Datadog monitors, dashboards, and PagerDuty services, ensuring your observability setup is version-controlled, auditable, and repeatable.

Datadog: Unified Monitoring and Analytics

Datadog provides a SaaS-based monitoring and analytics platform for cloud-scale applications. It offers a unified view of metrics, logs, and traces from your entire EKS stack, including Kubernetes control plane, worker nodes, and applications. Its robust alerting engine and extensive integrations make it an ideal choice for EKS observability.

PagerDuty: Intelligent Incident Response

PagerDuty is a leading incident management platform that transforms digital signals into actionable insights, ensuring the right people are alerted at the right time. Integrating Datadog with PagerDuty enables automated incident creation, escalation policies, and on-call scheduling, streamlining your incident response workflow.

Prerequisites

Before you begin, ensure you have the following:

An active AWS account with permissions to manage EKS and IAM resources.
An existing AWS EKS cluster. (This guide focuses on observability, not EKS cluster creation itself).
A Datadog account with API and Application Keys.
A PagerDuty account with an Admin API Key.
Terraform CLI installed (v1.0.0 or higher).
AWS CLI installed and configured.
Helm CLI installed (for Datadog Agent deployment).

Step-by-Step Automation Guide

1. Configure Terraform Providers

Start by defining the required Terraform providers for AWS, Datadog, and PagerDuty. Store your API keys securely, preferably using environment variables or a secrets manager like AWS Secrets Manager.

provider "aws" {
  region = "us-east-1" # Or your desired region
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty Admin API Token"
  type        = string
  sensitive   = true
}
    

2. Automate PagerDuty Service Creation with Terraform

Define your PagerDuty escalation policies and services using Terraform. This ensures consistent incident routing and on-call schedules.

# Create a PagerDuty User (if not existing)
resource "pagerduty_user" "devops_engineer" {
  name  = "DevOps Engineer"
  email = "devops@example.com"
}

# Create a PagerDuty Team
resource "pagerduty_team" "platform_team" {
  name        = "Platform Team"
  description = "Manages EKS infrastructure and observability."
}

# Create an Escalation Policy
resource "pagerduty_escalation_policy" "eks_critical_policy" {
  name      = "EKS Critical Alerts"
  team_id   = pagerduty_team.platform_team.id
  num_loops = 2

  rule {
    delay_in_minutes = 5
    target {
      type = "user"
      id   = pagerduty_user.devops_engineer.id
    }
  }

  rule {
    delay_in_minutes = 15
    target {
      type = "team"
      id   = pagerduty_team.platform_team.id
    }
  }
}

# Create a PagerDuty Service for EKS Observability
resource "pagerduty_service" "eks_observability_service" {
  name                      = "EKS Observability Alerts"
  auto_resolve_timeout      = "14400" # 4 hours
  acknowledgement_timeout   = "600"   # 10 minutes
  escalation_policy         = pagerduty_escalation_policy.eks_critical_policy.id
  alert_creation            = "create_alerts_and_incidents"
  incident_urgency_rule {
    type    = "constant"
    urgency = "high"
  }
  team    = pagerduty_team.platform_team.id
}
    

3. Deploy Datadog Agent to EKS via Terraform and Helm

The Datadog Agent collects metrics, logs, and traces from your EKS cluster. Deploy it using the official Datadog Helm chart. Terraform can manage Helm chart deployments using the Helm provider.

provider "kubernetes" {
  host                   = data.aws_eks_cluster.eks_cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.eks_cluster_auth.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.eks_cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.eks_cluster_auth.token
  }
}

# Data sources for existing EKS cluster info
data "aws_eks_cluster" "eks_cluster" {
  name = "your-eks-cluster-name" # Replace with your EKS cluster name
}

data "aws_eks_cluster_auth" "eks_cluster_auth" {
  name = "your-eks-cluster-name" # Replace with your EKS cluster name
}

# Deploy Datadog Agent using Helm
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Or dedicated monitoring namespace
  version    = "2.33.0" # Use a stable, recent version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "datadog.kubeStateMetricsCore.enabled"
    value = "true"
  }

  set {
    name  = "datadog.logs.enabled"
    value = "true"
  }
  set {
    name  = "datadog.logs.containerCollectAll"
    value = "true"
  }

  set {
    name  = "datadog.apm.enabled"
    value = "true"
  }

  set {
    name  = "datadog.processAgent.enabled"
    value = "true"
  }

  set {
    name  = "datadog.networkMonitoring.enabled"
    value = "true"
  }

  # Enable Datadog Cluster Agent
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  # Enable metrics collection for EKS control plane
  set {
    name  = "clusterChecksRunner.enabled"
    value = "true"
  }

  # Tags for all metrics, logs, traces
  set {
    name  = "tags"
    value = "{env:production,cluster:your-eks-cluster-name,team:platform}"
  }
}
    

4. Create Datadog Monitors and Dashboards with Terraform

Now, let's define critical Datadog monitors using Terraform. These monitors will automatically trigger alerts and, when configured, integrate with PagerDuty. You can also create dashboards to visualize your EKS health.

Example: EKS Node Not Ready Monitor

This monitor alerts if any EKS node becomes "NotReady", indicating a potential infrastructure issue.

resource "datadog_monitor" "eks_node_not_ready" {
  name               = "[EKS] Node Not Ready - {{kube_cluster_name.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):sum:kubernetes.node.ready{*} by {host} < 1"
  message            = <Explanation of the Datadog Monitor:
`query`: Targets the `kubernetes.node.ready` metric, which indicates the health status of a node. It alerts if the average value over 5 minutes drops below 1 for any host.
`message`: This is critical for PagerDuty integration. The `@webhook-pagerduty-service-id-` syntax directly tells Datadog to send this alert to the specified PagerDuty service. We use Terraform interpolation to dynamically insert the PagerDuty service ID.
`tags`: Helps categorize and filter your monitors in Datadog.
`priority`: Defines the severity in Datadog.
Example: EKS Pod Restarts MonitorDetects excessive pod restarts, often a sign of application instability.

resource "datadog_monitor" "eks_pod_restarts" {
  name               = "[EKS] High Pod Restarts - {{kube_cluster_name.name}} ({{kube_namespace.name}}/{{kube_app.name}})"
  type               = "metric alert"
  query              = "sum(last_5m):kubernetes.containers.restarts{kube_cluster_name:your-eks-cluster-name} by {kube_namespace,kube_app} > 5"
  message            = <Creating a Datadog Dashboard for EKSYou can also define comprehensive dashboards using Terraform to visualize key EKS metrics.

resource "datadog_dashboard" "eks_overview_dashboard" {
  title        = "EKS Cluster Overview - ${data.aws_eks_cluster.eks_cluster.name}"
  description  = "High-level overview of EKS cluster health."
  layout_type  = "ordered"
  is_read_only = false
  tags         = ["eks", "kubernetes", "overview", "terraform"]

  widget {
    definition {
      title      = "CPU Utilization"
      type       = "timeseries"
      requests {
        q = "sum:kubernetes.cpu.usage.total{*} by {host}"
      }
    }
  }

  widget {
    definition {
      title      = "Memory Utilization"
      type       = "timeseries"
      requests {
        q = "sum:kubernetes.memory.usage.total{*} by {host}"
      }
    }
  }

  widget {
    definition {
      title      = "Pod Restarts (Last Hour)"
      type       = "timeseries"
      requests {
        q = "sum:kubernetes.containers.restarts{*} by {kube_namespace,kube_app}"
        aggregator = "sum"
        change_type = "1 hour"
      }
    }
  }

  widget {
    definition {
      title      = "Node Status"
      type       = "hostmap"
      requests {
        q = "kubernetes.node.ready{*}"
      }
      node_type = "host"
      group_by = ["host"]
      style {
        palette = "white_on_green"
      }
    }
  }
}
    
Deployment and ValidationOnce your Terraform configurations are ready, deploy them:
Initialize Terraform: Navigate to your Terraform root directory and run `terraform init`.
Review Plan: Execute `terraform plan -var="datadog_api_key=" -var="datadog_app_key=" -var="pagerduty_api_token="`. Carefully review the changes Terraform proposes.
Apply Changes: If the plan looks correct, run `terraform apply -var="datadog_api_key=" -var="datadog_app_key=" -var="pagerduty_api_token="`. Confirm with `yes`.
Validation Steps:
Verify Datadog Agent: Check your EKS cluster pods with `kubectl get pods -n default | grep datadog`. All Datadog pods should be running.
Verify Datadog Monitors/Dashboards: Log into your Datadog account. You should see the newly created monitors and dashboards under "Monitors" and "Dashboards" sections, respectively.
Verify PagerDuty Services: Log into your PagerDuty account. Confirm the "EKS Observability Alerts" service and its associated escalation policy are present.
Test Alerting: Trigger a test alert (e.g., scale down a node or intentionally cause a pod restart) to ensure PagerDuty incidents are created as expected.


Best Practices for EKS ObservabilityCentralized Logging: Ensure all container logs are collected and forwarded to Datadog for easy searching and analysis.
Distributed Tracing: Instrument your applications to send traces to Datadog APM for end-to-end visibility into service requests.
Resource Tagging: Consistently tag your AWS resources and Kubernetes objects to enable granular filtering and correlation in Datadog.
Review and Refine Alerts: Regularly review your Datadog monitors to reduce alert fatigue and ensure they remain relevant to your operational needs.
Automate Runbooks: For common issues detected by Datadog, integrate PagerDuty with automation tools to trigger self-healing actions.
Version Control Everything: Treat all your observability configurations (Terraform files, Datadog dashboard JSON, PagerDuty service definitions) as code in a Git repository.
Troubleshooting Common IssuesDatadog Agent Pods Not RunningSymptom: `datadog-agent` pods are in `Pending` or `CrashLoopBackOff` state.
Solution:
Check pod logs: `kubectl logs -f  -n default`.
Verify `datadog.apiKey` and `datadog.appKey` in Helm release values. Incorrect keys are a common cause.
Ensure your EKS IAM role associated with the nodes has permissions to pull images from ECR (if you're using a private registry) and other necessary AWS API calls.
Datadog Monitors Not Triggering PagerDutySymptom: Datadog alerts are visible but no PagerDuty incident is created.
Solution:
Verify the PagerDuty integration in Datadog. Go to Integrations -> PagerDuty in Datadog and ensure it's configured correctly.
Double-check the `@webhook-pagerduty-service-id-` syntax in your Datadog monitor message. The `service_id` must exactly match the ID of your PagerDuty service created by Terraform.
Check Datadog event stream for any errors related to sending webhooks to PagerDuty.
ConclusionAutomating AWS EKS observability with Terraform, Datadog, and PagerDuty provides a powerful, scalable, and reliable solution for managing your cloud-native environments. By treating your observability stack as code, you gain consistency, reduce operational overhead, and significantly improve your team's ability to detect, diagnose, and resolve incidents efficiently. Embrace this integrated approach to build a resilient and highly observable EKS infrastructure.

CloudStack YAML