Automating AWS EKS Observability with Terraform, Datadog, and PagerDuty

In the dynamic world of cloud-native applications, maintaining robust observability for Kubernetes clusters is paramount. AWS EKS (Elastic Kubernetes Service) provides a powerful foundation for container orchestration, but ensuring its health, performance, and security requires a comprehensive monitoring and incident response strategy. This guide delves into automating EKS observability using a powerful triad: Terraform for infrastructure as code, Datadog for comprehensive monitoring and APM, and PagerDuty for streamlined incident management.

Architecture Pro-Tip:

Always define your observability stack as code alongside your core infrastructure. This "Observability as Code" approach ensures consistency, version control, and rapid disaster recovery for your monitoring, logging, and alerting configurations. Treat your Datadog dashboards, monitors, and PagerDuty services with the same rigor as your EKS cluster definition.

Why Automate EKS Observability?

The complexity of distributed systems running on Kubernetes makes manual monitoring an impossible task. Automation is not just an efficiency gain; it's a necessity for reliable operations.

Challenges of Manual EKS Observability

Scalability Issues: Manually configuring monitoring for new services or pods becomes unsustainable as your cluster scales.
Configuration Drift: Inconsistent monitoring setups across environments lead to blind spots and unreliable alerts.
Slow Incident Response: Lack of automated alerting and on-call rotation can delay critical incident resolution.
Incomplete Visibility: Disjointed tools for logs, metrics, and traces lead to fragmented insights.

Benefits of an Automated Approach

Consistency & Reproducibility: Terraform ensures your entire observability stack is defined, versioned, and deployed consistently.
Faster Time to Value: Rapidly deploy comprehensive monitoring and alerting for new services or clusters.
Reduced Operational Overhead: Minimize manual configuration tasks and errors.
Enhanced Reliability: Proactive monitoring and swift incident response prevent minor issues from becoming major outages.
Unified Visibility: Datadog centralizes metrics, logs, and traces for end-to-end visibility across your EKS environment.

Core Components: Terraform, Datadog, and PagerDuty

This guide leverages three industry-leading tools, each playing a crucial role in establishing a robust, automated observability pipeline.

Terraform for Infrastructure as Code (IaC)

Terraform, by HashiCorp, is the backbone of our automation strategy. It allows us to define and provision infrastructure – including AWS EKS clusters, Datadog configurations, and PagerDuty services – using a declarative configuration language. This ensures our entire observability setup is version-controlled, auditable, and easily repeatable.

Datadog for Comprehensive Monitoring & APM

Datadog provides a unified platform for monitoring, logging, and tracing. Its Kubernetes integration is deep, collecting metrics from the control plane, Kubelet, cAdvisor, and applications. Datadog allows us to:

Collect thousands of metrics from EKS, nodes, pods, and containers.
Aggregate logs from all sources within the cluster.
Perform distributed tracing for microservices running on EKS.
Create sophisticated dashboards and powerful monitors with anomaly detection.

PagerDuty for Incident Management

Once Datadog detects an issue, PagerDuty ensures the right person or team is notified immediately. It provides intelligent alert routing, on-call schedules, escalation policies, and incident tracking, transforming raw alerts into actionable incidents. Integrating Datadog with PagerDuty closes the loop from detection to resolution.

Prerequisites

Before you begin, ensure you have the following:

An AWS account with appropriate IAM permissions to create EKS clusters, VPCs, EC2 instances, etc.
AWS CLI installed and configured.
Terraform CLI installed (v1.0+ recommended).
A Datadog account with an API Key and Application Key.
A PagerDuty account with an API Key and a service integration key (for Datadog).
Basic understanding of AWS EKS, Kubernetes, Terraform, and Docker.

Step-by-Step Implementation Guide

We'll walk through the process of setting up your EKS cluster, deploying the Datadog Agent, and configuring monitors and alerts, all managed by Terraform.

1. Project Setup

Create a project directory and initialize your Terraform configuration.

mkdir eks-observability-automation
cd eks-observability-automation
touch main.tf variables.tf outputs.tf providers.tf

2. Configure Terraform Providers

Define the AWS, Datadog, and PagerDuty providers in your providers.tf file.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
    pagerduty = {
      source  = "PagerDuty/pagerduty"
      version = "~> 2.0"
    }
  }
}

provider "aws" {
  region = "us-east-1" # Or your preferred region
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

Your variables.tf should define these keys:

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

3. AWS EKS Cluster Setup (Terraform)

While a full EKS cluster definition is extensive, here's a simplified snippet for your main.tf. This assumes you have VPC and IAM roles defined elsewhere or inline.

# Simplified EKS Cluster and Node Group
resource "aws_eks_cluster" "main" {
  name     = "my-eks-cluster"
  role_arn = aws_iam_role.eks_cluster_role.arn

  vpc_config {
    subnet_ids         = [aws_subnet.private[0].id, aws_subnet.private[1].id]
    security_group_ids = [aws_security_group.eks_cluster_sg.id]
  }

  enabled_cluster_log_types = ["api", "audit", "authenticator", "controllerManager", "scheduler"]

  # ... other configurations like Kubernetes version, encryption ...
}

resource "aws_eks_node_group" "main" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "my-node-group"
  node_role_arn   = aws_iam_role.eks_nodegroup_role.arn
  subnet_ids      = [aws_subnet.private[0].id, aws_subnet.private[1].id]

  instance_types = ["t3.medium"]
  desired_size   = 2
  max_size       = 3
  min_size       = 1

  # ... other configurations ...
}

# Output Kubeconfig details for connecting
output "kubeconfig" {
  value = <<-EOT
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ${base64decode(aws_eks_cluster.main.certificate_authority.0.data)}
    server: ${aws_eks_cluster.main.endpoint}
  name: arn:aws:eks:${var.aws_region}:${data.aws_caller_identity.current.account_id}:cluster/${aws_eks_cluster.main.name}
contexts:
- context:
    cluster: arn:aws:eks:${var.aws_region}:${data.aws_caller_identity.current.account_id}:cluster/${aws_eks_cluster.main.name}
    user: arn:aws:eks:${var.aws_region}:${data.aws_caller_identity.current.account_id}:cluster/${aws_eks_cluster.main.name}
  name: arn:aws:eks:${var.aws_region}:${data.aws_caller_identity.current.account_id}:cluster/${aws_eks_cluster.main.name}
current-context: arn:aws:eks:${var.aws_region}:${data.aws_caller_identity.current.account_id}:cluster/${aws_eks_cluster.main.name}
kind: Config
preferences: {}
users:
- name: arn:aws:eks:${var.aws_region}:${data.aws_caller_identity.current.account_id}:cluster/${aws_eks_cluster.main.name}
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      command: aws
      args:
        - "eks"
        - "get-token"
        - "--cluster-name"
        - "${aws_eks_cluster.main.name}"
        - "--region"
        - "${var.aws_region}"
      installHint: |
        The aws-cli must be installed and configured to use this cluster.
        See https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html
EOT
  sensitive = true
}

4. Datadog Agent Deployment (Terraform)

The Datadog Agent is typically deployed as a DaemonSet within Kubernetes. We can manage this deployment using the kubernetes_manifest resource (or by applying a Helm chart via the helm_release resource). For simplicity and broad compatibility, we'll demonstrate using the kubernetes_manifest which requires the Kubernetes provider configured to connect to your EKS cluster.

# Add Kubernetes Provider (requires kubectl and AWS credentials configured to access EKS)
provider "kubernetes" {
  host                   = aws_eks_cluster.main.endpoint
  cluster_ca_certificate = base64decode(aws_eks_cluster.main.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.main.token
}

data "aws_eks_cluster_auth" "main" {
  name = aws_eks_cluster.main.name
}

# Deploy Datadog Agent using Helm (Recommended)
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog" # Create this namespace if it doesn't exist
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "datadog.clusterName"
    value = aws_eks_cluster.main.name
  }

  set {
    name  = "kubeStateMetricsCore.enabled"
    value = "true"
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "clusterChecksRunner.enabled"
    value = "true"
  }
}

Note: You'll need the helm Terraform provider configured separately or install it as a plugin. The above snippet assumes the Kubernetes provider is already configured to access your EKS cluster.

5. Datadog Dashboards & Monitors (Terraform)

Now that the Datadog Agent is deployed, we can use the Datadog Terraform provider to define dashboards and monitors.

Example: Datadog EKS Overview Dashboard

Create a dashboard to visualize key EKS metrics in main.tf:

resource "datadog_dashboard" "eks_overview" {
  title        = "EKS Cluster: ${aws_eks_cluster.main.name} Overview"
  description  = "Key metrics for EKS Cluster ${aws_eks_cluster.main.name}"
  layout_type  = "ordered"
  is_read_only = true

  widget {
    host_map_definition {
      title      = "Node Health"
      node_type  = "host"
      scope      = ["kubernetes_cluster:${aws_eks_cluster.main.name}"]
      group      = ["instance_type"]
      metric     = "system.cpu.idle"
      fill_empty_cells = true
      style {
        palette = "host_map"
        palette_flip = false
        fill_min     = "0"
        fill_max     = "100"
      }
    }
  }

  widget {
    timeseries_definition {
      title = "Kubernetes Pod Restarts"
      request {
        q = "sum:kubernetes.containers.restarts{kubernetes_cluster:${aws_eks_cluster.main.name}} by {host}"
        display_type = "line"
      }
    }
  }

  # ... Add more widgets for CPU, Memory, Network, Kube-API metrics, etc. ...
}

Example: Datadog EKS Node CPU Monitor

Configure a monitor to alert if node CPU utilization is too high:

resource "datadog_monitor" "eks_node_cpu_high" {
  name               = "[EKS - ${aws_eks_cluster.main.name}] High CPU Utilization on Node {{host.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster:${aws_eks_cluster.main.name}} by {host} < 20"
  message            = "CPU utilization is above 80% for {{host.name}} in EKS cluster ${aws_eks_cluster.main.name}.\n@pagerduty-eks-oncall"
  escalation_message = "CPU utilization remains high after 10 minutes. Please investigate urgently.\n@pagerduty-eks-oncall"
  notify_no_data     = true
  no_data_timeframe  = 10
  renotify_interval  = 0
  timeout_h          = 0
  thresholds {
    critical = "20"
    warning  = "30"
  }
  tags               = ["environment:production", "service:eks", "cluster:${aws_eks_cluster.main.name}"]
}

The @pagerduty-eks-oncall tag in the message will be used to integrate with PagerDuty.

6. PagerDuty Integration (Terraform)

First, define a PagerDuty service and an integration with Datadog.

# PagerDuty service for EKS alerts
resource "pagerduty_service" "eks_service" {
  name        = "EKS Cluster ${aws_eks_cluster.main.name} Service"
  auto_resolve_timeout = "14400" # 4 hours
  acknowledgement_timeout = "1800"  # 30 minutes
  escalation_policy = data.pagerduty_escalation_policy.default.id # Reference an existing policy
}

# PagerDuty integration for Datadog
resource "pagerduty_extension" "datadog_integration" {
  name = "Datadog Integration for EKS"
  endpoint = "https://events.pagerduty.com/integration/${pagerduty_service_integration.datadog_eks.integration_key}/enqueue"
  type = "generic_v2"
  service = pagerduty_service.eks_service.id
}

resource "pagerduty_service_integration" "datadog_eks" {
  name    = "Datadog"
  service = pagerduty_service.eks_service.id
  type    = "generic_events_api_v2" # Use Events API v2
}

# Data source to get an existing escalation policy (replace with your actual policy name/ID)
data "pagerduty_escalation_policy" "default" {
  name = "Default Escalation Policy"
}

Next, you need to configure Datadog to use this integration. This is typically done within Datadog's UI by adding an integration, or via the Datadog API if a Terraform provider for this specific part is available. For Datadog monitors, you can directly reference PagerDuty via the @pagerduty-[service_name] notification syntax, where [service_name] is the routing key or integration name configured in Datadog's PagerDuty integration settings. In our example, we used @pagerduty-eks-oncall, implying a PagerDuty integration named "eks-oncall" is set up in Datadog.

Ready-to-Use Configuration Snippet

Here’s a conceptual summary of the core Terraform configuration, omitting detailed IAM, VPC, and other AWS-specific setup for brevity, focusing on the Datadog and PagerDuty parts. You'd integrate this with your existing EKS setup.

# main.tf (excerpt)

# Assumes aws_eks_cluster.main is defined and accessible

# --- Datadog Agent Deployment (via Helm) ---
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set { name  = "datadog.apiKey"; value = var.datadog_api_key; sensitive = true }
  set { name  = "datadog.appKey"; value = var.datadog_app_key; sensitive = true }
  set { name  = "datadog.clusterName"; value = aws_eks_cluster.main.name }
  set { name  = "kubeStateMetricsCore.enabled"; value = "true" }
  set { name  = "clusterAgent.enabled"; value = "true" }
}

# --- PagerDuty Service & Integration ---
resource "pagerduty_service" "eks_service" {
  name              = "EKS Cluster ${aws_eks_cluster.main.name} Service"
  escalation_policy = data.pagerduty_escalation_policy.default.id
  auto_resolve_timeout = "14400"
  acknowledgement_timeout = "1800"
}

resource "pagerduty_service_integration" "datadog_eks" {
  name    = "Datadog"
  service = pagerduty_service.eks_service.id
  type    = "generic_events_api_v2"
}

# --- Datadog Monitor for EKS Node CPU ---
resource "datadog_monitor" "eks_node_cpu_high" {
  name               = "[EKS - ${aws_eks_cluster.main.name}] High CPU Utilization on Node {{host.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster:${aws_eks_cluster.main.name}} by {host} < 20"
  message            = "CPU utilization above 80% on {{host.name}}.\n@pagerduty-${replace(lower(pagerduty_service.eks_service.name), "/[^a-z0-9]+/", "-")}"
  notify_no_data     = true
  no_data_timeframe  = 10
  thresholds { critical = "20"; warning = "30" }
  tags               = ["environment:production", "service:eks", "cluster:${aws_eks_cluster.main.name}"]
}

# --- Datadog Dashboard for EKS Overview ---
resource "datadog_dashboard" "eks_overview" {
  title        = "EKS Cluster: ${aws_eks_cluster.main.name} Overview"
  description  = "Key metrics for EKS Cluster ${aws_eks_cluster.main.name}"
  layout_type  = "ordered"
  is_read_only = true
  widget {
    timeseries_definition {
      title = "Node CPU Usage"
      request {
        q = "avg:system.cpu.idle{kubernetes_cluster:${aws_eks_cluster.main.name}} by {host}"
        display_type = "line"
      }
    }
  }
}

To apply this configuration, run the following commands:

terraform init
terraform plan -var='datadog_api_key=your_dd_api_key' -var='datadog_app_key=your_dd_app_key' -var='pagerduty_api_token=your_pd_api_token'
terraform apply -var='datadog_api_key=your_dd_api_key' -var='datadog_app_key=your_dd_app_key' -var='pagerduty_api_token=your_pd_api_token'

Advanced Observability & Best Practices

This foundational setup can be expanded significantly for more granular control and deeper insights.

Custom Metrics and Logs

Application-Specific Metrics: Instrument your applications with Datadog's libraries to collect custom metrics and integrate with APM.
Structured Logging: Ensure applications log in JSON format for easier parsing and querying in Datadog Log Management.
Custom Checks: Extend Datadog Agent with custom checks for specific application health or business metrics.

Distributed Tracing (APM)

For microservices architectures on EKS, distributed tracing is critical. Datadog APM integrates seamlessly to visualize service dependencies, identify latency bottlenecks, and troubleshoot distributed transactions. Ensure your application code is instrumented with Datadog's APM libraries.

Cost Optimization Observability

Monitor resource utilization (kubernetes.cpu.usage, kubernetes.memory.usage) and set up alerts for under-utilized resources or unexpected cost spikes, potentially integrating with AWS Cost Explorer data in Datadog.

Troubleshooting Common Issues

Datadog Agent Not Reporting: Check the Datadog Agent pod logs (kubectl logs -f datadog-agent-xxxx), ensure API/App keys are correct, and verify network connectivity to Datadog endpoints.
PagerDuty Alerts Not Firing: Confirm the Datadog-PagerDuty integration is correctly set up in Datadog, verify the notification syntax in your Datadog monitor message (@pagerduty-[service_name]), and check PagerDuty incident logs.
Terraform Apply Errors: Review the error messages carefully. Common issues include IAM permissions, incorrect AWS region, or syntax errors in HCL. Use terraform plan to catch issues before applying.
Kubernetes Provider Authentication: Ensure your local kubectl is configured to access the EKS cluster and that your AWS CLI credentials have permissions to run aws eks get-token.

Conclusion

Automating AWS EKS observability with Terraform, Datadog, and PagerDuty provides a robust, scalable, and resilient foundation for managing your cloud-native workloads. By treating your entire observability stack as code, you gain consistency, reduce operational toil, and significantly improve your team's ability to quickly detect, diagnose, and resolve issues. This integrated approach is essential for maintaining high availability and performance in today's complex Kubernetes environments.

Embrace this automated paradigm to empower your DevOps teams, ensuring your EKS clusters are not just running, but are observable, reliable, and ready for whatever comes next.

CloudStack YAML