Automating AWS EKS Observability with Terraform, Datadog, and PagerDuty

In today's dynamic cloud-native landscape, ensuring the reliability and performance of Kubernetes clusters is paramount. AWS Elastic Kubernetes Service (EKS) offers a robust foundation, but effective observability—understanding the internal state of your system from its external outputs—is crucial for maintaining health and responding swiftly to issues. This guide will walk you through automating comprehensive EKS observability using a powerful combination of Terraform for infrastructure as code, Datadog for monitoring and alerting, and PagerDuty for streamlined incident management.

Architecture Pro-Tip:

Always design your observability stack with a "shift-left" mindset. Integrate monitoring and alerting definitions directly into your Infrastructure as Code (IaC) pipelines. This ensures that every new EKS cluster, service, or deployment automatically comes with its predefined observability components, minimizing human error and accelerating time to detection for potential issues. Prioritize unified dashboards and clear escalation paths from the outset.

Why Automate EKS Observability?

Manual configuration of monitoring tools, alert thresholds, and incident response workflows is prone to errors, inconsistency, and significant time investment, especially in environments with multiple EKS clusters or rapidly evolving microservices. Automation offers:

Consistency: Ensures uniform observability standards across all your EKS deployments.
Speed: Rapidly deploy and update monitoring configurations without manual intervention.
Reliability: Reduces human error and ensures that critical alerts are never missed.
Scalability: Easily extend observability to new services or clusters as your infrastructure grows.
Auditability: Terraform provides a clear, version-controlled record of your observability setup.

Core Components Overview

AWS EKS: The Foundation

AWS EKS provides a managed Kubernetes control plane, abstracting away the complexity of managing the master nodes. However, visibility into the health and performance of your worker nodes, pods, and applications running on EKS remains essential.

Datadog: Unified Monitoring and Alerting

Datadog offers a comprehensive SaaS platform for monitoring cloud applications, servers, and infrastructure. It provides full-stack visibility with metrics, logs, traces, and UX monitoring, making it ideal for complex Kubernetes environments. Its robust alerting engine allows for sophisticated thresholding and anomaly detection.

PagerDuty: Intelligent Incident Management

PagerDuty is a leading incident management platform that transforms any signal into an actionable incident. It offers intelligent alerting, on-call scheduling, and automated escalations, ensuring that critical issues from Datadog (or other sources) reach the right people at the right time.

Terraform: Infrastructure as Code (IaC)

Terraform, by HashiCorp, allows you to define and provision infrastructure using a declarative configuration language. Crucially, Terraform has providers for AWS, Datadog, and PagerDuty, enabling you to manage your entire observability stack alongside your EKS cluster itself.

Prerequisites

Before you begin, ensure you have the following:

An AWS Account with appropriate IAM permissions to create/manage EKS clusters.
A Datadog Account with an API key and Application key.
A PagerDuty Account with an API key.
Terraform CLI installed (v1.0.0+ recommended).
kubectl CLI installed and configured to connect to your EKS cluster.
Helm CLI installed (v3+ recommended) for deploying the Datadog Agent.
An existing AWS EKS cluster, or the ability to create one via Terraform. This guide assumes you have an EKS cluster ready or are provisioning one alongside these observability components.

Step-by-Step Automation Guide

1. Initialize Terraform Providers

Start by defining your AWS, Datadog, and PagerDuty providers in your main.tf file. Securely manage your API keys using environment variables or a secrets manager.

provider "aws" {
  region = "us-east-1"
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_token
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

2. Deploy Datadog Agent to EKS using Helm

While the Datadog Agent itself is deployed using Helm, we can use Terraform's null_resource and local-exec provisioner to automate its deployment post-EKS creation. Alternatively, if your EKS cluster is managed by Terraform, you can use the Terraform Helm provider to deploy the agent directly.

First, ensure your EKS cluster output provides the necessary Kubeconfig or context for kubectl and helm.

Here's how you might set up the Helm chart deployment via Terraform (assuming kubeconfig is configured):

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Or your dedicated monitoring namespace
  version    = "2.33.0" # Use a specific version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "datadog.site"
    value = "datadoghq.com" # Or your specific Datadog site (e.g., eu.datadoghq.com)
  }

  set {
    name  = "clusterAgent.enabled"
    value = true
  }

  set {
    name  = "kubeStateMetrics.enabled"
    value = true
  }

  # Ensure the agent collects logs
  set {
    name  = "logs.enabled"
    value = true
  }
  set {
    name  = "logs.containerCollectAll"
    value = true
  }

  # Ensure the agent collects APM traces
  set {
    name  = "apm.enabled"
    value = true
  }

  # Depending on your EKS worker node types, you might need to specify image
  # set {
  #   name  = "datadog.image.repository"
  #   value = "gcr.io/datadog/agent"
  # }
  # set {
  #   name  = "datadog.image.tag"
  #   value = "7.48.0"
  # }

  # Optional: For Fargate, use the Datadog Cluster Agent and enable Fargate support
  # set {
  #   name  = "fargate.enabled"
  #   value = true
  # }
}

3. Automate Datadog Monitors and Dashboards with Terraform

Once the Datadog Agent is collecting data, you can define monitors and dashboards using Terraform. This ensures that every EKS deployment includes a baseline of critical alerts and visualization.

Example: Datadog Monitor for EKS Node CPU Utilization

resource "datadog_monitor" "eks_node_cpu_high" {
  name               = "[EKS] High Node CPU Utilization on {{host.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster_name:${var.eks_cluster_name}} by {host} < 10" # Alert if idle CPU < 10%
  message            = "CPU utilization for node {{host.name}} in EKS cluster ${var.eks_cluster_name} is consistently high. @webhook-pagerduty-eks-alerts"
  tags               = ["environment:${var.environment}", "service:eks", "severity:critical"]
  priority           = 1
  restricted_roles   = []

  thresholds {
    critical = 10
    warning  = 20
  }

  notify_no_data = false
  renotify_interval = 60 # Minutes
  require_full_window = true
  timeout_h = 0
}

variable "eks_cluster_name" {
  description = "Name of the EKS cluster"
  type        = string
}

variable "environment" {
  description = "Deployment environment (e.g., dev, staging, prod)"
  type        = string
}

Example: Datadog Dashboard for EKS Cluster Overview

resource "datadog_dashboard" "eks_overview_dashboard" {
  title        = "EKS Cluster Overview - ${var.eks_cluster_name}"
  description  = "High-level overview of EKS cluster health and performance."
  layout_type  = "ordered"
  is_read_only = true

  widget {
    id   = 123456789 # Placeholder for dynamic ID in practice
    type = "group"
    layout {
      x      = 0
      y      = 0
      width  = 4
      height = 2
    }
    widget {
      definition {
        title = "Node CPU Utilization"
        type  = "timeseries"
        request {
          query {
            query_string = "avg:kubernetes.cpu.usage.total{kubernetes_cluster_name:${var.eks_cluster_name}}"
          }
          display_type = "line"
        }
      }
    }
    widget {
      definition {
        title = "Node Memory Utilization"
        type  = "timeseries"
        request {
          query {
            query_string = "avg:kubernetes.memory.usage.total{kubernetes_cluster_name:${var.eks_cluster_name}}"
          }
          display_type = "line"
        }
      }
    }
  }

  # Add more widgets for pods, deployments, network, etc.
  # For simplicity, only two widgets are shown.
}

4. Integrate PagerDuty for Incident Management

To route critical Datadog alerts to your on-call teams, you'll integrate PagerDuty. This involves creating a PagerDuty service and an integration key (e.g., for Datadog's webhook) using Terraform, then configuring Datadog to use this integration.

Example: PagerDuty Service and Integration

resource "pagerduty_service" "eks_monitoring_service" {
  name                   = "EKS Cluster Monitoring - ${var.eks_cluster_name}"
  auto_resolve_timeout   = "14400" # 4 hours
  acknowledgement_timeout = "600"  # 10 minutes
  escalation_policy      = pagerduty_escalation_policy.devops_ep.id
  description            = "Handles incidents related to AWS EKS cluster ${var.eks_cluster_name} infrastructure and applications."
}

resource "pagerduty_integration" "datadog_eks_integration" {
  name        = "Datadog Integration for EKS ${var.eks_cluster_name}"
  type        = "generic_events_api_inbound_integration"
  service     = pagerduty_service.eks_monitoring_service.id
  description = "Receives alerts from Datadog for EKS Cluster ${var.eks_cluster_name}."
}

# Example Escalation Policy (replace with your actual policy or create a new one)
resource "pagerduty_escalation_policy" "devops_ep" {
  name = "DevOps Team Escalation Policy"
  rule {
    escalation_delay_in_minutes = 10
    target {
      id   = pagerduty_user.devops_lead.id
      type = "user"
    }
  }
  # Add more rules for escalation
}

# Example PagerDuty User (replace with your actual user or data source)
resource "pagerduty_user" "devops_lead" {
  name  = "DevOps Lead"
  email = "devops.lead@example.com"
}

output "pagerduty_datadog_integration_url" {
  description = "The URL for Datadog to send events to PagerDuty."
  value       = "https://events.pagerduty.com/generic/2010-04-15/create_event.json" # Generic V1 API endpoint
}

output "pagerduty_integration_key" {
  description = "The integration key for the Datadog PagerDuty integration."
  value       = pagerduty_integration.datadog_eks_integration.integration_key
  sensitive   = true
}

After applying the above Terraform, you'll get the pagerduty_integration_key as an output. You'll then configure a Datadog webhook integration to send alerts to PagerDuty. You can automate this step using the Datadog provider's datadog_integration_webhook resource.

Configure Datadog Webhook to PagerDuty

resource "datadog_integration_webhook" "pagerduty_webhook" {
  name = "pagerduty-eks-alerts"
  url  = "${output.pagerduty_datadog_integration_url.value}?service_key=${output.pagerduty_integration_key.value}"
}

Notice the @webhook-pagerduty-eks-alerts in the Datadog monitor message above. This tells Datadog to send alerts via the webhook we just defined, effectively routing them to PagerDuty.

5. Deploy and Verify

Once all your Terraform configurations are ready:

Run terraform init to initialize your providers.
Run terraform plan to review the changes.
Run terraform apply to provision your Datadog and PagerDuty resources.
Verify the Datadog Agent pods are running correctly in your EKS cluster (kubectl get pods -n default | grep datadog).
Check your Datadog account for new monitors and dashboards.
Verify the PagerDuty service and integration are created.
Trigger a test alert (e.g., by intentionally causing high CPU on a node or using Datadog's test alert feature) to ensure PagerDuty receives it.

Benefits of this Automated Approach

End-to-End Observability: From metric collection to incident response, the entire flow is automated.
Developer Self-Service: Developers can define their service's observability requirements alongside their application code and EKS deployments.
Disaster Recovery Readiness: Rebuilding your EKS environment and its observability stack becomes a simple terraform apply command.
Cost Optimization: Efficiently manage Datadog resources, preventing alert fatigue and unnecessary monitoring of deprecated resources.

Troubleshooting and Best Practices

Common Issues:

Datadog Agent not reporting: Check pod logs (kubectl logs <datadog-agent-pod>), ensure correct API/APP keys, and verify network connectivity to Datadog endpoints.
PagerDuty alerts not firing: Double-check the Datadog webhook URL and integration key. Ensure the Datadog monitor's message references the correct webhook name (e.g., @webhook-pagerduty-eks-alerts).
Terraform state locking: Use a remote backend (like S3 with DynamoDB locking) for collaborative environments to prevent state corruption.

Best Practices:

Version Control Everything: Store all Terraform configurations in a Git repository.
Modularize Terraform: Break down your configurations into reusable modules for EKS, Datadog monitors, PagerDuty services, etc.
Use Service Accounts: For Helm deployments, use Kubernetes Service Accounts with appropriate IAM roles (via IRSA) for enhanced security.
Secrets Management: Never hardcode API keys. Use environment variables, AWS Secrets Manager, or HashiCorp Vault.
Granular Alerts: Avoid "noisy" alerts. Start with critical alerts and refine thresholds as you understand your system's baseline.
Regularly Review: Periodically review your monitors, dashboards, and escalation policies to ensure they remain relevant.

Conclusion

Automating AWS EKS observability with Terraform, Datadog, and PagerDuty transforms your operational capabilities. It shifts your organization from reactive firefighting to proactive incident prevention and rapid resolution, ensuring higher availability and reliability for your cloud-native applications. By embracing Infrastructure as Code for your entire observability stack, you build a resilient, scalable, and auditable system that can confidently grow with your business needs.

CloudStack YAML