Automating Full-Stack Observability for AWS EKS with Terraform, Datadog, and PagerDuty

Architecture Pro-Tip: Always embrace Infrastructure as Code (IaC) for your observability stack. By defining Datadog monitors, dashboards, and PagerDuty services in Terraform, you gain version control, auditability, and the ability to replicate your observability setup across environments consistently and efficiently. This shifts observability left, integrating it into your CI/CD pipelines.

Automating Full-Stack Observability for AWS EKS with Terraform, Datadog, and PagerDuty

In today's dynamic cloud-native landscape, ensuring the reliability and performance of applications running on Amazon Elastic Kubernetes Service (EKS) is paramount. Full-stack observability—encompassing metrics, logs, and traces—provides the critical insights needed to understand system behavior, diagnose issues, and optimize resource utilization. Automating the deployment and management of this observability stack through Infrastructure as Code (IaC) is not just a best practice, it's a necessity for modern DevOps teams.

This guide delves into a robust, automated solution for achieving full-stack observability on AWS EKS by integrating three industry-leading tools: Terraform for declarative infrastructure management, Datadog for comprehensive monitoring and analytics, and PagerDuty for incident response and alerting. By the end, you'll have a clear understanding of how to set up an automated, intelligent monitoring and incident management system for your EKS clusters.

Why Automate Full-Stack Observability for EKS?

Complexity of EKS: Kubernetes introduces significant complexity, with numerous components generating vast amounts of data. Manual monitoring is unsustainable.
Speed & Scale: Cloud-native environments evolve rapidly. Automation ensures observability keeps pace with infrastructure changes without manual overhead.
Consistency: IaC guarantees uniform observability configurations across development, staging, and production environments, reducing configuration drift.
Faster MTTR: Automated alerts and incident routing to PagerDuty drastically cut down Mean Time To Resolution (MTTR) for critical issues.
Cost Efficiency: Proactive identification of resource bottlenecks and underutilized services can lead to significant cost savings.

The Core Components: Terraform, Datadog, and PagerDuty

1. Terraform: Infrastructure as Code (IaC) Orchestration

Terraform, by HashiCorp, is the backbone of our automation strategy. It allows you to define and provision infrastructure using a declarative configuration language (HCL). For this solution, Terraform will manage:

The deployment of the Datadog Agent onto your EKS cluster.
The creation and management of Datadog monitors, dashboards, and integrations.
The definition of PagerDuty services, escalation policies, and users.

2. Datadog: Full-Stack Monitoring and Analytics

Datadog is a leading monitoring and analytics platform that provides end-to-end visibility across your entire stack. For EKS, Datadog offers:

Kubernetes Integration: Collects metrics, logs, and events from EKS nodes, pods, containers, and services.
APM & Distributed Tracing: Insights into application performance, service dependencies, and request flows.
Log Management: Centralized log aggregation, indexing, and analysis for rapid troubleshooting.
Network Performance Monitoring: Visualizes network traffic and connectivity within the cluster.
Synthetic Monitoring & RUM: Proactive checks on application availability and real user experience.

3. PagerDuty: Intelligent Incident Response

PagerDuty is an incident management platform that integrates seamlessly with monitoring tools like Datadog to provide reliable alerting, on-call scheduling, and incident orchestration. Its key benefits include:

Automated On-Call Scheduling: Ensures the right person is notified at the right time.
Multi-Channel Notifications: Alerts via SMS, phone calls, email, and push notifications.
Escalation Policies: Automatically escalates incidents if the primary responder doesn't acknowledge or resolve them.
Runbook Automation: Facilitates quicker incident resolution with predefined actions and playbooks.

Architectural Overview

The integration forms a robust closed-loop system:

EKS Cluster: Hosts your applications and the Datadog Agent.
Datadog Agent: Deployed as a DaemonSet on EKS, collecting metrics, logs, and traces from nodes, pods, and applications.
Datadog Platform: Ingests, processes, visualizes, and analyzes data. Datadog Monitors continuously evaluate incoming data against predefined thresholds.
Terraform: Deploys the Datadog Agent, defines Datadog Monitors/Dashboards, and configures PagerDuty Services/Escalation Policies.
Datadog-PagerDuty Integration: When a Datadog Monitor triggers an alert, it sends a notification to the configured PagerDuty service.
PagerDuty: Receives the incident, determines the correct on-call responder based on schedules and escalation policies, and notifies them.
Responder: Acknowledges, resolves, or escalates the incident, leveraging Datadog dashboards for context and diagnostics.

Prerequisites

An active AWS Account with an existing EKS Cluster.
A Datadog Account (with API and Application keys).
A PagerDuty Account (with an API key).
Terraform CLI installed (v1.0+ recommended).
AWS CLI configured with appropriate permissions.
kubectl installed and configured to connect to your EKS cluster.
Helm CLI installed.

Step-by-Step Implementation Guide

1. Configure Datadog API & Application Keys

You'll need your Datadog API Key and Application Key to allow Terraform to interact with Datadog. Find these in your Datadog account under Organization Settings > API Keys.

It's best practice to store these securely, for example, using environment variables or a secrets manager:

export DD_API_KEY="your_datadog_api_key"
export DD_APP_KEY="your_datadog_app_key"
export DD_SITE="datadoghq.com" # or eu.datadoghq.com, us3.datadoghq.com etc.
    

2. Configure PagerDuty API Key

For Terraform to manage PagerDuty resources, you need a PagerDuty API Key. Create a "Global API Key" under Integrations > API Access Keys in your PagerDuty account. Store it securely:

export PAGERDUTY_TOKEN="your_pagerduty_api_key"
    

3. Terraform Project Structure

Create a directory for your Terraform project. A typical structure might look like this:

.
├── main.tf
├── variables.tf
├── providers.tf
├── datadog_agent.tf
├── datadog_monitors.tf
├── pagerduty_incidents.tf
└── outputs.tf

4. Initialize Terraform Providers

In providers.tf, define the necessary providers: aws, helm, kubernetes, datadog, and pagerduty. Configure the Kubernetes and Helm providers to connect to your EKS cluster.

# providers.tf
provider "aws" {
  region = var.aws_region
}

data "aws_eks_cluster" "cluster" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = var.eks_cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  token                  = data.aws_eks_cluster_auth.cluster.token
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    token                  = data.aws_eks_cluster_auth.cluster.token
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
  api_url = "https://api.${var.datadog_site}"
}

provider "pagerduty" {
  token = var.pagerduty_token
}

# variables.tf (excerpt)
variable "aws_region" {
  description = "AWS region for EKS cluster"
  type        = string
  default     = "us-east-1"
}

variable "eks_cluster_name" {
  description = "Name of the existing EKS cluster"
  type        = string
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "datadog_site" {
  description = "Datadog site (e.g., datadoghq.com, eu.datadoghq.com)"
  type        = string
  default     = "datadoghq.com"
}

variable "pagerduty_token" {
  description = "PagerDuty Global API Key"
  type        = string
  sensitive   = true
}
    

5. Terraform Configuration: Deploying the Datadog Agent to EKS

The Datadog Agent is deployed as a DaemonSet to ensure it runs on every EKS node, collecting metrics, logs, and traces. We'll use the Helm provider for this.

In datadog_agent.tf:

# datadog_agent.tf
resource "kubernetes_secret" "datadog_api_key" {
  metadata {
    name = "datadog-api-key"
  }

  data = {
    "api-key" = var.datadog_api_key
    "app-key" = var.datadog_app_key
  }
  type = "Opaque"
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Consider a dedicated 'datadog' namespace
  version    = "2.33.10" # Use a specific, stable version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "datadog.site"
    value = var.datadog_site
  }

  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "clusterChecksRunner.enabled"
    value = "true"
  }

  set {
    name  = "logs.enabled"
    value = "true"
  }

  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }

  set {
    name  = "apm.enabled"
    value = "true"
  }

  set {
    name  = "processAgent.enabled"
    value = "true"
  }

  set {
    name  = "networkMonitoring.enabled"
    value = "true"
  }

  set {
    name  = "env[DD_KUBERNETES_KUBELET_TLS_VERIFY]"
    value = "false" # Adjust based on your Kubelet config, true is preferred for security
  }

  # Ensure the Datadog Agent has proper IAM permissions via IRSA
  # This typically requires creating an AWS IAM Role and a Kubernetes Service Account
  # and linking them. For brevity, this example assumes manual IRSA setup or
  # relies on node instance profiles.
  # Example: serviceAccount.create=true, serviceAccount.name=datadog-agent,
  # serviceAccount.annotations={"eks.amazonaws.com/role-arn"="arn:aws:iam::123456789012:role/datadog-eks-irsa"}

}
    

Important: For production environments, use IAM Roles for Service Accounts (IRSA) to grant your Datadog Agent the necessary AWS permissions (e.g., to collect CloudWatch metrics, SQS, etc.) without exposing AWS credentials. You would configure the serviceAccount.annotations in the Helm release values.

6. Terraform Configuration: Datadog Monitors and Dashboards

Now, let's define critical monitors and a dashboard to visualize EKS health. In datadog_monitors.tf:

# datadog_monitors.tf
resource "datadog_monitor" "eks_node_cpu_utilization" {
  name               = "[EKS] Node CPU Utilization High ({{aws_eks_cluster_name.name}})"
  type               = "metric alert"
  query              = "avg(last_5m):avg:kubernetes.node.cpu.usage{aws_eks_cluster_name:${var.eks_cluster_name}} by {host} > 80"
  message            = "CPU utilization for node {{host.name}} in EKS cluster ${var.eks_cluster_name} is over 80%. @slack-devops @pagerduty"
  monitor_thresholds {
    critical = 80
    warning  = 70
  }
  notify_no_data       = false
  new_group_delay      = 15
  renotify_interval    = 60
  evaluation_delay     = 900
  include_tags         = true
  require_full_window  = false
  force_delete         = false
  # Refer to PagerDuty integration below to send alerts
}

resource "datadog_monitor" "eks_pod_restarts" {
  name               = "[EKS] High Pod Restarts ({{kube_namespace.name}}/{{kube_app.name}})"
  type               = "log alert"
  query              = "logs(\"status:error service:kube_state_metrics EKS_CLUSTER_NAME:${var.eks_cluster_name} \"restartCount:*>0\").index(\"main\").rollup(\"count\").by(\"kube_namespace\",\"kube_app\").last(\"5m\") > 5"
  message            = "Multiple pod restarts detected in namespace {{kube_namespace.name}}, application {{kube_app.name}}. Investigate pod logs. @slack-devops @pagerduty"
  monitor_thresholds {
    critical = 5
  }
  notify_no_data       = false
  new_group_delay      = 15
  renotify_interval    = 60
  include_tags         = true
  require_full_window  = false
  force_delete         = false
}

# Example Dashboard
resource "datadog_dashboard" "eks_overview" {
  title        = "[EKS] Cluster Overview - ${var.eks_cluster_name}"
  description  = "High-level overview of EKS cluster health"
  layout_type  = "ordered"
  is_read_only = false

  widget {
    definition {
      title = "Node CPU Utilization"
      type  = "timeseries"
      request {
        q     = "avg:kubernetes.node.cpu.usage{aws_eks_cluster_name:${var.eks_cluster_name}} by {host}"
        display_type = "area"
      }
    }
  }

  widget {
    definition {
      title = "Node Memory Utilization"
      type  = "timeseries"
      request {
        q     = "avg:kubernetes.node.memory.usage{aws_eks_cluster_name:${var.eks_cluster_name}} by {host}"
        display_type = "area"
      }
    }
  }

  widget {
    definition {
      title = "Pod Count"
      type  = "query_value"
      request {
        q = "sum:kubernetes.pod.running{aws_eks_cluster_name:${var.eks_cluster_name}}"
      }
    }
  }

  widget {
    definition {
      title = "Kubernetes Events"
      type  = "event_stream"
      query = "tags:aws_eks_cluster_name:${var.eks_cluster_name} kubernetes"
    }
  }

  widget {
    definition {
      title = "Top Pods by CPU"
      type  = "toplist"
      request {
        q = "top(avg:kubernetes.cpu.usage{aws_eks_cluster_name:${var.eks_cluster_name}} by {pod_name}, 10, 'sum', 'desc')"
      }
    }
  }

  # Add more widgets for logs, network, APM, etc.
}
    

7. Terraform Configuration: PagerDuty Services and Escalation Policies

To route incidents effectively, define a PagerDuty service and an escalation policy. In pagerduty_incidents.tf:

# pagerduty_incidents.tf
# First, define an escalation policy. This assumes you have users/teams already configured in PagerDuty.
# For simplicity, we create a basic policy and a dummy user. In a real scenario, use existing users/teams.
resource "pagerduty_user" "devops_engineer" {
  name  = "DevOps Engineer"
  email = "devops-engineer@example.com"
  # Set role, job_title as needed
}

resource "pagerduty_escalation_policy" "eks_observability_policy" {
  name      = "EKS Observability Escalation Policy"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 15
    target {
      type = "user"
      id   = pagerduty_user.devops_engineer.id
    }
    # You can add more targets (users, schedules) here for different escalation levels
  }
}

# Then, create a service that uses this policy
resource "pagerduty_service" "eks_observability_service" {
  name                     = "EKS Cluster Observability - ${var.eks_cluster_name}"
  auto_resolve_timeout     = "14400" # 4 hours
  acknowledgement_timeout  = "600"   # 10 minutes
  escalation_policy        = pagerduty_escalation_policy.eks_observability_policy.id
  alert_creation           = "create_alerts_and_incidents"
  status                   = "active"

  incident_urgency_rule {
    type    = "constant"
    urgency = "high" # or "low"
  }
}

# Now, create a Datadog integration for this PagerDuty service
resource "pagerduty_extension" "datadog_integration" {
  name           = "Datadog to EKS Observability"
  endpoint_url   = "https://app.datadoghq.com/webhook/pagerduty" # This is the Datadog webhook URL for PagerDuty
  extension_type = "datadog_extension"
  service        = pagerduty_service.eks_observability_service.id
}

# Output the integration key for Datadog
output "pagerduty_datadog_integration_key" {
  value       = pagerduty_service.eks_observability_service.integration[0].integration_key
  description = "PagerDuty Integration Key for Datadog"
  sensitive   = true
}
    

8. Connecting Datadog Monitors to PagerDuty

With the PagerDuty service and integration defined, update your Datadog monitors to send alerts to PagerDuty. You'll reference the PagerDuty integration that Terraform just created. The message field in your Datadog monitors should include @pagerduty.

To fully automate the connection, you can define the PagerDuty integration within Datadog using Terraform's datadog_integration_pagerduty resource. However, using @pagerduty in the monitor message is the most common and flexible way.

Ensure your Datadog account has the PagerDuty integration set up via the UI Integrations > PagerDuty and provides the integration key obtained from the pagerduty_service output.

# Add this to your datadog_monitors.tf or a new file if preferred
resource "datadog_integration_pagerduty" "main" {
  api_key = var.pagerduty_token # PagerDuty API Key is used by Datadog to verify
  # If using the `pagerduty_service` to get the integration key,
  # you would typically manually add this key to Datadog's PagerDuty integration settings
  # or you could try to pass it via a Datadog monitor's `options.notify_audit` if there was a direct way.
  # For full automation, ensure Datadog has the PagerDuty integration configured (often via UI initially).
  # The `@pagerduty` tag in monitor messages then uses this pre-configured integration.
}
    

The @pagerduty tag within the message of a datadog_monitor resource will automatically trigger a PagerDuty incident once the integration is configured in the Datadog UI with the correct PagerDuty API key (or if you use the datadog_integration_pagerduty resource with an account-wide PagerDuty API key).

9. Deploying the Observability Stack

Navigate to your Terraform project directory and run the standard commands:

terraform init
terraform plan -var "eks_cluster_name=your-eks-cluster-name" -var "datadog_api_key=$DD_API_KEY" -var "datadog_app_key=$DD_APP_KEY" -var "pagerduty_token=$PAGERDUTY_TOKEN"
terraform apply -var "eks_cluster_name=your-eks-cluster-name" -var "datadog_api_key=$DD_API_KEY" -var "datadog_app_key=$DD_APP_KEY" -var "pagerduty_token=$PAGERDUTY_TOKEN"
    

Remember to replace your-eks-cluster-name with your actual EKS cluster name.

Validation and Testing

Verify Datadog Agent: Check your EKS cluster with kubectl get pods -n default (or your chosen namespace) to ensure Datadog Agent pods are running.
Datadog UI: Log into Datadog. You should see host metrics, container maps, and Kubernetes dashboards populating. Verify your custom dashboards and monitors exist.
PagerDuty UI: Confirm the new service, escalation policy, and integration are visible.
Simulate an Alert: Intentionally trigger a monitor (e.g., scale down pods, consume CPU on a node). Verify an incident is created in PagerDuty and notifications are sent.

Best Practices and Advanced Considerations

Granular Permissions (IRSA): Leverage IAM Roles for Service Accounts (IRSA) for the Datadog Agent to grant least-privilege AWS access without directly exposing credentials.
Resource Tagging: Implement consistent AWS resource tagging and use these tags in Datadog for powerful filtering and dashboarding.
Custom Metrics & Tracing: Instrument your applications to send custom metrics and distributed traces to Datadog for deeper application-level insights.
Log Enrichment: Use Datadog processors to enrich logs with valuable context (e.g., Kubernetes metadata, trace IDs) for faster troubleshooting.
Cost Optimization: Regularly review Datadog usage, especially log ingestion and custom metrics, to optimize costs.
Monitoring-as-Code (MaC): Treat your Datadog monitors and dashboards as code within your version control system, just like your infrastructure.
Synthetic Monitoring: Implement Datadog Synthetics to proactively test critical endpoints and user journeys from an outside-in perspective.
Automated Remediation: Explore integrating PagerDuty with automation tools (e.g., AWS Lambda, Ansible) to trigger self-healing actions for common issues.

Troubleshooting Common Issues

Datadog Agent Not Reporting:
- Check Helm release status: helm status datadog.
- Inspect Datadog Agent pod logs: kubectl logs -f <datadog-agent-pod-name>. Look for API key errors or connection issues.
- Verify Kubernetes permissions for the Agent Service Account.
Datadog Monitors Not Triggering/Sending to PagerDuty:
- Double-check monitor query and thresholds in Datadog UI.
- Ensure @pagerduty is correctly specified in the monitor message.
- Verify Datadog's PagerDuty integration is configured with the correct PagerDuty API key (or integration key if specified).
- Check PagerDuty incident logs for any incoming event errors.
Terraform Apply Errors:
- Provider Authentication: Ensure AWS, Datadog, and PagerDuty API keys/tokens are correctly set as environment variables or passed as Terraform variables.
- Kubernetes Connectivity: Confirm kubectl can connect to your EKS cluster and the Terraform Kubernetes/Helm providers are configured correctly.
- Resource Conflicts: If you're managing resources both manually and with Terraform, you might encounter conflicts. Import existing resources into Terraform state or clean them up.

Conclusion

Automating full-stack observability for AWS EKS using Terraform, Datadog, and PagerDuty provides a powerful, scalable, and resilient solution for managing modern cloud-native applications. By codifying your monitoring, alerting, and incident response, you empower your DevOps teams to rapidly deploy, observe, and maintain highly available systems with confidence.

Embrace this automated approach to shift observability left, reduce manual toil, and ensure your EKS clusters and the applications running on them are performing optimally, 24/7.

CloudStack YAML