Automating AWS EKS Observability: A Comprehensive Guide with Terraform, Datadog, and PagerDuty

In the dynamic landscape of cloud-native applications, managing Kubernetes clusters efficiently is paramount. AWS Elastic Kubernetes Service (EKS) provides a robust platform, but ensuring its continuous health and performance requires sophisticated observability. This guide delves into automating AWS EKS observability using a powerful trio: Terraform for infrastructure-as-code, Datadog for comprehensive monitoring and alerting, and PagerDuty for streamlined incident response.

Architecture Pro-Tip: Embrace GitOps for Observability

Treat your observability configurations (Datadog monitors, dashboards, PagerDuty services, escalation policies) as code. Store them in a Git repository alongside your infrastructure and application code. This GitOps approach ensures a single source of truth, enables version control, facilitates rollbacks, and promotes collaborative development, leading to more resilient and auditable observability pipelines.

Why Automate EKS Observability?

Manual configuration of monitoring and alerting systems for complex Kubernetes environments is prone to errors, inconsistency, and can't scale with your infrastructure. Automating this process offers significant advantages:

Consistency and Reliability: Terraform ensures that your observability stack is deployed uniformly across environments, reducing misconfigurations.
Speed and Efficiency: Rapidly provision and update monitoring configurations as your EKS clusters evolve.
Auditability and Version Control: All changes are tracked in Git, providing a clear history and easy rollbacks.
Reduced Operational Overhead: Free up your DevOps and SRE teams to focus on innovation rather than repetitive setup tasks.
Scalability: Easily extend observability to new clusters or services without manual intervention.

Core Components Overview

AWS EKS: The Foundation

AWS EKS is a managed service that makes it easy to run Kubernetes on AWS without needing to install, operate, and maintain your own Kubernetes control plane. It integrates seamlessly with other AWS services, providing a robust and scalable environment for containerized applications.

Datadog: Comprehensive Monitoring and Analytics

Datadog is a leading monitoring and analytics platform for cloud-scale applications. It provides end-to-end visibility across infrastructure, applications, and logs. For EKS, Datadog collects metrics, traces, and logs from your Kubernetes clusters, nodes, pods, and containers, offering dashboards, alerts, and AI-driven insights.

PagerDuty: Incident Management and On-Call Automation

PagerDuty is an incident management platform that helps teams detect, triage, and resolve incidents faster. By integrating with Datadog, PagerDuty ensures that critical alerts from your EKS environment are routed to the right on-call team members, escalating them according to predefined policies until acknowledged and resolved.

Terraform: Infrastructure as Code (IaC)

Terraform by HashiCorp is an open-source IaC tool that allows you to define and provision infrastructure using a declarative configuration language. It supports a vast ecosystem of providers, including AWS, Datadog, and PagerDuty, making it the ideal choice for automating the entire observability stack.

Prerequisites

Before you begin, ensure you have the following:

An AWS Account with administrative privileges.
A Datadog Account with API and Application keys.
A PagerDuty Account with an API token.
Terraform CLI (v1.0+) installed.
Kubectl CLI installed and configured to connect to your EKS cluster.
A basic understanding of AWS EKS, Datadog, PagerDuty, and Terraform.

Step 1: Setting up Terraform for AWS EKS and IAM Roles

While this guide assumes you have an EKS cluster, we'll quickly cover the necessary IAM setup for Datadog. The Datadog Agent requires specific permissions to collect metrics from your EKS cluster and AWS services.

EKS Cluster IAM for Datadog Agent

You'll need an IAM Role with a policy that allows the Datadog Agent to collect metrics from AWS services and describe EKS resources. This role will be associated with a Kubernetes Service Account which the Datadog Agent will use.

Create a file named main.tf:

resource "aws_iam_role" "datadog_agent_role" {
  name_prefix        = "datadog-agent-eks-"
  assume_role_policy = jsonencode({
    Version   = "2012-10-17"
    Statement = [
      {
        Effect    = "Allow"
        Principal = {
          Federated = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${replace(data.aws_eks_cluster.current.identity[0].oidc[0].issuer, "https://", "")}"
        }
        Action    = "sts:AssumeRoleWithWebIdentity"
        Condition = {
          StringEquals = {
            "${replace(data.aws_eks_cluster.current.identity[0].oidc[0].issuer, "https://", "")}:sub" : "system:serviceaccount:datadog:datadog-agent"
          }
        }
      }
    ]
  })
}

resource "aws_iam_policy" "datadog_agent_policy" {
  name_prefix = "datadog-agent-eks-policy-"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = [
          "ec2:DescribeInstances",
          "ec2:DescribeTags",
          "ec2:DescribeVolumes",
          "ec2:DescribeVpcs",
          "eks:DescribeCluster",
          "autoscaling:DescribeAutoScalingGroups",
          "autoscaling:DescribeLaunchConfigurations",
          "logs:DescribeLogGroups",
          "logs:DescribeLogStreams",
          "logs:FilterLogEvents",
          "logs:GetLogEvents",
          "rds:ListTagsForResource",
          "rds:DescribeDBInstances",
          "ecs:List*",
          "ecs:Describe*",
          "elasticloadbalancing:Describe*",
          "sqs:ListQueues",
          "sqs:GetQueueAttributes",
          "sns:ListTopics",
          "sns:GetTopicAttributes",
          "s3:ListAllMyBuckets",
          "s3:GetBucketLocation",
          "s3:GetBucketTagging",
          "s3:GetBucketLogging",
          "s3:GetBucketVersioning",
          "s3:GetBucketWebsite",
          "s3:GetEncryptionConfiguration",
          "lambda:ListFunctions",
          "lambda:GetFunctionConfiguration",
          "apigateway:GET",
          "tag:GetResources",
          "tag:GetTagKeys",
          "tag:GetTagValues"
        ]
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "datadog_agent_attach" {
  role       = aws_iam_role.datadog_agent_role.name
  policy_arn = aws_iam_policy.datadog_agent_policy.arn
}

data "aws_caller_identity" "current" {}
data "aws_eks_cluster" "current" {
  name = "your-eks-cluster-name" # Replace with your EKS cluster name
}
    

Explanation: This Terraform code sets up an IAM role that the Datadog Agent can assume via OIDC (OpenID Connect) provider associated with your EKS cluster. Replace "your-eks-cluster-name" with the actual name of your EKS cluster.

Step 2: Integrating Datadog Monitors and Dashboards with Terraform

Now we'll use Terraform to provision Datadog resources like monitors and dashboards. First, configure the Datadog provider.

Datadog Provider Configuration

Add the following to your main.tf or a new providers.tf file:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
    pagerduty = {
      source  = "PagerDuty/pagerduty"
      version = "~> 2.0"
    }
  }
}

provider "aws" {
  region = "us-east-1" # Specify your AWS region
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
  api_url = "https://api.datadoghq.com/" # Or your Datadog site URL
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}
    

Important: Pass datadog_api_key and datadog_app_key securely, for example, via environment variables (TF_VAR_datadog_api_key) or a CI/CD secret manager, not directly in your code.

Deploying Datadog Agent to EKS

While the Datadog Agent itself is typically deployed to EKS via Helm charts or Kubernetes manifests, Terraform can help manage the configuration. The Helm chart usually takes care of creating the Kubernetes Service Account and associating it with the IAM Role you created:

# Example for deploying Datadog Agent using Helm, potentially via Terraform's Helm provider
# This is an illustrative example; actual implementation depends on your setup.

# module "datadog_agent" {
#   source = "terraform-aws-modules/eks/aws//modules/helm_release"
#   version = "~> 19.0" # Use a compatible version

#   name       = "datadog"
#   repository = "https://helm.datadoghq.com"
#   chart      = "datadog"
#   namespace  = "datadog"
#   create_namespace = true

#   values = [
#     yamlencode({
#       datadog = {
#         apiKey = var.datadog_api_key
#         appKey = var.datadog_app_key
#         site   = "datadoghq.com" # or "datadoghq.eu"
#         kubelet = {
#           host = {
#             detectFrom = "env"
#           }
#         }
#         tags = [
#           "env:production",
#           "cluster_name:${data.aws_eks_cluster.current.name}"
#         ]
#       }
#       agents = {
#         podLabelsAsTags = {
#           "app.kubernetes.io/name" = "kube_app_name"
#         }
#       }
#       clusterAgent = {
#         enabled = true
#         metrics = {
#           enabled = true
#         }
#         processAgent = {
#           enabled = true
#         }
#         admissionController = {
#           enabled = true
#         }
#         rbac = {
#           create = true
#           serviceAccount = {
#             create = true
#             name = "datadog-agent" # Must match service account name in IAM role
#             annotations = {
#               "eks.amazonaws.com/role-arn" = aws_iam_role.datadog_agent_role.arn
#             }
#           }
#         }
#       }
#       # ... other Datadog agent configurations
#     })
#   ]
# }
    

Note: The Helm module configuration is commented out as deploying Helm charts with Terraform can vary. The crucial part here is the eks.amazonaws.com/role-arn annotation on the Service Account, linking it to the IAM role.

Terraform for Datadog Monitors

Let's create a Datadog monitor to alert on high EKS node CPU utilization.

resource "datadog_monitor" "eks_node_cpu_utilization" {
  name                = "EKS Node CPU Utilization High on {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:system.cpu.idle{kube_cluster_name:${data.aws_eks_cluster.current.name}} by {host} < 10" # Alert if idle < 10% (i.e., usage > 90%)
  message             = "EKS node {{host.name}} CPU utilization is high ({{value}}% idle)! PagerDuty: @pagerduty-your-service-integration-key"
  tags                = ["eks", "cpu", "alert"]
  renotify_interval   = 60 # minutes
  notify_no_data      = false
  new_group_delay     = 60
  no_data_timeframe   = 30 # minutes
  include_tags        = true
  require_full_window = false
  timeout_h           = 0

  threshold_windows {
    recovery_window = "last_15m"
  }

  monitor_thresholds {
    warning  = 20 # Warning if idle < 20%
    critical = 10 # Critical if idle < 10%
  }
}
    

Note: In the message field, @pagerduty-your-service-integration-key is a placeholder. You'd typically configure Datadog's PagerDuty integration in the Datadog UI first, then refer to it by its name or a specific key if Datadog supports that directly in the message syntax. Alternatively, you can use a notification group or webhook that forwards to PagerDuty.

Terraform for Datadog Dashboards

Automate the creation of informative EKS dashboards.

resource "datadog_dashboard" "eks_overview_dashboard" {
  title        = "EKS Cluster Overview - ${data.aws_eks_cluster.current.name}"
  description  = "A dashboard showing key metrics for EKS cluster: ${data.aws_eks_cluster.current.name}"
  layout_type  = "ordered"
  is_read_only = false
  widget {
    alert_value_definition {
      alert_id = datadog_monitor.eks_node_cpu_utilization.id
      title    = "EKS Node CPU Utilization Alert"
      live_span = "1h"
    }
  }

  widget {
    timeseries_definition {
      title     = "EKS Node CPU Utilization"
      live_span = "1h"
      request {
        q        = "avg:system.cpu.idle{kube_cluster_name:${data.aws_eks_cluster.current.name}} by {host}"
        display_type = "area"
        style {
          palette = "blue"
          type    = "solid"
          width   = "normal"
        }
      }
    }
  }

  # Add more widgets for Memory, Disk, Network, Pods, etc.
}
    

Step 3: Integrating PagerDuty for Incident Response

Now, let's configure PagerDuty services and escalation policies using Terraform, and ensure Datadog can trigger incidents.

PagerDuty Provider Configuration

Add the PagerDuty provider to your providers.tf or main.tf:

provider "pagerduty" {
  token = var.pagerduty_token
}

variable "pagerduty_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}
    

Important: Provide pagerduty_token securely.

Terraform for PagerDuty Services and Escalation Policies

First, define an escalation policy, then a service that uses it.

resource "pagerduty_user" "oncall_engineer" {
  name  = "On-Call Engineer"
  email = "oncall@example.com" # Replace with a real email
  role  = "user"
}

resource "pagerduty_escalation_policy" "eks_critical_policy" {
  name      = "EKS Critical Escalation Policy"
  num_loops = 2
  rule {
    delay_in_minutes = 5
    target {
      type = "user"
      id   = pagerduty_user.oncall_engineer.id
    }
  }
}

resource "pagerduty_service" "eks_observability_service" {
  name                    = "EKS Observability Service - ${data.aws_eks_cluster.current.name}"
  escalation_policy       = pagerduty_escalation_policy.eks_critical_policy.id
  auto_resolve_timeout_s  = 1800 # Auto-resolve after 30 minutes if not acknowledged
  acknowledgement_timeout_s = 600 # Auto-escalate after 10 minutes if not acknowledged

  incident_urgency_rule {
    type    = "constant"
    urgency = "high"
  }
}

resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog Integration"
  service_id  = pagerduty_service.eks_observability_service.id
  type        = "generic_events_api_inbound_integration"
}
    

Explanation: This creates a PagerDuty user, an escalation policy that targets that user, and a service that uses this policy. Finally, it sets up a generic events API integration for Datadog. The integration_key from pagerduty_service_integration.datadog_integration.integration_key is what you'd use in Datadog to send alerts.

Connecting Datadog to PagerDuty

In Datadog, go to Integrations -> Integrations, search for PagerDuty, and configure it. When adding a new PagerDuty integration, you will provide the integration_key from the pagerduty_service_integration resource created above. Once configured, you can then specify @pagerduty-YOUR_PAGERDUTY_SERVICE_NAME (e.g., @pagerduty-EKS Observability Service - my-cluster) in your Datadog monitor messages to trigger PagerDuty incidents.

Implementing the Solution: Ready-to-use Configuration

Here's a consolidated example of the Terraform configuration to bring it all together. Remember to replace placeholders and manage sensitive variables securely.

# main.tf
# Providers configuration
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
    pagerduty = {
      source  = "PagerDuty/pagerduty"
      version = "~> 2.0"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
  api_url = "https://api.datadoghq.com/"
}

provider "pagerduty" {
  token = var.pagerduty_token
}

# Data sources for current AWS account and EKS cluster
data "aws_caller_identity" "current" {}
data "aws_eks_cluster" "current" {
  name = var.eks_cluster_name
}

# AWS IAM Role for Datadog Agent
resource "aws_iam_role" "datadog_agent_role" {
  name_prefix        = "datadog-agent-eks-${var.eks_cluster_name}-"
  assume_role_policy = jsonencode({
    Version   = "2012-10-17"
    Statement = [
      {
        Effect    = "Allow"
        Principal = {
          Federated = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${replace(data.aws_eks_cluster.current.identity[0].oidc[0].issuer, "https://", "")}"
        }
        Action    = "sts:AssumeRoleWithWebIdentity"
        Condition = {
          StringEquals = {
            "${replace(data.aws_eks_cluster.current.identity[0].oidc[0].issuer, "https://", "")}:sub" : "system:serviceaccount:datadog:datadog-agent"
          }
        }
      }
    ]
  })
}

resource "aws_iam_policy" "datadog_agent_policy" {
  name_prefix = "datadog-agent-eks-policy-${var.eks_cluster_name}-"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = [
          "ec2:DescribeInstances", "ec2:DescribeTags", "eks:DescribeCluster",
          "autoscaling:DescribeAutoScalingGroups", "logs:FilterLogEvents",
          "rds:DescribeDBInstances", "ecs:List*", "elasticloadbalancing:Describe*",
          "sqs:ListQueues", "s3:ListAllMyBuckets", "lambda:ListFunctions",
          "apigateway:GET", "tag:GetResources"
        ]
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "datadog_agent_attach" {
  role       = aws_iam_role.datadog_agent_role.name
  policy_arn = aws_iam_policy.datadog_agent_policy.arn
}

# PagerDuty Setup
resource "pagerduty_user" "oncall_engineer" {
  name  = "Primary On-Call"
  email = "primary.oncall@example.com"
  role  = "user"
}

resource "pagerduty_escalation_policy" "eks_critical_policy" {
  name      = "${var.eks_cluster_name} - EKS Critical Policy"
  num_loops = 2
  rule {
    delay_in_minutes = 5
    target {
      type = "user"
      id   = pagerduty_user.oncall_engineer.id
    }
  }
}

resource "pagerduty_service" "eks_observability_service" {
  name                    = "${var.eks_cluster_name} EKS Observability Service"
  escalation_policy       = pagerduty_escalation_policy.eks_critical_policy.id
  auto_resolve_timeout_s  = 1800
  acknowledgement_timeout_s = 600
  incident_urgency_rule {
    type    = "constant"
    urgency = "high"
  }
}

resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog Integration for EKS"
  service_id  = pagerduty_service.eks_observability_service.id
  type        = "generic_events_api_inbound_integration"
}

# Datadog Monitor for EKS Node CPU
resource "datadog_monitor" "eks_node_cpu_utilization" {
  name                = "[EKS - ${var.eks_cluster_name}] High Node CPU Utilization on {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:system.cpu.idle{kube_cluster_name:${var.eks_cluster_name}} by {host} < 10"
  message             = "EKS node {{host.name}} CPU utilization is high ({{value}}% idle)! PagerDuty: @pagerduty-${pagerduty_service.eks_observability_service.name}"
  tags                = ["eks", "cpu", "alert", "cluster:${var.eks_cluster_name}"]
  renotify_interval   = 60
  notify_no_data      = false
  new_group_delay     = 60
  no_data_timeframe   = 30
  include_tags        = true
  require_full_window = false
  timeout_h           = 0

  monitor_thresholds {
    warning  = 20
    critical = 10
  }
}

# Datadog Dashboard for EKS Overview
resource "datadog_dashboard" "eks_overview_dashboard" {
  title        = "EKS Cluster Overview - ${var.eks_cluster_name}"
  description  = "Key metrics for EKS cluster: ${var.eks_cluster_name}"
  layout_type  = "ordered"
  is_read_only = false
  widget {
    alert_value_definition {
      alert_id = datadog_monitor.eks_node_cpu_utilization.id
      title    = "EKS Node CPU Utilization Alert Status"
      live_span = "1h"
    }
  }
  widget {
    timeseries_definition {
      title     = "EKS Node CPU Utilization by Host"
      live_span = "1h"
      request {
        q        = "avg:system.cpu.idle{kube_cluster_name:${var.eks_cluster_name}} by {host}"
        display_type = "line"
      }
    }
  }
  # More widgets can be added here
}

# variables.tf
variable "aws_region" {
  description = "AWS region for the EKS cluster."
  type        = string
  default     = "us-east-1"
}

variable "eks_cluster_name" {
  description = "The name of your AWS EKS cluster."
  type        = string
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}
    

To deploy this configuration:

Save the code above into main.tf and variables.tf files.
Initialize Terraform: terraform init
Plan the changes: terraform plan -var="eks_cluster_name=your-cluster-name" (provide other variables via environment variables or a .tfvars file).
Apply the configuration: terraform apply -var="eks_cluster_name=your-cluster-name"

Testing and Validation

After applying the Terraform configuration:

Verify IAM Role: Confirm the datadog_agent_role exists in AWS IAM.
Check Datadog Resources: Log into your Datadog account. You should see the EKS overview dashboard and the node CPU utilization monitor.
Confirm PagerDuty Setup: Log into PagerDuty. Verify the new user, escalation policy, and EKS observability service. Note the integration key for Datadog.
Trigger a Test Alert: Manually trigger a test alert from Datadog or simulate high CPU usage on an EKS node to ensure PagerDuty incidents are created and escalated correctly.

Best Practices for EKS Observability

Granular Monitoring: Beyond nodes, monitor pods, containers, deployments, services, and Kubernetes events.
Log Management: Centralize EKS logs (control plane and application logs) in Datadog for correlation with metrics and traces.
Distributed Tracing: Instrument your applications to capture traces, providing end-to-end visibility into request flows.
Synthetic Monitoring: Proactively test your application's availability and performance from an end-user perspective.
Alert Fatigue Prevention: Tune your monitors and escalation policies carefully. Use composite monitors, machine learning-driven alerts, and deduplication in PagerDuty.
Cost Optimization: Regularly review Datadog and PagerDuty usage. Optimize log ingestion, metric cardinality, and monitor frequency.
Security Monitoring: Integrate security tools and logs into Datadog for a unified security and operational view.

Troubleshooting Common Issues

Terraform Apply Fails: Double-check API keys, region, cluster names, and IAM permissions. Ensure your local AWS credentials have permissions to create IAM roles.
Datadog Agent Not Reporting:
- Verify the Datadog Agent pods are running in your EKS cluster: kubectl get pods -n datadog.
- Check agent logs: kubectl logs <datadog-agent-pod-name> -n datadog. Look for errors related to API keys, network connectivity, or permissions.
- Ensure the Kubernetes Service Account annotations for the IAM role are correct.
Datadog Alerts Not Triggering PagerDuty:
- Confirm the PagerDuty integration is correctly configured in Datadog (Integrations > PagerDuty).
- Ensure the @pagerduty-YOUR_SERVICE_NAME syntax in your Datadog monitor message matches the name used in your Datadog PagerDuty integration.
- Check PagerDuty's incident logs for any received events.

Conclusion and Next Steps

Automating AWS EKS observability with Terraform, Datadog, and PagerDuty significantly enhances your operational posture, ensuring consistent monitoring, rapid incident response, and a scalable foundation for your cloud-native applications. By treating your observability configuration as code, you gain the benefits of version control, collaboration, and reliability.

Next Steps:

Extend your Terraform configurations to include more specific Datadog monitors for application-level metrics, Kubernetes events, and resource quotas.
Implement Datadog log collection and create log-based monitors.
Explore Datadog's APM (Application Performance Monitoring) and Distributed Tracing for deeper application insights.
Refine PagerDuty escalation policies to include multiple teams and on-call schedules.
Integrate this automation into your CI/CD pipeline for true GitOps-driven observability.

CloudStack YAML

Automating AWS EKS Observability with Terraform, Datadog, and PagerDuty