Terraform for Unified AWS EKS Observability with Datadog and PagerDuty

Architecture Pro-Tip: Layered Observability Strategy

For truly unified EKS observability, aim for a layered approach:
1. Infrastructure Metrics: AWS CloudWatch (via Datadog integration) for underlying EC2 instances, EBS, VPC.
2. Kubernetes Metrics: Datadog Agent for EKS control plane and node health, pod metrics, events.
3. Application Metrics: APM (traces), custom metrics, and logs from your EKS-deployed applications.
4. Synthetics & RUM: External checks for user experience.
Consolidate all these data sources into Datadog, then use PagerDuty for actionable, high-priority alerts. Terraform ensures this entire stack is consistently deployed.

Terraform for Unified AWS EKS Observability with Datadog and PagerDuty

In the dynamic world of cloud-native applications, managing and monitoring Amazon Elastic Kubernetes Service (AWS EKS) environments can be incredibly complex. Ensuring high availability, performance, and rapid incident response demands a robust, unified observability strategy. This technical guide will walk you through leveraging Terraform to seamlessly integrate Datadog for comprehensive monitoring and PagerDuty for efficient incident management within your AWS EKS clusters, establishing a powerful, automated observability pipeline.

Why Unified EKS Observability is Critical

EKS clusters comprise numerous moving parts: nodes, pods, services, deployments, ingress controllers, and underlying AWS infrastructure. Without a centralized view, troubleshooting becomes a nightmare, leading to increased Mean Time To Resolution (MTTR) and potential service degradation. A unified approach brings several advantages:

  • Single Pane of Glass: Correlate metrics, logs, and traces across your entire EKS stack.
  • Faster Root Cause Analysis: Quickly pinpoint issues from infrastructure to application layer.
  • Proactive Incident Response: Leverage intelligent alerting to detect problems before they impact users.
  • Reduced Alert Fatigue: Consolidate, enrich, and route alerts effectively to the right teams.
  • Automated Scalability & Efficiency: Infrastructure as Code (IaC) ensures consistent, repeatable deployments.

Key Components of Our Observability Stack

AWS EKS: The Foundation

Amazon Elastic Kubernetes Service (EKS) provides a managed Kubernetes control plane, simplifying the deployment and management of containerized applications at scale. Our focus here is to equip EKS with robust monitoring capabilities.

Datadog: Comprehensive Monitoring and Analytics

Datadog is a leading monitoring and analytics platform for cloud-scale applications. It collects metrics, logs, and traces from your EKS cluster, underlying AWS resources, and applications, providing real-time visibility through customizable dashboards, powerful querying, and sophisticated alerting mechanisms. Key Datadog features we'll utilize include:

  • AWS Integration: Pulls CloudWatch metrics and metadata for EC2, ELB, RDS, etc.
  • Datadog Agent: Deployed as a DaemonSet on EKS, collecting host, container, and application metrics, logs, and traces.
  • EKS Integration: Provides deeper insights into Kubernetes objects and events.
  • Monitors & Alerts: Define thresholds and anomaly detection for critical metrics.

PagerDuty: Incident Management and On-Call Automation

PagerDuty acts as the central hub for incident response. When Datadog detects a critical issue, it triggers an incident in PagerDuty, which then intelligently routes the alert to the right on-call team member based on schedules and escalation policies. This minimizes response times and ensures critical issues are never missed. We'll use:

  • Services: Represent components or applications that might generate incidents.
  • Escalation Policies: Define who gets notified and in what order.
  • Integrations: Connects with monitoring tools like Datadog to receive alerts.

Terraform: Infrastructure as Code for Automation

Terraform, by HashiCorp, allows you to define and provision your infrastructure using a declarative configuration language. By codifying our observability stack, we gain:

  • Reproducibility: Easily replicate the setup across different environments (dev, staging, prod).
  • Version Control: Track changes to your observability configuration like any other code.
  • Reduced Manual Error: Eliminate human error associated with manual configuration.
  • Auditability: See who changed what and when.

Prerequisites

  • AWS Account: With appropriate IAM permissions to manage EKS, CloudWatch, and create IAM roles/policies.
  • Datadog Account: Access to your Datadog organization and an API key and Application key.
  • PagerDuty Account: Access to your PagerDuty organization and an API key.
  • Terraform CLI: Version 1.0+ installed on your local machine.
  • kubectl CLI: Configured to connect to your AWS EKS cluster (for verifying Datadog Agent deployment). Ensure your kubeconfig is correctly set up.
  • Existing AWS EKS Cluster: This guide assumes you have an operational EKS cluster. If not, you can create one using Terraform as well, but that's beyond the scope of this particular guide.

Terraform Configuration: Step-by-Step Guide

We'll structure our Terraform project to manage different aspects of the observability stack.

1. Project Structure

Create a new directory for your Terraform project: 

mkdir terraform-eks-observability
cd terraform-eks-observability

2. Provider Configuration & Variables

First, define your Terraform providers for AWS, Datadog, and PagerDuty, along with necessary variables for API keys and other configurations.

3. Datadog AWS & EKS Integration

Configure Datadog to integrate with your AWS account to pull CloudWatch metrics and metadata. We'll also deploy the Datadog Agent to your EKS cluster and set up Datadog's PagerDuty integration.

4. PagerDuty Service & Escalation Policy

Create a PagerDuty service that will receive incidents, and define an escalation policy to ensure the right team members are notified.

5. Connecting Datadog Monitors to PagerDuty

Finally, define Datadog monitors that trigger alerts when certain conditions are met and configure them to send notifications to your PagerDuty service.

Copy & Paste Terraform Configuration

Place the following code into your `terraform-eks-observability` directory, typically in files like `main.tf`, `variables.tf`, `datadog.tf`, and `pagerduty.tf`.
Remember to replace placeholder values like `your-aws-region`, `your-eks-cluster-name`, `your-datadog-api-key`, etc., with your actual credentials and environment specifics. For sensitive keys, consider using Terraform Cloud, AWS Secrets Manager, or environment variables.

# main.tf


terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
    pagerduty = {
      source  = "PagerDuty/pagerduty"
      version = "~> 1.15"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_key
}

# Configure the Kubernetes provider to connect to the EKS cluster
# This assumes your local kubeconfig is already set up to access the EKS cluster
data "aws_eks_cluster" "eks_cluster" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "eks_cluster_auth" {
  name = var.eks_cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.eks_cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.eks_cluster_auth.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.eks_cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.eks_cluster_auth.token
  }
}

# variables.tf


variable "aws_region" {
  description = "The AWS region where your EKS cluster resides."
  type        = string
  default     = "us-east-1" # Change this to your region
}

variable "eks_cluster_name" {
  description = "The name of your existing EKS cluster."
  type        = string
  default     = "my-observability-eks-cluster" # Change this to your EKS cluster name
}

variable "datadog_api_key" {
  description = "Your Datadog API Key."
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Your Datadog Application Key."
  type        = string
  sensitive   = true
}

variable "pagerduty_api_key" {
  description = "Your PagerDuty API Key (e.g., v2 API token)."
  type        = string
  sensitive   = true
}

variable "pagerduty_escalation_policy_id" {
  description = "The ID of the PagerDuty escalation policy to use for the service. You can obtain this from the PagerDuty UI or a 'data' source."
  type        = string
  # Example to fetch an existing policy by name:
  # data "pagerduty_escalation_policy" "default" { name = "Default" }
  # default = data.pagerduty_escalation_policy.default.id
  default     = "P123456" # <<< REPLACE with your actual PagerDuty Escalation Policy ID >>>
}

variable "pagerduty_user_id" {
  description = "The ID of a PagerDuty user to be associated with an escalation policy (if creating a new one). You can obtain this from the PagerDuty UI or a 'data' source."
  type        = string
  # Example to fetch an existing user by email:
  # data "pagerduty_user" "admin" { email = "admin@example.com" }
  # default = data.pagerduty_user.admin.id
  default     = "U123456" # <<< REPLACE with your actual PagerDuty User ID >>>
}

# datadog.tf


# Data source for AWS account ID
data "aws_caller_identity" "current" {}

# Ensure the Datadog Agent namespace exists
resource "kubernetes_namespace" "datadog_namespace" {
  metadata {
    name = "datadog"
  }
}

# --- Datadog AWS Integration ---
resource "aws_iam_role" "datadog_integration" {
  name = "DatadogIntegrationRole-${var.eks_cluster_name}"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          AWS = "464622532012" # Datadog AWS integration account ID
        }
        Condition = {
          StringEquals = {
            "sts:ExternalId" = var.datadog_api_key # Use your Datadog API Key as External ID
          }
        }
      },
    ]
  })
}

resource "aws_iam_role_policy" "datadog_integration_policy" {
  name = "DatadogIntegrationPolicy-${var.eks_cluster_name}"
  role = aws_iam_role.datadog_integration.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "ec2:Describe*", "rds:Describe*", "s3:GetBucketLocation", "s3:ListAllMyBuckets",
          "autoscaling:Describe*", "elb:Describe*", "elasticache:Describe*",
          "route53:ListHostedZones", "route53:ListResourceRecordSets",
          "dynamodb:ListTables", "dynamodb:DescribeTable",
          "lambda:ListFunctions", "lambda:GetFunctionConfiguration",
          "tag:GetResources", "ecs:Describe*", "eks:Describe*",
          "cloudwatch:ListMetrics", "cloudwatch:GetMetricStatistics",
          "cloudwatch:GetMetricData", "logs:DescribeLogGroups", "logs:FilterLogEvents"
        ]
        Effect   = "Allow"
        Resource = "*"
      },
    ]
  })
}

resource "datadog_integration_aws" "aws_integration" {
  account_id   = data.aws_caller_identity.current.account_id
  role_name    = aws_iam_role.datadog_integration.name
  host_tags    = ["environment:production", "project:eks-observability"]
  filter_tags  = ["eks_cluster_name:${var.eks_cluster_name}"]
  excluded_regions = ["ap-northeast-3"] # Exclude regions you don't monitor if desired
}

# --- Datadog Agent Deployment on EKS via Helm ---
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = kubernetes_namespace.datadog_namespace.metadata[0].name # Use the created namespace
  version    = "3.17.0" # Specify a stable chart version for consistency

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "kubeStateMetricsExternal.enabled"
    value = "true"
  }

  set {
    name  = "targetSystem"
    value = "linux"
  }

  # Enable APM, Log collection, and process monitoring
  set {
    name  = "apm.enabled"
    value = "true"
  }
  set {
    name  = "logAgent.enabled"
    value = "true"
  }
  set {
    name  = "logAgent.containerCollectAll"
    value = "true"
  }
  set {
    name  = "processAgent.enabled"
    value = "true"
  }
  set {
    name  = "processAgent.processCollection.enabled"
    value = "true"
  }

  # EKS-specific configuration and tags
  set {
    name  = "tags"
    value = "eks_cluster_name:${var.eks_cluster_name},environment:production"
  }

  set {
    name = "datadog.site"
    value = "datadoghq.com" # Or datadoghq.eu, us3.datadoghq.com, etc.
  }
}

# --- Datadog PagerDuty Integration ---
# This resource configures the PagerDuty integration within Datadog.
# It uses the PagerDuty Service name and the integration key from the PagerDuty service integration.
resource "datadog_integration_pagerduty" "pagerduty_integration_link" {
  services = [
    {
      service_name = pagerduty_service.eks_observability_service.name
      service_key  = pagerduty_service_integration.datadog_integration.integration_key
    }
  ]
}

# pagerduty.tf


# --- PagerDuty Service and Integration ---

# Optionally, create an escalation policy if you don't have one and want to manage it via Terraform.
# If you use this, comment out or remove the 'pagerduty_escalation_policy_id' variable and use
# 'pagerduty_escalation_policy.eks_observability_ep.id' below.
# resource "pagerduty_escalation_policy" "eks_observability_ep" {
#   name      = "EKS Observability Escalation Policy for ${var.eks_cluster_name}"
#   num_loops = 2 # Number of times to repeat the policy if incident is not acknowledged
#
#   rule {
#     delay_in_minutes = 5
#     target {
#       type = "user_reference"
#       id   = var.pagerduty_user_id # <<< Make sure this user ID is valid >>>
#     }
#   }
#   # Add more rules for different users/teams or notification methods if needed
# }

resource "pagerduty_service" "eks_observability_service" {
  name                      = "${var.eks_cluster_name}-EKS-Observability"
  auto_resolve_timeout      = "14400" # Auto-resolve after 4 hours if not addressed
  acknowledgement_timeout   = "600"   # Auto-escalate if not acknowledged in 10 minutes
  escalation_policy         = var.pagerduty_escalation_policy_id # Or pagerduty_escalation_policy.eks_observability_ep.id if creating above
  description               = "Handles incidents for AWS EKS cluster: ${var.eks_cluster_name}"
}

# This creates a PagerDuty integration specifically for Datadog Events API V2.
# It provides an 'integration_key' that Datadog uses to send events.
resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog Integration"
  service_id  = pagerduty_service.eks_observability_service.id
  type        = "datadog_events_api_v2_inbound_integration"
}

output "pagerduty_service_name" {
  description = "The name of the PagerDuty service created for EKS observability."
  value       = pagerduty_service.eks_observability_service.name
}

output "pagerduty_datadog_integration_key" {
  description = "The integration key for the Datadog PagerDuty service integration."
  value       = pagerduty_service_integration.datadog_integration.integration_key
  sensitive   = true
}

# datadog_monitors.tf


# Define a clean name for the Datadog PagerDuty integration handle
# Datadog expects clean names for @pagerduty- handles (lowercase, no special chars except hyphens/underscores)
locals {
  pagerduty_integration_handle = lower(replace(pagerduty_service.eks_observability_service.name, "/[^a-zA-Z0-9_-]/", ""))
}

# --- Example Datadog Monitor for EKS Node CPU Utilization ---
resource "datadog_monitor" "eks_node_cpu_utilization" {
  name                = "EKS Node CPU Utilization High on ${var.eks_cluster_name}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.cpu.usage.total{eks_cluster_name:${var.eks_cluster_name}} by {host} > 80"
  message             = <<EOF
EKS node CPU utilization is consistently high (above 80%) for 5 minutes.
Host: {{host.name}}
Current utilization: {{value}}%

@pagerduty-${local.pagerduty_integration_handle}
EOF
  escalation_message  = "CPU utilization remains high. Escalating to PagerDuty again."
  monitor_thresholds {
    critical = 80
    warning  = 70
  }
  tags                = ["environment:production", "eks", "cpu", "alert-team:sre", "eks_cluster_name:${var.eks_cluster_name}"]
  priority            = 1
  notify_no_data      = false
  new_group_delay     = 60 # seconds
  no_data_timeframe   = 20 # minutes
  renotify_interval   = 0  # minutes, 0 for no re-notification
  include_tags        = true
  require_full_window = false
  timeout_h           = 0  # 0 to keep the alert active indefinitely until resolved
}

# --- Example Datadog Monitor for EKS Pod Restarts ---
resource "datadog_monitor" "eks_pod_restarts" {
  name                = "EKS Pod Restarts High on ${var.eks_cluster_name}"
  type                = "metric alert"
  query               = "sum(last_5m):avg:kubernetes.containers.restarts{eks_cluster_name:${var.eks_cluster_name}} by {kube_deployment} > 3"
  message             = <<EOF
High number of pod restarts detected for deployment: {{kube_deployment.name}} in EKS cluster: ${var.eks_cluster_name}.
Check logs for: {{kube_deployment.name}}
@pagerduty-${local.pagerduty_integration_handle}
EOF
  escalation_message  = "Pod restarts continue. Escalating to PagerDuty again."
  monitor_thresholds {
    critical = 3
    warning  = 1
  }
  tags                = ["environment:production", "eks", "pod-restarts", "alert-team:devops", "eks_cluster_name:${var.eks_cluster_name}"]
  priority            = 2
  notify_no_data      = false
  new_group_delay     = 60
  no_data_timeframe   = 20
  renotify_interval   = 0
  include_tags        = true
  require_full_window = false
  timeout_h           = 0
}

Deployment Steps

Once you've placed the Terraform files and updated the variables (especially sensitive ones, consider environment variables like TF_VAR_datadog_api_key), follow these steps to deploy your observability stack:

1. Initialize Terraform

This command downloads the necessary provider plugins. 

terraform init

2. Review the Plan

Always review the execution plan before applying to understand what changes Terraform will make. 

terraform plan

3. Apply the Configuration

Confirm the plan and apply the changes. 

terraform apply

You'll be prompted to type yes to confirm the application.

4. Verify Datadog Agent Deployment (Optional but Recommended)

Use kubectl to ensure the Datadog Agent pods are running correctly in your EKS cluster. 

kubectl get pods -n datadog
kubectl logs -n datadog -l app=datadog-agent-cluster-agent
kubectl logs -n datadog -l app=datadog-agent

Verifying Your Unified Observability Setup

  • Datadog Dashboards: Log into your Datadog account. You should start seeing metrics, logs, and traces from your EKS cluster, nodes, pods, and applications populating your dashboards. Look for the "Kubernetes Overview" and "EKS Overview" dashboards, which should now show data.
  • Datadog Integrations: Verify that the AWS integration is active and collecting data (Integrations > AWS). Also, check the PagerDuty integration status (Integrations > PagerDuty). The service created by Terraform should appear here.
  • PagerDuty Service: In PagerDuty, navigate to Services and find the newly created service for your EKS cluster. Ensure it has the correct escalation policy.
  • Trigger an Alert: For testing, you could temporarily lower a monitor threshold in Datadog (or artificially induce high CPU load on an EKS node) to confirm that an incident is triggered in PagerDuty and the on-call team is notified.

Best Practices and Advanced Considerations

  • Secrets Management: Never hardcode sensitive API keys. Use environment variables (TF_VAR_KEY), AWS Secrets Manager, or HashiCorp Vault with Terraform to manage secrets securely.
  • Custom Monitors: Extend your Datadog monitoring with custom metrics from your applications, synthetic checks for external endpoint availability, and Real User Monitoring (RUM) for frontend performance.
  • Alert Fatigue: Continuously refine your Datadog monitors and PagerDuty escalation policies. Implement intelligent alert grouping, deduplication, and suppression rules to reduce noise and ensure only actionable alerts reach on-call teams.
  • Cost Optimization: Monitor your Datadog and AWS costs. Datadog allows fine-grained control over what metrics/logs are ingested. Optimize AWS resource tagging for better cost allocation.
  • GitOps Workflow: Integrate your Terraform configurations into a GitOps pipeline (e.g., using GitHub Actions, GitLab CI, Argo CD, or Atlantis) for automated, version-controlled deployments and changes.
  • Distributed Tracing: Ensure your applications are instrumented for APM/Distributed Tracing (e.g., OpenTelemetry, Datadog APM libraries) to gain end-to-end visibility into request flows.

Conclusion

By unifying AWS EKS observability with Datadog and PagerDuty, all orchestrated by Terraform, you establish a resilient, automated, and proactive monitoring and incident response system. This setup not only provides deep insights into your Kubernetes environment but also empowers your teams to detect, diagnose, and resolve issues faster, ensuring optimal application performance and reliability. Embrace Infrastructure as Code to keep your observability strategy robust and scalable as your cloud-native infrastructure evolves.

Comments

Post a Comment

Popular posts from this blog

Terraform Configuration for Datadog-PagerDuty Incident Management on AWS EKS

Terraform Configuration for Datadog-PagerDuty Incident Management on AWS EKS In the fast-paced world of cloud-native operations, ensuring high availability and rapid incident response for applications running on AWS EKS is paramount. This comprehensive guide details how to leverage Terraform to seamlessly integrate Datadog for robust monitoring and PagerDuty for efficient incident management, creating a resilient and automated incident response pipeline for your Kubernetes clusters. Architecture Pro-Tip: Modular Design for Scalability When building your incident management infrastructure with Terraform, consider a modular design. Separate your Datadog monitors, PagerDuty services, and AWS EKS resources into distinct Terraform modules. This approach enhances reusability, improves readability, and makes it significantly easier to manage configurations across multiple environments (development, staging, production) or different EKS clusters. Always priori...
Read more

Terraform-Managed AWS EKS Observability and Incident Response with Datadog and PagerDuty

Terraform-Managed AWS EKS Observability and Incident Response with Datadog and PagerDuty In the dynamic world of cloud-native applications, managing AWS Elastic Kubernetes Service (EKS) requires robust observability and a streamlined incident response strategy. This guide delves into integrating Datadog for comprehensive monitoring and PagerDuty for efficient incident management, all orchestrated declaratively using Terraform. We'll explore how to establish a resilient system that not only detects issues promptly but also ensures critical alerts reach the right teams for rapid resolution, maintaining high availability and operational excellence. Architecture Pro-Tip: Always centralize your observability configuration within your Infrastructure as Code (IaC) repository. Treat your Datadog monitors, dashboards, and PagerDuty services as code alongside your EKS clusters. This ensures version control, simplifies rollbacks, an...
Read more

Terraform for Production AWS EKS Observability with Datadog, Prometheus, and PagerDuty Integration

Terraform for Production AWS EKS Observability with Datadog, Prometheus, and PagerDuty Integration In the dynamic world of cloud-native applications, maintaining peak performance and rapid incident response for AWS Elastic Kubernetes Service (EKS) is paramount. Observability, often misunderstood as mere monitoring, extends to understanding the internal state of a system from its external outputs. This guide provides a comprehensive, technical walkthrough on leveraging Terraform to provision and manage a robust observability stack for your production EKS clusters, integrating the power of Datadog for holistic monitoring, Prometheus for deep Kubernetes-native metrics, and PagerDuty for automated incident management. Architecture Pro-Tip: For a truly resilient and scalable production observability strategy on AWS EKS, aim for a layered approach. Use Datadog as your primary pane of glass for aggregated metrics, logs, and traces, leveraging its native integrat...
Read more