Terraform for AWS EKS: Enterprise Observability with Datadog APM and PagerDuty Alerting

In the dynamic landscape of modern cloud infrastructure, managing containerized applications on Kubernetes demands robust observability and incident response strategies. AWS EKS provides a powerful platform for deploying scalable microservices, but without proper monitoring and alerting, operational stability can be compromised. This comprehensive guide details how to leverage Terraform for declarative infrastructure provisioning, integrating Datadog APM for deep application performance monitoring, and establishing proactive incident management with PagerDuty for enterprise-grade observability on AWS EKS.

Architecture Pro-Tip: Layered Observability Strategy

For true enterprise observability, adopt a layered approach:

Infrastructure Layer: Monitor EKS cluster health, node metrics, and underlying AWS services (EC2, VPC).
Kubernetes Layer: Track Pod, Deployment, Service, and Namespace metrics, events, and logs.
Application Layer (APM): Gain deep insights into application traces, service dependencies, latency, and error rates.
Synthetic Monitoring & RUM: Proactively test endpoints and monitor real user experience.

Consolidate these data points into a single pane of glass like Datadog to provide context-rich alerts to PagerDuty.

Why Enterprise Observability Matters for EKS

Microservices architectures on Kubernetes offer agility and scalability but introduce complexity. Without a unified observability strategy, teams struggle with:

Mean Time To Resolution (MTTR): Difficulty in quickly identifying root causes of issues.
Operational Blind Spots: Lack of visibility into application performance, infrastructure health, and user experience.
Alert Fatigue: Disjointed monitoring tools generating excessive, unactionable alerts.
Compliance and Auditing: Inability to easily track changes and performance over time.

By integrating Terraform, Datadog APM, and PagerDuty, organizations can achieve a robust, automated, and proactive approach to managing their EKS environments.

Key Components Explained

Terraform: Infrastructure as Code (IaC)

Terraform is an open-source IaC tool that allows you to define and provision cloud and on-prem resources using a high-level configuration language. For EKS, Terraform enables:

Declarative Provisioning: Define your desired EKS cluster state (VPC, subnets, IAM roles, EKS cluster, node groups) and let Terraform create/update it.
Version Control: Manage your infrastructure definitions in Git, enabling collaboration, change tracking, and rollbacks.
Reusability: Create modules for common infrastructure patterns.
Consistency: Ensure identical environments across development, staging, and production.

Datadog APM: Application Performance Monitoring for Kubernetes

Datadog offers a comprehensive monitoring platform, with its APM (Application Performance Monitoring) component being crucial for microservices. For EKS, Datadog APM provides:

Distributed Tracing: Visualize requests across services, identify bottlenecks, and pinpoint errors.
Service Maps: Understand application dependencies and health at a glance.
Code-Level Visibility: Drill down into specific method calls and database queries.
Kubernetes Integration: Collects metrics, logs, and events from EKS, Pods, Deployments, and containers out-of-the-box.
Unified Dashboarding: Correlate APM data with infrastructure metrics and logs.

PagerDuty: Incident Management and On-Call Alerting

PagerDuty is a leading incident management platform that transforms monitoring signals into actionable incidents. It enables:

Intelligent Alerting: Consolidate alerts from various monitoring tools (like Datadog) and apply noise reduction.
On-Call Management: Automate scheduling, escalation policies, and notifications.
Incident Response: Facilitate rapid response, collaboration, and post-incident analysis.
Integrations: Seamlessly connect with hundreds of monitoring, ticketing, and collaboration tools.

Prerequisites

Before you begin, ensure you have:

An AWS Account with administrative access.
Terraform CLI installed (v1.0+ recommended).
kubectl CLI installed and configured.
A Datadog Account with your API Key and Application Key.
A PagerDuty Account with an API Key (for Terraform provider) or an integration key (for Datadog service).

Step-by-Step Implementation with Terraform

1. Provisioning AWS EKS with Terraform

We'll use the official Terraform AWS EKS module for simplicity and best practices.

First, set up your AWS provider and define variables:

provider "aws" {
  region = var.aws_region
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
  }
}

variable "aws_region" {
  description = "AWS region"
  type        = string
  default     = "us-east-1"
}

variable "cluster_name" {
  description = "EKS cluster name"
  type        = string
  default     = "datadog-eks-observability"
}
    

Next, define your VPC and EKS cluster configuration:

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "${var.cluster_name}-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["${var.aws_region}a", "${var.aws_region}b", "${var.aws_region}c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]

  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true

  tags = {
    "kubernetes.io/cluster/${var.cluster_name}" = "owned"
  }
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0"

  cluster_name    = var.cluster_name
  cluster_version = "1.28"

  vpc_id                   = module.vpc.vpc_id
  subnet_ids               = module.vpc.private_subnets
  control_plane_subnet_ids = module.vpc.private_subnets # For Fargate profiles or private access

  enable_irsa = true

  eks_managed_node_groups = {
    general = {
      min_size     = 1
      max_size     = 3
      desired_size = 2

      instance_types = ["t3.medium"]
      capacity_type  = "ON_DEMAND"
    }
  }

  tags = {
    Environment = "Dev"
    Project     = "ObservabilityGuide"
  }
}

# Configure kubernetes provider to connect to the EKS cluster
data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}
    

2. Integrating Datadog APM with Terraform and Helm

The Datadog Agent is typically deployed as a DaemonSet on Kubernetes. We'll use the Helm provider for Terraform to deploy the Datadog Agent chart. This also creates the necessary Kubernetes RBAC resources.

Ensure you have your Datadog API and APP keys ready. These should be managed securely, ideally via AWS Secrets Manager and referenced in Terraform.

resource "kubernetes_secret" "datadog_api_key" {
  metadata {
    name      = "datadog-api-key"
    namespace = "default" # Or a dedicated monitoring namespace
  }
  data = {
    "api-key" = var.datadog_api_key
    "app-key" = var.datadog_app_key
  }
}

# Deploy Datadog Agent using Helm
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Or a dedicated monitoring namespace, ensure it exists
  version    = "2.33.0" # Use a stable, recent version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key # Use a secure variable or secret reference
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key # Use a secure variable or secret reference
  }
  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
  }
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "clusterAgent.metricsProvider.enabled"
    value = "true"
  }
  set {
    name  = "apm.enabled"
    value = "true" # Enable APM
  }
  set {
    name  = "logs.enabled"
    value = "true" # Enable log collection
  }
  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }
  set {
    name  = "processAgent.enabled"
    value = "true"
  }
  set {
    name  = "systemProbe.enabled"
    value = "true"
  }
  set {
    name  = "targetSystem"
    value = "linux"
  }

  values = [
    # Custom values can go here, e.g., for specific tags or resource limits
    # file("path/to/custom-values.yaml")
  ]
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}
    

With the Datadog Agent deployed, APM tracing can begin. Ensure your applications are instrumented with Datadog's APM libraries (e.g., for Java, Python, Node.js, Go) to send traces to the Agent.

3. Configuring PagerDuty Alerting with Terraform

To integrate PagerDuty, we'll first provision a PagerDuty Service and Escalation Policy using the PagerDuty Terraform provider. Then, we'll link Datadog monitors to this service.

terraform {
  required_providers {
    pagerduty = {
      source  = "PagerDuty/pagerduty"
      version = "~> 1.15"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
  }
}

variable "pagerduty_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

variable "pagerduty_user_email" {
  description = "Email of an existing PagerDuty user to add to the escalation policy"
  type        = string
}

provider "pagerduty" {
  token = var.pagerduty_token
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

# Create a PagerDuty escalation policy
resource "pagerduty_escalation_policy" "eks_observability_policy" {
  name      = "EKS Observability Policy"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 5
    target {
      type = "user"
      id   = data.pagerduty_user.example_user.id
    }
  }
}

# Data source to fetch an existing PagerDuty user
data "pagerduty_user" "example_user" {
  email = var.pagerduty_user_email
}

# Create a PagerDuty service
resource "pagerduty_service" "eks_observability_service" {
  name                    = "EKS Cluster Observability"
  auto_resolve_timeout    = 60
  acknowledgement_timeout = 30
  escalation_policy       = pagerduty_escalation_policy.eks_observability_policy.id
}

# Create a Datadog integration for PagerDuty
resource "datadog_integration_pagerduty" "pagerduty_integration" {
  api_token = var.pagerduty_token # PagerDuty API token for Datadog integration
}
    

With the PagerDuty service created, you can now define Datadog monitors that alert this service.

Ready-to-Use Configuration: Datadog APM Monitor and PagerDuty Alert

Here's how to define a Datadog monitor with Terraform that triggers an incident in PagerDuty when an EKS application's error rate exceeds a threshold.

# Datadog APM monitor for high error rate
resource "datadog_monitor" "high_app_error_rate" {
  name                = "[EKS APM] High Error Rate for ${var.cluster_name}-web-app"
  type                = "query alert"
  query               = "sum:trace.flask.request.errors{env:prod,service:web-app}.as_count().rollup(sum, 300) > 5" # Example query
  message             = "High error rate detected for the web-app service in EKS cluster ${var.cluster_name}. @pagerduty-${pagerduty_service.eks_observability_service.name}"
  
  tags                = ["environment:production", "application:web-app", "team:devops", "eks_cluster:${var.cluster_name}"]
  notify_no_data      = false
  new_group_delay     = 60
  no_data_timeframe   = 20
  renotify_interval   = 0
  timeout_h           = 0
  
  thresholds {
    critical = 5
    warning  = 2
  }
  
  priority = 1 # PagerDuty will respect Datadog's priority for incidents

  # Integration with PagerDuty service created earlier
  # The @pagerduty-servicename syntax links the alert to the PagerDuty service
  # Ensure the PagerDuty integration in Datadog is configured with an API token
}

output "eks_cluster_endpoint" {
  description = "The endpoint URL for the EKS cluster."
  value       = module.eks.cluster_endpoint
}

output "pagerduty_service_name" {
  description = "The name of the PagerDuty service configured for EKS observability."
  value       = pagerduty_service.eks_observability_service.name
}

output "datadog_monitor_id" {
  description = "The ID of the Datadog monitor for application error rates."
  value       = datadog_monitor.high_app_error_rate.id
}
    

To apply this configuration:

Save the code in main.tf files within a directory.
Initialize Terraform: terraform init
Review the plan: terraform plan -var="datadog_api_key=YOUR_DD_API_KEY" -var="datadog_app_key=YOUR_DD_APP_KEY" -var="pagerduty_token=YOUR_PD_TOKEN" -var="pagerduty_user_email=YOUR_PD_USER_EMAIL" (replace with actual keys/emails, or use environment variables/secret management).
Apply the changes: terraform apply -var="datadog_api_key=..." -var="datadog_app_key=..." -var="pagerduty_token=..." -var="pagerduty_user_email=..."

Once applied, your AWS EKS cluster will be provisioned, the Datadog Agent will be deployed for APM, and a Datadog monitor will be set up to alert your PagerDuty service for high application error rates.

Best Practices for Enterprise Observability

Tagging Strategy: Implement a consistent tagging strategy across AWS resources, Kubernetes objects, and Datadog metrics (e.g., env:prod, service:web-app, team:sre). This enables powerful filtering and correlation.
Resource Limits: Set appropriate CPU and memory limits/requests for your Pods to prevent resource contention and improve cluster stability.
SLOs and Alerts: Define Service Level Objectives (SLOs) for your critical services and create Datadog monitors to alert when these SLOs are at risk, not just when systems fail.
Synthetic Monitoring: Complement APM with Datadog Synthetic Monitoring to proactively test user journeys and API endpoints from various global locations.
Security & Compliance: Ensure all API keys and sensitive information are stored securely (e.g., AWS Secrets Manager, HashiCorp Vault) and never hardcoded. Rotate keys regularly.
Cost Optimization: Monitor Datadog usage and optimize data ingestion. Right-size your EKS nodes based on actual workload demands to manage AWS costs effectively.

Troubleshooting and FAQs

Datadog Agent is not collecting metrics/traces.

Check Agent Status: Run kubectl get pods -n default | grep datadog to ensure Datadog Agent Pods are running. Use kubectl logs <datadog-agent-pod> -n default to check logs for errors.

API Key: Double-check that the datadog.apiKey and datadog.appKey values passed to the Helm chart are correct and have the necessary permissions.

Application Instrumentation: Ensure your application code is correctly instrumented with Datadog's APM client libraries and that traces are configured to be sent to the Datadog Agent's APM port (default 8126).

RBAC: Verify the Datadog Agent's ServiceAccount has the necessary RBAC permissions to collect Kubernetes metrics (kubectl describe clusterrole datadog-agent).

PagerDuty incidents are not being created from Datadog alerts.

Datadog-PagerDuty Integration: In Datadog, go to Integrations -> PagerDuty and ensure the integration is active and correctly configured with your PagerDuty API token.

Monitor Message: Verify that your Datadog monitor's message explicitly includes @pagerduty-YOUR_SERVICE_NAME, where YOUR_SERVICE_NAME matches the exact name of your PagerDuty service.

PagerDuty Service: Check the PagerDuty service's Integration section to see if it's receiving events. If it's a "Datadog" integration type, ensure the keys match.

Terraform apply fails with EKS or Kubernetes errors.

IAM Permissions: Ensure the AWS credentials used by Terraform have sufficient permissions to create EKS clusters, IAM roles, VPCs, and related resources.

Kubernetes Provider: The Kubernetes provider relies on the EKS cluster being available and accessible. If EKS provisioning fails, the Kubernetes provider will fail. Address EKS issues first.

Helm Chart Version: Always check the latest stable version of the Datadog Helm chart. Incompatible versions can cause deployment issues.

Conclusion

Establishing robust enterprise observability for AWS EKS is paramount for maintaining the health, performance, and reliability of your microservices. By orchestrating your infrastructure with Terraform, gaining deep insights with Datadog APM, and streamlining incident response with PagerDuty, your DevOps and SRE teams can move from reactive firefighting to proactive, data-driven operations. This integrated approach ensures faster MTTR, reduced operational burden, and a more stable environment for your critical applications.

Embrace Infrastructure as Code to automate the deployment of your observability stack, ensuring consistency, scalability, and adherence to best practices across all your EKS environments.

CloudStack YAML