Terraform AWS EKS Enterprise Observability: Datadog, Prometheus, and PagerDuty Automation

Architecture Pro-Tip:

For enterprise-grade observability on AWS EKS, treat your monitoring and alerting infrastructure as code. Leverage Terraform to provision not just your EKS clusters, but also the deployment of Datadog Agents, Prometheus scraping configurations, and PagerDuty integrations. This ensures consistency, repeatability, and version control for your entire operational stack, drastically reducing configuration drift and accelerating incident response.

Terraform AWS EKS Enterprise Observability: Datadog, Prometheus, and PagerDuty Automation

In today's dynamic cloud-native landscape, running mission-critical applications on AWS EKS (Elastic Kubernetes Service) demands robust observability. For enterprise environments, a reactive approach to incidents is insufficient. Proactive monitoring, comprehensive logging, distributed tracing, and automated incident response are paramount. This guide will walk you through establishing a formidable observability stack on AWS EKS using Terraform for infrastructure as code (IaC), integrating industry-leading tools like Datadog for unified monitoring, leveraging Prometheus for granular metrics collection, and automating incident management with PagerDuty.

Why Enterprise Observability Matters for AWS EKS

The inherent complexity of Kubernetes clusters, with their ephemeral pods, dynamic scaling, and distributed services, makes traditional monitoring approaches obsolete. Enterprise observability for EKS provides:

End-to-End Visibility: From infrastructure (AWS resources) to the application layer (Kubernetes pods, services, deployments), understanding the health and performance of every component.
Faster MTTR (Mean Time To Resolution): Quickly identify, diagnose, and resolve issues before they impact end-users or business operations.
Proactive Anomaly Detection: Leverage AI/ML-driven insights to predict potential outages or performance bottlenecks.
Cost Optimization: Identify underutilized resources and optimize scaling strategies based on real-time data.
Compliance and Security: Maintain audit trails and monitor for security breaches across your Kubernetes environment.

The Power Trio: Datadog, Prometheus, and PagerDuty

These three tools, orchestrated by Terraform, form a comprehensive and automated observability solution:

Datadog: The Unified Observability Platform
Datadog provides a single pane of glass for metrics, logs, traces, synthetic monitoring, and security monitoring. Its robust integrations with AWS, Kubernetes, and various application technologies make it ideal for consolidating data from complex EKS environments. It can also scrape Prometheus metrics directly.
Prometheus: Granular Metrics Collection
While Datadog is an excellent aggregator, Prometheus excels at open-source time-series data collection. Many Kubernetes operators and custom applications expose metrics in the Prometheus format. Datadog's Agent can be configured to scrape these Prometheus endpoints, centralizing all your metrics in Datadog.
PagerDuty: Intelligent Incident Management
PagerDuty transforms Datadog alerts into actionable incidents, routing them to the right on-call teams, escalating as needed, and facilitating communication during critical events. This automation ensures no alert goes unnoticed and speeds up incident resolution.

Terraforming Your Observability Foundation on AWS EKS

Automating the deployment and configuration of your observability stack with Terraform brings immense benefits, ensuring consistency, repeatability, and version control. This section outlines the key Terraform resources and configurations.

Prerequisites

Before you begin, ensure you have:

An AWS Account with appropriate permissions.
Terraform CLI installed.
kubectl configured to interact with your EKS cluster.
A Datadog account with API and Application Keys.
A PagerDuty account with an API key (for programmatic integration).

1. AWS EKS Cluster Provisioning (Terraform)

While the full EKS cluster creation is outside the scope of this focused guide, it's the foundational step. We recommend using the terraform-aws-modules/eks/aws module for a robust and opinionated EKS deployment. Ensure your EKS cluster is operational and kubeconfig is updated.

2. Deploying Datadog Agent to EKS via Terraform and Helm

The Datadog Agent is crucial for collecting metrics, logs, and traces from your EKS cluster. We'll use Terraform's Kubernetes provider and Helm provider to deploy the Datadog Agent Helm chart.

Terraform Configuration Example:

Below is a ready-to-use Terraform configuration to deploy the Datadog Agent, enable basic Prometheus scraping, and configure a sample Datadog monitor that integrates with PagerDuty. Remember to replace placeholder values with your actual API keys and cluster details.

resource "kubernetes_namespace" "datadog_agent" {
  metadata {
    name = "datadog"
  }
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  namespace  = kubernetes_namespace.datadog_agent.metadata[0].name
  chart      = "datadog"
  repository = "https://helm.datadoghq.com"
  version    = "2.39.0" # Use the latest stable version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    type  = "string"
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    type  = "string"
  }

  # Enable APM, Log Collection, and Process Agent
  set {
    name  = "apm.enabled"
    value = "true"
  }
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }
  set {
    name  = "processAgent.enabled"
    value = "true"
  }

  # Enable Prometheus endpoint scraping by Datadog Agent
  # This tells Datadog Agent to discover services with specific annotations
  # and scrape their /metrics endpoint.
  set {
    name  = "datadog.kubeStateMetricsCore.enabled"
    value = "true"
  }
  set {
    name  = "prometheusScrape.enabled"
    value = "true"
  }
  set {
    name  = "datadog.config.use_dogstatsd"
    value = "true"
  }
  # Recommended for EKS: Set site to your Datadog region (e.g., datadoghq.com, eu.datadoghq.com)
  set {
    name  = "datadog.site"
    value = var.datadog_site
  }

  # Kubernetes event collection
  set {
    name  = "datadog.collectEvents"
    value = "true"
  }

  # RBAC must be enabled for Datadog Agent
  set {
    name  = "rbac.create"
    value = "true"
  }
}

# Configure Datadog Provider
provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
  api_url = "https://api.${var.datadog_site}"
}

# Ensure PagerDuty integration is set up in Datadog manually or via another Terraform resource
# For this example, we assume a PagerDuty integration named "PagerDuty-Primary" exists in Datadog.

resource "datadog_monitor" "high_cpu_alert" {
  name                = "[EKS-${var.eks_cluster_name}] High CPU Utilization Alert"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.eks_cluster_name}} by {kube_deployment} > 80"
  message             = "High CPU usage detected on EKS deployment {{kube_deployment.name}} in cluster ${var.eks_cluster_name}. Current value: {{value}}%. @webhook-PagerDuty-Primary"
  tags                = ["env:${var.environment}", "service:kubernetes", "severity:critical"]
  renotify_interval   = 60
  no_data_timeframe   = 20
  include_tags        = true
  require_full_window = true
  force_delete        = false

  monitor_thresholds {
    critical = 80
    warning  = 70
  }

  # This assumes you have a PagerDuty integration configured in Datadog,
  # typically named "PagerDuty" or "PagerDuty-Primary".
  # The @webhook-PagerDuty-Primary sends the alert to PagerDuty.
}

variable "datadog_api_key" {
  description = "Your Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Your Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "datadog_site" {
  description = "Datadog site (e.g., datadoghq.com, eu.datadoghq.com)"
  type        = string
  default     = "datadoghq.com"
}

variable "eks_cluster_name" {
  description = "The name of your EKS cluster"
  type        = string
}

variable "environment" {
  description = "Environment tag (e.g., prod, dev)"
  type        = string
  default     = "dev"
}

Explanation of the Terraform Configuration:

kubernetes_namespace: Creates a dedicated namespace for the Datadog Agent.
helm_release "datadog_agent":
- Deploys the Datadog Agent using its official Helm chart.
- datadog.apiKey and datadog.appKey are set using Terraform variables for security.
- Key features like APM, log collection (logs.containerCollectAll: true), and the Process Agent are enabled.
- prometheusScrape.enabled: true configures the Datadog Agent to discover and scrape Prometheus metrics from services annotated with prometheus.io/scrape: "true" and prometheus.io/port.
- rbac.create: true ensures the necessary Kubernetes RBAC resources are created for the Agent.
datadog_monitor "high_cpu_alert":
- Defines a Datadog "metric alert" monitor.
- The query targets average Kubernetes CPU usage across deployments in your specific EKS cluster.
- The message includes placeholders ({{kube_deployment.name}}, {{value}}) for dynamic content and crucially, @webhook-PagerDuty-Primary. This directs the alert to a pre-configured PagerDuty integration within Datadog.
- tags are essential for filtering and organizing monitors within Datadog.
Variables: Placeholders for sensitive data and environment-specific values.

3. Integrating PagerDuty with Datadog (Conceptual)

While the Datadog monitor configuration above sends alerts to PagerDuty via a webhook, you'll need to set up the PagerDuty integration within Datadog itself. This typically involves:

In Datadog, navigate to Integrations -> PagerDuty.
Configure the integration, selecting the desired PagerDuty service(s) where incidents should be routed.
You can also use the datadog_integration_pagerduty Terraform resource to automate this setup, but it requires a PagerDuty service key or API key to be managed securely.

Deep Dive into Configuration & Best Practices

Datadog Agent Helm Values for EKS

The values.yaml file (or set parameters in Terraform Helm release) for the Datadog Agent chart is extensive. Key considerations for EKS include:

Metrics Collection: Ensure kubeStateMetricsCore.enabled is true for core Kubernetes metrics.
Log Collection: Configure logs.enabled and logs.containerCollectAll. Consider filtering logs for cost optimization.
APM and Tracing: Enable apm.enabled and ensure your applications are instrumented with Datadog APM libraries.
Prometheus Scraping: As shown, prometheusScrape.enabled: true is vital. Services exposing Prometheus metrics should have annotations like:
annotations: prometheus.io/scrape: "true" prometheus.io/port: "8080"
Resource Limits: Set appropriate CPU and memory limits for the Agent DaemonSet to prevent resource contention.

Datadog Monitor Best Practices

Tagging: Use consistent tags (e.g., env:prod, service:api, cluster:my-eks) across all your resources and monitors for easy filtering and organization.
Composite Monitors: Combine multiple alerts into a single, high-fidelity incident using composite monitors.
No Data Alerts: Configure no_data_timeframe to detect when metrics stop reporting, indicating a potential agent or service failure.
Anomaly Detection: Leverage Datadog's machine learning capabilities for monitors that adapt to dynamic baselines, reducing alert fatigue.

PagerDuty Integration Best Practices

Dedicated Services: Create distinct PagerDuty services for different critical applications or teams to ensure alerts are routed correctly.
Escalation Policies: Define clear escalation policies within PagerDuty to ensure alerts are acknowledged and addressed promptly.
Runbooks: Link runbooks to PagerDuty services to provide on-call teams with immediate steps for troubleshooting.

Benefits of This Automated Approach

Consistency: Ensures all EKS clusters have the same observability configuration, regardless of environment.
Reliability: Reduces human error in configuration and deployment.
Scalability: Easily apply observability to new clusters or scale existing ones.
Version Control: All observability configurations are tracked in Git, allowing for rollbacks and audits.
Faster Incident Response: Automated alerting and incident routing mean quicker detection and resolution.

Troubleshooting and FAQ

Q: My Datadog Agent isn't reporting metrics. What should I check?
A: Verify that the datadog.apiKey and datadog.appKey are correct and not exposed. Check the Datadog Agent pod logs (kubectl logs -n datadog -l app=datadog --tail 100) for errors. Ensure network policies aren't blocking outbound traffic to Datadog endpoints. Confirm RBAC permissions are sufficient for the agent to access Kubernetes API resources.

Q: Prometheus metrics aren't appearing in Datadog.
A: Ensure prometheusScrape.enabled: true in your Helm chart values. Verify that your application's Kubernetes service or pod has the correct Prometheus annotations (prometheus.io/scrape: "true" and prometheus.io/port). Check Datadog Agent logs for any scraping errors.

Q: PagerDuty incidents aren't triggering from Datadog alerts.
A: Confirm the @webhook-PagerDuty-Primary (or whatever name you used) notification syntax in your Datadog monitor message is correct. Check the Datadog PagerDuty integration status in the Datadog UI to ensure it's active and correctly configured to your PagerDuty service.

Q: How do I manage sensitive API keys securely in Terraform?
A: Use Terraform variables and provide values via environment variables (TF_VAR_datadog_api_key), a CI/CD secrets manager (e.g., AWS Secrets Manager, HashiCorp Vault), or a terraform.tfvars file that is properly secured and not committed to version control.

Conclusion

Building an enterprise-grade observability stack for AWS EKS is a critical endeavor for any organization running cloud-native applications. By leveraging Terraform for declarative infrastructure management, Datadog for unified monitoring, its powerful Prometheus scraping capabilities, and PagerDuty for automated incident response, you can achieve unparalleled visibility and operational efficiency. This automated approach ensures your teams can confidently deploy, manage, and scale applications on Kubernetes, knowing that issues will be detected and addressed swiftly, minimizing impact on your business and customers.

CloudStack YAML