Terraform AWS EKS Observability: Datadog, Prometheus, PagerDuty Integration

Terraform AWS EKS Observability: Datadog, Prometheus, PagerDuty Integration Guide

In the dynamic world of cloud-native applications, maintaining robust observability for your Amazon Elastic Kubernetes Service (EKS) clusters is paramount. This comprehensive guide details how to establish a sophisticated observability stack using Terraform for infrastructure as code, Datadog for holistic monitoring, Prometheus for deep Kubernetes-native metrics, and PagerDuty for reliable incident response. By integrating these powerful tools, you can ensure high availability, proactive issue detection, and rapid resolution for your EKS workloads.

Architecture Pro-Tip: Unified Observability Strategy

For large-scale EKS deployments, consider a hybrid observability approach. Leverage Datadog as your primary pane of glass for consolidated metrics, logs, traces, and RUM (Real User Monitoring). Complement this with Prometheus for granular, high-cardinality custom metrics or specific Kubernetes internal components where deep, raw metric access is critical. Ensure consistent tagging across all tools to facilitate cross-platform correlation and reduce troubleshooting time.

Why an Integrated Observability Stack for AWS EKS?

Running containerized applications on AWS EKS introduces complexity that necessitates a robust monitoring strategy. Without proper observability, diagnosing performance bottlenecks, security vulnerabilities, or application errors can be a daunting and time-consuming task. An integrated stack provides:

• End-to-End Visibility: From infrastructure (AWS EC2, VPC) to EKS control plane, worker nodes, and individual pod/container metrics.
• Proactive Issue Detection: Real-time alerts on anomalies, resource exhaustion, or service degradations.
• Faster MTTR (Mean Time To Resolution): Consolidated dashboards, logs, and traces help engineers quickly pinpoint root causes.
• Automated Incident Response: Seamless escalation of critical issues to on-call teams.
• Infrastructure as Code (IaC): Terraform ensures consistent, repeatable, and auditable deployment of observability components.

Prerequisites

Before diving into the configuration, ensure you have the following:

• An active AWS account with administrative access.
• Terraform installed (v1.0+ recommended).
• kubectl configured to interact with your EKS cluster.
• A Datadog account with an API key and Application key.
• A PagerDuty account with an integration key for Datadog or Prometheus Alertmanager.
• An existing AWS EKS cluster, or the ability to provision one using Terraform.
• Helm CLI installed (for local testing/chart inspection, though Terraform will manage releases).

Core Components and Their Roles

Terraform: Infrastructure and Observability as Code

Terraform acts as the orchestrator, defining and managing the entire lifecycle of your AWS EKS cluster and its observability components. This includes IAM roles, Kubernetes resources (Deployments, DaemonSets, Services), and even Datadog monitors.

Datadog: Unified Monitoring and APM

Datadog provides a comprehensive platform for metrics, logs, traces, and security monitoring across your EKS environment. The Datadog Agent, deployed as a DaemonSet, collects data from nodes, pods, and applications. Key features include:

• Infrastructure Monitoring: CPU, memory, network I/O for nodes and containers.
• Kubernetes Integration: Monitors EKS control plane and component health.
• APM (Application Performance Monitoring): Distributed tracing for microservices.
• Log Management: Centralized log collection, parsing, and analysis.
• Network Performance Monitoring: Visibility into network traffic between services.
• Dashboards & Alerting: Customizable visualizations and sophisticated anomaly detection.

Prometheus: Open-Source Monitoring and Alerting

Prometheus is an open-source monitoring system with a powerful query language (PromQL). It excels at collecting time-series data from various targets. For EKS, the kube-prometheus-stack (which includes Prometheus, Grafana, and Alertmanager) is a popular choice for:

• Scraping Metrics: Collects metrics from Kubernetes components (kube-state-metrics, cAdvisor), custom application metrics, and node exporters.
• Alertmanager: Handles alerts generated by Prometheus, sending them to external systems like PagerDuty.
• Grafana: Provides powerful dashboards for visualizing Prometheus metrics.

While Datadog offers a full-stack solution, Prometheus can be valuable for specific use cases, such as maintaining an in-cluster, open-source-driven monitoring solution, or for highly custom metrics that you prefer to manage internally before potentially forwarding to Datadog.

PagerDuty: Incident Management and On-Call Automation

PagerDuty is a leading incident management platform that integrates with monitoring tools to provide reliable alerting, on-call scheduling, and automated escalation. When a critical alert is triggered in Datadog or Prometheus, PagerDuty ensures the right team member is notified through their preferred channel (phone, SMS, email, push notification), facilitating a rapid response.

Terraform Implementation: Deploying the Observability Stack

We will use Terraform to deploy the Datadog Agent and the Prometheus stack to your EKS cluster, and then configure a Datadog monitor to integrate with PagerDuty.

1. Configure AWS and Kubernetes Providers

First, set up your Terraform providers for AWS and Kubernetes. Ensure your Kubernetes provider is configured to connect to your EKS cluster.

provider "aws" { region = "us-east-1" } data "aws_eks_cluster" "eks_cluster" { name = var.cluster_name } data "aws_eks_cluster_auth" "eks_cluster_auth" { name = var.cluster_name } provider "kubernetes" { host = data.aws_eks_cluster.eks_cluster.endpoint cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data) token = data.aws_eks_cluster_auth.eks_cluster_auth.token } provider "helm" { kubernetes { host = data.aws_eks_cluster.eks_cluster.endpoint cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data) token = data.aws_eks_cluster_auth.eks_cluster_auth.token } } provider "datadog" { api_key = var.datadog_api_key app_key = var.datadog_app_key } variable "cluster_name" { description = "Name of the EKS cluster" type = string } variable "datadog_api_key" { description = "Datadog API Key" type = string sensitive = true } variable "datadog_app_key" { description = "Datadog Application Key" type = string sensitive = true } variable "pagerduty_service_integration_key" { description = "PagerDuty service integration key for Datadog" type = string sensitive = true }

2. Deploy Datadog Agent via Helm Chart

Use the Terraform helm_release resource to deploy the Datadog Agent. This example enables APM, log collection, and a few common integrations. Replace <YOUR_DATADOG_API_KEY> and <YOUR_DATADOG_APP_KEY> with your actual keys, or better yet, pass them as sensitive variables.

3. Ready-to-Use Configuration: Deploying Datadog, Prometheus, and PagerDuty Integrations

# main.tf # ---------------------------------------------------------------------------------------------------------------------- # DATADOG AGENT DEPLOYMENT # Deploys the Datadog Agent as a DaemonSet to collect metrics, logs, and traces from EKS nodes and pods. # ---------------------------------------------------------------------------------------------------------------------- resource "helm_release" "datadog_agent" { name = "datadog" repository = "https://helm.datadoghq.com" chart = "datadog" namespace = "datadog" create_namespace = true set { name = "datadog.apiKey" value = var.datadog_api_key sensitive = true } set { name = "datadog.appKey" value = var.datadog_app_key sensitive = true } values = [ "${file("datadog_values.yaml")}" ] } # datadog_values.yaml # (Place this content into a file named datadog_values.yaml in the same directory) # For more advanced configurations, refer to the Datadog Helm chart documentation. # Example: Custom checks, APM, Logging, Process Monitoring, etc. # --- # agent: # image: # tag: 7.49.0 # Specify agent version # clusterName: ${var.cluster_name} # logLevel: INFO # criSocket: /var/run/dockershim.sock # Or /var/run/containerd/containerd.sock for containerd # config: # collectEvents: true # checksd: # "kubernetes_state.yaml": |- # init_config: # instances: # - collectors: # - nodes # - deployments # - replicasets # - hpa # - pods # apm: # enabled: true # hostPortConfig: # enabled: true # logs: # enabled: true # containerCollectAll: true # processAgent: # enabled: true # kubeStateMetrics: # enabled: true # clusterAgent: # enabled: true # metricsProvider: # enabled: true # createServiceAccount: true # serviceAccountName: datadog-cluster-agent-metrics-provider # rbac: # create: true # serviceAccountName: datadog-agent # ---------------------------------------------------------------------------------------------------------------------- # ---------------------------------------------------------------------------------------------------------------------- # PROMETHEUS STACK DEPLOYMENT # Deploys kube-prometheus-stack (Prometheus, Grafana, Alertmanager) to collect Kubernetes-native metrics. # ---------------------------------------------------------------------------------------------------------------------- resource "helm_release" "prometheus_stack" { name = "prometheus" repository = "https://prometheus-community.github.io/helm-charts" chart = "kube-prometheus-stack" namespace = "monitoring" create_namespace = true values = [ "${file("prometheus_values.yaml")}" ] # Ensure Prometheus is deployed after Datadog if there are any resource conflicts or shared dependencies depends_on = [helm_release.datadog_agent] } # prometheus_values.yaml # (Place this content into a file named prometheus_values.yaml in the same directory) # Example: Basic configuration, exposing Grafana via ClusterIP (for internal access) # For external access, you might configure an Ingress or LoadBalancer. # --- # alertmanager: # enabled: true # alertmanagerSpec: # storage: # volumeClaimTemplate: # spec: # accessModes: ["ReadWriteOnce"] # resources: # requests: # storage: 1Gi # config: # global: # resolve_timeout: 5m # route: # group_by: ['alertname', 'cluster', 'service'] # group_wait: 30s # group_interval: 5m # repeat_interval: 1h # receiver: 'pagerduty' # receivers: # - name: 'pagerduty' # pagerduty_configs: # - service_key: ${var.pagerduty_service_integration_key} # PagerDuty integration key for Alertmanager # # The above line `service_key` can be used to send alerts directly from Prometheus Alertmanager to PagerDuty. # # However, for a unified approach, we recommend Datadog for PagerDuty integration. # # This is shown here for completeness if you prefer Prometheus-native alerting. # # You would need a separate PagerDuty integration for Alertmanager. # # prometheus: # prometheusSpec: # serviceMonitorSelectorNilUsesHelmValues: false # podMonitorSelectorNilUsesHelmValues: false # probeSelectorNilUsesHelmValues: false # ruleSelectorNilUsesHelmValues: false # storageSpec: # volumeClaimTemplate: # spec: # accessModes: ["ReadWriteOnce"] # resources: # requests: # storage: 10Gi # ingress: # enabled: false # # grafana: # enabled: true # adminPassword: "your-strong-grafana-password" # CHANGE THIS IN PRODUCTION! # service: # type: ClusterIP # Use LoadBalancer or Ingress for external access # ingress: # enabled: false # ---------------------------------------------------------------------------------------------------------------------- # ---------------------------------------------------------------------------------------------------------------------- # DATADOG MONITOR & PAGERDUTY INTEGRATION # Configures a Datadog monitor to trigger an alert and send it to PagerDuty. # This assumes you have already configured a PagerDuty integration within Datadog # and have its service integration key. # ---------------------------------------------------------------------------------------------------------------------- resource "datadog_monitor" "high_cpu_usage" { name = "[EKS-${var.cluster_name}] High CPU Usage on Node" type = "metric alert" message = "CPU usage on node {{host.name}} is above 80% for more than 5 minutes. @pagerduty-your-service" # Use your PagerDuty integration name/handle escalation_message = "CPU usage remains high, escalating to on-call." query = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster_name:${var.cluster_name}} by {host} < 20" monitor_thresholds { critical = 20 warning = 30 } notify_no_data = false new_group_delay = 60 no_data_timeframe = 10 # Tags for better organization and filtering in Datadog tags = [ "env:production", "service:eks-platform", "team:devops", "alert-type:cpu", ] # Configuration for sending alerts to PagerDuty # The `@pagerduty-your-service` in the message field is the primary way # Datadog integrates with PagerDuty for a specific monitor. # Ensure you have set up a PagerDuty integration in Datadog's integrations page # and named it appropriately. # For more advanced PagerDuty integration (e.g., routing keys), you might # leverage Datadog's integrations API or webhooks. }

Explanation of Configuration:

• Datadog Agent: The helm_release resource deploys the Datadog Agent using its official Helm chart. The values attribute points to a local datadog_values.yaml file for more detailed configurations like enabling APM, logs, and specific integrations. Replace the API and APP keys with your sensitive variables.
• Prometheus Stack: Similarly, the kube-prometheus-stack is deployed via Helm. The prometheus_values.yaml file can be customized for storage, Grafana access, and Alertmanager configurations. Note the commented-out PagerDuty configuration within Alertmanager; if you want Prometheus to directly send alerts, you'd enable that with a dedicated PagerDuty integration key for Alertmanager.
• Datadog Monitor with PagerDuty: The datadog_monitor resource defines a simple CPU alert. The crucial part for PagerDuty integration is the message field: @pagerduty-your-service. This special mention directs Datadog to send the alert to your configured PagerDuty service integration. Ensure you replace your-service with the actual name or handle you've configured in Datadog for your PagerDuty integration.

4. Deploying the Terraform Configuration

Initialize your Terraform workspace and apply the configuration:

terraform init terraform plan terraform apply --auto-approve

Validation and Testing

Verify Datadog Agent Deployment

Check if Datadog pods are running in the datadog namespace:

kubectl get pods -n datadog

Then, navigate to your Datadog dashboard. You should see your EKS nodes and pods reporting metrics, logs, and traces.

Verify Prometheus Stack Deployment

Check if Prometheus, Grafana, and Alertmanager pods are running in the monitoring namespace:

kubectl get pods -n monitoring

To access Grafana (if using ClusterIP), use port forwarding:

kubectl port-forward svc/prometheus-grafana 3000:80 -n monitoring

Then access http://localhost:3000 in your browser. Log in with admin and your specified password.

Test PagerDuty Integration

Manually trigger an alert or simulate high CPU usage on a node. Verify that an incident is created in PagerDuty and that your on-call team is notified.

Best Practices for EKS Observability

• Unified Tagging: Implement consistent resource tagging (e.g., environment, service, team, owner) across AWS, Kubernetes, Datadog, and Prometheus. This is crucial for filtering, dashboarding, and cost attribution.
• Granular IAM: Configure least-privilege IAM roles for your Datadog Agent and Prometheus components to access only necessary AWS and Kubernetes resources.
• Log Retention Policies: Define appropriate log retention policies in Datadog (and S3 if forwarding directly from EKS for archival) to balance cost and compliance requirements.
• Alert Fatigue Management: Fine-tune your Datadog and Prometheus alerts to be actionable. Use composite monitors, anomaly detection, and correlation to reduce noise. Leverage PagerDuty's incident grouping and routing capabilities.
• Version Control: Keep all your Terraform configurations in a version control system (like Git) to track changes, enable collaboration, and facilitate rollbacks.
• Security Hardening: Regularly update Helm charts and agent versions. Configure network policies to restrict access to monitoring endpoints (e.g., Grafana, Prometheus UI).

Troubleshooting Common Issues

• Datadog Agent Not Reporting:
  • Check datadog.apiKey and datadog.appKey in your Helm values for correctness.
  • Inspect Datadog Agent pod logs: kubectl logs -f -n datadog <datadog-agent-pod-name>. Look for connectivity errors.
  • Verify necessary IAM permissions for the EKS node group to allow communication with Datadog endpoints.
• Prometheus Not Scraping Metrics:
  • Access the Prometheus UI (via port-forwarding) and check the "Targets" section for scrape errors.
  • Ensure ServiceMonitors and PodMonitors are correctly defined and have appropriate labels to be discovered by Prometheus.
  • Check network policies that might be blocking Prometheus from reaching target endpoints.
• PagerDuty Alerts Not Firing:
  • Datadog: Verify the @pagerduty-your-service tag in your Datadog monitor message matches an active PagerDuty integration name in Datadog. Test the integration from the Datadog UI.
  • Prometheus Alertmanager: Ensure the service_key in Alertmanager's configuration is correct for your PagerDuty integration key. Check Alertmanager logs for errors.
  • Confirm that the monitor's alert conditions are actually being met.

Conclusion

Establishing robust observability for AWS EKS is a critical step in managing healthy, high-performing cloud-native applications. By leveraging Terraform for declarative infrastructure management, Datadog for comprehensive full-stack monitoring, Prometheus for deep Kubernetes-native insights, and PagerDuty for efficient incident response, you create a powerful, integrated solution. This guide provides the foundational steps to automate the deployment and configuration of this stack, empowering your DevOps teams with the visibility and alerting capabilities needed to maintain operational excellence in your EKS environments.