Terraform for AWS EKS: Unified Observability with Datadog, Prometheus, and PagerDuty

In the dynamic landscape of cloud-native applications, managing and monitoring Kubernetes clusters, especially on AWS EKS, demands a robust and unified observability strategy. This guide delves into leveraging Terraform for declarative infrastructure management, integrating powerful tools like Datadog for comprehensive monitoring, Prometheus for metrics collection and alerting, and PagerDuty for incident response, all to achieve unparalleled operational visibility and efficiency.

Architecture Pro-Tip:

Design your observability stack with a clear hierarchy: Metrics, Logs, and Traces are foundational. Use a centralized platform like Datadog for aggregation and visualization across all three, while leveraging Prometheus for granular, real-time Kubernetes-native metrics and Alertmanager for rule-based incident triggering. Always automate the deployment and configuration of these tools using Infrastructure as Code (IaC) principles with Terraform to ensure consistency, repeatability, and version control across environments.

The Pillars of Unified Observability for EKS

Unified observability goes beyond simply collecting data; it's about correlating metrics, logs, and traces to provide a holistic view of your system's health and performance. For AWS EKS, this means understanding everything from cluster health to individual pod behavior and application performance. Here's how our chosen tools fit together:

Terraform: Infrastructure as Code for Observability

Terraform allows you to define and provision your entire AWS EKS infrastructure, including the observability stack, using a declarative configuration language. This ensures that your monitoring and alerting systems are always in sync with your infrastructure, preventing configuration drift and enabling rapid, repeatable deployments.

Datadog: Comprehensive Cloud Monitoring

Datadog offers an end-to-end monitoring solution for EKS, capable of collecting metrics, logs, and traces from your Kubernetes cluster, applications, and AWS infrastructure. Its powerful dashboards, anomaly detection, and synthetic monitoring capabilities provide deep insights into application and infrastructure performance.

Metrics: Collects host-level, container-level, and application-level metrics.
Logs: Centralizes logs from all EKS pods and nodes for correlation and analysis.
APM & Tracing: Provides distributed tracing to understand request flows across microservices.
Network Performance: Monitors network traffic and dependencies within the cluster.

Prometheus & Alertmanager: Kubernetes-Native Monitoring and Alerting

Prometheus is the de facto standard for Kubernetes monitoring, excelling at time-series data collection and powerful query capabilities. Paired with Alertmanager, it provides sophisticated, rule-based alerting that can be integrated with various notification channels, including PagerDuty.

Metrics Collection: Scrapes metrics from Kubernetes components, nodes, and applications via exporters.
Query Language (PromQL): A powerful query language for slicing, dicing, and aggregating metrics.
Alerting Rules: Defines conditions for alerts based on metric thresholds.
Alertmanager: Handles routing, deduplication, grouping, and silencing of alerts.

PagerDuty: Incident Management and On-Call Automation

PagerDuty transforms alerts into actionable incidents, routing them to the right team members based on on-call schedules. It facilitates rapid response, collaboration, and post-incident analysis, significantly reducing Mean Time To Resolution (MTTR).

On-Call Scheduling: Manages complex on-call rotations and escalation policies.
Incident Routing: Directs alerts from monitoring tools to the appropriate teams.
Automated Notifications: Notifies responders via multiple channels (SMS, phone, email, push).
Stakeholder Communication: Keeps relevant parties informed during incidents.

Benefits of a Unified Observability Stack

Adopting this integrated approach provides several critical advantages:

Faster Incident Resolution: Correlated data across platforms means quicker identification of root causes.
Proactive Issue Detection: Advanced alerting from Prometheus and Datadog helps catch problems before they impact users.
Reduced Alert Fatigue: Intelligent grouping and deduplication by Alertmanager and PagerDuty reduce noise.
Operational Consistency: Terraform ensures your observability stack is deployed consistently across all environments.
Improved Collaboration: Centralized data and incident management streamline communication between DevOps, SRE, and development teams.

Prerequisites

Before you begin, ensure you have the following:

An active AWS Account with necessary IAM permissions.
Terraform CLI installed (v1.0+ recommended).
Kubectl configured to interact with your EKS cluster.
An existing AWS EKS cluster. This guide assumes the EKS cluster is already provisioned.
A Datadog API Key and Application Key.
A PagerDuty API Key and service integration key.

Terraform Configuration for Unified Observability

We will use Terraform to deploy the Datadog Agent, Prometheus Operator, and configure PagerDuty integrations within your EKS cluster.

1. Setting up Kubernetes and Helm Providers

Your Terraform configuration needs to interact with your EKS cluster. Ensure your AWS provider is configured to allow `kubectl` to connect. You'll need the Kubernetes and Helm providers.

provider "kubernetes" {
  host                   = data.aws_eks_cluster.eks.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.eks.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.eks.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.eks.token
  }
}

data "aws_eks_cluster" "eks" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "eks" {
  name = var.eks_cluster_name
}
    

2. Deploying Datadog Agent with Terraform (Helm)

The Datadog Agent is deployed as a DaemonSet to ensure it runs on every node, collecting metrics, logs, and traces. We'll use the Helm provider to manage its deployment.

resource "helm_release" "datadog_agent" {
  name             = "datadog"
  namespace        = "datadog"
  create_namespace = true
  repository       = "https://helm.datadoghq.com"
  chart            = "datadog"
  version          = "2.33.0" # Use a stable, recent version

  values = [
    yamlencode({
      datadog = {
        apiKey                  = var.datadog_api_key
        appKey                  = var.datadog_app_key
        site                    = "datadoghq.com" # Or datadoghq.eu, etc.
        clusterName             = var.eks_cluster_name
        logLevel                = "INFO"
        tags                    = ["env:${var.environment}", "cluster:${var.eks_cluster_name}"]
      }
      agents = {
        enabled = true
        kubeStateMetricsCore = {
          enabled = true
        }
        processAgent = {
          enabled = true
        }
      }
      clusterAgent = {
        enabled = true
        metrics = {
          enabled = true
        }
      }
      logs = {
        enabled = true
        containerCollectAll = true
      }
      apm = {
        enabled = true
      }
      kubeStateMetrics = {
        enabled = true
      }
      networkMonitoring = {
        enabled = true
      }
    })
  ]
}
    

3. Deploying Prometheus Operator with Terraform (Helm)

The Prometheus Operator simplifies the deployment and management of Prometheus and Alertmanager instances. We'll use the Helm provider for this as well.

Before you deploy, consider creating a dedicated namespace for monitoring components:

resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = "monitoring"
  }
}

resource "helm_release" "kube_prometheus_stack" {
  name             = "kube-prometheus-stack"
  namespace        = kubernetes_namespace.monitoring.metadata[0].name
  create_namespace = false # Namespace created above
  repository       = "https://prometheus-community.github.io/helm-charts"
  chart            = "kube-prometheus-stack"
  version          = "56.0.0" # Use a stable, recent version

  values = [
    yamlencode({
      alertmanager = {
        enabled = true
        ingress = {
          enabled = false # Enable if you need external access
        }
      }
      grafana = {
        enabled = false # Datadog for dashboards, disable Grafana to avoid duplication, or enable if desired
      }
      prometheus = {
        enabled = true
        ingress = {
          enabled = false # Enable if you need external access
        }
        prometheusSpec = {
          retention = "15d"
          ruleSelector = {
            matchLabels = {
              prometheus = "kube-prometheus-stack-prometheus"
              role       = "alert-rules"
            }
          }
          serviceMonitorSelector = {
            matchLabels = {
              release = "kube-prometheus-stack"
            }
          }
        }
      }
      kubeApiServer = {
        enabled = true
      }
      kubelet = {
        enabled = true
      }
      kubeControllerManager = {
        enabled = true
      }
      kubeScheduler = {
        enabled = true
      }
      kubeProxy = {
        enabled = true
      }
      kubeStateMetrics = {
        enabled = true
      }
      nodeExporter = {
        enabled = true
      }
    })
  ]
}
    

4. Integrating Prometheus Alertmanager with PagerDuty using Terraform

To send Prometheus alerts to PagerDuty, we need to configure Alertmanager with a PagerDuty receiver and then define an Alertmanager configuration secret. First, set up the PagerDuty service via Terraform:

# Create a PagerDuty service
resource "pagerduty_service" "eks_monitoring_service" {
  name                     = "${var.eks_cluster_name}-monitoring"
  auto_resolve_timeout     = "14400" # 4 hours
  acknowledgement_timeout  = "600"   # 10 minutes
  escalation_policy        = var.pagerduty_escalation_policy_id # Replace with your Escalation Policy ID
  alert_creation           = "create_alerts_and_incidents"
  incident_urgency_rule {
    type    = "constant"
    urgency = "high"
  }
}

# Create a PagerDuty integration for Prometheus Alertmanager
resource "pagerduty_service_integration" "alertmanager_integration" {
  name    = "Prometheus Alertmanager"
  type    = "generic_events_api_v2"
  service = pagerduty_service.eks_monitoring_service.id
}
    

Next, configure Alertmanager using a Kubernetes Secret. The `kube-prometheus-stack` chart allows injecting this configuration.

# Alertmanager configuration to send alerts to PagerDuty
resource "kubernetes_secret" "alertmanager_config" {
  metadata {
    name      = "alertmanager-kube-prometheus-stack-alertmanager"
    namespace = kubernetes_namespace.monitoring.metadata[0].name
    labels = {
      app              = "kube-prometheus-stack-alertmanager"
      app.kubernetes.io/instance = "kube-prometheus-stack"
      app.kubernetes.io/name = "alertmanager"
      app.kubernetes.io/part-of = "kube-prometheus-stack"
      app.kubernetes.io/version = "0.27.0" # Match your Alertmanager version if possible
      operator.prometheus.io/name = "k8s"
      operator.prometheus.io/part-of = "kube-prometheus-stack"
      release            = "kube-prometheus-stack"
    }
  }

  data = {
    "alertmanager.yaml" = yamlencode({
      global = {
        resolve_timeout = "5m"
      }
      route = {
        group_by           = ["alertname", "cluster", "service"]
        group_wait         = "30s"
        group_interval     = "5m"
        repeat_interval    = "4h"
        receiver           = "pagerduty"
        routes = [
          {
            match = {
              severity = "critical"
            }
            receiver = "pagerduty"
          },
          {
            match = {
              severity = "warning"
            }
            receiver = "pagerduty"
          }
        ]
      }
      receivers = [
        {
          name = "pagerduty"
          pagerduty_configs = [
            {
              routing_key  = pagerduty_service_integration.alertmanager_integration.integration_key
              url          = "https://events.pagerduty.com/v2/enqueue"
              severity     = "{{ .CommonLabels.severity }}"
              component    = "{{ .CommonLabels.kubernetes_pod_name }}"
              group        = "{{ .CommonLabels.kubernetes_namespace }}"
              class        = "{{ .CommonLabels.alertname }}"
              # Customize summary and details for PagerDuty events
              description  = "{{ .CommonLabels.alertname }} on {{ .CommonLabels.cluster }} ({{ .CommonLabels.kubernetes_namespace }}/{{ .CommonLabels.kubernetes_pod_name }}) is in state {{ .Status }}"
              details = {
                message = "{{ .CommonAnnotations.message }}"
                summary = "{{ .CommonAnnotations.summary }}"
                dashboard = "{{ if .CommonAnnotations.dashboard }}{{ .CommonAnnotations.dashboard }}{{ else }}N/A{{ end }}"
                link      = "{{ if .CommonAnnotations.runbook_url }}{{ .CommonAnnotations.runbook_url }}{{ else }}N/A{{ end }}"
                # Add more relevant alert details here
              }
            }
          ]
        }
      ]
    })
  }

  type = "Opaque"
  depends_on = [
    helm_release.kube_prometheus_stack,
    pagerduty_service_integration.alertmanager_integration
  ]
}
    

Important: The `pagerduty_escalation_policy_id` for the PagerDuty service resource needs to be an existing Escalation Policy ID from your PagerDuty account. The Alertmanager secret name `alertmanager-kube-prometheus-stack-alertmanager` must precisely match what the `kube-prometheus-stack` Helm chart expects for its Alertmanager configuration.

5. Prometheus Alerting Rules for PagerDuty

Define Prometheus alerting rules using `PrometheusRule` resources. These rules will trigger alerts that Alertmanager then sends to PagerDuty.

resource "kubernetes_manifest" "example_cpu_alert" {
  manifest = {
    apiVersion = "monitoring.coreos.com/v1"
    kind       = "PrometheusRule"
    metadata = {
      name      = "eks-node-alerts"
      namespace = kubernetes_namespace.monitoring.metadata[0].name
      labels = {
        release = "kube-prometheus-stack"
        prometheus = "kube-prometheus-stack-prometheus"
        role       = "alert-rules"
      }
    }
    spec = {
      groups = [
        {
          name = "node-cpu-utilization"
          rules = [
            {
              alert = "HighCpuUsage"
              expr  = "sum(node_cpu_seconds_total{mode!=\"idle\"}) by (instance) / sum(node_cpu_seconds_total) by (instance) * 100 > 90"
              for   = "5m"
              labels = {
                severity = "critical"
                owner    = "devops-team"
              }
              annotations = {
                summary     = "High CPU usage on node {{ $labels.instance }}"
                description = "Node {{ $labels.instance }} has been experiencing high CPU usage for over 5 minutes. Current usage: {{ $value }}%"
                runbook_url = "https://your-runbook-link.com/cpu-troubleshooting"
              }
            },
            {
              alert = "NodeMemoryPressure"
              expr  = "node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 10"
              for   = "10m"
              labels = {
                severity = "warning"
                owner    = "devops-team"
              }
              annotations = {
                summary     = "Node {{ $labels.instance }} is experiencing high memory pressure."
                description = "Node {{ $labels.instance }} has less than 10% memory available for 10 minutes. Current available: {{ $value }}%"
                runbook_url = "https://your-runbook-link.com/memory-troubleshooting"
              }
            }
          ]
        }
      ]
    }
  }
  depends_on = [helm_release.kube_prometheus_stack]
}
    

Unified Observability Workflow

With all components deployed and configured by Terraform, here’s how the unified observability workflow operates:

Metrics Collection: Datadog Agent and Prometheus (via exporters and ServiceMonitors) continuously scrape metrics from EKS nodes, pods, and applications.
Log Collection: Datadog Agent collects all relevant logs from containers and nodes, sending them to Datadog.
APM & Tracing: Datadog APM agents within your applications send traces to Datadog for performance monitoring.
Alerting:
- Prometheus: Evaluates defined `PrometheusRule`s. If a condition is met, it sends an alert to Alertmanager.
- Datadog: Monitors all collected data. Datadog monitors (configured directly in Datadog UI or via Datadog Terraform provider) can also trigger alerts.
Incident Management:
- Alertmanager: Receives alerts from Prometheus, deduplicates, groups, and routes them to PagerDuty based on the `alertmanager.yaml` configuration.
- Datadog: Can be configured to send alerts directly to PagerDuty via its own integration.
- PagerDuty: Receives events, creates incidents, notifies on-call teams according to escalation policies, and tracks incident lifecycle.
Visualization & Debugging:
- Datadog Dashboards: Provide a unified view of all metrics, logs, and traces, allowing for quick correlation and debugging.
- Prometheus UI: Useful for ad-hoc PromQL queries and debugging specific metric issues.

Troubleshooting and Best Practices

Common Issues:

Missing Datadog Metrics/Logs: Check Datadog Agent logs (`kubectl logs -n datadog -l app.kubernetes.io/name=datadog-agent`). Ensure correct API/App keys and site are configured. Verify network policies aren't blocking outbound traffic to Datadog endpoints.
Prometheus Not Scraping: Verify `ServiceMonitor` resources are correctly defined for your applications and that labels match the `serviceMonitorSelector` in the Prometheus configuration. Check Prometheus targets UI.
Alerts Not Reaching PagerDuty:
- Check Alertmanager logs for errors.
- Ensure the `alertmanager.yaml` secret is correctly mounted and configured with the right PagerDuty routing key.
- Verify the PagerDuty integration key is valid and the service is active in PagerDuty.
- Test the PagerDuty integration using `curl` or a test alert from Alertmanager.
Terraform Kubernetes Provider Issues: Ensure your `kubectl` context is correctly set up for the EKS cluster and that the AWS credentials used by Terraform have permission to get EKS cluster details and generate tokens.

Best Practices:

Version Control Everything: Treat your observability configuration (Terraform code, Alertmanager rules, Datadog dashboards if using API) as code.
Least Privilege: Ensure Datadog Agent and Prometheus have only the necessary IAM and Kubernetes RBAC permissions.
Tagging Consistency: Use consistent tagging conventions for AWS resources and Kubernetes labels; this greatly aids in correlation and filtering in Datadog and Prometheus.
Alerting Granularity: Start with critical alerts for PagerDuty and use less intrusive notifications (Slack, email) for warnings or informational alerts.
Runbooks: For every critical alert, have a clear runbook linked in the alert description to guide responders.
Regular Review: Periodically review your alerts, dashboards, and incident response processes to ensure they remain effective and relevant.
Automate Testing: Implement automated tests for your alerting rules to ensure they trigger as expected.

Conclusion

Achieving unified observability on AWS EKS with Terraform, Datadog, Prometheus, and PagerDuty is a strategic move towards a more resilient, performant, and manageable cloud-native infrastructure. By treating your observability stack as Infrastructure as Code, you gain the benefits of automation, consistency, and scalability, ultimately empowering your teams to detect, diagnose, and resolve issues with unparalleled speed and efficiency. Embrace these tools to transform your operational practices and ensure the continuous health of your EKS workloads.

CloudStack YAML