Unified AWS EKS Observability with Prometheus and Datadog via Terraform

In the dynamic world of cloud-native applications, maintaining robust observability for Kubernetes clusters is paramount. AWS Elastic Kubernetes Service (EKS) provides a powerful foundation, but understanding the health, performance, and behavior of your applications and infrastructure requires a sophisticated monitoring strategy. This guide details how to establish a comprehensive and unified observability stack for AWS EKS, leveraging the strengths of Prometheus for deep Kubernetes-native metrics and Datadog for holistic aggregation of metrics, logs, and traces, all orchestrated seamlessly using Terraform for Infrastructure as Code (IaC).

Architecture Pro-Tip: Always prioritize a layered observability approach. While native Kubernetes tools like Prometheus excel at infrastructure and application metrics, integrating with a comprehensive platform like Datadog unifies logs, traces, and advanced analytics, providing a single pane of glass for faster incident response and deeper insights across your entire cloud environment. Automate its deployment with Terraform for consistency and scalability.

The Observability Challenge in EKS

Operating applications on Kubernetes can be complex. You need to monitor not just your application performance, but also the underlying cluster health, node resources, pod lifecycles, and network interactions. Traditional monitoring tools often fall short in this highly ephemeral, distributed environment. A unified strategy is essential to avoid tool sprawl, reduce context switching for engineers, and accelerate issue resolution.

Why a Unified Approach? Prometheus + Datadog

The Power of Prometheus

Prometheus has become the de-facto standard for Kubernetes monitoring due to its cloud-native design principles.

Rich Metrics: Scrapes metrics from Kubernetes components (kube-state-metrics, cAdvisor), nodes, pods, and custom application endpoints.
Powerful Query Language (PromQL): Enables complex data analysis and aggregation.
Alerting: Integrated with Alertmanager for sophisticated alert routing and notification.
Open Source: Large community support and extensibility.

Datadog: The Unified Platform

While Prometheus excels at metric collection, Datadog provides a comprehensive, unified platform for:

Metrics Aggregation: Collects metrics from EKS, Prometheus (via integrations), AWS services, and custom applications.
Log Management: Centralized log collection, indexing, and analysis from all EKS components and applications.
Application Performance Monitoring (APM): Distributed tracing for microservices running on EKS.
Synthetic Monitoring: Proactive testing of application availability and performance.
Network Performance Monitoring: Visibility into EKS service mesh and network flows.
Security Monitoring: Detection of security threats and misconfigurations.
Unified Dashboards & Alerts: Single pane of glass for all observability data, reducing context switching.

Terraform: Infrastructure as Code for Observability

Deploying and managing observability tools manually can be error-prone and time-consuming. Terraform provides the perfect solution:

Automation: Automate the deployment of Prometheus, Datadog Agent, and their configurations.
Consistency: Ensure identical observability setups across environments (dev, staging, prod).
Version Control: Manage your observability infrastructure like application code.
Reproducibility: Easily recreate or scale your monitoring stack.
Drift Detection: Identify and rectify configuration drift.

Architectural Overview

The unified architecture involves:

AWS EKS Cluster: Your managed Kubernetes environment.
Prometheus Operator (via kube-prometheus-stack): Deploys and manages Prometheus, Alertmanager, Grafana, and related service monitors within your EKS cluster. It scrapes metrics from Kubernetes components and applications.
Datadog Agent: Deployed as a DaemonSet and Cluster Agent within EKS. The DaemonSet runs on each node to collect host-level metrics, logs, and traces from pods. The Cluster Agent aggregates cluster-level metrics and orchestrates data collection. It can also scrape Prometheus metrics and forward them to Datadog.
Terraform: Manages the lifecycle of both Prometheus (Helm chart) and Datadog Agent (Helm chart) deployments on EKS, including necessary IAM roles for service accounts (IRSA) if applicable.
Datadog SaaS Platform: The cloud-based analytics and visualization backend for all collected data.

Prerequisites

Before you begin, ensure you have the following:

An active AWS Account with necessary permissions to create/manage EKS clusters and IAM resources.
An existing AWS EKS Cluster. This guide assumes an operational EKS cluster.
Terraform CLI (v1.0.0+) installed and configured with AWS credentials.
Kubectl CLI configured to connect to your EKS cluster.
Helm CLI (v3.0.0+) installed (Terraform will use the Helm provider).
A Datadog Account with your API Key and Application Key.

Implementing Unified Observability with Terraform

We'll use Terraform to deploy the Prometheus kube-prometheus-stack Helm chart and the Datadog Agent Helm chart. This approach provides a robust, version-controlled method for managing your observability infrastructure.

Step 1: Configure AWS EKS Cluster (Base)

This guide assumes your EKS cluster is already provisioned, perhaps using a dedicated Terraform module (e.g., terraform-aws-modules/eks/aws). We will focus on integrating observability components into this existing cluster.

Ensure your Terraform configuration has access to the EKS cluster details, typically via data sources or output variables from your EKS module.

Step 2: Deploying Prometheus (kube-prometheus-stack)

The kube-prometheus-stack Helm chart is a comprehensive solution that bundles Prometheus, Alertmanager, Grafana, and Kubernetes exporters. We'll deploy it into a dedicated monitoring namespace.

Step 3: Deploying Datadog Agent and Cluster Agent

The Datadog Agent is deployed via its official Helm chart. It requires your Datadog API and Application keys. We'll enable critical features like log collection, APM, and process monitoring. The Cluster Agent helps consolidate cluster-level metrics and reduces agent overhead.

Ready-to-Use Terraform Configuration

Below is a consolidated Terraform configuration to deploy both Prometheus (via kube-prometheus-stack) and the Datadog Agent to your EKS cluster. Save this as main.tf in your Terraform project directory.

provider "aws" {
  region = var.aws_region
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}

# --- EKS Cluster Data Sources (Adapt to your existing EKS setup) ---
data "aws_eks_cluster" "cluster" {
  name = var.cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = var.cluster_name
}

# --- Prometheus (kube-prometheus-stack) Deployment ---
resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = "monitoring"
  }
  depends_on = [data.aws_eks_cluster.cluster]
}

resource "helm_release" "kube_prometheus_stack" {
  name       = "kube-prometheus-stack"
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  namespace  = kubernetes_namespace.monitoring.metadata.0.name
  version    = "55.9.1" # Use a specific, stable version
  timeout    = 600

  set {
    name  = "prometheus.prometheusSpec.serviceMonitorSelectorNilUsesPodsTargetLabels"
    value = "false"
  }
  set {
    name  = "prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.storageClassName"
    value = "gp2" # Or your preferred storage class
  }
  set {
    name  = "prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage"
    value = "50Gi"
  }
  set {
    name  = "grafana.enabled"
    value = "true"
  }
  # Consider setting up Grafana ingress for external access if needed.
  # Example:
  # set {
  #   name  = "grafana.ingress.enabled"
  #   value = "true"
  # }
  # set {
  #   name  = "grafana.ingress.annotations.kubernetes\\.io/ingress\\.class"
  #   value = "nginx"
  # }
  # set {
  #   name  = "grafana.ingress.hosts[0]"
  #   value = "grafana.${var.your_domain}"
  # }

  depends_on = [kubernetes_namespace.monitoring]
}

# --- Datadog Agent Deployment ---
resource "kubernetes_namespace" "datadog" {
  metadata {
    name = "datadog"
  }
  depends_on = [data.aws_eks_cluster.cluster]
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = kubernetes_namespace.datadog.metadata.0.name
  version    = "3.2.0" # Use a specific, stable version
  timeout    = 600

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "datadog.site"
    value = var.datadog_site # e.g., "datadoghq.com" or "eu.datadoghq.com"
  }

  set {
    name  = "kubeStateMetricsExternal.enabled"
    value = "true" # Enables Datadog to collect KSM metrics directly
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "agents.enabled"
    value = "true"
  }

  set {
    name  = "logs.enabled"
    value = "true"
  }

  set {
    name  = "logs.containerCollectAll"
    value = "true" # Collect logs from all containers
  }

  set {
    name  = "apm.enabled"
    value = "true"
  }

  set {
    name  = "processAgent.enabled"
    value = "true"
  }

  set {
    name  = "criSocketPath"
    value = "/var/run/containerd/containerd.sock" # Adjust based on your container runtime
  }

  # For fine-grained IAM permissions (IRSA) for Datadog Agent
  # If you need Datadog Agent to access AWS services (e.g., CloudWatch, S3 logs)
  # you'd configure an IAM role and associate it with the service accounts
  # for both the Datadog Agent and Cluster Agent.
  # Example for Cluster Agent:
  # serviceAccount:
  #   create: true
  #   name: datadog-cluster-agent
  #   annotations:
  #     eks.amazonaws.com/role-arn: "arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/datadog-cluster-agent-irsa"

  # Example for Node Agent:
  # agents:
  #   serviceAccount:
  #     create: true
  #     name: datadog-agent
  #     annotations:
  #       eks.amazonaws.com/role-arn: "arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/datadog-node-agent-irsa"

  depends_on = [kubernetes_namespace.datadog]
}

# --- Variables ---
variable "aws_region" {
  description = "AWS region for the EKS cluster."
  type        = string
  default     = "us-east-1"
}

variable "cluster_name" {
  description = "The name of your EKS cluster."
  type        = string
}

variable "datadog_api_key" {
  description = "Your Datadog API Key."
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Your Datadog Application Key."
  type        = string
  sensitive   = true
}

variable "datadog_site" {
  description = "The Datadog site to send metrics to (e.g., 'datadoghq.com', 'eu.datadoghq.com')."
  type        = string
  default     = "datadoghq.com"
}

# variable "your_domain" { # Uncomment if using Grafana ingress example
#   description = "Your domain for Grafana ingress."
#   type        = string
# }
    

To deploy this configuration:

Replace var.cluster_name, var.datadog_api_key, and var.datadog_app_key with your actual values. You can pass these as environment variables (e.g., TF_VAR_datadog_api_key="YOUR_KEY"), via a .tfvars file, or directly in the code (though not recommended for sensitive data).
Initialize Terraform: terraform init
Review the plan: terraform plan
Apply the changes: terraform apply

Verifying Your Unified Observability Stack

Prometheus & Grafana Validation

After successful deployment, you can verify Prometheus data:

Port-forward to Grafana: Find the Grafana service (kubectl get svc -n monitoring) and port-forward to access its UI (kubectl port-forward svc/kube-prometheus-stack-grafana 3000:80 -n monitoring).
Access Prometheus UI: Similarly, port-forward to the Prometheus service (e.g., kubectl port-forward svc/kube-prometheus-stack-prometheus 9090:9090 -n monitoring) and navigate to http://localhost:9090. Query some basic Kubernetes metrics (e.g., up{job="kubernetes-nodes"}).

Datadog UI Validation

Infrastructure -> Kubernetes: You should see your EKS cluster, nodes, and pods reporting metrics.
Logs -> Search: Filter by source:kubernetes or specific pod names to see container logs flowing in.
APM -> Services: If you have instrumented applications, you should see traces appearing.
Integrations: Confirm the Kubernetes integration is active.

Advanced Configuration & Best Practices

Resource Optimization: Adjust CPU/memory requests and limits for Prometheus and Datadog agents based on your cluster size and workload to prevent resource contention.
Persistent Storage: Ensure Prometheus uses persistent storage (e.g., AWS EBS via storage classes) for long-term metric retention.
Prometheus Remote Write: Consider configuring Prometheus to remote write metrics to Datadog (or another long-term store) for extended retention and unified querying, if needed, though Datadog Agent can also scrape Prometheus metrics directly.
IRSA for Datadog Agent: For enhanced security, configure IAM Roles for Service Accounts (IRSA) for the Datadog Agent to grant it minimal necessary AWS permissions without using IAM access keys. This is crucial if your Datadog Agent needs to pull data from other AWS services (e.g., CloudWatch, SQS).
Alerting Strategy: Leverage both Prometheus Alertmanager for Kubernetes-specific alerts and Datadog Monitors for cross-service, aggregated, and AI-driven alerting.
Cost Management: Monitor Datadog ingestion volumes (metrics, logs, traces) to optimize costs. Configure log exclusions or sampling if necessary.
Security: Regularly update Helm charts and agent versions to patch vulnerabilities. Restrict access to Grafana and Prometheus UIs.

Troubleshooting & FAQs

Datadog Agent pods not starting?

Check pod logs (kubectl logs -n datadog -l app=datadog) for API key errors or missing permissions. Ensure your datadog_api_key and datadog_app_key are correct and sensitive variables are handled properly by Terraform. Verify resource requests/limits.

Prometheus not scraping metrics?

Access the Prometheus UI (via port-forward) and navigate to "Status" -> "Targets." Check the status of your scraping targets. Common issues include incorrect service monitor configurations, network policies blocking Prometheus, or applications not exposing metrics in the correct format/port.

High resource usage by agents?

Monitor the CPU and memory usage of Datadog and Prometheus pods. Adjust the resource requests and limits in the Helm chart values. For Datadog, consider disabling certain features (e.g., APM, process monitoring, log collection) if not immediately needed, or tune collection intervals.

Terraform apply fails with EKS authentication errors?

Ensure your AWS credentials are correctly configured and have permission to access the EKS cluster. Your kubeconfig should be set up, and the user/role running Terraform should be mapped in your EKS aws-auth ConfigMap.

Conclusion

Achieving unified observability in AWS EKS is a critical step towards maintaining healthy, performant, and reliable cloud-native applications. By combining the deep, Kubernetes-native metric collection of Prometheus with the comprehensive log, trace, and advanced analytics capabilities of Datadog, and deploying it all with the consistency and automation of Terraform, you create a powerful, future-proof observability stack. This integrated approach not only streamlines operations but also empowers your teams with the insights needed to innovate faster and resolve issues with unparalleled efficiency.

CloudStack YAML