Architecture Pro-Tip: When designing observability for EKS, always prioritize a unified data plane. Consolidating metrics, logs, and traces into a single platform like Datadog, even when leveraging specialized tools like Prometheus for specific metrics, drastically reduces operational complexity and accelerates incident resolution. Ensure robust tagging strategies from the outset for effective filtering and analysis.

Terraform-Managed AWS EKS Observability with Datadog, Prometheus, and PagerDuty Alerts

In the dynamic world of cloud-native applications, maintaining robust observability for Kubernetes clusters is paramount. AWS EKS, as a managed Kubernetes service, simplifies cluster operations, but the responsibility for monitoring and alerting on application and infrastructure health still rests with you. This comprehensive guide will walk you through implementing a sophisticated observability stack for your Terraform-managed AWS EKS clusters, leveraging the power of Datadog for unified monitoring, Prometheus for specialized metrics, and PagerDuty for critical incident alerting.

Why Unified EKS Observability is Critical

Modern distributed applications running on Kubernetes generate vast amounts of telemetry data: metrics, logs, and traces. Without a cohesive strategy to collect, analyze, and act upon this data, teams can quickly become overwhelmed, leading to slow incident response, increased downtime, and degraded user experience. A unified observability approach provides:

End-to-End Visibility: From infrastructure (EC2, EKS control plane) to application code.
Proactive Issue Detection: Identify anomalies and potential problems before they impact users.
Faster MTTR (Mean Time To Resolution): Quickly pinpoint root causes during incidents.
Performance Optimization: Data-driven insights to improve resource utilization and application efficiency.
Compliance & Auditing: Maintain historical records for regulatory requirements.

Core Components of Our Observability Stack

This guide focuses on integrating three powerful tools into your EKS environment:

Terraform: For declarative provisioning and management of your AWS EKS cluster, its associated resources, and the deployment of observability agents.
Datadog: A comprehensive SaaS monitoring and analytics platform that consolidates metrics, logs, and traces from your EKS cluster, applications, and AWS services. It offers powerful dashboards, machine learning-driven anomaly detection, and robust alerting capabilities.
Prometheus (via kube-state-metrics): While Datadog can collect most metrics, kube-state-metrics exposes a wealth of Kubernetes-native metrics (like pod phase, deployment status) in a Prometheus-compatible format, which Datadog can easily scrape and ingest. This provides deeper Kubernetes internal state visibility.
PagerDuty: The industry-leading incident management platform. Datadog will integrate directly with PagerDuty to route critical alerts to on-call teams, ensuring rapid notification and escalation.

Prerequisites

Before you begin, ensure you have the following:

An active AWS account with appropriate IAM permissions to create EKS clusters and related resources.
Terraform (v1.0+) installed and configured.
kubectl installed and configured to interact with your EKS cluster.
Helm (v3+) installed.
A Datadog account with an API key and Application key.
A PagerDuty account with an Integration Key for a service.

Step-by-Step Implementation Guide

1. Provision AWS EKS Cluster with Terraform

First, we'll provision our EKS cluster using Terraform. This example assumes you have a VPC, subnets, and an IAM role for EKS already set up. If not, you'd include those resources in your Terraform configuration.

A typical EKS Terraform setup involves modules for cleaner organization. For brevity, here's a simplified representation:

resource "aws_eks_cluster" "main" {
  name     = "my-eks-cluster"
  role_arn = aws_iam_role.eks_master.arn
  version  = "1.28" # Or your desired version

  vpc_config {
    subnet_ids         = var.private_subnet_ids
    security_group_ids = [aws_security_group.eks_cluster.id]
  }

  tags = {
    Environment = "production"
    ManagedBy   = "Terraform"
  }
}

resource "aws_eks_node_group" "default" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "default-node-group"
  node_role_arn   = aws_iam_role.eks_nodes.arn
  subnet_ids      = var.private_subnet_ids
  instance_types  = ["t3.medium"]
  disk_size       = 20
  capacity_type   = "ON_DEMAND"

  scaling_config {
    desired_size = 2
    max_size     = 5
    min_size     = 2
  }

  labels = {
    "app" = "general"
  }

  tags = {
    Environment = "production"
    ManagedBy   = "Terraform"
  }
}

output "kubeconfig" {
  value = <<-EOT
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ${aws_eks_cluster.main.certificate_authority.0.data}
    server: ${aws_eks_cluster.main.endpoint}
  name: ${aws_eks_cluster.main.name}
contexts:
- context:
    cluster: ${aws_eks_cluster.main.name}
    user: ${aws_eks_cluster.main.name}
  name: ${aws_eks_cluster.main.name}
current-context: ${aws_eks_cluster.main.name}
kind: Config
preferences: {}
users:
- name: ${aws_eks_cluster.main.name}
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      command: aws
      args:
        - "eks"
        - "get-token"
        - "--cluster-name"
        - "${aws_eks_cluster.main.name}"
        - "--region"
        - "${var.aws_region}"
      installHint: |
        The aws-cli must be installed to use this authenticator;
        see https://docs.aws.amazon.com/cli/latest/userguide/install-aws-cli.html
EOT
  sensitive = true
}
    

After applying this Terraform, configure your kubectl to connect to the new cluster using the generated kubeconfig output.

2. Deploy Datadog Agent with Terraform (Helm Provider)

The Datadog Agent is the backbone of your Datadog observability. We'll deploy it to EKS using Terraform's Helm provider, ensuring all components (metrics, logs, APM, process) are enabled.

Ensure you have your Datadog API and Application keys ready. Store them securely, e.g., in AWS Secrets Manager or environment variables, and reference them in Terraform.

# main.tf (excerpt)

# Configure Kubernetes provider
resource "kubernetes_secret" "datadog_api_key" {
  metadata {
    name      = "datadog-api-key"
    namespace = "default" # Or your observability namespace
  }
  data = {
    "api-key" = var.datadog_api_key
  }
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Or your observability namespace
  version    = "3.1.20"  # Use a specific, stable version

  set {
    name  = "datadog.apiKeyExistingSecret"
    value = kubernetes_secret.datadog_api_key.metadata.0.name
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key # Often not directly needed by agent, but good practice to reference if using Datadog provider
    # value = var.datadog_app_key is more for Datadog provider resources like monitors, not the agent helm chart itself.
    # For helm chart, api_key_existing_secret is usually enough. Let's simplify this.
  }

  set {
    name  = "clusterAgent.enabled"
    value = true
  }
  set {
    name  = "clusterAgent.metricsProvider.enabled"
    value = true
  }
  set {
    name  = "datadog.site"
    value = "datadoghq.com" # Or your Datadog site, e.g., us3.datadoghq.com
  }

  # Enable APM, logs, and process collection
  set {
    name  = "apm.enabled"
    value = true
  }
  set {
    name  = "logs.enabled"
    value = true
  }
  set {
    name  = "logs.containerCollectAll"
    value = true
  }
  set {
    name  = "processAgent.enabled"
    value = true
  }

  # Enable Prometheus integration for scraping
  set {
    name  = "datadog.confd.prometheus_kube_state_metrics.yaml"
    value = <<-EOT
instances:
  - prometheus_url: http://kube-state-metrics.kube-system.svc.cluster.local:8080/metrics
    tags:
      - custom_tag:kube_state_metrics
    EOT
  }

  # Add tags to all collected metrics/logs/traces
  set {
    name  = "datadog.tags[0]"
    value = "environment:production"
  }
  set {
    name  = "datadog.tags[1]"
    value = "cluster_name:my-eks-cluster"
  }

  # Values for variables (e.g., in variables.tf)
  # variable "datadog_api_key" {
  #   description = "Datadog API Key"
  #   type        = string
  #   sensitive   = true
  # }
  # variable "datadog_app_key" {
  #   description = "Datadog Application Key (for Datadog provider resources)"
  #   type        = string
  #   sensitive   = true
  # }
}
    

This configuration deploys the Datadog Agent, Cluster Agent, enables APM, log collection, process monitoring, and sets up a basic Prometheus scrape configuration for kube-state-metrics (which we'll deploy next). Remember to replace placeholders with your actual values.

3. Deploy `kube-state-metrics` with Terraform (Helm Provider)

kube-state-metrics is an add-on that listens to the Kubernetes API server and generates metrics about the state of Kubernetes objects (pods, deployments, nodes, etc.). Datadog can then scrape these metrics.

# main.tf (excerpt)

resource "helm_release" "kube_state_metrics" {
  name       = "kube-state-metrics"
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-state-metrics"
  namespace  = "kube-system" # Standard namespace for KSM
  version    = "2.22.0"      # Use a specific, stable version

  set {
    name  = "podAnnotations.prometheus.io/scrape"
    value = "true"
  }
  set {
    name  = "podAnnotations.prometheus.io/port"
    value = "8080"
  }
  set {
    name  = "prometheus.io/probe"
    value = "true" # This is a common annotation for Datadog's autodiscovery
  }
}
    

After applying, kube-state-metrics will be running in the kube-system namespace, exposing metrics on port 8080. The Datadog Agent's Prometheus check (configured in the previous step) will automatically pick these up.

4. Configure Datadog Observability

With the agents deployed, Datadog will start ingesting data. Here's how to make the most of it:

Metrics: Explore the Metrics Explorer to visualize host, container, pod, and application metrics. Datadog provides out-of-the-box dashboards for EKS, Kubernetes, and various integrations.
Logs: Use the Log Explorer to search, filter, and analyze logs from all your containers. Create log patterns and facets for easier navigation and anomaly detection.
APM & Tracing: If your applications are instrumented with Datadog APM libraries, you'll see traces, service maps, and detailed performance insights under the APM section.
Dashboards: Create custom dashboards to visualize key performance indicators (KPIs) relevant to your services. Leverage Datadog's templating variables for dynamic views across environments or clusters.
Monitors: Set up monitors for critical metrics (CPU utilization, memory usage, network errors, pod restarts), log anomalies, or trace errors.

You can also manage Datadog resources like dashboards, monitors, and synthetic tests directly via the Terraform Datadog Provider for GitOps best practices.

5. Setting up Datadog Monitors and PagerDuty Alerts

Integrating PagerDuty ensures that critical alerts from Datadog reach the right people immediately. First, configure the PagerDuty integration in Datadog:

In Datadog, navigate to Integrations -> Integrations.
Search for "PagerDuty" and install the integration.
Add a new account, providing a service name and the PagerDuty integration key (API Key or Events API V2 Routing Key).

Now, create a Datadog monitor that uses this integration. Here's an example of a critical EKS node CPU utilization monitor, expressed as a Terraform resource:

# main.tf (excerpt)

# Ensure you have the Datadog provider configured with your API and App keys
# provider "datadog" {
#   api_key = var.datadog_api_key
#   app_key = var.datadog_app_key
#   api_url = "https://api.datadoghq.com/" # Or your specific Datadog site
# }

resource "datadog_monitor" "eks_node_cpu_critical" {
  name                = "[EKS] Node CPU Critical - {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:system.cpu.idle{eks_cluster_name:my-eks-cluster} by {host} < 20" # Alert if idle CPU is less than 20% (i.e., usage > 80%)
  message             = <<-EOT
@pagerduty-your_service_name
EKS Node CPU usage is critically high on {{host.name}} ({{value}}% used).
This could indicate resource contention or an overloaded node.
View CPU Usage in Datadog
EOT
  tags                = ["environment:production", "severity:critical", "service:eks"]
  require_full_window = true
  thresholds {
    critical = 20
  }
  notify_no_data      = false
  new_group_delay     = 60
  no_data_timeframe   = 30
  renotify_interval   = 0
  escalation_message  = "Node CPU still critical after 15 minutes. Investigating further."
}

resource "datadog_monitor" "eks_pod_restarts_critical" {
  name                = "[EKS] Pod Restarts Critical - {{kube_container_name}} on {{kube_pod_name}}"
  type                = "metric alert"
  query               = "sum(last_5m):kubernetes.containers.restarts{eks_cluster_name:my-eks-cluster} by {kube_container_name,kube_pod_name} > 3"
  message             = <<-EOT
@pagerduty-your_service_name
EKS Pod {{kube_pod_name}}/{{kube_container_name}} is experiencing excessive restarts ({{value}} in 5 minutes).
This indicates a stability issue within the application or infrastructure.
View Containers in Datadog
EOT
  tags                = ["environment:production", "severity:critical", "service:eks", "alert_type:restarts"]
  thresholds {
    critical = 3
  }
  notify_no_data    = false
  new_group_delay   = 60
  renotify_interval = 0
}
    

In the `message` field, @pagerduty-your_service_name ensures the alert is routed to the specified PagerDuty service. Replace your_service_name with the actual name configured in Datadog for your PagerDuty integration.

Best Practices for EKS Observability

Tagging Consistency: Implement a strict tagging policy across all AWS and Kubernetes resources. This allows for powerful filtering, aggregation, and cost allocation in Datadog.
SLOs & SLIs: Define Service Level Objectives (SLOs) and Service Level Indicators (SLIs) for your critical applications and services, and configure Datadog monitors to track them.
Cost Optimization: Monitor Datadog agent resource consumption (CPU/Memory) on your EKS nodes. Optimize logging ingestion filters to avoid sending unnecessary data to Datadog.
Security Best Practices: Use Kubernetes Secrets for sensitive information like API keys. Implement fine-grained RBAC for the Datadog Agent and Cluster Agent.
Automate Everything: Leverage Terraform for managing not just your EKS infrastructure but also your Datadog dashboards, monitors, and PagerDuty integrations for true Infrastructure-as-Code.
Regular Review: Periodically review your monitors and dashboards. Remove irrelevant ones and create new ones as your application and infrastructure evolve.

Troubleshooting Common Issues

No Datadog Data:
- Verify the Datadog Agent pods are running and healthy: kubectl get pods -n default -l app=datadog.
- Check agent logs for errors: kubectl logs <datadog-agent-pod-name> -n default.
- Ensure the correct Datadog API key is provided and valid.
- Check network connectivity from EKS nodes to Datadog endpoints.
Prometheus Metrics Missing:
- Verify kube-state-metrics pods are running in kube-system.
- Confirm the Datadog Agent's Prometheus check configuration (datadog.confd.prometheus_kube_state_metrics.yaml) is correct and points to the right service/port.
- Access http://kube-state-metrics.kube-system.svc.cluster.local:8080/metrics from a debug pod within your cluster to ensure metrics are exposed.
PagerDuty Alerts Not Firing:
- Check Datadog's PagerDuty integration status under Integrations -> PagerDuty.
- Ensure the @pagerduty-your_service_name syntax in the monitor message is correct and matches the service name configured in the integration.
- Verify the PagerDuty integration key is valid and associated with the correct PagerDuty service.

Conclusion

Establishing a robust and automated observability pipeline is non-negotiable for critical AWS EKS workloads. By harnessing Terraform for infrastructure-as-code, Datadog for unified monitoring, Prometheus (via kube-state-metrics) for detailed Kubernetes state, and PagerDuty for reliable incident alerting, you empower your DevOps teams with the insights and tools needed to proactively manage, troubleshoot, and optimize your cloud-native applications. This setup not only improves your operational efficiency but also significantly enhances the reliability and performance of your EKS deployments.

CloudStack YAML