Terraform Configuration for AWS EKS Production Observability: Datadog & PagerDuty Integration

In the dynamic world of cloud-native applications, maintaining robust observability for Kubernetes clusters is paramount. AWS Elastic Kubernetes Service (EKS) provides a powerful foundation, but its full potential is unlocked when coupled with comprehensive monitoring and incident response systems. This guide delves into configuring Datadog for deep Kubernetes observability and integrating it with PagerDuty for streamlined incident management, all orchestrated predictably and scalably with Terraform.

Architecture Pro-Tip: Observability-First Design

Integrate observability tools and practices from the very beginning of your AWS EKS cluster deployment. Treating observability as a core component of your Infrastructure as Code (IaC) ensures consistent, reliable monitoring and incident response capabilities, preventing costly blind spots in production and fostering a proactive operational culture.

Why Terraform for EKS Observability?

Terraform, as an Infrastructure as Code (IaC) tool, brings immense value to managing complex cloud environments like AWS EKS, especially when it comes to observability. Its declarative nature ensures that your monitoring and alerting infrastructure is defined, version-controlled, and deployed consistently across environments.

Consistency: Define Datadog agents, monitors, dashboards, and PagerDuty services uniformly.
Automation: Automate the deployment and updates of observability components alongside your EKS cluster.
Version Control: Track changes to your observability stack, enabling rollbacks and clear audit trails.
Scalability: Easily replicate configurations for multiple clusters or environments.
Reduced Manual Error: Eliminate the risks associated with manual configuration.

Prerequisites

Before you begin, ensure you have the following:

An active AWS Account with appropriate IAM permissions to manage EKS and related resources.
Terraform CLI installed (version 1.0+ recommended).
An existing AWS EKS cluster. This guide assumes your EKS cluster is already provisioned or will be provisioned using Terraform separately.
A Datadog Account with an API Key and Application Key.
A PagerDuty Account with an API Key.
Helm CLI installed for local chart value generation, though Terraform will deploy it.
kubectl configured to connect to your EKS cluster.

Core Components & Their Roles

Datadog for Comprehensive Monitoring

Datadog provides end-to-end visibility across your EKS environment, from node metrics to container logs and application traces. Key components include:

Datadog Agent: A DaemonSet deployed on your EKS cluster that collects metrics, logs, and traces from nodes, pods, and services.
Monitors: Alerting rules defined in Datadog based on collected metrics or logs.
Dashboards: Visual representations of your EKS cluster's health and performance.

PagerDuty for Incident Management

PagerDuty acts as your incident response hub, ensuring that critical alerts from Datadog reach the right on-call team members promptly. Its role involves:

Services: Represent specific applications or components that PagerDuty monitors.
Integrations: Connect monitoring tools (like Datadog) to PagerDuty services to trigger incidents.
Escalation Policies: Define how incidents are escalated through teams or individuals until acknowledged.

Terraform Configuration Walkthrough

1. Provider Configuration

First, define the necessary providers: aws, kubernetes, helm, datadog, and pagerduty. Ensure your API keys and AWS credentials are securely passed, ideally via environment variables or a secrets manager.

provider "aws" {
  region = "us-east-1" # Replace with your AWS region
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.eks_cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.eks_cluster_auth.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.eks_cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.eks_cluster_auth.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

data "aws_eks_cluster" "eks_cluster" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "eks_cluster_auth" {
  name = var.eks_cluster_name
}

variable "eks_cluster_name" {
  description = "Name of the existing EKS cluster"
  type        = string
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

2. Deploying the Datadog Agent with Helm

The Datadog Agent is best deployed as a DaemonSet using its official Helm chart. This ensures an agent runs on every worker node in your EKS cluster. You'll need to pass your Datadog API key and enable various features like APM, logging, and process monitoring.

resource "kubernetes_namespace" "datadog" {
  metadata {
    name = "datadog"
  }
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = kubernetes_namespace.datadog.metadata.0.name

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "datadog.site"
    value = "datadoghq.com" # Or eu.datadoghq.com for EU site
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "kubeStateMetricsExternal.enabled"
    value = "true"
  }

  set {
    name  = "targetSystem"
    value = "linux"
  }

  set {
    name  = "datadog.logs.enabled"
    value = "true"
  }

  set {
    name  = "datadog.logs.containerCollectAll"
    value = "true"
  }

  set {
    name  = "datadog.apm.enabled"
    value = "true"
  }

  set {
    name  = "datadog.processAgent.enabled"
    value = "true"
  }

  # EKS-specific configuration
  set {
    name  = "datadog.kubelet.tlsVerify"
    value = "false"
  }

  set {
    name  = "datadog.criSocketPath"
    value = "/var/run/dockershim.sock" # Adjust based on your EKS runtime (containerd/dockershim)
  }

  # RBAC for Datadog Agent
  set {
    name  = "rbac.create"
    value = "true"
  }
}

3. Configuring PagerDuty Service and Integration

Next, define your PagerDuty service. This service will represent your EKS cluster or a specific application running on it. You'll also create an integration for Datadog, which will generate a unique integration key.

resource "pagerduty_service" "eks_observability_service" {
  name                     = "EKS Production Observability - ${var.eks_cluster_name}"
  auto_resolve_timeout     = 14400 # 4 hours
  acknowledgement_timeout = 600   # 10 minutes
  escalation_policy        = var.pagerduty_escalation_policy_id # Replace with your Escalation Policy ID
  alert_creation           = "create_alerts_and_incidents"
}

resource "pagerduty_integration" "datadog_integration" {
  name    = "Datadog Integration"
  type    = "generic_events_api_inbound_integration"
  service = pagerduty_service.eks_observability_service.id
}

variable "pagerduty_escalation_policy_id" {
  description = "The ID of the PagerDuty escalation policy to associate with the service."
  type        = string
}

output "pagerduty_datadog_integration_key" {
  description = "PagerDuty integration key for Datadog"
  value       = pagerduty_integration.datadog_integration.integration_key
  sensitive   = true
}

4. Integrating Datadog with PagerDuty

With the PagerDuty integration key, you can now configure Datadog to send alerts to PagerDuty. This is done using the datadog_integration_pagerduty resource, mapping a Datadog integration to your PagerDuty service.

resource "datadog_integration_pagerduty" "pagerduty_setup" {
  services = [
    {
      service_name = pagerduty_service.eks_observability_service.name
      service_key  = pagerduty_integration.datadog_integration.integration_key
    }
  ]
}

5. Defining Datadog Monitors

Finally, create your Datadog monitors. These define the conditions under which an alert should be triggered. In the message field, you'll reference the PagerDuty service using the @pagerduty-{{SERVICE_NAME}} syntax to ensure alerts are routed correctly.

Below is an example for a critical EKS node CPU utilization monitor:

resource "datadog_monitor" "eks_node_cpu_utilization" {
  name                = "[EKS] Production Node CPU Critical - {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster_name:${var.eks_cluster_name},kube_namespace:default} by {host} < 20"
  message             = <<EOT
  @pagerduty-${pagerduty_service.eks_observability_service.name}
  High CPU utilization detected on EKS node {{host.name}} in cluster ${var.eks_cluster_name}.
  Current idle CPU is {{value}}%. Investigate immediately.
  EOT
  tags                = ["environment:production", "team:devops", "kubernetes_cluster:${var.eks_cluster_name}"]
  no_data_timeframe   = 20
  new_host_delay      = 300
  notify_no_data      = false
  renotify_interval   = 0
  escalation_message  = "CPU utilization remains high. Escalating to next level."
  include_tags        = true
  require_full_window = false

  thresholds {
    critical = 20
    warning  = 30
  }
}

resource "datadog_monitor" "eks_node_memory_utilization" {
  name                = "[EKS] Production Node Memory Critical - {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:system.mem.used{kubernetes_cluster_name:${var.eks_cluster_name}} by {host} > 90"
  message             = <<EOT
  @pagerduty-${pagerduty_service.eks_observability_service.name}
  High Memory utilization detected on EKS node {{host.name}} in cluster ${var.eks_cluster_name}.
  Current memory usage is {{value}}%. Investigate immediately.
  EOT
  tags                = ["environment:production", "team:devops", "kubernetes_cluster:${var.eks_cluster_name}"]
  no_data_timeframe   = 20
  new_host_delay      = 300
  notify_no_data      = false
  renotify_interval   = 0
  escalation_message  = "Memory utilization remains high. Escalating to next level."
  include_tags        = true
  require_full_window = false

  thresholds {
    critical = 90
    warning  = 80
  }
}

Applying the Configuration

To deploy this configuration, save the code blocks into .tf files (e.g., main.tf, variables.tf). Then, initialize and apply Terraform:

terraform init
terraform plan
terraform apply

Ensure you provide the sensitive variables (Datadog API/App keys, PagerDuty API token, Escalation Policy ID) either through environment variables (TF_VAR_datadog_api_key, etc.) or interactively during terraform apply.

Best Practices for Production Observability

Granular Monitoring: Beyond node-level metrics, deploy Datadog integrations for specific applications (e.g., Redis, PostgreSQL) and custom metrics for your microservices.
Log Management: Ensure all EKS pod logs are collected by Datadog and indexed for searchable troubleshooting. Implement log-based monitors for critical application errors.
Distributed Tracing (APM): Instrument your applications for APM to gain visibility into request flows, latency, and error rates across microservices.
Synthetic Monitoring: Use Datadog Synthetics to proactively test your application's availability and performance from an end-user perspective.
Alert Fatigue Reduction: Carefully tune your Datadog monitors to reduce noise. Use composite monitors, anomaly detection, and suppression rules.
Role-Based Access Control (RBAC): Implement strict RBAC for your Kubernetes and Datadog/PagerDuty users. Terraform can manage Datadog user roles and PagerDuty teams.
Tagging Strategy: Consistently tag all resources (AWS, Kubernetes, Datadog monitors/dashboards) with environment, team, service, and application details for better filtering and context.
Documentation: Maintain clear documentation for your observability stack, including alert runbooks for PagerDuty incidents.

Troubleshooting & FAQ

Q: Datadog Agent pods are not running or are in a CrashLoopBackOff state.

A: Check the logs of the Datadog Agent pods (kubectl logs -n datadog <pod-name>). Common issues include incorrect API/App keys, insufficient RBAC permissions, or issues with the CRI socket path. Ensure the datadog.criSocketPath in the Helm chart values matches your EKS runtime (e.g., /var/run/dockershim.sock for Docker, /var/run/containerd/containerd.sock for Containerd).

Q: Datadog monitors are not triggering PagerDuty incidents.

A: Verify the following:

The Datadog monitor's message field correctly references the PagerDuty service (e.g., @pagerduty-EKS Production Observability - my-cluster). The service name must match exactly.
The datadog_integration_pagerduty resource successfully linked Datadog and PagerDuty. Check the Datadog UI under Integrations -> PagerDuty.
The PagerDuty service and its associated escalation policy are correctly configured and active.

Q: Terraform fails to authenticate with Kubernetes or Helm.

A: Ensure your AWS credentials are correctly configured for the region. The aws_eks_cluster_auth data source relies on your AWS CLI/SDK configuration to generate the EKS token. Also, verify that the IAM user/role running Terraform has permissions to access the EKS cluster and its authentication endpoint.

Conclusion

Establishing robust production observability for AWS EKS is a non-negotiable aspect of modern cloud operations. By leveraging Terraform to configure Datadog for comprehensive monitoring and PagerDuty for efficient incident response, you create an "observability as code" pipeline that is automated, consistent, and scalable. This approach not only enhances operational efficiency but also significantly improves your team's ability to quickly detect, diagnose, and resolve issues, ensuring the reliability and performance of your mission-critical Kubernetes workloads.

CloudStack YAML