Standardizing AWS EKS Observability and PagerDuty Integration with Terraform and Datadog

Architecture Pro-Tip:

Always design your observability stack with a "single pane of glass" philosophy. Integrating all logs, metrics, and traces into a centralized platform like Datadog, managed by Infrastructure as Code (Terraform), ensures consistency, reduces operational overhead, and accelerates incident response when PagerDuty is triggered. Prioritize automation from day one to avoid configuration drift and manual errors.

In the dynamic landscape of cloud-native applications, maintaining robust observability for Amazon Elastic Kubernetes Service (EKS) clusters is paramount. As organizations scale, the challenge isn't just about collecting data, but standardizing how that data is collected, monitored, and acted upon. This guide provides a comprehensive technical approach to achieving standardized AWS EKS observability and integrating with PagerDuty for incident management, all orchestrated using Terraform and leveraging Datadog as the central monitoring platform.

The Imperative for Standardized EKS Observability

Unstandardized observability leads to fragmented insights, slower incident resolution, and increased operational burden. For EKS environments, this often manifests as:

Inconsistent Monitoring: Different teams or clusters use varying tools or configurations, making cross-cluster analysis difficult.
Manual Configuration Drift: Lack of Infrastructure as Code (IaC) leads to manual changes that are undocumented and prone to error.
Delayed Incident Response: Poorly defined alerts or notification pathways result in missed critical events.
High Cognitive Load: Engineers spend more time understanding different monitoring setups than resolving issues.

Standardization with Terraform, Datadog, and PagerDuty addresses these challenges by automating the deployment of observability agents, defining consistent monitoring rules, and establishing clear incident response workflows.

Key Technologies in Focus

Terraform: Infrastructure as Code (IaC) for Everything

Terraform is central to this strategy, enabling the declarative definition of your entire observability stack. This includes:

Deploying the Datadog Agent to EKS clusters via Helm.
Configuring Datadog monitors, dashboards, and integration settings.
Managing PagerDuty services, escalation policies, and schedules (often indirectly via Datadog's integration).

Using Terraform ensures repeatability, version control, and auditability for your observability infrastructure.

Datadog: Unified Observability Platform

Datadog provides an end-to-end view of your EKS applications and infrastructure. It collects:

Metrics: From Kubernetes components, nodes, pods, and applications.
Logs: Consolidated from all containers and services.
Traces: Distributed tracing for microservices.
Events: Kubernetes events for operational context.

Its powerful alerting engine and rich visualization capabilities make it ideal for EKS monitoring.

PagerDuty: Incident Management and On-Call Automation

When critical issues are detected by Datadog, PagerDuty steps in to ensure the right people are notified at the right time. Its capabilities include:

Intelligent alerting and notification across multiple channels.
On-call scheduling and escalation policies.
Post-incident analysis and reporting.

The integration between Datadog and PagerDuty is crucial for turning detected problems into actionable incidents.

Implementing Standardized EKS Observability with Terraform

Step 1: Datadog Agent Deployment on EKS

The Datadog Agent is deployed as a DaemonSet to collect cluster-level metrics, logs, and APM traces. Leveraging the official Datadog Helm chart is the recommended approach, which can be managed via Terraform's helm_release resource.

Before deployment, ensure you have your Datadog API and APP keys. These are sensitive and should be managed securely, e.g., via AWS Secrets Manager and referenced in your Terraform setup.

Step 2: Configuring Datadog Monitors via Terraform

Terraform allows you to define Datadog monitors declaratively. This ensures that all EKS clusters adhere to the same alerting standards. Common monitors include:

Node Health: CPU/memory utilization, disk pressure.
Pod/Container Health: Restarts, OOMKills, CPU/memory requests/limits.
Kubernetes Control Plane: API server latency, scheduler/controller-manager health.
Application-Specific Metrics: HTTP error rates, latency, custom business metrics.

The datadog_monitor resource is used for this, allowing you to specify query logic, thresholds, and notification messages.

Step 3: Integrating PagerDuty with Datadog

The primary integration point is from Datadog to PagerDuty. Datadog acts as the alert generator, and PagerDuty as the incident responder. This integration is set up in Datadog, often via its UI or by using the datadog_integration_pagerduty Terraform resource (though simpler PagerDuty services are often managed directly in PagerDuty and linked via a Datadog monitor message).

Within your Datadog monitors, you'll specify @pagerduty-your-service-name in the notification message to route alerts to specific PagerDuty services. Each PagerDuty service should correspond to a logical component or team responsible for incidents related to that service.

Step 4: Centralized Terraform Modules for Reusability

To truly standardize, encapsulate your EKS observability configurations within reusable Terraform modules. This means having modules for:

Datadog Agent: A module that deploys the Datadog Agent Helm chart with common configurations.
Standard EKS Monitors: A module that deploys a set of baseline Datadog monitors applicable to all EKS clusters (e.g., node resource alerts, API server health).
PagerDuty Integration Setup: While often simpler, a module could manage the Datadog-PagerDuty integration details.

This approach allows you to deploy a fully configured observability stack with minimal code duplication across multiple EKS clusters or environments.

Ready-to-Use Configuration Examples

Here are Terraform configuration snippets illustrating the key components:

Terraform for Datadog Agent Deployment (using Helm)

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog" # Ensure this namespace exists or is created
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    # Best practice: use Kubernetes secrets for sensitive values
    # valueFrom = {
    #   secretKeyRef = {
    #     name = "datadog-api-key"
    #     key  = "api-key"
    #   }
    # }
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    # Similar to API key, use secrets
  }

  set {
    name  = "clusterAgent.enabled"
    value = true
  }

  set {
    name  = "kubeStateMetricsExternal.enabled"
    value = false # Datadog Agent usually includes its own KSM
  }

  set {
    name  = "targetSystem"
    value = "linux"
  }

  set {
    name  = "datadog.site"
    value = "datadoghq.com" # Or eu.datadoghq.com, etc.
  }

  # Add any additional EKS-specific settings
  set {
    name  = "providers.aws.tag_collection_enabled"
    value = true
  }

  set {
    name  = "tags"
    value = "{env:${var.environment},cluster_name:${var.cluster_name}}"
  }
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog APP Key"
  type        = string
  sensitive   = true
}

variable "environment" {
  description = "Environment tag for Datadog"
  type        = string
}

variable "cluster_name" {
  description = "EKS Cluster Name tag for Datadog"
  type        = string
}
    

Terraform for a Standard Datadog Monitor (EKS Node CPU Utilization)

resource "datadog_monitor" "eks_node_cpu_high" {
  name               = "[EKS-${var.environment}-${var.cluster_name}] High Node CPU Utilization"
  type               = "metric alert"
  query              = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.cluster_name},kube_namespace:default} by {host} > 80"
  message            = <Terraform for Datadog-PagerDuty Integration (Optional, often done via UI)While individual PagerDuty services are typically managed within PagerDuty and referenced in Datadog monitors, you can configure the overarching integration via Terraform:

# This resource configures the global Datadog-PagerDuty integration.
# Note: You typically only need ONE of these per Datadog organization.
# Individual PagerDuty services are then referenced in Datadog monitor messages.
resource "datadog_integration_pagerduty" "main" {
  # This token is a PagerDuty API token for the Datadog integration,
  # not a service integration key.
  api_token = var.pagerduty_api_token
}

variable "pagerduty_api_token" {
  description = "PagerDuty API token (for Datadog global integration)"
  type        = string
  sensitive   = true
}
    
Best Practices for EKS ObservabilityTagging Consistency: Enforce consistent tagging for all AWS resources and Kubernetes objects. This is crucial for filtering and grouping in Datadog.
Alert Fatigue Management: Optimize alert thresholds and use composite monitors to reduce alert noise. PagerDuty escalation policies should be carefully crafted.
Granular Permissions: Apply the principle of least privilege to both Datadog API keys and Kubernetes service accounts for the Datadog Agent.
Version Control Everything: Store all Terraform configurations in a Git repository. Implement pull request workflows for changes.
Regular Review: Periodically review your monitors, dashboards, and PagerDuty policies. As your EKS environment evolves, so should your observability.


Troubleshooting Common IssuesDatadog Agent Not Reporting: Check Kubernetes logs for the Datadog Agent pods (kubectl logs -n datadog -l app=datadog). Verify API and APP keys are correct and have access. Ensure network policies allow egress to Datadog endpoints.
Alerts Not Triggering PagerDuty: Confirm the @pagerduty-your-service-name syntax is correct in your Datadog monitor message. Check the PagerDuty integration status in Datadog. Verify the PagerDuty service exists and has active on-call schedules.
Terraform Apply Failures: Ensure your AWS credentials have sufficient permissions to create IAM roles/policies, EKS resources, and manipulate Datadog/PagerDuty resources. Check provider configurations for correct API keys/tokens.
Missing Metrics/Logs: Review Datadog Agent configuration for specific integrations (e.g., APM, log collection paths). Check for resource limits on Datadog Agent pods that might be causing throttling.
ConclusionStandardizing AWS EKS observability and PagerDuty integration with Terraform and Datadog is not just a best practice; it's a critical enabler for reliable, scalable cloud-native operations. By embracing Infrastructure as Code, you establish a consistent, automated, and auditable observability pipeline that accelerates incident resolution, minimizes operational toil, and provides deep insights into the health and performance of your EKS clusters. This approach empowers your DevOps teams to focus on innovation rather than firefighting.

CloudStack YAML