Terraform-managed AWS EKS Observability with Datadog and PagerDuty Incident Response

Terraform-Managed AWS EKS Observability with Datadog and PagerDuty Incident Response

In today's fast-paced cloud-native landscape, ensuring the health, performance, and reliability of your Kubernetes clusters is paramount. AWS Elastic Kubernetes Service (EKS) offers a robust foundation, but true operational excellence demands a comprehensive observability strategy coupled with efficient incident response. This guide provides a detailed technical walkthrough on how to implement a fully automated, Terraform-managed observability stack for AWS EKS using Datadog for monitoring and PagerDuty for streamlined incident management.

Architecture Pro-Tip

Design your observability stack from the ground up, integrating it into your Infrastructure as Code (IaC) early. This ensures consistent, reproducible deployments and avoids manual configuration drift. Prioritize agent-based collection for deep insights and API-driven integrations for platform-level metrics. Centralize your monitoring and alerting platforms to reduce alert fatigue and accelerate Mean Time To Resolution (MTTR).

Why Terraform for EKS Observability?

Terraform's declarative approach allows you to define and manage your entire cloud infrastructure, including your EKS cluster, its associated services, and crucially, your observability tooling. This offers significant advantages:

Consistency: Ensures identical observability setups across development, staging, and production environments.
Version Control: Track changes to your observability configuration alongside your application code.
Automation: Automate the deployment and updates of monitoring agents, dashboards, and alert rules.
Auditability: Maintain a clear audit trail of all infrastructure and observability changes.

Prerequisites

Before you begin, ensure you have the following:

An active AWS account with administrative access.
Terraform CLI installed (v1.0.0+ recommended).
AWS CLI configured with appropriate credentials.
kubectl installed and configured to connect to your EKS cluster.
A Datadog account with API and Application keys.
A PagerDuty account with an integration key (or API access for programmatic setup).
An existing AWS EKS cluster, or the Terraform configuration to provision one. This guide assumes an operational EKS cluster.

Integrating Datadog for EKS Observability

Datadog provides comprehensive monitoring capabilities for EKS, collecting metrics, logs, and traces from your cluster, nodes, pods, and applications. We'll deploy the Datadog Agent as a DaemonSet across your EKS nodes and configure Datadog monitors using Terraform.

1. Datadog API and Application Keys

You'll need your Datadog API key and Application key for Terraform to interact with the Datadog API. It's best practice to manage these securely, for example, using AWS Secrets Manager or environment variables.

Create a datadog_provider.tf file:

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

Pass these values securely when running Terraform (e.g., via environment variables or a .tfvars file that is not committed to version control).

2. Deploying the Datadog Agent with Terraform

The Datadog Agent can be deployed on EKS using a Helm chart. Terraform can manage Helm chart deployments using the helm_release resource.

First, ensure you have the Kubernetes and Helm providers configured in your Terraform project, pointing to your EKS cluster:

provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.this.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.this.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.this.token
  }
}

data "aws_eks_cluster" "this" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "this" {
  name = var.eks_cluster_name
}

variable "eks_cluster_name" {
  description = "Name of the EKS cluster"
  type        = string
}

Now, define the helm_release for the Datadog Agent. This example configures basic metrics, logs, APM, and cluster agent. Remember to replace placeholders with your actual Datadog API key and site.

Terraform Configuration for Datadog Agent

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "datadog.site"
    value = "datadoghq.com" # Or eu.datadoghq.com, us3.datadoghq.com, etc.
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "clusterAgent.metricsProvider.enabled"
    value = "true"
  }

  set {
    name  = "agent.apm.enabled"
    value = "true"
  }

  set {
    name  = "agent.log.enabled"
    value = "true"
  }

  set {
    name  = "agent.log.logsEnabled"
    value = "true"
  }

  set {
    name  = "agent.log.containerCollectAll"
    value = "true"
  }

  set {
    name  = "targetSystem"
    value = "linux"
  }

  # Add other configurations as needed, e.g., service accounts, RBAC, resource limits
}

3. Configuring Datadog Monitors with Terraform

Once the Datadog Agent is collecting data, you can define monitors (alerts) using Terraform's datadog_monitor resource. Here's an example for monitoring EKS node CPU utilization.

resource "datadog_monitor" "eks_node_cpu_utilization" {
  name               = "EKS Node CPU Utilization High on {{host.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster_name:${var.eks_cluster_name}} by {host} < 10" # Alert if idle CPU is < 10%

  message            = <Notice the @webhook-pagerduty-eks-incidents in the message. This is how Datadog sends alerts to PagerDuty. We'll set up this integration next.
Setting Up PagerDuty for Incident ResponsePagerDuty acts as your central hub for incident management, routing critical alerts from Datadog to the right on-call teams and facilitating rapid response.
1. PagerDuty Provider ConfigurationSimilar to Datadog, configure the PagerDuty provider in Terraform with your PagerDuty API token. Store this token securely.

provider "pagerduty" {
  token = var.pagerduty_token
}

variable "pagerduty_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}
2. Creating PagerDuty Services and Escalation PoliciesA PagerDuty Service represents a system or application that needs to be monitored. An Escalation Policy defines how incidents on that service are escalated through teams or individuals.

resource "pagerduty_escalation_policy" "eks_critical_incidents" {
  name      = "EKS Critical Incident Policy"
  num_loops = 2

  rule {
    delay = 10 # minutes
    target {
      type = "user" # Or "team" if you have teams configured
      id   = pagerduty_user.devops_on_call.id # Replace with actual user ID
    }
  }

  rule {
    delay = 20 # minutes
    target {
      type = "user"
      id   = pagerduty_user.devops_manager.id # Replace with actual user ID
    }
  }
}

resource "pagerduty_user" "devops_on_call" {
  name  = "DevOps On-Call User"
  email = "devops-oncall@example.com"
  # Other user attributes...
}

resource "pagerduty_user" "devops_manager" {
  name  = "DevOps Manager"
  email = "devops-manager@example.com"
  # Other user attributes...
}

resource "pagerduty_service" "eks_observability_service" {
  name                    = "EKS Cluster Observability"
  escalation_policy       = pagerduty_escalation_policy.eks_critical_incidents.id
  auto_resolve_timeout    = "14400" # 4 hours
  acknowledgement_timeout = "600"   # 10 minutes
}
Note: You'd typically manage users, teams, and schedules in PagerDuty, often through a separate Terraform setup or directly in the UI, then reference their IDs here.
3. Integrating Datadog with PagerDutyDatadog integrates with PagerDuty primarily through webhooks or its native integration. For simplicity and granular control via Terraform, we'll use a webhook-based approach.
First, create a PagerDuty integration key for your service. This key is used by Datadog to send incident data to PagerDuty.

resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog Integration"
  service_id  = pagerduty_service.eks_observability_service.id
  type        = "generic_events_api_v2"
}
Now, in Datadog, you need to create a webhook integration that points to PagerDuty's Events API V2. You can achieve this using the datadog_integration_webhook resource in Terraform.

resource "datadog_integration_webhook" "pagerduty_eks" {
  name = "pagerduty-eks-incidents" # This name is used in the @webhook-pagerduty-eks-incidents tag in Datadog monitors

  url = "https://events.pagerduty.com/v2/enqueue"

  custom_headers {
    name  = "Content-Type"
    value = "application/json"
  }

  encode_as_form = false

  custom_payload {
    key   = "routing_key"
    value = pagerduty_service_integration.datadog_integration.integration_key
  }

  custom_payload {
    key   = "event_action"
    value = "$EVENT_TYPE" # Datadog macro for event type (trigger, resolve)
  }

  custom_payload {
    key   = "payload.summary"
    value = "$ALERT_TITLE"
  }

  custom_payload {
    key   = "payload.timestamp"
    value = "$LAST_UPDATED"
  }

  custom_payload {
    key   = "payload.severity"
    value = "$ALERT_STATUS" # Map Datadog statuses to PagerDuty severities
  }

  custom_payload {
    key   = "payload.source"
    value = "$ORG_NAME"
  }

  custom_payload {
    key   = "payload.component"
    value = "$TAGS"
  }

  custom_payload {
    key   = "payload.group"
    value = "$ALERT_ID"
  }

  custom_payload {
    key   = "links[0].href"
    value = "$LINK"
  }

  custom_payload {
    key   = "links[0].text"
    value = "View in Datadog"
  }

  # Add more custom payload fields for richer incident context
}
With this webhook configured, any Datadog monitor that includes @webhook-pagerduty-eks-incidents in its message will trigger an event in PagerDuty using the defined custom payload, creating or updating an incident.
Advanced Observability and Alerting PatternsBeyond basic metrics, Datadog offers powerful features that can be leveraged with Terraform:
Log Management: Collect and analyze Kubernetes logs for troubleshooting and security auditing. Configure log processing pipelines and log-based monitors.
APM & Distributed Tracing: Instrument your applications to trace requests across microservices running on EKS, identifying latency bottlenecks.
Synthetic Monitoring: Simulate user journeys or API calls to proactively detect availability and performance issues from an external perspective.
Custom Dashboards: Create comprehensive dashboards using datadog_dashboard to visualize EKS health, application performance, and service-level objectives (SLOs).
Anomaly Detection: Leverage Datadog's machine learning capabilities to alert on unusual patterns in your metrics that might indicate emerging issues.
End-to-End Incident Management WorkflowWith Terraform managing your observability and incident response configurations, here's how a typical incident workflow unfolds:
Detection: A critical metric (e.g., EKS node CPU utilization, pod restarts, application error rates) configured by a datadog_monitor resource breaches its threshold.
Alert Generation: Datadog triggers an alert, formatted with the custom payload defined in your datadog_integration_webhook resource.
Incident Creation: PagerDuty receives the webhook event and, based on the routing_key, creates an incident on the eks_observability_service.
On-Call Notification: PagerDuty's escalation policy for the service notifies the appropriate on-call engineer(s) via their preferred methods (SMS, phone call, push notification).
Investigation & Remediation: The on-call engineer acknowledges the incident in PagerDuty, reviews the Datadog alert context (via direct links), and investigates the EKS cluster using Datadog dashboards and logs. Remediation actions are taken.
Resolution: Once the issue is resolved and metrics return to normal, Datadog sends a "resolve" event to PagerDuty, automatically resolving the incident.
Post-Mortem: Teams conduct post-incident reviews to identify root causes, improve systems, and refine observability and alerting configurations (often updated via Terraform).


Troubleshooting and Best PracticesCommon IssuesDatadog Agent Not Reporting: Check Kubernetes logs for the Datadog Agent pods for errors. Ensure the API key is correct and network policies allow egress to Datadog endpoints. Verify RBAC permissions for the agent service account.
PagerDuty Incidents Not Triggering: Verify the Datadog webhook configuration, especially the URL and integration key. Ensure the Datadog monitor's message correctly includes the webhook tag (e.g., @webhook-pagerduty-eks-incidents). Check PagerDuty's incident logs for incoming events.
Terraform Apply Errors: Ensure AWS credentials are configured correctly. Verify that your EKS cluster name and details are accurate in your Terraform data sources. Check provider versions.
Best PracticesSecure Secrets: Never hardcode API or application keys. Use a secrets manager (AWS Secrets Manager, HashiCorp Vault) and integrate with Terraform.
Granular Alerts: Create specific monitors for different components and severities. Avoid "noisy" alerts by tuning thresholds carefully.
Clear Messages: Ensure your Datadog monitor messages provide actionable context, including links to relevant dashboards or runbooks.
On-Call Rotations: Implement clear on-call schedules and rotations in PagerDuty to prevent burnout and ensure 24/7 coverage.
Tagging Strategy: Use consistent tagging across AWS, EKS, Datadog, and PagerDuty (e.g., environment, service, team). This allows for powerful filtering, grouping, and cost attribution.
Automated Remediation: For recurring, well-understood incidents, explore automated remediation actions triggered by PagerDuty or Datadog webhooks.
ConclusionBy leveraging Terraform to manage your AWS EKS observability stack with Datadog and PagerDuty, you create a robust, automated, and scalable solution. This approach ensures your EKS clusters are continuously monitored, critical issues are promptly detected, and on-call teams are effectively alerted and empowered to respond quickly. Embracing IaC for your observability tooling is a crucial step towards achieving true operational excellence and reliability in your cloud-native environments.
Start implementing these patterns today to bring unparalleled visibility and incident response capabilities to your AWS EKS deployments!

CloudStack YAML