Terraform-Managed AWS EKS Observability and Incident Response with Datadog and PagerDuty

In the dynamic world of cloud-native applications, managing AWS Elastic Kubernetes Service (EKS) requires robust observability and a streamlined incident response strategy. This guide delves into integrating Datadog for comprehensive monitoring and PagerDuty for efficient incident management, all orchestrated declaratively using Terraform. We'll explore how to establish a resilient system that not only detects issues promptly but also ensures critical alerts reach the right teams for rapid resolution, maintaining high availability and operational excellence.

Architecture Pro-Tip:

Always centralize your observability configuration within your Infrastructure as Code (IaC) repository. Treat your Datadog monitors, dashboards, and PagerDuty services as code alongside your EKS clusters. This ensures version control, simplifies rollbacks, and promotes consistency across environments, significantly reducing configuration drift and human error. Leverage Terraform modules for reusable patterns to streamline multi-cluster or multi-region deployments.

The Pillars: Terraform, Datadog, PagerDuty for EKS

Achieving optimal operational efficiency for AWS EKS involves a synergistic combination of powerful tools:

Terraform: Your Infrastructure as Code (IaC) orchestrator, managing EKS clusters, networking, and critical integrations in a declarative, repeatable manner. It allows you to provision and configure Datadog and PagerDuty resources alongside your infrastructure.
Datadog: A unified observability platform offering end-to-end monitoring for EKS. This includes metrics, logs, traces, synthetic monitoring, and network performance monitoring, giving you deep insights into your Kubernetes workloads and AWS infrastructure.
PagerDuty: The incident management and on-call automation platform that takes Datadog alerts and transforms them into actionable incidents, ensuring the right people are notified at the right time through their preferred communication channels.

Prerequisites

Before diving into the configuration, ensure you have the following:

An AWS Account with administrative access.
Terraform CLI (v1.0+) installed.
AWS CLI configured with appropriate credentials.
A Datadog Account with an API and Application key.
A PagerDuty Account with an API key.
An existing AWS EKS cluster, or the ability to provision one using Terraform (which we'll assume for our examples).

Step 1: Setting Up Terraform Providers

Begin by configuring the necessary Terraform providers for AWS, Datadog, and PagerDuty. Store your API keys securely, preferably using environment variables or a secrets management solution like AWS Secrets Manager.

        
provider "aws" {
  region = "us-east-1"
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}
        
    

Step 2: Integrating Datadog with AWS EKS

Observability begins with data collection. Datadog integrates with EKS in two primary ways: via the Datadog Agent deployed within your EKS cluster and via AWS integration for cloud-level metrics and logs.

2.1 Deploying the Datadog Agent to EKS

The Datadog Agent, typically deployed as a DaemonSet using Helm, collects metrics, logs, and traces from your Kubernetes nodes and pods. While the Helm chart handles the Kubernetes deployment, Terraform can manage the Kubernetes resources if you prefer. For simplicity, we'll assume you deploy the agent via Helm, but ensure its configuration points to your Datadog API key.

For advanced setups, consider using the official Datadog Terraform module for Kubernetes.

2.2 Configuring AWS Integration for Datadog

To monitor your AWS infrastructure components (EC2 instances, S3 buckets, RDS databases, etc.) that support your EKS cluster, you need to grant Datadog access to your AWS account. This is done by creating an IAM role that Datadog can assume.

Terraform Configuration for Datadog AWS Integration and a Basic Monitor

        
resource "aws_iam_role" "datadog_integration_role" {
  name = "DatadogIntegrationRole"

  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect = "Allow",
        Principal = {
          AWS = "arn:aws:iam::464622532012:root" # Datadog's AWS account ID
        },
        Action = "sts:AssumeRole",
        Condition = {
          StringEquals = {
            "sts:ExternalId" = var.datadog_external_id # Generated by Datadog
          }
        }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "datadog_read_only_policy" {
  role       = aws_iam_role.datadog_integration_role.name
  policy_arn = "arn:aws:iam::aws:policy/ReadOnlyAccess" # Use a more restrictive policy if possible
}

# This variable should be obtained from Datadog UI after initiating AWS integration
variable "datadog_external_id" {
  description = "External ID provided by Datadog for AWS integration"
  type        = string
  sensitive   = true
}

resource "datadog_integration_aws" "aws_integration" {
  account_id  = data.aws_caller_identity.current.account_id
  role_name   = aws_iam_role.datadog_integration_role.name
  host_tags   = ["environment:production", "source:terraform"]
  filter_tags = ["region:us-east-1"]
}

data "aws_caller_identity" "current" {}

resource "datadog_monitor" "high_cpu_usage" {
  name          = "EKS High CPU Usage (Terraform)"
  type          = "query alert"
  query         = "avg(last_5m):avg:kubernetes.cpu.usage.total{*} by {kube_cluster_name} > 80"
  message       = "High CPU usage detected on EKS cluster {{kube_cluster_name.name}}. Investigate immediately. @slack-devops @webhook-pagerduty"
  tags          = ["team:devops", "service:eks", "severity:high"]
  priority      = 1
  notify_no_data = false
  new_group_delay = 60
  no_data_timeframe = 10
  require_full_window = false
  evaluation_delay = 300
}
        
    

Important: The datadog_external_id is crucial for secure cross-account access and needs to be generated within your Datadog account when setting up the AWS integration. After applying the above Terraform, you will typically go to Datadog's AWS integration page, select "IAM Role" setup, and find your unique External ID there. Update your Terraform variable with this ID. For the aws_iam_role_policy_attachment, consider using more granular policies than `ReadOnlyAccess` for production environments.

Step 3: Integrating PagerDuty for Incident Response

PagerDuty acts as the central hub for incident management, ensuring that critical alerts from Datadog are escalated effectively.

3.1 Setting up PagerDuty Services and Escalation Policies

With Terraform, you can define your PagerDuty services, escalation policies, and users, bringing your entire on-call schedule and alerting rules under IaC.

Terraform Configuration for PagerDuty Service and Escalation Policy

        
resource "pagerduty_user" "devops_engineer_john" {
  name  = "John Doe"
  email = "john.doe@example.com"
}

resource "pagerduty_team" "devops_team" {
  name        = "DevOps Team"
  description = "Team responsible for EKS infrastructure and applications."
}

resource "pagerduty_team_membership" "john_devops_membership" {
  user_id = pagerduty_user.devops_engineer_john.id
  team_id = pagerduty_team.devops_team.id
}

resource "pagerduty_escalation_policy" "eks_primary_escalation" {
  name      = "EKS Primary Escalation Policy"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 15
    target {
      type = "user_reference"
      id   = pagerduty_user.devops_engineer_john.id
    }
  }

  rule {
    escalation_delay_in_minutes = 30
    target {
      type = "team_reference"
      id   = pagerduty_team.devops_team.id
    }
  }
}

resource "pagerduty_service" "eks_observability_service" {
  name                    = "EKS Observability"
  auto_resolve_timeout_minutes = 120 # Automatically resolve after 2 hours if no action
  acknowledgement_timeout_minutes = 30 # Acknowledge within 30 minutes
  escalation_policy       = pagerduty_escalation_policy.eks_primary_escalation.id
}

# Output the integration key for Datadog
output "pagerduty_integration_key_eks_observability" {
  description = "The integration key for the EKS Observability PagerDuty service."
  value       = pagerduty_service.eks_observability_service.integrations[0].integration_key
  sensitive   = true
}
        
    

Step 4: Orchestrating Alerts - Datadog to PagerDuty

With both Datadog and PagerDuty configured, the final step is to create a Datadog monitor that triggers an incident in PagerDuty. This is achieved by using a PagerDuty integration within Datadog.

4.1 Configuring the Datadog-PagerDuty Integration Channel

First, ensure your Datadog account has a PagerDuty integration configured. This involves providing the PagerDuty integration key (obtained from the PagerDuty service).

You can configure a webhook in Datadog that points to PagerDuty's Events API or use the native PagerDuty integration within Datadog. For simplicity and reliability, Datadog's native PagerDuty integration is recommended.

4.2 Terraform for a Datadog Monitor Triggering PagerDuty

Modify an existing Datadog monitor or create a new one to include the PagerDuty service in its notification message. Datadog's native PagerDuty integration is typically referenced as @pagerduty-<service-name> or by adding the PagerDuty webhook.

Troubleshooting and Best Practices

Common Troubleshooting Steps:

Datadog Agent Not Reporting: Verify the Kubernetes DaemonSet is running, check agent logs for API key errors or network connectivity issues to Datadog endpoints. Ensure `DD_KUBERNETES_COLLECT_CONTAINER_TAGS` is enabled.
AWS Integration Issues: Double-check the IAM role's `assume_role_policy` and `sts:ExternalId`. Confirm the `ReadOnlyAccess` policy (or custom policy) is attached.
Datadog Monitors Not Firing: Review the monitor query in Datadog UI to ensure it matches expected metric patterns. Check evaluation delays and notification settings.
PagerDuty Incidents Not Triggering: Verify the @pagerduty notification syntax in your Datadog monitor message. Ensure the PagerDuty integration key used in Datadog corresponds to the correct PagerDuty service. Check PagerDuty's API logs for incoming events.
Terraform Plan/Apply Errors: Always perform a terraform plan before terraform apply. Address any syntax errors, provider authentication failures, or resource conflicts.

Best Practices for EKS Observability and Incident Response:

Granular Monitoring: Beyond cluster-level metrics, set up monitors for individual deployments, services, and pods. Focus on key performance indicators (KPIs) like latency, error rates, traffic, and saturation (RED method).
SLOs and SLIs: Define Service Level Objectives (SLOs) and Service Level Indicators (SLIs) for your critical EKS services. Create Datadog monitors to alert when SLIs are breached.
Runbooks: For every PagerDuty service, attach clear, concise runbooks that guide on-call engineers through initial triage, common fixes, and escalation paths.
Automated Testing: Regularly test your alerts and incident response workflows, especially after making changes to your EKS cluster, Datadog monitors, or PagerDuty escalation policies.
Cost Optimization: Monitor Datadog usage and optimize log retention, metric cardinality, and agent configuration to manage costs effectively.
Security: Ensure least-privilege for all IAM roles and API keys. Rotate keys regularly.

Conclusion

By leveraging Terraform to manage your AWS EKS observability and incident response configuration with Datadog and PagerDuty, you achieve a robust, auditable, and repeatable system. This Infrastructure as Code approach ensures that your monitoring and alerting infrastructure evolves seamlessly with your EKS clusters, enabling your teams to detect, diagnose, and resolve issues with unparalleled speed and efficiency. Embrace these tools to transform your operational practices and elevate the reliability of your cloud-native applications.

CloudStack YAML