Terraform-Managed AWS EKS Cluster Observability with Datadog and PagerDuty Integration

Architecture Pro-Tip:

Always manage your observability stack components, including agents, integrations, and monitor definitions, as Infrastructure as Code (IaC). Leveraging Terraform for Datadog agent deployment on EKS, AWS IAM roles for Datadog integration, and Datadog monitors ensures consistency, version control, and auditability. This approach significantly reduces configuration drift and streamlines disaster recovery or multi-environment deployments.

Terraform-Managed AWS EKS Cluster Observability with Datadog and PagerDuty Integration

In today's dynamic cloud-native environments, robust observability is not just a luxury; it's a critical necessity. Managing AWS EKS clusters with Terraform offers significant advantages in automation and consistency. Extending this IaC philosophy to your observability stack ensures that your monitoring, logging, and incident response capabilities are as resilient and well-defined as your infrastructure. This guide delves into establishing comprehensive observability for your Terraform-managed AWS EKS clusters using Datadog for all-in-one monitoring and PagerDuty for streamlined incident management.

The Imperative of EKS Observability

Kubernetes, and specifically AWS EKS, introduces layers of abstraction and complexity. Microservices, dynamic scheduling, and ephemeral pods demand sophisticated tools to understand system behavior. Observability, encompassing metrics, logs, and traces, provides the necessary visibility to:

Identify Performance Bottlenecks: Pinpoint resource contention, slow queries, or inefficient code within your applications and cluster.
Debug and Troubleshoot Faster: Correlate events across different services and infrastructure components to quickly diagnose root causes.
Proactive Anomaly Detection: Detect unusual patterns that might indicate impending issues before they impact users.
Optimize Resource Utilization: Gain insights to right-size your EKS nodes and pod requests/limits, saving costs.
Ensure Compliance and Security: Monitor access patterns, network flows, and system health for security audits.

Datadog: Your Unified Observability Platform for EKS

Datadog offers a unified platform that brings together infrastructure monitoring, application performance monitoring (APM), log management, network performance monitoring, and more. For EKS, Datadog provides out-of-the-box integrations that allow you to collect a rich set of data points:

Kubernetes Integration: Automatically collects metrics from kube-state-metrics, cAdvisor, and the Kubernetes API server.
AWS Integration: Gathers metrics and events from various AWS services, including EC2, CloudWatch, VPC, and, crucially, EKS Control Plane logs.
APM and Tracing: Provides deep visibility into distributed applications running on EKS, allowing you to trace requests end-to-end.
Log Management: Aggregates, processes, and analyzes logs from your pods, nodes, and EKS control plane.
Network Performance Monitoring: Visualizes network traffic and connectivity within and across your EKS cluster.
Synthetics and RUM: Proactively test application availability and performance from a user's perspective.

Integrating Datadog with AWS EKS

The primary mechanism for integrating Datadog with EKS involves two key steps:

AWS Integration: Configure an AWS IAM role for Datadog to pull metrics and logs from your AWS account, including EKS control plane metrics and CloudWatch logs.
Datadog Agent Deployment: Deploy the Datadog Agent as a DaemonSet across your EKS cluster nodes to collect metrics, logs, and traces from the cluster, nodes, pods, and containers.

PagerDuty: Streamlining Incident Response

When an issue arises, knowing about it quickly is paramount. PagerDuty excels in transforming monitoring signals into actionable incidents, ensuring the right teams are notified at the right time through their preferred communication channels.

Intelligent Alerting: Aggregates and enriches alerts, reducing noise and focusing on critical incidents.
On-Call Management: Manages complex on-call schedules, ensuring 24/7 coverage.
Automated Escalations: Defines escalation policies to ensure incidents are resolved promptly.
Cross-Tool Integration: Seamlessly integrates with a vast ecosystem of monitoring, chat, and ticketing tools.

Integrating Datadog with PagerDuty

Datadog's integration with PagerDuty is straightforward. You configure PagerDuty as a notification channel within Datadog, allowing your Datadog monitors to trigger incidents directly in PagerDuty, complete with contextual information.

Terraform: Infrastructure as Code for Observability

Managing your EKS cluster and its observability stack with Terraform brings the benefits of IaC to the forefront. This includes:

Version Control: All configurations are stored in Git, enabling collaboration, review, and easy rollback.
Reproducibility: Easily recreate environments (e.g., development, staging, production) with identical monitoring setups.
Automation: Eliminate manual configuration, reducing human error and speeding up deployments.
Auditability: Track changes to your observability configuration over time.

Key Terraform Resources

To achieve full observability with Terraform, Datadog, and PagerDuty, you'll leverage several Terraform providers:

AWS Provider: For EKS cluster creation, IAM roles, and other AWS services.
Helm Provider: To deploy the Datadog Agent using its official Helm chart onto your EKS cluster.
Datadog Provider: To manage Datadog resources like monitors (`datadog_monitor`), dashboards (`datadog_dashboard`), and AWS integration (`datadog_integration_aws`).
PagerDuty Provider: To manage PagerDuty services, escalation policies, and users (optional, as typically Datadog integrates with an existing PagerDuty service).

Step-by-Step Implementation Guide

Prerequisites

Before you begin, ensure you have the following:

An AWS account with appropriate permissions.
Terraform CLI installed (v1.0+ recommended).
kubectl and helm CLIs configured to interact with your EKS cluster.
A Datadog account with an API key and Application key.
A PagerDuty account with an Integration Key for Datadog.

1. Configure Terraform Providers

Set up your providers in your main.tf file. Ensure you replace placeholders with your actual Datadog API/APP keys and PagerDuty integration key.

provider "aws" {
  region = "us-east-1"
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.main.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.main.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.main.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.main.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

# Ensure PagerDuty integration is set up in Datadog UI,
# then reference the service key or use the PagerDuty provider for service creation.
# For simplicity, we'll assume a PagerDuty service exists and its key is used in Datadog monitor.

2. Integrate Datadog with AWS (IAM Role)

Create an IAM role that Datadog can assume to collect data from your AWS account. This example grants read-only access for CloudWatch metrics and EKS API access.

# Data source to fetch existing EKS cluster details
data "aws_eks_cluster" "main" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "main" {
  name = var.eks_cluster_name
}

# Datadog AWS Integration Role
resource "aws_iam_role" "datadog_integration" {
  name = "${var.eks_cluster_name}-datadog-integration-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action = "sts:AssumeRole",
        Effect = "Allow",
        Principal = {
          AWS = "arn:aws:iam::464622532012:root" # Datadog's AWS account ID
        }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "datadog_readonly" {
  policy_arn = "arn:aws:iam::aws:policy/ReadOnlyAccess" # For general metrics, logs
  role       = aws_iam_role.datadog_integration.name
}

# Attach specific policy for EKS control plane logging if needed (e.g., Kubelet logs from CloudWatch)
# resource "aws_iam_policy" "datadog_eks_control_plane_policy" {
#   name        = "${var.eks_cluster_name}-datadog-eks-control-plane-policy"
#   description = "IAM policy for Datadog to access EKS control plane logs"
#   policy      = jsonencode({
#     Version = "2012-10-17",
#     Statement = [
#       {
#         Action = [
#           "logs:FilterLogEvents",
#           "logs:GetLogEvents",
#           "logs:DescribeLogGroups",
#           "logs:DescribeLogStreams"
#         ],
#         Effect   = "Allow",
#         Resource = "arn:aws:logs:*:*:log-group:/aws/eks/${var.eks_cluster_name}/cluster:*"
#       },
#     ],
#   })
# }
# resource "aws_iam_role_policy_attachment" "datadog_eks_control_plane_attachment" {
#   policy_arn = aws_iam_policy.datadog_eks_control_plane_policy.arn
#   role       = aws_iam_role.datadog_integration.name
# }


resource "datadog_integration_aws" "main" {
  account_id = data.aws_caller_identity.current.account_id # Your AWS account ID
  role_name  = aws_iam_role.datadog_integration.name
  host_tags = [
    "env:${var.environment}",
    "eks_cluster:${var.eks_cluster_name}"
  ]
}

3. Deploy Datadog Agent to EKS using Helm and Terraform

Use the Helm provider to deploy the Datadog Agent. Configure it with your Datadog API key, enable APM, logs, and process monitoring.

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true
  version    = "2.33.0" # Use a stable and recent version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "datadog.site"
    value = "datadoghq.com" # or eu.datadoghq.com, etc.
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "kubeStateMetricsExternal.enabled"
    value = "true"
  }

  set {
    name  = "agents.logLevel"
    value = "INFO"
  }

  set {
    name  = "datadog.apm.enabled"
    value = "true"
  }

  set {
    name  = "datadog.logs.enabled"
    value = "true"
  }

  set {
    name  = "datadog.logs.containerCollectAll"
    value = "true"
  }

  set {
    name  = "datadog.processAgent.enabled"
    value = "true"
  }

  set {
    name  = "datadog.dogstatsd.tags"
    value = "env:${var.environment},cluster:${var.eks_cluster_name}"
  }

  set {
    name  = "clusterAgent.metricsProvider.enabled"
    value = "true"
  }

  set {
    name  = "datadog.tags"
    value = "{env:${var.environment},cluster:${var.eks_cluster_name}}"
  }
}

4. Define Datadog Monitors with PagerDuty Integration

Now, let's define a sample Datadog monitor for high CPU utilization on EKS nodes and configure it to send alerts to PagerDuty. You will need to have a PagerDuty service integrated within Datadog first (this is typically done via the Datadog UI under Integrations -> PagerDuty, where you provide the PagerDuty integration key for a specific service).

resource "datadog_monitor" "high_node_cpu" {
  name               = "[EKS] High Node CPU Utilization on {{host.name}} ({{cluster}})"
  type               = "metric alert"
  query              = "avg(last_5m):avg:system.cpu.idle{cluster:${var.eks_cluster_name}} by {host} < 10" # Less than 10% idle means > 90% utilization
  message            = <
        Note: The @pagerduty-{{var.pagerduty_service_name}} syntax in the message block tells Datadog to send an alert to the PagerDuty service configured with that name in Datadog's integrations. Ensure the pagerduty_service_name variable reflects the actual name you've given your PagerDuty service in Datadog.
    
5. Apply Terraform Configuration
        Initialize your Terraform workspace, review the plan, and apply the changes:
    

terraform init
terraform plan
terraform apply --auto-approve

        This will:
    
Create the necessary IAM role for Datadog in AWS.
Configure the AWS integration in Datadog using the created role.
Deploy the Datadog Agent Helm chart to your EKS cluster.
Create the Datadog monitors, configured to send alerts to PagerDuty.
Testing and Validation
        After applying the Terraform configuration, verify the setup:
    
Datadog Agent: Check your EKS cluster pods: kubectl get pods -n datadog. You should see Datadog agent pods running.
Datadog UI: Navigate to the Infrastructure section in Datadog. You should see your EKS nodes reporting metrics. Check the Kubernetes integration dashboard for cluster-wide metrics.
Logs: In Datadog Log Explorer, filter by source:kubernetes to see logs from your containers.
Monitors: In Datadog Monitors list, confirm your Terraform-defined monitors are present and active.
PagerDuty Integration: Artificially trigger a monitor (e.g., by simulating high CPU load on a node) to ensure PagerDuty incidents are created.


Troubleshooting Common Issues
            Datadog Agent Pods Not Running: Check kubectl describe pod datadog-agent-xxxx -n datadog for errors. Ensure the API key is correct. Check node taints/tolerations.
        

            No Data in Datadog:
            Verify the Datadog AWS integration. Check the IAM role's permissions and trust policy.
Ensure Datadog Agent pods are healthy and their logs (kubectl logs datadog-agent-xxxx -n datadog) don't show connection issues to Datadog endpoints.

        

            PagerDuty Alerts Not Triggering:
            Confirm the Datadog monitor is actually triggering (check Datadog events).
Verify the PagerDuty integration name in the Datadog monitor's message matches the one configured in Datadog's integrations.
Check PagerDuty's event log to see if events are being received.

        
Conclusion
        By leveraging Terraform, you can seamlessly integrate Datadog for comprehensive observability and PagerDuty for efficient incident response into your AWS EKS clusters. This IaC-driven approach ensures that your monitoring and alerting infrastructure is robust, scalable, and fully automated, allowing your teams to focus on innovation rather than manual configuration. A well-orchestrated observability stack is foundational for maintaining the health, performance, and reliability of your cloud-native applications.
    

CloudStack YAML