Production-Ready Terraform Configuration for AWS EKS Observability with Datadog and PagerDuty

In the rapidly evolving landscape of cloud-native applications, maintaining robust observability for Kubernetes clusters is paramount. AWS Elastic Kubernetes Service (EKS) provides a powerful platform for deploying containerized workloads, but without a comprehensive monitoring and incident management strategy, operational resilience can be severely compromised. This guide provides a detailed, production-ready Terraform configuration to integrate Datadog for deep EKS observability and PagerDuty for streamlined incident response, ensuring your applications remain performant and available.

Architecture Pro-Tip: Layered Observability Strategy

For optimal EKS observability, adopt a layered approach:

Infrastructure Layer: Monitor EKS control plane logs (CloudWatch), EC2 worker node metrics, and VPC flow logs.
Kubernetes Layer: Track cluster-level metrics (API server, scheduler, controller manager), node health, pod status, and resource utilization.
Application Layer: Collect application logs, custom metrics (OpenTelemetry/Prometheus), and distributed traces from your microservices.
Security Layer: Integrate security tools and monitor Kubernetes audit logs for suspicious activities.

Datadog excels at unifying these layers, providing a single pane of glass for comprehensive insights.

Why Comprehensive Observability for AWS EKS?

Operating Kubernetes at scale introduces complexity. Distributed systems, ephemeral containers, and dynamic networking demand sophisticated tools to understand their behavior. Datadog offers an all-in-one monitoring solution for EKS, providing:

Unified Metrics, Logs, and Traces: Correlate data across your entire stack.
Kubernetes-Native Monitoring: Deep insights into Pods, Deployments, Services, Nodes, and namespaces.
Network Performance Monitoring: Visibility into Kubernetes network traffic.
Custom Dashboards & Alerts: Tailor views and notifications to your specific needs.
Container Security: Real-time threat detection and vulnerability management.

When an incident occurs, swift and accurate response is critical. PagerDuty is an industry leader in incident management, enabling:

Automated Incident Routing: Alerts from Datadog are transformed into actionable incidents and routed to the right on-call team.
On-Call Scheduling: Manage complex schedules and escalation policies.
Communication & Collaboration: Facilitate rapid communication during outages.
Post-Mortem Analysis: Tools for reviewing incidents and implementing improvements.

Prerequisites

Before deploying the Terraform configuration, ensure you have the following in place:

AWS Account: With necessary permissions to create EKS clusters, IAM roles, and other AWS resources.
Terraform Installed: Version 1.0 or higher.
AWS CLI Configured: Authenticated with your AWS account.
Kubectl Installed: For interacting with your EKS cluster.
Helm Installed: For deploying Kubernetes applications.
Existing AWS EKS Cluster: This guide assumes you have an operational EKS cluster. If not, consider using Terraform to provision one first.
Datadog Account: With API and Application keys.
PagerDuty Account: With a Global API Key or an Integration Key for Datadog.

Core Components and Integration Strategy

Our solution leverages several key components to achieve end-to-end observability and incident management:

Datadog Agent: Deployed as a DaemonSet on EKS worker nodes to collect host-level metrics, logs, and traces.
Datadog Cluster Agent: Deployed as a Deployment, responsible for cluster-level collection (e.g., Kubernetes API server metrics, events, leader election).
Datadog Operator: Simplifies deployment and management of Datadog components within Kubernetes.
AWS IAM Roles for Service Accounts (IRSA): Securely grants AWS permissions to Kubernetes service accounts without managing AWS credentials in pods. This is crucial for the Datadog Agent to access CloudWatch, S3, etc.
Datadog API & Application Keys: For authenticating the Datadog Agent and managing Datadog resources via Terraform.
PagerDuty Service & Integration: Defines the service to be monitored and the specific integration point for Datadog alerts.
Terraform Providers:
- aws: For AWS resources like IAM roles.
- kubernetes: For interacting with the EKS cluster (e.g., creating service accounts, secrets).
- helm: For deploying the Datadog Agent via its Helm chart.
- datadog: For creating Datadog monitors, dashboards, and integrations.
- pagerduty: For managing PagerDuty services, escalation policies, and integrations.

Terraform Module Structure Overview

For production readiness, it's best to organize your Terraform code into logical modules. A typical structure might look like this:

main.tf: Main entry point, variable declarations.
providers.tf: Provider configurations (AWS, Kubernetes, Datadog, PagerDuty, Helm).
eks.tf: (Optional) If you're provisioning EKS here, otherwise inputs for an existing cluster.
datadog_agent.tf: IAM roles, Kubernetes Service Accounts, and Helm chart deployment for the Datadog Agent.
datadog_monitors.tf: Datadog monitors and dashboards.
pagerduty.tf: PagerDuty services, escalation policies, and Datadog integration.
variables.tf: Input variables for customization.
outputs.tf: Output values.

Configuring Datadog Observability with Terraform

1. IAM Role for Service Account (IRSA) for Datadog Agent

The Datadog Agent needs permissions to collect metrics and logs from AWS services, such as CloudWatch, EC2, and S3. Using IRSA is the most secure way to grant these permissions.

2. Datadog Agent Deployment via Helm

We'll use the Terraform Helm provider to deploy the official Datadog Agent Helm chart. This allows us to configure all necessary parameters (API key, cluster name, IRSA details) directly through Terraform.

3. Datadog Monitors and Dashboards

Leverage the Datadog Terraform provider to define critical monitors for EKS health, resource utilization, application performance, and more. This ensures your monitoring configuration is version-controlled and deployed consistently.

Configuring PagerDuty Incident Management with Terraform

1. PagerDuty Service and Escalation Policy

Define a PagerDuty service that represents your EKS cluster or the critical applications running on it. Attach an escalation policy that dictates who gets notified and when.

2. Datadog Integration in PagerDuty

Create a PagerDuty integration specifically for Datadog. This will generate a unique integration key that Datadog will use to send alerts.

3. Datadog Notification Configuration

Update your Datadog monitors to send notifications to PagerDuty using the integration key. This can be done directly within the monitor definition or via a global integration.

Ready-to-Use Terraform Configuration Example

Below is a simplified, yet comprehensive, example demonstrating how to set up the core components. Remember to replace placeholder values (e.g., YOUR_DD_API_KEY, YOUR_PD_API_KEY, YOUR_EKS_CLUSTER_NAME) with your actual credentials and details. For a full production deployment, externalize sensitive values using Terraform variables and secrets management (e.g., AWS Secrets Manager, HashiCorp Vault).

provider "aws" {
  region = var.aws_region
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_key
}

# --- Data Sources for EKS Cluster Details ---
data "aws_eks_cluster" "cluster" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = var.eks_cluster_name
}

# --- IAM Role for Datadog Agent (IRSA) ---
resource "aws_iam_policy" "datadog_agent_policy" {
  name_prefix = "${var.eks_cluster_name}-datadog-agent"
  policy      = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action = [
          "ec2:DescribeInstances",
          "ec2:DescribeVolumes",
          "ec2:DescribeTags",
          "autoscaling:DescribeAutoScalingGroups",
          "logs:DescribeLogGroups",
          "logs:DescribeLogStreams",
          "logs:FilterLogEvents",
          "cloudwatch:ListMetrics",
          "cloudwatch:GetMetricData",
          "s3:ListAllMyBuckets" # Example for S3 monitoring, customize as needed
        ],
        Effect   = "Allow",
        Resource = "*"
      },
    ]
  })
}

resource "aws_iam_role" "datadog_agent_role" {
  name               = "${var.eks_cluster_name}-datadog-agent-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect = "Allow",
        Principal = {
          Federated = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${replace(data.aws_eks_cluster.cluster.identity.0.oidc.0.issuer, "https://", "")}"
        },
        Action = "sts:AssumeRoleWithWebIdentity",
        Condition = {
          StringEquals = {
            "${replace(data.aws_eks_cluster.cluster.identity.0.oidc.0.issuer, "https://", "")}:sub" = "system:serviceaccount:default:datadog-agent" # Replace "default" if you use another namespace
            "${replace(data.aws_eks_cluster.cluster.identity.0.oidc.0.issuer, "https://", "")}:aud" = "sts.amazonaws.com"
          }
        }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "datadog_agent_attachment" {
  role       = aws_iam_role.datadog_agent_role.name
  policy_arn = aws_iam_policy.datadog_agent_policy.arn
}

data "aws_caller_identity" "current" {}

# --- Kubernetes Service Account for Datadog Agent ---
resource "kubernetes_service_account" "datadog_agent_sa" {
  metadata {
    name      = "datadog-agent"
    namespace = "default" # Ensure this matches your Helm chart namespace
    annotations = {
      "eks.amazonaws.com/role-arn" = aws_iam_role.datadog_agent_role.arn
    }
  }
}

# --- Deploy Datadog Agent using Helm ---
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Ensure this matches your service account namespace
  version    = "2.33.0"  # Specify a stable version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "datadog.kubeStateMetricsCore.enabled"
    value = true
  }

  set {
    name  = "datadog.clusterAgent.enabled"
    value = true
  }

  set {
    name  = "datadog.clusterAgent.kubeStateMetricsEnabled"
    value = true
  }

  set {
    name  = "datadog.clusterAgent.metricsProvider.enabled"
    value = true
  }

  set {
    name  = "datadog.containerRuntime"
    value = "docker" # Or "containerd" depending on your EKS version
  }

  set {
    name  = "datadog.tags[0]"
    value = "eks_cluster:${var.eks_cluster_name}"
  }

  # IRSA Configuration
  set {
    name  = "serviceAccount.create"
    value = false
  }

  set {
    name  = "serviceAccount.name"
    value = kubernetes_service_account.datadog_agent_sa.metadata.0.name
  }
}

# --- PagerDuty Configuration ---
resource "pagerduty_service" "eks_observability_service" {
  name                     = "${var.eks_cluster_name}-Observability"
  escalation_policy        = pagerduty_escalation_policy.devops_escalation_policy.id
  description              = "Monitors AWS EKS cluster ${var.eks_cluster_name} and its workloads."
  auto_resolve_timeout     = 14400 # 4 hours
  acknowledgement_timeout = 600   # 10 minutes
}

resource "pagerduty_escalation_policy" "devops_escalation_policy" {
  name      = "EKS DevOps Team Escalation"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "user" # Or "schedule"
      id   = var.pagerduty_user_id # Replace with a valid PagerDuty User ID or Schedule ID
    }
  }

  rule {
    escalation_delay_in_minutes = 30
    target {
      type = "user"
      id   = var.pagerduty_team_lead_user_id # Replace with a valid PagerDuty User ID
    }
  }
}

# Datadog Integration in PagerDuty
resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog"
  service_id  = pagerduty_service.eks_observability_service.id
  type        = "generic_events_api" # Use generic_events_api for Datadog
}

# --- Datadog Monitor Example (High CPU Utilization on EKS Node) ---
resource "datadog_monitor" "eks_node_cpu_high" {
  name                = "[EKS: ${var.eks_cluster_name}] High CPU Utilization on Node"
  type                = "metric alert"
  message             = "EKS node {{host.name}} is experiencing high CPU utilization (over 80%). Investigate immediately. @pagerduty-eks-observability"
  query               = "avg(last_5m):avg:system.cpu.idle{eks_cluster:${var.eks_cluster_name}} by {host} < 20" # < 20 means > 80% used
  monitor_threshold_window = "15m"

  thresholds {
    critical = 20
    warning  = 30
  }

  notify_no_data    = false
  no_data_timeframe = 20

  # This is how you link to PagerDuty.
  # The @pagerduty-NAME will automatically route to the PagerDuty integration linked above.
  # Ensure the PagerDuty integration name matches what you'll configure in Datadog UI
  # or use the pagerduty_service_integration resource. This example assumes
  # a manual setup of the Datadog PagerDuty integration.
  # For full automation, you might need to manage the integration via Datadog's API (not directly available in Datadog provider for this type of integration yet)
  # or use the email integration type with a PagerDuty email service.
  # For simplicity, we are showing the common usage with the PagerDuty @notification.
  # The integration should be set up in Datadog UI under Integrations -> PagerDuty,
  # mapping the PagerDuty service to a Datadog handle like '@pagerduty-eks-observability'.
}

# --- Variables ---
variable "aws_region" {
  description = "AWS region for EKS cluster"
  type        = string
  default     = "us-east-1"
}

variable "eks_cluster_name" {
  description = "Name of the existing EKS cluster"
  type        = string
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_key" {
  description = "PagerDuty Global API Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_user_id" {
  description = "PagerDuty User ID for initial escalation"
  type        = string
}

variable "pagerduty_team_lead_user_id" {
  description = "PagerDuty User ID for team lead escalation"
  type        = string
}
    

Deployment Steps

Follow these steps to deploy the configuration:

Save the code: Save the Terraform configuration into .tf files (e.g., main.tf, variables.tf).
Initialize Terraform: Navigate to your Terraform directory and run terraform init.
Set variables: Provide your specific values for the variables. You can use a terraform.tfvars file or pass them via command line:
# Example terraform.tfvars eks_cluster_name = "my-production-eks-cluster" datadog_api_key = "YOUR_DD_API_KEY" datadog_app_key = "YOUR_DD_APP_KEY" pagerduty_api_key = "YOUR_PD_GLOBAL_API_KEY" pagerduty_user_id = "PXXXXXX" # PagerDuty User ID pagerduty_team_lead_user_id = "PXXXXXY" # PagerDuty User ID
Review the plan: Run terraform plan to see the changes Terraform will apply.
Apply the configuration: If the plan looks correct, execute terraform apply and type yes when prompted.
Configure Datadog-PagerDuty Integration:
- In Datadog, go to Integrations -> Integrations and search for PagerDuty.
- Add a new PagerDuty integration, providing the Service Key generated by the pagerduty_service_integration resource (you might need to output it from Terraform if using API, otherwise use the key from PagerDuty UI once created).
- Map the PagerDuty service to a Datadog @handle (e.g., @pagerduty-eks-observability). This handle is used in the Datadog monitor message.

Validation and Testing

After deployment, verify the setup:

Check Datadog Agent Status: In your EKS cluster, run kubectl get pods -n default | grep datadog. All Datadog Agent pods (daemonset and cluster agent) should be running and healthy.
Verify Datadog Data: Log into your Datadog account. You should see metrics, logs, and traces from your EKS cluster, nodes, and pods. Navigate to Infrastructure List or Kubernetes dashboard.
Test Datadog Monitor: Manually trigger a condition that would cause the CPU alert to fire (e.g., run a CPU-intensive workload on a node).
Verify PagerDuty Incident: A PagerDuty incident should be created and routed according to your escalation policy.

Advanced Considerations and Best Practices

Secrets Management: Never hardcode API keys. Use AWS Secrets Manager, HashiCorp Vault, or environment variables in CI/CD pipelines to inject sensitive data.
Module Reusability: Encapsulate this configuration into a reusable Terraform module for multiple EKS clusters or environments.
Environment Separation: Use separate AWS accounts or VPCs for production, staging, and development environments. Parameterize your Terraform code accordingly.
Fine-grained IAM Permissions: Refine the IAM policy for the Datadog Agent to the absolute minimum necessary permissions (least privilege principle).
Datadog Integrations: Explore other Datadog integrations for AWS services (RDS, Lambda, S3) and application-specific metrics.
Custom Monitors: Develop more sophisticated Datadog monitors for application-specific SLOs/SLIs.
PagerDuty Schedules & On-Call: Configure PagerDuty schedules, users, and teams comprehensively to match your operational structure.
GitOps Workflow: Integrate this Terraform configuration into a GitOps workflow (e.g., with Argo CD or Flux CD for Kubernetes-level resources, or dedicated CI/CD for Terraform) for automated deployments and change management.
Cost Management: Monitor Datadog usage to optimize costs, especially for log ingestion and custom metrics.

Troubleshooting Common Issues

Datadog Agent Pods Not Running:
Symptom: kubectl get pods shows Datadog pods in pending or error state. Solution: Check kubectl describe pod <pod-name> and kubectl logs <pod-name>. Common causes include insufficient resources, incorrect API/APP keys, or issues with IRSA permissions. Ensure the EKS OIDC provider is correctly configured.
No Data in Datadog:
Symptom: Datadog dashboards show no data for your EKS cluster. Solution: Verify the Datadog Agent is running and has network connectivity to Datadog endpoints. Check agent logs for errors related to API key authentication or metric submission. Confirm correct tags are applied.
PagerDuty Incidents Not Triggering:
Symptom: Datadog alerts but no PagerDuty incident is created. Solution: Ensure the Datadog-PagerDuty integration in Datadog is correctly configured with the PagerDuty service key. Verify that the @pagerduty-<handle> tag in your Datadog monitor message matches the handle configured in Datadog's PagerDuty integration settings. Check PagerDuty's event log for incoming events.
Terraform Authentication Errors:
Symptom: Terraform fails with AWS, Kubernetes, Datadog, or PagerDuty authentication errors. Solution: Double-check your AWS CLI configuration, Kubernetes context, and ensure all API keys (Datadog, PagerDuty) are correct and have the necessary permissions. For Kubernetes, ensure your local kubeconfig is updated and points to the correct EKS cluster.

Conclusion

Achieving production-ready observability for AWS EKS requires a robust strategy and reliable tooling. By leveraging Terraform to automate the deployment of Datadog for comprehensive monitoring and PagerDuty for effective incident response, you can establish a resilient operational framework for your Kubernetes workloads. This guide provides the foundation for integrating these critical tools, empowering your teams to proactively manage cluster health, rapidly detect issues, and minimize downtime for your cloud-native applications. Continuously refine your monitoring and alerting strategies to adapt to the evolving needs of your infrastructure and applications.

CloudStack YAML