Terraform for AWS EKS Monitoring and Alerting with Datadog and PagerDuty

Managing Kubernetes clusters, especially AWS EKS, demands sophisticated monitoring and robust alerting to ensure high availability and performance. As applications scale and microservices architectures become more prevalent, the complexity of observing these systems grows exponentially. This guide demonstrates how to leverage Terraform for Infrastructure as Code (IaC) to seamlessly integrate Datadog for comprehensive monitoring and PagerDuty for effective incident management within your AWS EKS environment.

Architecture Pro-Tip: Observability as Code

Treat your monitoring and alerting configurations as critical infrastructure. By defining Datadog monitors, PagerDuty services, and EKS integrations through Terraform, you achieve "Observability as Code." This approach ensures consistency, version control, auditability, and faster recovery. It also promotes a shift-left strategy, allowing developers to define monitoring alongside their application deployments, fostering greater ownership and operational excellence.

Why Terraform, Datadog, and PagerDuty for EKS?

Terraform: Infrastructure as Code for Consistency

Terraform allows you to define and provision your AWS infrastructure, including EKS clusters, and crucially, your monitoring tools, using declarative configuration files. This means your entire observability stack, from Datadog agents to specific alert thresholds, can be version-controlled, reviewed, and deployed reliably across environments.

Automation: Eliminate manual configuration errors and speed up deployment.
Version Control: Track changes, revert to previous states, and collaborate effectively.
Repeatability: Spin up identical monitoring configurations for multiple clusters or environments.

Datadog: Comprehensive EKS Observability

Datadog provides a unified platform for metrics, logs, traces, and user experience monitoring. For EKS, it offers deep insights into cluster health, node performance, pod statuses, and application-level metrics, all visualized through powerful dashboards and intelligent monitors.

Full-Stack Visibility: Monitor everything from the AWS infrastructure layer to individual application containers.
Kubernetes-Native Monitoring: Auto-discovery of services, rich metadata, and pre-built dashboards for EKS.
Advanced Alerting: Create sophisticated monitors based on various data sources with intelligent anomaly detection.

PagerDuty: Streamlined Incident Response

PagerDuty takes the alerts from Datadog and routes them to the right on-call teams based on schedules, escalation policies, and incident priorities. It ensures that critical issues are never missed and are addressed promptly.

Intelligent Alert Routing: Deliver alerts to the correct personnel based on defined schedules.
Escalation Policies: Automatically escalate incidents if not acknowledged within a set timeframe.
Incident Lifecycle Management: Track, manage, and resolve incidents efficiently.

Prerequisites

Before you begin, ensure you have the following:

An active AWS account with sufficient permissions to create and manage EKS clusters and IAM resources.
Terraform CLI installed (version 1.0+ recommended).
An active Datadog account with API and Application keys.
An active PagerDuty account with a service to integrate.
aws-cli configured with appropriate credentials.
kubectl installed and configured to connect to your EKS cluster.
helm CLI installed (for Datadog Agent deployment).

Step-by-Step Implementation with Terraform

1. Project Setup and Providers Configuration

Start by creating a new directory for your Terraform project and defining the required providers: aws, datadog, pagerduty, and kubernetes/helm for EKS interactions.

provider "aws" {
  region = "us-east-1"
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.example.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.example.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.example.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.example.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.example.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.example.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_token
}

# Data sources to retrieve EKS cluster details
data "aws_eks_cluster" "example" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "example" {
  name = var.eks_cluster_name
}

variable "eks_cluster_name" {
  description = "The name of your EKS cluster."
  type        = string
}

variable "datadog_api_key" {
  description = "Your Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Your Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_token" {
  description = "Your PagerDuty API Token"
  type        = string
  sensitive   = true
}
    

Make sure to replace us-east-1 with your AWS region and set the eks_cluster_name variable. The Datadog and PagerDuty API keys should be provided via environment variables or a terraform.tfvars file, ensuring they are marked as sensitive.

2. Deploying the Datadog Agent to EKS

The Datadog Agent is typically deployed as a DaemonSet within your Kubernetes cluster. Using the Helm provider in Terraform is the most straightforward way to manage its lifecycle.

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "datadog.site"
    value = "datadoghq.com" # Or eu.datadoghq.com, us3.datadoghq.com, etc.
  }

  set {
    name  = "kubeStateMetricsCore.enabled"
    value = "true"
  }

  set {
    name  = "datadog.tags[0]"
    value = "environment:${var.environment}"
  }
  set {
    name  = "datadog.tags[1]"
    value = "cluster:${var.eks_cluster_name}"
  }
  # Add other relevant configurations like APM, Log Collection, CRI Socket paths as needed
}

variable "environment" {
  description = "The deployment environment (e.g., dev, staging, prod)."
  type        = string
}
    

This configuration deploys the Datadog Agent, including the cluster agent and kube-state-metrics core, enabling comprehensive monitoring of your EKS cluster and its workloads. It also adds useful tags for filtering and organization in Datadog.

3. Integrating Datadog with PagerDuty

Before creating monitors, set up the integration between Datadog and PagerDuty. This is done by creating a PagerDuty service in PagerDuty and then connecting it via Datadog's integration.

resource "pagerduty_user" "devops_engineer" {
  name  = "DevOps Engineer"
  email = "devops@example.com"
}

resource "pagerduty_team" "devops_team" {
  name        = "DevOps Team"
  description = "Team responsible for DevOps infrastructure."
}

resource "pagerduty_team_membership" "devops_engineer_membership" {
  user_id = pagerduty_user.devops_engineer.id
  team_id = pagerduty_team.devops_team.id
  role    = "observer"
}

resource "pagerduty_escalation_policy" "eks_escalation_policy" {
  name      = "${var.eks_cluster_name}-EKS Escalation Policy"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "user"
      id   = pagerduty_user.devops_engineer.id
    }
  }
}

resource "pagerduty_service" "eks_monitoring_service" {
  name                    = "${var.eks_cluster_name}-EKS Monitoring Service"
  description             = "Service for EKS cluster monitoring alerts."
  auto_resolve_timeout    = "14400" # 4 hours
  acknowledgement_timeout = "600"   # 10 minutes
  escalation_policy       = pagerduty_escalation_policy.eks_escalation_policy.id
  team                    = pagerduty_team.devops_team.id
}

resource "datadog_integration_pagerduty" "pagerduty_integration" {
  services {
    service_name = pagerduty_service.eks_monitoring_service.name
    service_key  = pagerduty_service.eks_monitoring_service.integration_keys[0].id
  }
}
    

This block creates a PagerDuty user, team, an escalation policy, and a service specifically for EKS monitoring alerts. Finally, it uses the datadog_integration_pagerduty resource to link this PagerDuty service with your Datadog account. Replace placeholder emails and adjust escalation policies as per your organizational needs.

Terraform Configuration Example: EKS Monitoring with Datadog and PagerDuty

Here's a ready-to-use example of Terraform code combining the above steps, including a few critical Datadog monitors configured to alert via the PagerDuty service. This example assumes you have an EKS cluster named my-eks-cluster and environment set to prod.

# main.tf

# --- AWS Provider Configuration ---
provider "aws" {
  region = "us-east-1" # Or your desired region
}

data "aws_eks_cluster" "example" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "example" {
  name = var.eks_cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.example.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.example.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.example.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.example.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.example.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.example.token
  }
}

# --- Datadog Provider Configuration ---
provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

# --- PagerDuty Provider Configuration ---
provider "pagerduty" {
  token = var.pagerduty_token
}

# --- Datadog Agent Deployment on EKS via Helm ---
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "datadog.site"
    value = "datadoghq.com"
  }

  set {
    name  = "kubeStateMetricsCore.enabled"
    value = "true"
  }

  set {
    name  = "datadog.tags[0]"
    value = "environment:${var.environment}"
  }
  set {
    name  = "datadog.tags[1]"
    value = "cluster:${var.eks_cluster_name}"
  }

  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }
  # Enable APM if needed
  set {
    name  = "apm.enabled"
    value = "true"
  }
}

# --- PagerDuty Service Configuration ---
resource "pagerduty_user" "devops_engineer" {
  name  = "DevOps Engineer"
  email = "devops@example.com" # Replace with a real email
}

resource "pagerduty_team" "devops_team" {
  name        = "DevOps Team"
  description = "Team responsible for EKS infrastructure and applications."
}

resource "pagerduty_team_membership" "devops_engineer_membership" {
  user_id = pagerduty_user.devops_engineer.id
  team_id = pagerduty_team.devops_team.id
  role    = "observer" # Or 'responder'
}

resource "pagerduty_escalation_policy" "eks_critical_policy" {
  name      = "${var.eks_cluster_name}-EKS Critical Escalation Policy"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "user"
      id   = pagerduty_user.devops_engineer.id
    }
  }
}

resource "pagerduty_service" "eks_critical_service" {
  name                    = "${var.eks_cluster_name}-EKS Critical Alerts"
  description             = "PagerDuty service for critical EKS cluster alerts."
  auto_resolve_timeout    = "14400" # 4 hours
  acknowledgement_timeout = "600"   # 10 minutes
  escalation_policy       = pagerduty_escalation_policy.eks_critical_policy.id
  team                    = pagerduty_team.devops_team.id
}

# --- Datadog PagerDuty Integration ---
resource "datadog_integration_pagerduty" "main_integration" {
  services {
    service_name = pagerduty_service.eks_critical_service.name
    service_key  = pagerduty_service.eks_critical_service.integration_keys[0].id
  }
}

# --- Datadog Monitors for EKS ---

# EKS Node Not Ready
resource "datadog_monitor" "eks_node_not_ready" {
  name = "[${var.environment}] EKS Node Not Ready on {{cluster.name}}"
  type = "metric alert"
  query = "sum(last_5m):kubernetes.node.not_ready{cluster_name:${var.eks_cluster_name},environment:${var.environment}} > 0"
  message = "One or more EKS nodes are not ready in cluster {{cluster.name}}.\n\n@pagerduty-${pagerduty_service.eks_critical_service.name}"
  tags = ["eks", "node", "critical", "pagerduty", "environment:${var.environment}", "cluster:${var.eks_cluster_name}"]
  notify_no_data = false
  new_group_delay = 60
  no_data_timeframe = 20
  renotify_interval = 0
  escalation_message = "Node(s) still not ready, escalating to management."
}

# EKS Critical Pod Restart Rate
resource "datadog_monitor" "eks_critical_pod_restart_rate" {
  name = "[${var.environment}] High Critical Pod Restart Rate on {{cluster_name.name}}"
  type = "metric alert"
  query = "sum(last_5m):kubernetes.pod.restarts{environment:${var.environment},cluster:${var.eks_cluster_name},kube_namespace:my-critical-namespace} > 5" # Adjust namespace and threshold
  message = "Critical pods in 'my-critical-namespace' are restarting too frequently in cluster {{cluster_name.name}}.\n\n@pagerduty-${pagerduty_service.eks_critical_service.name}"
  tags = ["eks", "pod", "restart", "critical", "pagerduty", "environment:${var.environment}", "cluster:${var.eks_cluster_name}"]
  notify_no_data = false
  new_group_delay = 60
  no_data_timeframe = 20
  renotify_interval = 0
}

# EKS Node CPU Utilization Exceeded
resource "datadog_monitor" "eks_node_cpu_utilization" {
  name = "[${var.environment}] EKS Node CPU Utilization High on {{host.name}}"
  type = "metric alert"
  query = "avg(last_5m):system.cpu.idle{environment:${var.environment},cluster:${var.eks_cluster_name}} by {host} > 80" # Alert if CPU idle is < 20% (i.e. utilization > 80%)
  message = "Node {{host.name}} in EKS cluster {{cluster.name}} is experiencing high CPU utilization (over 80%).\n\n@pagerduty-${pagerduty_service.eks_critical_service.name}"
  tags = ["eks", "node", "cpu", "performance", "pagerduty", "environment:${var.environment}", "cluster:${var.eks_cluster_name}"]
  notify_no_data = false
  new_group_delay = 60
  no_data_timeframe = 20
  renotify_interval = 0
  thresholds {
    critical = 80
    warning  = 70
  }
}

# variables.tf
variable "eks_cluster_name" {
  description = "The name of your EKS cluster."
  type        = string
  default     = "my-eks-cluster" # Example
}

variable "environment" {
  description = "The deployment environment (e.g., dev, staging, prod)."
  type        = string
  default     = "prod" # Example
}

variable "datadog_api_key" {
  description = "Your Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Your Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_token" {
  description = "Your PagerDuty API Token"
  type        = string
  sensitive   = true
}

# terraform.tfvars (example - do not commit sensitive data)
# datadog_api_key = "YOUR_DATADOG_API_KEY"
# datadog_app_key = "YOUR_DATADOG_APP_KEY"
# pagerduty_token = "YOUR_PAGERDUTY_TOKEN"
# eks_cluster_name = "your-actual-eks-cluster-name"
# environment = "production"
    

To deploy this configuration:

Save the code blocks into main.tf and variables.tf in your project directory.
Create a terraform.tfvars file (or use environment variables) to provide your sensitive API keys and cluster name.
Run terraform init to initialize the providers.
Run terraform plan to preview the changes.
Run terraform apply to provision the resources.

Datadog Monitors Explained

The example provides three essential monitors:

EKS Node Not Ready: Alerts if any Kubernetes node reports a NotReady status, indicating a severe cluster health issue.
EKS Critical Pod Restart Rate: Triggers if pods in a specified namespace (e.g., for critical applications) are restarting too frequently, suggesting application instability.
EKS Node CPU Utilization High: Notifies if a node's CPU usage consistently exceeds a high threshold, pointing to potential performance bottlenecks.

Notice the @pagerduty-{{SERVICE_NAME}} syntax in the monitor's message. This is how Datadog automatically routes the alert to the specified PagerDuty service.

Best Practices for EKS Monitoring with Terraform

Tagging Strategy: Consistently apply tags (e.g., environment, cluster, service, owner) to all resources managed by Terraform. This improves filtering, cost allocation, and monitor specificity in Datadog.
Modularity: Break down your Terraform configuration into logical modules (e.g., eks-cluster, datadog-agents, datadog-monitors, pagerduty-services). This enhances reusability and maintainability.
Alert Fatigue: Design your monitors carefully. Start with critical alerts that require immediate action and gradually refine thresholds to minimize false positives. Use Datadog's anomaly detection and forecast monitors for smarter alerting.
Dashboarding: While Terraform doesn't directly manage Datadog dashboards (though it can using datadog_dashboard), ensure your monitoring setup is complemented by informative dashboards for quick visual inspection and troubleshooting.
Security: Always use sensitive variables for API keys and tokens. Store them securely using tools like AWS Secrets Manager, HashiCorp Vault, or your CI/CD's secret management.
Continuous Improvement: Regularly review your monitors and alerting policies. As your EKS environment evolves, so should your observability strategy.

Troubleshooting Common Issues

Datadog Agent Not Reporting:
- Check Kubernetes logs for the Datadog Agent pods (kubectl logs -n datadog -l app=datadog).
- Verify that datadog.apiKey and datadog.appKey are correctly passed to the Helm chart.
- Ensure the Datadog Agent has network connectivity to Datadog's endpoints.
PagerDuty Alerts Not Triggering:
- Confirm the datadog_integration_pagerduty resource applied successfully and the integration shows active in Datadog UI.
- Double-check the PagerDuty service name in the Datadog monitor's message (e.g., @pagerduty-EKS Critical Alerts). It must exactly match the name configured in Datadog.
- Verify your PagerDuty service has an active escalation policy and on-call schedule.
Terraform EKS Provider Authentication Issues:
- Ensure your aws-cli is configured with credentials that have permission to describe EKS clusters (eks:DescribeCluster) and generate EKS authentication tokens (eks:DescribeCluster implicitly allows sts:GetCallerIdentity needed by aws-iam-authenticator).
- Your local kubeconfig context might be interfering; ensure Terraform is using the credentials it expects.

Conclusion

By adopting Terraform for managing your AWS EKS monitoring and alerting stack, you streamline operations, enhance reliability, and gain unparalleled visibility into your containerized workloads. The combination of Datadog for deep observability and PagerDuty for critical incident response, all orchestrated through Infrastructure as Code, establishes a robust and scalable foundation for any modern DevOps team. Embrace this approach to build resilient, self-healing EKS environments and significantly improve your mean time to resolution (MTTR).

CloudStack YAML