Automated AWS EKS Monitoring and Alerting with Terraform, Datadog, and PagerDuty

In the dynamic landscape of cloud-native applications, maintaining robust observability for Kubernetes clusters is paramount. AWS Elastic Kubernetes Service (EKS) provides a managed Kubernetes control plane, but ensuring its health, performance, and security requires a sophisticated monitoring and alerting strategy. This guide details how to implement a fully automated, scalable, and resilient monitoring and alerting solution for AWS EKS using Terraform for Infrastructure as Code (IaC), Datadog for comprehensive observability, and PagerDuty for incident management.

Architecture Pro-Tip: Unified Observability Strategy

Before diving into implementation, define a clear observability strategy. Aim for a unified platform like Datadog to collect metrics, logs, and traces from your EKS cluster, applications, and underlying AWS infrastructure. This consolidated view simplifies correlation, speeds up root cause analysis, and reduces tool sprawl. Ensure your alerting policies are aligned with your service level objectives (SLOs) and escalation paths are well-defined in PagerDuty.

Why Automated EKS Monitoring is Critical

AWS EKS, while simplifying Kubernetes management, still presents complexities that demand vigilant monitoring. Workloads scale, pods crash, nodes can become unhealthy, and resource contention can cripple applications. Manual monitoring is not scalable or reliable. Automation ensures:

Proactive Issue Detection: Identify problems before they impact users.
Reduced MTTR (Mean Time To Resolution): Quicker identification and diagnosis of incidents.
Operational Efficiency: Free up engineering teams from manual checks.
Consistent Configuration: IaC ensures monitoring setups are standardized across environments.
Cost Optimization: Identify resource bottlenecks or underutilized resources.

Core Technologies Overview

AWS EKS (Elastic Kubernetes Service)

A fully managed Kubernetes service that allows you to run Kubernetes on AWS without needing to install, operate, and maintain your own Kubernetes control plane.

Terraform

An open-source IaC tool that enables you to define and provision infrastructure (including EKS clusters, Datadog configurations, and PagerDuty services) using a declarative configuration language.

Datadog

A leading cloud-native observability platform that unifies metrics, logs, traces, and synthetics into a single pane of glass. It provides deep visibility into EKS clusters, applications, and underlying AWS services.

PagerDuty

An incident management platform that provides real-time alerts, on-call scheduling, and automated escalations, ensuring critical issues are addressed promptly by the right team members.

Prerequisites

Before you begin, ensure you have the following:

An AWS Account with administrative access.
An active Datadog Account with API and APP keys.
An active PagerDuty Account with an API token and appropriate service setup.
Terraform CLI installed (version 1.0+ recommended).
Kubectl CLI installed and configured to connect to your EKS cluster.
Helm CLI installed (for Datadog Agent deployment).
An existing AWS EKS cluster. This guide assumes your EKS cluster is already provisioned.

Step-by-Step Implementation Guide

1. Configure Terraform Providers and Variables

Set up your Terraform providers for AWS, Datadog, and PagerDuty. Store sensitive API keys as environment variables or using a secrets manager like AWS Secrets Manager or HashiCorp Vault, rather than hardcoding them.

providers.tf

provider "aws" {
  region = var.aws_region
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.eks_cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.eks_cluster_auth.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.eks_cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.eks_cluster_auth.token
  }
}

variables.tf

variable "aws_region" {
  description = "AWS region for EKS cluster"
  type        = string
  default     = "us-east-1"
}

variable "eks_cluster_name" {
  description = "Name of the existing EKS cluster"
  type        = string
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

variable "pagerduty_team_name" {
  description = "PagerDuty team name to assign service to"
  type        = string
  default     = "DevOps Team"
}

2. Deploy Datadog Agent to EKS using Terraform and Helm

The Datadog Agent is deployed as a DaemonSet on your EKS cluster to collect metrics, logs, and traces. We'll use the Terraform Helm provider to manage the Datadog Agent Helm chart.

First, retrieve EKS cluster details:

data "aws_eks_cluster" "eks_cluster" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "eks_cluster_auth" {
  name = var.eks_cluster_name
}

Now, deploy the Datadog Agent:

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }

  # Enable APM for trace collection
  set {
    name  = "apm.enabled"
    value = "true"
  }

  # Enable log collection
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }

  # Enable Process Agent for full visibility
  set {
    name  = "processAgent.enabled"
    value = "true"
  }

  # Enable Live Container Monitoring
  set {
    name  = "liveContainerMonitoring.enabled"
    value = "true"
  }

  # EKS specific configurations (IRSA for IAM roles, etc.)
  # This typically requires an IAM role with required permissions (e.g., EC2 read-only, CloudWatch read-only)
  # and associating it with the Datadog Agent ServiceAccount.
  # For simplicity, we assume default permissions or an existing IAM role is configured for the NodeGroup.
  # For IRSA, you'd add:
  # set {
  #   name  = "datadog.kubelet.tlsVerify"
  #   value = "false" # Use for self-signed Kubelet certs on some EKS setups
  # }
  # set {
  #   name  = "rbac.create"
  #   value = "true"
  # }
  # set {
  #   name  = "clusterAgent.rbac.create"
  #   value = "true"
  # }
  # set {
  #   name  = "clusterAgent.apm.enabled"
  #   value = "true"
  # }
  # set {
  #   name  = "clusterAgent.metricsProvider.enabled"
  #   value = "true"
  # }
  # set {
  #   name  = "targetSystem"
  #   value = "linux"
  # }

  # For full IRSA setup, you would create an IAM role and service account
  # and set `serviceAccount.create=true` and `serviceAccount.annotations."eks.amazonaws.com/role-arn" = "arn:aws:iam::..."`
  # via the Helm values.
}

3. Integrate Datadog with PagerDuty using Terraform

To route Datadog alerts to PagerDuty, you need to set up the integration within Datadog and specify a PagerDuty service.

First, define a PagerDuty service that Datadog will integrate with. This service will receive incidents when Datadog monitors trigger alerts.

resource "pagerduty_user" "team_member" {
  name  = "Datadog Integrator"
  email = "datadog-integrator@example.com" # Replace with a valid email
}

resource "pagerduty_escalation_policy" "default" {
  name      = "Default EKS Escalation Policy"
  num_loops = 2
  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "user"
      id   = pagerduty_user.team_member.id
    }
  }
}

resource "pagerduty_service" "eks_monitoring_service" {
  name                    = "EKS Cluster Monitoring Service"
  auto_resolve_timeout    = "14400" # 4 hours
  acknowledgement_timeout = "600"   # 10 minutes
  escalation_policy       = pagerduty_escalation_policy.default.id
}

resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog"
  service_id  = pagerduty_service.eks_monitoring_service.id
  type        = "datadog_api_inbound_integration"
}

# The Datadog PagerDuty integration doesn't directly use this resource
# but rather leverages the `pagerduty` notification channel specified in Datadog monitors.
# The `pagerduty_service_integration` provides the integration key needed by Datadog.
output "pagerduty_datadog_integration_key" {
  description = "The integration key for Datadog to PagerDuty."
  value       = pagerduty_service_integration.datadog_integration.integration_key
  sensitive   = true
}

Next, configure the Datadog PagerDuty integration. This is typically done through the Datadog UI or via a Datadog API call if not directly supported by the Terraform provider for all aspects. However, the `datadog_monitor` resource can directly leverage PagerDuty notification channels. We can also ensure the integration exists using the `datadog_integration_pagerduty` resource.

resource "datadog_integration_pagerduty" "pagerduty_integration" {
  services = [
    {
      service_name = pagerduty_service.eks_monitoring_service.name
      service_key  = pagerduty_service_integration.datadog_integration.integration_key
    }
  ]
}

4. Define Datadog Monitors with PagerDuty Alerting

Now, define critical EKS monitors in Datadog using Terraform. Each monitor can be configured to send alerts to the PagerDuty service we just created. Here are a few essential examples:

# Monitor: EKS Node CPU Utilization High
resource "datadog_monitor" "eks_node_cpu_high" {
  name                = "[EKS] High Node CPU Utilization on {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster_name:${var.eks_cluster_name}} by {host} < 10" # Less than 10% idle means >90% usage
  message             = "EKS Node {{host.name}} is experiencing high CPU utilization ({{value}}%). Investigate potential runaway processes or resource constraints. @pagerduty-${pagerduty_service.eks_monitoring_service.name}"
  tags                = ["environment:production", "service:eks", "alert-type:performance"]
  monitor_threshold_windows {
    recovery_window = "10m"
  }
  renotify_interval   = 60
  no_data_timeframe   = 20
  include_tags        = true
  notification_prefetch = true
  escalation_message = "CPU still high after 60 minutes. Please escalate."
  thresholds {
    critical = 10
    warning  = 20
  }
}

# Monitor: EKS Node Memory Utilization High
resource "datadog_monitor" "eks_node_memory_high" {
  name                = "[EKS] High Node Memory Utilization on {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:system.mem.used{kubernetes_cluster_name:${var.eks_cluster_name}} by {host} > 90" # Greater than 90% usage
  message             = "EKS Node {{host.name}} is experiencing high Memory utilization ({{value}}%). Check for memory leaks or pod misconfigurations. @pagerduty-${pagerduty_service.eks_monitoring_service.name}"
  tags                = ["environment:production", "service:eks", "alert-type:performance"]
  monitor_threshold_windows {
    recovery_window = "10m"
  }
  renotify_interval   = 60
  no_data_timeframe   = 20
  include_tags        = true
  notification_prefetch = true
  thresholds {
    critical = 90
    warning  = 80
  }
}

# Monitor: EKS Pod Restarts Rate High
resource "datadog_monitor" "eks_pod_restarts_high" {
  name                = "[EKS] High Pod Restart Rate in {{kube_namespace.name}}/{{kube_app.name}}"
  type                = "metric alert"
  query               = "sum(last_5m):sum:kubernetes.pod.restarts{kubernetes_cluster_name:${var.eks_cluster_name}} by {kube_namespace,kube_app} > 3" # More than 3 restarts in 5 minutes
  message             = "Pod {{kube_app.name}} in namespace {{kube_namespace.name}} is restarting frequently ({{value}} restarts). Investigate application stability. @pagerduty-${pagerduty_service.eks_monitoring_service.name}"
  tags                = ["environment:production", "service:eks", "alert-type:availability"]
  monitor_threshold_windows {
    recovery_window = "10m"
  }
  renotify_interval   = 60
  no_data_timeframe   = 20
  include_tags        = true
  notification_prefetch = true
  thresholds {
    critical = 3
    warning  = 1
  }
}

# Monitor: EKS Node Not Ready
resource "datadog_monitor" "eks_node_not_ready" {
  name                = "[EKS] Node {{host.name}} is Not Ready"
  type                = "metric alert"
  query               = "max(last_5m):max:kubernetes.node.ready{kubernetes_cluster_name:${var.eks_cluster_name}} by {host} == 0" # 0 means not ready
  message             = "EKS Node {{host.name}} is reporting Not Ready. This impacts workload scheduling. @pagerduty-${pagerduty_service.eks_monitoring_service.name}"
  tags                = ["environment:production", "service:eks", "alert-type:availability"]
  monitor_threshold_windows {
    recovery_window = "10m"
  }
  renotify_interval   = 30
  no_data_timeframe   = 10
  include_tags        = true
  notification_prefetch = true
  thresholds {
    critical = 0
  }
}

Comprehensive Terraform Configuration Example

Here's a consolidated view of the Terraform setup. Save these into files like main.tf, variables.tf, and providers.tf in your Terraform project directory. Remember to populate the sensitive variables through environment variables (e.g., TF_VAR_datadog_api_key) or a secrets management solution.

# main.tf

# EKS Cluster Data Source
data "aws_eks_cluster" "eks_cluster" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "eks_cluster_auth" {
  name = var.eks_cluster_name
}

# Deploy Datadog Agent using Helm
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }

  set {
    name  = "apm.enabled"
    value = "true"
  }

  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }

  set {
    name  = "processAgent.enabled"
    value = "true"
  }

  set {
    name  = "liveContainerMonitoring.enabled"
    value = "true"
  }

  # Cluster Agent for advanced features and reduced agent footprint
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "clusterAgent.metricsProvider.enabled"
    value = "true"
  }

  # RBAC for Datadog Agent
  set {
    name  = "rbac.create"
    value = "true"
  }
}

# PagerDuty Resources
resource "pagerduty_user" "team_member" {
  name  = "Datadog Integrator"
  email = "datadog-integrator@example.com"
}

resource "pagerduty_escalation_policy" "default" {
  name      = "Default EKS Escalation Policy"
  num_loops = 2
  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "user"
      id   = pagerduty_user.team_member.id
    }
  }
}

resource "pagerduty_service" "eks_monitoring_service" {
  name                    = "EKS Cluster Monitoring Service"
  auto_resolve_timeout    = "14400"
  acknowledgement_timeout = "600"
  escalation_policy       = pagerduty_escalation_policy.default.id
  alert_creation          = "create_alerts_and_incidents"
}

resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog"
  service_id  = pagerduty_service.eks_monitoring_service.id
  type        = "datadog_api_inbound_integration"
}

# Datadog PagerDuty Integration (ensures Datadog knows about the PagerDuty service)
resource "datadog_integration_pagerduty" "pagerduty_integration" {
  services = [
    {
      service_name = pagerduty_service.eks_monitoring_service.name
      service_key  = pagerduty_service_integration.datadog_integration.integration_key
    }
  ]
}

# Datadog Monitors for EKS
resource "datadog_monitor" "eks_node_cpu_high" {
  name                = "[EKS] High Node CPU Utilization on {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster_name:${var.eks_cluster_name}} by {host} < 10"
  message             = "EKS Node {{host.name}} is experiencing high CPU utilization ({{value}}%). Investigate. @pagerduty-${pagerduty_service.eks_monitoring_service.name}"
  tags                = ["environment:production", "service:eks", "alert-type:performance", "severity:high"]
  monitor_threshold_windows { recovery_window = "10m" }
  renotify_interval   = 60
  no_data_timeframe   = 20
  include_tags        = true
  notification_prefetch = true
  thresholds { critical = 10; warning  = 20 }
}

resource "datadog_monitor" "eks_node_memory_high" {
  name                = "[EKS] High Node Memory Utilization on {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:system.mem.used{kubernetes_cluster_name:${var.eks_cluster_name}} by {host} > 90"
  message             = "EKS Node {{host.name}} is experiencing high Memory utilization ({{value}}%). Check for memory leaks. @pagerduty-${pagerduty_service.eks_monitoring_service.name}"
  tags                = ["environment:production", "service:eks", "alert-type:performance", "severity:high"]
  monitor_threshold_windows { recovery_window = "10m" }
  renotify_interval   = 60
  no_data_timeframe   = 20
  include_tags        = true
  notification_prefetch = true
  thresholds { critical = 90; warning  = 80 }
}

resource "datadog_monitor" "eks_pod_restarts_high" {
  name                = "[EKS] High Pod Restart Rate in {{kube_namespace.name}}/{{kube_app.name}}"
  type                = "metric alert"
  query               = "sum(last_5m):sum:kubernetes.pod.restarts{kubernetes_cluster_name:${var.eks_cluster_name}} by {kube_namespace,kube_app} > 3"
  message             = "Pod {{kube_app.name}} in namespace {{kube_namespace.name}} is restarting frequently ({{value}} restarts). Investigate. @pagerduty-${pagerduty_service.eks_monitoring_service.name}"
  tags                = ["environment:production", "service:eks", "alert-type:availability", "severity:critical"]
  monitor_threshold_windows { recovery_window = "10m" }
  renotify_interval   = 60
  no_data_timeframe   = 20
  include_tags        = true
  notification_prefetch = true
  thresholds { critical = 3; warning = 1 }
}

resource "datadog_monitor" "eks_node_not_ready" {
  name                = "[EKS] Node {{host.name}} is Not Ready"
  type                = "metric alert"
  query               = "max(last_5m):max:kubernetes.node.ready{kubernetes_cluster_name:${var.eks_cluster_name}} by {host} == 0"
  message             = "EKS Node {{host.name}} is reporting Not Ready. This impacts workload scheduling. @pagerduty-${pagerduty_service.eks_monitoring_service.name}"
  tags                = ["environment:production", "service:eks", "alert-type:availability", "severity:critical"]
  monitor_threshold_windows { recovery_window = "10m" }
  renotify_interval   = 30
  no_data_timeframe   = 10
  include_tags        = true
  notification_prefetch = true
  thresholds { critical = 0 }
}

# outputs.tf
output "pagerduty_datadog_integration_key" {
  description = "The integration key for Datadog to PagerDuty."
  value       = pagerduty_service_integration.datadog_integration.integration_key
  sensitive   = true
}

To deploy this configuration:

Save the code into your Terraform project.
Initialize Terraform: terraform init
Review the plan: terraform plan -var="eks_cluster_name=your-eks-cluster-name"
Apply the changes: terraform apply -var="eks_cluster_name=your-eks-cluster-name"

Testing and Validation

After applying the Terraform configuration:

Datadog Agent: Verify that Datadog Agents are running correctly on all EKS nodes using kubectl get pods -n datadog. Check the Datadog UI for incoming metrics, logs, and traces.
Datadog Monitors: Navigate to the "Monitors" section in Datadog to confirm that your newly created monitors are listed and in an "OK" state.
PagerDuty Integration: In PagerDuty, confirm the "EKS Cluster Monitoring Service" exists. Trigger a test alert (e.g., by simulating high CPU usage on a non-critical node if possible, or using a Datadog monitor test feature) to ensure it creates an incident in PagerDuty and triggers the escalation policy.

Best Practices for EKS Monitoring and Alerting

Granular Monitoring: Beyond cluster-level metrics, monitor individual deployments, services, and pods for specific application health and performance indicators.
Custom Dashboards: Create Datadog dashboards tailored to different roles (e.g., SRE, Developer, Management) for quick insights.
Log Management: Ensure comprehensive log collection from all containers and system components. Use Datadog's log processing pipelines for enrichment and filtering.
Distributed Tracing: Instrument your applications for distributed tracing (e.g., OpenTelemetry, Datadog APM) to gain end-to-end visibility into requests across microservices.
Runbook Automation: For common alerts, attach runbooks to PagerDuty services or Datadog monitor messages to guide responders.
Alert Fatigue: Regularly review and fine-tune your alerts to minimize noise and prevent alert fatigue. Focus on actionable alerts.
Cost Monitoring: Monitor AWS EKS costs alongside performance to optimize resource allocation.
Security Monitoring: Integrate security tools and audit logs (e.g., Kubernetes Audit Logs, Falco, Trivy) into Datadog for a unified security observability posture.

Troubleshooting and Common Issues

Datadog Agent Pods Not Running: Check kubectl describe pod <datadog-agent-pod-name> -n datadog for events or errors. Ensure correct API/APP keys, sufficient node resources, and proper RBAC permissions.
No Metrics in Datadog: Verify network connectivity from EKS nodes to Datadog endpoints. Check Datadog Agent logs for errors (kubectl logs <datadog-agent-pod-name> -n datadog). Ensure correct IAM roles are attached to EKS node groups or service accounts for AWS integration.
PagerDuty Incidents Not Triggering: Double-check the @pagerduty-<service-name> tag in your Datadog monitor message. Ensure the Datadog-PagerDuty integration key is correctly configured in Datadog. Check PagerDuty event logs for incoming events.
Terraform Provider Authentication: Ensure environment variables for API keys are correctly set (e.g., DATADOG_API_KEY, DATADOG_APP_KEY, PAGERDUTY_TOKEN) or passed via -var flags.

Conclusion

Automating AWS EKS monitoring and alerting with Terraform, Datadog, and PagerDuty establishes a robust, scalable, and reliable observability foundation. By leveraging Infrastructure as Code, you ensure consistency, auditability, and rapid deployment across your environments. This integrated approach empowers your DevOps teams to maintain high availability, proactively address performance bottlenecks, and respond to incidents with agility, ultimately delivering a superior experience for your users. Embrace this pattern to elevate your cloud-native operations to the next level.

CloudStack YAML