Terraform for AWS EKS Datadog Monitoring & PagerDuty Alerting

Terraform for AWS EKS: Comprehensive Datadog Monitoring & PagerDuty Alerting

In the dynamic world of cloud-native applications, maintaining robust observability and rapid incident response for your Kubernetes clusters is paramount. AWS Elastic Kubernetes Service (EKS) provides a managed platform for running Kubernetes, but effective monitoring and alerting require specialized tools. This guide will walk you through leveraging Terraform to establish a seamless, automated monitoring and alerting pipeline for AWS EKS using Datadog for comprehensive observability and PagerDuty for reliable incident management.

Architecture Pro-Tip: Always treat your monitoring and alerting infrastructure as code. Integrating Datadog and PagerDuty into your Terraform workflows ensures consistency, version control, and rapid disaster recovery for your observability stack, aligning perfectly with modern GitOps principles for AWS EKS deployments.

Prerequisites

Before you begin, ensure you have the following:

An active AWS Account with administrative privileges.
A Datadog Account with API and Application Keys.
A PagerDuty Account with a Service and Integration Key.
Terraform CLI (v1.0+) installed.
AWS CLI configured with appropriate credentials.
Kubectl configured to connect to your AWS EKS cluster.
Helm CLI installed (for Datadog Agent deployment).

Why Terraform, Datadog, and PagerDuty?

This powerful combination delivers a robust solution for EKS observability:

Terraform: Enables Infrastructure as Code (IaC) for your entire monitoring setup, from AWS resources to Datadog monitors and PagerDuty services. This guarantees repeatability, version control, and simplified management.
Datadog: A comprehensive monitoring platform offering metrics, logs, traces, and synthetics for EKS clusters. It provides deep visibility into Kubernetes nodes, pods, deployments, and the underlying AWS infrastructure.
PagerDuty: An industry-leading incident management platform that orchestrates timely alerts, escalations, and on-call rotations, ensuring critical issues from Datadog are never missed and are addressed promptly by the right team.

Key Concepts for Integrated Monitoring

Datadog Agent on EKS

The Datadog Agent is a lightweight open-source software that collects events and metrics from hosts and sends them to Datadog. For EKS, it's typically deployed as a DaemonSet, ensuring an agent runs on every node to collect node-level metrics, pod-level metrics, and logs. It can also integrate with Kubernetes API for cluster-level events and states.

Datadog AWS Integration

Beyond the EKS cluster itself, your applications rely on various AWS services (EC2, RDS, S3, etc.). Datadog's AWS integration allows you to pull metrics and logs from these services directly into Datadog, providing a holistic view of your entire cloud environment. This is typically configured via an AWS IAM Role, granting Datadog read-only access to CloudWatch metrics and other relevant services.

Datadog Monitors & PagerDuty Integration

Datadog monitors allow you to define alert conditions based on various data sources (metrics, logs, traces, Uptime). When a condition is met, a notification is triggered. By integrating with PagerDuty, these notifications can be transformed into actionable incidents, leveraging PagerDuty's powerful on-call scheduling, escalation policies, and notification channels (SMS, phone calls, push notifications).

Step-by-Step Implementation with Terraform

We will use a modular approach for clarity. Assume your EKS cluster is already provisioned; this guide focuses on the monitoring and alerting components.

1. Configure AWS, Datadog, and PagerDuty Providers

First, set up your Terraform providers. Ensure your AWS credentials are configured (e.g., via environment variables or ~/.aws/credentials). Your Datadog and PagerDuty API keys should be stored securely, ideally using a secrets manager or environment variables, and referenced in variables.tf.

2. Integrate Datadog with Your AWS Account

This step establishes the necessary IAM role and policy in your AWS account, allowing Datadog to pull metrics from CloudWatch and other AWS services. The datadog_integration_aws resource handles this.

3. Deploy Datadog Agent to AWS EKS

The Datadog Agent is deployed using its official Helm chart. Terraform's helm_release resource provides a declarative way to manage Helm deployments within your cluster. We'll configure it with your Datadog API key and other necessary parameters.

# main.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.0"
    }
    pagerduty = {
      source  = "pagerduty/pagerduty"
      version = "~> 1.0"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

# Assume EKS cluster is already provisioned and its data is available
# Or define it here if you manage EKS with Terraform
# For this guide, we'll assume it's imported or fetched.
# Example: data "aws_eks_cluster" "eks_cluster" { name = var.eks_cluster_name }
data "aws_eks_cluster" "example" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "example" {
  name = var.eks_cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.example.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.example.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.example.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.example.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.example.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.example.token
  }
}

# datadog_aws_integration.tf
resource "datadog_integration_aws" "main" {
  account_id = data.aws_caller_identity.current.account_id
  role_name  = "DatadogIntegrationRole"
  host_tags  = ["env:production", "project:eks-monitoring"]
}

data "aws_caller_identity" "current" {}

# datadog_agent.tf
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }

  set {
    name  = "datadog.site"
    value = "datadoghq.com" # or eu.datadoghq.com
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "targetSystem"
    value = "linux"
  }

  set {
    name  = "kubeStateMetricsExternal.enabled"
    value = "true"
  }

  set {
    name  = "prometheusScrape.enabled"
    value = "true"
  }

  # RBAC configuration for Kubernetes integration
  set {
    name  = "rbac.create"
    value = "true"
  }

  # Enable APM and Log collection
  set {
    name  = "apm.enabled"
    value = "true"
  }

  set {
    name  = "logs.enabled"
    value = "true"
  }

  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }
  
  # Set EKS specific tags
  set {
    name  = "tags"
    value = "{env:production, cluster_name:${var.eks_cluster_name}}"
  }
}

# datadog_monitors.tf
resource "datadog_monitor" "eks_node_cpu_utilization" {
  name               = "[EKS] High Node CPU Utilization on {{host.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.eks_cluster_name}} by {host} > 80"
  message            = <<EOF
EKS Node CPU usage is high on {{host.name}} ({{value}}%).
@pagerduty-${var.pagerduty_service_name}
EOF
  tags               = ["env:production", "kubernetes", "eks", "cpu", "alert"]
  restricted_roles   = []
  escalation_message = "CPU still high after 15 minutes, please investigate immediately."

  monitor_thresholds {
    critical = 80
    warning  = 70
  }

  renotify_interval = 60
  no_data_timeframe = 10
}

resource "datadog_monitor" "eks_node_memory_utilization" {
  name               = "[EKS] High Node Memory Utilization on {{host.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:kubernetes.memory.usage.total{cluster_name:${var.eks_cluster_name}} by {host} > 80"
  message            = <<EOF
EKS Node Memory usage is high on {{host.name}} ({{value}}%).
@pagerduty-${var.pagerduty_service_name}
EOF
  tags               = ["env:production", "kubernetes", "eks", "memory", "alert"]
  restricted_roles   = []

  monitor_thresholds {
    critical = 80
    warning  = 70
  }

  renotify_interval = 60
  no_data_timeframe = 10
}

resource "datadog_monitor" "eks_node_not_ready" {
  name               = "[EKS] Node Not Ready in {{cluster_name}}"
  type               = "metric alert"
  query              = "min(last_5m):kubernetes.node.status.not_ready{cluster_name:${var.eks_cluster_name}} by {host} > 0"
  message            = <<EOF
EKS Node {{host.name}} is in a NotReady state. This indicates a serious issue.
@pagerduty-${var.pagerduty_service_name}
EOF
  tags               = ["env:production", "kubernetes", "eks", "node", "status", "alert"]
  restricted_roles   = []

  monitor_thresholds {
    critical = 0.5 # Greater than 0
  }

  no_data_timeframe = 10
}

# pagerduty_integration.tf (This is handled implicitly by datadog_monitor's message if using the email integration)
# For more advanced PagerDuty features, you'd typically define the service and then use
# an `datadog_monitor_pagerduty_integration` resource (or similar if available for direct API integration).
# For simplicity, we are using Datadog's built-in PagerDuty email integration via `@pagerduty-service_name` in the message.
# Ensure you have an Email Integration configured in PagerDuty for your service, and its email address is configured in Datadog.

# Example PagerDuty Service (if you want to manage it with Terraform)
# resource "pagerduty_service" "eks_alerts_service" {
#   name                       = "EKS Monitoring Alerts"
#   auto_resolve_timeout       = "14400" # 4 hours
#   acknowledgement_timeout    = "600"   # 10 minutes
#   escalation_policy          = pagerduty_escalation_policy.default.id # Assumes an existing or defined EP
# }

# data "pagerduty_escalation_policy" "default" {
#   name = "Default Escalation Policy"
# }

# variables.tf
variable "aws_region" {
  description = "AWS region for resource deployment."
  type        = string
  default     = "us-east-1"
}

variable "eks_cluster_name" {
  description = "Name of the existing EKS cluster."
  type        = string
}

variable "datadog_api_key" {
  description = "Datadog API Key."
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key."
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token."
  type        = string
  sensitive   = true
}

variable "pagerduty_service_name" {
  description = "The name of the PagerDuty service to integrate with Datadog. Used in monitor messages."
  type        = string
  default     = "eks-monitoring-service" # Ensure this matches a service name in your Datadog PagerDuty integration.
}
    

To apply this configuration:

Save the above code into .tf files (e.g., main.tf, variables.tf, etc.).
Run terraform init to initialize the providers.
Set your sensitive variables either via environment variables (e.g., TF_VAR_datadog_api_key="your_key") or a terraform.tfvars file (ensure it's not committed to VCS).
Execute terraform plan to review the changes.
Finally, run terraform apply to deploy the Datadog Agent, AWS integration, and monitors.

4. Define Datadog Monitors for EKS

In the example code, we defined three essential monitors:

High Node CPU Utilization: Alerts if any EKS node's CPU usage exceeds 80%.
High Node Memory Utilization: Alerts if any EKS node's memory usage exceeds 80%.
Node Not Ready: Critical alert if an EKS node transitions to a NotReady state.

The message field in each monitor includes @pagerduty-${var.pagerduty_service_name}. This is Datadog's standard way to integrate with PagerDuty via an email integration. Ensure you have a PagerDuty service configured in Datadog with the matching name. Datadog will send an email to PagerDuty, which will then trigger an incident.

5. Set up PagerDuty Service & Integration

While we've used Datadog's built-in PagerDuty email integration for simplicity in the monitors, for a more robust integration, you would typically:

Create a PagerDuty service (e.g., "EKS Monitoring Alerts") with a dedicated escalation policy.
Add a "Datadog" integration to this PagerDuty service to get an Integration Key.
In Datadog, configure a PagerDuty integration using this key. This allows Datadog to directly trigger PagerDuty incidents via API calls, offering more control (e.g., sending custom details, acknowledging incidents).
Then, in your datadog_monitor resources, instead of @pagerduty-service_name, you would use a dedicated notification channel for PagerDuty configured within Datadog that points to your API-based integration.

Best Practices and Advanced Considerations

Granular Access Control (IAM)

For production environments, follow the principle of least privilege. The IAM role created for Datadog's AWS integration should only have read-only access to necessary services (CloudWatch, EC2, EKS, etc.). Avoid granting overly broad permissions.

Cost Optimization

Datadog pricing is based on hosts, custom metrics, logs, and traces. Monitor your Datadog usage closely. Configure the Datadog Agent to only collect what's truly essential. Filter unnecessary logs and metrics at the source.

Scaling and Multi-Cluster Deployments

Terraform's modularity shines here. Encapsulate your Datadog Agent and monitor definitions into reusable modules. This allows you to easily deploy the same monitoring stack across multiple EKS clusters, ensuring consistent observability standards.

Advanced Alerting and Dashboards

Beyond basic resource utilization, consider creating Datadog monitors for:

Application-specific metrics: Custom metrics from your applications.
Log-based alerts: For specific error patterns or security events in your application logs.
APM metrics: Latency, error rates, and throughput for critical services.
Kubernetes event alerts: Pod failures, OOMKilled events, `CrashLoopBackOff`.

Create comprehensive Datadog dashboards to visualize your EKS cluster's health, performance, and application behavior, providing invaluable insights during incidents and for proactive monitoring.

Conclusion

Automating your AWS EKS monitoring and alerting with Terraform, Datadog, and PagerDuty empowers your DevOps teams with unparalleled visibility and swift incident response capabilities. By codifying your observability stack, you achieve consistency, reduce manual errors, and accelerate your path to a truly resilient cloud-native infrastructure. Start with the foundational monitors, then iteratively expand your coverage to include application-specific metrics and advanced alerting scenarios, ensuring your EKS clusters are always performing optimally and critical issues are addressed before they impact your users.

CloudStack YAML