Terraform for AWS EKS: Integrated Datadog Monitoring and PagerDuty Alerting

In the dynamic landscape of cloud-native applications, managing and monitoring Kubernetes clusters effectively is paramount for maintaining high availability and performance. AWS Elastic Kubernetes Service (EKS) provides a robust platform for deploying containerized workloads, but its true power is unlocked when integrated with comprehensive observability and incident management tools. This guide delves into leveraging Terraform to provision AWS EKS clusters and seamlessly integrate Datadog for powerful monitoring and PagerDuty for reliable alerting, creating a resilient and automated operational framework.

Architecture Pro-Tip:

Always design your EKS monitoring strategy with a 'shift-left' approach. Integrate Datadog agents and relevant monitors during the initial Terraform provisioning phase, rather than as an afterthought. This ensures your clusters are observable from day one, minimizing blind spots and enabling proactive issue detection. Furthermore, standardize your Datadog monitor definitions across environments using Terraform to enforce consistency and reduce operational overhead.

Why Integrate Terraform, AWS EKS, Datadog, and PagerDuty?

The synergy between these tools offers an unparalleled operational advantage:

Terraform for Infrastructure as Code (IaC): Automates the provisioning and management of EKS clusters, Datadog resources, and PagerDuty services, ensuring consistency, repeatability, and version control.
AWS EKS for Scalable Kubernetes: A fully managed Kubernetes service that simplifies the deployment, management, and scaling of containerized applications on AWS.
Datadog for Comprehensive Observability: Offers deep insights into EKS performance, health, and security through metric, log, and trace collection, custom dashboards, and intelligent monitoring.
PagerDuty for Incident Management: Transforms Datadog alerts into actionable incidents, ensuring the right teams are notified immediately and follow predefined response workflows.

This integrated approach significantly enhances reliability, reduces mean time to resolution (MTTR), and empowers DevOps teams with better visibility and control over their Kubernetes environments.

Prerequisites

Before diving into the configuration, ensure you have the following:

An AWS Account with programmatic access.
Terraform (v1.0.0+) installed.
AWS CLI and kubectl configured.
A Datadog Account with an API key and Application key.
A PagerDuty Account with an API token.
Basic understanding of AWS, Kubernetes, Terraform, Datadog, and PagerDuty concepts.

Step-by-Step Implementation

Step 1: Set up AWS EKS with Terraform

We'll start by provisioning an EKS cluster using the popular terraform-aws-modules/eks/aws module. This module simplifies EKS cluster and node group creation.

First, define your AWS provider and a basic VPC structure (if not already existing).

Step 2: Integrate Datadog Monitoring

Integrating Datadog involves several key components: setting up the Datadog provider, deploying the Datadog Agent to your EKS cluster, and defining Datadog monitors.

Datadog Provider: Configure the Datadog Terraform provider with your API and Application keys.
Datadog Agent Deployment: The Datadog Agent is typically deployed as a DaemonSet within your Kubernetes cluster. You can achieve this using Kubernetes manifests applied via Terraform's kubernetes_manifest resource or by utilizing the Datadog Helm chart through a helm_release resource.
EKS-specific Integrations: Datadog offers native integrations for AWS services (e.g., CloudWatch, EC2, EBS) and EKS-specific metrics (kube-state-metrics). Ensure the IAM role associated with your EKS cluster or worker nodes has appropriate permissions to allow Datadog to collect these metrics.
Datadog Monitors: Define monitors for critical EKS metrics like node CPU/memory utilization, pod restarts, deployment availability, and network issues.

Step 3: Configure PagerDuty Alerting

To route Datadog alerts to PagerDuty, you'll need to:

PagerDuty Provider: Configure the PagerDuty Terraform provider with your API token.
PagerDuty Service: Create a PagerDuty service that represents your EKS cluster or a specific application running on it. This service will have an integration key (e.g., Datadog integration) and an associated escalation policy.
Link Datadog Monitors to PagerDuty: Within your Datadog monitor definitions, specify the PagerDuty service integration as a notification channel.

Consolidated Terraform Configuration Example

Below is a comprehensive Terraform configuration demonstrating how to set up the necessary providers, deploy the Datadog Agent to EKS, create a Datadog monitor, and integrate it with PagerDuty. This example assumes your EKS cluster and VPC are already provisioned, focusing purely on the monitoring and alerting stack.

`main.tf`

resource "kubernetes_namespace" "datadog_agent" {
  metadata {
    name = "datadog"
  }
}

resource "kubernetes_secret" "datadog_api_key" {
  metadata {
    name      = "datadog-secret"
    namespace = kubernetes_namespace.datadog_agent.metadata[0].name
  }

  data = {
    "api-key" = var.datadog_api_key
    "app-key" = var.datadog_app_key
  }
}

resource "kubernetes_manifest" "datadog_agent_daemonset" {
  manifest = {
    apiVersion = "apps/v1"
    kind       = "DaemonSet"
    metadata = {
      name      = "datadog-agent"
      namespace = kubernetes_namespace.datadog_agent.metadata[0].name
      labels = {
        app = "datadog-agent"
      }
    }
    spec = {
      selector = {
        matchLabels = {
          app = "datadog-agent"
        }
      }
      template = {
        metadata = {
          labels = {
            app = "datadog-agent"
          }
        }
        spec = {
          serviceAccountName = "datadog-agent" # Assumes a service account exists or is created elsewhere
          containers = [
            {
              name  = "agent"
              image = "gcr.io/datadog-prod/agent:latest"
              env = [
                {
                  name  = "DD_API_KEY"
                  valueFrom = {
                    secretKeyRef = {
                      name = kubernetes_secret.datadog_api_key.metadata[0].name
                      key  = "api-key"
                    }
                  }
                },
                {
                  name  = "DD_APP_KEY"
                  valueFrom = {
                    secretKeyRef = {
                      name = kubernetes_secret.datadog_api_key.metadata[0].name
                      key  = "app-key"
                    }
                  }
                },
                {
                  name  = "DD_KUBERNETES_KUBELET_HOST"
                  value = "$(HOST_IP)"
                },
                {
                  name  = "DD_KUBERNETES_KUBELET_PORT"
                  value = "10250"
                },
                {
                  name  = "DD_CLUSTER_NAME"
                  value = var.eks_cluster_name
                },
                {
                  name  = "DD_LOGS_ENABLED"
                  value = "true"
                },
                {
                  name  = "DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL"
                  value = "true"
                }
                # Add more Datadog agent configuration as needed
              ]
              resources = {
                requests = {
                  memory = "256Mi"
                  cpu    = "200m"
                }
                limits = {
                  memory = "512Mi"
                  cpu    = "500m"
                }
              }
              volumeMounts = [
                {
                  name      = "procfs"
                  mountPath = "/host/proc"
                  readOnly  = true
                },
                {
                  name      = "cgroups"
                  mountPath = "/host/sys/fs/cgroup"
                  readOnly  = true
                },
                {
                  name      = "docker-sock"
                  mountPath = "/var/run/docker.sock"
                  readOnly  = true
                }
              ]
            }
          ]
          volumes = [
            {
              name = "procfs"
              hostPath = {
                path = "/proc"
              }
            },
            {
              name = "cgroups"
              hostPath = {
                path = "/sys/fs/cgroup"
              }
            },
            {
              name = "docker-sock"
              hostPath = {
                path = "/var/run/docker.sock"
              }
            }
          ]
        }
      }
    }
  }
}

# PagerDuty Service for EKS alerts
resource "pagerduty_service" "eks_monitoring_service" {
  name        = "${var.eks_cluster_name}-monitoring"
  description = "PagerDuty service for critical AWS EKS cluster alerts."
  escalation_policy = var.pagerduty_escalation_policy_id # Replace with your Escalation Policy ID
  alert_creation = "create_alerts_and_incidents"
}

# PagerDuty Integration for Datadog
resource "pagerduty_extension" "datadog_integration" {
  name       = "${pagerduty_service.eks_monitoring_service.name} - Datadog"
  endpoint   = "https://api.pagerduty.com/integrations/v1/datadog"
  type       = "datadog"
  service_id = pagerduty_service.eks_monitoring_service.id
}

# Datadog Monitor for high EKS Node CPU usage
resource "datadog_monitor" "high_eks_node_cpu" {
  name               = "[EKS-${var.eks_cluster_name}] High Node CPU Usage"
  type               = "metric alert"
  query              = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.eks_cluster_name}} by {host} > ${var.cpu_alert_threshold_percent}"
  message            = "CPU usage for node {{host.name}} in EKS cluster ${var.eks_cluster_name} is above ${var.cpu_alert_threshold_percent}%! @pagerduty-${pagerduty_service.eks_monitoring_service.name} @webhook-${pagerduty_extension.datadog_integration.id}"
  escalation_message = "CPU usage is still high, escalating to on-call."
  tags               = ["environment:${var.environment}", "service:eks", "alert-type:cpu"]
  priority           = 2
  notify_no_data     = false
  renotify_interval  = 60 # minutes

  monitor_thresholds {
    critical = var.cpu_alert_threshold_percent
  }
}

# Datadog Monitor for EKS Pod Restarts
resource "datadog_monitor" "eks_pod_restarts" {
  name               = "[EKS-${var.eks_cluster_name}] Frequent Pod Restarts"
  type               = "log alert"
  query              = "logs(\"status:error kubernetes.pod.restarts:>0 cluster_name:${var.eks_cluster_name}\").index(\"main\").rollup(\"count\").last(\"5m\") > 5"
  message            = "Multiple pod restarts detected in EKS cluster ${var.eks_cluster_name}. @pagerduty-${pagerduty_service.eks_monitoring_service.name} @webhook-${pagerduty_extension.datadog_integration.id}"
  tags               = ["environment:${var.environment}", "service:eks", "alert-type:pod-restarts"]
  priority           = 3
  notify_no_data     = false
  renotify_interval  = 30

  monitor_thresholds {
    critical = 5
  }
}

`variables.tf`

variable "region" {
  description = "AWS region"
  type        = string
  default     = "us-east-1"
}

variable "eks_cluster_name" {
  description = "Name of the EKS cluster"
  type        = string
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

variable "pagerduty_escalation_policy_id" {
  description = "PagerDuty Escalation Policy ID for the EKS service"
  type        = string
}

variable "cpu_alert_threshold_percent" {
  description = "CPU usage threshold for critical alert in percent"
  type        = number
  default     = 85
}

variable "environment" {
  description = "Environment tag (e.g., dev, prod)"
  type        = string
  default     = "dev"
}

`providers.tf`

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.20"
    }
    datadog = {
      source  = "datadog/datadog"
      version = "~> 3.0"
    }
    pagerduty = {
      source  = "pagerduty/pagerduty"
      version = "~> 2.0"
    }
  }
}

provider "aws" {
  region = var.region
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

data "aws_eks_cluster" "cluster" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = var.eks_cluster_name
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_token
}

Explanation of the Configuration:

Kubernetes Provider: Dynamically configured using EKS cluster details fetched via AWS data sources, ensuring secure communication with your cluster.
Datadog Agent Deployment: The kubernetes_manifest resource creates a Kubernetes Namespace and Secret for Datadog API keys, then deploys the Datadog Agent DaemonSet. This ensures an agent runs on every worker node to collect metrics, logs, and traces. Remember to define a ServiceAccount with appropriate permissions (e.g., cluster-info, kube-state-metrics access) for the Datadog Agent.
PagerDuty Service and Integration: A pagerduty_service is created to represent your EKS monitoring context, linked to a predefined escalation policy. The pagerduty_extension resource then establishes a Datadog integration for this service, providing a unique webhook endpoint for Datadog alerts.
Datadog Monitors: Two example datadog_monitor resources are defined: one for high CPU usage on EKS nodes and another for frequent pod restarts. The message field is critical, including @pagerduty-${pagerduty_service.eks_monitoring_service.name} to notify the PagerDuty service and @webhook-${pagerduty_extension.datadog_integration.id} to ensure the alert is sent via the configured integration.

Deployment and Validation

Follow these steps to deploy and validate your setup:

Initialize Terraform: Run terraform init in your project directory.
Review Plan: Execute terraform plan -var="eks_cluster_name=your-eks-cluster-name" -var="pagerduty_escalation_policy_id=your-pagerduty-ep-id" -var="datadog_api_key=..." -var="datadog_app_key=..." -var="pagerduty_token=..." to review the resources Terraform will create.
Apply Configuration: Apply the changes with terraform apply. Confirm with yes.
Verify Datadog Agents: After application, run kubectl get pods -n datadog. You should see Datadog Agent pods running on each worker node.
Check Datadog UI: Log into your Datadog account. Verify that your EKS cluster is reporting metrics, logs, and traces. Check the "Monitors" section to see your newly created alerts.
Verify PagerDuty: Confirm the new service and integration appear in your PagerDuty console.
Trigger a Test Alert: To test the full pipeline, consider temporarily setting a very low CPU alert threshold for one node or intentionally causing pod restarts to observe the alert flow from Datadog to PagerDuty.

Advanced Considerations

Security Best Practices: Use IAM roles for service accounts (IRSA) for the Datadog Agent instead of direct API keys in secrets. Restrict permissions to the minimum necessary.
Log Management: Configure Datadog Agent for advanced log collection and processing, including custom parsing rules and log-based metrics.
Application Performance Monitoring (APM): Instrument your applications for distributed tracing with Datadog APM to get end-to-end visibility.
Cost Optimization: Monitor Datadog usage and optimize agent configurations to control costs. Review PagerDuty alert noise to prevent alert fatigue.
Custom Dashboards: Create comprehensive Datadog dashboards for your EKS clusters, combining metrics, logs, and traces into a unified view.
Health Checks & Synthetics: Extend monitoring with Datadog Synthetic Monitoring for external uptime checks and API endpoint validation.

Troubleshooting Common Issues

Datadog Agent Pods Not Running: Check kubectl describe pod <agent-pod-name> -n datadog for events, image pull errors, or resource constraints. Ensure the ServiceAccount has necessary permissions.
No Data in Datadog: Verify API and Application keys. Check agent logs (kubectl logs <agent-pod-name> -n datadog) for connection errors to Datadog endpoints or issues collecting metrics. Ensure security groups allow outbound traffic to Datadog.
PagerDuty Alerts Not Triggering: Double-check the Datadog monitor's message syntax for the PagerDuty integration (e.g., @pagerduty-servicename and @webhook-integrationid). Confirm the PagerDuty service and integration are correctly configured and enabled.
Terraform Kubernetes Provider Errors: Ensure your AWS CLI context is correctly set to the region where your EKS cluster resides, and you have valid credentials to fetch EKS cluster details.

Conclusion

By integrating Terraform, AWS EKS, Datadog, and PagerDuty, organizations can build a highly observable, resilient, and automated cloud-native infrastructure. This guide provides a robust framework to establish comprehensive monitoring and incident response for your EKS clusters, empowering your teams to focus on innovation rather than firefighting. Embrace Infrastructure as Code to drive consistency, reduce operational overhead, and ensure that your critical applications running on AWS EKS are always performant and reliable.

CloudStack YAML