Automating AWS EKS Observability with Terraform, Datadog, and PagerDuty

Architecture Pro-Tip

For robust EKS observability, always deploy the Datadog Agent as a DaemonSet across your Kubernetes nodes and the Cluster Agent as a Deployment. This ensures comprehensive metric collection from every node and centralizes cluster-level metadata. Leverage Terraform to manage these deployments and all related Datadog resources (monitors, dashboards) and PagerDuty services, promoting an immutable infrastructure approach and enabling GitOps for your observability stack.

Automating AWS EKS Observability with Terraform, Datadog, and PagerDuty

In the dynamic landscape of cloud-native applications, managing and monitoring Kubernetes clusters, especially those hosted on AWS EKS, presents significant challenges. Ensuring high availability, performance, and rapid incident response requires a robust observability strategy. This guide delves into automating AWS EKS observability using a powerful trifecta: Terraform for Infrastructure as Code (IaC), Datadog for comprehensive monitoring and analytics, and PagerDuty for streamlined incident management.

The Imperative for Automated EKS Observability

As EKS clusters scale, manual configuration of monitoring tools becomes unsustainable and error-prone. Automation is critical for:

Consistency and Reproducibility: Ensure every cluster adheres to the same observability standards.
Scalability: Easily extend monitoring and alerting to new services and clusters without manual overhead.
Reduced Mean Time To Resolution (MTTR): Proactive monitoring and automated incident routing accelerate problem identification and resolution.
Security and Compliance: Enforce monitoring best practices across your infrastructure programmatically.

Core Components of Our Solution

This guide leverages three industry-leading tools:

Terraform: An open-source IaC tool from HashiCorp, used to provision and manage infrastructure on AWS, deploy Datadog agents to EKS, configure Datadog monitors, and set up PagerDuty services and escalation policies.
Datadog: A comprehensive monitoring, security, and analytics platform for cloud-scale applications. It provides full-stack visibility into EKS clusters, including metrics, logs, traces, APM, and user experience monitoring.
PagerDuty: A leading incident management platform that provides on-call scheduling, automated alerting, and intelligent incident routing, ensuring critical issues are quickly escalated to the right teams.

Prerequisites

Before you begin, ensure you have the following:

An active AWS account with necessary permissions to create EKS clusters, IAM roles, and other resources.
Terraform CLI installed (v1.0.0+ recommended).
AWS CLI configured with appropriate credentials.
A Datadog account with API and Application keys.
A PagerDuty account with an API token.
kubectl CLI installed and configured.
Helm CLI installed (for Datadog agent deployment via Helm).

Step-by-Step Implementation Guide

1. Terraform Setup for AWS EKS Cluster

First, you'll need a Terraform configuration to provision your EKS cluster. While the full EKS setup is extensive, we'll assume an existing EKS cluster or a basic setup similar to:

        # provider.tf
        provider "aws" {
          region = "us-east-1"
        }

        # Assuming EKS is already defined or provisioned via a separate module
        data "aws_eks_cluster" "example" {
          name = "my-eks-cluster"
        }

        data "aws_eks_cluster_auth" "example" {
          name = "my-eks-cluster"
        }
    

Ensure your kubeconfig is updated to interact with this cluster.

2. Integrating Datadog with EKS via Terraform

The Datadog Agent is crucial for collecting metrics, logs, and traces from your Kubernetes environment. We'll deploy it using the Helm chart via Terraform.

        # providers.tf (add datadog and helm providers)
        provider "datadog" {
          api_key = var.datadog_api_key
          app_key = var.datadog_app_key
        }

        provider "helm" {
          kubernetes {
            host                   = data.aws_eks_cluster.example.endpoint
            cluster_ca_certificate = base64decode(data.aws_eks_cluster.example.certificate_authority[0].data)
            token                  = data.aws_eks_cluster_auth.example.token
          }
        }

        # main.tf (for Datadog Agent deployment)
        resource "helm_release" "datadog_agent" {
          name       = "datadog"
          repository = "https://helm.datadoghq.com"
          chart      = "datadog"
          namespace  = "datadog"
          create_namespace = true

          set {
            name  = "datadog.apiKey"
            value = var.datadog_api_key
            sensitive = true
          }

          set {
            name  = "datadog.appKey"
            value = var.datadog_app_key
            sensitive = true
          }

          set {
            name  = "clusterAgent.enabled"
            value = "true"
          }

          set {
            name  = "agents.enabled"
            value = "true"
          }

          set {
            name  = "kubeStateMetricsExternal.enabled"
            value = "true"
          }

          set {
            name  = "clusterAgent.metricsProvider.enabled"
            value = "true"
          }

          set {
            name  = "logs.enabled"
            value = "true"
          }

          set {
            name  = "logs.containerCollectAll"
            value = "true"
          }

          # ... additional configurations like APM, network monitoring, etc.
        }
    

This configuration deploys the Datadog Agent as a DaemonSet on your EKS nodes and the Cluster Agent as a Deployment, enabling comprehensive metric and log collection.

3. Configuring Datadog Monitors via Terraform

Now, let's define some essential Datadog monitors using Terraform to alert on critical EKS health metrics.

        resource "datadog_monitor" "high_cpu_usage" {
          name               = "[EKS] High CPU Usage on Node {{host.name}}"
          type               = "metric alert"
          query              = "avg(last_5m):avg:system.cpu.idle{*} by {host} < 10"
          message            = "CPU utilization is critically high on {{host.name}}. Investigate immediately. @slack-channel @pagerduty-service"
          tags               = ["environment:production", "service:kubernetes", "team:devops"]
          escalation_message = "Still high after 15 minutes! Please address."
          notify_no_data     = false
          renotify_interval  = 60
          thresholds {
            warning          = 15
            critical         = 10
          }
        }

        resource "datadog_monitor" "pod_restart_rate" {
          name               = "[EKS] High Pod Restart Rate"
          type               = "metric alert"
          query              = "sum(last_5m):sum:kubernetes.pod.restarts_total{*} by {kube_namespace,kube_deployment} > 5"
          message            = "Deployment {{kube_deployment.name}} in {{kube_namespace.name}} is experiencing a high restart rate. @slack-channel @pagerduty-service"
          tags               = ["environment:production", "service:kubernetes", "team:devops"]
          escalation_message = "Pod restarts continue. This is critical!"
          notify_no_data     = false
          renotify_interval  = 30
          thresholds {
            warning          = 3
            critical         = 5
          }
        }
    

4. Integrating PagerDuty for Incident Response

To integrate PagerDuty, we'll define a PagerDuty service and an escalation policy using its Terraform provider. This service will receive alerts from Datadog.

        # providers.tf (add pagerduty provider)
        provider "pagerduty" {
          token = var.pagerduty_token
        }

        # main.tf (for PagerDuty service and policy)
        resource "pagerduty_user" "devops_engineer_one" {
          name  = "DevOps Engineer One"
          email = "devops.one@example.com"
          role  = "user"
        }

        resource "pagerduty_user" "devops_engineer_two" {
          name  = "DevOps Engineer Two"
          email = "devops.two@example.com"
          role  = "user"
        }

        resource "pagerduty_escalation_policy" "devops_policy" {
          name      = "DevOps EKS Escalation Policy"
          num_loops = 2

          rule {
            delay    = 5
            target {
              type = "user"
              id   = pagerduty_user.devops_engineer_one.id
            }
          }

          rule {
            delay    = 10
            target {
              type = "user"
              id   = pagerduty_user.devops_engineer_two.id
            }
          }
        }

        resource "pagerduty_service" "eks_observability_service" {
          name                    = "EKS Observability Service"
          auto_resolve_timeout    = 3600
          acknowledgement_timeout = 600
          escalation_policy       = pagerduty_escalation_policy.devops_policy.id
        }

        resource "pagerduty_service_integration" "datadog_integration" {
          name      = "Datadog Integration"
          type      = "datadog_inbound_integration"
          service   = pagerduty_service.eks_observability_service.id
        }
    

5. Connecting Datadog Monitors to PagerDuty

Finally, update your Datadog monitors to send alerts to the newly created PagerDuty integration. This involves updating the message field of your datadog_monitor resources to include the PagerDuty service notification handle.

The integration automatically creates a service in Datadog. You reference it in the Datadog monitor's message like @pagerduty-EKS_Observability_Service (replace with the actual service name). If you prefer, you can use the integration key directly from the pagerduty_service_integration resource. The example monitors above already include @pagerduty-service, which you'd replace with your actual PagerDuty service name configured in Datadog.

Ready-to-Use Configuration Example (`main.tf`)

This consolidated example illustrates how you might structure your main.tf for the core observability setup. Remember to replace placeholder values and extend with your specific cluster details and monitoring requirements.

        # main.tf

        # Variable Definitions (e.g., in variables.tf)
        variable "aws_region" {
          description = "AWS region"
          type        = string
          default     = "us-east-1"
        }

        variable "eks_cluster_name" {
          description = "Name of the EKS cluster"
          type        = string
          default     = "my-eks-cluster"
        }

        variable "datadog_api_key" {
          description = "Datadog API Key"
          type        = string
          sensitive   = true
        }

        variable "datadog_app_key" {
          description = "Datadog Application Key"
          type        = string
          sensitive   = true
        }

        variable "pagerduty_token" {
          description = "PagerDuty API Token"
          type        = string
          sensitive   = true
        }

        # Providers Configuration
        provider "aws" {
          region = var.aws_region
        }

        provider "datadog" {
          api_key = var.datadog_api_key
          app_key = var.datadog_app_key
        }

        provider "pagerduty" {
          token = var.pagerduty_token
        }

        provider "helm" {
          kubernetes {
            host                   = data.aws_eks_cluster.example.endpoint
            cluster_ca_certificate = base64decode(data.aws_eks_cluster.example.certificate_authority[0].data)
            token                  = data.aws_eks_cluster_auth.example.token
          }
        }

        # Data Sources for EKS Cluster Details
        data "aws_eks_cluster" "example" {
          name = var.eks_cluster_name
        }

        data "aws_eks_cluster_auth" "example" {
          name = var.eks_cluster_name
        }

        # 1. Datadog Agent Deployment via Helm
        resource "helm_release" "datadog_agent" {
          name       = "datadog"
          repository = "https://helm.datadoghq.com"
          chart      = "datadog"
          namespace  = "datadog"
          create_namespace = true

          set {
            name  = "datadog.apiKey"
            value = var.datadog_api_key
            sensitive = true
          }

          set {
            name  = "datadog.appKey"
            value = var.datadog_app_key
            sensitive = true
          }

          set {
            name  = "clusterAgent.enabled"
            value = "true"
          }

          set {
            name  = "agents.enabled"
            value = "true"
          }

          set {
            name  = "logs.enabled"
            value = "true"
          }

          set {
            name  = "logs.containerCollectAll"
            value = "true"
          }

          # Ensure you have the Datadog API & APP keys available as Terraform variables
          # via environment variables or a .tfvars file.
        }

        # 2. PagerDuty Service and Escalation Policy
        resource "pagerduty_user" "devops_oncall" {
          name  = "On-Call DevOps"
          email = "oncall@example.com" # Replace with a real email
          role  = "user"
        }

        resource "pagerduty_escalation_policy" "eks_observability_policy" {
          name      = "EKS Observability On-Call"
          num_loops = 1

          rule {
            delay    = 0
            target {
              type = "user"
              id   = pagerduty_user.devops_oncall.id
            }
          }
        }

        resource "pagerduty_service" "eks_critical_service" {
          name                    = "EKS Critical Service"
          auto_resolve_timeout    = 14400 # 4 hours
          acknowledgement_timeout = 600 # 10 minutes
          escalation_policy       = pagerduty_escalation_policy.eks_observability_policy.id
        }

        resource "pagerduty_service_integration" "datadog_integration" {
          name      = "Datadog Integration for EKS Alerts"
          type      = "datadog_inbound_integration"
          service   = pagerduty_service.eks_critical_service.id
        }

        # 3. Datadog Monitor for EKS Node CPU (alerting to PagerDuty)
        resource "datadog_monitor" "eks_node_high_cpu" {
          name               = "[EKS] Node High CPU Alert - {{host.name}}"
          type               = "metric alert"
          query              = "avg(last_5m):avg:system.cpu.idle{*} by {host} < 10"
          message            = "High CPU usage on EKS node {{host.name}}. Investigate! @pagerduty-${pagerduty_service.eks_critical_service.name}"
          tags               = ["environment:production", "service:eks-control-plane", "priority:high"]
          escalation_message = "CPU still high after 15 minutes. Escalating!"
          notify_no_data     = false
          renotify_interval  = 60
          thresholds {
            warning          = 15
            critical         = 10
          }
        }

        # To apply this:
        # terraform init
        # terraform apply
    

Troubleshooting and Best Practices

Common Issues

Permissions: Ensure the IAM role associated with your EKS worker nodes has the necessary permissions for Datadog to collect metrics (e.g., Kube-state-metrics access).
API/APP Keys: Verify your Datadog API and APP keys are correct and have the necessary permissions within Datadog. Similarly for PagerDuty token.
Helm Chart Values: Double-check the set values in your helm_release resource for the Datadog Agent. Incorrect values can lead to incomplete data collection.
Network Connectivity: Ensure your EKS cluster can reach Datadog's intake endpoints and PagerDuty's API.
PagerDuty Service Name in Datadog: When referencing PagerDuty in Datadog monitor messages, use the exact integration name as it appears in Datadog (e.g., @pagerduty-EKS_Critical_Service).

Best Practices

Modularize Terraform: Break down your Terraform configuration into logical modules (e.g., EKS cluster, Datadog setup, PagerDuty setup) for better organization and reusability.
Version Control: Store all your Terraform code in a Git repository to track changes, enable collaboration, and facilitate rollbacks.
Automated CI/CD: Integrate Terraform into your CI/CD pipeline to automatically deploy and manage your observability stack.
Granular Monitoring: Don't just monitor nodes; include critical pod metrics, deployment health, and application-specific custom metrics.
Alert Fatigue: Carefully tune your Datadog monitors to avoid alert fatigue. Use thresholds, anomaly detection, and composite monitors effectively.
Documentation: Maintain clear documentation for your observability setup, including alert thresholds, escalation policies, and runbooks.
Security: Store sensitive credentials (API keys, tokens) securely using services like AWS Secrets Manager or HashiCorp Vault, and integrate them with Terraform.

Conclusion

Automating AWS EKS observability with Terraform, Datadog, and PagerDuty empowers teams to achieve unparalleled visibility and incident response capabilities. By codifying your monitoring and alerting infrastructure, you ensure consistency, scalability, and resilience across your cloud-native deployments. This integrated approach not only reduces operational overhead but also significantly improves your team's ability to maintain healthy, high-performing EKS environments, ultimately driving business continuity and innovation.

CloudStack YAML