Terraform for AWS EKS: Datadog Monitoring and PagerDuty Incident Automation

Terraform for AWS EKS: Seamless Datadog Monitoring and PagerDuty Incident Automation

In the dynamic landscape of cloud-native applications, maintaining high availability and performance for Kubernetes clusters is paramount. AWS Elastic Kubernetes Service (EKS) provides a robust platform for deploying scalable applications, but effective monitoring and incident response are crucial for operational excellence. This comprehensive guide will walk you through leveraging Terraform to provision and manage Datadog monitoring for your AWS EKS clusters, coupled with PagerDuty incident automation, ensuring a proactive and efficient DevOps workflow.

By adopting an Infrastructure as Code (IaC) approach, you can codify your observability and incident management configurations, achieving consistency, version control, and automation across your environments.

Architecture Pro-Tip: For critical production EKS environments, always separate your core EKS cluster definition from your observability and incident response configurations into distinct Terraform modules. This enhances modularity, allows independent updates, and improves security by adhering to the principle of least privilege for different CI/CD pipelines or teams.

Prerequisites

  • An active AWS Account with necessary permissions.
  • Terraform CLI installed (v1.0+ recommended).
  • AWS CLI configured with credentials.
  • An existing AWS EKS cluster. This guide assumes your EKS cluster is already provisioned.
  • Datadog Account with API and Application Keys.
  • PagerDuty Account with API Token and Service IDs.
  • kubectl configured to interact with your EKS cluster.

Core Concepts: IaC, Observability, and Incident Management

Infrastructure as Code (IaC) with Terraform

Terraform enables you to define and provision infrastructure using a declarative configuration language. For EKS, this means not only managing the cluster itself but also extending to operational tools like Datadog and PagerDuty. Codifying these configurations ensures reproducibility, consistency, and traceability.

Datadog for EKS Observability

Datadog offers a unified platform for monitoring, logging, and tracing. For EKS, its agent collects metrics, logs, and events from Kubernetes nodes, pods, and applications. Critical metrics include CPU/memory utilization, network I/O, pod health, deployment statuses, and container logs.

PagerDuty for Incident Automation

PagerDuty provides a robust incident response platform. By integrating with Datadog, alerts can be automatically escalated to the right teams, reducing MTTR (Mean Time To Resolution). Terraform can manage PagerDuty services, escalation policies, and users, streamlining on-call rotations and notification rules.

Terraform Setup for Datadog Monitoring on AWS EKS

Integrating Datadog with your EKS cluster via Terraform involves two main steps: deploying the Datadog Agent to your cluster and then defining Datadog monitors using Terraform.

Step 1: Deploying the Datadog Agent

The Datadog Agent is typically deployed as a DaemonSet in Kubernetes, ensuring an agent runs on every node to collect host-level metrics and system-level events. While you can use helm_release within Terraform to deploy the Datadog Helm chart, we'll demonstrate a direct Terraform approach for clarity, assuming necessary Kubernetes provider configuration.

  • Kubernetes Provider: Ensure your Terraform is configured to interact with your EKS cluster.
  • Datadog API Keys: Securely pass your Datadog API and Application keys.

Step 2: Configuring Datadog Monitors with Terraform

Once the agent is collecting data, you can define specific monitors to alert on critical conditions. Terraform’s Datadog provider allows you to codify these alerts.

Key EKS Metrics to Monitor:

  • Node Health: CPU/Memory utilization (kubernetes.node.cpu.usage, kubernetes.node.memory.usage), node readiness.
  • Pod Health: Pod restarts (kubernetes.pod.restarts), pod CPU/memory requests/limits.
  • Deployment/DaemonSet Status: Unavailable pods (kubernetes.deployment.unavailable_replicas).
  • Network Metrics: Latency, throughput.
  • EKS Control Plane: API server availability, controller manager health (often monitored via AWS CloudWatch integration).

PagerDuty Incident Automation with Terraform

Connecting Datadog alerts to PagerDuty automates the incident response workflow. When a Datadog monitor triggers, PagerDuty creates an incident, notifies the appropriate on-call team, and manages escalation.

Step 1: Setting up PagerDuty Services and Escalation Policies

Using the Terraform PagerDuty provider, you can define your services, escalation policies, and even schedules. A "service" in PagerDuty typically represents a component or application that needs to be monitored and has a dedicated on-call team.

Step 2: Connecting Datadog Alerts to PagerDuty

This connection is made within the Datadog monitor definition. You specify the PagerDuty integration as a notification channel. When the monitor's conditions are met, Datadog sends an event to PagerDuty's API, triggering an incident.

Ready-to-Use Terraform Configuration

Below is a simplified, illustrative Terraform configuration. Remember to replace placeholder values and adapt it to your specific EKS setup and security best practices (e.g., using AWS Secrets Manager for API keys).

main.tf (Terraform Providers & Variables)

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
    pagerduty = {
      source  = "PagerDuty/pagerduty"
      version = "~> 1.15"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

data "aws_eks_cluster" "main" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "main" {
  name = var.eks_cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.main.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.main.token
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

resource "kubernetes_namespace" "datadog" {
  metadata {
    name = "datadog"
  }
}

resource "kubernetes_secret" "datadog_api_key" {
  metadata {
    name      = "datadog-secret"
    namespace = kubernetes_namespace.datadog.metadata[0].name
  }
  data = {
    "api-key" = base64encode(var.datadog_api_key)
    "app-key" = base64encode(var.datadog_app_key)
  }
}

resource "kubernetes_manifest" "datadog_agent_daemonset" {
  manifest = {
    "apiVersion" = "apps/v1"
    "kind"       = "DaemonSet"
    "metadata"   = {
      "name"      = "datadog-agent"
      "namespace" = kubernetes_namespace.datadog.metadata[0].name
      "labels"    = {
        "app" = "datadog-agent"
      }
    }
    "spec" = {
      "selector" = {
        "matchLabels" = {
          "app" = "datadog-agent"
        }
      }
      "template" = {
        "metadata" = {
          "labels" = {
            "app" = "datadog-agent"
          }
        }
        "spec" = {
          "serviceAccountName" = "datadog-agent" # Assumes you've created this SA with necessary permissions
          "containers" = [
            {
              "name"  = "agent"
              "image" = "gcr.io/datadog/agent:latest"
              "env"   = [
                {
                  "name"  = "DD_KUBERNETES_KUBELET_HOST"
                  "value" = "$(HOST_IP)"
                },
                {
                  "name"  = "DD_API_KEY"
                  "valueFrom" = {
                    "secretKeyRef" = {
                      "name" = kubernetes_secret.datadog_api_key.metadata[0].name
                      "key"  = "api-key"
                    }
                  }
                },
                {
                  "name"  = "DD_APP_KEY"
                  "valueFrom" = {
                    "secretKeyRef" = {
                      "name" = kubernetes_secret.datadog_api_key.metadata[0].name
                      "key"  = "app-key"
                    }
                  }
                },
                {
                  "name"  = "DD_KUBERNETES_COLLECT_CONTAINER_LABELS"
                  "value" = "true"
                },
                {
                  "name"  = "DD_LOGS_ENABLED"
                  "value" = "true"
                },
                {
                  "name"  = "DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL"
                  "value" = "true"
                },
                {
                  "name"  = "DD_PROCESS_AGENT_ENABLED"
                  "value" = "true"
                },
                {
                  "name"  = "DD_TAGS"
                  "value" = "environment:prod,cluster_name:${var.eks_cluster_name}"
                },
              ]
              "resources" = {
                "requests" = {
                  "memory" = "256Mi"
                  "cpu"    = "200m"
                }
              }
              "volumeMounts" = [
                {
                  "name"      = "proc"
                  "mountPath" = "/host/proc"
                  "readOnly"  = true
                },
                {
                  "name"      = "cgroup"
                  "mountPath" = "/host/sys/fs/cgroup"
                  "readOnly"  = true
                },
                {
                  "name"      = "root"
                  "mountPath" = "/host/root"
                  "readOnly"  = true
                },
                {
                  "name"      = "var-run-docker-sock"
                  "mountPath" = "/var/run/docker.sock"
                  "readOnly"  = true
                },
              ]
            }
          ]
          "volumes" = [
            {
              "name"     = "proc"
              "hostPath" = { "path" = "/proc" }
            },
            {
              "name"     = "cgroup"
              "hostPath" = { "path" = "/sys/fs/cgroup" }
            },
            {
              "name"     = "root"
              "hostPath" = { "path" = "/" }
            },
            {
              "name"     = "var-run-docker-sock"
              "hostPath" = { "path" = "/var/run/docker.sock" }
            },
          ]
        }
      }
    }
  }
}

# PagerDuty Service & Escalation Policy
resource "pagerduty_user" "ops_engineer" {
  name  = "Ops Engineer"
  email = "ops_engineer@example.com" # Replace with a real email
}

resource "pagerduty_escalation_policy" "devops_policy" {
  name      = "DevOps EKS Critical Policy"
  num_loops = 2
  rule {
    escalation_delay_in_minutes = 5
    target {
      type = "user_reference"
      id   = pagerduty_user.ops_engineer.id
    }
  }
}

resource "pagerduty_service" "eks_monitoring_service" {
  name                       = "EKS Cluster Monitoring"
  auto_resolve_timeout       = "14400" # 4 hours
  acknowledgement_timeout    = "600"   # 10 minutes
  escalation_policy          = pagerduty_escalation_policy.devops_policy.id
  alert_creation             = "create_alerts_and_incidents"
  alert_grouping             = "time"
  alert_grouping_timeout     = 5
  incident_urgency_rule {
    type = "constant"
    urgency = "high"
  }
}

# Datadog Monitor for EKS Node CPU Usage
resource "datadog_monitor" "eks_node_cpu_high" {
  name               = "[EKS] Node CPU Usage High - {{host.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:kubernetes.node.cpu.usage{cluster_name:${var.eks_cluster_name}} by {host} > 80"
  message            = "CPU usage on node {{host.name}} is above 80% for 5 minutes! @pagerduty-EKS-Cluster-Monitoring" # PagerDuty integration name must match
  tags               = ["environment:prod", "eks", "cpu"]
  new_group_delay    = 60
  new_host_delay     = 300
  notification_noreeporting = true
  renotify_interval  = 0
  no_data_timeframe  = 20
  include_tags       = true
  require_full_window = true
  monitor_thresholds {
    critical = 80
    warning  = 70
  }
  options {
    threshold_windows {
      recovery_window = "5m"
      trigger_window  = "5m"
    }
  }
}

# Datadog Monitor for EKS Pod Restarts
resource "datadog_monitor" "eks_pod_restarts" {
  name               = "[EKS] Pod Restart Count High - {{kube_container_name}}"
  type               = "metric alert"
  query              = "sum(last_15m):sum:kubernetes.pod.restarts{cluster_name:${var.eks_cluster_name}} by {kube_container_name} > 3"
  message            = "Container {{kube_container_name}} in pod {{kube_pod_name}} has restarted more than 3 times in 15 minutes! @pagerduty-EKS-Cluster-Monitoring"
  tags               = ["environment:prod", "eks", "pod_health"]
  new_group_delay    = 60
  new_host_delay     = 300
  notification_noreeporting = true
  renotify_interval  = 0
  no_data_timeframe  = 20
  include_tags       = true
  require_full_window = true
  monitor_thresholds {
    critical = 3
    warning  = 2
  }
}

# variables.tf
variable "aws_region" {
  description = "AWS region for the EKS cluster"
  type        = string
  default     = "us-east-1"
}

variable "eks_cluster_name" {
  description = "Name of the existing EKS cluster"
  type        = string
  # IMPORTANT: Replace with your actual EKS cluster name
  default     = "my-production-eks-cluster"
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

Testing and Validation

After applying your Terraform configuration, it's crucial to validate the setup:

  • Datadog Agent: Verify the Datadog Agent pods are running correctly in your EKS cluster: kubectl get pods -n datadog. Check Datadog's "Infrastructure" -> "Hosts" or "Containers" sections to see if data is flowing.
  • Datadog Monitors: Navigate to Datadog's "Monitors" -> "Manage Monitors" and confirm your new monitors are listed and in an "OK" state.
  • PagerDuty Integration: Ensure the Datadog integration is correctly configured in PagerDuty (Services -> Service Integrations). Trigger a test alert from Datadog or simulate a critical condition in your EKS cluster to verify an incident is created in PagerDuty.

Best Practices for Production Environments

  • Secret Management: Never hardcode API keys. Use AWS Secrets Manager, HashiCorp Vault, or environment variables in your CI/CD pipeline to inject sensitive data.
  • Modular Terraform: Break down your Terraform configuration into reusable modules (e.g., an eks-datadog-agent module, an datadog-monitors module).
  • Role-Based Access Control (RBAC): Grant the Datadog Agent only the necessary Kubernetes RBAC permissions.
  • Alert Fatigue: Fine-tune your Datadog monitor thresholds and PagerDuty escalation policies to minimize alert fatigue. Start with higher thresholds and gradually lower them as you understand your system's baseline.
  • Synthetic Monitoring: Complement your infrastructure monitoring with synthetic checks to actively test application endpoints from an end-user perspective.
  • Version Control: Store all Terraform configurations in a Git repository and use a CI/CD pipeline for automated deployments.

Troubleshooting and FAQ

Q: Datadog Agent pods are not running.

A: Check kubectl logs <datadog-agent-pod-name> -n datadog and kubectl describe pod <datadog-agent-pod-name> -n datadog. Common issues include incorrect API/APP keys, insufficient RBAC permissions for the agent's ServiceAccount, or resource constraints on the nodes.

Q: PagerDuty incidents are not triggering.

A: Verify the @pagerduty-SERVICE-NAME tag in your Datadog monitor message matches the PagerDuty integration name exactly. Also, ensure your PagerDuty service is enabled and its integration key is correctly configured in Datadog (Integrations > PagerDuty). Check Datadog's event stream for monitor triggers.

Q: How do I manage multiple environments (dev, staging, prod)?

A: Use Terraform workspaces or a directory-per-environment structure, combined with variable files (.tfvars), to manage different configurations and API keys for each environment.

Conclusion

By meticulously defining your AWS EKS monitoring and incident response workflows using Terraform, Datadog, and PagerDuty, you establish a robust, scalable, and automated observability framework. This IaC approach reduces manual overhead, minimizes human error, and ensures your critical EKS workloads are continuously monitored, with incidents escalated swiftly to the right personnel, ultimately leading to improved reliability and operational efficiency.

Comments

Post a Comment

Popular posts from this blog

Terraform Configuration for Datadog-PagerDuty Incident Management on AWS EKS

Terraform Configuration for Datadog-PagerDuty Incident Management on AWS EKS In the fast-paced world of cloud-native operations, ensuring high availability and rapid incident response for applications running on AWS EKS is paramount. This comprehensive guide details how to leverage Terraform to seamlessly integrate Datadog for robust monitoring and PagerDuty for efficient incident management, creating a resilient and automated incident response pipeline for your Kubernetes clusters. Architecture Pro-Tip: Modular Design for Scalability When building your incident management infrastructure with Terraform, consider a modular design. Separate your Datadog monitors, PagerDuty services, and AWS EKS resources into distinct Terraform modules. This approach enhances reusability, improves readability, and makes it significantly easier to manage configurations across multiple environments (development, staging, production) or different EKS clusters. Always priori...
Read more

Terraform-Managed AWS EKS Observability and Incident Response with Datadog and PagerDuty

Terraform-Managed AWS EKS Observability and Incident Response with Datadog and PagerDuty In the dynamic world of cloud-native applications, managing AWS Elastic Kubernetes Service (EKS) requires robust observability and a streamlined incident response strategy. This guide delves into integrating Datadog for comprehensive monitoring and PagerDuty for efficient incident management, all orchestrated declaratively using Terraform. We'll explore how to establish a resilient system that not only detects issues promptly but also ensures critical alerts reach the right teams for rapid resolution, maintaining high availability and operational excellence. Architecture Pro-Tip: Always centralize your observability configuration within your Infrastructure as Code (IaC) repository. Treat your Datadog monitors, dashboards, and PagerDuty services as code alongside your EKS clusters. This ensures version control, simplifies rollbacks, an...
Read more

Terraform for Production AWS EKS Observability with Datadog, Prometheus, and PagerDuty Integration

Terraform for Production AWS EKS Observability with Datadog, Prometheus, and PagerDuty Integration In the dynamic world of cloud-native applications, maintaining peak performance and rapid incident response for AWS Elastic Kubernetes Service (EKS) is paramount. Observability, often misunderstood as mere monitoring, extends to understanding the internal state of a system from its external outputs. This guide provides a comprehensive, technical walkthrough on leveraging Terraform to provision and manage a robust observability stack for your production EKS clusters, integrating the power of Datadog for holistic monitoring, Prometheus for deep Kubernetes-native metrics, and PagerDuty for automated incident management. Architecture Pro-Tip: For a truly resilient and scalable production observability strategy on AWS EKS, aim for a layered approach. Use Datadog as your primary pane of glass for aggregated metrics, logs, and traces, leveraging its native integrat...
Read more