Terraform for AWS EKS Datadog Monitoring and PagerDuty Alerting

Architecture Pro-Tip: Embrace Infrastructure as Code (IaC) for your observability stack. By defining Datadog monitors, dashboards, and agent deployments via Terraform, you ensure consistency, version control, and auditability. This "Monitoring as Code" approach is crucial for reliable, scalable, and reproducible monitoring across multiple AWS EKS environments, enabling true GitOps for your entire platform.

Terraform for AWS EKS Datadog Monitoring and PagerDuty Alerting: A Comprehensive Guide

In modern cloud-native environments, robust monitoring and effective alerting are non-negotiable for maintaining the health and performance of your applications. AWS EKS (Elastic Kubernetes Service) provides a powerful container orchestration platform, but without proper observability, managing its complexity can be challenging. This guide will walk you through setting up comprehensive monitoring for your AWS EKS clusters using Datadog, integrating with PagerDuty for critical incident management, all automated and managed efficiently with Terraform.

By the end of this guide, you will have a clear understanding of how to declare your entire EKS monitoring and alerting infrastructure as code, ensuring consistency, repeatability, and a streamlined DevOps workflow.

Why Terraform, Datadog, and PagerDuty for EKS?

Terraform: Automates the provisioning and management of your infrastructure, including the deployment of monitoring agents, creation of monitors, and configuration of dashboards across AWS and Datadog. This eliminates manual errors and enables version control for your observability stack.
Datadog: A leading monitoring and analytics platform that provides deep visibility into your EKS clusters, applications, and underlying AWS infrastructure. It aggregates metrics, logs, and traces, offering rich dashboards, anomaly detection, and powerful alerting capabilities.
PagerDuty: An incident management platform that integrates seamlessly with Datadog to ensure that critical alerts reach the right people, at the right time, through various communication channels, facilitating rapid response and resolution.

Prerequisites

Before you begin, ensure you have the following:

An active AWS Account with permissions to manage EKS clusters, IAM roles, and EC2 instances.
An active Datadog Account with API and Application Keys.
An active PagerDuty Account with a configured service to integrate with Datadog.
Terraform CLI installed (v1.0.0 or higher).
AWS CLI configured with appropriate credentials.
kubectl installed and configured to access your EKS cluster.
Helm CLI installed (v3.0.0 or higher).

Core Concepts and Integration Points

Terraform Provider Configuration

We'll use the AWS, Datadog, and Helm Terraform providers to manage resources. Your Datadog API and Application keys should be kept secure, ideally sourced from environment variables or a secrets manager like AWS Secrets Manager.

Datadog Agent Deployment on EKS

The Datadog Agent is deployed as a DaemonSet in your EKS cluster, ensuring an agent runs on every node. It collects metrics, logs, and traces, and sends them to the Datadog platform. The recommended way to deploy the Agent on Kubernetes is via its Helm chart.

Datadog Monitors for EKS Health

Datadog monitors continuously evaluate metrics and logs against defined thresholds. For EKS, critical monitors include CPU/memory utilization, pod restarts, node readiness, API server latency, and deployment failures.

Datadog Dashboards for EKS Visualization

Dashboards provide a consolidated view of your EKS cluster's health and performance. Terraform can define entire dashboards, ensuring consistent visualization across environments.

PagerDuty Integration

Datadog integrates directly with PagerDuty. When a Datadog monitor triggers an alert, it can send an incident to a specified PagerDuty service, initiating your incident response workflow. This is typically configured within the Datadog monitor definition itself by mentioning the PagerDuty service name in the notification message.

Step-by-Step Implementation with Terraform

1. Configure Terraform Providers

Start by defining the required providers and their configurations. Ensure your AWS EKS cluster is already provisioned and your `kubeconfig` is set up correctly for Terraform to interact with it.

resource "aws_eks_cluster" "example" {
  # ... your EKS cluster definition ...
}

# Example: Data source to get current Kubernetes context
data "kubernetes_secret" "kubeconfig" {
  metadata {
    name      = "default-token-xxxxx" # Adjust to your cluster's actual secret name if needed
    namespace = "default"
  }
}

# You might typically use a kubeconfig file or environment variables
# For simplicity, assuming kubectl context is already set to your EKS cluster.

provider "aws" {
  region = "us-east-1" # Replace with your AWS region
}

provider "kubernetes" {
  host                   = aws_eks_cluster.example.endpoint
  cluster_ca_certificate = base64decode(aws_eks_cluster.example.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.example.token
}

data "aws_eks_cluster_auth" "example" {
  name = aws_eks_cluster.example.name
}

provider "helm" {
  kubernetes {
    host                   = aws_eks_cluster.example.endpoint
    cluster_ca_certificate = base64decode(aws_eks_cluster.example.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.example.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
  site    = var.datadog_site # e.g., "datadoghq.com"
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "datadog_site" {
  description = "Datadog site (e.g., datadoghq.com, eu.datadoghq.com)"
  type        = string
  default     = "datadoghq.com"
}
    

2. Deploy Datadog Agent to EKS using Helm

The `helm_release` resource allows you to deploy the Datadog Agent Helm chart. You'll pass your Datadog API key and other configurations via `set` blocks or a `values` file.

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog" # Recommended namespace
  create_namespace = true
  version    = "3.1.20" # Specify a stable version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "datadog.site"
    value = var.datadog_site
  }

  set {
    name  = "targetSystem"
    value = "linux"
  }

  # Enable APM for trace collection
  set {
    name  = "apm.enabled"
    value = "true"
  }

  # Enable Live Process Monitoring
  set {
    name  = "processAgent.enabled"
    value = "true"
  }

  # Enable Kubernetes State Metrics Core collection
  set {
    name  = "kubeStateMetricsCore.enabled"
    value = "true"
  }

  # Enable EKS specific features (e.g., metadata collection)
  set {
    name  = "datadog.kubernetes.clusterName"
    value = aws_eks_cluster.example.name
  }

  # RBAC settings for the agent
  set {
    name  = "rbac.create"
    value = "true"
  }

  # If you want to enable Admission Controller for injecting APM libs
  set {
    name  = "admissionController.enabled"
    value = "true"
  }

  # Integrate with AWS services for enhanced metrics
  set {
    name = "aws.hostTags"
    value = "true"
  }

  # Ensure the agent has the correct service account for EKS
  # This typically assumes an IAM role for service accounts (IRSA) is configured on your EKS cluster
  set {
    name  = "clusterAgent.rbac.create"
    value = "true"
  }

  # Set an appropriate resource limit for the Datadog Agent
  set {
    name  = "agents.container.resources.limits.cpu"
    value = "500m"
  }
  set {
    name  = "agents.container.resources.limits.memory"
    value = "512Mi"
  }
  set {
    name  = "agents.container.resources.requests.cpu"
    value = "100m"
  }
  set {
    name  = "agents.container.resources.requests.memory"
    value = "256Mi"
  }

  # ... other Datadog Agent configurations ...
}
    

3. Configure Datadog Monitors with Terraform for PagerDuty Alerting

Now, let's define some essential EKS monitors. The `message` field is crucial for integrating with PagerDuty. You'll reference your PagerDuty service name (which you set up in Datadog's PagerDuty integration) using the `@pagerduty-` syntax.

# Datadog PagerDuty Integration Setup (Pre-requisite, usually manual or through a different Terraform resource if available)
# In Datadog, go to Integrations -> PagerDuty, and set up your service integration.
# Let's assume you've named your PagerDuty integration 'my-eks-oncall-team'.

# Monitor: EKS Node Not Ready
resource "datadog_monitor" "eks_node_not_ready" {
  name                = "EKS Cluster: Node {{kubernetes.node}} is Not Ready - {{kubernetes.cluster_name}}"
  type                = "metric alert"
  query               = "avg(last_5m):kubernetes.node.ready.count{kubernetes.cluster_name:${aws_eks_cluster.example.name}} by {kubernetes.node} < 1"
  message             = "EKS Node {{kubernetes.node}} in cluster {{kubernetes.cluster_name}} is not ready. Investigate node health. @pagerduty-my-eks-oncall-team"
  threshold_warnings  = 1
  threshold_critical  = 1
  include_tags        = true
  no_data_timeframe   = 20
  renotify_interval   = 60 # Renotify every 60 minutes if alert persists

  tags = [
    "environment:production",
    "service:eks",
    "severity:critical",
    "component:cluster-health",
  ]
}

# Monitor: EKS High Pod Restart Rate
resource "datadog_monitor" "eks_high_pod_restart_rate" {
  name                = "EKS Cluster: High Pod Restart Rate for {{kube_container_name}} in {{kube_namespace}} - {{kubernetes.cluster_name}}"
  type                = "metric alert"
  query               = "sum(last_5m):kubernetes.pod.restarts_total{kubernetes.cluster_name:${aws_eks_cluster.example.name}} by {kube_container_name,kube_namespace} > 5"
  message             = "Pod {{kube_container_name}} in namespace {{kube_namespace}} has restarted >5 times in the last 5 minutes in cluster {{kubernetes.cluster_name}}. @pagerduty-my-eks-oncall-team"
  threshold_warnings  = 3
  threshold_critical  = 5
  include_tags        = true
  notify_no_data      = false
  renotify_interval   = 30

  tags = [
    "environment:production",
    "service:eks",
    "severity:high",
    "component:application-health",
  ]
}

# Monitor: EKS Cluster CPU Utilization
resource "datadog_monitor" "eks_high_cluster_cpu" {
  name               = "EKS Cluster CPU Utilization High - {{kubernetes.cluster_name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:kubernetes.cpu.usage.total{kubernetes.cluster_name:${aws_eks_cluster.example.name}} by {kubernetes.cluster_name} > 75"
  message            = "EKS Cluster {{kubernetes.cluster_name}} CPU utilization is at {{value}}%. Consider scaling or investigating workload. @pagerduty-my-eks-oncall-team"
  threshold_warnings = 70
  threshold_critical = 80
  include_tags       = true
  notify_no_data     = true

  tags = [
    "environment:production",
    "service:eks",
    "severity:warning",
    "component:resource-utilization",
  ]
}
    

4. Configure Datadog Dashboards with Terraform

Dashboards provide a visual overview of your EKS cluster's health. While complex dashboards can be extensive, here's a simple example to get you started. You can use the Datadog UI to design a dashboard and then export its JSON, which can be easily converted to HCL for Terraform.

resource "datadog_dashboard" "eks_overview_dashboard" {
  title        = "EKS Cluster Overview - ${aws_eks_cluster.example.name}"
  description  = "High-level overview of EKS cluster health and performance."
  layout_type  = "ordered"
  is_read_only = false

  widget {
    # Cluster CPU Usage
    timeseries_definition {
      title = "Cluster CPU Usage (Avg)"
      request {
        q     = "avg:kubernetes.cpu.usage.total{kubernetes.cluster_name:${aws_eks_cluster.example.name}}"
        display_type = "line"
      }
      right_yaxis {
        min    = "0"
        max    = "100"
        scale  = "linear"
        include_zero = true
      }
    }
  }

  widget {
    # Cluster Memory Usage
    timeseries_definition {
      title = "Cluster Memory Usage (Avg)"
      request {
        q     = "avg:kubernetes.memory.usage.total{kubernetes.cluster_name:${aws_eks_cluster.example.name}}"
        display_type = "line"
      }
      right_yaxis {
        min    = "0"
        max    = "100"
        scale  = "linear"
        include_zero = true
      }
    }
  }

  widget {
    # Pod Count
    timeseries_definition {
      title = "Total Pods Running"
      request {
        q     = "sum:kubernetes.pods.running{kubernetes.cluster_name:${aws_eks_cluster.example.name}}"
        display_type = "bars"
      }
    }
  }

  widget {
    # Node Count
    timeseries_definition {
      title = "Total Ready Nodes"
      request {
        q     = "sum:kubernetes.node.ready.count{kubernetes.cluster_name:${aws_eks_cluster.example.name}}"
        display_type = "bars"
      }
    }
  }

  # Add more widgets as needed for detailed insights
  # e.g., Network I/O, Disk usage, specific service metrics, etc.

  tags = [
    "environment:production",
    "service:eks",
    "cluster:${aws_eks_cluster.example.name}",
  ]
}
    

Applying Your Terraform Configuration

Once your `main.tf` file (or modularized configuration) is ready, follow these steps:

Initialize Terraform: Navigate to your project directory in the terminal and run `terraform init`. This downloads the necessary providers.
Plan Changes: Run `terraform plan -var="datadog_api_key=" -var="datadog_app_key="`. Review the proposed changes to ensure they match your expectations. (Alternatively, set these as environment variables: `DD_API_KEY`, `DD_APP_KEY`).
Apply Changes: Execute `terraform apply -var="datadog_api_key=" -var="datadog_app_key="`. Type `yes` when prompted to confirm the deployment.

Terraform will now provision the Datadog Agent, monitors, and dashboards as defined. You should see the Datadog Agent pods running in your EKS cluster and new monitors/dashboards appearing in your Datadog account.

Best Practices

Secrets Management: Never hardcode API keys or sensitive information. Use environment variables, AWS Secrets Manager, or HashiCorp Vault.
Modularize Terraform: For larger setups, break down your Terraform configuration into reusable modules (e.g., `eks-datadog-agent`, `eks-datadog-monitors`).
Version Control: Store all your Terraform code in Git for versioning, collaboration, and auditing.
Granular Permissions: Ensure the IAM role associated with your EKS nodes and the Datadog Agent service account has only the necessary permissions.
Test Monitors: After deployment, test your monitors by intentionally creating a condition that should trigger an alert to verify PagerDuty integration.

Troubleshooting and FAQ

Q: Datadog Agent pods are not running or are in a CrashLoopBackOff state.

Check Pod Logs: Use `kubectl logs -f -n datadog`. Look for errors related to API keys, permissions, or configuration.
Verify API/App Keys: Ensure `datadog_api_key` and `datadog_app_key` are correct and have no leading/trailing spaces.
RBAC Permissions: Confirm the Datadog Agent's Service Account has the necessary Kubernetes RBAC permissions. The Helm chart usually sets these up, but verify if you have custom RBAC policies.
Resource Limits: If your nodes are resource-constrained, the agent might fail to start. Adjust resource requests/limits in the Helm chart values.

Q: My Datadog monitors are not triggering PagerDuty incidents.

PagerDuty Integration in Datadog: Double-check that the PagerDuty integration is properly configured in Datadog's UI (Integrations -> PagerDuty). Ensure the "service_name" you are using in the `@pagerduty-` tag in your monitor message matches exactly what you configured in Datadog.
Monitor Firing: Verify that the Datadog monitor itself is actually triggering (e.g., check its status in Datadog UI, or temporarily lower thresholds to force a trigger).
Permissions: Ensure the Datadog API key has sufficient permissions to create events or trigger integrations.

Q: Why are my EKS metrics not appearing in Datadog?

Agent Health: Check the Datadog Agent pods for healthy status.
Connectivity: Ensure your EKS nodes and pods can reach Datadog endpoints over the internet (or via a private link if configured).
Helm Chart Values: Review the `set` values in your `helm_release` resource. Ensure `datadog.site`, `datadog.apiKey`, `datadog.appKey`, and Kubernetes-specific settings like `datadog.kubernetes.clusterName` are correct.

Conclusion

By leveraging Terraform, Datadog, and PagerDuty, you can establish a robust, automated, and observable AWS EKS environment. This guide provided a foundational approach to deploying the Datadog Agent, configuring essential monitors, and building informative dashboards, all managed as code. This "Monitoring as Code" paradigm is essential for any modern DevOps team aiming for high availability, quick incident resolution, and efficient management of their cloud-native infrastructure.

Continuously refine your monitors, dashboards, and alerting thresholds as your EKS workloads evolve to maintain optimal observability and incident response capabilities.

CloudStack YAML