Terraform for AWS EKS Production Readiness: Datadog Monitoring and PagerDuty Alerting

Achieving production readiness for AWS EKS clusters demands robust observability and incident management. This guide provides a comprehensive, hands-on approach to deploying and configuring Datadog for monitoring and PagerDuty for critical alerting, all managed through Terraform. By infrastructure-as-code (IaC), you'll ensure consistency, repeatability, and scalability for your Kubernetes environments.

Architecture Pro-Tip: Observability-First Design

When designing your EKS infrastructure, treat observability as a core component, not an afterthought. Integrate monitoring agents, log shippers, and trace collectors from day one. Define your SLOs/SLIs early and build alerts around them, ensuring that your monitoring solution is tailored to your business critical applications. This proactive approach drastically reduces MTTR (Mean Time To Recovery) and improves system reliability.

The Observability Imperative for EKS Production

In a dynamic containerized environment like AWS EKS, maintaining visibility into your applications and infrastructure is paramount. Production readiness isn't just about deploying code; it's about confidently operating and recovering it. Datadog offers a unified platform for metrics, logs, and traces across your EKS clusters, while PagerDuty ensures critical issues are never missed, escalating them to the right on-call personnel.

Why Datadog for EKS?

Unified Platform: Consolidates metrics, logs, and traces from Kubernetes, AWS, and your applications.
Rich Kubernetes Integration: Automatic discovery of pods, services, deployments, and nodes with deep-dive dashboards.
Advanced Alerting: Machine learning-driven alerts, anomaly detection, and granular notification options.
Customizable Dashboards: Visualize the health and performance of your entire EKS ecosystem.

Why PagerDuty for Alerting?

Reliable Incident Management: Guarantees delivery of critical alerts through multiple channels (SMS, phone call, email, push notifications).
On-Call Scheduling: Manages complex on-call rotations and escalation policies.
Seamless Integrations: Works out-of-the-box with Datadog and hundreds of other monitoring tools.
Incident Response Automation: Facilitates runbook execution and stakeholder communication.

Prerequisites and Setup

Before diving into Terraform, ensure you have the following:

AWS Account: With permissions to manage EKS, IAM, and other AWS resources.
Terraform CLI: Installed and configured for your AWS account.
Kubectl: Configured to interact with your EKS cluster.
Datadog Account: With an API Key and Application Key.
PagerDuty Account: With a Service and Integration Key (for Datadog integration).

For Datadog and PagerDuty, you'll need to retrieve API keys/tokens and integration keys, typically found in their respective settings/integrations sections.

Terraform for EKS Base Infrastructure

While this guide focuses on observability, it assumes you have an existing AWS EKS cluster. You can provision an EKS cluster using the terraform-aws-modules/eks/aws module or eksctl. Ensure your Terraform configuration outputs the EKS cluster name and OIDC provider URL, as these are useful for configuring Kubernetes resources and IAM roles for service accounts (IRSA).

Deploying Datadog Agent on EKS with Terraform

The Datadog Agent is deployed as a DaemonSet to collect metrics, logs, and traces from your Kubernetes nodes and pods. The recommended way to deploy it is via its Helm chart. We'll use Terraform's helm provider to manage this deployment.

Setting up the Kubernetes and Helm Providers

Your Terraform configuration needs to interact with your EKS cluster. Ensure your `kubernetes` and `helm` providers are configured to point to your EKS cluster. This typically involves using the output of your EKS module to get the cluster endpoint and certificate authority data.

Terraform Configuration for Datadog Agent

Below is a basic Terraform configuration snippet to deploy the Datadog Agent using the Helm provider. Replace placeholders with your actual values.

resource "kubernetes_secret" "datadog_api_key" {
  metadata {
    name      = "datadog-secret"
    namespace = "datadog" # Ensure this namespace exists or is created
  }

  data = {
    "api-key" = var.datadog_api_key
  }
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  version    = "2.35.0" # Use a specific, tested version

  depends_on = [kubernetes_secret.datadog_api_key]

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key # Or reference the secret directly
    type  = "string"
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    type  = "string"
  }

  values = [
    yamlencode({
      # General agent settings
      datadog = {
        site = var.datadog_site # E.g., "us5.datadoghq.com", "app.datadoghq.com"
        tags = [
          "env:${var.environment}",
          "cluster:${var.eks_cluster_name}"
        ]
        # Enable logs collection (recommended)
        logs = {
          enabled = true
          containerCollectAll = true
          autoMultiLineLogDetection = true
        }
        # Enable APM tracing
        apm = {
          enabled = true
        }
        # Enable process monitoring
        processAgent = {
          enabled = true
        }
        # Enable network performance monitoring
        networkMonitoring = {
          enabled = true
        }
        # Optionally enable security features
        securityAgent = {
          compliance = {
            enabled = true
          }
          runtime = {
            enabled = true
          }
        }
      }

      # Cluster Agent settings
      clusterAgent = {
        enabled = true
        metricsProvider = {
          enabled = true
          wpaController = true # Enable Horizontal Pod Autoscaling based on Datadog metrics
        }
      }

      # Node Agent settings
      agents = {
        kubeStateMetrics = {
          enabled = true # Collects metrics from kube-state-metrics
        }
      }

      # RBAC for Datadog Agent
      rbac = {
        create = true
      }
    })
  ]
}

Important considerations:

Namespaces: Ensure the `datadog` namespace exists or is created by your Terraform.
API/App Keys: Best practice is to manage these as Kubernetes secrets or AWS Secrets Manager and reference them securely. For simplicity, this example uses `var.datadog_api_key`.
`values` block: Use `yamlencode` for complex Helm `values` to keep your Terraform code clean and readable.
Version Pinning: Always pin the Helm chart version to prevent unexpected upgrades.
IRSA (IAM Roles for Service Accounts): For enhanced security, configure IRSA for the Datadog Agent. This example skips it for brevity but is highly recommended for production.

Terraform for Datadog Monitoring Configuration

With the Datadog Agent deployed, we can now use the datadog provider to define monitors and dashboards as code.

Configuring the Datadog Provider

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
  api_url = "https://${var.datadog_site}" # E.g., "https://api.datadoghq.com"
}

Creating Datadog Monitors with Terraform

Let's create a critical monitor for high EKS node CPU utilization. This will be integrated with PagerDuty later.

resource "datadog_monitor" "eks_node_high_cpu" {
  name               = "[EKS] High Node CPU Utilization on {{host.name}}"
  type               = "metric alert"
  message            = "CPU utilization on EKS node {{host.name}} is above {{threshold}}% for 5 minutes. @pagerduty-eks-oncall"
  query              = "avg(last_5m):avg:system.cpu.idle{cluster_name:${var.eks_cluster_name}} by {host} < 20" # Less than 20% idle = >80% utilized

  monitor_threshold_windows {
    recovery_window = "10m"
  }

  thresholds {
    critical = 80
    warning  = 70
  }

  include_tags       = true
  require_full_window = true
  renotify_interval  = 60 # Renotify every 60 minutes if issue persists

  # Notification options
  escalation_message = "EKS node CPU remains critical. Investigate immediately!"
  notify_audit       = false
  notify_no_data     = false
  no_data_timeframe  = 20 # After 20 minutes of no data, trigger alert
  timeout_h          = 0

  tags = [
    "env:${var.environment}",
    "service:eks",
    "severity:critical",
    "pagerduty:true",
    "monitor_owner:devops-team"
  ]
}

This monitor triggers if any EKS node in the specified cluster maintains over 80% CPU utilization for 5 minutes. The message includes an `@pagerduty-eks-oncall` handle, which we'll configure next.

Integrating Datadog with PagerDuty for Alerting

To route critical Datadog alerts to PagerDuty, you need to configure the integration within Datadog. This is also manageable via the Datadog Terraform provider.

Setting up PagerDuty Service and Integration Key

In PagerDuty:

Create a new service (e.g., "EKS Critical Alerts").
Add a new integration to this service, choosing "Datadog" as the integration type.
Copy the generated Integration Key. This is crucial for Terraform.

Terraform Configuration for Datadog-PagerDuty Integration

We'll use the `datadog_integration_pagerduty` resource to set up the connection and then define a notification channel.

resource "datadog_integration_pagerduty" "pagerduty_integration" {
  api_token = var.pagerduty_api_token # PagerDuty API token for Datadog integration, not the service integration key
}

resource "datadog_integration_pagerduty_service" "eks_oncall_service" {
  service_name = "EKS Critical Alerts" # Name of the service in PagerDuty
  service_key  = var.pagerduty_eks_integration_key
}

# This step is typically done manually in Datadog UI or implied by using the service_name in the monitor message.
# For PagerDuty notifications, you simply use the @pagerduty- handle in your monitor message.
# So if your service_name in Datadog PagerDuty integration is 'EKS On-Call', you'd use @pagerduty-EKS-On-Call

# For a more dynamic approach for handling service names in monitor message, you could pass it as a variable.
# Example: monitor_message = "CPU utilization on EKS node {{host.name}} is above {{threshold}}% for 5 minutes. @pagerduty-${datadog_integration_pagerduty_service.eks_oncall_service.service_name_in_datadog}"
# Note: Datadog automatically creates a 'notification handle' like @pagerduty-Service_Name when you add a PagerDuty integration service.
# The `service_name_in_datadog` property is often lowercased and hyphenated by Datadog.
# It's safest to verify the exact handle in your Datadog UI after integration setup.
# For our example, let's assume the handle is @pagerduty-eks-oncall based on a PagerDuty service named "EKS Oncall".

Note: The `api_token` for `datadog_integration_pagerduty` is a PagerDuty API Token (typically a read-only one for integrations, or a generic one). The `service_key` for `datadog_integration_pagerduty_service` is the integration key you obtained from the specific PagerDuty service. Ensure you use the correct key for each resource.

Putting It All Together: A Complete Terraform Example

Here's how a consolidated `main.tf` might look, along with `variables.tf` and `providers.tf` for a clearer picture.

`versions.tf` (Provider Configuration)

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

provider "kubernetes" {
  host                   = var.eks_cluster_endpoint
  cluster_ca_certificate = base64decode(var.eks_cluster_ca_certificate)
  token                  = data.aws_eks_cluster_auth.this.token
}

provider "helm" {
  kubernetes {
    host                   = var.eks_cluster_endpoint
    cluster_ca_certificate = base64decode(var.eks_cluster_ca_certificate)
    token                  = data.aws_eks_cluster_auth.this.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
  api_url = "https://${var.datadog_site}"
}

data "aws_eks_cluster_auth" "this" {
  name = var.eks_cluster_name
}

`variables.tf`

variable "aws_region" {
  description = "AWS region for EKS cluster"
  type        = string
  default     = "us-east-1"
}

variable "eks_cluster_name" {
  description = "Name of the EKS cluster"
  type        = string
}

variable "eks_cluster_endpoint" {
  description = "EKS cluster endpoint"
  type        = string
}

variable "eks_cluster_ca_certificate" {
  description = "EKS cluster CA certificate"
  type        = string
}

variable "environment" {
  description = "Deployment environment (e.g., dev, staging, prod)"
  type        = string
  default     = "production"
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "datadog_site" {
  description = "Datadog site URL (e.g., 'api.datadoghq.com', 'us5.datadoghq.com')"
  type        = string
  default     = "api.datadoghq.com"
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token for Datadog integration"
  type        = string
  sensitive   = true
}

variable "pagerduty_eks_integration_key" {
  description = "PagerDuty Integration Key for the EKS Critical Alerts service"
  type        = string
  sensitive   = true
}

`main.tf` (Consolidated)

# Create datadog namespace if it doesn't exist
resource "kubernetes_namespace" "datadog" {
  metadata {
    name = "datadog"
  }
}

# Kubernetes Secret for Datadog API Key
resource "kubernetes_secret" "datadog_api_key_secret" {
  metadata {
    name      = "datadog-secret"
    namespace = kubernetes_namespace.datadog.metadata[0].name
  }

  data = {
    "api-key" = var.datadog_api_key
  }
}

# Deploy Datadog Agent using Helm
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = kubernetes_namespace.datadog.metadata[0].name
  version    = "2.35.0" # Always pin chart versions

  depends_on = [kubernetes_secret.datadog_api_key_secret]

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    type  = "string"
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    type  = "string"
  }

  values = [
    yamlencode({
      datadog = {
        site = var.datadog_site
        tags = [
          "env:${var.environment}",
          "cluster:${var.eks_cluster_name}"
        ]
        logs = {
          enabled           = true
          containerCollectAll = true
        }
        apm = {
          enabled = true
        }
        processAgent = {
          enabled = true
        }
        networkMonitoring = {
          enabled = true
        }
      }
      clusterAgent = {
        enabled = true
        metricsProvider = {
          enabled = true
        }
      }
      agents = {
        kubeStateMetrics = {
          enabled = true
        }
      }
      rbac = {
        create = true
      }
    })
  ]
}

# Datadog PagerDuty Integration
resource "datadog_integration_pagerduty" "pagerduty_integration" {
  api_token = var.pagerduty_api_token
}

resource "datadog_integration_pagerduty_service" "eks_oncall_service" {
  service_name = "EKS Critical Alerts"
  service_key  = var.pagerduty_eks_integration_key
  # You can specify a custom key for this service as well, e.g., "eks-oncall"
  # This custom key would then be used in the monitor message: @pagerduty-eks-oncall
  # For simplicity, we assume the default name used by Datadog based on the service_name.
}


# Datadog Monitor for High EKS Node CPU Utilization
resource "datadog_monitor" "eks_node_high_cpu" {
  name               = "[EKS][${var.environment}] High Node CPU Utilization on {{host.name}}"
  type               = "metric alert"
  message            = "CPU utilization on EKS node {{host.name}} is above {{threshold}}% for 5 minutes. @pagerduty-eks-critical-alerts" # Use the handle created by Datadog
  query              = "avg(last_5m):avg:system.cpu.idle{cluster_name:${var.eks_cluster_name},env:${var.environment}} by {host} < 20"

  monitor_threshold_windows {
    recovery_window = "10m"
  }

  thresholds {
    critical = 80
    warning  = 70
  }

  include_tags       = true
  require_full_window = true
  renotify_interval  = 60

  escalation_message = "EKS node CPU remains critical. Investigate immediately!"
  notify_audit       = false
  notify_no_data     = false
  no_data_timeframe  = 20
  timeout_h          = 0

  tags = [
    "env:${var.environment}",
    "service:eks",
    "severity:critical",
    "pagerduty:true",
    "monitor_owner:devops-team"
  ]
}

# Example: Datadog Monitor for EKS Node Not Ready
resource "datadog_monitor" "eks_node_not_ready" {
  name               = "[EKS][${var.environment}] Node Not Ready Detected on {{host.name}}"
  type               = "metric alert"
  message            = "EKS node {{host.name}} is reporting NotReady status for over 5 minutes. @pagerduty-eks-critical-alerts"
  query              = "avg(last_5m):avg:kubernetes.node.ready{cluster_name:${var.eks_cluster_name},env:${var.environment}} by {host} == 0"

  monitor_threshold_windows {
    recovery_window = "5m"
  }

  thresholds {
    critical = 0.5 # A value less than 1 (meaning not ready)
  }

  include_tags       = true
  require_full_window = true
  renotify_interval  = 30

  escalation_message = "Node NotReady persists. Check AWS Console for instance health!"
  notify_audit       = false
  notify_no_data     = true # Important for status-based alerts
  no_data_timeframe  = 10
  timeout_h          = 0

  tags = [
    "env:${var.environment}",
    "service:eks",
    "severity:critical",
    "pagerduty:true",
    "monitor_owner:devops-team"
  ]
}

Best Practices for Production Readiness

Tag Everything: Consistently use tags (env, service, owner, cluster) across AWS resources, Kubernetes objects, and Datadog monitors for easier filtering, cost allocation, and organization.
Granular Alerts: Start with critical alerts and gradually refine them. Avoid alert fatigue by setting appropriate thresholds and using composite monitors for complex scenarios.
Runbooks: For every critical alert, have a documented runbook detailing steps for diagnosis and resolution. Link these in your PagerDuty service or Datadog monitor messages.
Cost Optimization: Datadog can be expensive. Monitor your usage, especially for logs and custom metrics. Define clear data retention policies.
Security Best Practices: Use IAM Roles for Service Accounts (IRSA) for your Datadog Agent to securely grant permissions without relying on long-lived AWS credentials. Store API/App keys securely in AWS Secrets Manager or Vault, retrieving them dynamically with Terraform data sources.
Dashboard as Code: Beyond monitors, define your Datadog dashboards using Terraform's `datadog_dashboard` resource for consistent visualization of your EKS health.

Troubleshooting and FAQ

Datadog Agent Pods Not Running

Check `kubectl get pods -n datadog`: Look for pending or crashlooping pods.
Inspect logs (`kubectl logs -f -n datadog`): Common issues include incorrect API keys, RBAC permissions, or resource constraints.
Verify `datadog.apiKey` in Helm values: Ensure it's correctly passed and not truncated.

Metrics Not Appearing in Datadog

Check Agent status: Run `kubectl exec -it -n datadog -- agent status` to see if the agent is collecting and sending data.
Firewall rules: Ensure your EKS cluster nodes have outbound access to Datadog's ingest endpoints (e.g., `https://api.datadoghq.com` or your specific site).
Correct `datadog_site`: Double-check the `datadog_site` variable matches your Datadog region.

PagerDuty Alerts Not Triggering

Datadog Monitor State: Verify the Datadog monitor is actually triggering (e.g., in an ALERT or WARNING state) in the Datadog UI.
PagerDuty Integration: In Datadog, go to Integrations -> PagerDuty. Ensure your PagerDuty service is correctly listed and has a valid integration key.
Notification Handle: Confirm the `@pagerduty-` handle used in the Datadog monitor message exactly matches the one generated by Datadog for your PagerDuty service (case-sensitive and hyphenation matters).
PagerDuty Service Configuration: In PagerDuty, check the service's escalation policy and on-call schedule.

Conclusion

By leveraging Terraform, you can achieve a truly production-ready AWS EKS environment with fully automated Datadog monitoring and PagerDuty alerting. This Infrastructure as Code approach not only streamlines deployment but also enforces consistent observability standards, making your EKS operations more resilient, transparent, and manageable. Continuously review and refine your monitors and alerting strategies as your EKS workloads evolve to maintain optimal production readiness.

CloudStack YAML