Terraform for AWS EKS: Production Monitoring with Datadog and PagerDuty

Architecture Pro-Tip: When designing your monitoring and incident response strategy for AWS EKS, always strive for infrastructure-as-code (IaC) principles. Treat your monitoring configurations, alert definitions, and incident management services (like Datadog and PagerDuty) as code. This ensures version control, auditability, repeatability, and consistency across environments, significantly reducing manual errors and accelerating disaster recovery efforts. Automate not just the deployment of your applications, but also the observability and incident workflow around them.

Terraform for AWS EKS: Production Monitoring with Datadog and PagerDuty

In the demanding world of cloud-native applications, maintaining robust observability and a rapid incident response mechanism is paramount, especially for critical workloads running on AWS Elastic Kubernetes Service (EKS). This guide provides a comprehensive, technical walkthrough on how to leverage Terraform to provision and manage production-grade monitoring for your EKS clusters using industry leaders Datadog for observability and PagerDuty for incident management. By codifying your monitoring infrastructure, you gain unparalleled consistency, scalability, and reliability.

Why Terraform, Datadog, and PagerDuty for EKS?

Managing Kubernetes clusters in production environments introduces significant complexity. From resource utilization to application health, dozens of metrics need constant vigilance.

Terraform: Enables Infrastructure as Code (IaC) for EKS, Datadog monitors, and PagerDuty services, ensuring repeatable deployments and version control.
Datadog: Provides end-to-end visibility across your EKS infrastructure, applications, and logs. It offers comprehensive dashboards, anomaly detection, and powerful alerting capabilities.
PagerDuty: Automates incident response, ensuring the right people are notified at the right time through flexible on-call schedules, escalation policies, and integrations.

Together, these tools form a formidable stack for maintaining high availability and operational excellence for your Kubernetes workloads on AWS.

Prerequisites

Before diving into the configurations, ensure you have the following in place:

An AWS account with necessary permissions to create EKS clusters and associated resources.
Terraform (v1.0+) installed and configured with AWS credentials.
kubectl installed and configured to interact with your EKS cluster.
A Datadog account with API and Application keys.
A PagerDuty account with a User API Token and an associated service/team.
An existing AWS EKS cluster, or the ability to provision one using Terraform (which is beyond the scope of this specific monitoring guide but assumed).

Step 1: Integrating Datadog for EKS Monitoring with Terraform

The first step in achieving comprehensive observability is deploying the Datadog Agent to your EKS cluster. The Datadog Agent collects metrics, logs, and traces from your nodes, pods, and applications. We'll deploy it using the official Datadog Helm chart, managed by Terraform's Helm provider.

1.1. Configure Terraform Providers

Ensure your Terraform configuration includes the AWS, Helm, and Kubernetes providers. The Kubernetes provider will use credentials derived from your AWS EKS cluster. You'll also need the Datadog provider to manage monitors.

provider "aws" {
  region = "us-east-1"
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.example.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.example.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.example.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.example.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.example.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.example.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

data "aws_eks_cluster" "example" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "example" {
  name = var.eks_cluster_name
}
    

1.2. Deploy Datadog Agent with Terraform Helm Provider

This Terraform configuration uses the Helm provider to deploy the Datadog Agent. Remember to replace placeholder values like <YOUR_DATADOG_API_KEY> and <YOUR_DATADOG_APP_KEY> with your actual keys, preferably via environment variables or a secure secret management system.

Terraform Configuration for Datadog Agent

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  version    = "2.33.0" # Use a specific, stable version
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    type  = "string"
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    type  = "string"
  }

  # Enable Kubernetes monitoring features
  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "clusterChecksRunner.enabled"
    value = "true"
  }

  set {
    name  = "processAgent.enabled"
    value = "true"
  }

  # Enable APM for distributed tracing
  set {
    name  = "apm.enabled"
    value = "true"
  }

  # Enable log collection
  set {
    name  = "logs.enabled"
    value = "true"
  }

  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }

  # Ensure hostPort is not enabled for security in EKS
  set {
    name  = "hostPortEnabled"
    value = "false"
  }

  # Optional: Enable admission controller for advanced features like APM injection
  set {
    name  = "admissionController.enabled"
    value = "true"
  }
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "eks_cluster_name" {
  description = "The name of your EKS cluster"
  type        = string
}
    

After applying this configuration (terraform init, terraform apply), the Datadog Agent will be deployed as a DaemonSet across your EKS nodes, and the Cluster Agent as a Deployment. You should soon see your EKS cluster data flowing into Datadog.

1.3. Create Datadog Monitors with Terraform

Once data is flowing, you can define specific monitors in Datadog to alert on critical conditions. Here's an example of a Datadog monitor created using Terraform to alert on high CPU utilization across your EKS nodes.

resource "datadog_monitor" "high_node_cpu" {
  name               = "[EKS] High Node CPU Utilization on {{host.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster_name:${var.eks_cluster_name}} by {host} < 20"
  message            = "CPU utilization on host {{host.name}} is over 80%. Consider scaling or investigating. @pagerduty-EKS-Critical-Service"
  tags               = ["environment:${var.environment}", "service:eks-platform", "severity:critical"]
  priority           = 1
  restricted_roles   = ["${var.datadog_restricted_role_id}"] # Optional: Restrict visibility to specific roles

  monitor_thresholds {
    critical = 20
    warning  = 30
  }

  # Configure downtime for no data
  new_group_delay = 120
  new_host_delay  = 300
  no_data_timeframe = 20

  # Re-notify every 60 minutes if the alert persists
  renotify_interval = 60

  # Auto-resolve after 10 minutes if conditions are met
  timeout_h = 10
}

variable "environment" {
  description = "The deployment environment (e.g., prod, staging)"
  type        = string
}

variable "datadog_restricted_role_id" {
  description = "Optional: ID of a Datadog role to restrict monitor access"
  type        = string
  default     = null
}
    

Notice the @pagerduty-EKS-Critical-Service in the message field. This is how Datadog integrates with PagerDuty, by tagging the alert message with the PagerDuty integration key or service name. We'll set this up next.

Step 2: Integrating PagerDuty for Incident Management with Terraform

PagerDuty acts as your central nervous system for incident response. We'll use Terraform to define a PagerDuty service that will receive alerts from Datadog and trigger incidents.

2.1. Configure Terraform PagerDuty Provider

Add the PagerDuty provider to your Terraform configuration. You'll need a PagerDuty User API Token.

provider "pagerduty" {
  token = var.pagerduty_auth_token
}

variable "pagerduty_auth_token" {
  description = "PagerDuty User API Token"
  type        = string
  sensitive   = true
}
    

2.2. Create PagerDuty Service with Terraform

A PagerDuty service represents a component or application that you want to monitor. When an alert related to this service is received, PagerDuty creates an incident and follows its associated escalation policy.

resource "pagerduty_service" "eks_critical_service" {
  name                      = "EKS Critical Workloads Monitoring - ${var.environment}"
  auto_resolve_timeout      = 14400 # 4 hours
  acknowledgement_timeout   = 1800  # 30 minutes
  escalation_policy         = pagerduty_escalation_policy.eks_critical_policy.id
  alert_creation_parameters {
    type = "create_incidents_and_alert_then_send"
  }
}

resource "pagerduty_escalation_policy" "eks_critical_policy" {
  name      = "EKS Critical Policy - ${var.environment}"
  num_loops = 2 # Escalate twice

  rule {
    escalation_delay_in_minutes = 15
    target {
      type = "user"
      id   = var.pagerduty_oncall_user_id # ID of the primary on-call user
    }
  }

  rule {
    escalation_delay_in_minutes = 30
    target {
      type = "team"
      id   = var.pagerduty_ops_team_id # ID of the operations team
    }
  }
}

variable "pagerduty_oncall_user_id" {
  description = "ID of the primary on-call user in PagerDuty"
  type        = string
}

variable "pagerduty_ops_team_id" {
  description = "ID of the operations team in PagerDuty for secondary escalation"
  type        = string
}
    

This configuration creates a PagerDuty service and a basic escalation policy. You'll need to obtain the IDs for your PagerDuty users and teams.

2.3. Link Datadog to PagerDuty

After creating the PagerDuty service, you need to integrate it with Datadog. Datadog provides a native integration with PagerDuty. You can configure this integration in the Datadog UI (Integrations -> PagerDuty) or use the Datadog Terraform provider. For a clean IaC approach, we prefer the latter.

resource "datadog_integration_pagerduty" "eks_critical_pagerduty_integration" {
  services {
    service_name = pagerduty_service.eks_critical_service.name
    service_key  = pagerduty_service.eks_critical_service.integration_key # This key is specific to the Datadog integration for the PagerDuty service
  }
}
    

This Terraform resource links your newly created PagerDuty service with Datadog. The integration_key from the PagerDuty service is crucial here. Once this is applied, you can use the @pagerduty-EKS-Critical-Workloads-Monitoring-prod (or whatever your service name is) in your Datadog monitor messages to trigger incidents.

Step 3: Production Monitoring Best Practices

Deploying the agents and setting up basic alerts is just the beginning. For a truly resilient production environment, consider these best practices:

Service Level Objectives (SLOs) and Service Level Indicators (SLIs): Define clear SLOs for your EKS applications and create Datadog monitors based on SLIs (e.g., request latency, error rate, uptime).
Comprehensive Alerting: Move beyond basic CPU/memory alerts. Monitor application-specific metrics, Kubernetes events, pod restarts, deployment failures, and saturated resources.
Alert Fatigue Reduction:
- Tune Thresholds: Continuously adjust monitor thresholds to reduce noise.
- Deduplication & Grouping: Leverage Datadog's capabilities to group related alerts.
- Clear Messages & Runbooks: Ensure every alert has a clear, actionable message and links to a runbook or documentation for faster resolution.
Synthetic Monitoring: Implement Datadog synthetic tests for external and internal endpoints to proactively detect issues before they impact users.
Distributed Tracing: Utilize Datadog APM to get deep visibility into application performance bottlenecks and dependencies within your EKS microservices.
Log Management: Centralize all EKS and application logs in Datadog for easier troubleshooting and correlation with metrics and traces.
Cost Optimization Monitoring: Monitor AWS costs related to EKS and identify opportunities for resource optimization.
Security Monitoring: Integrate Datadog Security Monitoring to detect threats and vulnerabilities across your EKS cluster.

Troubleshooting and FAQs

Q: Datadog Agent isn't reporting data. What should I check?

API Key/App Key: Double-check that your Datadog API and Application keys are correctly set in the Helm release values.
Pod Status: Run kubectl get pods -n datadog to ensure Datadog Agent pods are running and healthy. Check logs for errors using kubectl logs <datadog-agent-pod> -n datadog.
Network Connectivity: Verify that your EKS nodes have outbound network access to Datadog endpoints.
RBAC Permissions: Ensure the service account used by the Datadog Agent has the necessary Kubernetes RBAC permissions. The Helm chart usually handles this, but custom setups might interfere.

Q: PagerDuty incidents are not being triggered by Datadog alerts.

Monitor Message: Confirm the @pagerduty-<YOUR_SERVICE_NAME> tag in your Datadog monitor message exactly matches the name of your PagerDuty service as configured in Datadog's integration.
Datadog PagerDuty Integration: In Datadog, navigate to Integrations -> PagerDuty and ensure your PagerDuty service is listed and configured correctly. Verify the integration_key is correct.
Escalation Policy: Check your PagerDuty service's escalation policy and on-call schedules to ensure there are users available to be notified.
Monitor State: Ensure the Datadog monitor is actually triggering an alert (e.g., check its status in Datadog).

Conclusion

Establishing robust production monitoring for AWS EKS is a critical component of operational excellence. By adopting an Infrastructure as Code approach with Terraform, you can seamlessly integrate Datadog for comprehensive observability and PagerDuty for efficient incident response. This not only streamlines your DevOps workflows but also ensures that your critical Kubernetes applications remain performant, reliable, and secure. Continuously iterate on your monitoring strategy, refine your alerts, and keep your incident response procedures sharp to minimize downtime and maintain a healthy, observable cloud-native environment.

CloudStack YAML