Terraform for Kubernetes: Centralized Observability & Incident Management with Prometheus, Datadog, and PagerDuty

In today's dynamic cloud-native landscape, managing Kubernetes clusters efficiently demands robust observability and incident management. As applications scale and microservices proliferate, a fragmented approach to monitoring and alerting leads to blind spots and delayed incident resolution. This guide demonstrates how to leverage Terraform to provision and configure a centralized observability and incident management solution for Kubernetes, integrating industry-leading tools like Prometheus, Datadog, and PagerDuty.

Architecture Pro-Tip: Adopt a GitOps Philosophy

For maximum consistency and traceability, manage all your observability and incident management configurations, including Terraform code, Kubernetes manifests, and Helm charts, within a Git repository. Implement a GitOps workflow where changes to this repository automatically trigger updates to your infrastructure and applications. This ensures version control, peer review, and a single source of truth for your entire operational setup, crucial for complex Kubernetes environments.

Why Centralized Observability and Incident Management?

Kubernetes, while powerful, introduces significant operational complexity. Workloads are ephemeral, distributed, and highly dynamic. Centralized observability provides a holistic view of your system's health, performance, and behavior, enabling proactive identification and resolution of issues. When incidents do occur, a well-defined incident management pipeline ensures rapid response, clear communication, and efficient resolution, minimizing downtime and business impact.

Unified Visibility: Aggregate metrics, logs, and traces from all Kubernetes components and applications.
Faster MTTR (Mean Time To Resolution): Quickly pinpoint the root cause of issues with comprehensive data.
Proactive Alerting: Set up intelligent alerts to notify teams before minor issues escalate.
Streamlined On-Call: Automate incident routing, escalation, and post-mortems.
Improved Collaboration: Foster better communication between development, operations, and SRE teams.

The Toolchain: Prometheus, Datadog, and PagerDuty

This guide focuses on a robust combination of tools, each excelling in its specific domain:

Prometheus: Kubernetes-Native Monitoring

Prometheus is an open-source monitoring system with a dimensional data model, flexible query language (PromQL), and an alert manager. It's the de facto standard for Kubernetes monitoring, capable of scraping metrics from pods, nodes, and internal Kubernetes components.

Key Features: Multi-dimensional data model, powerful PromQL, push gateway for short-lived jobs, service discovery via Kubernetes API.
Role in Our Stack: Primary collector of raw, granular Kubernetes infrastructure and application metrics.

Datadog: Comprehensive Observability Platform

Datadog is a SaaS-based monitoring and analytics platform that brings together metrics, logs, traces, and synthetics into a single pane of glass. It provides powerful dashboards, anomaly detection, and extensive integrations.

Key Features: Unified dashboards, AI-powered anomaly detection, APM, log management, network performance monitoring, real user monitoring, cloud integration.
Role in Our Stack: Aggregator of metrics (including potentially from Prometheus via integrations), log management, distributed tracing, high-level dashboards, and the primary alerting engine for complex scenarios.

PagerDuty: Real-time Incident Management

PagerDuty is an incident management platform that orchestrates and automates response to critical incidents. It provides on-call scheduling, escalations, communication automation, and post-incident analysis tools.

Key Features: On-call scheduling, multi-channel notifications, automated escalations, incident conferencing, runbook automation.
Role in Our Stack: The final destination for critical alerts from Datadog, ensuring the right team member is notified and incidents are managed according to defined policies.

Terraform: Infrastructure as Code for Observability

Terraform, HashiCorp's open-source Infrastructure as Code (IaC) tool, allows you to define and provision infrastructure using a declarative configuration language. By using Terraform, you can:

Version Control: Track changes to your observability setup in Git.
Automation: Eliminate manual configuration errors and speed up deployments.
Reproducibility: Easily recreate environments or deploy consistent configurations across multiple clusters.
Collaboration: Facilitate teamwork with standardized definitions.

We will use the following Terraform providers:

Kubernetes Provider: To interact with the Kubernetes API for deploying resources.
Helm Provider: To deploy Prometheus and Datadog agents via Helm charts.
Datadog Provider: To configure Datadog monitors, dashboards, and integrations.
PagerDuty Provider: To set up PagerDuty services, escalation policies, and users.

Step-by-Step Implementation with Terraform

Prerequisites:

An existing Kubernetes cluster (EKS, GKE, AKS, or on-prem).
Terraform CLI installed.
kubectl configured to access your cluster.
Datadog API and Application keys.
PagerDuty API key and an existing PagerDuty account.

1. Deploying Prometheus Operator to Kubernetes

We'll use the Helm provider to deploy the kube-prometheus-stack, which includes Prometheus, Grafana, and Alertmanager.

Create a main.tf:

resource "helm_release" "kube_prometheus_stack" {
  name       = "kube-prometheus-stack"
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  namespace  = "monitoring"
  create_namespace = true
  version    = "45.2.0" # Use a specific version
  
  set {
    name  = "grafana.enabled"
    value = "true"
  }
  set {
    name  = "prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues"
    value = "false"
  }
  set {
    name  = "prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues"
    value = "false"
  }
}

2. Integrating Datadog for Centralized Observability

First, deploy the Datadog Agent to your Kubernetes cluster using the Helm provider. Then, use the Datadog provider to configure monitors and dashboards.

Add to main.tf:

resource "kubernetes_secret" "datadog_api_key" {
  metadata {
    name = "datadog-api-key"
    namespace = "datadog"
  }
  data = {
    "api-key" = var.datadog_api_key
  }
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true
  version    = "2.32.0" # Use a specific version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }
  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
  }
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "processAgent.enabled"
    value = "true"
  }
}

resource "datadog_monitor" "high_cpu_usage" {
  name                = "Kubernetes Node High CPU Usage (Managed by Terraform)"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.cpu.usage.total{*} by {host} > 80"
  message             = "@webhook-pagerduty @slack-channel-devops Node {{host.name}} has high CPU usage!"
  tags                = ["env:production", "service:kubernetes", "terraform"]
  renotify_interval   = 60
  no_data_timeframe   = 20
  include_tags        = true
  require_full_window = false
  notify_audit        = false
  notify_no_data      = false
  timeout_h           = 0
  escalation_message  = "CPU usage remains high for {{host.name}} after 1 hour. Please investigate immediately."
}

3. Setting up PagerDuty for Incident Management

We'll use the PagerDuty provider to create a service, an escalation policy, and integrate it with Datadog.

Add to main.tf:

resource "pagerduty_team" "devops_team" {
  name        = "DevOps Team"
  description = "Team responsible for Kubernetes infrastructure and operations."
}

resource "pagerduty_escalation_policy" "kubernetes_critical" {
  name      = "Kubernetes Critical Escalation Policy"
  team_ids  = [pagerduty_team.devops_team.id]

  rule {
    # Notify primary on-call immediately
    target {
      type = "user"
      id   = var.pagerduty_primary_oncall_user_id
    }
    delay = 0
  }

  rule {
    # Escalate to secondary on-call after 15 minutes
    target {
      type = "user"
      id   = var.pagerduty_secondary_oncall_user_id
    }
    delay = 15
  }
}

resource "pagerduty_service" "kubernetes_observability_service" {
  name                    = "Kubernetes Observability Service"
  auto_resolve_timeout_minutes = 60
  acknowledgement_timeout_minutes = 30
  escalation_policy       = pagerduty_escalation_policy.kubernetes_critical.id
  team                    = pagerduty_team.devops_team.id
}

resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog Integration"
  service     = pagerduty_service.kubernetes_observability_service.id
  type        = "datadog_inbound_integration"
  vendor      = "P80KGFN" # This is the vendor ID for Datadog in PagerDuty
}

# Output the PagerDuty integration key to be used in Datadog webhooks
output "pagerduty_datadog_integration_key" {
  value       = pagerduty_service_integration.datadog_integration.integration_key
  description = "The PagerDuty integration key for Datadog. Use this in Datadog webhook notifications."
  sensitive   = true
}

Now, you need to configure Datadog to use this integration key. In Datadog, go to Integrations > Webhooks and create a new webhook pointing to https://events.pagerduty.com/generic/2010-04-15/create_event.json with the PagerDuty integration key as a parameter. Then, reference this webhook in your Datadog monitor's message, e.g., @webhook-pagerduty-kubernetes.

Variables and Providers Configuration (`variables.tf` and `providers.tf`)

Create a variables.tf file:

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

variable "pagerduty_primary_oncall_user_id" {
  description = "PagerDuty User ID for Primary On-Call"
  type        = string
}

variable "pagerduty_secondary_oncall_user_id" {
  description = "PagerDuty User ID for Secondary On-Call"
  type        = string
}

Create a providers.tf file:

terraform {
  required_providers {
    kubernetes = {
      source = "hashicorp/kubernetes"
      version = "~> 2.0"
    }
    helm = {
      source = "hashicorp/helm"
      version = "~> 2.0"
    }
    datadog = {
      source = "DataDog/datadog"
      version = "~> 3.0"
    }
    pagerduty = {
      source = "pagerduty/pagerduty"
      version = "~> 1.0"
    }
  }
}

provider "kubernetes" {
  # Configuration picked up from kubeconfig
}

provider "helm" {
  kubernetes {
    # Configuration picked up from kubeconfig
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_token
}

Deployment Steps:

Initialize Terraform: terraform init
Review the plan: terraform plan
Apply the configuration: terraform apply (You'll be prompted for sensitive variables or provide them via TF_VAR_... environment variables)

Best Practices for Centralized Observability & Incident Management

Tagging & Naming Conventions: Implement consistent tags (e.g., env:production, service:auth, team:sre) across all resources and tools. This is crucial for filtering, dashboarding, and correlating data.
Alert Fatigue Mitigation:
- Define clear alert thresholds based on SLOs/SLAs.
- Use Datadog's anomaly detection and composite monitors to reduce noise.
- Leverage PagerDuty's incident grouping and suppression rules.
- Ensure alerts are actionable and provide sufficient context.
Runbook Automation: For common incidents, create automated runbooks in PagerDuty or link to documentation that guides responders through resolution steps.
Regular Review: Periodically review your monitors, alerts, and escalation policies. As your system evolves, so should your observability strategy.
Security: Store API keys and sensitive information securely, preferably using a secrets management solution like HashiCorp Vault or Kubernetes Secrets (for Kubernetes components) and referencing them in Terraform.
Cost Management: Monitor your Datadog usage, especially for logs and custom metrics, as costs can scale rapidly. Optimize data ingestion where possible.

Troubleshooting Common Issues

Terraform Apply Fails:
- Check provider authentication (kubeconfig for Kubernetes/Helm, API keys for Datadog/PagerDuty).
- Review error messages carefully; they often point to specific resource misconfigurations.
- Ensure Helm chart versions are compatible and repositories are correct.
Datadog Agent Not Reporting:
- Verify datadog.apiKey and datadog.appKey are correct in the Helm release.
- Check Datadog Agent pod logs for errors (kubectl logs -n datadog -l app=datadog).
- Ensure network policies aren't blocking outbound traffic from the Datadog Agent to Datadog's endpoints.
Alerts Not Triggering PagerDuty:
- Confirm the PagerDuty integration key in Datadog's webhook configuration is correct.
- Verify the Datadog monitor's message references the correct webhook (e.g., @webhook-pagerduty-kubernetes).
- Check PagerDuty's "Recent Incidents" or "Service Events" logs for any received events or errors.
- Ensure the Datadog monitor itself is triggering as expected within Datadog.
Prometheus Metrics Missing:
- Check Prometheus targets in the Prometheus UI to see if they are UP.
- Ensure ServiceMonitor or PodMonitor resources are correctly defined and selected by Prometheus.
- Verify applications are exposing metrics in Prometheus format on the expected ports/paths.

Conclusion

By leveraging Terraform, you can provision and manage a sophisticated, centralized observability and incident management system for your Kubernetes environments with consistency and confidence. The combination of Prometheus for granular Kubernetes metrics, Datadog for unified observability and intelligent alerting, and PagerDuty for streamlined incident response creates a robust operational framework.

Embracing Infrastructure as Code for these critical tools not only automates their deployment but also embeds best practices into your operational workflow, leading to increased stability, faster resolution times, and ultimately, a more reliable and resilient cloud-native infrastructure.

CloudStack YAML