Centralized Alerting for Kubernetes with Prometheus and PagerDuty via Terraform

Centralized Alerting for Kubernetes: Prometheus, PagerDuty, and Terraform Integration Guide

In the dynamic world of cloud-native applications, maintaining system reliability and ensuring rapid incident response are paramount. For Kubernetes environments, a robust alerting system is not just a luxury but a necessity. This comprehensive guide details how to build a powerful, centralized alerting solution for your Kubernetes clusters using Prometheus for monitoring, PagerDuty for incident management, and Terraform for declarative infrastructure-as-code (IaC) automation.

Architecture Pro-Tip:

Always prioritize Infrastructure as Code (IaC) for every layer of your monitoring and alerting stack. This ensures reproducibility, version control, and consistent deployments across environments. Managing Kubernetes resources, Prometheus configurations, and PagerDuty services through Terraform reduces manual errors, accelerates provisioning, and enables a true GitOps workflow for your operational tooling. Treat your alerting configuration with the same rigor as your application code.

Why Centralized Alerting Matters for Kubernetes

Kubernetes, while powerful, introduces complexity. Workloads are ephemeral, scaling is dynamic, and failures can cascade. A centralized alerting system:

• Ensures Observability: Provides a single pane of glass for critical metrics and events.
• Reduces Alert Fatigue: Intelligent routing and deduplication prevent unnecessary notifications.
• Accelerates Incident Response: Automates on-call rotations and escalations, directing alerts to the right team members swiftly.
• Promotes Standardization: Enforces consistent alerting policies across services and teams.
• Improves Reliability: Proactive identification of issues before they impact end-users.

Core Components Explained

Prometheus: The Monitoring Powerhouse

Prometheus is an open-source systems monitoring and alerting toolkit. It collects and stores metrics as time series data, exposing a powerful query language (PromQL). Its integrated Alertmanager component handles alerts sent by client applications like Prometheus server. It deduplicates, groups, and routes them to appropriate receivers like email, Slack, or PagerDuty.

PagerDuty: Incident Management and On-Call Automation

PagerDuty is a leading incident management platform that transforms any signal into an actionable incident. It offers robust features like on-call scheduling, escalation policies, automated notifications across multiple channels, and incident conferencing. Integrating Prometheus with PagerDuty ensures that critical alerts are never missed and always reach the right person at the right time.

Terraform: Infrastructure as Code for Everything

Terraform by HashiCorp is an open-source IaC tool that allows you to define and provision infrastructure using a declarative configuration language. With Terraform, you can manage Kubernetes resources (via the Kubernetes provider), deploy Helm charts (via the Helm provider), and even configure PagerDuty services and escalation policies (via the PagerDuty provider). This unified approach simplifies setup, version control, and consistency.

Prerequisites

Before we begin, ensure you have the following:

• An active Kubernetes cluster (e.g., EKS, GKE, AKS, or Kubeadm).
• kubectl configured to connect to your cluster.
• Helm 3 installed.
• Terraform CLI installed (version 1.0+ recommended).
• A PagerDuty account with API access (Admin or Owner role typically required to create API keys).
• A PagerDuty API Key (for the PagerDuty Terraform provider).
• A PagerDuty Routing Key (for the Prometheus Alertmanager integration).

Step-by-Step Implementation Guide

1. Set Up PagerDuty Service and Integration

First, we'll configure PagerDuty. For the purpose of this guide, we'll set up a basic escalation policy and a service. In a real-world scenario, you'd likely have existing escalation policies or more complex setups.

1. Generate a PagerDuty API Key: Go to your PagerDuty account -> Integrations -> API Access Keys -> Create New API Key. Save this key; it's sensitive.
2. Manually create a PagerDuty Service (or use Terraform): For initial testing, you might create a service manually. Go to Services -> Service Directory -> +New Service. Name it "Kubernetes Alerts" or similar. Add an integration of type "Events API v2". Save the generated Routing Key.

Later, we'll demonstrate how to manage this with Terraform.

2. Deploy Prometheus and Alertmanager to Kubernetes

We'll use the robust kube-prometheus-stack Helm chart, which bundles Prometheus, Alertmanager, Grafana, and default dashboards/alert rules.

Create a file named prometheus-values.yaml:

alertmanager: enabled: true ingress: enabled: false # Set to true and configure if you need external access config: # We'll inject this via Terraform later # For now, let's keep it minimal or use default # (PagerDuty configuration comes in the next step) global: resolve_timeout: 5m route: group_by: ['alertname', 'cluster', 'service'] group_wait: 30s group_interval: 5m repeat_interval: 12h receiver: 'default-pagerduty' # This receiver will be defined in the secret receivers: - name: 'default-pagerduty' webhook_configs: - url: 'https://events.pagerduty.com/v2/enqueue' # Placeholder for routing_key, will be replaced by actual secret send_resolved: true prometheus: enabled: true ingress: enabled: false # Set to true and configure if you need external access grafana: enabled: true adminPassword: prom-operator # CHANGE THIS IN PRODUCTION! ingress: enabled: false # Set to true and configure if you need external access defaultRules: create: true kubeEtcd: enabled: true kubeScheduler: enabled: true kubeProxy: enabled: true kubeControllerManager: enabled: true

Deploy the chart (for now, without the PagerDuty specific config, which will come from Terraform):

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update helm install prometheus-stack prometheus-community/kube-prometheus-stack \ --namespace monitoring --create-namespace \ -f prometheus-values.yaml

Verify the deployment:

kubectl get pods -n monitoring kubectl get svc -n monitoring

3. Configure Alertmanager for PagerDuty via Terraform

Now we'll use Terraform to manage the PagerDuty integration for Alertmanager. Alertmanager configuration is typically stored in a Kubernetes Secret (or ConfigMap), which the kube-prometheus-stack chart can be configured to consume. We will create a Kubernetes Secret containing the PagerDuty integration details.

Centralized Terraform Configuration

Here's a consolidated Terraform setup that:

1. Configures the PagerDuty provider.
2. Creates an Escalation Policy and a Service in PagerDuty.
3. Creates a Kubernetes Secret for Alertmanager with the PagerDuty routing key.
4. Deploys the kube-prometheus-stack Helm chart, referencing the Alertmanager secret.
5. Defines a basic Prometheus Alert Rule for demonstration.

main.tf

provider "pagerduty" {
  token = var.pagerduty_api_token
}

provider "kubernetes" {
  # Configuration picked up from KUBECONFIG environment variable or default ~/.kube/config
}

provider "helm" {
  kubernetes {
    # Configuration picked up from KUBECONFIG environment variable or default ~/.kube/config
  }
}

# 1. PagerDuty Escalation Policy
resource "pagerduty_escalation_policy" "devops_escalation" {
  name      = "DevOps Team Escalation Policy"
  num_loops = 2
  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "user"
      id   = var.pagerduty_user_id # Replace with a valid PagerDuty user ID
    }
  }
}

# 2. PagerDuty Service
resource "pagerduty_service" "kubernetes_alerts_service" {
  name                    = "Kubernetes Alerts"
  auto_resolve_timeout_in_minutes = 1440 # 24 hours
  acknowledgement_timeout_in_minutes = 30
  escalation_policy       = pagerduty_escalation_policy.devops_escalation.id
}

# 3. PagerDuty Service Integration (Events API v2)
resource "pagerduty_service_integration" "kubernetes_alertmanager_integration" {
  name    = "Prometheus Alertmanager"
  service = pagerduty_service.kubernetes_alerts_service.id
  type    = "events_api_v2"
}

# 4. Kubernetes Secret for Alertmanager Configuration
resource "kubernetes_secret" "alertmanager_pagerduty_config" {
  metadata {
    name      = "alertmanager-pagerduty-config"
    namespace = "monitoring"
  }

  data = {
    "alertmanager.yaml" = yamlencode({
      global = {
        resolve_timeout = "5m"
      }
      route = {
        group_by       = ["alertname", "namespace", "severity"]
        group_wait     = "30s"
        group_interval = "5m"
        repeat_interval = "4h"
        receiver       = "pagerduty-receiver"
        routes = [
          {
            match = {
              severity = "critical"
            }
            receiver = "pagerduty-receiver"
          },
          {
            match = {
              severity = "warning"
            }
            receiver = "default-receiver" # Optional: for non-critical alerts
          }
        ]
      }
      receivers = [
        {
          name = "pagerduty-receiver"
          pagerduty_configs = [
            {
              routing_key = pagerduty_service_integration.kubernetes_alertmanager_integration.routing_key
              severity    = "{{ .CommonLabels.severity }}"
            }
          ]
        },
        {
          name = "default-receiver" # Placeholder for non-critical notifications (e.g., Slack)
        }
      ]
      templates = []
    })
  }

  type = "Opaque"
}

# 5. Helm Chart Deployment for kube-prometheus-stack
resource "helm_release" "kube_prometheus_stack" {
  name       = "prometheus-stack"
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  namespace  = "monitoring"
  create_namespace = true
  version    = "57.0.0" # Use a stable, recent version
  values = [
    yamlencode({
      alertmanager = {
        enabled = true
        configExternalURL = "http://alertmanager-prometheus-stack.monitoring.svc.cluster.local:9093" # Internal URL
        configFromSecret = kubernetes_secret.alertmanager_pagerduty_config.metadata[0].name
        ingress = {
          enabled = false
        }
      }
      prometheus = {
        enabled = true
        prometheusSpec = {
          serviceMonitorSelectorNilUsesHelmValues = false
          podMonitorSelectorNilUsesHelmValues     = false
          ruleSelectorNilUsesHelmValues           = false
          storageSpec = {
            volumeClaimTemplate = {
              spec = {
                storageClassName = var.storage_class_name
                resources = {
                  requests = {
                    storage = "50Gi"
                  }
                }
              }
            }
          }
        }
        ingress = {
          enabled = false
        }
      }
      grafana = {
        enabled = true
        adminPassword = var.grafana_admin_password
        ingress = {
          enabled = false
        }
      }
      defaultRules = {
        create = true
      }
    })
  ]
}

# 6. Example Prometheus Alert Rule (optional, but good for testing)
resource "kubernetes_manifest" "high_cpu_alert" {
  manifest = {
    apiVersion = "monitoring.coreos.com/v1"
    kind       = "PrometheusRule"
    metadata = {
      name      = "kubernetes-high-cpu"
      namespace = "monitoring"
      labels = {
        prometheus = "prometheus-stack-kube-prom"
        role       = "alert-rules"
      }
    }
    spec = {
      groups = [
        {
          name = "kubernetes.rules"
          rules = [
            {
              alert = "KubeContainerCPUUsageHigh"
              expr  = "sum(rate(container_cpu_usage_seconds_total{namespace=~\"prod|staging\", container!=\"\"}[5m])) by (namespace, pod, container) * 100 > 80"
              for   = "5m"
              labels = {
                severity = "critical"
              }
              annotations = {
                summary     = "High CPU usage detected on container {{ $labels.container }} in pod {{ $labels.pod }}"
                description = "Container {{ $labels.container }} in pod {{ $labels.pod }} (namespace {{ $labels.namespace }}) has been using more than 80% CPU for 5 minutes."
              }
            }
          ]
        }
      ]
    }
  }
}

output "pagerduty_service_url" {
  description = "URL to the created PagerDuty service."
  value       = pagerduty_service.kubernetes_alerts_service.html_url
}

output "pagerduty_integration_key" {
  description = "The integration key for Prometheus Alertmanager to send events to PagerDuty."
  value       = pagerduty_service_integration.kubernetes_alertmanager_integration.routing_key
  sensitive   = true
}
        

variables.tf

variable "pagerduty_api_token" {
  description = "The PagerDuty API token."
  type        = string
  sensitive   = true
}

variable "pagerduty_user_id" {
  description = "The PagerDuty User ID to assign to the escalation policy (e.g., 'PXXXXXXXXXXXXXX')."
  type        = string
}

variable "storage_class_name" {
  description = "The StorageClass name for Prometheus persistent volumes."
  type        = string
  default     = "standard" # Adjust based on your cluster's StorageClass
}

variable "grafana_admin_password" {
  description = "Admin password for Grafana. CHANGE THIS FOR PRODUCTION!"
  type        = string
  sensitive   = true
  default     = "prom-operator"
}
        

To apply this configuration:

# Initialize Terraform
terraform init

# Plan the changes
terraform plan -var="pagerduty_api_token=YOUR_PD_API_KEY" -var="pagerduty_user_id=YOUR_PD_USER_ID" -var="grafana_admin_password=YOUR_GRAFANA_PASSWORD"

# Apply the changes
terraform apply -var="pagerduty_api_token=YOUR_PD_API_KEY" -var="pagerduty_user_id=YOUR_PD_USER_ID" -var="grafana_admin_password=YOUR_GRAFANA_PASSWORD"
        

Testing and Validation

After applying the Terraform configuration, your Kubernetes cluster should have Prometheus and Alertmanager running, with Alertmanager configured to send alerts to your PagerDuty service. You can test this:

1. Port-forward Alertmanager: kubectl -n monitoring port-forward svc/prometheus-stack-kube-prom-alertmanager 9093:9093. Access Alertmanager UI at http://localhost:9093.
2. Verify Alert Rules: Check Prometheus UI (port-forward svc/prometheus-stack-kube-prom-prometheus 9090:9090) under "Alerts" to see if your custom rule (e.g., KubeContainerCPUUsageHigh) is loaded.
3. Trigger a Test Alert:
   • Simulate high CPU: Run a busybox pod with an infinite loop to stress a CPU. For example: kubectl run busybox-stress --rm -it --image=busybox -- /bin/sh -c 'while true; do dd if=/dev/zero of=/dev/null &; done'. Let it run for longer than your alert's for duration (e.g., 5 minutes).
   • Manual Alerting (advanced): Use the Alertmanager API directly for a quick test:
     curl -X POST -H 'Content-Type: application/json' \ -d '[{ "labels": { "alertname": "TestAlert", "severity": "critical", "namespace": "monitoring", "instance": "test-instance" }, "annotations": { "summary": "This is a test alert from Alertmanager." } }]' http://localhost:9093/api/v2/alerts
4. Check PagerDuty: Shortly after the alert fires, you should see a new incident appear in your PagerDuty service.

Best Practices

• Granular Alert Rules: Create specific and actionable alert rules that pinpoint issues rather than general symptoms.
• Define Clear Severities: Use Prometheus labels (e.g., severity: critical, severity: warning) to categorize alerts and map them to appropriate PagerDuty urgency levels.
• Test Thoroughly: Regularly test your entire alerting pipeline to ensure alerts are firing, routing, and escalating as expected.
• Maintain On-Call Schedules: Keep your PagerDuty on-call schedules and escalation policies up-to-date.
• Silence and Maintenance Windows: Leverage Alertmanager's silencing capabilities and PagerDuty's maintenance windows to prevent alert storms during planned maintenance.
• Version Control Everything: Store all your Terraform code, Helm values, and PrometheusRules in a Git repository.
• Review and Refine: Regularly review your alerts and incident reports. Tune thresholds, add new alerts for recurring issues, and remove noisy ones.

Troubleshooting / FAQ

Alerts not firing in PagerDuty

• Check Alertmanager UI: Access the Alertmanager UI (via port-forwarding) to see if alerts are reaching Alertmanager and if there are any errors in the PagerDuty receiver configuration.
• Validate Alertmanager Config: Ensure the alertmanager.yaml inside the Kubernetes Secret is correctly formatted and contains the correct PagerDuty routing key. Check logs of the Alertmanager pod.
• Network Connectivity: Ensure Alertmanager pods can reach events.pagerduty.com (check firewall rules, network policies, egress configurations).
• Prometheus Rules: Verify that your Prometheus alert rules are actually firing (check Prometheus UI -> Alerts tab).
• PagerDuty API Key/Routing Key: Double-check that your PagerDuty API token (for Terraform) and Routing Key (for Alertmanager) are correct and active.

Terraform issues

• Provider Configuration: Ensure your Kubernetes and PagerDuty providers are correctly configured in main.tf.
• Sensitive Variables: Pass sensitive variables (like API keys) securely using environment variables or Terraform's input prompt rather than hardcoding them.
• State Management: For production, configure remote state storage (e.g., S3, Azure Blob Storage, GCS) for your Terraform state to enable collaboration and prevent data loss.

Conclusion

By integrating Prometheus for robust metric collection and alerting, PagerDuty for efficient incident response, and Terraform for declarative, automated deployments, you can establish a highly reliable and maintainable centralized alerting system for your Kubernetes environments. This approach not only streamlines operations but also significantly improves your team's ability to quickly detect, respond to, and resolve critical incidents, ensuring high availability and reliability for your applications.