Automated Alerting with Prometheus and PagerDuty on Kubernetes using Terraform

In modern cloud-native environments, reliable and automated alerting is paramount for maintaining system health and ensuring rapid response to incidents. This guide provides a comprehensive technical walkthrough on how to set up an advanced alerting system for your Kubernetes clusters, leveraging the power of Prometheus for monitoring, PagerDuty for incident management, and Terraform for infrastructure as code (IaC).

Architecture Pro-Tip

For robust alerting, always design for observability, not just notifications. Integrate your monitoring system (Prometheus) deeply with your incident management platform (PagerDuty) to ensure critical alerts trigger immediate human intervention. Use Terraform to codify this entire alerting pipeline, making it repeatable, auditable, and scalable across multiple environments. Focus on actionable alerts, establish clear escalation policies, and regularly review your alert rules to combat alert fatigue.

Why This Stack?

Combining Prometheus, PagerDuty, and Terraform offers a robust, scalable, and automated solution for incident response:

• Prometheus: A leading open-source monitoring solution, widely adopted in Kubernetes for its powerful multi-dimensional data model, flexible query language (PromQL), and built-in Alertmanager.
• PagerDuty: An industry-standard incident management platform that transforms machine data into actionable incidents, ensuring the right person is notified at the right time through flexible on-call schedules and escalation policies.
• Terraform: The de facto tool for Infrastructure as Code, allowing you to define and provision cloud and on-premises resources (including Kubernetes configurations and PagerDuty services) in a declarative manner. This ensures consistency, version control, and automation.

Prerequisites

Before you begin, ensure you have the following:

• A running Kubernetes cluster (e.g., EKS, GKE, AKS, or on-prem).
• kubectl configured to connect to your cluster.
• helm (version 3+) installed.
• Terraform (version 1.0+) installed.
• A PagerDuty account with administrative access to create services, escalation policies, and API keys.
• A PagerDuty API Token (Admin API Key), which will be used by Terraform. Set it as an environment variable: export PAGERDUTY_TOKEN="YOUR_PD_API_TOKEN".
• A PagerDuty User ID to assign to the escalation policy. You can find this in the PagerDuty UI under your user profile URL (e.g., https://your-domain.pagerduty.com/users/P1234567, where P1234567 is the ID).

Core Components Overview

1. PagerDuty Setup

We'll use Terraform to automate the creation of:

• Escalation Policy: Defines the order in which users are notified of an incident.
• Service: Represents a system, application, or component that you want to monitor and manage incidents for.
• Service Integration: Specifically a "Generic Events API V2" integration, which provides an API key (service key) that Alertmanager will use to send events to PagerDuty.

2. Prometheus and Alertmanager on Kubernetes

We'll deploy the kube-prometheus-stack Helm chart, which includes:

• Prometheus: Scrapes metrics from your Kubernetes cluster.
• Alertmanager: Handles alerts sent by Prometheus, deduplicating, grouping, and routing them to the correct receiver (in this case, PagerDuty).
• Prometheus Operator: Manages Prometheus instances, Alertmanager instances, and related custom resources.

The key is to configure Alertmanager with the PagerDuty service key obtained from the Terraform-managed integration.

Terraform Configuration for Automated Alerting

This section provides the complete Terraform configuration to set up your PagerDuty service and deploy the Prometheus stack with integrated alerting to Kubernetes.

Project Structure

Organize your Terraform files like this:

.
├── main.tf
└── variables.tf
    

main.tf: Core Resources

This file defines the providers, PagerDuty resources (escalation policy, service, integration), and the Helm release for the Prometheus stack.

# main.tf # Define required providers and their versions terraform { required_providers { pagerduty = { source = "PagerDuty/pagerduty" version = "~> 1.14.0" } kubernetes = { source = "hashicorp/kubernetes" version = "~> 2.23.0" } helm = { source = "hashicorp/helm" version = "~> 2.11.0" } } } # --- PagerDuty Provider --- # Assumes PAGERDUTY_TOKEN environment variable is set for authentication provider "pagerduty" {} # --- Kubernetes Provider --- # Assumes kubectl is configured (e.g., KUBECONFIG env var or default path) provider "kubernetes" {} # --- Helm Provider --- provider "helm" {} # --- PagerDuty Resources --- # Create an escalation policy resource "pagerduty_escalation_policy" "default_policy" { name = var.pagerduty_escalation_policy_name num_loops = 2 # Number of times the policy will repeat after the last step is reached rule { escalation_delay_in_minutes = 15 # Wait 15 minutes before escalating target { type = "user" id = var.pagerduty_user_id # Replace with a valid PagerDuty user ID (e.g., P123ABCD) } } } # Create a PagerDuty service for Kubernetes alerts resource "pagerduty_service" "prometheus_k8s_alerts_service" { name = var.pagerduty_service_name auto_resolve_timeout = 14400 # 4 hours - incidents resolve automatically if not triggered again acknowledgement_timeout = 600 # 10 minutes - if an alert is acknowledged within this time, it won't escalate further escalation_policy = pagerduty_escalation_policy.default_policy.id } # Create a PagerDuty service integration for Prometheus (Generic Events API V2) resource "pagerduty_service_integration" "prometheus_integration" { name = "Prometheus Alerts" type = "generic_events_api_v2" service = pagerduty_service.prometheus_k8s_alerts_service.id } # --- Helm Chart for Prometheus Stack --- # Deploy the kube-prometheus-stack which includes Prometheus, Alertmanager, etc. resource "helm_release" "prometheus_stack" { name = "prometheus-stack" repository = "https://prometheus-community.github.io/helm-charts" chart = "kube-prometheus-stack" version = "50.0.0" # Use a recent stable version suitable for your K8s version namespace = "monitoring" create_namespace = true # Create the namespace if it doesn't exist # Inject Alertmanager configuration to send alerts to PagerDuty values = [<# Replace with your external URL if applicable prometheus: prometheusSpec: # Example alert rule for testing. This rule will always fire. # In a real scenario, you would have actual metrics-based rules. additionalPrometheusRules: - name: general.rules groups: - name: demo-alerts rules: - alert: HighCPUUsage expr: vector(1) # An expression that is always true for demonstration for: 1m # Alert after 1 minute of being true labels: severity: critical priority: p1 annotations: summary: "Simulated high CPU usage detected on Kubernetes node" description: "This is a test alert from Prometheus. Please investigate if this were real!" EOT ] }

variables.tf: Input Variables

Define variables for sensitive information and configurable names.

# variables.tf # PagerDuty API token (sensitive, provided via env var PAGERDUTY_TOKEN) variable "pagerduty_token" { description = "The PagerDuty API token." type = string sensitive = true } # Name for the PagerDuty escalation policy variable "pagerduty_escalation_policy_name" { description = "The name for the PagerDuty escalation policy." type = string default = "Kubernetes Alerts Policy" } # Name for the PagerDuty service variable "pagerduty_service_name" { description = "The name for the PagerDuty service." type = string default = "Kubernetes Monitoring" } # PagerDuty user ID to be included in the escalation policy variable "pagerduty_user_id" { description = "The ID of the PagerDuty user to add to the escalation policy (e.g., P123ABCD)." type = string # IMPORTANT: Replace "YOUR_PAGERDUTY_USER_ID" with an actual user ID from your PagerDuty account. # You can find this in the URL when viewing a user's profile (e.g., P1234567). }

Deployment Steps

Follow these steps to deploy your automated alerting system:

1. Save the Files: Save the above content into main.tf and variables.tf in a new directory.
2. Set Environment Variable: Ensure your PagerDuty API token is set as an environment variable:
   export PAGERDUTY_TOKEN="YOUR_PAGERDUTY_ADMIN_API_KEY"

   And populate the pagerduty_user_id variable in variables.tf with a valid PagerDuty user ID.

3. Initialize Terraform: Navigate to your project directory and run:
   terraform init
4. Review the Plan: Examine the changes Terraform will apply:
   terraform plan
5. Apply the Configuration: Apply the changes to create the PagerDuty resources and deploy Prometheus to Kubernetes:
   terraform apply --auto-approve

Testing and Validation

Once Terraform successfully applies the configuration:

1. Verify PagerDuty Setup: Log in to your PagerDuty account. You should see a new escalation policy and a service named "Kubernetes Monitoring" (or whatever you configured) with a "Prometheus Alerts" integration.
2. Verify Kubernetes Deployment: Check the status of your Prometheus pods:
   kubectl get pods -n monitoring

   All pods (Prometheus, Alertmanager, Grafana, etc.) should be running.

3. Trigger and Verify Alert: The example alert rule HighCPUUsage in main.tf is designed to always be true. After the for: 1m duration, Prometheus will send this alert to Alertmanager, which in turn will forward it to PagerDuty.

   Within a few minutes of successful deployment, you should receive an incident notification from PagerDuty based on your escalation policy. Log in to PagerDuty to see the triggered incident.

4. Access Alertmanager UI (Optional): To see alerts directly in Alertmanager, you can port-forward the Alertmanager service:
   kubectl -n monitoring port-forward svc/prometheus-stack-kube-pr-alertmanager 9093:9093

   Then open http://localhost:9093 in your browser to view active and inhibited alerts.

Best Practices for Alerting

• Actionable Alerts: Ensure every alert has a clear purpose and a defined response plan. Avoid "noisy" alerts that don't indicate an immediate problem.
• Runbooks: Link runbooks or documentation directly from your PagerDuty incidents to guide responders through troubleshooting and resolution.
• SLOs/SLAs: Define Service Level Objectives (SLOs) and Service Level Agreements (SLAs) for your services and configure alerts based on deviations from these targets.
• Deduplication and Grouping: Alertmanager's powerful grouping features help prevent alert storms by bundling similar alerts into a single PagerDuty incident. Configure group_by, group_wait, and group_interval wisely.
• Regular Review: Periodically review your alert rules, escalation policies, and on-call schedules to ensure they remain relevant and effective.
• Testing: Regularly test your alerting pipeline (e.g., using a dedicated test alert or chaos engineering practices) to confirm it functions as expected.

Troubleshooting Common Issues

1. PagerDuty Incident Not Triggering

• Check Alertmanager Logs: Look for errors in the Alertmanager pod logs: kubectl logs -n monitoring prometheus-stack-kube-pr-alertmanager-0.
• Verify Service Key: Double-check that the service_key in your Alertmanager configuration within main.tf matches the integration key generated by PagerDuty.
• Prometheus to Alertmanager Connectivity: Ensure Prometheus is correctly configured to send alerts to Alertmanager. Check Prometheus configuration via its UI (port-forward svc/prometheus-stack-kube-pr-prometheus 9090:9090 and navigate to 'Status' -> 'Targets' and 'Alerts').
• Alert Rule Firing: In the Prometheus UI, check the 'Alerts' tab to confirm your test alert is in a 'FIRING' state.

2. Terraform Apply Errors

• Provider Authentication: Ensure PAGERDUTY_TOKEN environment variable is set correctly and the pagerduty_user_id variable is valid. For Kubernetes, verify kubectl context is correct.
• Helm Chart Version: If you face issues with the Helm chart, try a different stable version or check the chart's documentation for breaking changes.

Conclusion

By following this guide, you've successfully implemented a powerful and automated alerting system for your Kubernetes cluster. Leveraging Prometheus for robust monitoring, PagerDuty for efficient incident management, and Terraform for declarative infrastructure, you've established a resilient pipeline that ensures critical issues are detected and escalated to the right teams promptly. This setup not only improves operational efficiency but also contributes significantly to the reliability and stability of your cloud-native applications. Continue to refine your alert rules and escalation policies to adapt to your evolving system needs and maintain a healthy, responsive environment.