Terraform Configuration for Automated Kubernetes Observability with Prometheus and PagerDuty

In the dynamic world of cloud-native applications, maintaining robust observability is paramount. Kubernetes, while offering unparalleled scalability and resilience, introduces complexity that demands sophisticated monitoring and incident management strategies. This guide provides a comprehensive, technical walkthrough on leveraging Terraform to automate the deployment and configuration of Prometheus for Kubernetes observability, seamlessly integrating with PagerDuty for automated incident response. This Infrastructure as Code (IaC) approach ensures consistency, repeatability, and efficient management of your monitoring stack.

Architecture Pro-Tip:

For mission-critical Kubernetes environments, a resilient observability architecture is non-negotiable. Deploy Prometheus in a highly available setup (e.g., using Thanos for long-term storage and global view), configure Alertmanager with multiple receivers, and ensure your PagerDuty services reflect your on-call rotations and escalation policies accurately. Automating this entire stack with Terraform not only reduces manual errors but also enables rapid disaster recovery and consistent deployments across different environments (dev, staging, production).

Understanding the Core Components

Before diving into the Terraform code, let's briefly review the roles of the key players in our observability stack:

Kubernetes: The container orchestration platform where your applications run. Our goal is to monitor its health and the performance of workloads within it.
Prometheus: An open-source monitoring system and time-series database. It scrapes metrics from configured targets (Kubernetes nodes, pods, applications) and stores them.
Alertmanager: A component of the Prometheus ecosystem that handles alerts sent by Prometheus server. It de-duplicates, groups, and routes them to appropriate receivers like email, Slack, or PagerDuty.
Kube-state-metrics: A service that listens to the Kubernetes API server and generates metrics about the state of objects (deployments, pods, nodes, etc.) from the API.
Node Exporter: A Prometheus exporter that exposes a wide variety of hardware and OS metrics (CPU, memory, disk I/O, network stats) from Linux and other Unix-like systems.
PagerDuty: An incident management platform that aggregates alerts from various monitoring tools, applies on-call schedules, and escalates incidents to the right teams.
Terraform: An open-source IaC tool that allows you to define and provision infrastructure using a declarative configuration language. It's the automation backbone of our solution.

Prerequisites

Before you begin, ensure you have the following:

An existing Kubernetes cluster (EKS, GKE, AKS, or on-prem).
kubectl configured to connect to your cluster.
Terraform CLI installed (version 1.0+ recommended).
Helm CLI installed (version 3+ recommended).
A PagerDuty account with an API key (for Terraform) and a Routing Key (for Alertmanager).
Basic understanding of Kubernetes, Prometheus, and PagerDuty concepts.

Terraform Setup: Providers and Variables

We'll use three primary Terraform providers:

kubernetes: To interact with your Kubernetes cluster.
helm: To deploy Prometheus and its components using their official Helm chart.
pagerduty: To programmatically create services and integration keys.

Start by defining your providers and input variables:

provider "kubernetes" {
  # Kubernetes configuration typically picked up from ~/.kube/config
  # or environment variables (KUBE_CONFIG_PATH, KUBE_CONFIG_DATA, etc.)
  # For specific configurations (e.g., EKS, GKE), you might add:
  # host                   = var.kubernetes_host
  # token                  = var.kubernetes_token
  # cluster_ca_certificate = base64decode(var.kubernetes_ca_cert)
}

provider "helm" {
  kubernetes {
    # Helm provider uses the same Kubernetes config
    # host                   = var.kubernetes_host
    # token                  = var.kubernetes_token
    # cluster_ca_certificate = base64decode(var.kubernetes_ca_cert)
  }
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

variable "pagerduty_api_token" {
  description = "Your PagerDuty API token with appropriate permissions."
  type        = string
  sensitive   = true
}

variable "pagerduty_team_id" {
  description = "The ID of the PagerDuty team responsible for this service."
  type        = string
}

variable "pagerduty_escalation_policy_id" {
  description = "The ID of the PagerDuty escalation policy for this service."
  type        = string
}

variable "kubernetes_namespace" {
  description = "The Kubernetes namespace to deploy Prometheus and Alertmanager into."
  type        = string
  default     = "monitoring"
}

variable "pagerduty_routing_key" {
  description = "The PagerDuty integration routing key for Alertmanager to send events."
  type        = string
  sensitive   = true
}

Automating PagerDuty Service Creation

First, let's create the PagerDuty service that will receive alerts. This ensures your service is consistently configured with the correct team and escalation policy.

resource "pagerduty_service" "kubernetes_observability" {
  name                      = "Kubernetes Observability Alerts"
  auto_resolve_timeout_days = 7
  acknowledgement_timeout_minutes = 15
  escalation_policy         = var.pagerduty_escalation_policy_id
  team                      = var.pagerduty_team_id
  description               = "Paging service for critical Kubernetes cluster and application alerts."
}

resource "pagerduty_extension" "kubernetes_observability_extension" {
  name    = "${pagerduty_service.kubernetes_observability.name} Alerts Webhook"
  endpoint = "https://events.pagerduty.com/v2/enqueue" # PagerDuty Events API v2
  type    = "generic_events_api_v2"
  service = pagerduty_service.kubernetes_observability.id
}

Note: While `pagerduty_extension` is shown here for completeness, Alertmanager directly uses a routing key to send events to PagerDuty's Events API. The `pagerduty_service` resource primarily creates the service to attach the routing key to.

Deploying Prometheus with Helm via Terraform

We'll use the official Prometheus community Helm chart. This chart is comprehensive, deploying Prometheus, Alertmanager, Kube-state-metrics, and Node Exporter. The key is to provide a custom `values.yaml` content to configure Alertmanager to send alerts to PagerDuty and define essential Prometheus alert rules.

Prometheus Alertmanager PagerDuty Configuration

The following configuration snippet for Alertmanager within the `values.yaml` integrates with PagerDuty. Replace `YOUR_PAGERDUTY_ROUTING_KEY` with your actual key.

alertmanager:
  enabled: true
  config:
    global:
      resolve_timeout: 5m
    route:
      group_by: ['alertname', 'cluster', 'service']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 1h
      receiver: 'pagerduty'
      routes:
      - match:
          severity: 'critical'
        receiver: 'pagerduty'
      - match:
          severity: 'warning'
        receiver: 'pagerduty'
    receivers:
    - name: 'pagerduty'
      pagerduty_configs:
      - service_key: "{{ .Values.alertmanager.pagerduty.routingKey }}" # Placeholder for Terraform dynamic injection
        url: 'https://events.pagerduty.com/v2/enqueue'
        client: 'Prometheus Alertmanager'
        client_url: 'http://{{ .Values.alertmanager.fullnameOverride }}.{{ .Release.Namespace }}.svc.cluster.local:9093' # Example internal URL
        description: '{{ .CommonLabels.alertname }} ({{ .CommonLabels.instance }})'
        details:
          message: '{{ .CommonLabels.alertname }} on {{ .CommonLabels.instance }} is {{ .Status }}'
          severity: '{{ .CommonLabels.severity | toLower }}'
          summary: '{{ .CommonLabels.alertname }}: {{ .Annotations.summary }}'
          description: '{{ .Annotations.description }}'
          link: '{{ .GeneratorURL }}'

Essential Prometheus Alert Rules

Include common Kubernetes and node health alert rules directly in the Prometheus configuration. These are defined within the `server.statefulset.alerting.rules` section of the `values.yaml`.

server:
  statefulset:
    alerting:
      rules:
        default: true # Enable default rules from the chart
        additional:
        - name: kubernetes-alerts
          rules:
          - alert: KubePodCrashLooping
            expr: sum(kube_pod_container_status_restarts_total) by (namespace, pod, container) > 5 and kube_pod_container_status_running == 0
            for: 5m
            labels:
              severity: critical
            annotations:
              summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) is crashlooping"
              description: "Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) has restarted more than 5 times in 5 minutes."
          - alert: KubeNodeNotReady
            expr: kube_node_status_condition{condition="Ready", status="true"} == 0
            for: 5m
            labels:
              severity: critical
            annotations:
              summary: "Node {{ $labels.node }} is not ready"
              description: "Node {{ $labels.node }} has been in a NotReady state for more than 5 minutes."
          - alert: KubeDeploymentReplicasMismatch
            expr: kube_deployment_spec_replicas != kube_deployment_status_replicas_available
            for: 1m
            labels:
              severity: warning
            annotations:
              summary: "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has replica mismatch"
              description: "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has {{ $value }} unavailable replicas."
          # Add more alerts as needed, e.g., CPU/Memory utilization, Disk usage

Implementing the Terraform Configuration

Now, let's assemble the full Terraform configuration (`main.tf`) that pulls everything together. This will create the Kubernetes namespace, deploy the Prometheus Helm chart with the custom values, and configure PagerDuty.

# main.tf

# Define providers (as shown in "Terraform Setup" section)
provider "kubernetes" {}
provider "helm" { kubernetes {} }
provider "pagerduty" { token = var.pagerduty_api_token }

# Variables (as shown in "Terraform Setup" section)
variable "pagerduty_api_token" { sensitive = true }
variable "pagerduty_team_id" {}
variable "pagerduty_escalation_policy_id" {}
variable "kubernetes_namespace" { default = "monitoring" }
variable "pagerduty_routing_key" { sensitive = true }

# Create Kubernetes Namespace for monitoring components
resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = var.kubernetes_namespace
  }
}

# --- PagerDuty Service Configuration ---
resource "pagerduty_service" "kubernetes_observability" {
  name                      = "Kubernetes Observability Alerts"
  auto_resolve_timeout_days = 7
  acknowledgement_timeout_minutes = 15
  escalation_policy         = var.pagerduty_escalation_policy_id
  team                      = var.pagerduty_team_id
  description               = "Paging service for critical Kubernetes cluster and application alerts."
}

resource "pagerduty_extension" "kubernetes_observability_extension" {
  name    = "${pagerduty_service.kubernetes_observability.name} Alerts Webhook"
  endpoint = "https://events.pagerduty.com/v2/enqueue"
  type    = "generic_events_api_v2"
  service = pagerduty_service.kubernetes_observability.id
}

# --- Helm Chart Deployment for Prometheus Stack ---
resource "helm_release" "prometheus" {
  name       = "kube-prometheus-stack"
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  namespace  = kubernetes_namespace.monitoring.metadata[0].name
  version    = "58.1.0" # Use a specific chart version for stability
  create_namespace = false # The namespace is managed by kubernetes_namespace resource

  values = [
    < 5 and kube_pod_container_status_running == 0
            for: 5m
            labels:
              severity: critical
            annotations:
              summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) is crashlooping"
              description: "Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) has restarted more than 5 times in 5 minutes."
          - alert: KubeNodeNotReady
            expr: kube_node_status_condition{condition="Ready", status="true"} == 0
            for: 5m
            labels:
              severity: critical
            annotations:
              summary: "Node {{ $labels.node }} is not ready"
              description: "Node {{ $labels.node }} has been in a NotReady state for more than 5 minutes."
          - alert: KubeDeploymentReplicasMismatch
            expr: kube_deployment_spec_replicas != kube_deployment_status_replicas_available
            for: 1m
            labels:
              severity: warning
            annotations:
              summary: "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has replica mismatch"
              description: "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has {{ $value }} unavailable replicas."
EOF
  ]
}

# Output relevant information
output "pagerduty_service_url" {
  description = "URL to the created PagerDuty service."
  value       = pagerduty_service.kubernetes_observability.html_url
}

output "prometheus_ingress_host" {
  description = "Prometheus Ingress Host"
  value       = "prometheus.yourdomain.com" # As defined in values
}

output "grafana_ingress_host" {
  description = "Grafana Ingress Host"
  value       = "grafana.yourdomain.com" # As defined in values
}

Important considerations for the `values` block:

Ingress: I've enabled Ingress for Alertmanager, Prometheus, and Grafana. You'll need an Ingress controller (like NGINX Ingress) installed in your cluster and proper DNS records pointing to your cluster's Ingress IP. Replace `yourdomain.com` with your actual domain.
Storage: Adjust `storageClassName` and `storage` requests for Prometheus persistent volumes based on your cluster's capabilities and monitoring needs.
Grafana Password: CRITICAL: Change `your_strong_grafana_password` to a secure, randomly generated password. Consider using a Kubernetes secret for this, managed outside of raw Terraform for production.
Alertmanager Routing Key: The PagerDuty routing key is dynamically injected using `${var.pagerduty_routing_key}`.

Deployment Steps

Follow these steps to deploy your automated observability stack:

Save the code: Save the Terraform configuration in a file named `main.tf` (or similar) in an empty directory.
Initialize Terraform: Open your terminal in the directory where you saved `main.tf` and run:
terraform init
Set Variables: Provide your sensitive PagerDuty API token and routing key, and other required variables. You can do this via environment variables (e.g., `TF_VAR_pagerduty_api_token="YOUR_API_TOKEN"`), a `terraform.tfvars` file (ensure it's not committed to VCS), or by passing them on the command line.
export TF_VAR_pagerduty_api_token="pd_your_api_token" export TF_VAR_pagerduty_team_id="PTxxxxxxx" # Get from PagerDuty console URL (teams/PTxxxxxxx) export TF_VAR_pagerduty_escalation_policy_id="PExxxxxxx" # Get from PagerDuty console URL (escalation_policies/PExxxxxxx) export TF_VAR_pagerduty_routing_key="your_integration_routing_key" # Get from your PagerDuty service integration
Review the plan:
terraform plan
Review the proposed changes carefully.
Apply the configuration:
terraform apply
Type `yes` when prompted to confirm the changes.

Verification

After `terraform apply` completes:

Kubernetes Pods: Check if all Prometheus stack pods are running:
kubectl get pods -n monitoring
You should see pods for Prometheus, Alertmanager, Grafana, Kube-state-metrics, and Node Exporter in a `Running` state.
Prometheus UI: Access the Prometheus UI via the configured Ingress host (e.g., `prometheus.yourdomain.com`). Verify targets are being scraped.
Alertmanager UI: Access the Alertmanager UI (e.g., `alertmanager.yourdomain.com`). Check the configuration and active alerts.
PagerDuty Service: Log into your PagerDuty account. You should see the newly created service under the specified team.
Test an Alert: Intentionally trigger an alert (e.g., scale down a critical deployment to zero replicas, which should trigger `KubeDeploymentReplicasMismatch`). Verify that an incident is created in PagerDuty.

Troubleshooting and Best Practices

Common Issues:

Helm Release Failure: Check `kubectl get events -n monitoring` for issues during Helm chart deployment. Examine the pod logs for specific errors.
PagerDuty Integration: Ensure `pagerduty_api_token` has sufficient permissions. Double-check `pagerduty_routing_key` accuracy. If alerts aren't firing, check Alertmanager logs for connectivity issues or configuration parsing errors.
Prometheus Scrape Issues: In the Prometheus UI, navigate to "Status" -> "Targets" to verify all expected targets are healthy and being scraped.
Ingress Not Working: Confirm your Ingress controller is running. Check Ingress resources: `kubectl get ingress -n monitoring`. Verify DNS records point to your Ingress controller's external IP/hostname.

Best Practices:

Version Pinning: Always pin your Helm chart versions and Terraform provider versions (`version = "X.Y.Z"`) to ensure reproducible deployments.
Separate Environments: Use different Terraform workspaces or separate configurations for development, staging, and production environments.
Secrets Management: For production, avoid passing sensitive values like `pagerduty_api_token`, `pagerduty_routing_key`, and `grafana_admin_password` directly as CLI variables. Utilize a dedicated secrets management solution (e.g., HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, Google Secret Manager) and integrate with Terraform.
Custom Alert Rules: Develop application-specific alert rules based on service-level objectives (SLOs) and service-level indicators (SLIs) for comprehensive monitoring.
Role-Based Access Control (RBAC): Implement strict RBAC for both Kubernetes (for Prometheus components) and PagerDuty (for user access).
Testing Alerts: Regularly test your alert configurations to ensure they trigger correctly and route to PagerDuty as expected.

Conclusion

Automating Kubernetes observability with Terraform, Prometheus, and PagerDuty provides a powerful, scalable, and reliable foundation for managing the health of your cloud-native applications. By adopting an Infrastructure as Code approach, you gain consistency, reduce operational overhead, and significantly enhance your team's ability to detect, diagnose, and resolve incidents swiftly. This guide offers a robust starting point, enabling you to build upon this configuration with more advanced metrics, custom alert rules, and sophisticated PagerDuty escalation policies tailored to your specific organizational needs.

CloudStack YAML