Terraform for Deploying Prometheus and PagerDuty Alerting on AWS EKS

In the dynamic landscape of cloud-native applications, robust monitoring and efficient incident response are paramount. This comprehensive guide details how to leverage Terraform for automating the deployment of Prometheus for metrics collection and PagerDuty for incident alerting on Amazon Elastic Kubernetes Service (EKS). By integrating these powerful tools, organizations can achieve high observability and significantly reduce mean time to resolution (MTTR) for critical issues, ensuring application reliability and operational excellence.

Architecture Pro-Tip: Modular Infrastructure Design

For complex cloud environments, always aim for a modular Terraform project structure. Separate your EKS cluster definition, monitoring components, and application deployments into distinct modules. This approach enhances reusability, simplifies management, and allows different teams to own specific parts of the infrastructure without interfering with others. Consider using Terraform workspaces for managing different environments (dev, staging, prod) within the same configuration.

Why Terraform, Prometheus, and PagerDuty on EKS?

Terraform for Infrastructure as Code (IaC): Automates the provisioning and management of cloud resources, ensuring consistency, version control, and auditability for your EKS cluster and its deployed services.
AWS EKS for Managed Kubernetes: Provides a highly available and scalable Kubernetes control plane, simplifying cluster operations and allowing focus on application development.
Prometheus for Observability: An open-source monitoring system for collecting metrics from your Kubernetes cluster, applications, and infrastructure. Its powerful querying language (PromQL) and alert definitions are key for proactive issue detection.
PagerDuty for Incident Management: A leading incident response platform that ensures the right people are notified at the right time. Integrating Prometheus Alertmanager with PagerDuty streamlines alert routing, on-call scheduling, and escalation policies.

Prerequisites

Before you begin, ensure you have the following tools and access configured:

AWS Account: With appropriate permissions to create EKS clusters, IAM roles, and other AWS resources.
AWS CLI: Configured with your credentials.
Terraform CLI: Version 1.0 or higher installed.
kubectl: Command-line tool for interacting with Kubernetes clusters.
Helm CLI: The package manager for Kubernetes.
Existing AWS EKS Cluster: This guide assumes you have an EKS cluster already running. If not, you can provision one using Terraform (e.g., via the terraform-aws-modules/eks/aws module).
PagerDuty Service: Create a new service in PagerDuty and obtain an "Events API (V2) Integration Key" for it.

Core Concepts and Architecture

Our setup involves the following components working in tandem:

EKS Cluster: The foundation hosting our monitoring stack.
Prometheus (via kube-prometheus-stack Helm chart): Deploys Prometheus, Alertmanager, Grafana, and related Kubernetes exporters (kube-state-metrics, node-exporter) into your cluster. Prometheus scrapes metrics.
Alertmanager: Receives alerts from Prometheus, deduplicates, groups, and routes them to notification receivers. Here, PagerDuty is our primary receiver.
PagerDuty: Ingests alerts from Alertmanager, triggers incidents, and manages on-call rotations and escalation policies.

Terraform Project Structure

A recommended directory structure for your Terraform project:

main.tf: Main configuration for providers, EKS data sources, and resource definitions.
variables.tf: Input variables for customization (e.g., EKS cluster name, PagerDuty key).
outputs.tf: Output values from the deployment (e.g., Kubernetes namespace).
versions.tf: Specifies Terraform and provider versions.

Step-by-Step Deployment Guide

1. Configure Terraform Providers and EKS Access

First, define your AWS, Kubernetes, and Helm providers. The Kubernetes and Helm providers need to be configured to connect to your EKS cluster. This typically involves using data sources to retrieve EKS cluster details and authentication tokens.

2. Deploy `kube-prometheus-stack` using Helm

We will use the helm_release resource to deploy the popular kube-prometheus-stack chart. This chart bundles Prometheus, Grafana, Alertmanager, and various exporters, providing a comprehensive monitoring solution out-of-the-box. We'll enable Alertmanager and configure it for PagerDuty.

3. Configure Alertmanager for PagerDuty Integration

The kube-prometheus-stack chart allows overriding Alertmanager's configuration via a custom YAML embedded in the values attribute of the helm_release resource. We will define a PagerDuty receiver and a routing rule for alerts.

4. Define PagerDuty Alerting Rules in Prometheus

While the kube-prometheus-stack provides many default rules, you can define custom PrometheusRule Kubernetes resources (or provide additional rule files via the Helm chart values) to trigger specific PagerDuty alerts based on your application's needs.

Ready-to-Use Terraform Configuration

Here's a complete Terraform configuration that provisions Prometheus and Alertmanager with PagerDuty integration on an existing EKS cluster. Ensure you replace placeholders like YOUR_EKS_CLUSTER_NAME and YOUR_PAGERDUTY_INTEGRATION_KEY.

resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = "monitoring"
  }
}

resource "helm_release" "kube_prometheus_stack" {
  name       = "kube-prometheus-stack"
  namespace  = kubernetes_namespace.monitoring.metadata[0].name
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  version    = "58.2.0" # Use a compatible chart version

  set {
    name  = "kube-state-metrics.fullnameOverride"
    value = "kube-state-metrics"
  }
  set {
    name  = "prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues"
    value = "false"
  }
  set {
    name  = "prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues"
    value = "false"
  }
  set {
    name  = "prometheus.prometheusSpec.ruleSelectorNilUsesHelmValues"
    value = "false"
  }
  set {
    name  = "grafana.enabled"
    value = "true"
  }

  values = [
    yamlencode({
      alertmanager = {
        enabled = true
        config = {
          # This config is passed directly to Alertmanager
          # Documentation: https://prometheus.io/docs/alerting/latest/configuration/
          global = {
            resolve_timeout = "5m"
          }
          route = {
            group_by        = ["alertname", "cluster", "service"]
            group_wait      = "30s"
            group_interval  = "5m"
            repeat_interval = "4h"
            receiver        = "pagerduty-receiver"
            routes = [
              {
                match = {
                  severity = "critical"
                }
                receiver = "pagerduty-receiver"
              }
              {
                match = {
                  severity = "warning"
                }
                receiver = "pagerduty-receiver"
                # Optional: Send warnings to a different PagerDuty service or delay
                # group_wait = "1m"
              }
            ]
          }
          receivers = [
            {
              name = "pagerduty-receiver"
              pagerduty_configs = [
                {
                  service_key = var.pagerduty_integration_key
                  # Optional: Set a custom client name
                  # client = "Prometheus-EKS-Monitoring"
                  # Optional: Severity mapping for PagerDuty
                  # severity = "error" # Defaults to "critical" for critical alerts, "warning" for warning alerts
                }
              ]
            }
          ]
          # Ensure a default catch-all route to prevent alerts from being dropped if no specific route matches
          # A default route is typically handled by the main 'route' block if no 'routes' are specified within it.
          # Here we ensure it by setting a primary receiver.
        }
        # Mount the configuration as a secret
        secrets = [
          {
            name = "alertmanager-pagerduty-secret"
            key  = "alertmanager.yaml"
            data = <Explanation of the Code:
kubernetes_namespace.monitoring: Creates a dedicated Kubernetes namespace for your monitoring stack, promoting isolation and organization.
helm_release.kube_prometheus_stack: This is the core resource. It deploys the kube-prometheus-stack Helm chart.
            name, namespace, repository, chart, version: Standard Helm release parameters.
set blocks: Override specific default values of the Helm chart, ensuring service monitor discovery works correctly within the cluster.
values = [yamlencode({...})]: This crucial block provides custom YAML configuration to the Helm chart.
                    alertmanager.enabled = true: Ensures Alertmanager is deployed.
alertmanager.config: Defines the Alertmanager configuration.
                            global.resolve_timeout: Sets how long Alertmanager waits for a resolving alert before sending a resolution notification.
route: Defines the primary routing tree for alerts. Alerts are grouped, waited, and repeated based on these settings.
receivers: Defines different notification channels. Our pagerduty-receiver uses pagerduty_configs.
service_key = var.pagerduty_integration_key: Your PagerDuty Events API V2 integration key, retrieved from a Terraform variable for security and flexibility.
The raw YAML embedded via the secrets block is the actual Alertmanager configuration that will be mounted, allowing for more complex configurations than direct config mapping.

                        
prometheus.prometheusSpec.additionalPrometheusRules (commented): An example of how to add custom Prometheus recording and alerting rules directly within your Terraform. These rules determine what conditions trigger an alert.

                

        
Configuration for variables.tf and versions.tfCreate these files in the same directory as main.tf:
variables.tf
variable "aws_region" {
  description = "AWS region"
  type        = string
  default     = "us-east-1"
}

variable "eks_cluster_name" {
  description = "The name of your existing EKS cluster"
  type        = string
}

variable "pagerduty_integration_key" {
  description = "PagerDuty Events API V2 Integration Key for the monitoring service"
  type        = string
  sensitive   = true
}
    
versions.tf
terraform {
  required_version = ">= 1.0.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.20"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.10"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

# Configure the Kubernetes provider to connect to the EKS cluster
# This dynamically fetches credentials for the EKS cluster
data "aws_eks_cluster" "cluster" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = var.eks_cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

# Configure the Helm provider to use the Kubernetes provider context
provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}
    
Deployment StepsFollow these commands to deploy your monitoring stack:
Initialize Terraform:
terraform init
Review the Plan:
terraform plan -var="eks_cluster_name=your-eks-cluster-name" -var="pagerduty_integration_key=your-pagerduty-key"
Apply the Configuration:
terraform apply -var="eks_cluster_name=your-eks-cluster-name" -var="pagerduty_integration_key=your-pagerduty-key"
After applying, Prometheus, Alertmanager, and Grafana will be deployed in the monitoring namespace of your EKS cluster.
Verification and TestingCheck Kubernetes Pods: Verify all monitoring pods are running:
kubectl get pods -n monitoring
Access Grafana: Port-forward Grafana to access its UI (default credentials: admin/prom-operator). Configure Prometheus as a data source and explore dashboards.
kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80
Access Prometheus UI:
kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090
Test Alerting: Trigger a test alert. You can create a simple PrometheusRule resource with an always-true expression to fire an alert immediately and verify it reaches PagerDuty.
Advanced Considerations and Best PracticesPersistent Storage: For production environments, configure persistent volumes for Prometheus and Grafana to ensure data is not lost across pod restarts. The Helm chart supports this.
Security: Implement strict network policies to control access to your monitoring stack. Use AWS IAM Roles for Service Accounts (IRSA) for fine-grained permissions for Prometheus and Alertmanager to access AWS resources if needed.
High Availability: Run multiple replicas of Prometheus and Alertmanager for redundancy, especially in critical production setups.
External Access to Grafana: For secure external access, consider placing Grafana behind an AWS Application Load Balancer (ALB) or NGINX Ingress Controller with TLS.
Thanos Integration: For long-term metrics storage and global query views across multiple clusters, consider integrating Thanos with Prometheus.
Custom Alert Rules: Develop custom Prometheus alert rules specific to your applications' SLOs (Service Level Objectives) and SLAs (Service Level Agreements).
PagerDuty Service Configuration: Tailor your PagerDuty service's escalation policies, on-call schedules, and suppression rules to match your operational requirements.


Troubleshooting and FAQ1. Alerts not appearing in PagerDuty:Ensure your pagerduty_integration_key is correct. Check Alertmanager logs (kubectl logs -n monitoring -l app.kubernetes.io/name=alertmanager) for errors sending notifications. Verify your Alertmanager configuration using the Alertmanager UI.
2. Prometheus not scraping metrics:Check Prometheus targets in its UI (http://localhost:9090/targets if port-forwarded). Ensure your services have correct annotations for Prometheus to discover them, or that ServiceMonitors/PodMonitors are correctly configured and pointing to your services.
3. Connectivity issues to EKS:Verify your kubectl context is correct (aws eks update-kubeconfig --name your-eks-cluster-name --region your-aws-region). Ensure the IAM user/role running Terraform has permissions to access the EKS cluster and its authentication token.
4. Helm chart deployment failures:Examine the Terraform apply output for errors. Check for resource conflicts or insufficient permissions within the EKS cluster for the Helm chart to create necessary resources.
ConclusionBy following this guide, you have successfully automated the deployment of a robust monitoring and alerting solution for your AWS EKS cluster using Terraform, Prometheus, and PagerDuty. This integration empowers your DevOps teams with unparalleled visibility into your Kubernetes workloads and a reliable, automated incident response workflow. Embracing IaC for your observability stack ensures consistency, scalability, and maintainability, paving the way for more resilient and efficient cloud-native operations.

CloudStack YAML