Prometheus Monitoring on AWS EKS with PagerDuty Alerting via Terraform: A Comprehensive Guide

In the dynamic landscape of cloud-native applications, robust monitoring and effective incident response are paramount. This guide provides a detailed, technical walkthrough on setting up a comprehensive monitoring solution for AWS EKS (Elastic Kubernetes Service) using Prometheus, integrating with PagerDuty for incident management, all provisioned and managed declaratively with Terraform. This architecture ensures high availability, scalability, and automated operational readiness for your Kubernetes workloads.

Architecture Pro-Tip: Federated Monitoring for Multi-EKS Deployments

For organizations managing multiple EKS clusters across different AWS accounts or regions, consider implementing a federated Prometheus setup. Deploy a dedicated Prometheus instance (or Thanos/Cortex for long-term storage and global view) in each cluster, scraping local metrics. Then, deploy a global Prometheus (or Thanos Query) that scrapes or queries these local instances, providing a centralized monitoring plane without sacrificing local autonomy or increasing cross-region data transfer costs unnecessarily. This approach simplifies alert routing and overall observability management.

Why This Stack? Prometheus, EKS, PagerDuty, Terraform

Each component in this stack plays a critical role in building a resilient and observable cloud-native environment:

Prometheus: An open-source monitoring system designed for reliability and scalability, ideal for collecting metrics from containerized workloads in Kubernetes. Its powerful query language (PromQL) and flexible alerting rules make it a DevOps staple.
AWS EKS: Amazon's managed Kubernetes service, simplifying the deployment, management, and scaling of containerized applications. EKS handles the Kubernetes control plane's heavy lifting, allowing teams to focus on application development.
PagerDuty: A leading incident management platform that consolidates alerts from various monitoring tools, intelligently routes them to the right teams, and facilitates faster incident resolution with on-call scheduling, escalations, and post-incident analysis.
Terraform: An Infrastructure as Code (IaC) tool that enables you to define and provision infrastructure using a high-level configuration language. Terraform ensures idempotent deployments, version control, and consistent environments, from cloud resources to application configurations.

Prerequisites

Before you begin, ensure you have the following tools and access configured:

An active AWS account with administrative access.
AWS CLI configured with appropriate credentials.
Kubectl installed and configured to connect to your EKS cluster.
Helm 3 installed.
Terraform CLI installed (v1.0+ recommended).
A PagerDuty account with permissions to create services and integrations.

Step-by-Step Implementation via Terraform

1. Terraform Setup and AWS EKS Cluster Provisioning (If Not Existing)

Start by defining your AWS provider and, if necessary, provisioning your EKS cluster. For brevity, we assume an existing EKS cluster and focus on the monitoring stack. If you need to provision EKS, use a module like terraform-aws-modules/eks/aws.

Create a main.tf, variables.tf, and outputs.tf in a new directory.

variable "aws_region" {
  description = "AWS region for deployments"
  type        = string
  default     = "us-east-1"
}

variable "eks_cluster_name" {
  description = "Name of the existing EKS cluster"
  type        = string
}

variable "pagerduty_api_token" {
  description = "PagerDuty API token"
  type        = string
  sensitive   = true
}

variable "pagerduty_team_id" {
  description = "PagerDuty Team ID to associate the service with"
  type        = string
}

variable "pagerduty_escalation_policy_id" {
  description = "PagerDuty Escalation Policy ID for the service"
  type        = string
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
    pagerduty = {
      source  = "PagerDuty/pagerduty"
      version = "~> 2.0"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

data "aws_eks_cluster" "cluster" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = var.eks_cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

2. Provision PagerDuty Service and Integration

We'll use Terraform to create a dedicated PagerDuty service for EKS alerts and an HTTP integration for Alertmanager.

resource "pagerduty_service" "eks_monitoring" {
  name                      = "${var.eks_cluster_name}-monitoring"
  auto_resolve_timeout      = 14400 # 4 hours
  acknowledgement_timeout   = 600   # 10 minutes
  escalation_policy         = var.pagerduty_escalation_policy_id
  description               = "PagerDuty service for EKS cluster monitoring alerts from Prometheus/Alertmanager."
  team                      = var.pagerduty_team_id
}

resource "pagerduty_service_integration" "alertmanager_integration" {
  name        = "Prometheus Alertmanager"
  type        = "generic_events_api_inbound_integration"
  service     = pagerduty_service.eks_monitoring.id
  vendor_type = "Prometheus" # Or generic_events_api for custom
}

output "pagerduty_integration_key" {
  description = "The PagerDuty integration key for Alertmanager."
  value       = pagerduty_service_integration.alertmanager_integration.integration_key
  sensitive   = true
}

3. Install Prometheus Stack on EKS using Helm

The kube-prometheus-stack Helm chart is a comprehensive solution, bundling Prometheus, Alertmanager, Grafana, and default dashboards/rules.

First, ensure the namespace exists:

resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = "monitoring"
  }
}

Now, the Helm chart. We'll inject the PagerDuty integration key into Alertmanager's configuration.

4. Terraform Configuration for Prometheus and Alertmanager

resource "helm_release" "kube_prometheus_stack" {
  name       = "prometheus-stack"
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  namespace  = kubernetes_namespace.monitoring.metadata[0].name
  version    = "56.6.0" # Use a specific chart version
  timeout    = 600

  values = [
    yamlencode({
      alertmanager = {
        enabled = true
        config = {
          global = {
            resolve_timeout = "5m"
          }
          route = {
            group_by        = ["alertname", "cluster", "namespace"]
            group_wait      = "30s"
            group_interval  = "5m"
            repeat_interval = "4h"
            receiver        = "pagerduty"
          }
          receivers = [
            {
              name = "pagerduty"
              pagerduty_configs = [
                {
                  service_key = "${pagerduty_service_integration.alertmanager_integration.integration_key}"
                  # You can add severity mapping here based on Prometheus labels
                  # severity = "{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toLower }}{{ else }}critical{{ end }}"
                }
              ]
            }
          ]
        }
        # Optionally expose Alertmanager UI
        ingress = {
          enabled = true
          hosts   = ["alertmanager.${var.eks_cluster_name}.yourdomain.com"] # Update with your domain
          annotations = {
            "kubernetes.io/ingress.class": "nginx" # Or your specific ingress controller
          }
        }
      }
      prometheus = {
        prometheusSpec = {
          retention = "15d" # Adjust retention based on your needs
          # Optionally expose Prometheus UI
          ingress = {
            enabled = true
            hosts   = ["prometheus.${var.eks_cluster_name}.yourdomain.com"] # Update with your domain
            annotations = {
              "kubernetes.io/ingress.class": "nginx"
            }
          }
          # If using EKS, ensure service accounts have proper IAM roles for scraping AWS services if needed
          # serviceAccount = {
          #   create = true
          #   name   = "prometheus-kube-prometheus-stack-prometheus"
          #   annotations = {
          #     "eks.amazonaws.com/role-arn" = aws_iam_role.prometheus_irsa_role.arn # Example IRSA
          #   }
          # }
        }
      }
      grafana = {
        enabled = true
        adminPassword = "your-strong-grafana-password" # Change this!
        ingress = {
          enabled = true
          hosts   = ["grafana.${var.eks_cluster_name}.yourdomain.com"] # Update with your domain
          annotations = {
            "kubernetes.io/ingress.class": "nginx"
          }
        }
      }
      kubeApiServer = {
        enabled = true
      }
      kubelet = {
        enabled = true
      }
      kubeControllerManager = {
        enabled = true
      }
      kubeEtcd = {
        enabled = true
      }
      kubeScheduler = {
        enabled = true
      }
      kubeProxy = {
        enabled = true
      }
      kubeStateMetrics = {
        enabled = true
      }
      nodeExporter = {
        enabled = true
      }
    })
  ]
}

Replace your-strong-grafana-password and yourdomain.com placeholders. For production, consider using AWS Secrets Manager to store sensitive information like Grafana passwords and retrieve them via Terraform or directly in Kubernetes.

5. Deploy Custom Prometheus Rules (Optional but Recommended)

While kube-prometheus-stack provides many default rules, you'll likely want to define custom alerts specific to your applications or cluster health. You can manage these using kubernetes_manifest or by adding them to the Helm chart values.

Example: Alerting on CPU utilization of a specific deployment.

resource "kubernetes_manifest" "example_cpu_alert_rule" {
  provider = kubernetes
  yaml_body = < 80
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "High CPU usage on {{ $labels.pod }} in default namespace"
        description: "{{ $labels.pod }} in default namespace has been utilizing > 80% of its CPU for 5 minutes."
YAML
}

Applying Your Configuration

Once your Terraform files are ready, initialize and apply the configuration:

terraform init
terraform plan (Review the changes carefully)
terraform apply

You will be prompted for your variables (eks_cluster_name, pagerduty_api_token, pagerduty_team_id, pagerduty_escalation_policy_id). For sensitive values like the API token, it's best to use environment variables (TF_VAR_pagerduty_api_token=...) or a .tfvars file with appropriate security measures.

Verification and Testing

After applying the Terraform configuration:

Kubernetes Pods: Verify all Prometheus stack pods are running in the monitoring namespace: kubectl get pods -n monitoring.
Alertmanager UI: Access the Alertmanager UI (via the configured ingress or port-forwarding) to confirm the PagerDuty receiver is configured correctly.
Prometheus UI: Access the Prometheus UI to ensure targets are being scraped and rules are loaded.
PagerDuty Service: Log into PagerDuty and confirm the new service and integration are active.
Test Alert: Create a temporary Prometheus rule that will always fire (e.g., vector(1) > 0) with a short for duration (e.g., 1s) to trigger a test incident in PagerDuty. Observe the alert flow and ensure the PagerDuty incident is created and resolved correctly.

Troubleshooting and Best Practices

Common Issues:

Incorrect PagerDuty Integration Key: Double-check the integration key used in Alertmanager configuration. It's a common source of failed alerts.
Firewall/Security Group Issues: Ensure your EKS cluster can reach PagerDuty's API endpoints (typically outbound HTTPS on port 443).
Prometheus Scrape Configuration: Verify that Prometheus is correctly discovering and scraping targets within your EKS cluster. Check the "Status -> Targets" page in the Prometheus UI.
PrometheusRule Syntax Errors: Invalid PromQL or YAML syntax in your PrometheusRule objects can prevent alerts from firing. Check Prometheus logs and the UI for errors.
Helm Chart Version Drift: Always specify a chart version (`version` attribute in `helm_release`) to ensure consistent deployments.

Best Practices:

Version Control: Store all your Terraform code in a Git repository.
Modularize Terraform: Break down your Terraform configuration into smaller, reusable modules (e.g., a module for EKS, one for monitoring, etc.).
State Management: Use a remote backend (like an S3 bucket with DynamoDB locking) for your Terraform state to enable collaboration and prevent state corruption.
Fine-tune Alerts: Start with critical alerts and progressively add more granular ones. Avoid alert fatigue by ensuring each alert is actionable.
Dashboarding with Grafana: Leverage Grafana (included in kube-prometheus-stack) to visualize your metrics and complement your alerting strategy. Import dashboards from the community or create custom ones.
Security: Implement OIDC for Kubernetes Service Account (KSA) roles (IRSA) for Prometheus to access AWS services securely, and restrict access to monitoring UIs.

Conclusion

By leveraging Prometheus for metric collection, Alertmanager for intelligent routing, and PagerDuty for incident management, all provisioned declaratively with Terraform, you establish a robust and automated observability framework for your AWS EKS environment. This setup empowers your DevOps teams with the insights and tools necessary to maintain application health, respond swiftly to issues, and ultimately deliver a superior user experience. Continuously refine your monitoring rules and incident response playbooks to adapt to the evolving needs of your cloud-native applications.

CloudStack YAML