Terraform to Deploy Prometheus on AWS EKS with PagerDuty Alerting Integration

Architecture Pro-Tip:

For robust production environments, consider deploying Prometheus in a high-availability configuration across multiple availability zones. Utilize AWS EBS volumes for persistent storage, snapshotting them regularly, or explore solutions like Thanos for long-term storage and global query views. Always configure dedicated IAM roles for your EKS nodes with least privilege access, and ensure network policies are in place to restrict access to your monitoring endpoints.

Terraform to Deploy Prometheus on AWS EKS with PagerDuty Alerting Integration

In the dynamic landscape of cloud-native applications, robust monitoring and incident management are paramount. This comprehensive guide details how to leverage Terraform Infrastructure as Code (IaC) to deploy Prometheus, the industry-standard monitoring solution, onto an AWS Elastic Kubernetes Service (EKS) cluster. Furthermore, we'll integrate Alertmanager with PagerDuty to ensure critical alerts are efficiently routed to your on-call teams, streamlining incident response and maintaining high service availability.

Why Terraform, EKS, Prometheus, and PagerDuty?

Terraform: Provides declarative configuration for infrastructure, enabling repeatable, version-controlled, and auditable deployments across AWS services.
AWS EKS: A fully managed Kubernetes service that simplifies the deployment, management, and scaling of containerized applications without needing to provision or maintain the Kubernetes control plane.
Prometheus: An open-source monitoring system with a powerful data model, flexible query language (PromQL), and robust alerting capabilities. It's the de-facto standard for Kubernetes monitoring.
PagerDuty: A leading incident management platform that consolidates alerts from various sources, applies intelligent routing, and facilitates structured incident response processes.

Prerequisites

Before you begin, ensure you have the following:

An active AWS Account with necessary permissions to create EKS clusters, VPCs, IAM roles, and EC2 instances.
AWS CLI configured with appropriate credentials.
Terraform CLI (version 1.0.0 or higher) installed.
kubectl CLI installed and configured to interact with Kubernetes clusters.
Helm CLI (version 3.x or higher) installed.
A PagerDuty account with permissions to create new services and retrieve integration keys.

Core Concepts and Components

AWS EKS Cluster Setup

Our Terraform configuration will provision a dedicated VPC, subnets, and security groups to host the EKS cluster. It will then define the EKS control plane and associated managed node groups (EC2 instances) where your applications and Prometheus will run.

Prometheus & Alertmanager Deployment with Helm

We will use the kube-prometheus-stack Helm chart, which bundles Prometheus, Alertmanager, Grafana, and various Kubernetes exporters. This chart simplifies the deployment and configuration of a comprehensive monitoring solution.

Prometheus: Scrapes metrics from defined targets (e.g., Kubernetes nodes, pods, services).
Alertmanager: Receives alerts from Prometheus, groups them, deduplicates, and routes them to notification receivers (like PagerDuty).
Grafana: Provides powerful visualization and dashboarding for your Prometheus metrics.

PagerDuty Integration

PagerDuty acts as the central hub for incident management. We'll configure Alertmanager to send critical alerts to a PagerDuty service using a secure integration key. This ensures that alerts trigger incidents, notify on-call teams, and facilitate tracking and resolution.

Step-by-Step Deployment Guide

Step 1: Set up PagerDuty Service

First, create a new service in PagerDuty that will receive alerts from Alertmanager. Navigate to Services > Service Directory > New Service. Give it a descriptive name (e.g., "EKS Prometheus Alerts"). For the integration type, select "Events API v2". Once the service is created, copy the Integration Key. You will need this for Alertmanager configuration.

Step 2: Prepare Terraform Configuration Files

Create a project directory, e.g., terraform-eks-prometheus-pagerduty, and within it, create the following files:

main.tf: Contains the core infrastructure definitions.
variables.tf: Defines input variables.
outputs.tf: Defines output values.
versions.tf: Specifies provider versions.

Step 3: Define Provider Configuration (versions.tf)

This file ensures you're using compatible versions of the Terraform providers.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

Step 4: Define Variables (variables.tf)

Customize these variables for your environment.

variable "aws_region" {
  description = "AWS region for deployment"
  type        = string
  default     = "us-east-1"
}

variable "cluster_name" {
  description = "Name of the EKS cluster"
  type        = string
  default     = "prometheus-eks-cluster"
}

variable "node_instance_type" {
  description = "EC2 instance type for EKS worker nodes"
  type        = string
  default     = "t3.medium"
}

variable "node_group_desired_size" {
  description = "Desired number of worker nodes in the EKS node group"
  type        = number
  default     = 2
}

variable "pagerduty_integration_key" {
  description = "PagerDuty Events API v2 Integration Key"
  type        = string
  sensitive   = true
}

Step 5: Main Configuration (main.tf)

This is the core of your Terraform deployment. It sets up the EKS cluster, node groups, and then deploys Prometheus and Alertmanager with PagerDuty integration.

Complete Terraform Configuration (main.tf)

# Create VPC for EKS
resource "aws_vpc" "eks_vpc" {
  cidr_block = "10.0.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name = "${var.cluster_name}-vpc"
  }
}

# Create Public Subnets
resource "aws_subnet" "public" {
  count             = 2 # Deploying in 2 availability zones
  vpc_id            = aws_vpc.eks_vpc.id
  cidr_block        = cidrsubnet(aws_vpc.eks_vpc.cidr_block, 8, count.index)
  availability_zone = data.aws_availability_zones.available.names[count.index]
  map_public_ip_on_launch = true

  tags = {
    Name = "${var.cluster_name}-public-subnet-${count.index}"
    "kubernetes.io/cluster/${var.cluster_name}" = "owned"
    "kubernetes.io/role/elb"                  = "1"
  }
}

# Create Internet Gateway
resource "aws_internet_gateway" "gw" {
  vpc_id = aws_vpc.eks_vpc.id

  tags = {
    Name = "${var.cluster_name}-igw"
  }
}

# Create Route Table for Public Subnets
resource "aws_route_table" "public_rt" {
  vpc_id = aws_vpc.eks_vpc.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.gw.id
  }

  tags = {
    Name = "${var.cluster_name}-public-rt"
  }
}

# Associate Route Table with Public Subnets
resource "aws_route_table_association" "public_rt_assoc" {
  count          = length(aws_subnet.public)
  subnet_id      = aws_subnet.public[count.index].id
  route_table_id = aws_route_table.public_rt.id
}

# Get list of availability zones
data "aws_availability_zones" "available" {}

# IAM Role for EKS Cluster
resource "aws_iam_role" "eks_cluster_role" {
  name = "${var.cluster_name}-cluster-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect    = "Allow"
        Principal = {
          Service = "eks.amazonaws.com"
        }
        Action    = "sts:AssumeRole"
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eks_cluster_role.name
}

resource "aws_iam_role_policy_attachment" "eks_vpc_resource_controller" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSVPCResourceController"
  role       = aws_iam_role.eks_cluster_role.name
}


# EKS Cluster
resource "aws_eks_cluster" "main" {
  name     = var.cluster_name
  role_arn = aws_iam_role.eks_cluster_role.arn
  version  = "1.27" # Specify your desired Kubernetes version

  vpc_config {
    subnet_ids = [for s in aws_subnet.public : s.id]
    security_group_ids = [aws_security_group.eks_cluster_sg.id]
    endpoint_private_access = false
    endpoint_public_access  = true
  }

  depends_on = [
    aws_iam_role_policy_attachment.eks_cluster_policy,
    aws_iam_role_policy_attachment.eks_vpc_resource_controller
  ]

  tags = {
    Name = var.cluster_name
  }
}

# EKS Cluster Security Group
resource "aws_security_group" "eks_cluster_sg" {
  name        = "${var.cluster_name}-cluster-sg"
  description = "Security group for EKS cluster"
  vpc_id      = aws_vpc.eks_vpc.id

  ingress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "${var.cluster_name}-cluster-sg"
  }
}


# IAM Role for EKS Worker Nodes
resource "aws_iam_role" "eks_node_role" {
  name = "${var.cluster_name}-node-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect    = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
        Action    = "sts:AssumeRole"
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_node_worker_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.eks_node_role.name
}

resource "aws_iam_role_policy_attachment" "eks_node_cni_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.eks_node_role.name
}

resource "aws_iam_role_policy_attachment" "eks_node_container_registry_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.eks_node_role.name
}

# EKS Node Group
resource "aws_eks_node_group" "main" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "${var.cluster_name}-nodes"
  node_role_arn   = aws_iam_role.eks_node_role.arn
  subnet_ids      = [for s in aws_subnet.public : s.id]
  instance_types  = [var.node_instance_type]

  scaling_config {
    desired_size = var.node_group_desired_size
    max_size     = 3
    min_size     = 1
  }

  depends_on = [
    aws_iam_role_policy_attachment.eks_node_worker_policy,
    aws_iam_role_policy_attachment.eks_node_cni_policy,
    aws_iam_role_policy_attachment.eks_node_container_registry_policy,
  ]

  labels = {
    "eks.amazonaws.com/nodegroup-type" = "general"
  }

  tags = {
    "eks.amazonaws.com/cluster-name" = aws_eks_cluster.main.name
  }
}

# Configure Kubernetes provider to connect to EKS
data "aws_eks_cluster" "cluster" {
  name = aws_eks_cluster.main.name
}

data "aws_eks_cluster_auth" "cluster_auth" {
  name = aws_eks_cluster.main.name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.cluster_auth.token
}

# Add Helm repository for Prometheus-community
resource "helm_repository" "prometheus_community" {
  name = "prometheus-community"
  url  = "https://prometheus-community.github.io/helm-charts"
}

# Deploy kube-prometheus-stack via Helm
resource "helm_release" "kube_prometheus_stack" {
  name       = "kube-prometheus-stack"
  repository = helm_repository.prometheus_community.name
  chart      = "kube-prometheus-stack"
  namespace  = "monitoring"
  create_namespace = true
  version    = "50.1.0" # Use a stable chart version

  values = [
    <<-EOF
      alertmanager:
        enabled: true
        alertmanagerSpec:
          replicas: 1
          storage:
            volumeClaimTemplate:
              spec:
                storageClassName: gp2 # Or your preferred storage class
                resources:
                  requests:
                    storage: 1Gi
          config:
            global:
              resolve_timeout: 5m
            route:
              group_by: ['alertname', 'cluster', 'service']
              group_wait: 30s
              group_interval: 5m
              repeat_interval: 1h
              receiver: 'pagerduty'
            receivers:
            - name: 'pagerduty'
              pagerduty_configs:
              - service_key: "${var.pagerduty_integration_key}"
                routing_key: "${var.pagerduty_integration_key}"
                severity: "critical"
                client: "Prometheus Alertmanager"
                client_url: "http://localhost:9093" # Replace if Alertmanager UI is exposed
      
      prometheus:
        enabled: true
        prometheusSpec:
          replicas: 1
          retention: 15d
          storageSpec:
            volumeClaimTemplate:
              spec:
                storageClassName: gp2 # Or your preferred storage class
                resources:
                  requests:
                    storage: 10Gi
          ruleSelectorNilUsesHelmValues: false
          serviceMonitorSelectorNilUsesHelmValues: false
          podMonitorSelectorNilUsesHelmValues: false
          probeSelectorNilUsesHelmValues: false
          # Example of a simple alert rule to demonstrate PagerDuty integration
          additionalAlertManagerConfigs:
          - name: "pagerduty-alerts"
            rules:
            - alert: HostHighCPUUsage
              expr: sum(rate(node_cpu_seconds_total{mode!="idle"}[5m])) by (instance) > 0.8
              for: 1m
              labels:
                severity: critical
              annotations:
                summary: "High CPU usage on {{ $labels.instance }}"
                description: "CPU usage is above 80% on instance {{ $labels.instance }}"
      
      grafana:
        enabled: true
        adminPassword: "strong-password-here" # CHANGE THIS IN PRODUCTION
        service:
          type: LoadBalancer # For external access, change to ClusterIP for internal access
          annotations:
            # If using AWS Load Balancer Controller, customize annotations as needed
            # service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    EOF
  ]

  depends_on = [
    aws_eks_node_group.main,
    kubernetes_namespace.monitoring # Ensure namespace is created first
  ]
}

# Create Kubernetes namespace for monitoring
resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = "monitoring"
  }
}

Step 6: Define Outputs (outputs.tf)

These outputs provide useful information after deployment.

output "eks_cluster_name" {
  description = "The name of the EKS cluster"
  value       = aws_eks_cluster.main.name
}

output "kubeconfig_command" {
  description = "Command to update your local kubeconfig"
  value       = "aws eks update-kubeconfig --region ${var.aws_region} --name ${aws_eks_cluster.main.name}"
}

output "grafana_url" {
  description = "The URL for Grafana (may take a few minutes to be available)"
  value       = "http://${helm_release.kube_prometheus_stack.status[0].load_balancer.ingress[0].hostname}"
}

output "prometheus_url" {
  description = "The URL for Prometheus (ClusterIP, may need port-forwarding)"
  value       = "To access: kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090"
}

output "alertmanager_url" {
  description = "The URL for Alertmanager (ClusterIP, may need port-forwarding)"
  value       = "To access: kubectl port-forward -n monitoring svc/kube-prometheus-stack-alertmanager 9093:9093"
}

Deployment Steps

Navigate to your project directory in the terminal and execute the following commands:

Initialize Terraform:
terraform init
Validate Configuration:
terraform validate
Review Plan:
terraform plan -var="pagerduty_integration_key=YOUR_PAGERDUTY_KEY"

Important: Replace YOUR_PAGERDUTY_KEY with the actual integration key obtained from PagerDuty. For production, use environment variables or a secrets manager.
Apply Configuration:
terraform apply -var="pagerduty_integration_key=YOUR_PAGERDUTY_KEY"

Type yes when prompted to confirm the deployment.

Post-Deployment Verification

After Terraform successfully applies the configuration, run the `kubeconfig_command` from the outputs to connect `kubectl` to your new EKS cluster:

aws eks update-kubeconfig --region us-east-1 --name prometheus-eks-cluster

Then, verify the deployments:

Check Kubernetes Pods:
kubectl get pods -n monitoring

You should see pods for Prometheus, Alertmanager, Grafana, and various exporters in a Running state.
Access Grafana:
Once the Grafana LoadBalancer service is provisioned (may take a few minutes), use the grafana_url output from Terraform. Log in with admin and the password you specified in the Helm values (strong-password-here by default, change this immediately!).
Test PagerDuty Alerting:
The example HostHighCPUUsage alert rule defined in the additionalAlertManagerConfigs section will trigger if any node's CPU usage is consistently high (over 80% for 1 minute). To manually test:
1. Identify a workload or create a dummy pod that can consume significant CPU.
2. Wait for Prometheus to scrape metrics and Alertmanager to fire the alert.
3. Verify that an incident is created in your PagerDuty service. You should receive notifications based on your PagerDuty escalation policies.
Alternatively, you can port-forward to Alertmanager UI:

kubectl port-forward -n monitoring svc/kube-prometheus-stack-alertmanager 9093:9093

Then visit http://localhost:9093 in your browser to see active alerts and configurations.

Best Practices and Advanced Considerations

Persistent Storage: For production, ensure you're using a robust storage class (e.g., GP2 or GP3 for EBS, or EFS) with sufficient capacity and IOPS for Prometheus and Alertmanager.
High Availability: Increase Prometheus and Alertmanager replicas in the Helm chart values for redundancy. Consider a multi-AZ EKS cluster for even greater resilience.
Custom Alert Rules: Define specific alert rules relevant to your applications and infrastructure by creating Kubernetes PrometheusRule objects or by extending the additionalPrometheusRules section in the Helm chart values.
Fine-tune Alertmanager Routes: Configure more sophisticated Alertmanager routing rules based on alert labels (e.g., routing critical alerts to PagerDuty, warning alerts to Slack).
EKS Logging: Integrate EKS control plane logs with CloudWatch Logs and use fluentd/fluentbit for node and application logs to ensure comprehensive observability.
Cost Optimization: Explore using EC2 Spot Instances for your EKS worker nodes (with appropriate tolerations and node selectors for critical workloads) to reduce costs.
Security: Implement Kubernetes Network Policies to restrict traffic between namespaces and ensure Prometheus can only scrape allowed targets. Use IAM Roles for Service Accounts (IRSA) for fine-grained permissions for your monitoring components.
Long-Term Storage: For historical data analysis beyond Prometheus's retention, consider integrating with solutions like Thanos or Cortex.

Troubleshooting Common Issues

EKS Cluster Creation Failure: Check IAM role permissions, VPC CIDR blocks, and subnet configurations. Ensure your AWS account has sufficient service limits.
Kubernetes Pods Not Running: Use kubectl describe pod <pod-name> -n monitoring to check events and logs. Common causes include insufficient resources, image pull errors, or incorrect configurations.
Prometheus Not Scraping Metrics: Verify that ServiceMonitor or PodMonitor resources are correctly defined and that their selectors match your application services/pods. Check Prometheus targets UI at http://localhost:9090/targets (after port-forwarding).
Alerts Not Reaching PagerDuty:
1. Confirm the pagerduty_integration_key is correct in your Alertmanager configuration.
2. Check Alertmanager logs for errors: kubectl logs -f -n monitoring <alertmanager-pod-name>.
3. Ensure network connectivity from the EKS cluster to PagerDuty's API endpoints (check security group egress rules).
4. Verify alert rules are correctly defined and firing in Prometheus and Alertmanager UI.

Conclusion

By following this guide, you have successfully deployed a robust, production-ready monitoring solution using Terraform to provision AWS EKS, Prometheus, and Alertmanager, integrated with PagerDuty for incident management. This architecture provides a solid foundation for observing your cloud-native applications, ensuring you have the visibility and alerting capabilities required to maintain high availability and quickly respond to operational issues. Embrace the power of Infrastructure as Code to automate and scale your observability practices, paving the way for more resilient and efficient operations.

CloudStack YAML