Automating AWS EKS Prometheus Monitoring and PagerDuty Alerting with Terraform

Architecture Pro-Tip: When designing your cloud-native monitoring strategy, prioritize a 'shift-left' approach by integrating observability tools directly into your IaC. This ensures consistency, reproducibility, and version control for your entire monitoring stack, from metric collection to alert routing. Decouple your alerting logic from raw Prometheus alerts into Alertmanager for flexible routing, deduplication, and suppression, vastly improving incident response efficiency.

Automating AWS EKS Prometheus Monitoring and PagerDuty Alerting with Terraform

In the dynamic world of cloud-native applications, robust monitoring and efficient incident management are paramount. For organizations leveraging AWS EKS for their Kubernetes workloads, a powerful combination of Prometheus for metrics and PagerDuty for incident alerting, all automated with Terraform, provides a comprehensive and scalable observability solution. This guide will walk you through the process of setting up and automating this critical stack.

Why Automate with Terraform?

Terraform, as an Infrastructure as Code (IaC) tool, allows you to define and provision your entire infrastructure declaratively. Automating your monitoring and alerting setup with Terraform brings several benefits:

Consistency: Ensure identical configurations across development, staging, and production environments.
Version Control: Track changes to your monitoring setup, enabling rollbacks and clear audit trails.
Efficiency: Rapidly deploy or update monitoring components with minimal manual effort.
Reduced Human Error: Eliminate manual configuration mistakes.
Scalability: Easily extend monitoring to new clusters or services.

Prerequisites

Before you begin, ensure you have the following:

An active AWS account with appropriate permissions to manage EKS and related resources.
AWS CLI configured on your local machine.
An existing AWS EKS cluster. This guide assumes your EKS cluster is already running.
kubectl installed and configured to connect to your EKS cluster.
helm CLI installed.
Terraform CLI installed (version 1.0+ recommended).
A PagerDuty account with an existing service and a new Events API (V2) integration key.

Core Components Overview

We'll be leveraging several key technologies:

AWS EKS: Managed Kubernetes service where our applications and monitoring stack will run.
Prometheus: The de-facto standard for Kubernetes monitoring, collecting metrics from various services.
Alertmanager: Handles alerts sent by Prometheus, deduping, grouping, and routing them to external notification systems like PagerDuty.
Grafana: Used for visualizing the metrics collected by Prometheus. (Included in kube-prometheus-stack).
kube-prometheus-stack (Helm Chart): A comprehensive collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules for monitoring Kubernetes and its workloads. This bundles Prometheus, Alertmanager, Grafana, node-exporter, kube-state-metrics, etc.
PagerDuty: An incident management platform that will receive critical alerts from Alertmanager and notify on-call teams.

Step-by-Step Implementation with Terraform

1. Initialize Terraform Project and Providers

Start by creating a new directory for your Terraform configuration. We'll need the aws and helm providers.

resource "aws_eks_cluster" "main" {
  # ... your EKS cluster definition or data source ...
  # Ensure you have the cluster configured and kubectl context set.
}

data "aws_eks_cluster" "main" {
  name = "your-eks-cluster-name" # Replace with your EKS cluster name
}

data "aws_eks_cluster_auth" "main" {
  name = "your-eks-cluster-name" # Replace with your EKS cluster name
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.main.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.main.token
    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args        = ["eks", "get-token", "--cluster-name", data.aws_eks_cluster.main.name]
    }
  }
}
    

This configuration sets up the helm provider to communicate with your EKS cluster using the AWS CLI for authentication.

2. Deploy `kube-prometheus-stack` using Helm

The kube-prometheus-stack Helm chart is the easiest way to deploy Prometheus, Alertmanager, and Grafana to your EKS cluster. We'll use the helm_release resource in Terraform.

resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = "monitoring"
  }
}

resource "helm_release" "prometheus_stack" {
  name       = "kube-prometheus-stack"
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  namespace  = kubernetes_namespace.monitoring.metadata[0].name
  version    = "58.1.0" # Use a specific chart version for stability

  values = [
    # Basic values, more can be added
    file("${path.module}/values.yaml")
  ]

  # Optional: Wait for resources to be ready
  wait = true
  timeout = 600
}
    

We're referencing a values.yaml file to keep our Helm chart configurations organized. This file will contain all the custom settings for Prometheus, Alertmanager, and Grafana.

3. Configure Alertmanager for PagerDuty Alerting

Inside your values.yaml file, you'll define the Alertmanager configuration. This includes receivers and routing rules to send alerts to PagerDuty. You'll need your PagerDuty Events API (V2) integration key.

# values.yaml for kube-prometheus-stack
alertmanager:
  enabled: true
  config:
    global:
      resolve_timeout: 5m
    route:
      group_by: ['alertname', 'cluster', 'service']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 1h
      receiver: 'default-pagerduty' # Default receiver for all alerts
      routes:
      - match:
          severity: critical
        receiver: 'pagerduty-critical' # Specific receiver for critical alerts
        continue: true # Continue evaluation for other rules if needed
      - match:
          severity: warning
        receiver: 'pagerduty-warning' # Specific receiver for warning alerts

    receivers:
    - name: 'default-pagerduty'
      pagerduty_configs:
      - service_key: "YOUR_PAGERDUTY_DEFAULT_INTEGRATION_KEY" # Replace with your PagerDuty default service key
        severity: "{{ .CommonLabels.severity | title }}" # Dynamically set severity based on alert label
        description: "{{ .CommonLabels.alertname }} in {{ .CommonLabels.namespace }}/{{ .CommonLabels.pod }}"
        details:
          message: "{{ .CommonLabels.alertname }} (severity: {{ .CommonLabels.severity }}) triggered for instance {{ .CommonLabels.instance }}"
          summary: "{{ .CommonLabels.alertname }} is {{ .Status }}"
          description: "{{ .Annotations.description }}"
          dashboard: "https://grafana.example.com/d/my-dashboard?var-instance={{ .CommonLabels.instance }}"
          playbook: "https://your-internal-wiki.com/playbooks/{{ .CommonLabels.alertname }}"
    - name: 'pagerduty-critical'
      pagerduty_configs:
      - service_key: "YOUR_PAGERDUTY_CRITICAL_INTEGRATION_KEY" # Replace with your PagerDuty critical service key
        severity: "critical"
        description: "[CRITICAL] {{ .CommonLabels.alertname }} in {{ .CommonLabels.namespace }}"
        # ... other details as above ...
    - name: 'pagerduty-warning'
      pagerduty_configs:
      - service_key: "YOUR_PAGERDUTY_WARNING_INTEGRATION_KEY" # Replace with your PagerDuty warning service key
        severity: "warning"
        description: "[WARNING] {{ .CommonLabels.alertname }} in {{ .CommonLabels.namespace }}"
        # ... other details as above ...
    

Important: Replace YOUR_PAGERDUTY_DEFAULT_INTEGRATION_KEY, YOUR_PAGERDUTY_CRITICAL_INTEGRATION_KEY, and YOUR_PAGERDUTY_WARNING_INTEGRATION_KEY with actual keys from your PagerDuty service integrations. Consider using Terraform secrets management (e.g., AWS Secrets Manager or Vault) for these sensitive keys instead of hardcoding them in the values.yaml.

4. Define Custom Prometheus Alerting Rules

While kube-prometheus-stack comes with a robust set of default rules, you'll often need to define custom alerts specific to your applications. These can also be managed within your values.yaml under the prometheus.prometheusSpec.ruleSelector.matchLabels or by deploying separate PrometheusRule objects. For simplicity, we'll embed a basic example within values.yaml.

# values.yaml continuation
prometheus:
  enabled: true
  prometheusSpec:
    ruleSelector:
      matchLabels:
        app: kube-prometheus-stack-prometheus
        release: kube-prometheus-stack # Default label from the chart
    additionalAlertManagerConfigs:
      - static_configs:
        - targets:
          - "alertmanager-operated:9093"
    additionalScrapeConfigs:
      # Example: scrape a custom application
      - job_name: 'my-custom-app'
        kubernetes_sd_configs:
        - role: endpoints
        relabel_configs:
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
          action: keep
          regex: "true"
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: "(.+)"
        - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
          action: replace
          target_label: __address__
          regex: "([^:]+)(?::\d+)?;(\d+)"
          replacement: "$1:$2"
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_service_name]
          action: replace
          target_label: kubernetes_service_name

prometheusOperator:
  enabled: true

# Define custom Prometheus rules directly
prometheus-rules:
  additionalRules:
    - name: custom-app-alerts
      groups:
        - name: general.rules
          rules:
            - alert: HighErrorRate
              expr: sum(rate(http_requests_total{job="my-custom-app", code=~"5xx"}[5m])) by (kubernetes_namespace, kubernetes_service_name) / sum(rate(http_requests_total{job="my-custom-app"}[5m])) by (kubernetes_namespace, kubernetes_service_name) > 0.05
              for: 5m
              labels:
                severity: critical
                tier: application
              annotations:
                summary: "High error rate (5xx) detected on {{ $labels.kubernetes_service_name }}"
                description: "The service {{ $labels.kubernetes_service_name }} in namespace {{ $labels.kubernetes_namespace }} is experiencing a 5xx error rate above 5% for more than 5 minutes. Investigate immediately."
            - alert: PodNotReady
              expr: kube_pod_status_phase{phase="Running"} == 0
              for: 2m
              labels:
                severity: warning
                tier: infrastructure
              annotations:
                summary: "Pod {{ $labels.pod }} in namespace {{ $labels.namespace }} is not running"
                description: "Pod {{ $labels.pod }} has been in a non-running phase for 2 minutes. Check pod status and logs."
    

This example shows how to add a custom scraping configuration for an application and define two simple alert rules: one for a high 5xx error rate and another for pods not running. These rules will be evaluated by Prometheus, and if triggered, alerts will be sent to Alertmanager, which then routes them to PagerDuty.

Ready-to-Use Terraform Configuration

Here's a consolidated example of your main Terraform configuration (e.g., main.tf) and the corresponding values.yaml for a complete setup. Remember to replace placeholders like your-eks-cluster-name and PagerDuty keys.

`main.tf` Example

# main.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.0"
    }
  }
}

provider "aws" {
  region = "us-east-1" # Specify your AWS region
}

# Data sources for your EKS cluster
data "aws_eks_cluster" "main" {
  name = "your-eks-cluster-name" # <<< REPLACE THIS
}

data "aws_eks_cluster_auth" "main" {
  name = "your-eks-cluster-name" # <<< REPLACE THIS
}

# Configure Helm provider to connect to EKS
provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.main.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.main.token
    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args        = ["eks", "get-token", "--cluster-name", data.aws_eks_cluster.main.name]
    }
  }
}

# Configure Kubernetes provider (optional, mainly for creating namespaces)
provider "kubernetes" {
  host                   = data.aws_eks_cluster.main.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.main.token
  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args        = ["eks", "get-token", "--cluster-name", data.aws_eks_cluster.main.name]
  }
}

# Create a namespace for monitoring components
resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = "monitoring"
  }
}

# Deploy kube-prometheus-stack using Helm
resource "helm_release" "prometheus_stack" {
  name       = "kube-prometheus-stack"
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  namespace  = kubernetes_namespace.monitoring.metadata[0].name
  version    = "58.1.0" # Always pin to a specific chart version
  values = [
    file("${path.module}/values.yaml")
  ]

  # Optional: Wait for resources to be ready and set a timeout
  wait       = true
  timeout    = 900
  depends_on = [kubernetes_namespace.monitoring]
}

output "grafana_url" {
  description = "The URL for Grafana dashboard. You might need to expose it via ingress."
  value = "http://localhost:3000 (port-forward manually: kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80)"
}
    

`values.yaml` Example

# values.yaml
alertmanager:
  enabled: true
  config:
    global:
      resolve_timeout: 5m
    route:
      group_by: ['alertname', 'cluster', 'service']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 1h
      receiver: 'pagerduty-critical' # Default to critical alerts
      routes:
      - match:
          severity: warning
        receiver: 'pagerduty-warning' # Route warnings to a specific PagerDuty key

    receivers:
    - name: 'pagerduty-critical'
      pagerduty_configs:
      - service_key: "YOUR_PAGERDUTY_CRITICAL_INTEGRATION_KEY" # <<< REPLACE THIS
        severity: "critical"
        description: "{{ .CommonLabels.alertname }} in {{ .CommonLabels.namespace }}/{{ .CommonLabels.pod }}"
        details:
          summary: "{{ .CommonLabels.alertname }} is {{ .Status }}"
          description: "{{ .Annotations.description }}"
          instance: "{{ .CommonLabels.instance }}"
          cluster: "{{ .CommonLabels.cluster }}"
          namespace: "{{ .CommonLabels.namespace }}"
          pod: "{{ .CommonLabels.pod }}"
          severity_label: "{{ .CommonLabels.severity }}"
          dashboard: "https://your-grafana-url.com/d/my-app?var-instance={{ .CommonLabels.instance }}"
          playbook: "https://your-internal-wiki.com/playbooks/{{ .CommonLabels.alertname | replace \" \" \"-\" }}"
    - name: 'pagerduty-warning'
      pagerduty_configs:
      - service_key: "YOUR_PAGERDUTY_WARNING_INTEGRATION_KEY" # <<< REPLACE THIS
        severity: "warning"
        description: "[WARNING] {{ .CommonLabels.alertname }} in {{ .CommonLabels.namespace }}/{{ .CommonLabels.pod }}"
        details:
          summary: "[WARNING] {{ .CommonLabels.alertname }} is {{ .Status }}"
          description: "{{ .Annotations.description }}"
          instance: "{{ .CommonLabels.instance }}"
          cluster: "{{ .CommonLabels.cluster }}"
          namespace: "{{ .CommonLabels.namespace }}"
          pod: "{{ .CommonLabels.pod }}"
          severity_label: "{{ .CommonLabels.severity }}"

grafana:
  enabled: true
  adminPassword: "your-strong-grafana-password" # <<< REPLACE THIS or use secrets
  service:
    type: ClusterIP # Use LoadBalancer or Ingress for external access
  ingress:
    enabled: false # Set to true and configure if you use an Ingress controller

prometheus:
  enabled: true
  prometheusSpec:
    # Example custom scrape config for an application (assuming Prometheus annotations on services)
    additionalScrapeConfigs:
      - job_name: 'kubernetes-service-endpoints'
        kubernetes_sd_configs:
          - role: endpoints
        relabel_configs:
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
            action: keep
            regex: "true"
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: "(.+)"
          - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
            action: replace
            target_label: __address__
            regex: "([^:]+)(?::\d+)?;(\d+)"
            replacement: "$1:$2"
          - action: labelmap
            regex: __meta_kubernetes_service_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_service_name]
            action: replace
            target_label: kubernetes_service_name

    # Mount additional Prometheus rules
    ruleSelector:
      matchLabels:
        prometheus: kube-prometheus-stack
        role: alert-rules
    additionalAlertManagerConfigs:
      - static_configs:
        - targets:
          - "kube-prometheus-stack-kube-pr-alertmanager.monitoring.svc:9093" # Adjust if your namespace/release name differs

prometheusOperator:
  enabled: true

# Add custom PrometheusRule objects via the chart
prometheus-rules:
  additionalRules:
    - name: custom-application-alerts
      groups:
        - name: application-errors
          rules:
            - alert: CriticalApplicationErrorRate
              expr: sum(rate(my_app_errors_total{environment="production"}[5m])) by (service) > 10
              for: 5m
              labels:
                severity: critical
                team: backend
              annotations:
                summary: "Critical error rate detected for {{ $labels.service }}"
                description: "The application service {{ $labels.service }} is experiencing a critical error rate above 10 errors/second for 5 minutes. Investigate immediately."
            - alert: HighLatencyApplication
              expr: histogram_quantile(0.99, sum by (le, service) (rate(my_app_request_duration_seconds_bucket{environment="production"}[5m]))) > 1
              for: 10m
              labels:
                severity: warning
                team: backend
              annotations:
                summary: "High latency detected for {{ $labels.service }}"
                description: "The 99th percentile request latency for service {{ $labels.service }} is above 1 second for 10 minutes."
    

5. Deploy with Terraform

Navigate to your Terraform project directory and execute the following commands:

terraform init
terraform plan
terraform apply --auto-approve
    

Terraform will now deploy the kube-prometheus-stack Helm chart to your EKS cluster, configuring Prometheus to scrape metrics, Alertmanager to route alerts to PagerDuty, and Grafana for visualization.

Testing and Validation

After deployment, it's crucial to verify that everything is working as expected:

Check Pod Status:
kubectl get pods -n monitoring

Ensure all pods (Prometheus, Alertmanager, Grafana, exporters) are running.
Access Grafana:
Port-forward the Grafana service:

kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80

Then open http://localhost:3000 in your browser. Log in with admin and your specified password. Verify that default dashboards are populated with data.
Check Prometheus UI:
Port-forward the Prometheus service:

kubectl -n monitoring port-forward svc/kube-prometheus-stack-prometheus 9090:9090

Access http://localhost:9090. Navigate to 'Status > Targets' to ensure your application metrics endpoints are being scraped. Check 'Alerts' to see the status of your configured rules.
Test Alertmanager and PagerDuty:
You can either intentionally trigger an alert (e.g., by scaling down a critical deployment to zero replicas to trigger a PodNotReady alert) or use the Alertmanager UI.

Port-forward Alertmanager:

kubectl -n monitoring port-forward svc/kube-prometheus-stack-kube-pr-alertmanager 9093:9093

Access http://localhost:9093. You can send a test alert from Prometheus's "Status > Runtime & Build Information" and look for the 'Test Alert' button or manually create a curl request to Alertmanager. Crucially, verify that alerts appear in your PagerDuty account.

Troubleshooting and Best Practices

Common Issues:

Helm Release Stuck: If terraform apply hangs, check Kubernetes events (kubectl get events -n monitoring) or pod logs for errors. Increase timeout in helm_release if needed.
Prometheus Not Scraping: Verify service/pod annotations (prometheus.io/scrape: "true", prometheus.io/port: "XXXX", etc.) for your applications. Check Prometheus UI targets.
Alerts Not Firing/Reaching PagerDuty:
- Check Prometheus UI 'Alerts' tab – are rules evaluating to 'FIRING'?
- Check Alertmanager UI – are alerts being received? Are they being routed correctly?
- Double-check your PagerDuty integration keys and network connectivity from Alertmanager pods.
- Review Alertmanager logs for errors related to PagerDuty integration.
Terraform EKS Authentication: Ensure your AWS CLI is configured for the correct region and account, and you have permission to run aws eks get-token.

Best Practices:

Secrets Management: Never hardcode PagerDuty integration keys or Grafana passwords directly in your values.yaml. Use Terraform's secret management integrations (e.g., AWS Secrets Manager, HashiCorp Vault) or Kubernetes Secrets with External Secrets Operator.
Role-Based Access Control (RBAC): Ensure your Prometheus and Alertmanager deployments have appropriate Kubernetes RBAC permissions to read metrics and send alerts. The kube-prometheus-stack chart typically handles this, but custom setups might require adjustments.
Persistence: For production environments, configure persistent storage for Prometheus and Alertmanager to retain metrics and alert states across pod restarts. This is typically done via Persistent Volume Claims (PVCs), which you can enable in the values.yaml.
External Access (Grafana/Prometheus): For production, expose Grafana and Prometheus UI securely via an Ingress controller (e.g., AWS ALB Ingress Controller) with TLS and authentication, rather than port-forwarding.
Alert Granularity: Craft alert rules carefully. Avoid noisy alerts, prioritize actionable ones, and use Alertmanager's grouping, inhibition, and silences effectively.
Terraform State Management: Use a remote backend (e.g., S3 with DynamoDB locking) for your Terraform state file, especially in team environments.

Conclusion

Automating AWS EKS Prometheus monitoring and PagerDuty alerting with Terraform establishes a robust, maintainable, and scalable observability foundation for your cloud-native applications. By defining your entire monitoring and alerting infrastructure as code, you gain consistency, version control, and operational efficiency, allowing your teams to focus on innovation rather than manual configurations. Implement these practices to achieve true DevOps maturity and ensure your critical services are always under vigilant watch.

CloudStack YAML