Terraform AWS EKS Prometheus PagerDuty Integration for Incident Response

Architecture Pro-Tip: Resilient Monitoring Design

For mission-critical applications on AWS EKS, ensure your Prometheus and Alertmanager deployments are highly available. Consider running multiple Prometheus replicas with Thanos for long-term storage and global query views, and deploy Alertmanager in a clustered configuration across different availability zones. Leverage AWS IAM Roles for Service Accounts (IRSA) for secure access to AWS services from your Kubernetes pods, enhancing both security posture and operational efficiency. Automate all of this with Terraform to guarantee consistent and repeatable deployments.

Terraform AWS EKS Prometheus PagerDuty Integration for Incident Response

In today's dynamic cloud environments, rapid incident detection and response are paramount for maintaining high availability and customer satisfaction. This comprehensive guide details how to seamlessly integrate AWS Elastic Kubernetes Service (EKS) with Prometheus for robust monitoring, Alertmanager for sophisticated alert routing, and PagerDuty for streamlined incident management, all orchestrated using Terraform. By implementing this Infrastructure as Code (IaC) approach, organizations can achieve predictable, scalable, and auditable incident response workflows for their AWS EKS clusters.

The Power of Integrated DevOps Tooling

Combining these industry-leading tools provides a powerful monitoring and incident response stack:

Terraform: Automates the provisioning and management of the entire infrastructure, from the EKS cluster to the deployment of monitoring agents and configuration of alerting rules. It ensures consistency and enables rapid iteration.
AWS EKS: Provides a managed Kubernetes service, simplifying the deployment, management, and scaling of containerized applications in the AWS cloud.
Prometheus & Alertmanager: An open-source monitoring and alerting toolkit. Prometheus collects metrics from your EKS cluster and applications, while Alertmanager de-duplicates, groups, and routes alerts to the appropriate receivers.
PagerDuty: A leading digital operations management platform that aggregates alerts from various monitoring systems, notifies on-call teams based on schedules, escalates incidents, and facilitates incident response.

Prerequisites for Implementation

Before diving into the configuration, ensure you have the following tools and accounts set up:

An active AWS account with administrative access.
Terraform CLI (v1.0.0 or later) installed and configured.
kubectl CLI installed and configured to connect to EKS.
Helm CLI (v3 or later) installed.
A PagerDuty account with permissions to create services and integration keys.
AWS CLI installed and configured.

Step-by-Step Integration Guide

1. PagerDuty Service Configuration

First, set up a dedicated service in PagerDuty to receive alerts from Alertmanager.

Log in to your PagerDuty account.
Navigate to Services > Service Directory > +New Service.
Give your service a descriptive name (e.g., "EKS Production Alerts") and assign an escalation policy.
On the integrations step, select "Events API v2" as the integration type.
Save the service. You will be provided with an Integration Key. Keep this key secure, as it will be used in your Alertmanager configuration.

2. Terraform for AWS EKS Cluster and Supporting Infrastructure

Use Terraform to provision your EKS cluster, its VPC, subnets, and node groups. This foundational step ensures a scalable and secure environment.

Key Terraform modules often include:

terraform-aws-modules/vpc/aws for networking.
terraform-aws-modules/eks/aws for the EKS cluster and node groups.

3. Terraform for Prometheus and Alertmanager Deployment on EKS

Once EKS is up and running, Terraform will deploy the Prometheus Operator, Prometheus, and Alertmanager using the Kubernetes and Helm providers. This includes creating necessary Kubernetes namespaces, service accounts, and RBAC roles.

The Prometheus community provides official Helm charts that simplify deployment. You'll primarily use the kube-prometheus-stack chart, which includes Prometheus, Grafana, and Alertmanager.

4. Integrating Alertmanager with PagerDuty

The core of the incident response integration lies in Alertmanager's configuration. You will provide the PagerDuty integration key obtained earlier to Alertmanager via a Kubernetes Secret or ConfigMap, which the Helm chart can then reference.

5. Defining Prometheus Alerting Rules

Finally, define your alerting rules within Prometheus. These rules, written in PromQL, will evaluate metrics and fire alerts to Alertmanager when specified conditions are met. Examples include high CPU utilization, low disk space, or pod failures.

Ready-to-Use Configuration Example

This example demonstrates the essential Terraform and Kubernetes manifests to achieve the integration. Replace placeholder values like <YOUR_PAGERDUTY_INTEGRATION_KEY> and <YOUR_AWS_REGION> with your actual details. For brevity, EKS cluster setup is simplified, focusing on the monitoring integration aspect.

resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = "monitoring"
  }
}

# Assume EKS cluster and kubeconfig are already set up and accessible
# You would typically use a dedicated Terraform module for EKS.

# --- PagerDuty Integration Key Secret ---
resource "kubernetes_secret" "pagerduty_integration_key" {
  metadata {
    name      = "pagerduty-integration-key"
    namespace = kubernetes_namespace.monitoring.metadata[0].name
  }
  data = {
    "PAGERDUTY_KEY" = "<YOUR_PAGERDUTY_INTEGRATION_KEY>"
  }
  type = "Opaque"
}

# --- Helm Chart for kube-prometheus-stack (includes Prometheus, Alertmanager, Grafana) ---
resource "helm_release" "kube_prometheus_stack" {
  name       = "kube-prom"
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  namespace  = kubernetes_namespace.monitoring.metadata[0].name
  version    = "55.9.1" # Use a specific stable version

  set {
    name  = "alertmanager.enabled"
    value = "true"
  }

  set {
    name  = "alertmanager.config.externalUrl"
    value = "http://<YOUR_ALERTMANAGER_EXTERNAL_URL>" # e.g., ingress URL
  }

  values = [
    templatefile("${path.module}/alertmanager_config.yaml", {
      pagerduty_integration_key = base64decode(kubernetes_secret.pagerduty_integration_key.data["PAGERDUTY_KEY"])
    }),
    templatefile("${path.module}/prometheus_rules.yaml", {})
  ]

  # Optional: Expose Alertmanager via AWS Load Balancer Controller
  # set {
  #   name  = "alertmanager.service.type"
  #   value = "ClusterIP"
  # }
  # set {
  #   name  = "alertmanager.ingress.enabled"
  #   value = "true"
  # }
  # set {
  #   name  = "alertmanager.ingress.annotations.kubernetes\\.io/ingress\\.class"
  #   value = "nginx" # or "alb" if using AWS Load Balancer Controller
  # }
  # set {
  #   name  = "alertmanager.ingress.hosts[0]"
  #   value = "alertmanager.yourdomain.com"
  # }
}

# --- alertmanager_config.yaml (referenced by helm_release values) ---
# content in file alertmanager_config.yaml
# route:
#   group_by: ['alertname', 'cluster', 'service']
#   group_wait: 30s
#   group_interval: 5m
#   repeat_interval: 4h
#   receiver: 'pagerduty'
#   routes:
#   - receiver: 'pagerduty'
#     match:
#       severity: 'critical'
#
# receivers:
# - name: 'pagerduty'
#   pagerduty_configs:
#   - service_key: "${pagerduty_integration_key}" # Injects the key from the secret
#     url: 'https://events.pagerduty.com/v2/enqueue'
#     send_resolved: true

# --- prometheus_rules.yaml (example rules, referenced by helm_release values) ---
# content in file prometheus_rules.yaml
# alertmanager:
#   enabled: true
# prometheus:
#   enabled: true
#   additionalPrometheusRules:
#   - name: general.rules
#     groups:
#     - name: kubernetes.rules
#       rules:
#       - alert: KubeHighCPUUsage
#         expr: sum(node_cpu_seconds_total{mode!="idle"}) by (instance) / sum(node_cpu_seconds_total) by (instance) * 100 > 80
#         for: 5m
#         labels:
#           severity: critical
#         annotations:
#           summary: "Node {{ $labels.instance }} has high CPU usage"
#           description: "CPU usage on node {{ $labels.instance }} is at {{ $value | humanize }}% for 5 minutes."
#       - alert: KubePodCrashLooping
#         expr: sum(increase(kube_pod_container_status_restarts_total[5m])) by (namespace, pod, container) > 0
#         for: 1m
#         labels:
#           severity: critical
#         annotations:
#           summary: "Pod {{ $labels.pod }} in namespace {{ $labels.namespace }} is crashlooping"
#           description: "Container {{ $labels.container }} in pod {{ $labels.pod }} is restarting frequently."
    

To deploy this configuration:

Save the Alertmanager configuration into a file named alertmanager_config.yaml in your Terraform module directory.
Save the Prometheus rules into a file named prometheus_rules.yaml in the same directory.
Update the placeholder for <YOUR_PAGERDUTY_INTEGRATION_KEY> and <YOUR_ALERTMANAGER_EXTERNAL_URL> (if you plan to expose it).
Run terraform init, terraform plan, and terraform apply.

Testing the Integration

After applying the Terraform configuration, verify the integration:

Check Kubernetes Resources: Ensure all pods for Prometheus and Alertmanager are running in the monitoring namespace. kubectl get pods -n monitoring
Access Prometheus/Alertmanager: Port-forward to the Prometheus or Alertmanager UI to inspect targets, alerts, and configurations. kubectl port-forward svc/kube-prom-kube-prometheus-prometheus 9090:9090 -n monitoring kubectl port-forward svc/kube-prom-kube-prometheus-alertmanager 9093:9093 -n monitoring
Trigger an Alert: Intentionally create a condition that should trigger one of your defined Prometheus alerts (e.g., deploy a pod that immediately crashes, or temporarily increase a resource limit on a test node to exceed a threshold).
Verify PagerDuty Incident: Within minutes of an alert firing in Prometheus, a corresponding incident should be created in your PagerDuty service. Check the PagerDuty UI.

Best Practices and Advanced Considerations

High Availability: Deploy multiple replicas for Prometheus and Alertmanager. For Prometheus, consider Thanos for global views and long-term storage.
Security: Implement strict network policies for your monitoring stack. Use AWS IAM Roles for Service Accounts (IRSA) to provide fine-grained AWS permissions to Prometheus components.
Cost Optimization: Monitor resource usage of your monitoring stack itself. Consider optimizing Prometheus storage using object storage (e.g., S3 with Thanos) for cost-effective long-term retention.
Advanced Alerting: Implement Service Level Objectives (SLOs) and Service Level Indicators (SLIs) for more business-contextual alerting. Use Alertmanager's sophisticated routing to direct alerts based on labels (e.g., team, severity).
Grafana Dashboards: Complement your alerting with comprehensive Grafana dashboards for visualization and deeper analysis of your EKS metrics. The kube-prometheus-stack conveniently bundles Grafana.

Troubleshooting Common Issues

PagerDuty Incident Not Firing:
- Check Alertmanager logs for errors sending to PagerDuty (kubectl logs -n monitoring -l app.kubernetes.io/name=alertmanager).
- Verify the PagerDuty integration key in your Kubernetes Secret and Alertmanager configuration.
- Ensure Alertmanager can reach events.pagerduty.com (check network policies, firewalls, DNS).
Prometheus Not Scraping Metrics:
- Check Prometheus UI under 'Status > Targets' for failing scrape jobs.
- Verify ServiceMonitor and PodMonitor resources are correctly defined and match your application labels.
- Ensure firewall rules or network policies aren't blocking Prometheus from reaching target endpoints.
Alerts Not Firing in Prometheus:
- Check Prometheus UI under 'Status > Rules' to see if your rules are loaded without errors.
- Verify that the metric data exists and the alert expression evaluates to true in the Prometheus graph explorer.
- Adjust the for clause in your alert rules if the condition is not sustained long enough.

Conclusion

By leveraging Terraform for IaC automation, you can establish a robust, scalable, and maintainable AWS EKS monitoring and incident response solution. Integrating Prometheus and Alertmanager with PagerDuty ensures that critical alerts from your Kubernetes cluster are not only detected but also efficiently routed to the right teams, minimizing downtime and improving operational resilience. This integrated approach is a cornerstone for any mature DevOps practice operating on cloud-native platforms.

CloudStack YAML