Automating AWS EKS Incident Response with Terraform, Prometheus, and PagerDuty

In today's fast-paced cloud-native environments, swift and efficient incident response is paramount. For organizations running critical workloads on AWS Elastic Kubernetes Service (EKS), manual incident resolution can lead to unacceptable downtime and operational overhead. This guide delves into automating EKS incident response workflows using a powerful trifecta: Terraform for infrastructure-as-code deployment, Prometheus for robust monitoring and alerting, and PagerDuty for reliable incident notification and management.

Architecture Pro-Tip:

For large-scale or multi-cluster EKS deployments, consider a centralized Prometheus/Alertmanager setup that scrapes metrics from multiple clusters using federated Prometheus instances or a Thanos sidecar. This centralizes your alerting configuration and PagerDuty integration, simplifying management and providing a single pane of glass for incident observability across your entire Kubernetes footprint.

The Challenge: Manual EKS Incident Response

Without automation, responding to incidents in an EKS environment is often a labor-intensive process:

Slow Detection: Relying on manual checks or basic log alerts can delay incident identification.
Alert Fatigue: Inefficient alerting systems can generate a flood of non-actionable notifications, desensitizing on-call teams.
Delayed Escalation: Routing incidents to the correct team members can be cumbersome and error-prone.
Inconsistent Workflows: Lack of standardized procedures can lead to varying response times and effectiveness.
Manual Remediation: Many initial response steps might be repetitive and could be automated.

The Solution: A Synergistic Approach

By integrating Terraform, Prometheus, and PagerDuty, we establish a robust, automated incident response pipeline for EKS:

Terraform: Deploys and manages the entire monitoring and alerting infrastructure on EKS, ensuring consistency, version control, and auditability. This includes the EKS cluster itself, associated IAM roles, and the deployment of Prometheus and Alertmanager via Helm charts.
Prometheus: Scrapes metrics from Kubernetes components (kube-state-metrics), nodes (node-exporter), and applications. It provides a powerful query language (PromQL) for defining precise alerting rules based on real-time data.
Alertmanager: Handles alerts sent by Prometheus, deduplicating, grouping, and routing them to the correct receivers. Its sophisticated routing tree ensures alerts reach the right on-call teams through PagerDuty.
PagerDuty: Serves as the central hub for incident management, handling on-call schedules, escalation policies, and various notification channels (SMS, phone, email, push). It ensures that critical alerts are acknowledged and acted upon promptly.

Prerequisites

Before you begin, ensure you have the following:

An active AWS account with administrative privileges.
Terraform CLI installed (v1.0+ recommended).
AWS CLI configured with appropriate credentials.
kubectl CLI installed and configured.
helm CLI installed.
A PagerDuty account with API access and an integration key (REST API integration).
An existing AWS EKS cluster or the ability to create one using Terraform. This guide assumes you have an EKS cluster ready or can adapt the Terraform code to create one.

Step-by-Step Implementation Guide

1. Configure PagerDuty Service

First, set up a new service in PagerDuty to receive alerts from Alertmanager. This service will have its own integration key.

Navigate to Services > Service Directory in your PagerDuty account.
Click +New Service.
Give it a name (e.g., "EKS Critical Alerts"), assign an escalation policy, and click Create Service.
On the service details page, go to the Integrations tab, click +Add an integration.
Search for "Prometheus" or "Generic Events API" and select it. Click Add Integration.
Save the generated Integration Key. You'll need this for Alertmanager configuration.

2. Deploy Prometheus and Alertmanager to EKS using Terraform and Helm

We'll use Terraform to deploy the kube-prometheus-stack Helm chart, which includes Prometheus, Alertmanager, Grafana, and various exporters.

# main.tf
# Define your AWS provider and EKS cluster details first (not shown for brevity)

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.current.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.current.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.current.token
  }
}

resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = "monitoring"
  }
}

resource "helm_release" "kube_prometheus_stack" {
  name       = "kube-prometheus-stack"
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  namespace  = kubernetes_namespace.monitoring.metadata.0.name
  version    = "55.0.0" # Use a specific version

  values = [
    file("values.yaml")
  ]

  depends_on = [kubernetes_namespace.monitoring]
}

# values.yaml (This file will contain the configuration for Alertmanager and Prometheus)
# This is passed to the helm_release as a file.
alertmanager:
  enabled: true
  alertmanagerSpec:
    routePrefix: /
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: gp2 # Or your preferred StorageClass
          resources:
            requests:
              storage: 1Gi
    config:
      global:
        resolve_timeout: 5m
      route:
        group_by: ['alertname', 'cluster', 'service']
        group_wait: 30s
        group_interval: 5m
        repeat_interval: 1h
        receiver: 'pagerduty'
        routes:
        - match:
            severity: critical
          receiver: 'pagerduty'
          continue: true
      receivers:
      - name: 'pagerduty'
        pagerduty_configs:
        - service_key: "{{ .Values.pagerduty.integrationKey }}"
          routing_key: "{{ .Values.pagerduty.routingKey | default .Values.pagerduty.integrationKey }}" # For V2 API, routing key is preferred
          client: 'Prometheus Alertmanager'
          client_url: 'http://{{ .Release.Name }}-grafana.{{ .Release.Namespace }}.svc.cluster.local'
# Add the integration key to the values.yaml or pass it as --set in helm_release
pagerduty:
  integrationKey: "YOUR_PAGERDUTY_INTEGRATION_KEY_HERE" # Replace with your actual PagerDuty Integration Key
  routingKey: "YOUR_PAGERDUTY_ROUTING_KEY_HERE" # Optional, if using PagerDuty Events API v2 with a routing key

prometheus:
  enabled: true
  prometheusSpec:
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: gp2
          resources:
            requests:
              storage: 10Gi
    ruleSelectorNilUsesHelmValues: false # Required for older versions, ensure it's compatible with your Helm chart version
    serviceMonitorSelectorNilUsesHelmValues: false # Required for older versions

# Example of how you might define an EKS cluster and auth data sources:
data "aws_eks_cluster" "current" {
  name = "your-eks-cluster-name"
}

data "aws_eks_cluster_auth" "current" {
  name = "your-eks-cluster-name"
}

Explanation:

The helm_release resource installs the kube-prometheus-stack chart into the monitoring namespace.
The values.yaml file customizes the Alertmanager configuration, specifically defining a pagerduty_configs receiver.
Replace "YOUR_PAGERDUTY_INTEGRATION_KEY_HERE" with the Integration Key obtained from PagerDuty. For the Events API v2, a routing key is often used, which can be the same as the integration key or a specific key for routing within a service.
Ensure your EKS cluster name is correctly specified in the aws_eks_cluster and aws_eks_cluster_auth data sources.

Apply this Terraform configuration:

terraform init
terraform plan
terraform apply --auto-approve

3. Configure Prometheus Alerting Rules

Prometheus uses rules to define when an alert should fire. These rules are typically stored in PrometheusRule custom resources, which are managed by the Prometheus Operator (included in kube-prometheus-stack). Here’s an example for a critical EKS node not ready:

# eks-alerts.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: kubernetes-apps-alerts
  namespace: monitoring
  labels:
    prometheus: kube-prometheus-stack
    role: alert-rules
spec:
  groups:
  - name: kubernetes-apps
    rules:
    - alert: KubeNodeNotReady
      expr: |
        kube_node_status_condition{condition="Ready", status="false"} == 1
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: Kubernetes Node {{ $labels.node }} is not ready (instance {{ $labels.instance }})
        description: A Kubernetes node has been unready for more than 5 minutes. This indicates a potential issue with the node or its underlying infrastructure.
        runbook_url: "https://your-company.com/runbooks/eks-node-not-ready"

    - alert: KubePodCrashLooping
      expr: |
        sum by (namespace, pod, container) (increase(kube_pod_container_status_restarts_total{job="kube-state-metrics", container=~".+"}[10m])) > 5
      for: 1m
      labels:
        severity: warning
      annotations:
        summary: Pod {{ $labels.namespace }}/{{ $labels.pod }} (container {{ $labels.container }}) is crashlooping
        description: Container {{ $labels.container }} in pod {{ $labels.namespace }}/{{ $labels.pod }} is restarting frequently.
        runbook_url: "https://your-company.com/runbooks/eks-pod-crashlooping"

Apply these rules to your EKS cluster:

kubectl apply -f eks-alerts.yaml

Key points for alerting rules:

expr: The PromQL query that defines the condition for the alert.
for: The duration the condition must be true before the alert fires. This helps reduce flapping alerts.
labels: Critical for routing. The severity: critical label here will be used by Alertmanager to send to PagerDuty as defined in values.yaml.
annotations: Provide valuable context to the on-call engineer, including summaries, descriptions, and links to runbooks.

4. Verify Integration

After applying the Terraform and Kubernetes configurations, verify that Prometheus and Alertmanager are running correctly:

kubectl get pods -n monitoring
kubectl get svc -n monitoring

You should see pods for Prometheus, Alertmanager, Grafana, and various exporters in the monitoring namespace. You can port-forward to the Alertmanager UI to inspect its configuration and status:

kubectl port-forward svc/kube-prometheus-stack-alertmanager 9093:9093 -n monitoring

Then navigate to http://localhost:9093 in your browser. Under the "Status" tab, you should see the configured PagerDuty receiver.

Testing the Incident Response Workflow

To test the full workflow, you can deliberately trigger an alert. For instance, to trigger the KubeNodeNotReady alert, you could stop the kubelet service on one of your EKS nodes (DO NOT do this in a production environment without proper planning!).

Alternatively, for a safer test, create a temporary PrometheusRule that triggers immediately:

# test-alert.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: test-critical-alert
  namespace: monitoring
  labels:
    prometheus: kube-prometheus-stack
    role: alert-rules
spec:
  groups:
  - name: test-group
    rules:
    - alert: TestCriticalAlert
      expr: vector(1) # This expression is always true
      for: 1s
      labels:
        severity: critical
      annotations:
        summary: This is a test critical alert for EKS incident response automation.
        description: Please acknowledge and resolve this test alert.

kubectl apply -f test-alert.yaml

Within a few minutes, Prometheus should detect this alert, send it to Alertmanager, which will then route it to PagerDuty. You should receive a PagerDuty incident notification according to your service's escalation policy.

Remember to delete the test alert rule afterwards:

kubectl delete -f test-alert.yaml

Best Practices for EKS Incident Automation

Refine Alerting Rules: Continuously review and optimize your Prometheus alerting rules to minimize false positives and ensure that only actionable alerts are sent to PagerDuty.
Leverage Alertmanager Grouping: Use Alertmanager's grouping features effectively to prevent alert storms during widespread outages. Group related alerts into a single PagerDuty incident.
Detailed Annotations: Provide rich context in alert annotations, including links to Grafana dashboards, runbooks, and relevant documentation.
Test Regularly: Periodically test your incident response pipeline, especially after making changes to your EKS cluster, monitoring stack, or PagerDuty configurations.
Integrate with Observability Tools: Beyond Prometheus, consider integrating logs (e.g., Fluentd/Loki), traces (e.g., Jaeger/OpenTelemetry), and other metrics for comprehensive incident investigation.
Automated Remediation (Advanced): For well-understood, low-risk incidents, explore automated remediation actions triggered by PagerDuty webhooks or specific alert conditions. This could involve auto-scaling, restarting pods, or rolling back deployments.
Version Control Everything: Store all your Terraform code, Helm values, and PrometheusRule definitions in Git to maintain a clear history, enable collaboration, and facilitate disaster recovery.

Conclusion

Automating AWS EKS incident response with Terraform, Prometheus, and PagerDuty is a critical step towards building a resilient, highly available, and efficiently managed cloud-native infrastructure. By codifying your monitoring and alerting, you empower your DevOps teams to detect, triage, and resolve issues faster, minimizing impact on your services and end-users. This guide provides a solid foundation; remember to continuously iterate and refine your setup to meet the evolving demands of your EKS workloads.

CloudStack YAML