Terraform-Driven AWS EKS Production Alerting with Prometheus and PagerDuty Integration

Architecture Pro-Tip:

Always design your alerting strategy with a "shift-left" mindset. Integrate monitoring and alerting definitions directly into your Infrastructure as Code (IaC) pipelines. This ensures consistency, version control, and rapid deployment of critical observability components, treating your alerts as first-class citizens alongside your application code and infrastructure.

Terraform-Driven AWS EKS Production Alerting with Prometheus and PagerDuty Integration

In the dynamic world of cloud-native applications, maintaining robust observability is paramount for ensuring service reliability and minimizing downtime. For organizations leveraging AWS EKS for their Kubernetes workloads, a powerful and automated alerting system is not just a luxury but a necessity. This comprehensive guide will walk you through setting up a production-grade alerting solution for AWS EKS, driven by Terraform, utilizing Prometheus for metric collection and Alertmanager for routing critical incidents to PagerDuty.

By the end of this guide, you will have a clear understanding of how to implement a fully automated, IaC-managed alerting pipeline that proactively notifies your on-call teams about potential issues, significantly enhancing your incident response capabilities.

Why Terraform, Prometheus, and PagerDuty for EKS Alerting?

• Terraform: Enables Infrastructure as Code (IaC) for your entire alerting stack, including EKS add-ons, Prometheus configurations, and even PagerDuty service definitions. This ensures repeatability, version control, and auditability.
• Prometheus: The de-facto standard for monitoring Kubernetes clusters. Its powerful multi-dimensional data model, flexible query language (PromQL), and built-in Alertmanager make it ideal for collecting and evaluating EKS metrics.
• Alertmanager: Handles alerts sent by Prometheus, deduping, grouping, and routing them to the correct receiver (e.g., PagerDuty, Slack, email). It's crucial for preventing alert fatigue.
• PagerDuty: A leading incident management platform that provides reliable, actionable notifications, on-call scheduling, escalation policies, and incident tracking, ensuring critical alerts reach the right people at the right time.

Prerequisites

Before diving into the implementation, ensure you have the following:

• An AWS account with appropriate permissions to create EKS clusters and associated resources.
• Terraform installed and configured for AWS authentication.
• kubectl installed and configured to connect to your EKS cluster.
• An existing AWS EKS cluster. (This guide assumes you have one. If not, you can create one using Terraform as well.)
• A PagerDuty account with API access and an existing service or the ability to create one.
• Familiarity with Kubernetes concepts (Pods, Deployments, Services, Ingress, CRDs).

Core Components Architecture

The architecture involves several interconnected components:

• EKS Cluster: Your Kubernetes environment hosting your applications and monitoring stack.
• Prometheus Operator: Deploys and manages Prometheus and Alertmanager instances within Kubernetes using Custom Resource Definitions (CRDs).
• Prometheus: Scrapes metrics from your EKS cluster (nodes, pods, services, Kube-state-metrics, Node Exporter).
• Alertmanager: Receives alerts from Prometheus, processes them, and sends notifications.
• PagerDuty: The final destination for critical alerts, managing on-call rotations and incident escalation.
• Terraform: Orchestrates the deployment of the Prometheus Operator, Prometheus, Alertmanager, and integrates with PagerDuty.

Step-by-Step Implementation Guide

Step 1: PagerDuty Service Configuration (Terraform)

First, let's use Terraform to define a PagerDuty service and an integration key. This ensures your incident management platform is configured as part of your IaC.

provider "pagerduty" { token = var.pagerduty_api_token } resource "pagerduty_team" "devops_team" { name = "DevOps Team" description = "Team responsible for EKS infrastructure and applications" } resource "pagerduty_user" "oncall_engineer" { name = "On-Call Engineer" email = "oncall@example.com" } resource "pagerduty_schedule" "devops_oncall_schedule" { name = "DevOps Primary On-Call" time_zone = "America/Los_Angeles" layer { name = "Primary Shift" start = "2023-01-01T09:00:00-08:00" rotation_turn_length_seconds = 604800 # 1 week rotation_virtual_start = "2023-01-01T09:00:00-08:00" users = [pagerduty_user.oncall_engineer.id] } } resource "pagerduty_escalation_policy" "devops_escalation_policy" { name = "DevOps EKS Escalation Policy" num_loops = 2 rule { escalation_delay_in_minutes = 5 target { type = "schedule" id = pagerduty_schedule.devops_oncall_schedule.id } } rule { escalation_delay_in_minutes = 10 target { type = "user" id = pagerduty_user.oncall_engineer.id # Fallback to a specific user or another schedule } } } resource "pagerduty_service" "eks_monitoring_service" { name = "EKS Production Monitoring" auto_resolve_timeout = 14400 # 4 hours acknowledgement_timeout = 600 # 10 minutes escalation_policy = pagerduty_escalation_policy.devops_escalation_policy.id description = "Service for critical alerts from AWS EKS via Prometheus Alertmanager." team = pagerduty_team.devops_team.id } resource "pagerduty_service_integration" "prometheus_integration" { name = "Prometheus Alertmanager Integration" service = pagerduty_service.eks_monitoring_service.id type = "generic_events_api_v2" } output "pagerduty_integration_key" { value = pagerduty_service_integration.prometheus_integration.integration_key description = "PagerDuty Integration Key for Prometheus Alertmanager" sensitive = true }

Explanation: This Terraform configuration sets up a PagerDuty team, an on-call schedule, an escalation policy, and finally, a service with a "Generic Events API V2" integration. The output `pagerduty_integration_key` is crucial for Alertmanager to send events to PagerDuty.

Step 2: Deploy Prometheus and Alertmanager with Prometheus Operator (Terraform)

The Prometheus Operator simplifies the deployment and management of Prometheus and Alertmanager on Kubernetes. We'll use the Terraform Kubernetes provider and Helm provider to deploy it.

First, ensure your Kubernetes provider is configured:

provider "kubernetes" { host = data.aws_eks_cluster.main.endpoint cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority[0].data) token = data.aws_eks_cluster_auth.main.token } provider "helm" { kubernetes { host = data.aws_eks_cluster.main.endpoint cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority[0].data) token = data.aws_eks_cluster_auth.main.token } } data "aws_eks_cluster" "main" { name = var.eks_cluster_name } data "aws_eks_cluster_auth" "main" { name = var.eks_cluster_name }

Terraform and Prometheus Configuration

Now, let's deploy the kube-prometheus-stack Helm chart, which includes Prometheus, Alertmanager, Grafana, and all necessary exporters and CRDs. We'll configure Alertmanager to use the PagerDuty integration key.

resource "kubernetes_namespace" "monitoring" { metadata { name = "monitoring" } } resource "helm_release" "kube_prometheus_stack" { name = "prometheus" namespace = kubernetes_namespace.monitoring.metadata[0].name chart = "kube-prometheus-stack" repository = "https://prometheus-community.github.io/helm-charts" version = "57.0.0" # Use a specific stable version timeout = 600 set { name = "defaultRules.create" value = "true" } set { name = "kubeControllerManager.enabled" value = "false" } set { name = "kubeEtcd.enabled" value = "false" } set { name = "kubeScheduler.enabled" value = "false" } set { name = "kubeProxy.enabled" value = "false" } values = [ yamlencode({ alertmanager = { enabled = true config = { global = { resolve_timeout = "5m" } route = { group_by = ["alertname", "cluster", "service"] group_wait = "30s" group_interval = "5m" repeat_interval = "4h" receiver = "pagerduty" } receivers = [ { name = "pagerduty" pagerduty_configs = [ { service_key = var.pagerduty_integration_key severity = "{{ .CommonLabels.severity | default \"critical\" }}" client = "Prometheus Alertmanager" client_url = "http://{{ .ExternalURL }}" } ] } ] } # Use a secret for Alertmanager config in production for better security # Alternatively, create a secret via Terraform and reference it here # extraSecretMounts: # - name: alertmanager-config # mountPath: /etc/alertmanager/config # subPath: alertmanager.yaml # secretName: alertmanager-config-secret # readOnly: true } prometheus = { prometheusSpec = { scrapeInterval = "30s" retention = "15d" # Optional: Persistent storage for Prometheus # storageSpec = { # volumeClaimTemplate = { # spec = { # storageClassName = "gp2" # Or your preferred storage class # accessModes = ["ReadWriteOnce"] # resources = { # requests = { # storage = "50Gi" # } # } # } # } # } serviceMonitorSelectorNilUsesPods = true } } grafana = { enabled = true # For production, enable ingress, configure persistent storage, and secure passwords } kubeApiServer = { enabled = false # Metrics often duplicated by kube-state-metrics } kubelet = { enabled = true # Essential for node/pod health } kubeStateMetrics = { enabled = true # Essential for Kubernetes resource metrics } nodeExporter = { enabled = true # Essential for host-level metrics } }) ] depends_on = [ kubernetes_namespace.monitoring ] } resource "kubernetes_secret" "pagerduty_integration_key_secret" { metadata { name = "pagerduty-integration-key" namespace = kubernetes_namespace.monitoring.metadata[0].name } data = { "integration_key" = var.pagerduty_integration_key # Pass your PagerDuty integration key as a variable } type = "Opaque" } # Example of a PrometheusRule resource for a specific alert resource "kubernetes_manifest" "high_cpu_usage_alert" { depends_on = [helm_release.kube_prometheus_stack] manifest = { apiVersion = "monitoring.coreos.com/v1" kind = "PrometheusRule" metadata = { name = "eks-node-cpu-alerts" namespace = kubernetes_namespace.monitoring.metadata[0].name labels = { release = "prometheus" # Match the label of your prometheus instance } } spec = { groups = [ { name = "node.rules" rules = [ { alert = "HighCPUUsage" expr = "(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode='idle'}[5m])) * 100)) > 80" for = "5m" labels = { severity = "critical" owner = "devops-team" } annotations = { summary = "High CPU usage on instance {{ $labels.instance }}" description = "CPU usage on node {{ $labels.instance }} has been above 80% for 5 minutes. Current value: {{ $value }}" runbook = "https://example.com/runbooks/high-cpu" } }, { alert = "KubePodCrashLooping" expr = "sum(increase(kube_pod_container_status_restarts_total{job='kube-state-metrics'}[5m])) by (namespace, pod, container) > 5" for = "5m" labels = { severity = "critical" owner = "devops-team" } annotations = { summary = "Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) is crashlooping" description = "Pod {{ $labels.namespace }}/{{ $labels.pod }} (container {{ $labels.container }}) has restarted more than 5 times in the last 5 minutes." runbook = "https://example.com/runbooks/pod-crashloop" } } ] } ] } } }

Explanation:

• The `helm_release` resource deploys the `kube-prometheus-stack` chart into the `monitoring` namespace.
• We disable certain components (`kubeControllerManager`, `kubeEtcd`, `kubeScheduler`, `kubeProxy`, `kubeApiServer`) that might not be necessary or are redundant in EKS, reducing resource consumption.
• The `alertmanager.config` block directly embeds the configuration for Alertmanager, defining a receiver named `pagerduty` that uses the `service_key` provided by the Terraform output from the PagerDuty integration.
• A `kubernetes_secret` is created to securely store the PagerDuty integration key. Although it's used directly in the `helm_release` values for simplicity here, in a real production environment, you might prefer to mount this secret into the Alertmanager pod and reference it in the configuration file, or use external secrets management.
• An example `PrometheusRule` Custom Resource is defined to create alerts for high CPU usage on nodes and container crash loops. These rules will be picked up by Prometheus.

Step 3: Apply the Terraform Configuration

Save the above configurations into `.tf` files (e.g., `main.tf`, `variables.tf`, `outputs.tf`). Ensure your `variables.tf` defines `pagerduty_api_token`, `pagerduty_integration_key` (which will be an output from the PagerDuty provider), and `eks_cluster_name`.

Run the following Terraform commands:

terraform init terraform plan terraform apply

Confirm the changes and type `yes` when prompted. Terraform will deploy the PagerDuty resources, the Prometheus Operator, Prometheus, Alertmanager, and your custom alerting rules to your EKS cluster.

Testing and Validation

Verify Prometheus and Alertmanager Deployment

• Check Pods:
  kubectl get pods -n monitoring
  You should see pods for Prometheus, Alertmanager, Grafana, kube-state-metrics, and node-exporter running.
• Access Prometheus UI: Port-forward the Prometheus service:
  kubectl -n monitoring port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090
  Then navigate to `http://localhost:9090` in your browser. Check the "Status -> Targets" page to ensure metrics are being scraped. Check the "Alerts" page to see if your `PrometheusRule` alerts are loaded.
• Access Alertmanager UI: Port-forward the Alertmanager service:
  kubectl -n monitoring port-forward svc/prometheus-kube-prometheus-alertmanager 9093:9093
  Navigate to `http://localhost:9093`. You should see the configured PagerDuty receiver under "Configuration".

Trigger a Test Alert

The easiest way to test the PagerDuty integration is to trigger a simple alert.

• Manual Alert in Prometheus: You can temporarily modify a `PrometheusRule` with a condition that is always true, or use the Prometheus UI's "Graph" tab to fire an alert manually (though this won't go through Alertmanager).
• Simulate CrashLooping Pod: Create a deployment with a misconfigured health check or a simple script that exits immediately. This will trigger the `KubePodCrashLooping` alert you defined.
  apiVersion: apps/v1 kind: Deployment metadata: name: crashloop-test namespace: default spec: replicas: 1 selector: matchLabels: app: crashloop-test template: metadata: labels: app: crashloop-test spec: containers: - name: always-fail image: busybox command: ["/bin/sh", "-c", "exit 1"]
  Apply this YAML: `kubectl apply -f crashloop-test.yaml`. After a few minutes, Prometheus should detect the crash loops, fire the alert, Alertmanager should process it, and you should receive an incident notification in PagerDuty.

Monitor your PagerDuty service's incidents for the triggered alert. You should see a new incident with details matching your alert's annotations.

Best Practices for Production Alerting

• Actionable Alerts: Every alert should have a clear owner, severity, and a link to a runbook or troubleshooting guide. Avoid "noisy" alerts that don't indicate an immediate problem.
• SLOs/SLIs Driven: Define Service Level Objectives (SLOs) and Service Level Indicators (SLIs) for your critical services. Base your alerts on deviations from these SLOs (e.g., latency exceeding a threshold, error rates increasing).
• Alert Grouping: Use Alertmanager's grouping features effectively to aggregate related alerts into a single incident, preventing alert storms during outages.
• Silence and Deduplication: Leverage Alertmanager's ability to silence alerts during maintenance windows or deduplicate alerts from multiple sources.
• Review and Refine: Regularly review your alerts. Are they still relevant? Are they too sensitive or not sensitive enough? Adjust thresholds and rules based on operational feedback.
• Granular Permissions: Ensure your Kubernetes service accounts and roles for Prometheus have only the necessary permissions to scrape metrics.
• External Secrets Management: For true production, consider using AWS Secrets Manager or HashiCorp Vault to store sensitive information like PagerDuty API tokens and integration keys, and retrieve them dynamically in your Terraform or Kubernetes deployments.
• Persistent Storage: For Prometheus, enable persistent storage to retain historical metric data across pod restarts, which is critical for long-term analysis and debugging.

Troubleshooting Common Issues

• No Alerts Firing:
  • Check Prometheus UI: Are your targets healthy? Are the `PrometheusRule` objects loaded under "Alerts" section? Is the alert state 'PENDING' or 'FIRING'?
  • Check metric values: Use the "Graph" tab in Prometheus to query the metric your alert is based on. Is it actually crossing the threshold?
  • Check Prometheus logs: `kubectl logs -n monitoring prometheus-kube-prometheus-prometheus-0`
• Alerts Firing but No PagerDuty Notification:
  • Check Alertmanager UI: Navigate to `http://localhost:9093` (via port-forward). Do you see the firing alerts? Is the PagerDuty receiver configured correctly under "Configuration"?
  • Check Alertmanager logs: `kubectl logs -n monitoring prometheus-kube-prometheus-alertmanager-0`. Look for errors related to sending notifications.
  • Verify PagerDuty Integration Key: Ensure the `service_key` in Alertmanager configuration exactly matches the integration key from PagerDuty.
  • Check PagerDuty Service: Is the service enabled? Are there any network policies preventing Alertmanager from reaching PagerDuty APIs?
• Terraform Errors:
  • Authentication issues: Ensure your AWS and PagerDuty API tokens are correctly configured and have the necessary permissions.
  • Kubernetes provider issues: Verify your EKS cluster name is correct and `kubectl` context works.
  • Helm chart issues: Check the Helm chart version and `values` syntax.

Conclusion

Implementing a robust, automated alerting system is fundamental for operating production-grade AWS EKS clusters. By combining the power of Terraform for Infrastructure as Code, Prometheus for comprehensive monitoring, Alertmanager for intelligent routing, and PagerDuty for reliable incident management, you can significantly enhance your team's ability to detect, respond to, and resolve issues efficiently.

This guide provides a solid foundation for building your EKS alerting solution. Remember to tailor the alerts to your specific application and infrastructure needs, and continuously refine your monitoring strategy as your systems evolve. A well-configured alerting system is a cornerstone of operational excellence in a cloud-native environment.