Terraform-Managed Prometheus and PagerDuty On-Call Integration for AWS EKS Production Clusters

In the fast-paced world of cloud-native applications, maintaining high availability and reliability for production clusters is paramount. AWS Elastic Kubernetes Service (EKS) provides a robust platform for deploying scalable microservices, but robust monitoring and an efficient incident response strategy are equally critical. Unmanaged incidents can quickly escalate, leading to costly downtime and reputational damage. This guide delves into a sophisticated yet automated approach to ensuring operational excellence: integrating Prometheus for comprehensive metric collection and alerting, with PagerDuty for streamlined on-call management and incident response, all provisioned and managed declaratively using Terraform. By embracing Infrastructure as Code (IaC) principles, organizations can achieve unparalleled consistency, reduce manual errors, and significantly lower their Mean Time To Resolution (MTTR) for critical issues within their EKS environments, paving the way for truly resilient and observable systems.

Architecture Pro-Tip: Achieving true resilience in AWS EKS production environments demands a proactive, IaC-driven approach to observability. By managing your Prometheus monitoring stack and PagerDuty incident response workflow entirely through Terraform, you not only ensure consistency and repeatability but also enable GitOps principles, allowing your infrastructure to evolve reliably with your application code. This 'shift-left' on incident preparedness dramatically improves MTTR and operational confidence, transforming how your team responds to critical events.

Understanding the Core Components: Prometheus, Alertmanager, PagerDuty, and Terraform

Before diving into the implementation details, it's crucial to understand the role each technology plays in this integrated solution. Prometheus serves as the backbone of our monitoring strategy, acting as a powerful open-source systems monitoring and alerting toolkit. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if certain conditions are met. Its robust query language (PromQL) allows for complex data analysis. Complementing Prometheus is Alertmanager, a separate component designed to handle alerts sent by client applications like Prometheus. Alertmanager takes care of deduplicating, grouping, and routing alerts to the correct receiver integration, which in our case, will be PagerDuty. This prevents alert storms and ensures that only actionable notifications reach the on-call team.

PagerDuty steps in as the incident management system. It integrates with monitoring tools to ingest alerts, applies on-call schedules, escalates incidents through various notification channels (SMS, phone, email, push), and tracks the entire incident lifecycle from detection to resolution. Its capabilities streamline the process of getting the right alert to the right person at the right time, minimizing the impact of production issues. Finally, Terraform is our chosen Infrastructure as Code tool. It allows us to define and provision our entire infrastructure stack, including cloud resources on AWS EKS, Helm chart deployments for Prometheus and Alertmanager, and even the PagerDuty services and integrations, using a declarative configuration language. This ensures that our monitoring and incident response setup is version-controlled, reproducible, and easily auditable, bringing consistency and reliability to our operational practices.

Prerequisites and Initial Setup for Terraform and Providers

To embark on this journey, several prerequisites must be met. You'll need an active AWS account with appropriate IAM permissions to manage EKS, EC2, and other related services. An existing AWS EKS cluster is assumed for this guide; if you don't have one, Terraform can certainly provision that too, but it falls outside the immediate scope of this specific integration. On your local machine or CI/CD environment, ensure you have the Terraform CLI installed (version 1.0 or higher is recommended), `kubectl` configured to interact with your EKS cluster, and `helm` CLI for managing Kubernetes packages. Furthermore, a PagerDuty account is required, and you should have administrative access to create services, escalation policies, and retrieve API keys.

Your Terraform project will need to be initialized with the necessary providers. We'll be utilizing the AWS provider for any AWS-specific resources, the Kubernetes provider for interacting with the EKS cluster (e.g., creating namespaces), the Helm provider for deploying Prometheus and Alertmanager, and crucially, the PagerDuty provider to programmatically manage PagerDuty services and integrations. It's good practice to define your provider versions explicitly to ensure consistent deployments. Below is a foundational `versions.tf` and `providers.tf` configuration that sets up the required providers, preparing your environment for the subsequent infrastructure definitions. Remember to configure your AWS credentials appropriately, for instance, via environment variables or an AWS CLI profile, and provide your PagerDuty API token securely, perhaps through a Terraform variable or environment variable.

provider "aws" { region = var.aws_region } provider "kubernetes" { host = data.aws_eks_cluster.cluster.endpoint cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data) token = data.aws_eks_cluster_auth.cluster.token } provider "helm" { kubernetes { host = data.aws_eks_cluster.cluster.endpoint cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data) token = data.aws_eks_cluster_auth.cluster.token } } provider "pagerduty" { token = var.pagerduty_api_token } data "aws_eks_cluster" "cluster" { name = var.eks_cluster_name } data "aws_eks_cluster_auth" "cluster" { name = var.eks_cluster_name }

Deploying Prometheus and Alertmanager on EKS with Terraform

Once your Terraform providers are configured, the next logical step is to deploy Prometheus and Alertmanager onto your AWS EKS cluster. The most efficient and widely accepted method for deploying complex applications on Kubernetes is through Helm charts. The Prometheus community maintains official Helm charts that simplify the deployment process, allowing for extensive customization through `values.yaml` files. Terraform's Helm provider allows us to treat these Helm chart deployments as declarative resources, fully integrating them into our Infrastructure as Code workflow. We'll deploy the kube-prometheus-stack, which includes Prometheus, Alertmanager, Grafana, and various exporters, providing a comprehensive monitoring solution out-of-the-box.

Within our Terraform configuration, we'll define a `helm_release` resource for the `kube-prometheus-stack`. This resource will point to the Helm chart repository and specify the desired version. Crucially, we'll pass a `values` block that customizes the deployment. For production clusters, considerations like persistent storage for Prometheus data (using AWS EBS or EFS, for example), resource limits and requests for all components, and specific Alertmanager configuration will be paramount. The `values` block is where we can explicitly enable or disable components, configure scraping targets, and, most importantly for this guide, define the initial configuration for Alertmanager, including its integration with PagerDuty. Managing these values directly within Terraform ensures that your monitoring stack is always deployed consistently with your infrastructure.

resource "kubernetes_namespace" "monitoring" { metadata { name = "monitoring" } } resource "helm_release" "kube_prometheus_stack" { name = "kube-prometheus-stack" repository = "https://prometheus-community.github.io/helm-charts" chart = "kube-prometheus-stack" namespace = kubernetes_namespace.monitoring.metadata.0.name version = "57.0.3" # Use a specific chart version for stability values = [ yamlencode({ # Prometheus configuration prometheus = { prometheusSpec = { retention = "15d" storageSpec = { volumeClaimTemplate = { spec = { storageClassName = "gp2" # Or your preferred storage class resources = { requests = { storage = "50Gi" } } } } } } } # Alertmanager configuration - initial setup, details for PagerDuty integration follow alertmanager = { enabled = true config = { global = { resolve_timeout = "5m" } route = { group_by = ["alertname", "cluster", "namespace"] group_wait = "30s" group_interval = "5m" repeat_interval = "4h" receiver = "default-pagerduty" # This receiver will be defined later } receivers = [ { name = "default-pagerduty" # PagerDuty integration will be configured here # pagerduty_configs = [ # { # service_key = var.pagerduty_integration_key # } # ] } ] } } # Disable Grafana if you manage it separately or don't need it grafana = { enabled = false } # Other customizations like nodeSelectors, tolerations, resource limits etc. }) ] }

Configuring PagerDuty Services and API Keys with Terraform

With Prometheus and Alertmanager ready on EKS, our focus shifts to configuring PagerDuty to receive and manage the incoming alerts. Leveraging Terraform to manage PagerDuty resources brings the same benefits of automation and consistency to our incident response workflow as it does to our infrastructure. The PagerDuty provider allows us to define escalation policies, services, and service integrations directly within our Terraform code. This eliminates manual configuration steps in the PagerDuty UI, ensuring that our on-call setup is version-controlled and reproducible across environments.

The first step is often to define an escalation policy. This policy dictates the order in which on-call responders are notified when an incident occurs. For an EKS production cluster, you might want multiple layers of escalation, starting with your primary DevOps or SRE team and escalating to management if an alert remains unacknowledged. Next, we create a PagerDuty service. A service represents a component or application that your team is responsible for. For EKS monitoring, it's common to have a dedicated service like "EKS Production Cluster Alerts." This service will then be linked to our escalation policy. Finally, and most critically for the integration, we create a service integration within that PagerDuty service. This integration defines how an external monitoring tool (like Alertmanager) sends alerts to PagerDuty. For Prometheus/Alertmanager, we typically use the PagerDuty Events API v2 integration, which generates a unique integration key. This key is what Alertmanager will use to send alerts to the correct PagerDuty service.

# Define an Escalation Policy for EKS Alerts resource "pagerduty_escalation_policy" "eks_escalation_policy" { name = "EKS Production Cluster Escalation Policy" num_loops = 2 # Number of times to repeat the entire escalation policy # Example: Notify primary on-call team, then secondary, then management rule { escalation_delay_in_minutes = 5 # Wait 5 minutes before escalating to next target target { type = "user" id = var.pagerduty_primary_oncall_user_id # PagerDuty User ID } } rule { escalation_delay_in_minutes = 10 # Wait another 10 minutes target { type = "user" id = var.pagerduty_secondary_oncall_user_id # PagerDuty User ID } } # Add more rules or target teams/schedules as needed } # Define a PagerDuty Service for EKS Monitoring resource "pagerduty_service" "eks_monitoring_service" { name = "EKS Production Cluster Monitoring" auto_resolve_timeout = "14400" # 4 hours (in seconds) acknowledgement_timeout = "1800" # 30 minutes (in seconds) escalation_policy = pagerduty_escalation_policy.eks_escalation_policy.id } # Create a Service Integration for Alertmanager (Events API V2) resource "pagerduty_service_integration" "alertmanager_integration" { name = "Alertmanager Integration" service_id = pagerduty_service.eks_monitoring_service.id type = "generic_events_api_v2" # Recommended for Alertmanager } # Output the integration key to be used by Alertmanager output "pagerduty_integration_key" { value = pagerduty_service_integration.alertmanager_integration.integration_key description = "The PagerDuty integration key for Alertmanager to send alerts." sensitive = true }

Integrating Alertmanager with PagerDuty for On-Call Notifications

Now that Prometheus and Alertmanager are deployed and PagerDuty services are configured via Terraform, the pivotal step is to connect Alertmanager to PagerDuty. This is achieved by updating Alertmanager's configuration to define a receiver that utilizes the PagerDuty integration key we obtained in the previous step. Alertmanager's configuration is typically defined in a YAML file, which can be passed to the Helm chart as part of its `values`. The `alertmanager.config` block within the `kube-prometheus-stack` Helm release is where this magic happens.

Inside the `config` block, we define `receivers` and `routes`. A `receiver` specifies where alerts should be sent when they match a particular route. For PagerDuty, we'll create a receiver with a `pagerduty_configs` section, supplying the `service_key` (which is our PagerDuty integration key). The `route` block determines how alerts are matched and routed to specific receivers. You can create complex routing trees based on alert labels (e.g., severity, team, cluster). For simplicity, we'll configure a default route to send all EKS-related alerts to our PagerDuty receiver. It’s imperative to manage the `service_key` securely, ideally by referencing the Terraform output as shown and ensuring it's never hardcoded or exposed unnecessarily. This integration solidifies the automated incident response chain, ensuring that when Prometheus detects a problem, Alertmanager processes it, and PagerDuty immediately notifies the on-call team.

# Update the helm_release for kube_prometheus_stack # This block replaces or merges with the earlier alertmanager section in values resource "helm_release" "kube_prometheus_stack" { # ... other configuration ... values = [ yamlencode({ alertmanager = { enabled = true config = { global = { resolve_timeout = "5m" } route = { group_by = ["alertname", "cluster", "namespace", "severity"] group_wait = "30s" group_interval = "5m" repeat_interval = "4h" receiver = "default-pagerduty" # All alerts go here by default routes = [ # Example of more granular routing { match = { severity = "critical" } receiver = "critical-pagerduty" # Route critical alerts to a dedicated receiver }, { match = { severity = "warning" } receiver = "warning-slack" # Route warnings to a Slack channel, for instance } ] } receivers = [ { name = "default-pagerduty" pagerduty_configs = [ { service_key = var.pagerduty_integration_key } ] }, { name = "critical-pagerduty" pagerduty_configs = [ { service_key = var.pagerduty_integration_key # Could be a different integration key for critical alerts severity = "critical" } ] }, { name = "warning-slack" slack_configs = [ # Example of another receiver type { channel = "#eks-warnings" api_url = var.slack_webhook_url send_resolved = true } ] } ] } } # ... rest of the values ... }) ] }

Defining Advanced Alerting Rules and Best Practices

Effective monitoring goes beyond simply deploying tools; it requires well-defined alerting rules that are both timely and actionable. With Prometheus, we can define powerful alerting rules using PromQL, which are then evaluated by the Prometheus server. When an alert condition is met, Prometheus sends an alert to Alertmanager, initiating the incident response workflow. It's essential to define rules that catch critical issues without overwhelming your on-call team with noise. Best practices suggest focusing on symptoms rather than causes (e.g., "high latency" instead of "CPU utilization on host X is high") and adhering to the "four golden signals" of monitoring: latency, traffic, errors, and saturation.

Alerting rules are typically grouped into rule files (e.g., `kubernetes.rules`, `application.rules`) and loaded by Prometheus. These rules can be managed within your Terraform deployment as Kubernetes ConfigMaps that the Prometheus server is configured to load. This keeps your alerting logic version-controlled alongside your infrastructure. Common EKS-specific alerts might include node failures, pod restarts, API server unavailability, high resource utilization (CPU, memory, disk), and application-specific error rates. Remember to use the `for` clause in your rules to prevent flapping alerts, ensuring an alert is sustained for a minimum duration before firing. Assigning appropriate severity labels (critical, warning) allows Alertmanager to route them effectively to PagerDuty or other notification channels, as configured in the previous step. Below is an example of how you might define a Kubernetes ConfigMap for Prometheus alerting rules, which Prometheus would then discover and load.

resource "kubernetes_config_map" "prometheus_rules" { metadata { name = "eks-prometheus-rules" namespace = kubernetes_namespace.monitoring.metadata.0.name labels = { "prometheus.k8s.io/rule" = "true" # Label to ensure Prometheus operator picks it up "release" = helm_release.kube_prometheus_stack.name } } data = { "kubernetes.rules" = < 5 for: 1m labels: severity: critical annotations: summary: "Pod Crashlooping ({{ $labels.namespace }}/{{ $labels.pod }})" description: "Pod {{ $labels.pod }} in namespace {{ $labels.namespace }} is crashlooping." EOF } } # Ensure Prometheus picks up these rules by configuring its scrape config or rule files # This is typically handled by the kube-prometheus-stack Helm chart if labels match, # or you may need to explicitly mount this ConfigMap into the Prometheus pod configuration.

Conclusion and Next Steps for Enhanced Observability

By following this comprehensive guide, you have successfully established a robust, Terraform-managed monitoring and incident response system for your AWS EKS production clusters. We've walked through the deployment of Prometheus and Alertmanager on Kubernetes using Helm charts, configured PagerDuty services and escalation policies, and meticulously integrated Alertmanager with PagerDuty for reliable on-call notifications—all through the declarative power of Infrastructure as Code. This approach not only automates the setup process but also guarantees consistency, reduces human error, and provides a version-controlled, auditable record of your observability infrastructure. The ability to manage these critical components alongside your core infrastructure code brings immense operational efficiency and confidence to your DevOps practices.

However, the journey towards ultimate observability and resilience is continuous. Consider these next steps to further enhance your system:

• Visualizations with Grafana: While not fully covered, the `kube-prometheus-stack` includes Grafana. Leverage it to build rich dashboards for real-time visualization of your EKS metrics and alert trends, providing deeper insights into cluster health.
• Advanced Alertmanager Routing: Explore more complex Alertmanager routing rules to direct specific alerts to different PagerDuty services, Slack channels, or webhooks based on team ownership, severity, or affected components.
• Runbook Automation: Integrate PagerDuty with runbook automation tools to automatically execute diagnostic commands or remediation steps for common, well-understood incidents.
• Cost Optimization: Continuously monitor the resource consumption of your monitoring stack and optimize Prometheus storage, retention policies, and resource requests to balance performance and cost efficiency.
• Security Best Practices: Ensure all components are secured with appropriate RBAC, network policies, and encrypted communication channels, especially for sensitive PagerDuty API keys and monitoring data.

By continually iterating on your monitoring and incident response strategy, you can build an even more resilient, self-healing, and observable AWS EKS environment, minimizing downtime and maximizing operational efficiency.