Terraform-Managed AWS EKS Prometheus Monitoring and PagerDuty Incident Automation

In the dynamic landscape of cloud-native applications, maintaining high availability and rapid incident response is paramount. This guide provides a comprehensive, technical walkthrough on establishing a robust monitoring and alerting solution for AWS EKS clusters using Prometheus, managed entirely by Terraform, and integrating it with PagerDuty for streamlined incident automation.

Architecture Pro-Tip: For mission-critical environments, consider deploying Prometheus and Alertmanager in a highly available configuration across multiple availability zones within your EKS cluster. Leverage external storage solutions like AWS S3 with Thanos for long-term metric retention and global query views, ensuring your observability stack is as resilient as your applications.

Why Terraform for EKS Monitoring?

Terraform, as an Infrastructure as Code (IaC) tool, offers unparalleled advantages in managing complex cloud environments. By codifying your Prometheus and Alertmanager deployments for AWS EKS, you gain:

Consistency: Ensure identical monitoring setups across development, staging, and production environments.
Version Control: Track changes, roll back configurations, and collaborate effectively using standard VCS practices.
Automation: Automate the deployment and scaling of your monitoring stack, reducing manual errors and operational overhead.
Scalability: Easily replicate or scale your monitoring infrastructure as your EKS footprint grows.

Understanding the Monitoring Stack Components

This solution leverages a powerful combination of open-source tools and cloud services:

AWS EKS: The managed Kubernetes service providing a robust and scalable platform for containerized applications.
Prometheus: An open-source monitoring system with a powerful data model and query language (PromQL) for collecting and storing time-series metrics.
Grafana: A popular open-source platform for analytics and interactive visualization, used to create dashboards from Prometheus data.
Alertmanager: Handles alerts sent by client applications like Prometheus. It deduces, groups, routes, and sends notifications to the correct receiver.
PagerDuty: An incident management platform that aggregates alerts from various monitoring systems, automates incident response, and facilitates on-call scheduling and communication.

Prerequisites

Before you begin, ensure you have the following:

An active AWS account with appropriate permissions.
AWS CLI configured.
Terraform (v1.0+) installed.
Kubectl installed and configured to connect to your EKS cluster.
Helm (v3+) installed.
A PagerDuty account with a service created and an integration key obtained.
An existing AWS EKS cluster. This guide assumes the EKS cluster and its node groups are already provisioned.

Step-by-Step Implementation

1. Configure AWS IAM for EKS and Kubernetes Service Accounts

Prometheus components will need permissions to interact with AWS resources (e.g., EBS for persistent storage, S3 for Thanos if expanded). We'll use IAM Roles for Service Accounts (IRSA) for secure access.

2. Deploy Prometheus and Grafana via Helm with Terraform

We'll leverage the official Prometheus community Helm chart, which bundles Prometheus, Alertmanager, and an optional Grafana instance.

3. Configure Alertmanager for PagerDuty Integration

The Alertmanager configuration will define how alerts are routed and to which receivers (e.g., PagerDuty).

4. Define Prometheus Rules and Alerts

Create Prometheus recording and alerting rules to monitor your EKS cluster and applications. These rules will fire alerts to Alertmanager when conditions are met.

Comprehensive Terraform Configuration Example

Below is a ready-to-use Terraform configuration demonstrating how to deploy the Prometheus stack with PagerDuty integration into an existing EKS cluster. Replace placeholders with your actual values.

resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = "monitoring"
  }
}

resource "helm_release" "kube_prometheus_stack" {
  name       = "kube-prometheus-stack"
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  namespace  = kubernetes_namespace.monitoring.metadata[0].name
  version    = "58.1.0" # Use a stable, recent version

  set {
    name  = "kubeControllerManager.enabled"
    value = "false"
  }
  set {
    name  = "kubeEtcd.enabled"
    value = "false"
  }
  set {
    name  = "kubeScheduler.enabled"
    value = "false"
  }
  set {
    name  = "grafana.enabled"
    value = "true"
  }
  set {
    name  = "grafana.service.type"
    value = "LoadBalancer"
  }
  set {
    name  = "alertmanager.enabled"
    value = "true"
  }
  set {
    name  = "alertmanager.alertmanagerSpec.storage.volumeClaimTemplate.spec.resources.requests.storage"
    value = "10Gi"
  }
  set {
    name  = "prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage"
    value = "50Gi"
  }

  values = [
    templatefile("${path.module}/templates/alertmanager-values.yaml", {
      pagerduty_service_key = var.pagerduty_service_key
    }),
    templatefile("${path.module}/templates/prometheus-rules.yaml", {})
  ]
}

# Define your Alertmanager values in templates/alertmanager-values.yaml
# ---
# alertmanager:
#   config:
#     global:
#       resolve_timeout: 5m
#     route:
#       receiver: 'pagerduty-receiver'
#       group_by: ['alertname', 'cluster', 'service']
#       group_wait: 30s
#       group_interval: 5m
#       repeat_interval: 1h
#       routes:
#         - receiver: 'pagerduty-receiver'
#           match:
#             severity: 'critical'
#     receivers:
#       - name: 'pagerduty-receiver'
#         pagerduty_configs:
#           - service_key: "{{ .pagerduty_service_key }}"
#             severity: '{{ .CommonLabels.severity }}'
#             description: '{{ .CommonLabels.alertname }} on {{ .CommonLabels.instance }}'
#             details:
#               summary: '{{ .CommonAnnotations.summary }}'
#               description: '{{ .CommonAnnotations.description }}'
#               cluster: '{{ .CommonLabels.cluster }}'
#               namespace: '{{ .CommonLabels.namespace }}'
#               pod: '{{ .CommonLabels.pod }}'
#
# Define your Prometheus rules in templates/prometheus-rules.yaml
# ---
# prometheus:
#   additionalPrometheusRules:
#     - name: kubernetes-apps
#       groups:
#         - name: k8s.rules
#           rules:
#           - alert: HighPodRestarts
#             expr: sum(increase(kube_pod_container_status_restarts_total{namespace="default"}[10m])) by (pod, namespace) > 5
#             for: 5m
#             labels:
#               severity: critical
#               cluster: "${var.eks_cluster_name}"
#             annotations:
#               summary: "Pod {{ $labels.pod }} in {{ $labels.namespace }} is restarting frequently"
#               description: "{{ $labels.pod }} has restarted {{ $value }} times in the last 10 minutes."
#           - alert: KubeApiServerLatencyHigh
#             expr: histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket{job="kubernetes-apiservers",verb=~"GET|PUT|POST|DELETE"}[5m])) by (job, verb, le)) > 0.5
#             for: 10m
#             labels:
#               severity: critical
#               cluster: "${var.eks_cluster_name}"
#             annotations:
#               summary: "High API Server latency for {{ $labels.verb }} requests"
#               description: "The 99th percentile latency for {{ $labels.verb }} requests to the Kubernetes API server has been above 0.5s for 10 minutes."
#
# Variables for Terraform
# variable "eks_cluster_name" {
#   description = "The name of your EKS cluster."
#   type        = string
# }
#
# variable "pagerduty_service_key" {
#   description = "PagerDuty integration key for the service."
#   type        = string
#   sensitive   = true
# }
    

Note: The YAML content for alertmanager-values.yaml and prometheus-rules.yaml shown within the code block are templates to be placed in a templates/ subdirectory relative to your Terraform module. Terraform's templatefile function will render these, substituting variables like pagerduty_service_key.

Deployment Steps

Save the Terraform code: Create a main.tf file with the above resource block. Create a variables.tf for your variables.
Create templates: Create a templates/ directory and place alertmanager-values.yaml and prometheus-rules.yaml inside it, using the example YAML provided.
Initialize Terraform: Run terraform init in your project directory.
Review the plan: Run terraform plan -var="pagerduty_service_key=YOUR_PAGERDUTY_KEY" -var="eks_cluster_name=YOUR_EKS_CLUSTER_NAME" to see the changes Terraform will apply.
Apply the configuration: Execute terraform apply -var="pagerduty_service_key=YOUR_PAGERDUTY_KEY" -var="eks_cluster_name=YOUR_EKS_CLUSTER_NAME". Confirm with yes.

Verifying the Setup

Kubernetes Pods: Check if Prometheus, Alertmanager, and Grafana pods are running in the monitoring namespace:
kubectl get pods -n monitoring
Grafana Access: Obtain the Grafana LoadBalancer URL:
kubectl get svc -n monitoring kube-prometheus-stack-grafana -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
Access Grafana using the default credentials (admin/prom-operator).
Alertmanager UI: Similarly, get the Alertmanager LoadBalancer URL:
kubectl get svc -n monitoring kube-prometheus-stack-alertmanager -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
Verify PagerDuty receiver configuration.
Test an Alert: Manually trigger a test alert in Prometheus or simulate a condition to verify PagerDuty integration.

Best Practices for Production Environments

Persistent Storage: Ensure Prometheus and Alertmanager use persistent volumes (e.g., AWS EBS CSI Driver) for state retention across restarts.
Resource Limits: Set appropriate CPU and memory limits for all monitoring components to prevent resource exhaustion.
Network Security: Restrict access to Grafana and Alertmanager UIs using AWS Security Groups, Network ACLs, or Kubernetes Ingress controllers with authentication.
High Availability: Deploy multiple replicas of Prometheus and Alertmanager for redundancy.
Externalized Configuration: For sensitive data like PagerDuty keys, use AWS Secrets Manager or Kubernetes Secrets and reference them securely in Terraform.
Thanos Integration: For large-scale EKS deployments, integrate Thanos with Prometheus for long-term storage, global query view, and high availability.

Troubleshooting Common Issues

Alerts Not Firing:
- Check Prometheus logs for rule evaluation errors:
  kubectl logs -f -n monitoring prometheus-kube-prometheus-stack-prometheus-0
- Verify Alertmanager configuration in its UI.
PagerDuty Incidents Not Created:
- Ensure the PagerDuty integration key is correct and assigned to the right service.
- Check Alertmanager logs for any errors communicating with PagerDuty.
- Verify network connectivity from EKS to PagerDuty API endpoints.
Prometheus Data Gaps:
- Check Prometheus targets status in its UI (/targets endpoint).
- Inspect logs of kube-state-metrics and node-exporter for issues.

Conclusion

Implementing a Terraform-managed Prometheus monitoring solution for AWS EKS, complete with PagerDuty incident automation, establishes a robust and reliable observability foundation. This approach not only streamlines operations through Infrastructure as Code but also ensures critical issues are identified and addressed swiftly, significantly enhancing the reliability and performance of your cloud-native applications. By following this guide, you can confidently deploy and manage an advanced monitoring stack tailored for your EKS environments, empowering your DevOps teams with actionable insights and automated incident response.

CloudStack YAML