Terraform for AWS EKS: Automated Prometheus Monitoring & PagerDuty Incident Response

In the dynamic landscape of cloud-native applications, maintaining robust observability and swift incident response for Kubernetes clusters is paramount. AWS EKS provides a managed Kubernetes service, but the responsibility for monitoring and alerting often falls to the user. This guide demonstrates how to leverage Terraform Infrastructure as Code (IaC) to deploy a comprehensive monitoring solution using Prometheus on AWS EKS, integrated with PagerDuty for automated incident management.

Architecture Pro-Tip: Modular Design & Secrets Management

For production environments, consider adopting a modular Terraform structure. Separate your EKS cluster definition, monitoring components, and application deployments into distinct modules. Furthermore, never hardcode sensitive information like PagerDuty API keys. Utilize AWS Secrets Manager or HashiCorp Vault, retrieved securely via Terraform data sources, to manage and inject secrets into your Kubernetes deployments, enhancing both security and maintainability.

Why Automated Monitoring and Incident Response?

Automating your monitoring and incident response stack offers numerous benefits:

Consistency and Reproducibility: Terraform ensures your monitoring setup is identical across all environments, eliminating configuration drift.
Speed and Efficiency: Deploy complex monitoring systems in minutes, not hours, allowing teams to focus on development rather than operational overhead.
Reliability: Automating the setup reduces human error, leading to more stable and reliable monitoring.
Scalability: Easily scale your monitoring solution as your EKS clusters and application footprint grow.
Proactive Incident Management: PagerDuty ensures critical alerts reach the right on-call personnel immediately, minimizing downtime and business impact.

Core Components Explained

1. AWS EKS (Elastic Kubernetes Service)

The managed Kubernetes service from AWS that hosts our containerized applications. We'll assume an existing EKS cluster or a cluster provisioned by a separate Terraform module for this guide.

2. Prometheus

An open-source monitoring system and time-series database. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true.

Prometheus Server: The core component that scrapes metrics.
Node Exporter: Collects host-level metrics from EKS worker nodes.
Kube-state-metrics: Generates metrics about the state of Kubernetes objects (e.g., deployments, pods, nodes).
Prometheus Operator: Simplifies the deployment and management of Prometheus and related components within Kubernetes.

3. Alertmanager

Handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration, such as email, Slack, or PagerDuty.

4. PagerDuty

An incident management platform that aggregates alerts from various monitoring tools, applies on-call schedules, and notifies the right team members through multiple channels (SMS, phone calls, email, push notifications) until an alert is acknowledged.

Prerequisites

Before you begin, ensure you have the following installed and configured:

Terraform CLI: Version 1.0 or higher.
AWS CLI: Configured with appropriate credentials and default region to interact with your EKS cluster.
kubectl: Configured to connect to your EKS cluster. You can update your kubeconfig using aws eks update-kubeconfig --name your-eks-cluster-name --region your-aws-region.
PagerDuty Account: With an existing service and a generated "Events API v2" integration key for that service.

Architecture Overview

Our architecture involves:

Terraform deploys the kube-prometheus-stack Helm chart onto the EKS cluster.
The Helm chart installs Prometheus, Alertmanager, Grafana, and other monitoring components.
Terraform also creates a Kubernetes Secret containing the Alertmanager configuration, including the PagerDuty receiver details.
Prometheus scrapes metrics from EKS components and applications.
When a Prometheus alert rule fires, it sends an alert to Alertmanager.
Alertmanager processes the alert, groups it, and routes it to PagerDuty using the configured integration key.
PagerDuty initiates an incident and notifies the on-call team.

Automating with Terraform: Ready-to-Use Configuration

Let's create the Terraform configuration files. We'll use a simple structure for this example. Create a directory named eks-monitoring and place the following files within it:

1. `main.tf`

This file defines our providers, data sources for the EKS cluster, the Helm release for kube-prometheus-stack, and the Kubernetes Secret for Alertmanager configuration.

provider "aws" {
  region = var.aws_region
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}

# Data source for the EKS cluster
data "aws_eks_cluster" "cluster" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = var.eks_cluster_name
}

# Create a Kubernetes Namespace for monitoring components
resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = "monitoring"
  }
}

# Deploy kube-prometheus-stack using Helm
resource "helm_release" "kube_prometheus_stack" {
  name       = "kube-prometheus-stack"
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  namespace  = kubernetes_namespace.monitoring.metadata.0.name

  # Custom values for the Helm chart
  values = [
    "${file("prometheus_values.yaml")}"
  ]

  set {
    name  = "alertmanager.config.externalUrl"
    value = "http://kube-prometheus-stack-alertmanager.monitoring.svc.cluster.local" # Internal URL for Alertmanager
  }
  set {
    name  = "prometheus.prometheusSpec.externalUrl"
    value = "http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local" # Internal URL for Prometheus
  }
}

# Kubernetes Secret for Alertmanager configuration
# This uses a template to inject the PagerDuty key
resource "kubernetes_secret" "alertmanager_config" {
  metadata {
    name      = "alertmanager-kube-prometheus-stack-alertmanager" # Must match the name expected by the Helm chart
    namespace = kubernetes_namespace.monitoring.metadata.0.name
  }

  data = {
    "alertmanager.yaml" = base64encode(templatefile("${path.module}/alertmanager_config.yaml", {
      pagerduty_integration_key = var.pagerduty_integration_key
    }))
  }

  type = "Opaque"
  depends_on = [
    helm_release.kube_prometheus_stack
  ]
}

# Example PrometheusRule for EKS Node CPU usage
resource "kubernetes_manifest" "node_cpu_high_alert" {
  manifest = {
    apiVersion = "monitoring.coreos.com/v1"
    kind       = "PrometheusRule"
    metadata = {
      name      = "eks-node-cpu-high"
      namespace = kubernetes_namespace.monitoring.metadata.0.name
      labels = {
        app_kubernetes_io/name       = "kube-prometheus-stack"
        app_kubernetes_io/instance   = "kube-prometheus-stack"
        prometheus                  = "kube-prometheus-stack"
        role                        = "alert-rules"
      }
    }
    spec = {
      groups = [
        {
          name = "node.rules"
          rules = [
            {
              alert = "KubeNodeCPUHigh"
              expr  = "(1 - sum(rate(node_cpu_seconds_total{mode='idle'}[5m])) by (instance)) * 100 > 80"
              for   = "5m"
              labels = {
                severity = "warning"
              }
              annotations = {
                summary     = "Node {{ $labels.instance }} CPU usage is high"
                description = "Node CPU usage for {{ $labels.instance }} is over 80% for 5 minutes."
              }
            },
          ]
        }
      ]
    }
  }
  depends_on = [
    helm_release.kube_prometheus_stack
  ]
}

2. `variables.tf`

Define input variables for our Terraform configuration.

variable "aws_region" {
  description = "AWS region for the EKS cluster"
  type        = string
  default     = "us-east-1" # Change to your region
}

variable "eks_cluster_name" {
  description = "Name of the existing AWS EKS cluster"
  type        = string
}

variable "pagerduty_integration_key" {
  description = "PagerDuty Events API v2 Integration Key"
  type        = string
  sensitive   = true # Mark as sensitive to prevent logging in plain text
}

3. `prometheus_values.yaml`

Custom values file for the kube-prometheus-stack Helm chart. We'll enable some components and ensure Alertmanager is configured to use our external secret for its configuration.

# Minimal values for kube-prometheus-stack
# Adjust these based on your specific requirements

defaultRules:
  create: true
  rules:
    alertmanager: true
    etcd: true
    general: true
    k8s: true
    kubeApiserver: true
    kubeControllerManager: true
    kubeScheduler: true
    kubeStateMetrics: true
    kubelet: true
    node: true
    prometheus: true
    prometheusOperator: true

prometheus:
  enabled: true
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false
    scrapeInterval: 30s
    evaluationInterval: 30s
    retention: 15d
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: gp2 # Or your preferred storage class
          resources:
            requests:
              storage: 50Gi

alertmanager:
  enabled: true
  # Disable default Alertmanager configuration to use our custom secret
  # We will manually inject the configuration via the 'alertmanager-kube-prometheus-stack-alertmanager' Secret resource
  config:
    # We will provide our own alertmanager.yaml via Kubernetes Secret
    # This prevents the chart from generating its own config and potentially overriding ours
    # However, to explicitly use an external secret, ensure the Helm chart's Alertmanager deployment
    # mounts this external secret correctly. The `kubernetes_secret` name MUST match the one expected
    # by the Helm chart. For kube-prometheus-stack, it defaults to 'alertmanager--alertmanager'.
    # Our `kubernetes_secret` resource is named accordingly.
    # No values needed here as it's provided externally.

grafana:
  enabled: true
  adminPassword: "your_secure_grafana_password" # Change this!
  service:
    type: ClusterIP # Use LoadBalancer if you want external access easily
  # Add more Grafana settings like dashboard import, datasources here

kubeControllerManager:
  enabled: true
kubeScheduler:
  enabled: true
kubeProxy:
  enabled: true
kubeEtcd:
  enabled: true
kubeStateMetrics:
  enabled: true
nodeExporter:
  enabled: true

4. `alertmanager_config.yaml`

This is the template for our Alertmanager configuration, including the PagerDuty receiver.

global:
  resolve_timeout: 5m

route:
  receiver: 'pagerduty-receiver'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  routes:
    # Send all critical alerts to PagerDuty immediately
    - match:
        severity: critical
      receiver: 'pagerduty-receiver'
      continue: true
    # You can add more specific routes here for different teams or services

receivers:
  - name: 'pagerduty-receiver'
    pagerduty_configs:
      - service_key: "${pagerduty_integration_key}"
        url: 'https://events.pagerduty.com/v2/enqueue' # PagerDuty Events API v2 URL

Deployment Steps

Follow these steps to deploy your automated monitoring solution:

1. Initialize Terraform

Navigate to your eks-monitoring directory and initialize Terraform:

terraform init

2. Plan the Deployment

Review the changes Terraform plans to make. Replace placeholders with your actual EKS cluster name and PagerDuty key.

terraform plan \
  -var="eks_cluster_name=YOUR_EKS_CLUSTER_NAME" \
  -var="pagerduty_integration_key=YOUR_PAGERDUTY_INTEGRATION_KEY" \
  -var="aws_region=your-aws-region"

Note: For a production setup, consider passing sensitive variables via environment variables (e.g., TF_VAR_pagerduty_integration_key) or a secure backend.

3. Apply the Configuration

Execute the deployment. Type yes when prompted.

terraform apply \
  -var="eks_cluster_name=YOUR_EKS_CLUSTER_NAME" \
  -var="pagerduty_integration_key=YOUR_PAGERDUTY_INTEGRATION_KEY" \
  -var="aws_region=your-aws-region"

Verification and Testing

1. Check Kubernetes Pods

Verify that all monitoring components are running in the monitoring namespace:

kubectl get pods -n monitoring

You should see pods for Prometheus, Alertmanager, Grafana, Kube-state-metrics, etc., in a Running state.

2. Access Grafana Dashboard

Port-forward to the Grafana service to access its UI:

kubectl port-forward svc/kube-prometheus-stack-grafana -n monitoring 3000:80

Then, open http://localhost:3000 in your browser. Log in with admin and the password you set in prometheus_values.yaml.

3. Verify Alertmanager Configuration

Port-forward to the Alertmanager service:

kubectl port-forward svc/kube-prometheus-stack-alertmanager -n monitoring 9093:9093

Access http://localhost:9093. Navigate to the "Status" tab and then "Receivers" to confirm your PagerDuty receiver is listed and correctly configured.

4. Test PagerDuty Integration

To test the PagerDuty integration, you can deliberately trigger an alert. For instance, scale down a critical deployment to 0 replicas, which might trigger a KubeDeploymentReplicasMismatch alert (if you have such a rule). Alternatively, you can temporarily modify a Prometheus rule to fire immediately for a simple condition, then revert it.

Once an alert fires and is routed through Alertmanager to PagerDuty, you should see an incident created in your PagerDuty service, and on-call team members will be notified.

Troubleshooting and Best Practices

Common Issues:

kubectl/Terraform authentication errors: Ensure your AWS CLI credentials are valid and kubeconfig is updated for your EKS cluster.
Helm Chart Installation Failures: Check Helm release logs for errors (helm history -n monitoring kube-prometheus-stack, helm get values -n monitoring kube-prometheus-stack, kubectl get events -n monitoring).
Alerts not reaching PagerDuty:
- Verify the pagerduty_integration_key is correct and belongs to an Events API v2 integration.
- Check Alertmanager logs for errors related to sending notifications.
- Ensure your Alertmanager configuration (alertmanager_config.yaml) is correctly applied via the Kubernetes Secret.
- Validate Prometheus rules are firing correctly by checking the Prometheus UI (http://localhost:9090/alerts after port-forwarding).
Resource Constraints: Prometheus can be resource-intensive. Ensure your EKS nodes have sufficient CPU and memory, and consider tuning Prometheus storage and retention settings.

Best Practices:

Version Control: Keep all your Terraform code in a Git repository.
Terraform State Management: Use a remote backend like AWS S3 with DynamoDB locking for production environments.
Secret Management: As mentioned, use AWS Secrets Manager or Vault for all sensitive data.
Custom Prometheus Rules: Develop custom Prometheus rules tailored to your application's specific SLOs (Service Level Objectives) and error budgets.
Granular PagerDuty Routing: Configure multiple PagerDuty services and integration keys, and use Alertmanager routing rules to direct alerts to the most appropriate on-call teams.
Regular Review: Periodically review your alert rules, thresholds, and PagerDuty schedules to ensure they remain relevant and effective.

Conclusion

By leveraging Terraform, you can fully automate the deployment and management of a robust Prometheus monitoring solution on AWS EKS, seamlessly integrated with PagerDuty for critical incident response. This approach not only streamlines operations but also ensures consistency, reliability, and scalability for your cloud-native applications. Embracing Infrastructure as Code for observability is a fundamental step towards building resilient and high-performing systems in the modern cloud environment.

Ready to elevate your EKS monitoring? Start by adapting this guide to your specific environment and discover the power of automated, proactive incident management.

CloudStack YAML