Terraform AWS EKS Production Alerting with Prometheus and PagerDuty

In the dynamic landscape of modern cloud infrastructure, ensuring high availability and rapid incident response for Kubernetes clusters is paramount. This comprehensive guide details how to establish a robust, production-grade alerting system for AWS EKS using Terraform for Infrastructure as Code (IaC), Prometheus for monitoring, and PagerDuty for incident management. By automating the deployment of these critical components, you can achieve consistent, scalable, and reliable alerting across your EKS environments.

Architecture Pro-Tip: Idempotent & Declarative Monitoring

Always strive for an idempotent and declarative monitoring setup. Using Terraform ensures that your entire alerting stack, from EKS cluster configuration to Prometheus and Alertmanager deployments, is version-controlled and reproducible. This approach minimizes configuration drift, simplifies disaster recovery, and allows for consistent deployments across development, staging, and production environments, significantly reducing operational overhead and increasing system reliability.

Prerequisites

Before diving into the implementation, ensure you have the following tools and access configured:

AWS Account: With administrative privileges to create EKS clusters, VPCs, IAM roles, and other necessary resources.
Terraform CLI: Version 1.0.0 or higher installed.
AWS CLI: Configured with appropriate credentials and a default region.
Kubectl: Installed and configured to interact with Kubernetes clusters.
Helm CLI: Version 3.x or higher installed, used for deploying Prometheus and Alertmanager.
PagerDuty Account: With an existing service and a generated integration key (or you can create one during setup).

Core Components Explained

AWS EKS (Elastic Kubernetes Service)

AWS EKS is a managed Kubernetes service that makes it easy to run Kubernetes on AWS without needing to install, operate, and maintain your own Kubernetes control plane. It provides a highly available and scalable control plane across multiple availability zones.

Prometheus

Prometheus is an open-source monitoring system with a dimensional data model, flexible query language (PromQL), efficient time-series database, and a modern alerting solution. It scrapes metrics from configured targets at specified intervals, evaluates rule expressions, and can trigger alerts.

Alertmanager

Alertmanager handles alerts sent by client applications like the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration, such as email, Slack, or PagerDuty. It also supports silencing and inhibition of alerts.

PagerDuty

PagerDuty is an incident management platform that helps teams detect and resolve incidents faster. It consolidates alerts from various monitoring tools, intelligently routes them to the right on-call personnel, and provides robust capabilities for incident response, on-call scheduling, and post-incident analysis.

Step-by-Step Implementation with Terraform

1. Project Setup and Provider Configuration

Start by setting up your Terraform project directory and defining the AWS and Kubernetes providers.

2. Deploying AWS EKS Cluster

We'll use the popular terraform-aws-modules/eks/aws module for a streamlined EKS deployment. This module handles the creation of the VPC, subnets, IAM roles, and the EKS cluster itself, including worker nodes.

3. Installing Prometheus and Alertmanager on EKS

The most effective way to deploy Prometheus and Alertmanager on Kubernetes is via the kube-prometheus-stack Helm chart. This chart provides a complete monitoring solution, including Prometheus, Alertmanager, Grafana, and various Kubernetes exporters.

We will use Terraform's Helm provider to deploy this chart. The key is to configure Alertmanager within the Helm values to integrate with PagerDuty.

4. Integrating with PagerDuty

To integrate Alertmanager with PagerDuty, you need a PagerDuty service with an integration key. This key will be supplied to Alertmanager's configuration, allowing it to send alerts directly to your PagerDuty service. Ensure you create a generic API service integration in PagerDuty to obtain this key.

5. Defining Alerting Rules

Prometheus uses PrometheusRule Kubernetes custom resources to define alerting rules. These rules evaluate metric expressions and, if conditions are met, send alerts to Alertmanager. Below, we'll provide an example rule for a common scenario.

Ready-to-Use Configuration

Below is a comprehensive Terraform configuration that orchestrates the deployment of an AWS EKS cluster, installs the kube-prometheus-stack via Helm, and configures Alertmanager for PagerDuty integration. Remember to replace placeholder values like <YOUR_PAGERDUTY_ROUTING_KEY>, <YOUR_AWS_REGION>, etc., with your actual production values.

resource "aws_vpc" "eks_vpc" {
  cidr_block = "10.0.0.0/16"
  tags = {
    Name = "eks-prod-alerting-vpc"
  }
}

resource "aws_subnet" "public_subnets" {
  count             = 2
  vpc_id            = aws_vpc.eks_vpc.id
  cidr_block        = "10.0.${count.index}.0/24"
  availability_zone = data.aws_availability_zones.available.names[count.index]
  map_public_ip_on_launch = true
  tags = {
    Name                                    = "eks-prod-alerting-public-${count.index}"
    "kubernetes.io/cluster/eks-prod-cluster" = "shared"
    "kubernetes.io/role/elb"                 = "1"
  }
}

resource "aws_internet_gateway" "eks_igw" {
  vpc_id = aws_vpc.eks_vpc.id
  tags = {
    Name = "eks-prod-alerting-igw"
  }
}

resource "aws_route_table" "public_route_table" {
  vpc_id = aws_vpc.eks_vpc.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.eks_igw.id
  }
}

resource "aws_route_table_association" "public_route_table_association" {
  count          = length(aws_subnet.public_subnets)
  subnet_id      = aws_subnet.public_subnets[count.index].id
  route_table_id = aws_route_table.public_route_table.id
}

resource "aws_security_group" "eks_cluster_sg" {
  vpc_id      = aws_vpc.eks_vpc.id
  name        = "eks-prod-cluster-sg"
  description = "Security group for EKS cluster"

  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
    description = "Allow inbound HTTPS for EKS API server"
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_iam_role" "eks_cluster_role" {
  name = "eks-prod-cluster-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "eks.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eks_cluster_role.name
}

resource "aws_iam_role_policy_attachment" "eks_service_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
  role       = aws_iam_role.eks_cluster_role.name
}

resource "aws_eks_cluster" "eks_cluster" {
  name     = "eks-prod-cluster"
  role_arn = aws_iam_role.eks_cluster_role.arn
  vpc_config {
    subnet_ids         = aws_subnet.public_subnets[*].id
    security_group_ids = [aws_security_group.eks_cluster_sg.id]
  }

  depends_on = [
    aws_iam_role_policy_attachment.eks_cluster_policy,
    aws_iam_role_policy_attachment.eks_service_policy,
  ]
}

resource "aws_iam_role" "eks_node_group_role" {
  name = "eks-prod-node-group-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_node_group_policy_cni" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.eks_node_group_role.name
}

resource "aws_iam_role_policy_attachment" "eks_node_group_policy_worker" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.eks_node_group_role.name
}

resource "aws_iam_role_policy_attachment" "eks_node_group_policy_registry" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.eks_node_group_role.name
}

resource "aws_eks_node_group" "eks_node_group" {
  cluster_name    = aws_eks_cluster.eks_cluster.name
  node_group_name = "eks-prod-node-group"
  node_role_arn   = aws_iam_role.eks_node_group_role.arn
  subnet_ids      = aws_subnet.public_subnets[*].id
  instance_types  = ["t3.medium"]

  scaling_config {
    desired_size = 2
    max_size     = 3
    min_size     = 1
  }

  depends_on = [
    aws_iam_role_policy_attachment.eks_node_group_policy_cni,
    aws_iam_role_policy_attachment.eks_node_group_policy_worker,
    aws_iam_role_policy_attachment.eks_node_group_policy_registry,
  ]
}

resource "helm_release" "prometheus_stack" {
  name       = "prometheus-stack"
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  namespace  = "monitoring"
  create_namespace = true

  set {
    name  = "kube-state-metrics.enabled"
    value = "true"
  }
  set {
    name  = "nodeExporter.enabled"
    value = "true"
  }
  set {
    name  = "grafana.enabled"
    value = "true"
  }

  values = [
    < # Replace with your PagerDuty integration key
              severity: 'critical' # or 'warning', 'info' based on alert
              details:
                cluster_name: "eks-prod-cluster"
                namespace: "{{ .CommonLabels.namespace }}"
                alertname: "{{ .CommonLabels.alertname }}"
                severity: "{{ .CommonLabels.severity }}"
                instance: "{{ .CommonLabels.instance }}"
                summary: "{{ .CommonAnnotations.summary }}"
                description: "{{ .CommonAnnotations.description }}"
EOF
  ]

  depends_on = [
    aws_eks_cluster.eks_cluster,
    aws_eks_node_group.eks_node_group
  ]
}

resource "kubernetes_manifest" "example_prometheus_rule" {
  # This creates a basic PrometheusRule to demonstrate functionality.
  # More complex rules would be defined in a dedicated file or more kubernetes_manifest blocks.
  yaml_body = <<-EOT
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: example-alerts
  namespace: monitoring # Ensure this matches your Prometheus namespace
  labels:
    prometheus: k8s
    role: alert-rules
spec:
  groups:
  - name: general.rules
    rules:
    - alert: KubePodCrashLooping
      expr: sum(increase(kube_pod_container_status_restarts_total[5m])) by (namespace, pod, container) > 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: Pod is crash looping
        description: Pod {{ $labels.namespace }}/{{ $labels.pod }} (container {{ $labels.container }}) is crash looping.
    - alert: KubeDeploymentReplicasMismatch
      expr: kube_deployment_spec_replicas{job="kube-state-metrics"} != kube_deployment_status_replicas_available{job="kube-state-metrics"}
      for: 1m
      labels:
        severity: warning
      annotations:
        summary: Deployment replica mismatch
        description: Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has replica mismatch (available: {{ $value }}, desired: {{ $labels.deployment_spec_replicas }}).
EOT

  depends_on = [
    helm_release.prometheus_stack
  ]
}

data "aws_availability_zones" "available" {
  state = "available"
}

output "eks_cluster_name" {
  description = "The name of the EKS cluster"
  value       = aws_eks_cluster.eks_cluster.name
}

output "kubeconfig" {
  description = "Kubectl configuration to connect to the EKS cluster"
  value = <<-EOT
apiVersion: v1
clusters:
- cluster:
    server: ${aws_eks_cluster.eks_cluster.endpoint}
    certificate-authority-data: ${aws_eks_cluster.eks_cluster.certificate_authority[0].data}
  name: ${aws_eks_cluster.eks_cluster.name}
contexts:
- context:
    cluster: ${aws_eks_cluster.eks_cluster.name}
    user: ${aws_eks_cluster.eks_cluster.name}
  name: ${aws_eks_cluster.eks_cluster.name}
current-context: ${aws_eks_cluster.eks_cluster.name}
kind: Config
preferences: {}
users:
- name: ${aws_eks_cluster.eks_cluster.name}
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      command: aws
      args:
        - "eks"
        - "get-token"
        - "--cluster-name"
        - "${aws_eks_cluster.eks_cluster.name}"
        - "--region"
        - "${data.aws_region.current.name}"
      env:
        - name: AWS_PROFILE
          value: "" # Optional: Specify if using a specific AWS CLI profile
EOT
}
    

Deploying and Testing Your Alerting System

To deploy this infrastructure, navigate to your Terraform project directory and execute the following commands:

terraform init: Initializes the Terraform project and downloads necessary providers.
terraform plan: Review the proposed changes before applying.
terraform apply --auto-approve: Applies the configuration and deploys your EKS cluster and monitoring stack.

Once deployed, configure your kubectl to connect to the new EKS cluster:

aws eks update-kubeconfig --name eks-prod-cluster --region <YOUR_AWS_REGION>
    

You can then check the deployed pods:

kubectl get pods -n monitoring
    

To verify Alertmanager configuration and potentially simulate an alert:

Access Alertmanager UI: Port-forward the Alertmanager service:
kubectl port-forward svc/prometheus-stack-kube-prom-alertmanager 9093 -n monitoring
Then access http://localhost:9093 in your browser.
Simulate an Alert: You can temporarily modify a PrometheusRule to trigger an alert immediately (e.g., change for: 5m to for: 0m for testing, then revert). Or, manually push an alert to Alertmanager using its API for advanced testing.
Verify PagerDuty: Check your PagerDuty service for new incidents.

Conclusion

By leveraging Terraform, Prometheus, and PagerDuty, you've established a robust, automated, and production-ready alerting system for your AWS EKS clusters. This infrastructure-as-code approach ensures consistency, simplifies management, and significantly enhances your team's ability to respond to critical incidents swiftly, thereby minimizing downtime and improving overall system reliability.

Next Steps and Best Practices

Advanced Alerting Rules: Develop a comprehensive suite of Prometheus alerting rules specific to your applications and infrastructure, covering error rates, latency, resource utilization, and custom business metrics.
Grafana Dashboards: Integrate Grafana (included in kube-prometheus-stack) to create rich, interactive dashboards for visualizing your EKS metrics and alert statuses.
Alertmanager Grouping and Inhibition: Fine-tune Alertmanager's routing, grouping, and inhibition rules to prevent alert storms and ensure that the right teams receive relevant notifications.
Cost Optimization: Monitor the cost of your EKS worker nodes and optimize instance types or leverage Karpenter for intelligent autoscaling.
Security Hardening: Implement least-privilege IAM roles, network policies, and regular security audits for your EKS cluster.
Persistent Storage for Prometheus: For production deployments, consider using AWS EBS or EFS CSI drivers for persistent storage for Prometheus to retain metrics data across pod restarts.

CloudStack YAML