Terraform AWS EKS Prometheus Monitoring and PagerDuty Alerting Configuration
Architecture Pro-Tip: Embrace Infrastructure as Code for Observability
Leveraging Terraform to provision your entire monitoring and alerting stack on AWS EKS ensures consistency, repeatability, and disaster recovery capabilities. By defining your Prometheus, Grafana, Alertmanager, and PagerDuty integrations as code, you create a self-documenting, auditable, and version-controlled observability platform. This approach significantly reduces configuration drift and streamlines scaling, allowing your DevOps teams to focus on innovation rather than manual setup.
Mastering Terraform AWS EKS Prometheus Monitoring with PagerDuty Alerting
In the dynamic world of cloud-native applications, robust monitoring and efficient incident response are paramount. This comprehensive guide details how to establish a powerful observability stack for your Amazon EKS clusters using Prometheus, Grafana, and Alertmanager, all provisioned and managed with Terraform. Furthermore, we will integrate this setup with PagerDuty for reliable, on-call alerting, ensuring that critical issues are addressed promptly. By treating your monitoring infrastructure as code, you gain unparalleled control, scalability, and consistency.
Prerequisites for Terraform EKS Monitoring
Before diving into the configuration, ensure you have the following:
- An AWS Account with administrative access.
- Terraform CLI (v1.0+) installed and configured.
- kubectl CLI installed and authenticated to your EKS cluster.
- Helm CLI (v3+) installed.
- An existing AWS EKS Cluster. This guide assumes your EKS cluster is already provisioned.
- A PagerDuty Account with an Integration Key for a new or existing service.
- Basic understanding of Kubernetes, Prometheus, and Terraform concepts.
Core Components of Our EKS Observability Stack
Our robust monitoring solution comprises several interconnected components:
- AWS EKS: The managed Kubernetes service hosting our applications and monitoring tools.
- Prometheus: The open-source monitoring system for collecting metrics from EKS components and applications. We'll leverage the
kube-prometheus-stackHelm chart, which includes Prometheus, Grafana, Alertmanager, node-exporter, and kube-state-metrics. - Grafana: For visualizing the collected metrics through customizable dashboards.
- Alertmanager: Handles routing and deduplicating alerts generated by Prometheus, sending them to PagerDuty.
- PagerDuty: The incident management platform responsible for escalating critical alerts to on-call teams.
Terraform Project Structure for EKS Monitoring
A well-organized Terraform project enhances maintainability and scalability. A typical structure might look like this:
main.tf: Main configuration, resource definitions.variables.tf: Input variables.outputs.tf: Output values.versions.tf: Terraform and provider versions.providers.tf: Provider configurations (AWS, Kubernetes, Helm).monitoring/: (Optional) A dedicated module for monitoring resources.
Step-by-Step Terraform Configuration
1. Configure AWS and Kubernetes Providers
First, set up your AWS provider and configure the Kubernetes and Helm providers to interact with your EKS cluster. This typically involves using an EKS Data Source to retrieve cluster details.
resource "aws_eks_cluster" "main" {
# ... (your EKS cluster definition or data source) ...
}
data "aws_eks_cluster" "main" {
name = aws_eks_cluster.main.name
}
data "aws_eks_cluster_auth" "main" {
name = aws_eks_cluster.main.name
}
provider "kubernetes" {
host = data.aws_eks_cluster.main.endpoint
cluster_ca_certificate = base64decode(data["aws_eks_cluster"].main.certificate_authority[0].data)
token = data.aws_eks_cluster_auth.main.token
}
provider "helm" {
kubernetes {
host = data.aws_eks_cluster.main.endpoint
cluster_ca_certificate = base64decode(data["aws_eks_cluster"].main.certificate_authority[0].data)
token = data.aws_eks_cluster_auth.main.token
}
}
2. Deploy Prometheus and Grafana using Helm
We'll use the official kube-prometheus-stack Helm chart, which bundles Prometheus, Grafana, Alertmanager, and necessary exporters. This simplifies deployment significantly.
Important: You'll need to define the namespace where these resources will be deployed and ensure it exists. You can create it with a kubernetes_namespace resource.
resource "kubernetes_namespace" "monitoring" {
metadata {
name = "monitoring"
}
}
resource "helm_release" "kube_prometheus_stack" {
name = "kube-prometheus-stack"
repository = "https://prometheus-community.github.io/helm-charts"
chart = "kube-prometheus-stack"
namespace = kubernetes_namespace.monitoring.metadata[0].name
version = "58.1.0" # Use a stable chart version
values = [
yamlencode({
# Global settings
fullnameOverride = "kube-prometheus-stack"
# Prometheus configuration
prometheus = {
prometheusSpec = {
# Use IRSA for AWS integrations (e.g., S3 for Thanos, or CloudWatch exporter if needed)
serviceAccountName = "kube-prometheus-stack-prometheus"
retention = "7d" # Adjust retention as needed
storageSpec = {
volumeClaimTemplate = {
spec = {
storageClassName = "gp2" # Or your preferred storage class
resources = {
requests = {
storage = "50Gi"
}
}
}
}
}
}
}
# Grafana configuration
grafana = {
enabled = true
adminPassword = "your-secure-grafana-password" # Change this! Use AWS Secrets Manager or Vault for production.
service = {
type = "LoadBalancer" # Expose Grafana via an AWS Load Balancer
}
ingress = {
enabled = false # Set to true and configure if you prefer Ingress over LoadBalancer
}
}
# Alertmanager configuration - basic setup, full config below
alertmanager = {
enabled = true
alertmanagerSpec = {
# ... (PagerDuty config will be added here or via a separate ConfigMap) ...
}
service = {
type = "ClusterIP" # Alertmanager usually accessed internally
}
}
# Node Exporter configuration
nodeExporter = {
enabled = true
}
# Kube-state-metrics configuration
kubeStateMetrics = {
enabled = true
}
# Blackbox Exporter configuration (optional, for external service monitoring)
# blackboxExporter = {
# enabled = true
# }
})
]
}
3. Configure Alertmanager for PagerDuty Integration
Alertmanager acts as the central hub for processing alerts. We'll configure it to send critical alerts to PagerDuty. For security, the PagerDuty integration key should be stored as a Kubernetes Secret and referenced in the Alertmanager configuration.
resource "kubernetes_secret" "pagerduty_api_key" {
metadata {
name = "pagerduty-api-key"
namespace = kubernetes_namespace.monitoring.metadata[0].name
}
data = {
# It's better to fetch this from AWS Secrets Manager or another secure vault
# For demonstration, replace "YOUR_PAGERDUTY_INTEGRATION_KEY" with your actual key
# base64encode is necessary for secret values
"PAGERDUTY_INTEGRATION_KEY" = base64encode("YOUR_PAGERDUTY_INTEGRATION_KEY")
}
type = "Opaque"
}
resource "kubernetes_config_map" "alertmanager_config" {
metadata {
name = "alertmanager-config"
namespace = kubernetes_namespace.monitoring.metadata[0].name
}
data = {
"alertmanager.yaml" = yamlencode({
global = {
resolve_timeout = "5m"
}
route = {
group_by = ["alertname", "cluster", "service"]
group_wait = "30s"
group_interval = "5m"
repeat_interval = "4h"
receiver = "pagerduty"
routes = [
{
match_re = {
severity = "critical|warning"
}
receiver = "pagerduty"
}
]
}
receivers = [
{
name = "pagerduty"
pagerduty_configs = [
{
service_key_file = "/etc/alertmanager/secrets/PAGERDUTY_INTEGRATION_KEY"
# Optional: You can specify a custom PagerDuty URL if needed
# url = "https://events.pagerduty.com/v2/enqueue"
}
]
}
]
templates = [ "/etc/alertmanager/config/*.tmpl" ] # Example if you use custom templates
})
}
# This config map will be mounted to the Alertmanager pod
# Ensure the helm_release for kube-prometheus-stack is updated to use this config map
# This is often done by setting `alertmanager.config` in the Helm values or
# by overriding the Alertmanager deployment to mount this config map and secret.
# For the `kube-prometheus-stack`, you'd typically pass this directly via `alertmanager.config` in values.
}
# The helm_release for kube-prometheus-stack needs to be updated to consume this config.
# A snippet showing how to integrate the PagerDuty secret into Alertmanager's pod:
# This part would typically be added to the `helm_release` values under `alertmanager.alertmanagerSpec`.
# For simplicity and to avoid excessive nesting, we assume the secret is mounted.
# The `kube-prometheus-stack` chart allows injecting additional secrets.
# For PagerDuty, we typically provide the secret directly to the alertmanager.yml via the `secret_key` field
# within the pagerduty_configs receiver.
# --- Example of how to integrate the Alertmanager config directly via Helm values ---
# This overrides the default Alertmanager configuration provided by the chart.
# Update the 'helm_release.kube_prometheus_stack' resource:
# Add or modify the 'alertmanager' section in 'values':
/*
alertmanager = {
enabled = true
config = {
global = {
resolve_timeout = "5m"
}
route = {
group_by = ["alertname", "cluster", "service"]
group_wait = "30s"
group_interval = "5m"
repeat_interval = "4h"
receiver = "pagerduty"
}
receivers = [
{
name = "pagerduty"
pagerduty_configs = [
{
service_key = var.pagerduty_integration_key # Pass this as a Terraform variable
}
]
}
]
}
}
*/
# --- End of example for Helm values integration ---
# For direct file mounting with secrets, you need to ensure the `kube-prometheus-stack`
# Alertmanager deployment has the secret mounted to `/etc/alertmanager/secrets/`.
# This often requires `alertmanager.additionalSecretMounts` in chart values.
# Given the `kube-prometheus-stack` chart complexity, directly injecting the service_key
# into the Helm values for Alertmanager is usually simpler than manual ConfigMap/Secret mounting.
# For this guide, we'll simplify and show the ideal alertmanager.yaml structure.
Note: Integrating the Alertmanager configuration, especially secrets, directly into the helm_release values for kube-prometheus-stack is often the cleanest method. The Helm chart typically provides options to define the Alertmanager configuration directly in its values, including referencing secrets.
4. Defining Prometheus Alerting Rules
Prometheus uses Alerting Rules to define conditions under which alerts are fired. These are typically defined as PrometheusRule custom resources. We can create these resources using Terraform's kubernetes_manifest or kubectl_manifest (if using the kubectl provider) resources.
resource "kubernetes_manifest" "high_cpu_alert" {
provider = kubernetes
manifest = {
apiVersion = "monitoring.coreos.com/v1"
kind = "PrometheusRule"
metadata = {
name = "kubernetes-high-cpu-alert"
namespace = kubernetes_namespace.monitoring.metadata[0].name
labels = {
"release" = helm_release.kube_prometheus_stack.name # Link to the Helm release
}
}
spec = {
groups = [
{
name = "kubernetes.rules"
rules = [
{
alert = "KubeHighCPUUsage"
expr = "sum(node_cpu_seconds_total{mode!=\"idle\",instance=~\"^(.*)\"}) by (instance) / sum(node_cpu_seconds_total{instance=~\"^(.*)\"}) by (instance) * 100 > 80"
for = "5m"
labels = {
severity = "warning"
}
annotations = {
summary = "High CPU usage on Kubernetes node {{ $labels.instance }}"
description = "Node {{ $labels.instance }} has been running with high CPU usage (above 80%) for 5 minutes."
}
},
{
alert = "KubeHighMemoryUsage"
expr = "(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 85"
for = "5m"
labels = {
severity = "critical"
}
annotations = {
summary = "High Memory usage on Kubernetes node {{ $labels.instance }}"
description = "Node {{ $labels.instance }} has been running with high memory usage (above 85%) for 5 minutes."
}
}
]
}
]
}
}
}
Ready-to-Use Terraform Configuration Snippets
Here are consolidated snippets representing the core Terraform resources discussed. Remember to replace placeholder values like "YOUR_PAGERDUTY_INTEGRATION_KEY" and "your-secure-grafana-password" with your actual, secure credentials.
variable "cluster_name" {
description = "The name of the EKS cluster."
type = string
}
variable "pagerduty_integration_key" {
description = "PagerDuty integration key for Alertmanager."
type = string
sensitive = true
}
variable "grafana_admin_password" {
description = "Grafana admin password."
type = string
sensitive = true
}
# --- Providers Setup ---
data "aws_eks_cluster" "selected" {
name = var.cluster_name
}
data "aws_eks_cluster_auth" "selected" {
name = var.cluster_name
}
provider "kubernetes" {
host = data.aws_eks_cluster.selected.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.selected.certificate_authority[0].data)
token = data.aws_eks_cluster_auth.selected.token
}
provider "helm" {
kubernetes {
host = data.aws_eks_cluster.selected.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.selected.certificate_authority[0].data)
token = data.aws_eks_cluster_auth.selected.token
}
}
# --- Kubernetes Namespace ---
resource "kubernetes_namespace" "monitoring" {
metadata {
name = "monitoring"
}
}
# --- Helm Release for Kube-Prometheus-Stack ---
resource "helm_release" "kube_prometheus_stack" {
name = "kube-prometheus-stack"
repository = "https://prometheus-community.github.io/helm-charts"
chart = "kube-prometheus-stack"
namespace = kubernetes_namespace.monitoring.metadata[0].name
version = "58.1.0" # Always pin your chart versions!
values = [
yamlencode({
fullnameOverride = "kube-prometheus-stack"
prometheus = {
prometheusSpec = {
retention = "7d"
serviceAccountName = "kube-prometheus-stack-prometheus"
storageSpec = {
volumeClaimTemplate = {
spec = {
storageClassName = "gp2"
resources = {
requests = {
storage = "50Gi"
}
}
}
}
}
}
}
grafana = {
enabled = true
adminPassword = var.grafana_admin_password
service = {
type = "LoadBalancer"
}
}
alertmanager = {
enabled = true
config = {
global = {
resolve_timeout = "5m"
}
route = {
group_by = ["alertname", "cluster", "service"]
group_wait = "30s"
group_interval = "5m"
repeat_interval = "4h"
receiver = "pagerduty"
routes = [
{
match_re = {
severity = "critical|warning"
}
receiver = "pagerduty"
}
]
}
receivers = [
{
name = "pagerduty"
pagerduty_configs = [
{
service_key = var.pagerduty_integration_key
# Optional: proxy_url, http_config (for custom TLS/auth)
}
]
}
]
}
}
nodeExporter = { enabled = true }
kubeStateMetrics = { enabled = true }
})
]
}
# --- Prometheus Alerting Rule ---
resource "kubernetes_manifest" "high_cpu_alert" {
provider = kubernetes
manifest = {
apiVersion = "monitoring.coreos.com/v1"
kind = "PrometheusRule"
metadata = {
name = "kubernetes-high-cpu-alert"
namespace = kubernetes_namespace.monitoring.metadata[0].name
labels = {
"release" = helm_release.kube_prometheus_stack.name
}
}
spec = {
groups = [
{
name = "kubernetes.rules"
rules = [
{
alert = "KubeHighCPUUsage"
expr = "sum(node_cpu_seconds_total{mode!=\"idle\",instance=~\"^(.*)\"}) by (instance) / sum(node_cpu_seconds_total{instance=~\"^(.*)\"}) by (instance) * 100 > 80"
for = "5m"
labels = {
severity = "warning"
}
annotations = {
summary = "High CPU usage on Kubernetes node {{ $labels.instance }}"
description = "Node {{ $labels.instance }} has been running with high CPU usage (above 80%) for 5 minutes."
}
}
]
}
]
}
}
}
output "grafana_url" {
description = "The URL to access Grafana."
value = one(flatten([
for service in helm_release.kube_prometheus_stack.status[0].resource_kind_values["Service"] :
service.spec.type == "LoadBalancer" ? service.status.load_balancer[0].ingress[0].hostname : null
if service.metadata.name == "kube-prometheus-stack-grafana"
]))
}
Post-Deployment Verification
After running terraform init, terraform plan, and terraform apply, perform these checks:
- Verify Pods: Run
kubectl get pods -n monitoringto ensure all Prometheus, Grafana, and Alertmanager pods are running. - Access Grafana: Use the outputted Grafana URL (or
kubectl port-forward) to access Grafana. Log in with the configured admin password. - Check Prometheus Targets: Within Grafana, navigate to the Prometheus data source and verify that targets (e.g., kube-state-metrics, node-exporter) are being scraped.
- Simulate an Alert: Temporarily lower the threshold of a Prometheus rule (e.g., for CPU usage) to trigger an alert. Check Alertmanager UI (port-forward 9093 to the Alertmanager service) to see the alert and verify it's sent to PagerDuty. Confirm receipt on your PagerDuty dashboard.
Advanced Considerations and Best Practices
- Persistence and Storage: For production, configure robust and cost-effective persistent storage (e.g., EBS gp3) for Prometheus. Consider Thanos or Cortex for long-term metric storage and global views.
- Security: Implement IAM Roles for Service Accounts (IRSA) for Prometheus to securely access AWS services (e.g., S3 for Thanos, CloudWatch). Apply network policies to restrict access to monitoring components.
- Custom Dashboards: Version control your Grafana dashboards and provision them via Terraform or directly through the Grafana sidecar.
- Advanced Alerting: Explore Alertmanager's capabilities for alert suppression, silencing, and complex routing rules based on labels.
- Cost Optimization: Right-size your Prometheus and Grafana deployments based on your cluster size and retention needs. Monitor the cost of underlying AWS resources (EBS, Load Balancers).
- GitOps Integration: For continuous deployment of Kubernetes manifests (like PrometheusRules), consider tools like Argo CD or Flux CD, which can reconcile desired states directly from Git repositories.
Troubleshooting Common Issues
- Prometheus not scraping targets:
- Check Prometheus UI (Status -> Targets) for failed scrapes.
- Verify service discovery (e.g.,
serviceMonitorSelectorin Prometheus config matches yourServiceMonitorresources). - Ensure firewall rules or network policies aren't blocking Prometheus from reaching targets.
- Alertmanager not sending alerts to PagerDuty:
- Check Alertmanager UI (Status -> Alerts) to see if alerts are firing.
- Review Alertmanager logs (
kubectl logs -f <alertmanager-pod> -n monitoring) for errors related to PagerDuty. - Verify the PagerDuty integration key is correct and accessible to Alertmanager (via secret mounting or Helm values).
- Ensure the PagerDuty service is configured correctly to receive events.
- EKS Cluster Authentication Issues (Terraform/Kubernetes providers):
- Ensure your AWS CLI is configured with appropriate credentials and region.
- Verify that the IAM user/role used by Terraform has permissions to describe EKS clusters and generate tokens (
eks:DescribeCluster,eks:ListClusters,eks:AccessCluster). - Confirm the cluster name in your Terraform configuration matches your EKS cluster.
Conclusion
By following this guide, you have successfully configured a robust, observable, and automated monitoring solution for your AWS EKS clusters using Terraform. This Infrastructure as Code approach for Prometheus, Grafana, and PagerDuty integration not only ensures consistency and reliability but also empowers your DevOps teams with critical insights and rapid incident response capabilities. Embrace the power of automation to build resilient cloud-native platforms.
Comments
Post a Comment