Building a Resilient AWS EKS Cluster with Terraform, Datadog, Prometheus, and PagerDuty
In today's fast-paced cloud native landscape, deploying and managing Kubernetes clusters requires not just robust infrastructure automation but also comprehensive observability and an agile incident response strategy. This guide provides a comprehensive, technical walkthrough on how to leverage Terraform for declarative infrastructure as code (IaC) to provision an AWS EKS cluster, integrate it with Datadog for end-to-end monitoring, Prometheus for powerful metrics collection and alerting, and finally, PagerDuty for automated incident escalation and response. By the end of this guide, you will have a production-ready EKS environment equipped with a sophisticated stack for proactive monitoring and streamlined incident management.
Architecture Pro-Tip: Always design your cloud infrastructure with modularity and security in mind. For EKS, separate your VPC, EKS cluster, and node group configurations into distinct Terraform modules. Implement least privilege IAM roles for service accounts (IRSA) and network policies within Kubernetes. Utilize private subnets for EKS worker nodes and leverage VPC endpoints for AWS services to enhance security and reduce data transfer costs. Consider using managed node groups or Fargate for simplified cluster operations and Karpenter for intelligent, cost-optimized node provisioning.
Prerequisites
Before you begin, ensure you have the following tools and accounts configured:
- AWS Account: With programmatic access and sufficient permissions to create EKS clusters, VPCs, EC2 instances, and IAM roles.
- AWS CLI: Configured with your credentials and default region.
- Terraform: Version 1.0 or newer installed.
- Kubectl: Installed and configured to interact with Kubernetes clusters.
- Helm: Version 3 or newer installed for deploying Kubernetes applications.
- Datadog Account: With an API key and Application key.
- PagerDuty Account: With an integration key for your service.
Core Components Overview
AWS EKS (Elastic Kubernetes Service)
EKS is a managed Kubernetes service that makes it easy to deploy, manage, and scale containerized applications using Kubernetes on AWS. It handles the availability and scalability of the Kubernetes control plane nodes.
Terraform
Terraform by HashiCorp is an open-source IaC tool that allows you to define and provision infrastructure using a high-level configuration language. It enables consistent and repeatable deployments across various cloud providers, including AWS.
Datadog
Datadog is a leading monitoring and analytics platform for cloud applications. It provides comprehensive visibility across servers, databases, tools, and services through a unified platform, offering metrics, logs, traces, and UX monitoring.
Prometheus
Prometheus is an open-source monitoring system and time-series database. It is highly popular in the Kubernetes ecosystem for collecting metrics from various services, enabling powerful querying and alerting capabilities.
PagerDuty
PagerDuty is an incident management platform that provides reliable notifications, automatic escalations, on-call scheduling, and other features to help teams detect and resolve incidents quickly.
Step-by-Step Implementation Guide
1. Terraform AWS EKS Cluster Setup
We will define our EKS cluster, VPC, and node groups using Terraform. Create a project directory, e.g., terraform-eks-datadog-pd, and place the following files within it.
main.tf: Core EKS Infrastructure
This file defines the AWS provider, creates a new VPC, subnets, and then provisions the EKS cluster and its associated worker node groups. We recommend using private subnets for worker nodes for enhanced security.
resource "aws_vpc" "eks_vpc" {
cidr_block = var.vpc_cidr
enable_dns_hostnames = true
enable_dns_support = true
tags = {
Name = "${var.cluster_name}-vpc"
}
}
resource "aws_subnet" "public" {
count = length(var.public_subnets_cidr)
vpc_id = aws_vpc.eks_vpc.id
cidr_block = var.public_subnets_cidr[count.index]
availability_zone = data.aws_availability_zones.available.names[count.index]
map_public_ip_on_launch = true
tags = {
Name = "${var.cluster_name}-public-subnet-${count.index}"
"kubernetes.io/cluster/${var.cluster_name}" = "shared"
"kubernetes.io/role/elb" = "1"
}
}
resource "aws_subnet" "private" {
count = length(var.private_subnets_cidr)
vpc_id = aws_vpc.eks_vpc.id
cidr_block = var.private_subnets_cidr[count.index]
availability_zone = data.aws_availability_zones.available.names[count.index]
tags = {
Name = "${var.cluster_name}-private-subnet-${count.index}"
"kubernetes.io/cluster/${var.cluster_name}" = "shared"
"kubernetes.io/role/internal-elb" = "1"
}
}
resource "aws_internet_gateway" "eks_igw" {
vpc_id = aws_vpc.eks_vpc.id
tags = {
Name = "${var.cluster_name}-igw"
}
}
resource "aws_route_table" "public_rt" {
vpc_id = aws_vpc.eks_vpc.id
route {
cidr_block = "0.0.0.0/0"
gateway_id = aws_internet_gateway.eks_igw.id
}
tags = {
Name = "${var.cluster_name}-public-rt"
}
}
resource "aws_route_table_association" "public_rt_assoc" {
count = length(aws_subnet.public)
subnet_id = aws_subnet.public[count.index].id
route_table_id = aws_route_table.public_rt.id
}
resource "aws_eip" "nat_eip" {
count = length(var.public_subnets_cidr)
vpc = true
tags = {
Name = "${var.cluster_name}-nat-eip-${count.index}"
}
}
resource "aws_nat_gateway" "eks_nat_gateway" {
count = length(var.public_subnets_cidr)
allocation_id = aws_eip.nat_eip[count.index].id
subnet_id = aws_subnet.public[count.index].id
tags = {
Name = "${var.cluster_name}-nat-gateway-${count.index}"
}
depends_on = [aws_internet_gateway.eks_igw]
}
resource "aws_route_table" "private_rt" {
count = length(var.private_subnets_cidr)
vpc_id = aws_vpc.eks_vpc.id
route {
cidr_block = "0.0.0.0/0"
nat_gateway_id = aws_nat_gateway.eks_nat_gateway[count.index].id
}
tags = {
Name = "${var.cluster_name}-private-rt-${count.index}"
}
}
resource "aws_route_table_association" "private_rt_assoc" {
count = length(aws_subnet.private)
subnet_id = aws_subnet.private[count.index].id
route_table_id = aws_route_table.private_rt[count.index].id
}
resource "aws_iam_role" "eks_cluster_role" {
name = "${var.cluster_name}-eks-cluster-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Principal = {
Service = "eks.amazonaws.com"
}
Action = "sts:AssumeRole"
},
]
})
}
resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
role = aws_iam_role.eks_cluster_role.name
}
resource "aws_iam_role_policy_attachment" "eks_service_policy" {
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
role = aws_iam_role.eks_cluster_role.name
}
resource "aws_iam_role" "eks_node_role" {
name = "${var.cluster_name}-eks-node-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Principal = {
Service = "ec2.amazonaws.com"
}
Action = "sts:AssumeRole"
},
]
})
}
resource "aws_iam_role_policy_attachment" "eks_node_policy" {
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
role = aws_iam_role.eks_node_role.name
}
resource "aws_iam_role_policy_attachment" "eks_cni_policy" {
policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
role = aws_iam_role.eks_node_role.name
}
resource "aws_iam_role_policy_attachment" "ec2_container_registry_readonly" {
policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
role = aws_iam_role.eks_node_role.name
}
resource "aws_eks_cluster" "main" {
name = var.cluster_name
role_arn = aws_iam_role.eks_cluster_role.arn
version = var.kubernetes_version
vpc_config {
subnet_ids = concat(aws_subnet.private[*].id, aws_subnet.public[*].id)
security_group_ids = [] # EKS creates a default SG. Add specific ones if needed.
}
depends_on = [
aws_iam_role_policy_attachment.eks_cluster_policy,
aws_iam_role_policy_attachment.eks_service_policy,
]
tags = {
Name = var.cluster_name
}
}
resource "aws_eks_node_group" "main" {
cluster_name = aws_eks_cluster.main.name
node_group_name = "${var.cluster_name}-node-group"
node_role_arn = aws_iam_role.eks_node_role.arn
subnet_ids = aws_subnet.private[*].id # Use private subnets for worker nodes
instance_types = [var.instance_type]
disk_size = var.disk_size
scaling_config {
desired_size = var.node_group_desired_size
min_size = var.node_group_min_size
max_size = var.node_group_max_size
}
update_config {
max_unavailable = 1
}
depends_on = [
aws_iam_role_policy_attachment.eks_node_policy,
aws_iam_role_policy_attachment.eks_cni_policy,
aws_iam_role_policy_attachment.ec2_container_registry_readonly,
]
tags = {
Name = "${var.cluster_name}-node-group"
}
}
data "aws_availability_zones" "available" {}
provider "aws" {
region = var.aws_region
}
variables.tf: Cluster Configuration Variables
Define configurable parameters for your EKS cluster.
variable "aws_region" {
description = "AWS region for deployment"
type = string
default = "us-east-1"
}
variable "cluster_name" {
description = "Name of the EKS cluster"
type = string
default = "my-datadog-eks"
}
variable "kubernetes_version" {
description = "Kubernetes version for the EKS cluster"
type = string
default = "1.28" # Or your desired version
}
variable "vpc_cidr" {
description = "CIDR block for the VPC"
type = string
default = "10.0.0.0/16"
}
variable "public_subnets_cidr" {
description = "List of CIDR blocks for public subnets"
type = list(string)
default = ["10.0.1.0/24", "10.0.2.0/24"]
}
variable "private_subnets_cidr" {
description = "List of CIDR blocks for private subnets"
type = list(string)
default = ["10.0.3.0/24", "10.0.4.0/24"]
}
variable "instance_type" {
description = "EC2 instance type for EKS worker nodes"
type = string
default = "t3.medium"
}
variable "disk_size" {
description = "Disk size (GB) for EKS worker nodes"
type = number
default = 20
}
variable "node_group_desired_size" {
description = "Desired number of worker nodes in the EKS node group"
type = number
default = 2
}
variable "node_group_min_size" {
description = "Minimum number of worker nodes in the EKS node group"
type = number
default = 1
}
variable "node_group_max_size" {
description = "Maximum number of worker nodes in the EKS node group"
type = number
default = 3
}
outputs.tf: Expose Cluster Information
These outputs are useful for configuring kubectl and subsequent integrations.
output "cluster_name" {
description = "The name of the EKS cluster"
value = aws_eks_cluster.main.name
}
output "kubeconfig_command" {
description = "Command to update kubeconfig for the EKS cluster"
value = "aws eks update-kubeconfig --region ${var.aws_region} --name ${aws_eks_cluster.main.name}"
}
output "cluster_endpoint" {
description = "The endpoint for the EKS cluster"
value = aws_eks_cluster.main.endpoint
}
output "cluster_security_group_id" {
description = "The security group ID of the EKS cluster"
value = aws_eks_cluster.main.vpc_config[0].cluster_security_group_id
}
output "vpc_id" {
description = "The ID of the VPC created for the EKS cluster"
value = aws_vpc.eks_vpc.id
}
Deployment Steps for EKS:
- Initialize Terraform:
terraform init
- Review the plan:
terraform plan
- Apply the configuration:
terraform apply --auto-approve
- Once applied, run the command from the
kubeconfig_command output to configure your kubectl: aws eks update-kubeconfig --region <your_region> --name <your_cluster_name>
- Verify cluster access:
kubectl get svc
2. Integrating Datadog for Comprehensive Monitoring
Datadog provides deep visibility into your EKS cluster, applications, and underlying infrastructure. We'll deploy the Datadog Agent using Helm.
Datadog Agent Deployment via Helm
First, add the Datadog Helm repository and create a datadog-values.yaml file. Replace <YOUR_DATADOG_API_KEY> and <YOUR_DATADOG_APP_KEY> with your actual keys.
# Add Datadog Helm repo
helm repo add datadog https://helm.datadoghq.com
helm repo update
# datadog-values.yaml
apiVersion: v2
name: datadog
description: A Helm chart for Datadog Agent
version: 1.0.0
# Values to override in Datadog Helm chart
# Run: helm install datadog -f datadog-values.yaml datadog/datadog --set datadog.apiKey="" --set datadog.appKey=""
datadog:
apiKey: "" # Ensure this is securely managed, e.g., via Kubernetes secrets or external secret management
appKey: "" # Ensure this is securely managed
site: "datadoghq.com" # Or your specific Datadog site (e.g., eu.datadoghq.com)
kubeStateMetricsCore:
enabled: true
clusterAgent:
enabled: true
admissionController:
enabled: true
mutateUnlabelled: false # Set to true to inject APM/logging into all pods
metricsProvider:
enabled: true
use-datadogmetric-crd: true
# Enable APM and Log Collection
apm:
enabled: true
socketEnabled: true
logs:
enabled: true
containerCollectAll: true
autoMultiLineLogFiles: true
# Enable process monitoring
processAgent:
enabled: true
# Enable network performance monitoring
networkMonitoring:
enabled: true
# Enable Live Processes (requires processAgent to be enabled)
liveProcesses:
enabled: true
# Enable system probes for security and network data
systemProbe:
enabled: true
appsec:
enabled: false # Enable if you need Application Security Monitoring
networkEnabled: true
runtimeEnabled: false # Enable if you need Runtime Security Monitoring
# Enable support for EKS Fargate if you're using it
# fargate:
# enabled: false
# Image configuration (optional, use if you need specific agent versions)
# agent:
# image:
# name: "gcr.io/datadog/agent"
# tag: "7.48.0" # Specify the desired agent version
}
Deploy the Datadog Agent using Helm:
helm install datadog -f datadog-values.yaml datadog/datadog --namespace datadog --create-namespace
Verify Datadog Agent pods are running:
kubectl get pods -n datadog
Once deployed, you should start seeing metrics, logs, and traces from your EKS cluster appear in your Datadog dashboard.
3. Setting Up Prometheus for Metrics and Alerting
Prometheus is a powerful tool for collecting metrics from your Kubernetes applications. We'll deploy the kube-prometheus-stack Helm chart, which includes Prometheus, Alertmanager, Grafana, and Kube-state-metrics.
Prometheus Operator Deployment via Helm
Add the Prometheus community Helm repository and deploy the stack. Create a prometheus-values.yaml file for customization.
# Add Prometheus community Helm repo
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
# prometheus-values.yaml
apiVersion: v2
name: kube-prometheus-stack
description: Prometheus stack for Kubernetes
version: 1.0.0
# Values to override in kube-prometheus-stack Helm chart
# Run: helm install prometheus -f prometheus-values.yaml prometheus-community/kube-prometheus-stack
kube-prometheus-stack:
prometheus:
prometheusSpec:
retention: 10d # Retain metrics for 10 days
storageSpec:
volumeClaimTemplate:
spec:
storageClassName: gp2 # Or your default storage class
resources:
requests:
storage: 50Gi # Adjust storage as needed
grafana:
enabled: true
adminPassword: "prom-admin-password" # Change this to a secure password
ingress:
enabled: false # Set to true and configure if you need external access to Grafana
annotations:
kubernetes.io/ingress.class: nginx
hosts:
- grafana.yourdomain.com
service:
type: ClusterIP # Change to LoadBalancer if you want easy external access
alertmanager:
enabled: true
alertmanagerSpec:
retention: 24h
storage:
volumeClaimTemplate:
spec:
storageClassName: gp2 # Or your default storage class
resources:
requests:
storage: 5Gi
config:
global:
resolve_timeout: 5m
route:
group_by: ['alertname', 'cluster', 'service']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receiver: 'default-receiver'
receivers:
- name: 'default-receiver'
# PagerDuty configuration will be added here in the next step
}
Deploy Prometheus stack:
helm install prometheus -f prometheus-values.yaml prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace
Verify Prometheus pods are running:
kubectl get pods -n monitoring
Prometheus will now start scraping metrics from your EKS cluster components. You can access Grafana (if enabled) to view dashboards.
4. Automating Incident Response with PagerDuty
To automate incident response, we'll integrate Prometheus Alertmanager with PagerDuty. When an alert fires in Prometheus that matches certain rules, Alertmanager will send an event to PagerDuty, triggering an incident.
PagerDuty Service Creation
In your PagerDuty account:
- Go to Services > Service Directory and click +New Service.
- Give your service a name (e.g., "EKS Critical Alerts") and assign it to a team.
- For "Integration Type," select Prometheus or Events API v2 if Prometheus isn't listed (and use a Generic Events API integration).
- Once created, you will get an Integration Key. Keep this key handy.
Integrating Prometheus Alertmanager with PagerDuty
We need to update the Alertmanager configuration with your PagerDuty integration key. You can do this by modifying the prometheus-values.yaml file we used earlier and upgrading the Helm release.
Locate the alertmanager.config section in your prometheus-values.yaml and add the PagerDuty receiver:
# prometheus-values.yaml (excerpt)
alertmanager:
enabled: true
alertmanagerSpec:
# ... existing configuration ...
config:
global:
resolve_timeout: 5m
route:
group_by: ['alertname', 'cluster', 'service']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receiver: 'pagerduty-receiver' # Change default-receiver to pagerduty-receiver
receivers:
- name: 'default-receiver' # Keep this or remove if not used
- name: 'pagerduty-receiver'
pagerduty_configs:
- service_key: "" # Replace with your PagerDuty Integration Key
severity: 'critical' # Default severity for alerts sent to PagerDuty
details:
cluster: '{{ .CommonLabels.cluster }}'
namespace: '{{ .CommonLabels.namespace }}'
pod: '{{ .CommonLabels.pod }}'
alertname: '{{ .CommonLabels.alertname }}'
summary: '{{ .CommonLabels.summary }}'
# Example for a simple alert rule (create a new file named rules.yaml if preferred)
# This can also be defined as a PrometheusRule Kubernetes object.
# We add this directly to the config for simplicity in this example.
templates:
- name: 'default.tmpl'
content: |
{{ define "__alertmanager" }}https://alertmanager.monitoring.svc.cluster.local{{ end }}
{{ define "__subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.service }} {{ .CommonLabels.cluster }}{{ end }}
{{ define "__description" }}{{ range .Alerts }}{{ .Annotations.description }}{{ end }}{{ end }}
{{ define "__text_alert_description" }}
{{- range .Alerts }}
*Alert:* {{ .Annotations.summary }}
*Description:* {{ .Annotations.description }}
*Severity:* {{ .Labels.severity | toUpper }}
*Instance:* {{ .Labels.instance }}
*Job:* {{ .Labels.job }}
*Starts At:* {{ .StartsAt | datetime "2006-01-02 15:04:05 MST" }}
{{- if .EndsAt.IsZero }}
{{- else }}*Ends At:* {{ .EndsAt | datetime "2006-01-02 15:04:05 MST" }}{{- end }}
{{- if .GeneratorURL }}*Source:* {{ .GeneratorURL }}{{- end }}
{{- end }}
{{ end }}
After updating prometheus-values.yaml, upgrade the Helm release:
helm upgrade prometheus -f prometheus-values.yaml prometheus-community/kube-prometheus-stack --namespace monitoring
Now, when a Prometheus alert fires that matches the Alertmanager's routing configuration, an incident will be automatically created in PagerDuty, notifying the on-call team.
Best Practices and Advanced Considerations
- Security Hardening: Implement Kubernetes Network Policies, use OPA/Kyverno for admission control, enable IRSA (IAM Roles for Service Accounts) for fine-grained permissions, and regularly scan images for vulnerabilities.
- Cost Optimization: Explore AWS Fargate for serverless Kubernetes, implement Karpenter for intelligent node provisioning, utilize EC2 Spot Instances for stateless workloads, and right-size your instances based on actual usage.
- CI/CD Integration: Integrate your Terraform code into a CI/CD pipeline (e.g., GitLab CI, GitHub Actions, AWS CodePipeline) for automated and consistent deployments.
- Terraform State Management: Store your Terraform state in a remote backend like an S3 bucket with DynamoDB locking to ensure consistency and prevent corruption in team environments.
- Custom Metrics and Dashboards: Beyond default metrics, instrument your applications to emit custom metrics. Create tailored Datadog dashboards and Prometheus recording rules/alerts to monitor specific business KPIs and application health.
- Log Management: Ensure all application logs are centralized and accessible. Datadog provides excellent log ingestion and analysis, but consider other tools like Fluent Bit/Fluentd for log forwarding.
- Distributed Tracing: Integrate Datadog APM with your applications to gain end-to-end visibility into requests across microservices, crucial for debugging complex distributed systems.
Conclusion
By following this guide, you have successfully established a powerful, observable, and resilient AWS EKS cluster. Leveraging Terraform's IaC capabilities ensures your infrastructure is consistently provisioned and easily replicable. Datadog provides the comprehensive monitoring and logging necessary for day-to-day operations and proactive issue identification. Prometheus offers granular metric collection and flexible alerting, while PagerDuty closes the loop with automated incident response, ensuring critical issues are addressed promptly. This integrated stack empowers DevOps teams to build, deploy, and operate cloud-native applications with confidence, minimizing downtime and accelerating recovery.
Comments
Post a Comment