Production-Grade AWS EKS Deployment with Terraform, Integrated Datadog Observability and PagerDuty Alerts

In today's fast-paced cloud-native landscape, deploying and managing Kubernetes clusters that meet production-grade standards for reliability, scalability, security, and observability is paramount. This comprehensive guide will walk you through setting up a robust AWS EKS (Elastic Kubernetes Service) cluster using Terraform, integrating Datadog for deep visibility into your infrastructure and applications, and configuring PagerDuty for proactive incident response.

Architecture Pro-Tip: Infrastructure as Code First

Always prioritize Infrastructure as Code (IaC) for your EKS deployments. Terraform ensures that your infrastructure is version-controlled, repeatable, and auditable. For production environments, consider a modular Terraform structure, separating VPC, EKS cluster, node groups, and addon configurations into distinct modules. This approach enhances maintainability, promotes reuse, and reduces the risk of configuration drift, making your deployments robust and predictable.

Why This Stack? Building for Reliability

Combining AWS EKS, Terraform, Datadog, and PagerDuty creates a powerful, integrated solution for managing critical workloads:

• AWS EKS: A managed Kubernetes service that simplifies the deployment, management, and scaling of Kubernetes applications in the AWS cloud. It removes the operational burden of managing the Kubernetes control plane.
• Terraform: An open-source IaC tool that allows you to define and provision datacenter infrastructure using a declarative configuration language. It enables consistent, reproducible, and version-controlled infrastructure deployments.
• Datadog: A comprehensive monitoring and analytics platform for cloud-scale applications. It provides full-stack observability with metrics, traces, and logs from your EKS cluster, applications, and AWS infrastructure.
• PagerDuty: An incident management platform that helps teams detect, triage, and resolve incidents faster. It integrates seamlessly with monitoring tools like Datadog to ensure critical alerts reach the right people immediately.

Prerequisites for Deployment

Before diving in, ensure you have the following:

• An active AWS account with programmatic access (configured AWS CLI).
• Terraform (v1.0.0 or higher) installed.
• kubectl installed and configured.
• aws-iam-authenticator or kubectl aws-auth plugin installed.
• Datadog account with API and Application keys.
• PagerDuty account with an Integration Key for Datadog.

Step 1: Building Your AWS EKS Cluster with Terraform

We'll define the necessary AWS resources using Terraform, including a VPC, subnets, IAM roles, and the EKS cluster itself. For simplicity, we'll use a single Terraform configuration, but in production, you'd likely modularize this.

Core Terraform Setup

Start by defining your AWS provider and any required variables.

VPC and Networking

EKS requires a robust VPC setup with public and private subnets. Nodes should ideally reside in private subnets for enhanced security, using NAT Gateways for outbound internet access.

EKS Cluster and Node Groups

We'll define the EKS control plane and a managed node group. Managed node groups simplify the scaling and updating of your worker nodes.

IAM Roles for EKS

Proper IAM roles are crucial for EKS. You need a role for the EKS control plane and another for the EKS node group to allow them to interact with other AWS services.

Ready-to-Use Terraform Configuration (main.tf)

Here’s a simplified Terraform configuration to get your EKS cluster up and running. Remember to replace placeholders and adapt it to your specific needs.

provider "aws" { region = "us-east-1" } resource "aws_vpc" "eks_vpc" { cidr_block = "10.0.0.0/16" enable_dns_hostnames = true enable_dns_support = true tags = { Name = "eks-prod-vpc" } } resource "aws_subnet" "public_subnets" { count = 2 vpc_id = aws_vpc.eks_vpc.id cidr_block = "10.0.${count.index + 1}.0/24" availability_zone = data.aws_availability_zones.available.names[count.index] map_public_ip_on_launch = true tags = { Name = "eks-prod-public-subnet-${count.index}" "kubernetes.io/cluster/${aws_eks_cluster.prod_eks.name}" = "owned" "kubernetes.io/role/elb" = "1" } } resource "aws_subnet" "private_subnets" { count = 2 vpc_id = aws_vpc.eks_vpc.id cidr_block = "10.0.${count.index + 10}.0/24" availability_zone = data.aws_availability_zones.available.names[count.index] tags = { Name = "eks-prod-private-subnet-${count.index}" "kubernetes.io/cluster/${aws_eks_cluster.prod_eks.name}" = "owned" "kubernetes.io/role/internal-elb" = "1" } } resource "aws_internet_gateway" "eks_igw" { vpc_id = aws_vpc.eks_vpc.id tags = { Name = "eks-prod-igw" } } resource "aws_eip" "nat_eip" { count = 2 vpc = true } resource "aws_nat_gateway" "eks_nat_gateway" { count = 2 allocation_id = aws_eip.nat_eip[count.index].id subnet_id = aws_subnet.public_subnets[count.index].id tags = { Name = "eks-prod-nat-gateway-${count.index}" } } resource "aws_route_table" "public_route_table" { vpc_id = aws_vpc.eks_vpc.id route { cidr_block = "0.0.0.0/0" gateway_id = aws_internet_gateway.eks_igw.id } tags = { Name = "eks-prod-public-rt" } } resource "aws_route_table_association" "public_route_table_association" { count = 2 subnet_id = aws_subnet.public_subnets[count.index].id route_table_id = aws_route_table.public_route_table.id } resource "aws_route_table" "private_route_table" { count = 2 vpc_id = aws_vpc.eks_vpc.id route { cidr_block = "0.0.0.0/0" nat_gateway_id = aws_nat_gateway.eks_nat_gateway[count.index].id } tags = { Name = "eks-prod-private-rt-${count.index}" } } resource "aws_route_table_association" "private_route_table_association" { count = 2 subnet_id = aws_subnet.private_subnets[count.index].id route_table_id = aws_route_table.private_route_table[count.index].id } data "aws_availability_zones" "available" {} resource "aws_iam_role" "eks_cluster_role" { name = "eks-cluster-role-prod" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [ { Effect = "Allow" Principal = { Service = "eks.amazonaws.com" } Action = "sts:AssumeRole" } ] }) } resource "aws_iam_role_policy_attachment" "eks_cluster_policy" { policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy" role = aws_iam_role.eks_cluster_role.name } resource "aws_iam_role" "eks_node_group_role" { name = "eks-node-group-role-prod" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [ { Effect = "Allow" Principal = { Service = "ec2.amazonaws.com" } Action = "sts:AssumeRole" } ] }) } resource "aws_iam_role_policy_attachment" "eks_node_group_worker_policy" { policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy" role = aws_iam_role.eks_node_group_role.name } resource "aws_iam_role_policy_attachment" "eks_node_group_cni_policy" { policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy" role = aws_iam_role.eks_node_group_role.name } resource "aws_iam_role_policy_attachment" "eks_node_group_registry_policy" { policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly" role = aws_iam_role.eks_node_group_role.name } resource "aws_eks_cluster" "prod_eks" { name = "prod-eks-cluster" role_arn = aws_iam_role.eks_cluster_role.arn vpc_config { subnet_ids = concat(aws_subnet.private_subnets[*].id, aws_subnet.public_subnets[*].id) security_group_ids = [] # EKS creates a default one } version = "1.28" # Specify your desired Kubernetes version depends_on = [ aws_iam_role_policy_attachment.eks_cluster_policy, ] tags = { Name = "prod-eks-cluster" } } resource "aws_eks_node_group" "prod_node_group" { cluster_name = aws_eks_cluster.prod_eks.name node_group_name = "prod-managed-ng" node_role_arn = aws_iam_role.eks_node_group_role.arn subnet_ids = aws_subnet.private_subnets[*].id # Deploy nodes in private subnets instance_types = ["t3.medium"] scaling_config { desired_size = 2 max_size = 4 min_size = 2 } remote_access { ec2_ssh_key = "your-ssh-key-name" # Optional: if you need SSH access to nodes } tags = { Name = "prod-eks-node-group" } depends_on = [ aws_iam_role_policy_attachment.eks_node_group_worker_policy, aws_iam_role_policy_attachment.eks_node_group_cni_policy, aws_iam_role_policy_attachment.eks_node_group_registry_policy, ] } output "eks_cluster_name" { value = aws_eks_cluster.prod_eks.name } output "kubeconfig_command" { value = "aws eks update-kubeconfig --region us-east-1 --name ${aws_eks_cluster.prod_eks.name}" }

To deploy, save this as main.tf, run terraform init, terraform plan, and then terraform apply.

After successful deployment, use the `kubeconfig_command` output to configure your kubectl context.

Step 2: Integrating Datadog for Comprehensive Observability

Datadog provides deep insights into your EKS cluster's health, performance, and application behavior. It collects metrics, logs, and traces, offering a unified view of your entire stack.

Datadog Agent Deployment on EKS

The Datadog Agent is typically deployed as a DaemonSet across all nodes in your EKS cluster to collect host-level metrics, events, and logs. We recommend using Helm for a streamlined deployment.

Pre-requisites for Datadog on EKS

• Ensure you have your Datadog API key and Application key ready.
• Grant necessary IAM permissions to your EKS node role for Datadog to pull metrics from AWS services (e.g., CloudWatch, EC2).

Deploying Datadog Agent via Helm

First, add the Datadog Helm repository:

helm repo add datadog https://helm.datadoghq.com helm repo update

Then, install the Datadog Agent, replacing <YOUR_DATADOG_API_KEY> and <YOUR_DATADOG_APP_KEY>:

helm install datadog-agent datadog/datadog \ --set datadog.apiKey=<YOUR_DATADOG_API_KEY> \ --set datadog.appKey=<YOUR_DATADOG_APP_KEY> \ --set datadog.site="datadoghq.com" \ --set targetSystem="linux" \ --set clusterAgent.enabled=true \ --set clusterAgent.createServiceAccount=true \ --set clusterChecksRunner.enabled=true \ --set clusterChecksRunner.createServiceAccount=true \ --set collectEvents=true \ --set metricsCollection.enabled=true \ --set logs.enabled=true \ --set logs.containerCollectAll=true \ --set apm.enabled=true \ --set processAgent.enabled=true \ --set networkMonitoring.enabled=true \ --set systemProbe.enabled=true \ --set kubeStateMetrics.enabled=true \ --set providers.eks.enabled=true \ --set targetKubelet="https://<YOUR_KUBERNETES_API_SERVER_IP>:10250" # Optional, if not using auto-discovery

Verify the deployment:

kubectl get pods -l app=datadog -n default

You should see Datadog Agent pods running on each of your EKS nodes, along with the cluster agent.

Step 3: Configuring PagerDuty for Incident Management

Integrating PagerDuty with Datadog ensures that critical alerts from your EKS cluster trigger immediate notifications and incident response workflows.

Datadog PagerDuty Integration Setup

In your Datadog account, navigate to Integrations > Integrations, search for PagerDuty, and click "Install". You'll need to provide your PagerDuty Integration Key (obtained from PagerDuty by adding a new service or integration to an existing service).

Defining Production Alerts in Datadog

Once integrated, you can configure any Datadog monitor to send alerts to PagerDuty. Here's an example of a common production alert: EKS node CPU utilization exceeding a threshold.

Example Datadog Monitor for PagerDuty Alert

You can define monitors directly in the Datadog UI or, for IaC consistency, use Terraform for Datadog monitors.

# This is a conceptual example for a Datadog monitor resource in Terraform. # Actual resource names and attributes may vary based on your Datadog provider version. resource "datadog_monitor" "eks_high_cpu_node_alert" { name = "[EKS Production] High CPU Utilization on Node: {{host.name}}" type = "metric alert" query = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster_name:prod-eks-cluster} by {host} < 10" # Alert if idle CPU < 10% (i.e., usage > 90%) message = "CPU utilization on host {{host.name}} is {{system.cpu.idle.avg_last_5m}}% idle. This indicates high load. @pagerduty-prod-team" tags = ["environment:production", "service:eks", "severity:critical"] threshold_warning = 20 threshold_critical = 10 renotify_interval = 60 notify_no_data = false require_full_window = false timeout_h = 0 # No auto-resolve timeout monitor_threshold_windows { recovery_window = "last_10m" } force_delete = false # Set to true to allow Terraform to destroy this monitor }

In the message field, @pagerduty-prod-team is a Datadog notification handle configured to send to your PagerDuty integration. This ensures that when the CPU threshold is breached, a critical incident is created in PagerDuty, triggering your on-call schedule.

Best Practices for Production-Grade Readiness

Beyond the initial setup, consider these practices for a truly production-grade EKS environment:

• Security Hardening:
  • Implement IAM roles for service accounts (IRSA) for fine-grained permissions for Kubernetes pods.
  • Use Network Policies to control pod-to-pod communication.
  • Regularly update EKS cluster and node group versions.
  • Encrypt EBS volumes for node groups and enforce secrets management (e.g., AWS Secrets Manager, HashiCorp Vault).
• Cost Optimization:
  • Utilize EC2 Spot Instances for stateless, fault-tolerant workloads via Karpenter or Cluster Autoscaler.
  • Right-size your EC2 instances in node groups based on actual workload requirements.
  • Implement Horizontal Pod Autoscaler (HPA) and Vertical Pod Autoscaler (VPA) for efficient resource allocation.
• High Availability & Disaster Recovery:
  • Distribute node groups across multiple Availability Zones.
  • Ensure your application deployments use appropriate anti-affinity rules and pod disruption budgets.
  • Regularly back up persistent data (e.g., using Velero for Kubernetes resources and volumes).
• CI/CD Integration:
  • Automate Terraform deployments via a CI/CD pipeline (e.g., GitLab CI, GitHub Actions, AWS CodePipeline).
  • Automate application deployments using tools like Argo CD or Flux for GitOps.

Conclusion

Deploying a production-grade AWS EKS cluster requires careful planning and robust tooling. By leveraging Terraform for infrastructure automation, Datadog for comprehensive observability, and PagerDuty for efficient incident response, you can build a highly resilient, scalable, and manageable Kubernetes platform.

This guide provides a solid foundation. Remember to continuously refine your configuration, monitoring, and alerting strategies based on your specific application needs and operational experience. A well-architected EKS environment, backed by powerful observability and incident management, is key to successful cloud-native operations.