Architecture Pro-Tip: Immutable Infrastructure & GitOps First

Always design your EKS infrastructure with immutability in mind. Leverage Terraform to provision and manage every component, from VPCs and subnets to EKS clusters, node groups, and even Kubernetes add-ons. Combine this with a GitOps approach (e.g., using ArgoCD or FluxCD) to manage your application deployments within EKS. This ensures your infrastructure and applications are always in a desired, version-controlled state, significantly reducing configuration drift and improving recovery times. Integrating observability and alerting at the IaC level ensures these critical components are never an afterthought.

Terraform AWS EKS Deployment with Datadog Observability and PagerDuty Alerting

Building robust, scalable, and observable Kubernetes clusters on AWS is a cornerstone of modern cloud-native architectures. This guide provides a comprehensive, technical walkthrough on deploying an AWS Elastic Kubernetes Service (EKS) cluster using Terraform, integrating end-to-end observability with Datadog, and ensuring critical incident management through PagerDuty. By codifying your entire infrastructure and monitoring stack, you achieve consistency, repeatability, and agility essential for high-performing DevOps teams.

Why Terraform, Datadog, and PagerDuty for EKS?

Each tool plays a pivotal role in creating a resilient and manageable EKS environment:

Terraform: Infrastructure as Code (IaC) for declarative, version-controlled provisioning of AWS resources, EKS clusters, and even Datadog/PagerDuty configurations.
Datadog: A unified observability platform offering comprehensive monitoring for metrics, logs, traces, and UX, providing deep insights into your EKS cluster and applications.
PagerDuty: An incident management platform that transforms Datadog alerts into actionable incidents, ensuring the right teams are notified immediately and efficiently.

Prerequisites

Before you begin, ensure you have the following:

AWS Account: With programmatic access and sufficient permissions to create EKS clusters, IAM roles, VPCs, EC2 instances, etc.
Datadog Account: With API and Application keys.
PagerDuty Account: With an API key for service and integration creation.
Terraform CLI: Installed (v1.0.0+ recommended).
AWS CLI: Configured with your AWS credentials.
kubectl: Installed for interacting with the EKS cluster.
Helm CLI: Installed for deploying the Datadog Agent.

Step 1: Core EKS Infrastructure with Terraform

We start by defining the fundamental network and compute resources for our EKS cluster. This includes a Virtual Private Cloud (VPC), subnets, security groups, and IAM roles.

1.1 VPC and Networking

A dedicated VPC is crucial for network isolation and control. EKS requires public and private subnets, along with an Internet Gateway and NAT Gateways for outbound access from private subnets.

1.2 IAM Roles for EKS

EKS needs specific IAM roles for the cluster itself and for the worker nodes to interact with other AWS services.

EKS Cluster Role: Allows the EKS control plane to manage resources.
EKS Node Group Role: Grants permissions to EC2 instances (worker nodes) to join the EKS cluster and access services like ECR, S3, etc.

1.3 EKS Cluster and Node Group

The heart of the deployment is the EKS cluster resource, followed by a managed node group for compute capacity. Managed node groups simplify node lifecycle management.

Step 2: Integrating Datadog Observability

Datadog provides deep insights into your EKS cluster's health, performance, and application behavior. We'll deploy the Datadog Agent using a Kubernetes Helm chart and configure basic monitors.

2.1 Datadog Agent Deployment

The Datadog Agent runs on each worker node and collects metrics, logs, and traces. We'll use the Helm provider in Terraform to deploy it.

2.2 Datadog Monitors with Terraform

Terraform can manage Datadog resources like monitors, dashboards, and integrations. This ensures your monitoring configuration is version-controlled and deployed alongside your infrastructure.

Step 3: PagerDuty for Incident Management

PagerDuty acts as the bridge between Datadog alerts and your on-call teams. We'll set up a PagerDuty service and an integration with Datadog using Terraform.

3.1 PagerDuty Service and Escalation Policy

A PagerDuty service represents a component or application that requires incident response. An escalation policy defines how incidents are routed to individuals or teams.

3.2 Datadog-PagerDuty Integration

We'll create a generic email integration in PagerDuty, which Datadog can then use to send alerts, ensuring that triggered Datadog monitors create incidents in PagerDuty.

Step 4: Comprehensive Terraform Configuration Example

Below is a simplified, yet comprehensive, example of the Terraform configuration to deploy EKS, Datadog Agent, and PagerDuty integration. Remember to replace placeholder values with your actual data.

# main.tf

# AWS Provider Configuration
provider "aws" {
  region = var.aws_region
}

# Datadog Provider Configuration
provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

# PagerDuty Provider Configuration
provider "pagerduty" {
  token = var.pagerduty_api_token
}

# VPC Module (using a community module for simplicity)
module "vpc" {
  source = "terraform-aws-modules/vpc/aws"
  version = "3.1.0"

  name = "${var.cluster_name}-vpc"
  cidr = "10.0.0.0/16"
  azs  = ["${var.aws_region}a", "${var.aws_region}b", "${var.aws_region}c"]

  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway = true
  single_nat_gateway = true

  tags = {
    Environment = var.environment
    Project     = var.cluster_name
  }
}

# EKS Cluster Module (using a community module for simplicity)
module "eks" {
  source = "terraform-aws-modules/eks/aws"
  version = "18.2.0"

  cluster_name    = var.cluster_name
  cluster_version = "1.23"
  vpc_id          = module.vpc.vpc_id
  subnet_ids      = module.vpc.private_subnets
  enable_irsa     = true # Enable IAM roles for service accounts

  # EKS Managed Node Group
  eks_managed_node_groups = {
    default = {
      min_size     = 1
      max_size     = 3
      desired_size = 2

      instance_types = ["t3.medium"]
      disk_size      = 20
      subnet_ids     = module.vpc.private_subnets
    }
  }

  tags = {
    Environment = var.environment
    Project     = var.cluster_name
  }
}

# Kubernetes Provider to interact with the EKS cluster
data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_id
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_id
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

# Helm Provider for Datadog Agent
provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}

# Deploy Datadog Agent via Helm
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "agents.config.logLevel"
    value = "INFO"
  }
  # Enable EKS-specific integrations
  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
  }
  set {
    name  = "prometheusScrape.enabled"
    value = "true"
  }
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "apm.enabled"
    value = "true"
  }
}

# PagerDuty Escalation Policy
resource "pagerduty_escalation_policy" "general_policy" {
  name      = "${var.cluster_name}-escalation-policy"
  num_loops = 2

  rule {
    time_unit = "minutes"
    start_delay_in_minutes = 5
    target {
      type = "user_reference"
      id   = var.pagerduty_user_id # Replace with a valid PagerDuty user ID
    }
  }
}

# PagerDuty Service for EKS Cluster
resource "pagerduty_service" "eks_cluster_service" {
  name                    = "${var.cluster_name}-EKS-Service"
  auto_resolve_timeout_minutes = 60 # Resolve after 60 minutes if not acknowledged
  acknowledgement_timeout_minutes = 30 # Escalate if not acknowledged in 30 minutes
  escalation_policy = pagerduty_escalation_policy.general_policy.id
}

# Datadog Monitor for EKS Node CPU Utilization
resource "datadog_monitor" "eks_node_cpu_high" {
  name                = "[EKS - ${var.environment}] High Node CPU Utilization on {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster_name:${var.cluster_name}} by {host} < 20"
  message             = <<EOF
    Node {{host.name}} in EKS cluster ${var.cluster_name} has high CPU utilization.
    @webhook-${pagerduty_service_integration.datadog_integration.id}
  EOF
  tags                = ["environment:${var.environment}", "eks", "cpu"]
  threshold_windows {
    recovery_window = "5m"
  }
  notify_no_data      = false
  new_group_delay     = 60
  no_data_timeframe   = 20
  renotify_interval   = 0
  timeout_h           = 0
  notification_preset_name = "hide_handles"
  require_full_window = true
  escalation_message  = "CPU utilization remains high. Escalating to on-call."
}

# PagerDuty integration for Datadog (Generic Webhook)
resource "pagerduty_service_integration" "datadog_integration" {
  name    = "Datadog"
  service = pagerduty_service.eks_cluster_service.id
  type    = "generic_events_api_inbound_integration"
}

# variables.tf

variable "aws_region" {
  description = "AWS region for deployment"
  type        = string
  default     = "us-east-1"
}

variable "cluster_name" {
  description = "Name for the EKS cluster"
  type        = string
  default     = "my-eks-cluster"
}

variable "environment" {
  description = "Deployment environment (e.g., dev, staging, prod)"
  type        = string
  default     = "dev"
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

variable "pagerduty_user_id" {
  description = "PagerDuty User ID for initial escalation target"
  type        = string
  sensitive   = true
}

# outputs.tf

output "eks_cluster_endpoint" {
  description = "The endpoint for the EKS cluster"
  value       = module.eks.cluster_endpoint
}

output "kubeconfig" {
  description = "Kubeconfig for the EKS cluster"
  value       = module.eks.kubeconfig
  sensitive   = true
}

output "pagerduty_integration_key" {
  description = "PagerDuty integration key for Datadog"
  value       = pagerduty_service_integration.datadog_integration.integration_key
  sensitive   = true
}
    

Step 5: Deployment

With your Terraform configuration ready, deploy your EKS cluster with integrated observability and alerting:

Initialize Terraform: Navigate to your Terraform project directory and run terraform init.
Plan the Deployment: Review the changes Terraform will apply by running terraform plan. This step helps verify your configuration.
Apply the Configuration: Execute terraform apply and confirm with yes. This will provision all resources.
Configure Kubeconfig: After successful deployment, update your local kubeconfig: aws eks update-kubeconfig --name ${var.cluster_name} --region ${var.aws_region}

Step 6: Verification

Verify the deployment:

EKS Cluster: Check cluster status: kubectl get nodes
Datadog Agent: Verify agent pods are running: kubectl get pods -n datadog. Then, log into Datadog and check the Infrastructure list for your EKS nodes and EKS dashboards.
PagerDuty: Log into PagerDuty to confirm the EKS service, escalation policy, and Datadog integration are created. Trigger a test alert from Datadog to verify PagerDuty incident creation.

Step 7: Best Practices and Advanced Considerations

To further enhance your EKS deployment:

GitOps: Implement GitOps with tools like ArgoCD or FluxCD to manage your Kubernetes application deployments from Git.
Cluster Autoscaling: Integrate the Kubernetes Cluster Autoscaler or Karpenter for dynamic scaling of your node groups based on demand.
AWS Load Balancer Controller: Use the AWS Load Balancer Controller to provision ALBs/NLBs directly from Kubernetes Ingresses or Services.
ExternalDNS: Automate DNS record management for your services.
Security: Implement Pod Security Standards (PSS), Network Policies, and regularly review IAM roles for least privilege.
Cost Optimization: Utilize Spot Instances for stateless workloads and monitor costs with Datadog's cloud cost management features.

Troubleshooting Common Issues

IAM Permissions: Most EKS deployment failures stem from incorrect IAM roles or policies. Ensure the EKS cluster and node group roles have all necessary permissions.
Network Connectivity: Verify security group rules and NACLs allow communication between control plane and worker nodes, and outbound access for Datadog agents.
Datadog Agent Not Reporting: Check agent logs (kubectl logs -f -n datadog <datadog-agent-pod>) for API key errors or network issues.
PagerDuty Alerts Not Firing: Ensure the Datadog monitor's message correctly references the PagerDuty integration key (e.g., @webhook-${pagerduty_integration_id}). Check Datadog Event Explorer for alert triggers and PagerDuty service logs for incoming events.

Conclusion

Deploying AWS EKS with Terraform, Datadog, and PagerDuty provides a robust, automated foundation for cloud-native applications. This setup ensures your infrastructure is defined as code, your services are deeply observable, and critical incidents are managed effectively. By embracing Infrastructure as Code and comprehensive observability, organizations can accelerate development, minimize downtime, and build highly resilient systems in the cloud.

CloudStack YAML