Terraform AWS EKS Cluster Provisioning with Datadog Observability and PagerDuty Integration

Architecture Pro-Tip: Always design your EKS infrastructure with modularity and security in mind. Utilize separate Terraform modules for VPC, EKS cluster, node groups, and addon deployments. Implement least privilege IAM roles for both the EKS cluster and its node groups. For critical production workloads, consider private EKS endpoints and strict network ACLs to enhance security posture.

Mastering Terraform AWS EKS with Datadog & PagerDuty Integration

In the fast-paced world of modern DevOps, robust and automated infrastructure provisioning, coupled with comprehensive observability and reliable incident management, is paramount. This guide provides a detailed, technical walkthrough on how to provision an AWS Elastic Kubernetes Service (EKS) cluster using Terraform, integrate Datadog for end-to-end observability, and establish seamless incident alerting with PagerDuty. This powerful combination ensures your Kubernetes workloads are not only deployed efficiently but also monitored proactively and managed effectively when issues arise.

Prerequisites

Before you begin, ensure you have the following:

An AWS Account with administrative access.
Terraform (v1.0+) installed locally.
AWS CLI configured with appropriate credentials.
kubectl installed for interacting with the Kubernetes cluster.
helm installed for deploying Kubernetes packages.
A Datadog Account with an API and Application Key.
A PagerDuty Account with permissions to create Services and Integration Keys.

Core Concepts Explained

Terraform: Infrastructure as Code (IaC)

Terraform is an open-source IaC tool that allows you to define and provision cloud and on-premise resources using a declarative configuration language. It enables consistent, repeatable, and version-controlled infrastructure deployments, making it ideal for managing complex environments like AWS EKS.

AWS EKS: Managed Kubernetes Service

AWS EKS provides a managed Kubernetes control plane, significantly simplifying the deployment, management, and scaling of Kubernetes applications in the AWS cloud. EKS handles patching, upgrades, and high availability of the control plane, allowing you to focus on your applications.

Datadog: Unified Observability Platform

Datadog is a comprehensive monitoring, logging, and security platform for cloud applications. It collects metrics, logs, and traces from your entire infrastructure and applications, providing real-time visibility and powerful analytics. For EKS, Datadog offers deep insights into cluster health, pod performance, network traffic, and more.

PagerDuty: Incident Management for Critical Alerts

PagerDuty is an incident management platform that centralizes alerts from various monitoring systems, routes them to the right on-call teams, and facilitates rapid incident resolution. Integrating Datadog with PagerDuty ensures that critical issues detected in your EKS cluster trigger immediate notifications to the responsible personnel.

Terraform Project Structure

A well-organized Terraform project structure is key for maintainability. We recommend the following layout:

main.tf: Defines the primary resources like the EKS cluster, node groups, and core configurations.
variables.tf: Declares input variables for customizable parameters (e.g., region, instance types).
outputs.tf: Specifies output values that can be referenced by other Terraform configurations or for external use (e.g., EKS cluster endpoint, kubeconfig).
providers.tf: Configures the AWS, Kubernetes, and Datadog providers.
vpc.tf: Defines the VPC, subnets, and associated networking components for EKS.
datadog.tf: Configures Datadog integrations and monitors.
pagerduty.tf: Sets up PagerDuty services and alert routing.

Provisioning AWS EKS with Terraform

The first step is to define the networking infrastructure and then the EKS cluster itself.

VPC and Networking for EKS

EKS clusters require a robust VPC setup. Best practice dictates using private subnets for your worker nodes and public subnets for load balancers or bastion hosts, along with NAT Gateways for outbound internet access from private subnets. You can leverage the official terraform-aws-modules/vpc/aws module for a streamlined approach.

EKS Cluster and Node Groups

With the VPC in place, you can define your EKS cluster. We'll use the terraform-aws-modules/eks/aws module, which simplifies EKS deployment significantly. It handles the EKS control plane, IAM roles, security groups, and optionally, managed node groups or Fargate profiles.

Integrating Datadog for Observability

Once your EKS cluster is operational, deploying the Datadog Agent is crucial for collecting metrics, logs, and traces.

Deploying the Datadog Agent

The Datadog Agent typically runs as a DaemonSet on your EKS worker nodes, ensuring an agent instance runs on every node. The recommended way to deploy it is via the Datadog Helm chart. You can manage this deployment directly with Terraform using the helm_release resource from the Kubernetes provider, or by applying Kubernetes manifests.

You'll need your Datadog API key and application key (or KMS encrypted versions) to configure the agent. Ensure the agent has appropriate IAM permissions if using IRSA (IAM Roles for Service Accounts) for enhanced security.

Configuring Datadog Monitors

After the agent is running, you can define Datadog monitors to alert on critical EKS metrics. Examples include high CPU/memory utilization, node not ready, pod restarts, or critical application logs. Terraform's Datadog provider (DataDog/datadog) allows you to define these monitors as code, enabling version control and automated deployment.

Seamless PagerDuty Incident Management

Integrating Datadog with PagerDuty transforms alerts into actionable incidents, ensuring your on-call teams are promptly notified of critical issues.

Setting up Datadog-PagerDuty Integration

PagerDuty integrates with Datadog primarily through a webhook. You define a PagerDuty service with a generic API integration, which provides an integration key. This key is then used in Datadog to configure an integration that sends alerts to PagerDuty. The Datadog Terraform provider can manage this integration as a datadog_integration_pagerduty resource.

Defining Alerting Strategies

When configuring your Datadog monitors, you can specify PagerDuty as an alert recipient. Use PagerDuty's escalation policies to ensure alerts are routed correctly and escalated if not acknowledged. Terraform allows you to define these escalation policies and services using the PagerDuty/pagerduty provider.

Ready-to-Use Terraform Configuration

Below is a simplified, yet comprehensive, Terraform configuration demonstrating how to provision an EKS cluster, deploy the Datadog Agent via Helm, set up a basic Datadog monitor, and integrate with PagerDuty. Remember to replace placeholder values (like YOUR_DD_API_KEY, YOUR_DD_APP_KEY, YOUR_PD_SERVICE_KEY, etc.) with your actual credentials and desired configurations.

// providers.tf
provider "aws" {
  region = "us-east-1"
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
  // To avoid circular dependency for initial setup, you might need to run
  // 'terraform apply' twice or configure kubeconfig manually for the first run.
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

// variables.tf
variable "cluster_name" {
  description = "Name of the EKS cluster"
  type        = string
  default     = "my-eks-cluster"
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

variable "pagerduty_service_name" {
  description = "Name for the PagerDuty service"
  type        = string
  default     = "EKS Cluster Alerts"
}

// main.tf - EKS Cluster & VPC
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "${var.cluster_name}-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway     = true
  single_nat_gateway     = true
  enable_dns_hostnames   = true
  enable_dns_support     = true
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0"

  cluster_name    = var.cluster_name
  cluster_version = "1.28"

  vpc_id                   = module.vpc.vpc_id
  subnet_ids               = module.vpc.private_subnets
  control_plane_subnet_ids = module.vpc.public_subnets

  eks_managed_node_groups = {
    default = {
      min_size     = 1
      max_size     = 3
      desired_size = 1

      instance_types = ["t3.medium"]
      ami_type       = "AL2_x86_64"
    }
  }

  tags = {
    Environment = "Dev"
    Project     = "EKS-Datadog-PagerDuty"
  }
}

data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_name
}

// datadog.tf - Datadog Agent & Monitor
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Or dedicated 'datadog' namespace
  version    = "2.38.0" # Use a specific chart version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
  }

  depends_on = [module.eks] # Ensure EKS is ready before deploying agent
}

resource "datadog_monitor" "eks_node_cpu_high" {
  name               = "EKS Node CPU Utilization High - {{host.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:kubernetes.cpu.usage.total{*} by {host} > 80"
  message            = "High CPU utilization detected on EKS node {{host.name}}! @pagerduty-EKS_Alerts"
  tags               = ["environment:dev", "service:eks"]
  escalation_message = "CPU utilization remains high, escalating to L2."

  # Attach to Datadog's PagerDuty integration
  renotify_interval = 10
  require_full_window = false
  notify_no_data = false
  new_group_delay = 60
  no_data_timeframe = 20

  # Example of integrating directly with PagerDuty via service name
  # The Datadog integration must be set up to recognize "pagerduty-EKS_Alerts"
}

// pagerduty.tf - PagerDuty Service & Integration
resource "pagerduty_team" "devops_team" {
  name        = "DevOps Team"
  description = "Team responsible for EKS infrastructure."
}

resource "pagerduty_escalation_policy" "eks_escalation_policy" {
  name      = "EKS Critical Escalation"
  num_loops = 2
  team {
    id   = pagerduty_team.devops_team.id
    type = "team_reference"
  }

  rule {
    escalation_delay_in_minutes = 5
    target {
      type = "user"
      # Replace with your PagerDuty User ID
      id   = "P2L4R7Y" 
    }
  }
}

resource "pagerduty_service" "eks_alerts_service" {
  name                    = var.pagerduty_service_name
  auto_resolve_timeout    = 60
  acknowledgement_timeout = 30
  escalation_policy       = pagerduty_escalation_policy.eks_escalation_policy.id
  description             = "Service for critical EKS cluster alerts from Datadog."
}

resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog Integration"
  service     = pagerduty_service.eks_alerts_service.id
  type        = "generic_events_api_inbound_integration"
}

// outputs.tf
output "kubeconfig" {
  description = "Kubernetes config for accessing the EKS cluster"
  value       = module.eks.kubeconfig
  sensitive   = true
}

output "eks_cluster_name" {
  description = "Name of the EKS cluster"
  value       = module.eks.cluster_name
}

output "pagerduty_integration_key" {
  description = "PagerDuty Integration Key for Datadog"
  value       = pagerduty_service_integration.datadog_integration.integration_key
  sensitive   = true
}
    

Deployment Steps

Follow these steps to deploy your EKS cluster with integrated observability and incident management:

Initialize Terraform: Navigate to your project directory and run terraform init to download the necessary providers and modules.
Plan the Deployment: Execute terraform plan to review the resources Terraform will create, modify, or destroy. This is a crucial step for verifying your configuration.
Apply the Configuration: Run terraform apply and confirm with yes when prompted. This will provision your AWS EKS cluster, deploy the Datadog Agent, and set up the Datadog monitor with PagerDuty integration.
Configure Kubeconfig: After the EKS cluster is created, update your kubeconfig file to interact with it. The EKS module automatically generates an output for this. You can use:
aws eks update-kubeconfig --name $(terraform output -raw eks_cluster_name) --region us-east-1
Or simply use the kubeconfig output directly.

Post-Deployment Verification

Verify that all components are running as expected:

EKS Cluster Status: Check your Kubernetes nodes using kubectl get nodes. You should see your worker nodes in a Ready state.
Datadog Agent Status: Verify the Datadog Agent pods are running: kubectl get pods -l app=datadog --namespace default (adjust namespace if changed).
Datadog Dashboard: Log in to your Datadog account. You should start seeing metrics, logs, and traces from your EKS cluster. Check the "Hosts" and "Kubernetes" dashboards.
PagerDuty Integration: Trigger a test alert from Datadog (e.g., manually lower a threshold for a monitor) and confirm that an incident is created in PagerDuty and routed to the correct team.

Troubleshooting and Best Practices

Common Pitfalls

IAM Permissions: Ensure the AWS user/role executing Terraform has sufficient permissions to create EKS, EC2, VPC, and IAM resources. The EKS cluster role and node group instance profiles also require specific permissions.
Networking Issues: Incorrect subnet configurations, security group rules, or NAT Gateway settings can prevent nodes from joining the cluster or agents from reporting data.
Datadog API/App Keys: Double-check that your Datadog API and Application keys are correct and have the necessary permissions.
Terraform State Locking: Use remote state (e.g., S3 backend with DynamoDB locking) for collaborative environments to prevent concurrent modifications and data corruption.

Security Considerations

Least Privilege: Apply the principle of least privilege to all IAM roles, Kubernetes service accounts (using IRSA), and Datadog/PagerDuty API keys.
Secrets Management: Store sensitive information like API keys in a secure secrets manager (AWS Secrets Manager, HashiCorp Vault) and retrieve them dynamically within Terraform, rather than hardcoding them.
Network Segmentation: Implement strict network policies within Kubernetes (NetworkPolicies) and at the VPC level (Security Groups, Network ACLs) to isolate workloads.
Regular Updates: Keep your EKS cluster, worker nodes, Datadog Agent, and Terraform providers updated to benefit from the latest security patches and features.

Conclusion

By following this guide, you have successfully provisioned a robust AWS EKS cluster using Terraform, integrated Datadog for comprehensive observability, and established a reliable incident response workflow with PagerDuty. This powerful combination automates your cloud infrastructure, provides deep insights into your Kubernetes workloads, and ensures that your critical services are always monitored and protected. Embrace these practices to build resilient, scalable, and observable cloud-native applications, enhancing your team's efficiency and reducing Mean Time To Resolution (MTTR).

CloudStack YAML