Terraform for AWS EKS Cluster Provisioning with Datadog Monitoring and PagerDuty Incident Response

In the world of modern cloud infrastructure, managing Kubernetes clusters efficiently and ensuring their reliability is paramount. This comprehensive guide delves into leveraging Terraform for automated provisioning of AWS EKS (Elastic Kubernetes Service) clusters, seamlessly integrating Datadog for robust monitoring, and configuring PagerDuty for proactive incident response. This powerful trio streamlines your DevOps workflow, enabling scalable, observable, and resilient Kubernetes deployments.

Architecture Pro-Tip: Immutable Infrastructure & Modular Design

Always design your Terraform configurations with immutable infrastructure principles in mind. Each component (VPC, EKS, Datadog agents, PagerDuty services) should be managed as a distinct module, promoting reusability and minimizing state drift. Isolate environments (dev, staging, prod) using separate workspaces or dedicated AWS accounts. This modular approach simplifies updates, rollbacks, and enhances overall infrastructure reliability and maintainability.

Why This Stack? The Power of Integration

Combining Terraform, AWS EKS, Datadog, and PagerDuty creates a formidable platform for managing containerized applications:

Terraform: Enables Infrastructure as Code (IaC), allowing you to define, provision, and manage your entire cloud environment in a declarative manner. This reduces manual errors, promotes version control, and accelerates deployment cycles.
AWS EKS: A fully managed Kubernetes service that simplifies the deployment, management, and scaling of containerized applications using Kubernetes on AWS. It handles the Kubernetes control plane's availability and patching.
Datadog: A leading monitoring and analytics platform that provides end-to-end observability for your EKS clusters, applications, and underlying AWS infrastructure. It offers comprehensive metrics, logs, traces, and synthetic monitoring.
PagerDuty: An incident management platform that provides real-time alerts, on-call scheduling, and automated escalation policies, ensuring that critical issues are addressed promptly by the right team members.

Prerequisites

Before you begin, ensure you have the following:

An AWS Account with programmatic access keys configured.
Terraform CLI installed (version 1.0+ recommended).
AWS CLI installed and configured.
kubectl CLI installed.
helm CLI installed (for Datadog Agent).
A Datadog Account with API and Application Keys.
A PagerDuty Account with an API Token.

Step-by-Step Guide: Provisioning and Integration

1. Project Setup and Provider Configuration

Create a new directory for your Terraform project. Initialize your providers, including AWS, Datadog, and PagerDuty. It's recommended to use an S3 backend for state management.

Create a main.tf:

provider "aws" {
  region = "us-east-1"
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

terraform {
  backend "s3" {
    bucket         = "my-terraform-state-bucket"
    key            = "eks/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "my-terraform-locks"
  }
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
    pagerduty = {
      source  = "PagerDuty/pagerduty"
      version = "~> 2.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
  }
}

Also, create a variables.tf for sensitive keys and other parameters:

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

variable "cluster_name" {
  description = "Name of the EKS cluster"
  type        = string
  default     = "my-eks-cluster"
}

variable "vpc_id" {
  description = "ID of the VPC to deploy EKS into"
  type        = string
}

variable "subnet_ids" {
  description = "List of private subnet IDs for EKS"
  type        = list(string)
}

And a terraform.tfvars (or pass these as environment variables) for values:

datadog_api_key     = "YOUR_DATADOG_API_KEY"
datadog_app_key     = "YOUR_DATADOG_APP_KEY"
pagerduty_api_token = "YOUR_PAGERDUTY_API_TOKEN"
vpc_id              = "vpc-0123456789abcdef0"
subnet_ids          = ["subnet-0abcdef1234567890", "subnet-0fedcba9876543210"]

2. Provisioning the AWS EKS Cluster

We'll use the official terraform-aws-modules/eks/aws module for simplicity, which handles VPC, EKS cluster, and node groups.

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.1"

  cluster_name    = var.cluster_name
  cluster_version = "1.28"

  vpc_id                   = var.vpc_id
  subnet_ids               = var.subnet_ids
  control_plane_subnet_ids = var.subnet_ids # For private EKS endpoint

  # EKS Cluster Security Group Rules
  cluster_security_group_additional_rules = {
    ingress_self_all = {
      description = "EKS Control Plane to EKS Workers"
      protocol    = "-1"
      from_port   = 0
      to_port     = 0
      type        = "ingress"
      self        = true
    }
  }

  eks_managed_node_groups = {
    default = {
      min_size     = 2
      max_size     = 5
      desired_size = 3

      instance_types = ["t3.medium"]
      capacity_type  = "ON_DEMAND"
      # Additional IAM policies for node group, e.g., for ECR access
      iam_role_additional_policies = {
        AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
      }
    }
  }

  tags = {
    Environment = "production"
    Project     = "EKS-Datadog-PagerDuty"
  }
}

output "kubeconfig" {
  description = "Kubectl configuration file content"
  value       = module.eks.kubeconfig
  sensitive   = true
}

output "cluster_endpoint" {
  description = "Endpoint for EKS control plane."
  value       = module.eks.cluster_endpoint
}

output "cluster_certificate_authority_data" {
  description = "Base64 encoded certificate data required to communicate with your cluster."
  value       = module.eks.cluster_certificate_authority_data
}

3. Integrating Datadog Monitoring

We'll deploy the Datadog Agent to the EKS cluster using the Helm provider and define a basic Datadog monitor via the Datadog provider.

First, configure the Kubernetes and Helm providers to connect to your newly created EKS cluster:

data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}

Now, deploy the Datadog Agent using Helm:

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "kubeStateMetricsExternal.enabled"
    value = "true"
  }

  set {
    name  = "targetSystem"
    value = "linux"
  }

  set {
    name = "tags"
    value = "{environment:production,cluster_name:${var.cluster_name}}"
  }
  # Enable APM, Log Collection, Process Monitoring
  set {
    name  = "datadog.apm.enabled"
    value = "true"
  }
  set {
    name  = "datadog.logs.enabled"
    value = "true"
  }
  set {
    name  = "datadog.logs.containerCollectAll"
    value = "true"
  }
  set {
    name  = "datadog.processAgent.enabled"
    value = "true"
  }
}

Create a basic Datadog monitor (e.g., for high CPU utilization in EKS nodes):

resource "datadog_monitor" "high_node_cpu" {
  name                = "[EKS] High Node CPU Utilization on ${var.cluster_name}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.cluster_name}} by {host} > 80"
  message             = "EKS Node CPU utilization is above 80% for {{host.name}} in cluster ${var.cluster_name}. @webhook-pagerduty"
  monitor_threshold_windows {
    recovery_window = "10m"
    trigger_window  = "5m"
  }
  require_full_window     = false
  notify_no_data          = false
  new_host_delay          = 300
  renotify_interval       = 0
  escalation_message      = "CPU is still high after 15 minutes, escalating to on-call."
  no_data_timeframe       = 20
  include_tags            = true
  enable_logs_sample      = true
  force_delete            = false # Set to true for easier testing, but use with caution in production

  tags = ["environment:production", "service:eks", "severity:high"]
}

4. Configuring PagerDuty Incident Response

Automate the creation of a PagerDuty service and an integration with Datadog.

resource "pagerduty_team" "devops" {
  name        = "DevOps Team"
  description = "Team responsible for EKS infrastructure"
}

resource "pagerduty_escalation_policy" "eks_primary_ep" {
  name      = "EKS Primary Escalation Policy"
  team_id   = pagerduty_team.devops.id
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 5
    target {
      type = "user" # Or team, schedule
      id   = "YOUR_PAGERDUTY_USER_ID" # Replace with actual PagerDuty user ID
    }
  }

  rule {
    escalation_delay_in_minutes = 15
    target {
      type = "user"
      id   = "YOUR_PAGERDUTY_SECONDARY_USER_ID" # Replace
    }
  }
}

resource "pagerduty_service" "eks_service" {
  name                      = "${var.cluster_name}-EKS Service"
  auto_resolve_timeout_days = 1
  acknowledgement_timeout_minutes = 10
  escalation_policy           = pagerduty_escalation_policy.eks_primary_ep.id
  alert_creation              = "create_alerts_and_incidents"
  description                 = "PagerDuty service for incidents related to ${var.cluster_name} EKS Cluster."
}

resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog Integration"
  service     = pagerduty_service.eks_service.id
  type        = "generic_events_api_inbound_integration"
}

resource "datadog_integration_webhook" "pagerduty_webhook" {
  name = "PagerDuty"
  url  = pagerduty_service_integration.datadog_integration.html_url # This URL contains the integration key
}

# Link Datadog monitor to PagerDuty via webhook
# The monitor message needs to include "@webhook-pagerduty" for Datadog to use this webhook.
# Update the datadog_monitor resource above with "message = ... @webhook-pagerduty"
}

Remember to replace YOUR_PAGERDUTY_USER_ID with actual user IDs from your PagerDuty account. You can find these IDs in the PagerDuty UI or via the PagerDuty API.

The @webhook-pagerduty in the Datadog monitor message is crucial. Datadog will use this keyword to send alerts to the webhook configured above, which in turn routes them to your PagerDuty service.

5. Apply Your Configuration

Navigate to your project directory in the terminal and execute:

terraform init
terraform plan
terraform apply --auto-approve

This will provision the EKS cluster, deploy the Datadog Agent, set up Datadog monitors, and configure PagerDuty services and integrations.

Ready-to-Use Configuration Example (`main.tf`)

Here's a consolidated example of the Terraform code, combining the EKS cluster, Datadog Agent deployment, Datadog monitor, and PagerDuty service definitions into a single file for quick setup. Remember to fill in your VPC/subnet details and API keys in terraform.tfvars.

# main.tf for AWS EKS, Datadog, and PagerDuty Integration

# AWS Provider Configuration
provider "aws" {
  region = "us-east-1"
}

# Datadog Provider Configuration
provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

# PagerDuty Provider Configuration
provider "pagerduty" {
  token = var.pagerduty_api_token
}

# Terraform Backend Configuration (S3 for state management)
terraform {
  backend "s3" {
    bucket         = "my-terraform-state-bucket-unique-name" # Replace with your unique S3 bucket name
    key            = "eks/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "my-terraform-locks-unique-name" # Replace with your unique DynamoDB table name
  }
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
    pagerduty = {
      source  = "PagerDuty/pagerduty"
      version = "~> 2.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
  }
}

# Variables Definition
variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

variable "pagerduty_user_id" {
  description = "PagerDuty User ID for primary escalation"
  type        = string
  sensitive   = true
}

variable "pagerduty_secondary_user_id" {
  description = "PagerDuty User ID for secondary escalation"
  type        = string
  sensitive   = true
}

variable "cluster_name" {
  description = "Name of the EKS cluster"
  type        = string
  default     = "my-eks-datadog-pd-cluster"
}

variable "vpc_id" {
  description = "ID of the VPC to deploy EKS into"
  type        = string
  default     = "vpc-0abcdef1234567890" # Replace with your VPC ID
}

variable "subnet_ids" {
  description = "List of private subnet IDs for EKS"
  type        = list(string)
  default     = ["subnet-0abcdef1234567890", "subnet-0fedcba9876543210"] # Replace with your Subnet IDs
}

# AWS EKS Cluster Provisioning
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.1"

  cluster_name    = var.cluster_name
  cluster_version = "1.28"

  vpc_id                   = var.vpc_id
  subnet_ids               = var.subnet_ids
  control_plane_subnet_ids = var.subnet_ids # For private EKS endpoint

  cluster_security_group_additional_rules = {
    ingress_self_all = {
      description = "EKS Control Plane to EKS Workers"
      protocol    = "-1"
      from_port   = 0
      to_port     = 0
      type        = "ingress"
      self        = true
    }
  }

  eks_managed_node_groups = {
    default = {
      min_size     = 2
      max_size     = 5
      desired_size = 3

      instance_types = ["t3.medium"]
      capacity_type  = "ON_DEMAND"
      iam_role_additional_policies = {
        AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
      }
    }
  }

  tags = {
    Environment = "production"
    Project     = "EKS-Datadog-PagerDuty"
  }
}

# Kubernetes and Helm Provider Configuration to connect to EKS
data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}

# Datadog Agent Deployment via Helm
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "kubeStateMetricsExternal.enabled"
    value = "true"
  }

  set {
    name  = "targetSystem"
    value = "linux"
  }

  set {
    name = "tags"
    value = "{environment:production,cluster_name:${var.cluster_name}}"
  }
  set {
    name  = "datadog.apm.enabled"
    value = "true"
  }
  set {
    name  = "datadog.logs.enabled"
    value = "true"
  }
  set {
    name  = "datadog.logs.containerCollectAll"
    value = "true"
  }
  set {
    name  = "datadog.processAgent.enabled"
    value = "true"
  }
}

# PagerDuty Team and Escalation Policy
resource "pagerduty_team" "devops" {
  name        = "DevOps Team"
  description = "Team responsible for EKS infrastructure"
}

resource "pagerduty_escalation_policy" "eks_primary_ep" {
  name      = "EKS Primary Escalation Policy - ${var.cluster_name}"
  team_id   = pagerduty_team.devops.id
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 5
    target {
      type = "user"
      id   = var.pagerduty_user_id
    }
  }

  rule {
    escalation_delay_in_minutes = 15
    target {
      type = "user"
      id   = var.pagerduty_secondary_user_id
    }
  }
}

# PagerDuty Service and Datadog Integration
resource "pagerduty_service" "eks_service" {
  name                      = "${var.cluster_name}-EKS Service"
  auto_resolve_timeout_days = 1
  acknowledgement_timeout_minutes = 10
  escalation_policy           = pagerduty_escalation_policy.eks_primary_ep.id
  alert_creation              = "create_alerts_and_incidents"
  description                 = "PagerDuty service for incidents related to ${var.cluster_name} EKS Cluster."
}

resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog Integration"
  service     = pagerduty_service.eks_service.id
  type        = "generic_events_api_inbound_integration"
}

# Datadog Webhook for PagerDuty
resource "datadog_integration_webhook" "pagerduty_webhook" {
  name = "PagerDuty"
  url  = pagerduty_service_integration.datadog_integration.html_url
}

# Datadog Monitor for High EKS Node CPU
resource "datadog_monitor" "high_node_cpu" {
  name                = "[EKS] High Node CPU Utilization on ${var.cluster_name}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.cluster_name}} by {host} > 80"
  message             = "EKS Node CPU utilization is above 80% for {{host.name}} in cluster ${var.cluster_name}. @webhook-pagerduty"
  monitor_threshold_windows {
    recovery_window = "10m"
    trigger_window  = "5m"
  }
  require_full_window     = false
  notify_no_data          = false
  new_host_delay          = 300
  renotify_interval       = 0
  escalation_message      = "CPU is still high after 15 minutes, escalating to on-call."
  no_data_timeframe       = 20
  include_tags            = true
  enable_logs_sample      = true
  force_delete            = false

  tags = ["environment:production", "service:eks", "severity:high", "cluster:${var.cluster_name}"]
}

# Outputs
output "kubeconfig" {
  description = "Kubectl configuration file content"
  value       = module.eks.kubeconfig
  sensitive   = true
}

output "cluster_endpoint" {
  description = "Endpoint for EKS control plane."
  value       = module.eks.cluster_endpoint
}

Troubleshooting and Best Practices

Common Issues:

IAM Permissions: Ensure your AWS user or role has sufficient permissions to create EKS clusters, VPC resources, and IAM roles. Likewise, Datadog and PagerDuty API keys/tokens need appropriate scopes.
Networking: EKS requires specific networking configurations. Verify your VPC, subnets, and security groups allow proper communication between the control plane and worker nodes, and outbound access for the Datadog Agent.
Kubernetes Context: After terraform apply, you might need to manually update your kubeconfig file using aws eks update-kubeconfig --name <cluster-name> --region <aws-region> to interact with the cluster via kubectl.
Datadog Agent Connectivity: If Datadog isn't reporting data, check the agent logs (kubectl logs -f -l app=datadog-agent -n datadog) and ensure network policies aren't blocking outbound traffic to Datadog endpoints.

Best Practices:

Module Your Code: For larger, more complex setups, break down your Terraform configuration into reusable modules (e.g., EKS module, Datadog monitoring module).
Version Control: Keep your Terraform code in a Git repository.
CI/CD Integration: Automate Terraform deployments using CI/CD pipelines (e.g., GitLab CI, GitHub Actions, AWS CodePipeline).
State Locking: Always use a remote backend with state locking (like S3 with DynamoDB) to prevent concurrent modifications and state corruption.
Regular Audits: Periodically review your Datadog monitors and PagerDuty escalation policies to ensure they align with your operational needs.

Conclusion

Provisioning AWS EKS clusters with Terraform, combined with the powerful observability of Datadog and the robust incident response of PagerDuty, forms a highly effective and scalable foundation for your containerized applications. By adopting Infrastructure as Code, you gain consistency, repeatability, and agility, empowering your teams to manage complex Kubernetes environments with confidence. This integrated approach ensures that your EKS clusters are not only well-provisioned but also continuously monitored, with critical issues addressed swiftly, minimizing downtime and operational impact.

CloudStack YAML