Terraform Provisioning of AWS EKS with Integrated Datadog Monitoring and PagerDuty Alerts

Architecture Pro-Tip

Always design your infrastructure as code (IaC) with modularity and reusability in mind. Separate your VPC, EKS cluster, and application deployments into distinct Terraform modules. Implement strict IAM policies for Datadog and ensure secret management (e.g., AWS Secrets Manager, Vault) for API keys. Embrace GitOps for Kubernetes deployments to maintain declarative infrastructure and application states.

In today's dynamic cloud landscape, deploying and managing Kubernetes clusters efficiently is paramount. AWS Elastic Kubernetes Service (EKS) offers a robust, scalable platform for containerized applications. However, operational excellence extends beyond just provisioning; it demands comprehensive monitoring and proactive incident response. This guide provides a detailed, technical walkthrough on how to leverage Terraform for automating the deployment of an AWS EKS cluster, seamlessly integrating Datadog for advanced observability, and configuring PagerDuty for immediate incident notification.

Why Terraform, EKS, Datadog, and PagerDuty?

Terraform: Enables Infrastructure as Code (IaC), allowing you to define and provision cloud resources in human-readable configuration files. This ensures consistency, repeatability, and version control for your infrastructure.
AWS EKS: A managed Kubernetes service that simplifies the deployment, management, and scaling of Kubernetes applications in the AWS cloud, offloading operational burdens of the control plane.
Datadog: A leading monitoring and analytics platform providing end-to-end visibility across your entire technology stack. For EKS, it offers deep insights into cluster health, pod performance, network traffic, and custom application metrics.
PagerDuty: An incident management platform that provides reliable notifications, automatic escalations, and on-call scheduling to ensure critical alerts from Datadog reach the right teams immediately, minimizing downtime.

Prerequisites

Before you begin, ensure you have the following:

AWS Account: With programmatic access and sufficient permissions to create EKS clusters, VPCs, IAM roles, etc.
Terraform CLI: Version 1.0.0+ installed.
AWS CLI: Configured with your AWS credentials.
Kubectl: Installed and configured to interact with your Kubernetes cluster.
Datadog Account: With API and Application keys generated.
PagerDuty Account: With a service and integration key created for Datadog.
Helm CLI: For deploying the Datadog Agent into EKS.

Terraform Project Structure

A recommended project structure for clarity and maintainability:

.
├── main.tf
├── variables.tf
├── outputs.tf
├── providers.tf
├── versions.tf
└── modules/
    ├── vpc/
    │   ├── main.tf
    │   └── variables.tf
    └── eks/
        ├── main.tf
        └── variables.tf

Step 1: Provisioning AWS EKS with Terraform

1.1 Configure AWS Provider and Versions

Set up your AWS provider and define required Terraform and provider versions in versions.tf and providers.tf.

1.2 Create a Dedicated VPC for EKS

EKS requires a dedicated VPC with public and private subnets, NAT Gateways, and an Internet Gateway. Using a VPC module (e.g., terraform-aws-modules/vpc/aws) is highly recommended for best practices.

1.3 Define IAM Roles for EKS

EKS requires specific IAM roles:

EKS Cluster Role: Allows the EKS service to create and manage AWS resources on your behalf.
EKS Node Group Role: Assigned to the EC2 instances that function as worker nodes, granting them necessary permissions to join the cluster and interact with other AWS services.

1.4 Deploy the EKS Cluster and Node Groups

Utilize the terraform-aws-modules/eks/aws module for a robust and opinionated EKS deployment. This module simplifies the creation of the control plane and worker node groups.

Step 2: Integrating Datadog for EKS Monitoring

Datadog provides deep visibility into your EKS clusters by deploying an agent as a DaemonSet. You'll need your Datadog API and Application keys, which should be stored securely (e.g., in AWS Secrets Manager) and referenced in Terraform.

2.1 Configure Datadog Provider

Add the Datadog provider to your providers.tf, using variables for your API and APP keys.

2.2 Create IAM Policy for Datadog Agent (Optional but Recommended)

While the Datadog agent primarily collects data via the Kubernetes API, certain integrations (e.g., EC2, EBS, ALB) benefit from IAM permissions. Create a dedicated IAM policy and role for Datadog if you plan to enable these host-level integrations.

2.3 Deploy Datadog Agent via Helm

The most effective way to deploy the Datadog Agent to EKS is using its official Helm chart. Terraform can manage Helm chart deployments using the Helm provider.

Step 3: Configuring PagerDuty Alerts via Datadog

Datadog acts as the central hub for monitoring and alerting. We'll integrate Datadog with PagerDuty to ensure critical incidents are escalated efficiently.

3.1 Integrate Datadog with PagerDuty

First, ensure your Datadog account is integrated with PagerDuty. This is typically done through the Datadog UI (Integrations -> PagerDuty). You'll need your PagerDuty Service Key. Once integrated, Datadog can send events and alerts to specified PagerDuty services.

3.2 Create Datadog Monitors with PagerDuty Integration

With the Datadog provider, you can define monitors in Terraform. These monitors can be configured to trigger alerts based on specific metrics or logs from your EKS cluster and send notifications to PagerDuty.

Example Monitor: High CPU utilization on EKS nodes.
Notification: Configure the monitor to notify @pagerduty-your-service-name in the message body.

Ready-to-use Terraform Configuration

Below is a simplified, consolidated Terraform configuration demonstrating the core components discussed. Remember to adapt this for your specific environment, security requirements, and module usage.

# providers.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.0"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
  }
  required_version = "~> 1.0"
}

provider "aws" {
  region = var.aws_region
}

provider "kubernetes" {
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args        = ["eks", "get-token", "--region", var.aws_region, "--cluster-name", module.eks.cluster_id]
  }
}

provider "helm" {
  kubernetes {
    host                   = module.eks.cluster_endpoint
    cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args        = ["eks", "get-token", "--region", var.aws_region, "--cluster-name", module.eks.cluster_id]
    }
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

# variables.tf
variable "aws_region" {
  description = "AWS region"
  type        = string
  default     = "us-east-1"
}

variable "cluster_name" {
  description = "Name of the EKS cluster"
  type        = string
  default     = "my-datadog-eks-cluster"
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_service_name" {
  description = "Name of the PagerDuty service integrated with Datadog"
  type        = string
  default     = "my-eks-critical-alerts" # This should match your Datadog PagerDuty integration name
}

# main.tf (Consolidated for brevity - in production, use modules!)
data "aws_availability_zones" "available" {
  state = "available"
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "${var.cluster_name}-vpc"
  cidr = "10.0.0.0/16"

  azs             = slice(data.aws_availability_zones.available.names, 0, 3)
  public_subnets  = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  private_subnets = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    "kubernetes.io/cluster/${var.cluster_name}" = "owned"
  }
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0"

  cluster_name    = var.cluster_name
  cluster_version = "1.28" # Specify desired EKS version

  vpc_id                   = module.vpc.vpc_id
  subnet_ids               = module.vpc.private_subnets
  control_plane_subnet_ids = module.vpc.public_subnets # Public subnets for control plane access if desired

  eks_managed_node_groups = {
    default = {
      min_size     = 1
      max_size     = 3
      desired_size = 2

      instance_types = ["t3.medium"]
      capacity_type  = "ON_DEMAND"
    }
  }

  tags = {
    Environment = "Dev"
    Project     = "EKS-Datadog-PagerDuty"
  }
}

resource "helm_release" "datadog_agent" {
  name       = "datadog-agent"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "kubeStateMetricsExternal.enabled"
    value = "true"
  }

  set {
    name  = "datadog.site"
    value = "us5.datadoghq.com" # Example: check your Datadog region
  }

  set {
    name  = "tags"
    value = "{kube_cluster_name: ${var.cluster_name}}"
  }

  depends_on = [
    module.eks.cluster_id
  ]
}

resource "datadog_monitor" "eks_node_cpu_alert" {
  name               = "EKS Node CPU Utilization High (${var.cluster_name})"
  type               = "metric alert"
  query              = "avg(last_5m):avg:kubernetes.cpu.usage.total{kube_cluster_name:${var.cluster_name}} by {host} > 80"
  message            = "CPU usage on EKS node {{host.name}} is above 80% for the last 5 minutes. @pagerduty-${var.pagerduty_service_name}"
  tags               = ["environment:dev", "team:devops", "eks_cluster:${var.cluster_name}"]
  no_data_timeframe  = 20
  include_tags       = true
  notify_no_data     = false
  renotify_interval  = 60 # Renotify every 60 minutes if alert persists
  timeout_h          = 0 # No auto-resolve timeout
  evaluation_delay   = 300 # Evaluate after 5 minutes of data
}

output "kubeconfig" {
  description = "Kubeconfig command for EKS cluster"
  value       = "aws eks update-kubeconfig --region ${var.aws_region} --name ${module.eks.cluster_id}"
}

output "datadog_dashboard_url" {
  description = "Link to Datadog EKS Overview Dashboard (requires manual creation initially)"
  value       = "https://app.datadoghq.com/dash/list?filter=kubernetes&text=${var.cluster_name}"
}

Deployment and Verification

1. Initialize and Plan

Navigate to your Terraform project directory and run:

terraform init
terraform plan
    

Review the plan carefully to ensure all resources will be created as expected.

2. Apply Configuration

Execute the apply command to provision your infrastructure:

terraform apply --auto-approve
    

This process can take 15-25 minutes as EKS cluster creation is time-consuming.

3. Verify EKS Cluster

Once applied, configure your kubectl and check the cluster status:

$(terraform output -raw kubeconfig)
kubectl get nodes
kubectl get pods -n datadog
    

You should see your EKS nodes and Datadog agent pods running.

4. Verify Datadog Integration

Log in to your Datadog account. You should start seeing metrics and logs from your new EKS cluster under the Kubernetes integration dashboard. Verify that the Datadog monitor for EKS Node CPU Utilization has been created.

5. Test PagerDuty Alert

To test the PagerDuty integration, you can deliberately trigger the Datadog monitor (e.g., by running a CPU-intensive workload on an EKS node, or by adjusting the monitor threshold temporarily). Verify that an incident is created in PagerDuty.

Best Practices and Troubleshooting

Best Practices

State Management: Always use a remote backend (e.g., S3 with DynamoDB locking) for Terraform state to enable collaboration and prevent corruption.
Modularity: Break down your Terraform configuration into logical modules (VPC, EKS, Datadog, etc.) for better organization and reusability.
Security: Implement strict IAM roles with least privilege. Avoid hardcoding sensitive information; use environment variables or a secrets manager.
Cost Optimization: Choose appropriate instance types for EKS nodes, consider Graviton instances, and implement cluster autoscaling.
Alert Fatigue: Fine-tune your Datadog monitors to prevent alert storms. Use composite monitors and machine learning-driven anomaly detection where appropriate.

Troubleshooting

Terraform Errors: Read error messages carefully. Use terraform validate and terraform fmt.
EKS Cluster Issues: Check EKS control plane logs in CloudWatch. Ensure your IAM roles have correct trust policies and permissions.
Datadog Agent Not Reporting: Check Datadog Agent pod logs (kubectl logs -n datadog <datadog-pod-name>). Verify API/APP keys and network connectivity to Datadog endpoints.
PagerDuty Alerts Not Triggering: Confirm the Datadog-PagerDuty integration is correctly set up. Check Datadog monitor event streams for alerts, and ensure the @pagerduty-your-service-name notification is correctly included in the monitor message.

Conclusion

By following this guide, you've established a robust, automated pipeline for deploying and managing an AWS EKS cluster with comprehensive observability and incident response capabilities. Terraform streamlines infrastructure provisioning, Datadog provides invaluable insights into your Kubernetes workloads, and PagerDuty ensures critical issues are addressed promptly. This powerful combination significantly enhances your operational efficiency and helps maintain the reliability of your cloud-native applications. Continue to iterate on your configurations, integrate more services, and refine your monitoring strategies for optimal DevOps maturity.

CloudStack YAML