Terraform for AWS EKS Provisioning with Datadog and PagerDuty Incident Management

Architecture Pro-Tip: Always separate your core infrastructure (VPC, EKS cluster) from your application deployments. Use Terraform workspaces or distinct directories for environments (dev, staging, prod) and consider GitOps for application deployments atop EKS to maintain a clear separation of concerns and enhance security.

Terraform for AWS EKS Provisioning with Datadog and PagerDuty Incident Management

In today's dynamic cloud-native landscape, robust infrastructure automation, comprehensive observability, and efficient incident response are paramount for maintaining highly available and performant applications. This guide provides a detailed, technical walkthrough on leveraging Terraform for AWS EKS provisioning, seamlessly integrating Datadog for Kubernetes monitoring, and automating incident management with PagerDuty. This powerful trifecta ensures your Kubernetes clusters are not only provisioned consistently but also deeply observable and resilient against operational disruptions.

Why Terraform, EKS, Datadog, and PagerDuty?

Each component plays a critical role in a modern DevOps workflow:

Terraform: As an Infrastructure-as-Code (IaC) tool, Terraform enables declarative definition and provisioning of cloud resources. It ensures consistency, version control, and repeatability for your AWS EKS clusters and all supporting infrastructure.
AWS EKS: Amazon Elastic Kubernetes Service (EKS) provides a managed Kubernetes control plane, simplifying the deployment, management, and scaling of containerized applications without requiring you to install, operate, and maintain your own Kubernetes control plane.
Datadog: A leading monitoring and analytics platform that offers full-stack observability. For Kubernetes, Datadog provides unparalleled insights into cluster health, application performance, and log analysis, crucial for proactive issue identification.
PagerDuty: An incident management platform that automates the alerting and on-call scheduling process. By integrating with Datadog, PagerDuty ensures critical alerts are promptly escalated to the right teams, minimizing downtime and improving response times.

Solution Architecture Overview

Our integrated solution follows a clear architecture:

Core AWS Infrastructure: Terraform provisions a dedicated Virtual Private Cloud (VPC), subnets (public and private), NAT Gateways, Internet Gateways, and security groups to securely host the EKS cluster.
EKS Cluster and Node Groups: Terraform deploys the EKS control plane and associated worker node groups (e.g., using EC2 instances or Fargate).
Datadog Agent Deployment: Once EKS is up, Terraform (or Helm via Terraform) deploys the Datadog Agent as a DaemonSet across the worker nodes and a Cluster Agent to collect cluster-wide metrics, logs, and traces.
Datadog Monitors & Alerts: Datadog monitors are configured (potentially via Terraform or Datadog APIs) to detect anomalies and critical events within the EKS cluster and applications.
PagerDuty Integration: Datadog alerts are configured to trigger incidents in PagerDuty services. Terraform can also provision PagerDuty services, escalation policies, and users.
Incident Response: PagerDuty automates on-call notifications, escalations, and incident tracking, ensuring timely resolution.

Prerequisites

Before you begin, ensure you have the following:

AWS Account: With programmatic access and sufficient permissions (e.g., AdministratorAccess for simplicity, or finely-grained IAM policies).
Terraform CLI: Installed (v1.0.0+ recommended).
AWS CLI: Configured with your credentials.
Kubectl: Installed for interacting with the EKS cluster.
Helm CLI: Installed (v3+ recommended) for deploying Datadog agent.
Datadog API Key and Application Key: Obtain these from your Datadog account.
PagerDuty API Key: Obtain a full-access API key from your PagerDuty account.

Step-by-Step Terraform Configuration

We'll structure our Terraform configuration into logical files for clarity and maintainability.

1. Provider Configuration and Variables (`main.tf` & `variables.tf`)

Define your AWS region, and configure Datadog and PagerDuty providers.

2. Network Infrastructure (`vpc.tf`)

Provision a dedicated VPC, public and private subnets, an Internet Gateway, and NAT Gateways for outbound internet access from private subnets.

3. AWS EKS Cluster (`eks.tf`)

Create the EKS cluster and a default managed node group. We use the official terraform-aws-modules/eks/aws module for best practices.

4. Datadog Agent Deployment on EKS (`datadog.tf`)

Use the Kubernetes provider and Helm provider to deploy the Datadog Agent. This typically involves configuring an IAM role for service accounts (IRSA) for enhanced security.

5. PagerDuty Service and Integration (`pagerduty.tf`)

Define a PagerDuty service, an escalation policy, and link it with a Datadog integration key.

6. Datadog Monitors (`datadog_monitors.tf`)

Create example Datadog monitors that alert on critical EKS metrics and integrate with PagerDuty.

Complete Terraform Configuration Example

Here’s a consolidated example to get you started. Remember to replace placeholder values with your actual data.

# main.tf
provider "aws" {
  region = var.aws_region
}

provider "kubernetes" {
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  token                  = data.aws_eks_cluster_auth.this.token
}

provider "helm" {
  kubernetes {
    host                   = module.eks.cluster_endpoint
    cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
    token                  = data.aws_eks_cluster_auth.this.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_key
}

# variables.tf
variable "aws_region" { description = "AWS region" type = string default = "us-east-1" }
variable "project_name" { description = "Project name for resource tagging" type = string default = "eks-datadog-pd" }
variable "datadog_api_key" { sensitive = true }
variable "datadog_app_key" { sensitive = true }
variable "pagerduty_api_key" { sensitive = true }
variable "pagerduty_team_id" { description = "ID of the PagerDuty team responsible for incidents" type = string }
variable "pagerduty_user_ids" { description = "List of PagerDuty user IDs for escalation policy" type = list(string) }

# vpc.tf
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name                 = "${var.project_name}-vpc"
  cidr                 = "10.0.0.0/16"
  azs                  = ["${var.aws_region}a", "${var.aws_region}b", "${var.aws_region}c"]
  private_subnets      = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets       = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]
  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true
}

# eks.tf
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0"

  cluster_name    = "${var.project_name}-cluster"
  cluster_version = "1.28"
  vpc_id          = module.vpc.vpc_id
  subnet_ids      = module.vpc.private_subnets

  eks_managed_node_groups = {
    default = {
      min_size     = 2
      max_size     = 5
      desired_size = 3
      instance_types = ["t3.medium"]
    }
  }

  tags = {
    Project = var.project_name
  }
}

data "aws_eks_cluster_auth" "this" {
  name = module.eks.cluster_name
}

# datadog.tf (Using Helm to deploy Datadog Agent)
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true
  version    = "2.33.0" # Use a specific stable version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }
  set {
    name  = "datadog.kubeStateMetricsCore.enabled"
    value = "true"
  }
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "clusterAgent.kubeStateMetricsEnabled"
    value = "true"
  }
  set {
    name  = "agents.hostPort"
    value = "8125" # Required for host-level check and APM
  }
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "processAgent.enabled"
    value = "true"
  }
  set {
    name  = "apm.enabled"
    value = "true"
  }
}

# pagerduty.tf
resource "pagerduty_escalation_policy" "eks_escalation_policy" {
  name      = "${var.project_name}-EKS-Escalation-Policy"
  num_loops = 2
  team_id   = var.pagerduty_team_id
  rule {
    escalation_delay_in_minutes = 5
    target {
      type = "user_reference"
      id   = var.pagerduty_user_ids[0]
    }
  }
  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "user_reference"
      id   = var.pagerduty_user_ids[1] # Ensure this user exists or use a schedule ID
    }
  }
}

resource "pagerduty_service" "eks_monitoring_service" {
  name                    = "${var.project_name}-EKS-Monitoring"
  auto_resolve_timeout_minutes = 60 # Auto-resolve incidents after 60 min if no new events
  acknowledgement_timeout_minutes = 30
  escalation_policy      = pagerduty_escalation_policy.eks_escalation_policy.id
  team                   = var.pagerduty_team_id
}

resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog Integration"
  service_id  = pagerduty_service.eks_monitoring_service.id
  type        = "datadog_api_inbound_integration"
}

# datadog_monitors.tf
resource "datadog_monitor" "eks_node_cpu_utilization" {
  name               = "[EKS] High Node CPU Utilization on ${var.project_name}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${module.eks.cluster_name},kube_namespace:default} by {host} > 80"
  message            = "CPU usage on EKS node {{host.name}} is {{value}}% above 80%. @pagerduty-${pagerduty_service_integration.datadog_integration.integration_key}"
  escalation_message = "Still high CPU! Please investigate immediately. @slack-devops"
  tags               = ["environment:production", "service:eks", "team:devops"]
  renotify_interval  = 60
  notify_no_data     = false
  threshold_windows {
    recovery_window = "5m"
  }
  critical_threshold = 80
  warning_threshold  = 70
}

resource "datadog_monitor" "eks_critical_pod_restart" {
  name  = "[EKS] Critical Pod Restarts in ${var.project_name}"
  type  = "metric alert"
  query = "sum(last_5m):sum:kubernetes.pod.restarts{cluster_name:${module.eks.cluster_name},kube_namespace:default} > 5"
  message = "More than 5 pod restarts in the last 5 minutes in default namespace. Investigate. @pagerduty-${pagerduty_service_integration.datadog_integration.integration_key}"
  tags    = ["environment:production", "service:eks", "team:devops"]
  critical_threshold = 5
}

Deployment Steps

Follow these steps to deploy your EKS cluster with Datadog and PagerDuty integration:

Save the Configuration: Save the code above into .tf files (e.g., main.tf, variables.tf, vpc.tf, eks.tf, datadog.tf, pagerduty.tf, datadog_monitors.tf) in a dedicated directory.
Initialize Terraform: Open your terminal, navigate to the directory, and run terraform init to download providers and modules.
Review Plan: Execute terraform plan to see the infrastructure changes Terraform proposes. Carefully review the output to ensure it matches your expectations.
Apply Configuration: Run terraform apply and type yes when prompted to provision the resources. This step will create your VPC, EKS cluster, deploy the Datadog agent, and configure PagerDuty services and Datadog monitors.
Configure Kubeconfig: After the EKS cluster is created, update your kubeconfig file to interact with it:
aws eks update-kubeconfig --name ${var.project_name}-cluster --region ${var.aws_region}
Verify Deployment:
- Check EKS nodes: kubectl get nodes
- Check Datadog agents: kubectl get pods -n datadog (ensure pods are running)
- Log into Datadog and PagerDuty portals to confirm monitors, services, and integrations are visible.

Troubleshooting and Best Practices

IAM Permissions: Ensure your AWS credentials have sufficient permissions to create all specified resources. EKS-related IAM roles (cluster role, node group role) are crucial.
Provider Versions: Always pin your provider and module versions (e.g., version = "~> 5.0"). This prevents unexpected changes from new releases.
Secrets Management: Do not hardcode API keys. Use Terraform variables with the sensitive = true flag and retrieve values from secure sources like AWS Secrets Manager or HashiCorp Vault.
State Management: Use a remote backend (e.g., AWS S3 with DynamoDB locking) for Terraform state to enable team collaboration and prevent state corruption.
Datadog Agent Health: If Datadog metrics aren't appearing, check the agent logs: kubectl logs -f -n datadog <datadog-agent-pod-name>. Verify API and APP keys.
PagerDuty Escalation: Test your PagerDuty integration by manually triggering a Datadog test event or a low-severity alert to ensure the escalation policy works as expected.
Cost Optimization: Monitor your EKS node group sizes and types. Use Spot Instances for non-critical workloads or leverage Cluster Autoscaler and Karpenter for efficient scaling.

Conclusion

By meticulously defining your AWS EKS infrastructure with Terraform and integrating it with Datadog for observability and PagerDuty for automated incident response, you establish a robust, scalable, and resilient cloud-native platform. This comprehensive approach not only streamlines provisioning and operations but also significantly enhances your team's ability to monitor performance, detect issues proactively, and respond to incidents efficiently, ensuring high availability for your critical applications.

CloudStack YAML