Terraform-Managed AWS EKS with Datadog and PagerDuty Incident Automation

Terraform-Managed AWS EKS with Datadog and PagerDuty Incident Automation

In the fast-evolving world of cloud-native applications, managing complex infrastructures efficiently and ensuring high availability is paramount. This guide provides a comprehensive technical walkthrough on orchestrating a robust Amazon Elastic Kubernetes Service (AWS EKS) cluster using Terraform Infrastructure as Code (IaC), integrating advanced monitoring with Datadog, and automating incident response via PagerDuty. This powerful combination empowers DevOps teams to maintain resilient, observable, and rapidly recoverable Kubernetes environments.

Architecture Pro-Tip: Immutable Infrastructure & GitOps

Embrace immutable infrastructure principles by managing all resources, including Kubernetes manifests and Datadog monitors, exclusively through Terraform. Combine this with a GitOps workflow where all infrastructure and application changes are committed to a version-controlled repository. Tools like ArgoCD or Flux can then synchronize your EKS cluster state with your Git repository, ensuring consistency, auditability, and faster recovery from failures. This setup minimizes manual errors and standardizes your deployment processes.

Why This Stack Matters for Modern DevOps

Adopting this stack offers significant advantages for organizations seeking to optimize their cloud operations:

  • Automation & Consistency with Terraform: Define and provision your entire EKS infrastructure – cluster, node groups, IAM roles, networking, and even Kubernetes add-ons – as code. This eliminates manual configuration drift, speeds up deployments, and ensures environments are consistent across development, staging, and production.
  • Comprehensive Observability with Datadog: Gain deep insights into your EKS clusters, applications, and underlying AWS infrastructure. Datadog unifies metrics, logs, traces, and network performance data, allowing for proactive issue identification, root cause analysis, and performance optimization.
  • Automated Incident Response with PagerDuty: Transform critical alerts from Datadog into actionable incidents. PagerDuty's intelligent routing, on-call scheduling, and escalation policies ensure that the right team members are notified immediately, reducing mean time to resolution (MTTR) and minimizing service disruptions.
  • Scalability & Resilience: AWS EKS provides a highly available and scalable Kubernetes control plane, while Terraform enables elastic scaling of worker nodes. Datadog and PagerDuty ensure that this dynamic environment remains under constant surveillance and any anomalies trigger immediate response.

Prerequisites

Before diving into the configuration, ensure you have the following tools and accounts set up:

  • An AWS Account with programmatic access keys configured.
  • Terraform CLI installed (v1.0.x or higher).
  • AWS CLI installed and configured.
  • kubectl CLI installed.
  • Helm CLI installed (v3.x or higher).
  • A Datadog Account with an API Key and Application Key.
  • A PagerDuty Account with an API Key (for Terraform provider) and a Service Integration Key (for Datadog).

Core Components & Terraform Configuration Strategy

We'll break down the Terraform configuration into logical modules for better maintainability and reusability.

1. AWS EKS Cluster & Networking

This involves creating the VPC, subnets, EKS cluster, and associated IAM roles. We recommend using the terraform-aws-modules/eks/aws module for a streamlined setup.

2. Datadog Agent Deployment on EKS

The Datadog Agent runs on your EKS worker nodes, collecting metrics, logs, and traces. We'll deploy it using the Terraform helm_release resource.

3. Datadog Monitors & PagerDuty Integration

Define Datadog monitors to alert on critical EKS health metrics. These monitors will be configured to trigger incidents in PagerDuty via a dedicated service integration.

4. PagerDuty Services & Escalation Policies

Manage your PagerDuty services, escalation policies, and users directly through Terraform, ensuring your incident response configuration is also version-controlled.

Step-by-Step Terraform Implementation

Let's put theory into practice. We'll provide snippets for a modular Terraform setup.

Project Structure:

. ├── main.tf ├── variables.tf ├── outputs.tf ├── versions.tf ├── modules/ │ ├── eks/ │ │ ├── main.tf │ │ ├── variables.tf │ │ └── outputs.tf │ ├── datadog_agent/ │ │ ├── main.tf │ │ └── variables.tf │ └── datadog_monitors/ │ ├── main.tf │ └── variables.tf └── pagerduty/ ├── main.tf ├── variables.tf └── outputs.tf

1. Provider and Backend Configuration (versions.tf)

terraform {
  required_version = ">= 1.0.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
    datadog = {
      source  = "datadog/datadog"
      version = "~> 3.0"
    }
    pagerduty = {
      source  = "pagerduty/pagerduty"
      version = "~> 2.0"
    }
  }

  backend "s3" {
    bucket         = "my-terraform-state-bucket" # Replace with your S3 bucket name
    key            = "eks-datadog-pagerduty/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "my-terraform-state-lock" # Replace with your DynamoDB table name
  }
}

provider "aws" {
  region = var.aws_region
}

provider "kubernetes" {
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  token                  = data.aws_eks_cluster_auth.this.token
}

provider "helm" {
  kubernetes {
    host                   = module.eks.cluster_endpoint
    cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
    token                  = data.aws_eks_cluster_auth.this.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

data "aws_eks_cluster_auth" "this" {
  name = module.eks.cluster_id
}

2. EKS Cluster Module (modules/eks/main.tf)

This module provisions the EKS cluster, VPC, and node groups. Refer to the official `terraform-aws-modules/eks/aws` documentation for full options.

# modules/eks/main.tf
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "${var.cluster_name}-vpc"
  cidr = var.vpc_cidr_block

  azs             = var.vpc_azs
  private_subnets = var.vpc_private_subnets
  public_subnets  = var.vpc_public_subnets

  enable_nat_gateway = true
  single_nat_gateway = true
  enable_dns_hostnames = true
  enable_dns_support = true

  tags = {
    Environment = var.environment
    Project     = var.project
  }
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.16"

  cluster_name    = var.cluster_name
  cluster_version = var.kubernetes_version

  vpc_id                   = module.vpc.vpc_id
  subnet_ids               = module.vpc.private_subnets
  control_plane_subnet_ids = module.vpc.intra_subnets # For Fargate or dedicated control plane if needed

  cluster_endpoint_private_access = true
  cluster_endpoint_public_access  = true

  eks_managed_node_groups = {
    default = {
      min_size     = var.node_group_min_size
      max_size     = var.node_group_max_size
      desired_size = var.node_group_desired_size
      instance_types = [var.node_group_instance_type]
      capacity_type  = "ON_DEMAND"
      labels = {
        "datadog.com/node-agent" = "true"
      }
    }
  }

  tags = {
    Environment = var.environment
    Project     = var.project
  }
}

3. Datadog Agent Module (modules/datadog_agent/main.tf)

This deploys the Datadog Agent using Helm. Your Datadog API key is required.

# modules/datadog_agent/main.tf
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true
  version    = "2.33.10" # Use a stable version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    type  = "string"
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    type  = "string"
  }

  set {
    name  = "datadog.site"
    value = "us3.datadoghq.com" # Or your Datadog site (e.g., datadoghq.com, eu.datadoghq.com)
    type  = "string"
  }

  # Enable Kubernetes integrations
  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
  }
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "clusterAgent.metricsProvider.enabled"
    value = "true"
  }
  set {
    name  = "datadog.logs.enabled"
    value = "true"
  }
  set {
    name  = "datadog.logs.containerCollectAll"
    value = "true"
  }
  set {
    name  = "datadog.apm.enabled"
    value = "true"
  }
  set {
    name  = "datadog.processAgent.enabled"
    value = "true"
  }
  set {
    name  = "datadog.processAgent.containerCollection.enabled"
    value = "true"
  }

  # Ensure the Helm release waits for resources to be ready
  wait       = true
  timeout    = 300
}

4. PagerDuty Services & Escalation Policy (pagerduty/main.tf)

Define an escalation policy and a service for EKS alerts. This service will receive incidents from Datadog.

# pagerduty/main.tf
resource "pagerduty_escalation_policy" "eks_escalation_policy" {
  name      = "${var.cluster_name}-EKS-Escalation-Policy"
  num_loops = 2
  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "user"
      id   = var.pagerduty_user_id # ID of a PagerDuty user
    }
  }
}

resource "pagerduty_service" "eks_monitoring_service" {
  name                    = "${var.cluster_name}-EKS-Monitoring"
  auto_resolve_timeout    = 14400 # 4 hours
  acknowledgement_timeout = 600   # 10 minutes
  escalation_policy       = pagerduty_escalation_policy.eks_escalation_policy.id

  incident_urgency_rule {
    type    = "constant"
    urgency = "high"
  }
}

resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog Integration"
  vendor      = "P5G77I9" # PagerDuty vendor ID for Datadog
  service     = pagerduty_service.eks_monitoring_service.id
  type        = "generic_events_api_inbound_integration" # Or the specific Datadog integration type if available
}

output "pagerduty_datadog_integration_key" {
  description = "The integration key for Datadog to send events to PagerDuty."
  value       = pagerduty_service_integration.datadog_integration.integration_key
  sensitive   = true
}

5. Datadog Monitors Module (modules/datadog_monitors/main.tf)

Define critical Datadog monitors that integrate with the PagerDuty service. We'll create a basic CPU utilization monitor and a Pod restart monitor.

# modules/datadog_monitors/main.tf
resource "datadog_monitor" "eks_node_cpu_utilization" {
  name              = "[EKS] Node CPU Utilization High on {{host.name}}"
  type              = "metric alert"
  query             = "avg(last_5m):avg:system.cpu.idle{eks_cluster_name:${var.cluster_name}} by {host} < 20" # Alert if idle < 20% (i.e., usage > 80%)
  message           = <<EOM
@pagerduty-${var.pagerduty_service_name}
EKS Node CPU Utilization is high. Host: {{host.name}} ({{host.id}}). Current usage: {{value}}%
Please investigate for potential overloads or runaway processes.
EOM
  monitor_threshold_windows {
    recovery_window = "15m"
    trigger_window  = "10m"
  }
  escalation_message = "CPU utilization remains high. Escalating to next level."
  notify_no_data     = false
  renotify_interval  = 60
  no_data_timeframe  = 20
  include_tags       = true
  tags               = ["environment:${var.environment}", "eks", "cpu"]
  new_group_delay    = 300
  restricted_roles   = [] # Add specific role IDs if desired for restricted editing
}

resource "datadog_monitor" "eks_pod_restarts" {
  name    = "[EKS] High Pod Restart Rate in {{kube_cluster_name}} - {{kube_namespace}} / {{kube_deployment}}"
  type    = "metric alert"
  query   = "sum(last_5m):sum:kubernetes.containers.restarts{kube_cluster_name:${var.cluster_name}} by {kube_cluster_name,kube_namespace,kube_deployment} > 5" # More than 5 restarts in 5 minutes
  message = <<EOM
@pagerduty-${var.pagerduty_service_name}
High Pod restart rate detected in EKS cluster {{kube_cluster_name}}.
Namespace: {{kube_namespace}}
Deployment: {{kube_deployment}}
Total restarts in last 5 min: {{value}}
Investigate logs for Pods in this deployment.
EOM
  monitor_threshold_windows {
    recovery_window = "15m"
    trigger_window  = "10m"
  }
  escalation_message = "Pod restarts continue. Escalating."
  notify_no_data     = false
  renotify_interval  = 60
  no_data_timeframe  = 20
  include_tags       = true
  tags               = ["environment:${var.environment}", "eks", "kubernetes", "pod_restarts"]
  new_group_delay    = 300
  restricted_roles   = []
}

6. Root main.tf (main.tf)

Orchestrate your modules from the root `main.tf`.

# main.tf
module "eks" {
  source = "./modules/eks"

  cluster_name           = var.cluster_name
  kubernetes_version     = var.kubernetes_version
  aws_region             = var.aws_region
  environment            = var.environment
  project                = var.project
  vpc_cidr_block         = var.vpc_cidr_block
  vpc_azs                = var.vpc_azs
  vpc_private_subnets    = var.vpc_private_subnets
  vpc_public_subnets     = var.vpc_public_subnets
  node_group_min_size    = var.node_group_min_size
  node_group_max_size    = var.node_group_max_size
  node_group_desired_size= var.node_group_desired_size
  node_group_instance_type = var.node_group_instance_type
}

module "pagerduty" {
  source = "./pagerduty"

  cluster_name        = var.cluster_name
  pagerduty_user_id   = var.pagerduty_user_id
}

module "datadog_agent" {
  source = "./modules/datadog_agent"

  datadog_api_key = var.datadog_api_key
  datadog_app_key = var.datadog_app_key
}

module "datadog_monitors" {
  source = "./modules/datadog_monitors"

  cluster_name         = var.cluster_name
  environment          = var.environment
  pagerduty_service_name = module.pagerduty.pagerduty_eks_service_name
}

7. Variables (variables.tf)

Define all necessary variables, sensitive ones loaded from environment variables or a secure vault (e.g., AWS Secrets Manager, HashiCorp Vault).

# variables.tf
variable "aws_region" {
  description = "AWS region for the deployment."
  type        = string
  default     = "us-east-1"
}

variable "cluster_name" {
  description = "Name of the EKS cluster."
  type        = string
  default     = "my-eks-cluster"
}

variable "kubernetes_version" {
  description = "Kubernetes version for the EKS cluster."
  type        = string
  default     = "1.28"
}

variable "environment" {
  description = "Deployment environment (e.g., dev, prod)."
  type        = string
  default     = "dev"
}

variable "project" {
  description = "Project name."
  type        = string
  default     = "CloudNative"
}

variable "vpc_cidr_block" {
  description = "CIDR block for the VPC."
  type        = string
  default     = "10.0.0.0/16"
}

variable "vpc_azs" {
  description = "List of availability zones for the VPC."
  type        = list(string)
  default     = ["us-east-1a", "us-east-1b", "us-east-1c"]
}

variable "vpc_private_subnets" {
  description = "List of private subnets CIDR blocks."
  type        = list(string)
  default     = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
}

variable "vpc_public_subnets" {
  description = "List of public subnets CIDR blocks."
  type        = list(string)
  default     = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]
}

variable "node_group_min_size" {
  description = "Minimum size of the EKS managed node group."
  type        = number
  default     = 1
}

variable "node_group_max_size" {
  description = "Maximum size of the EKS managed node group."
  type        = number
  default     = 3
}

variable "node_group_desired_size" {
  description = "Desired size of the EKS managed node group."
  type        = number
  default     = 1
}

variable "node_group_instance_type" {
  description = "Instance type for EKS managed node group."
  type        = string
  default     = "t3.medium"
}

variable "datadog_api_key" {
  description = "Datadog API key."
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application key."
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty API token for Terraform provider."
  type        = string
  sensitive   = true
}

variable "pagerduty_user_id" {
  description = "ID of the PagerDuty user to be assigned to the escalation policy."
  type        = string
  # Example: "P012345" - Find this in the PagerDuty UI under Users -> User Details -> ID
}

Deployment Steps

Follow these steps to deploy your EKS cluster with integrated monitoring and incident automation:

  1. Configure AWS Credentials: Ensure your AWS CLI is configured with credentials that have sufficient permissions to create EKS clusters, VPCs, IAM roles, etc.
  2. Set Environment Variables: Export your Datadog and PagerDuty keys. 
    export TF_VAR_datadog_api_key="your_datadog_api_key"
export TF_VAR_datadog_app_key="your_datadog_app_key"
export TF_VAR_pagerduty_api_token="your_pagerduty_api_token"
export TF_VAR_pagerduty_user_id="your_pagerduty_user_id"
  3. Initialize Terraform: Navigate to your project root and initialize Terraform.
    4. terraform init
  4. Review the Plan: Generate an execution plan to see what resources Terraform will create.
    5. terraform plan -out tfplan
  5. Apply the Configuration: Execute the plan to provision the infrastructure.
    6. terraform apply "tfplan"
  6. Configure kubectl: After successful deployment, configure kubectl to connect to your new EKS cluster.
    7. aws eks update-kubeconfig --name $(terraform output -raw cluster_name) --region $(terraform output -raw aws_region)

Post-Deployment Verification

Confirm that all components are functioning as expected:

  • EKS Cluster Status: 
    kubectl get nodes
kubectl get pods -A

    You should see your EKS nodes and Datadog agent pods running in the datadog namespace.

  • Datadog Integration:

    Log in to your Datadog account. Navigate to Infrastructure -> Hosts. You should see your EKS nodes reporting metrics. Under Logs, you should see container logs. The monitors defined via Terraform should also appear under Monitors -> Manage Monitors.

  • PagerDuty Integration:

    Log in to PagerDuty. Confirm that the service and escalation policy created by Terraform exist. To test the integration, you could manually trigger an incident through Datadog or simulate a high CPU load on an EKS node to trigger the monitor.

Advanced Considerations & Best Practices

  • Security Hardening: Implement Kubernetes Network Policies, Pod Security Standards (or Admission Controllers like Kyverno/OPA Gatekeeper), and regularly audit IAM roles for least privilege. Consider AWS KMS for encrypting EKS secrets.
  • Cost Optimization: Explore using AWS Fargate for serverless worker nodes for specific workloads, or leverage Spot Instances with Karpenter for significant cost savings on managed node groups.
  • Observability Depth: Beyond basic metrics, implement Distributed Tracing for microservices (Datadog APM), establish custom metrics for business-critical application KPIs, and use Datadog Dashboards to visualize your EKS health comprehensively.
  • GitOps Workflow: Integrate Terraform state and Kubernetes manifests into a GitOps pipeline using tools like ArgoCD or Flux CD. This ensures that your desired state is always reflected in your cluster and provides a clear audit trail.
  • State Management: Always use a remote backend for Terraform state (e.g., S3 with DynamoDB locking) to enable team collaboration and prevent state corruption.
  • Automated Remediation: For certain predictable incidents, consider automating remediation steps using AWS Lambda or Kubernetes operators triggered by Datadog webhooks, further reducing MTTR.

Troubleshooting Common Issues

  • Terraform Apply Fails with EKS Issues:

    Ensure your IAM user/role has permissions for EKS, EC2, IAM, and VPC. Check AWS service quotas. EKS cluster creation can take 15-20 minutes; network issues or invalid subnets are common culprits.

  • Datadog Agent Pods Not Running:

    Check logs of Datadog agent pods: kubectl logs -n datadog -l app=datadog --tail 100. Verify your `datadog_api_key` and `datadog_app_key` are correct and accessible by the Helm release. Ensure your node groups have sufficient resources (CPU/memory) and the necessary IAM permissions for the Datadog agent.

  • Datadog Monitors Not Triggering PagerDuty:

    Verify the @pagerduty-${var.pagerduty_service_name} syntax in your Datadog monitor message matches the PagerDuty service name configured in Datadog's integrations. Check Datadog's event stream for alerts being triggered. Ensure the PagerDuty integration key is correctly set up in Datadog (this might be done manually in Datadog UI after Terraform creates the service integration, or ensure the vendor ID is correct).

  • `kubectl` Authorization Issues:

    After `aws eks update-kubeconfig`, if you still have issues, ensure your AWS credentials are valid and the IAM user/role you are using is mapped in the EKS aws-auth ConfigMap (which can also be managed by the `terraform-aws-modules/eks/aws` module).

Conclusion

This guide demonstrates how to build a robust, observable, and resilient AWS EKS environment using Terraform, Datadog, and PagerDuty. By embracing Infrastructure as Code and integrating powerful monitoring and incident response tools, organizations can achieve unparalleled control over their cloud-native infrastructure, reduce operational overhead, and ensure critical applications remain highly available and performant. This setup not only streamlines deployment but also significantly enhances your team's ability to quickly detect, diagnose, and resolve issues, paving the way for more efficient and reliable DevOps practices.

Comments

Post a Comment

Popular posts from this blog

Terraform Configuration for Datadog-PagerDuty Incident Management on AWS EKS

Terraform Configuration for Datadog-PagerDuty Incident Management on AWS EKS In the fast-paced world of cloud-native operations, ensuring high availability and rapid incident response for applications running on AWS EKS is paramount. This comprehensive guide details how to leverage Terraform to seamlessly integrate Datadog for robust monitoring and PagerDuty for efficient incident management, creating a resilient and automated incident response pipeline for your Kubernetes clusters. Architecture Pro-Tip: Modular Design for Scalability When building your incident management infrastructure with Terraform, consider a modular design. Separate your Datadog monitors, PagerDuty services, and AWS EKS resources into distinct Terraform modules. This approach enhances reusability, improves readability, and makes it significantly easier to manage configurations across multiple environments (development, staging, production) or different EKS clusters. Always priori...
Read more

Terraform-Managed AWS EKS Observability and Incident Response with Datadog and PagerDuty

Terraform-Managed AWS EKS Observability and Incident Response with Datadog and PagerDuty In the dynamic world of cloud-native applications, managing AWS Elastic Kubernetes Service (EKS) requires robust observability and a streamlined incident response strategy. This guide delves into integrating Datadog for comprehensive monitoring and PagerDuty for efficient incident management, all orchestrated declaratively using Terraform. We'll explore how to establish a resilient system that not only detects issues promptly but also ensures critical alerts reach the right teams for rapid resolution, maintaining high availability and operational excellence. Architecture Pro-Tip: Always centralize your observability configuration within your Infrastructure as Code (IaC) repository. Treat your Datadog monitors, dashboards, and PagerDuty services as code alongside your EKS clusters. This ensures version control, simplifies rollbacks, an...
Read more

Terraform for Production AWS EKS Observability with Datadog, Prometheus, and PagerDuty Integration

Terraform for Production AWS EKS Observability with Datadog, Prometheus, and PagerDuty Integration In the dynamic world of cloud-native applications, maintaining peak performance and rapid incident response for AWS Elastic Kubernetes Service (EKS) is paramount. Observability, often misunderstood as mere monitoring, extends to understanding the internal state of a system from its external outputs. This guide provides a comprehensive, technical walkthrough on leveraging Terraform to provision and manage a robust observability stack for your production EKS clusters, integrating the power of Datadog for holistic monitoring, Prometheus for deep Kubernetes-native metrics, and PagerDuty for automated incident management. Architecture Pro-Tip: For a truly resilient and scalable production observability strategy on AWS EKS, aim for a layered approach. Use Datadog as your primary pane of glass for aggregated metrics, logs, and traces, leveraging its native integrat...
Read more