Terraform for AWS EKS Datadog Observability and PagerDuty Incident Response Automation

In today's fast-paced cloud-native environments, robust observability and swift incident response are non-negotiable. This comprehensive guide delves into leveraging Terraform to establish a fully automated and observable AWS EKS cluster, integrated seamlessly with Datadog for monitoring and PagerDuty for incident response. By treating your observability and incident response infrastructure as code, you gain consistency, repeatability, and agility, critical for maintaining high availability and performance.

Architecture Pro-Tip:

Always design your observability and incident response tooling with a "shift-left" mindset. Integrate these configurations directly into your Infrastructure as Code (IaC) pipelines. This ensures that every new service or cluster automatically inherits the necessary monitoring and alerting capabilities from day one, reducing manual overhead and preventing blind spots in your production environment.

Why Terraform for EKS, Datadog, and PagerDuty?

Terraform, as the industry-standard Infrastructure as Code (IaC) tool, provides a declarative approach to managing cloud resources and third-party services. Its benefits for this specific use case are profound:

Consistency: Define your EKS clusters, Datadog agents, monitors, dashboards, and PagerDuty services in code, ensuring identical environments across development, staging, and production.
Version Control: Track changes to your infrastructure and observability configurations, enabling rollbacks and collaborative development.
Automation: Automate the provisioning and management of complex integrations, reducing manual errors and speeding up deployments.
Scalability: Easily scale your observability footprint as your EKS clusters grow, applying consistent monitoring policies across new services.
Reduced Toil: Eliminate repetitive manual tasks associated with setting up monitoring and alerting for new services or environments.

Prerequisites

Before diving into the configurations, ensure you have the following:

AWS Account: With programmatic access and sufficient permissions to create EKS clusters, IAM roles, and other AWS resources.
Terraform CLI: Installed and configured (version 1.0+ recommended).
AWS CLI: Configured for authentication with your AWS account.
Kubectl CLI: Installed to interact with your EKS cluster.
Helm CLI: Installed to manage Kubernetes packages (Datadog Agent).
Datadog Account: With an API Key and Application Key.
PagerDuty Account: With an API Token and necessary permissions to create services and escalation policies.

Core Components and Integration Strategy

1. AWS EKS Cluster Setup with Terraform

The foundation is an AWS EKS cluster. Terraform can provision the cluster, node groups, necessary IAM roles, and VPC networking. We'll use the `eks` module or direct resource declarations for this.

2. Integrating Datadog with EKS using Terraform

Datadog provides deep visibility into Kubernetes environments. Integrating it involves deploying the Datadog Agent and defining monitors and dashboards as code.

Datadog Agent Deployment: The Datadog Agent runs as a DaemonSet on your EKS worker nodes, collecting metrics, logs, and traces. Terraform can deploy this agent using the `helm_release` resource, referencing the official Datadog Helm chart.
Datadog Monitors & Dashboards as Code: Use the Datadog Terraform provider to define monitors (alerts) for critical metrics (CPU, memory, network, pod status) and create comprehensive dashboards.

3. Automating PagerDuty Incident Response with Terraform

PagerDuty is the industry leader for incident management. Terraform allows you to provision PagerDuty services, escalation policies, and users.

PagerDuty Service & Escalation Policy Setup: Define PagerDuty services that represent the components of your EKS application. Attach escalation policies to these services to determine who gets notified and when.
Connecting Datadog Alerts to PagerDuty: When a Datadog monitor's alert conditions are met, it needs to trigger an incident in PagerDuty. This is achieved by configuring the Datadog monitor to send notifications to a PagerDuty integration key.

Step-by-Step Terraform Configuration Example

Let's create a simplified Terraform configuration to demonstrate these integrations. For production environments, consider using the official AWS EKS Terraform module for a more robust cluster setup.

Project Structure:

Create a directory for your Terraform project (e.g., `eks-observability`). Inside, create the following files:

`main.tf`: Defines resources.
`variables.tf`: Declares input variables.
`outputs.tf`: Exports important values.
`versions.tf`: Specifies provider requirements.

`versions.tf`

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.20"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
    pagerduty = {
      source  = "PagerDuty/pagerduty"
      version = "~> 1.0"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

`variables.tf`

variable "aws_region" {
  description = "AWS region"
  type        = string
  default     = "us-east-1"
}

variable "cluster_name" {
  description = "Name of the EKS cluster"
  type        = string
  default     = "eks-observability-cluster"
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

variable "pagerduty_team_id" {
  description = "The ID of the PagerDuty team to associate with services/escalation policies"
  type        = string
  # Example: "P0M1L2N" - Replace with your actual Team ID
}

variable "pagerduty_user_id" {
  description = "The ID of the PagerDuty user to add to the escalation policy"
  type        = string
  # Example: "P1A2B3C" - Replace with your actual User ID
}

`main.tf`

This file will contain the EKS cluster, Datadog Agent, Datadog Monitor, and PagerDuty service definitions.

Terraform Code Block (main.tf)

# --- AWS EKS Cluster Setup (Simplified for brevity) ---
# In a real-world scenario, you would use the official AWS EKS module.
# This example provisions minimal resources for demonstration.

resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name = "${var.cluster_name}-vpc"
  }
}

resource "aws_subnet" "private" {
  count             = 2
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.0.${count.index + 1}.0/24"
  availability_zone = "${var.aws_region}${element(["a", "b"], count.index)}"

  tags = {
    Name = "${var.cluster_name}-private-subnet-${count.index}"
  }
}

resource "aws_iam_role" "eks_cluster_role" {
  name = "${var.cluster_name}-eks-cluster-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "eks.amazonaws.com"
        }
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eks_cluster_role.name
}

resource "aws_iam_role_policy_attachment" "eks_vpc_resource_controller_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSVPCResourceController"
  role       = aws_iam_role.eks_cluster_role.name
}


resource "aws_eks_cluster" "main" {
  name     = var.cluster_name
  role_arn = aws_iam_role.eks_cluster_role.arn
  version  = "1.28" # Specify desired Kubernetes version

  vpc_config {
    subnet_ids = aws_subnet.private[*].id
  }

  tags = {
    Environment = "dev"
    Project     = "observability"
  }

  depends_on = [
    aws_iam_role_policy_attachment.eks_cluster_policy,
    aws_iam_role_policy_attachment.eks_vpc_resource_controller_policy,
  ]
}

resource "aws_iam_role" "eks_nodegroup_role" {
  name = "${var.cluster_name}-eks-nodegroup-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_worker_node_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.eks_nodegroup_role.name
}

resource "aws_iam_role_policy_attachment" "eks_cni_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSCNIPolicy"
  role       = aws_iam_role.eks_nodegroup_role.name
}

resource "aws_iam_role_policy_attachment" "ec2_container_registry_readonly_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.eks_nodegroup_role.name
}

resource "aws_eks_node_group" "example" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "${var.cluster_name}-nodegroup"
  node_role_arn   = aws_iam_role.eks_nodegroup_role.arn
  subnet_ids      = aws_subnet.private[*].id
  instance_types  = ["t3.medium"]
  disk_size       = 20
  desired_size    = 2
  min_size        = 1
  max_size        = 3

  depends_on = [
    aws_iam_role_policy_attachment.eks_worker_node_policy,
    aws_iam_role_policy_attachment.eks_cni_policy,
    aws_iam_role_policy_attachment.ec2_container_registry_readonly_policy,
  ]
}

# --- Kubernetes Configuration for Kubeconfig ---
data "aws_eks_cluster" "cluster" {
  name = aws_eks_cluster.main.name
}

data "aws_eks_cluster_auth" "cluster" {
  name = aws_eks_cluster.main.name
}

# --- Datadog Agent Deployment via Helm ---
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Or a dedicated 'datadog' namespace
  version    = "2.33.0"  # Use a recent stable version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }
  set {
    name  = "datadog.site"
    value = "datadoghq.com" # Or your specific Datadog site (e.g., eu.datadoghq.com)
  }
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
  }
  set {
    name  = "targetSystem"
    value = "linux"
  }
  set {
    name  = "tags"
    value = "{cluster_name:${var.cluster_name}}"
  }

  depends_on = [
    aws_eks_node_group.example # Ensure nodes are ready before deploying agent
  ]
}

# --- PagerDuty Configuration ---

resource "pagerduty_user" "oncall_user" {
  # This assumes the user already exists in PagerDuty and you're referencing them.
  # For creating a new user, refer to PagerDuty provider docs.
  email = "your.oncall.email@example.com"
  name  = "Oncall Engineer"
}

resource "pagerduty_escalation_policy" "devops_ep" {
  name      = "${var.cluster_name}-DevOps-EP"
  team      = var.pagerduty_team_id
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 5
    target {
      type = "user"
      id   = pagerduty_user.oncall_user.id # Use ID of existing or created user
    }
  }

  # Add more rules for additional escalation levels if needed
}

resource "pagerduty_service" "eks_application_service" {
  name                     = "${var.cluster_name}-EKS-App-Service"
  auto_resolve_timeout     = 14400 # 4 hours
  acknowledgement_timeout  = 600   # 10 minutes
  escalation_policy        = pagerduty_escalation_policy.devops_ep.id
  incident_urgency_rule {
    type    = "constant"
    urgency = "high"
  }
  alert_creation         = "create_incidents_and_alert_the_designated_responders"
  alert_grouping_timeout = 5
  alert_grouping         = "time"
  # Add optional description, teams, and other settings as needed
}

# --- Datadog Monitor Configuration for PagerDuty Integration ---

resource "datadog_monitor" "high_cpu_alert" {
  name                = "EKS Cluster High CPU Alert - {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.cluster_name}} by {host} > 80"
  message             = "High CPU usage detected on EKS node {{host.name}} ({{value}}%). @pagerduty-{{resources.pagerduty_service_name.name}} Please investigate immediately. @slack-channel-name"
  escalation_message  = "CPU usage remains high after 15 minutes. Investigating now."
  tags                = ["environment:production", "team:devops", "service:eks-app"]
  renotify_interval   = 30
  timeout_h           = 0
  new_group_delay     = 60
  no_data_timeframe   = 20
  notify_no_data      = false
  require_full_window = true
  thresholds {
    critical = 80
    warning  = 70
  }
  # Integrate with PagerDuty using the PagerDuty integration name/key.
  # This typically looks like @pagerduty-integration_key_name or @pagerduty-service_name
  # The PagerDuty integration is usually configured within Datadog UI
  # The name used here (eks-app-service) should match the integration configured in Datadog UI
  # for the pagerduty_service.
}

# --- Datadog Dashboard Configuration ---
resource "datadog_dashboard" "eks_overview" {
  title        = "${var.cluster_name} EKS Cluster Overview"
  description  = "High-level overview of EKS cluster health"
  layout_type  = "ordered"
  is_read_only = false
  widget {
    definition {
      title     = "Cluster CPU Usage"
      type      = "timeseries"
      requests {
        q = "avg:kubernetes.cpu.usage.total{cluster_name:${var.cluster_name}}"
        display_type = "line"
      }
    }
  }
  widget {
    definition {
      title     = "Cluster Memory Usage"
      type      = "timeseries"
      requests {
        q = "avg:kubernetes.memory.usage.total{cluster_name:${var.cluster_name}}"
        display_type = "line"
      }
    }
  }
  widget {
    definition {
      title     = "Nodes Ready"
      type      = "query_value"
      requests {
        q = "sum:kubernetes.nodes.ready{cluster_name:${var.cluster_name}}.as_count()"
      }
    }
  }
  # Add more widgets as needed for a comprehensive dashboard
}

`outputs.tf`

output "eks_cluster_name" {
  description = "The name of the EKS cluster"
  value       = aws_eks_cluster.main.name
}

output "kubeconfig_command" {
  description = "Command to update your kubeconfig"
  value       = "aws eks update-kubeconfig --region ${var.aws_region} --name ${aws_eks_cluster.main.name}"
}

output "pagerduty_service_url" {
  description = "URL to the PagerDuty service"
  value       = "https://your-domain.pagerduty.com/service-directory/${pagerduty_service.eks_application_service.id}"
}

output "datadog_dashboard_url" {
  description = "URL to the Datadog dashboard"
  value       = datadog_dashboard.eks_overview.url
}

Deploying the Solution

With your Terraform configuration files in place, follow these steps to deploy your EKS cluster with integrated Datadog and PagerDuty:

Initialize Terraform: Navigate to your project directory and run terraform init. This downloads the necessary providers.
Plan the Deployment: Execute terraform plan to review the changes Terraform will make. This is a crucial step to ensure everything is configured as expected.
Apply the Configuration: If the plan looks good, apply it using terraform apply. You will be prompted to confirm the actions.
Update Kubeconfig: After the EKS cluster is created, use the output from kubeconfig_command to update your local Kubeconfig: aws eks update-kubeconfig --region <your-region> --name <your-cluster-name>.
Verify Kubernetes Resources: Run kubectl get nodes and kubectl get pods -n default (or your Datadog namespace) to confirm that the EKS nodes are ready and the Datadog Agent pods are running.

Testing and Validation

Once deployed, validate the integration:

Datadog Dashboards: Access your Datadog account. You should see the EKS Cluster Overview dashboard populated with metrics from your cluster.
Datadog Monitors: Verify that the "EKS Cluster High CPU Alert" monitor is active and correctly configured to notify PagerDuty.
PagerDuty Service: Check your PagerDuty account for the newly created service and escalation policy. Ensure the integration key is properly linked (usually visible in the Datadog UI under integrations).
Trigger an Alert (Optional): To test the PagerDuty integration, you could intentionally stress a node or deploy a high-CPU workload. This should trigger the Datadog monitor and, subsequently, a PagerDuty incident.

Advanced Concepts & Best Practices

To further enhance your setup, consider these advanced strategies:

Terraform Modules: Organize your Terraform code into reusable modules for EKS, Datadog configuration, and PagerDuty resources. This promotes modularity and reusability.
Remote State Management: Use AWS S3 and DynamoDB for Terraform remote state to enable collaboration and state locking.
GitOps Workflows: Implement a GitOps approach where all infrastructure and application changes are managed through Git, with automated reconciliation using tools like Argo CD or Flux.
Service-Specific Monitoring: Create dedicated Datadog monitors and PagerDuty services for individual microservices running on EKS, providing more granular alerting and ownership.
Custom Metrics and Logs: Extend Datadog's capabilities by sending custom application metrics and logs from your EKS workloads.
Security Best Practices: Employ IAM roles for service accounts (IRSA) for fine-grained permissions for your Kubernetes pods (e.g., Datadog Agent if needed for specific AWS API access).
Cost Optimization: Regularly review EKS node group sizing and Datadog/PagerDuty usage to optimize costs.

Troubleshooting Common Issues

Terraform Apply Fails: Check AWS IAM permissions. Ensure the user/role running Terraform has permissions to create EKS clusters, IAM roles, and other resources.
Datadog Agent Pods Not Running:
- Check kubectl describe pod <datadog-agent-pod> for error messages.
- Verify your datadog_api_key and datadog_app_key are correct.
- Ensure EKS worker nodes have internet access to reach Datadog endpoints.
- Check resource limits/requests if pods are pending due to insufficient resources.
Datadog Not Receiving Metrics:
- Confirm Datadog Agent pods are running and healthy.
- Verify your Datadog API key and application key.
- Check Datadog Agent logs for connectivity issues or configuration errors.
- Ensure security groups/network ACLs allow traffic from EKS nodes to Datadog's endpoints.
PagerDuty Incidents Not Triggering:
- Ensure the @pagerduty-<integration_name> in your Datadog monitor message correctly matches the integration configured in Datadog UI for your PagerDuty service.
- Check Datadog event logs for the monitor to see if alerts are being sent.
- Verify the PagerDuty service is configured with the correct integration type (e.g., Datadog).

Conclusion

By embracing Terraform for AWS EKS, Datadog observability, and PagerDuty incident response, organizations can achieve an unparalleled level of automation, control, and reliability. This guide provides a robust foundation for building cloud-native systems that are not only performant but also resilient and easily maintainable through the power of Infrastructure as Code. Invest in these integrations to empower your DevOps teams, minimize downtime, and ensure your critical applications are always under vigilant watch.

CloudStack YAML