Terraform for AWS EKS Cluster Provisioning with Datadog Observability and PagerDuty Integration

In the rapidly evolving landscape of cloud-native development, deploying and managing Kubernetes clusters efficiently and reliably is paramount. AWS Elastic Kubernetes Service (EKS) offers a powerful managed service for running Kubernetes, but its provisioning and ongoing management can still present complexities, especially when integrating essential operational tools. This guide delves into a robust, automated approach using Terraform to provision AWS EKS clusters, augmented with comprehensive observability provided by Datadog and a streamlined incident response workflow via PagerDuty. By leveraging Infrastructure as Code (IaC) principles, organizations can achieve repeatable, scalable, and auditable deployments, ensuring their critical applications operate smoothly while maintaining high levels of visibility and rapid incident resolution capabilities. This integrated strategy empowers DevOps teams to build resilient, high-performing cloud infrastructures with confidence and efficiency.

Architecture Pro-Tip: Immutable Infrastructure & Modular Design

Always strive for immutable infrastructure when provisioning EKS clusters. This means creating new infrastructure for updates rather than modifying existing resources. Combine this with a modular Terraform design, separating your VPC, EKS cluster, node groups, and observability tooling into distinct modules. This approach enhances reusability, reduces blast radius during changes, and simplifies maintenance, making your infrastructure highly resilient and easier to manage at scale.

Mastering EKS Provisioning with Terraform

Terraform, as a leading Infrastructure as Code (IaC) tool, provides an declarative syntax to define and provision cloud infrastructure. For AWS EKS, this means you can define your entire cluster—from the underlying Virtual Private Cloud (VPC) and subnets to the IAM roles and the EKS control plane itself—as code. This approach eliminates manual configuration errors, ensures consistency across environments, and enables version control for your infrastructure. Starting with a solid foundation is crucial. This typically involves setting up your AWS provider configuration, defining your network topology with a VPC, public and private subnets, and the necessary security groups and NAT gateways. Furthermore, appropriate IAM roles for the EKS control plane and node groups are indispensable for secure operations. The declarative nature of Terraform allows you to articulate exactly what you want your infrastructure to look like, and Terraform handles the intricate steps of making that a reality.

# main.tf - AWS Provider Configuration, VPC, and IAM for EKS

provider "aws" {
region = "us-east-1"
}

data "aws_availability_zones" "available" {
state = "available"
}

resource "aws_vpc" "eks_vpc" {
cidr_block = "10.0.0.0/16"
enable_dns_hostnames = true
enable_dns_support = true
tags = {
Name = "eks-vpc"
}
}

resource "aws_subnet" "public" {
count = 2
vpc_id = aws_vpc.eks_vpc.id
cidr_block = cidrsubnet(aws_vpc.eks_vpc.cidr_block, 8, count.index)
availability_zone = data.aws_availability_zones.available.names[count.index]
map_public_ip_on_launch = true
tags = {
Name = "eks-public-subnet-${count.index}"
"kubernetes.io/cluster/my-eks-cluster" = "shared"
"kubernetes.io/role/elb" = "1"
}
}

resource "aws_subnet" "private" {
count = 2
vpc_id = aws_vpc.eks_vpc.id
cidr_block = cidrsubnet(aws_vpc.eks_vpc.cidr_block, 8, count.index + 2)
availability_zone = data.aws_availability_zones.available.names[count.index]
tags = {
Name = "eks-private-subnet-${count.index}"
"kubernetes.io/cluster/my-eks-cluster" = "shared"
"kubernetes.io/role/internal-elb" = "1"
}
}

resource "aws_iam_role" "eks_cluster_role" {
name = "eks-cluster-role"
assume_role_policy = jsonencode({
Statement = [{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
Service = "eks.amazonaws.com"
}
}]
Version = "2012-10-17"
})
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
role = aws_iam_role.eks_cluster_role.name
}

resource "aws_iam_role_policy_attachment" "eks_service_policy" {
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
role = aws_iam_role.eks_cluster_role.name
}

Building Your EKS Cluster

With the foundational networking and IAM roles in place, the next step is to define the EKS cluster itself. The aws_eks_cluster resource in Terraform allows you to specify critical parameters such as the Kubernetes version, the VPC ID, and the subnet IDs where the cluster will operate. Equally important is the creation of node groups, which are the EC2 instances that will run your containerized applications. You can define managed node groups, which are easier to maintain, or self-managed node groups for more granular control. For managed node groups, you specify the instance type, desired capacity, and associate them with specific subnets and IAM roles. It's vital to ensure that the IAM role for the node group has the necessary permissions for EKS worker nodes to communicate with the control plane and other AWS services. This declarative approach simplifies updates and rollbacks, making cluster lifecycle management significantly more robust compared to manual methods.

# main.tf - EKS Cluster and Managed Node Group

resource "aws_eks_cluster" "my_eks_cluster" {
name = "my-eks-cluster"
role_arn = aws_iam_role.eks_cluster_role.arn
vpc_config {
subnet_ids = concat(aws_subnet.private[*].id, aws_subnet.public[*].id)
security_group_ids = [] # Optionally add specific security groups
endpoint_private_access = false
endpoint_public_access = true
}
version = "1.28"

tags = {
Name = "my-eks-cluster"
}
}

resource "aws_iam_role" "eks_nodegroup_role" {
name = "eks-nodegroup-role"
assume_role_policy = jsonencode({
Statement = [{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
Service = "ec2.amazonaws.com"
}
}]
Version = "2012-10-17"
})
}

resource "aws_iam_role_policy_attachment" "eks_nodegroup_policy_cni" {
policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
role = aws_iam_role.eks_nodegroup_role.name
}

resource "aws_iam_role_policy_attachment" "eks_nodegroup_policy_worker" {
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
role = aws_iam_role.eks_nodegroup_role.name
}

resource "aws_iam_role_policy_attachment" "eks_nodegroup_policy_registry" {
policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
role = aws_iam_role.eks_nodegroup_role.name
}

resource "aws_eks_node_group" "my_eks_nodegroup" {
cluster_name = aws_eks_cluster.my_eks_cluster.name
node_group_name = "my-eks-nodegroup"
node_role_arn = aws_iam_role.eks_nodegroup_role.arn
subnet_ids = aws_subnet.private[*].id
instance_types = ["t3.medium"]

scaling_config {
desired_size = 2
max_size = 3
min_size = 1
}

remote_access {
ec2_ssh_key = "my-ssh-keypair" # Replace with your SSH key pair name
}

tags = {
Name = "my-eks-nodegroup"
}
}

Integrating Datadog for Comprehensive Observability

Once your EKS cluster is operational, the next critical step is to gain deep visibility into its performance and health. Datadog offers a unified observability platform that collects metrics, logs, and traces from your entire infrastructure, including Kubernetes, applications, and AWS services. Integrating Datadog with EKS typically involves deploying the Datadog Agent as a DaemonSet across your cluster. This agent collects host-level metrics, events, and logs, as well as Kubernetes-specific data like pod metrics, service discovery, and resource utilization. Additionally, Datadog can ingest AWS service metrics through its native integration, providing a holistic view. Deploying the Datadog Agent via Terraform can be achieved using the Kubernetes provider and applying a manifest, or more commonly, leveraging the Helm provider to deploy the Datadog Helm chart. This ensures that every new EKS node automatically starts reporting data to Datadog, providing immediate insights.

# main.tf - Deploying Datadog Agent via Helm Release

resource "helm_release" "datadog_agent" {
name = "datadog"
repository = "https://helm.datadoghq.com"
chart = "datadog"
namespace = "datadog"
create_namespace = true

set {
name = "datadog.apiKey"
value = var.datadog_api_key
}
set {
name = "datadog.appKey"
value = var.datadog_app_key
}
set {
name = "clusterAgent.enabled"
value = "true"
}
set {
name = "kubeStateMetricsCore.enabled"
value = "true"
}
set {
name = "containerRuntime.enabled"
value = "true"
}
set {
name = "logAgent.enabled"
value = "true"
}
set {
name = "logAgent.containerCollectAll"
value = "true"
}
set {
name = "processAgent.enabled"
value = "true"
}
set {
name = "apm.enabled"
value = "true"
}
set {
name = "tags[0]"
value = "env:production"
}
set {
name = "tags[1]"
value = "cluster:${aws_eks_cluster.my_eks_cluster.name}"
}

# Ensure Kubernetes provider is configured to talk to the new EKS cluster
# This requires a 'kubernetes' provider block configured with EKS cluster details
depends_on = [
aws_eks_cluster.my_eks_cluster,
aws_eks_node_group.my_eks_nodegroup
]
}

# Variables for Datadog API/App keys (e.g., in variables.tf or environment variables)
variable "datadog_api_key" {
description = "Your Datadog API Key"
type = string
sensitive = true
}

variable "datadog_app_key" {
description = "Your Datadog Application Key"
type = string
sensitive = true
}

Real-time Incident Response with PagerDuty

Observability alone is not enough; a robust incident response mechanism is equally vital. PagerDuty is an industry-leading incident management platform that orchestrates responses to critical operational incidents. By integrating PagerDuty with Datadog, you can transform detected issues and anomalies into actionable alerts that escalate to the right teams at the right time. The integration allows Datadog monitors to trigger PagerDuty incidents, ensuring that engineers are notified promptly about critical problems in your EKS cluster or applications. Setting up this integration via Terraform involves defining the PagerDuty service and then configuring Datadog monitors to send alerts to this service. This automated setup ensures that your incident response workflows are consistent and reliable, eliminating manual steps and reducing mean time to acknowledge (MTTA) and mean time to resolution (MTTR).

# main.tf - Datadog-PagerDuty Integration via Terraform

provider "datadog" {
api_key = var.datadog_api_key
app_key = var.datadog_app_key
}

resource "datadog_integration_pagerduty" "pagerduty_integration" {
services = [{
service_name = "MyEKSClusterAlerts"
service_key = var.pagerduty_service_integration_key # PagerDuty integration key for a service
}]
}

# Example Datadog monitor that uses the PagerDuty integration
resource "datadog_monitor" "eks_node_cpu_utilization" {
name = "[EKS] High Node CPU Utilization on my-eks-cluster"
type = "query alert"
query = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:my-eks-cluster} by {host} > 80"
message = <<-EOT
@pagerduty-MyEKSClusterAlerts
High CPU utilization detected on EKS node {{host.name}} in cluster {{cluster_name.name}}.
Current utilization: {{value}}%. Please investigate immediately.
EOT
tags = ["environment:production", "team:devops", "service:eks"]
priority = 1
notify_no_data = false
new_group_delay = 60
no_data_timeframe = 20
renotify_interval = 0
timeout_h = 0
escalation_message = "CPU utilization remains high. Escalating to on-call."
}

variable "pagerduty_service_integration_key" {
description = "PagerDuty service integration key (from Datadog integration within PagerDuty)"
type = string
sensitive = true
}

Connecting the Dots: A Unified Workflow

The true power of this setup lies in its seamless integration. From the initial provisioning of the EKS cluster to continuous monitoring and automated incident response, every component works in concert. When you apply your Terraform configuration, it not only sets up your EKS infrastructure but also deploys the Datadog Agent, ensuring immediate data collection. Subsequently, Terraform can configure Datadog monitors with predefined thresholds and integrate them directly with PagerDuty services. This means that if a critical metric, like high CPU usage on an EKS node, crosses its defined threshold, Datadog automatically triggers an alert in PagerDuty. PagerDuty then takes over, initiating the on-call rotation, sending notifications (SMS, phone calls, email), and creating an incident that can be tracked and managed. This unified workflow drastically reduces the time from problem detection to resolution, minimizing potential downtime and operational impact.

# Simplified Workflow Overview

1. Terraform Apply: Provisions AWS EKS Cluster, VPC, IAM roles, and deploys Datadog Agent via Helm.

2. Datadog Agent: Starts collecting metrics, logs, and traces from EKS nodes, pods, and applications.

3. Terraform Apply (Datadog Monitors): Configures Datadog monitors, linking them to specific metrics and PagerDuty integration.

4. Incident Trigger: Datadog monitor detects an anomaly (e.g., node CPU > 80%).

5. PagerDuty Notification: Datadog sends an alert to PagerDuty via the configured integration.

6. On-Call Engagement: PagerDuty triggers an incident, notifies the appropriate on-call engineer according to schedules and escalation policies.

7. Resolution: Engineer investigates using Datadog dashboards and logs, resolves the issue, and marks the incident resolved in PagerDuty.

Best Practices and Conclusion

Implementing a robust EKS provisioning, observability, and incident response pipeline with Terraform, Datadog, and PagerDuty involves more than just writing code; it requires adhering to best practices to ensure long-term maintainability and reliability. Always use a modular Terraform structure, separating concerns into logical components like VPC, EKS cluster, node groups, and applications. Store sensitive information like API keys using a secure secrets manager like AWS Secrets Manager or HashiCorp Vault, referencing them in Terraform using data sources instead of hardcoding. Implement continuous integration and continuous deployment (CI/CD) pipelines for your Terraform code to automate deployments and enforce consistency. Regularly review and update your IAM policies to follow the principle of least privilege, ensuring components only have the permissions they absolutely need. Furthermore, proactively define Datadog monitors for key performance indicators (KPIs) and service level objectives (SLOs) relevant to your applications and infrastructure. Test your PagerDuty incident response flows periodically to ensure they function as expected and that your on-call teams are well-prepared. This comprehensive approach ensures that your cloud-native applications running on EKS are not only provisioned efficiently but also monitored thoroughly and supported by an effective incident management system.

By embracing Infrastructure as Code with Terraform for AWS EKS, combined with the powerful observability capabilities of Datadog and the agile incident management of PagerDuty, organizations can achieve a highly resilient, observable, and rapidly responsive cloud infrastructure. This integrated strategy is fundamental for modern DevOps practices, enabling teams to deliver higher quality services with greater confidence and operational excellence.

Key Best Practices:

• Modular Terraform: Break down your infrastructure into reusable modules.
• Secrets Management: Use AWS Secrets Manager or Vault for sensitive data.
• CI/CD Pipelines: Automate Terraform deployments for consistency and auditability.
• Least Privilege IAM: Grant only the necessary permissions to roles and users.
• Proactive Monitoring: Define Datadog monitors for key metrics and SLOs.
• Regular Incident Drills: Test PagerDuty escalation policies and on-call readiness.
• Version Control Everything: Treat your infrastructure code and configuration as critical assets under version control.