Automating AWS EKS Cluster Deployment with Terraform, Datadog, and PagerDuty

Architecture Pro-Tip: Modular Infrastructure as Code (IaC)

For large-scale, enterprise-grade deployments, always structure your Terraform configurations into logical modules. Separate concerns such as VPC, EKS cluster, node groups, and application deployments into distinct modules. This approach enhances reusability, simplifies maintenance, and promotes team collaboration while maintaining stringent security controls and cost observability. Leverage remote state management (e.g., S3 backend with DynamoDB locking) for robust collaboration and state integrity.

In today's fast-paced cloud-native landscape, the ability to rapidly deploy, manage, and monitor containerized applications is paramount. AWS Elastic Kubernetes Service (EKS) provides a robust platform for running Kubernetes on AWS, offering high availability and scalability. However, manually deploying and managing EKS, along with integrating observability and incident response, can be complex and error-prone. This guide outlines a comprehensive strategy for automating EKS cluster deployment using Terraform for Infrastructure as Code (IaC), Datadog for end-to-end monitoring, and PagerDuty for streamlined incident management, ensuring operational excellence from day one.

The Synergy of Automation: Terraform, Datadog, and PagerDuty

By combining these industry-leading tools, organizations can achieve a highly automated, observable, and resilient EKS environment:

Terraform: Automates the provisioning and management of the entire EKS infrastructure, including VPCs, subnets, security groups, IAM roles, the EKS control plane, and node groups. It ensures consistency, repeatability, and version control for your cloud infrastructure.
Datadog: Provides comprehensive observability across your EKS clusters and applications. It collects metrics, logs, and traces from Kubernetes components, containers, and applications, offering real-time dashboards, powerful analytics, and intelligent alerting capabilities.
PagerDuty: Acts as the central hub for incident management. By integrating with Datadog, critical alerts are transformed into actionable incidents, ensuring the right on-call team members are notified promptly via their preferred communication channels, facilitating rapid resolution.

Prerequisites

Before you begin, ensure you have the following:

An AWS account with appropriate programmatic access (IAM user or role).
Terraform CLI installed (version 1.0+ recommended).
kubectl CLI installed and configured.
AWS CLI installed and configured.
Datadog account with API and Application keys.
PagerDuty account with an integration key for Datadog.

Step-by-Step Implementation Guide

1. Provisioning EKS with Terraform

The core of our automation begins with Terraform. We will define the necessary AWS resources, including a Virtual Private Cloud (VPC), subnets (public and private for high availability), IAM roles for EKS, and the EKS cluster itself, along with managed node groups.

Create a directory for your Terraform configuration (e.g., eks-automation/) and add the following files:

main.tf: Defines the primary resources.
variables.tf: Declares input variables.
outputs.tf: Defines output values.
versions.tf: Specifies provider versions.

For simplicity, the example below consolidates key resources. In a production scenario, leverage the official Terraform AWS EKS module for a more robust and feature-rich setup.

# main.tf - Simplified EKS Cluster Deployment

provider "aws" {
  region = "us-east-1" # Specify your desired region
}

# VPC for EKS
resource "aws_vpc" "eks_vpc" {
  cidr_block = "10.0.0.0/16"
  enable_dns_hostnames = true
  tags = {
    Name = "eks-automated-vpc"
  }
}

# Public Subnets (for load balancers, etc.)
resource "aws_subnet" "public_subnets" {
  count = 2
  vpc_id = "${aws_vpc.eks_vpc.id}"
  cidr_block = "10.0.${count.index + 1}.0/24"
  availability_zone = "${data.aws_availability_zones.available.names[count.index]}"
  map_public_ip_on_launch = true
  tags = {
    Name = "eks-public-subnet-${count.index}"
  }
}

# Private Subnets (for EKS nodes)
resource "aws_subnet" "private_subnets" {
  count = 2
  vpc_id = "${aws_vpc.eks_vpc.id}"
  cidr_block = "10.0.${count.index + 10}.0/24"
  availability_zone = "${data.aws_availability_zones.available.names[count.index]}"
  tags = {
    Name = "eks-private-subnet-${count.index}"
  }
}

# EKS Cluster IAM Role
resource "aws_iam_role" "eks_cluster_role" {
  name = "eks-cluster-role"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "eks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role = "${aws_iam_role.eks_cluster_role.name}"
}

# EKS Cluster
resource "aws_eks_cluster" "automated_eks" {
  name = "automated-eks-cluster"
  role_arn = "${aws_iam_role.eks_cluster_role.arn}"
  vpc_config {
    subnet_ids = "${concat(aws_subnet.public_subnets.*.id, aws_subnet.private_subnets.*.id)}"
  }
  depends_on = [
    "aws_iam_role_policy_attachment.eks_cluster_policy"
  ]
}

# EKS Node Group IAM Role
resource "aws_iam_role" "eks_node_group_role" {
  name = "eks-node-group-role"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "eks_node_group_worker_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role = "${aws_iam_role.eks_node_group_role.name}"
}

resource "aws_iam_role_policy_attachment" "eks_node_group_cni_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role = "${aws_iam_role.eks_node_group_role.name}"
}

resource "aws_iam_role_policy_attachment" "eks_node_group_registry_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role = "${aws_iam_role.eks_node_group_role.name}"
}

# EKS Managed Node Group
resource "aws_eks_node_group" "automated_node_group" {
  cluster_name = "${aws_eks_cluster.automated_eks.name}"
  node_group_name = "default-node-group"
  node_role_arn = "${aws_iam_role.eks_node_group_role.arn}"
  subnet_ids = "${aws_subnet.private_subnets.*.id}"
  instance_types = ["t3.medium"] # Choose appropriate instance types
  scaling_config {
    desired_size = 2
    max_size = 3
    min_size = 1
  }
  depends_on = [
    "aws_iam_role_policy_attachment.eks_node_group_worker_policy",
    "aws_iam_role_policy_attachment.eks_node_group_cni_policy",
    "aws_iam_role_policy_attachment.eks_node_group_registry_policy",
  ]
}

data "aws_availability_zones" "available" {
  state = "available"
}

# Output the Kubeconfig setup command
output "configure_kubectl" {
  value = "aws eks update-kubeconfig --region ${var.aws_region} --name ${aws_eks_cluster.automated_eks.name}"
  description = "Command to configure kubectl for the EKS cluster."
}
    

Deployment Steps for Terraform:

Initialize Terraform: terraform init
Review the execution plan: terraform plan
Apply the configuration: terraform apply -auto-approve
After successful deployment, configure kubectl using the output command: aws eks update-kubeconfig --region us-east-1 --name automated-eks-cluster (adjust region if needed).

2. Integrating Datadog for EKS Observability

Once your EKS cluster is operational, the next crucial step is to deploy the Datadog Agent to collect metrics, logs, and traces. The Datadog Agent runs as a DaemonSet on your Kubernetes cluster, ensuring an agent instance runs on every node.

There are several ways to deploy the Datadog Agent, including Helm charts or Kubernetes manifests. We'll use Helm for its ease of management.

First, ensure you have Helm installed and add the Datadog Helm repository:

helm repo add datadog https://helm.datadoghq.com
helm repo update
    

Next, install the Datadog Agent, replacing <YOUR_DATADOG_API_KEY> and <YOUR_DATADOG_APP_KEY> with your actual keys:

helm install datadog-agent datadog/datadog \
  --set datadog.apiKey=<YOUR_DATADOG_API_KEY> \
  --set datadog.appKey=<YOUR_DATADOG_APP_KEY> \
  --set clusterAgent.enabled=true \
  --set clusterAgent.metricsProvider.enabled=true \
  --set targetSystem=linux \
  --set agents.sendHostInfo=true \
  --set datadog.site="datadoghq.com" # or eu.datadoghq.com, etc. \
  --set containerExclude=["kube_namespace:kube-system"] # Example exclusion
    

Verify the Datadog Agent pods are running:

kubectl get pods -n default -l app=datadog
    

Once deployed, Datadog will automatically start collecting data from your EKS cluster. You can then navigate to your Datadog dashboard to explore pre-built EKS dashboards, create custom monitors, and analyze logs and traces.

3. Integrating PagerDuty for Incident Management

To ensure critical issues detected by Datadog are acted upon immediately, integrate PagerDuty. This involves setting up a Datadog integration service in PagerDuty and configuring Datadog monitors to trigger PagerDuty incidents.

3.1. Configure PagerDuty Integration

Log in to your PagerDuty account.
Go to Services > Service Directory and create a new service or select an existing one.
Under the Integrations tab of your service, click + Add an integration.
Search for Datadog and click Add.
Copy the generated Integration Key. You'll need this in Datadog.

3.2. Configure Datadog Integration

Log in to your Datadog account.
Go to Integrations > Integrations.
Search for PagerDuty and click on its tile.
Click Install Integration (if not already installed).
Under the Configuration tab, add a new account and paste your PagerDuty Integration Key from step 3.1. Give it a meaningful name (e.g., "EKS Critical Alerts").
Click Install Integration to save.

3.3. Create Datadog Monitors to Alert PagerDuty

Now, you can configure any Datadog monitor to send alerts to PagerDuty. For example, let's create a monitor for high CPU utilization on EKS nodes:

In Datadog, go to Monitors > New Monitor.
Select Metric as the monitor type.
Configure your query, e.g., avg(system.cpu.idle) by {host} where system.cpu.idle drops below a certain threshold (meaning high CPU usage).
Set alert conditions (e.g., alert if avg(system.cpu.idle) is below 10 for 5 minutes).
In the Notify your team section, add @pagerduty-<YOUR_INTEGRATION_NAME> (e.g., @pagerduty-EKS Critical Alerts) to the notification message. This will trigger an incident in PagerDuty when the monitor alerts.
Customize your message and set other preferences, then click Save.

Troubleshooting and Best Practices

Troubleshooting Common Issues:

Terraform Apply Errors: Check IAM permissions, VPC CIDR block conflicts, or resource limits in your AWS account. Review the Terraform output carefully for specific error messages.
EKS Cluster Not Ready: EKS cluster creation can take 10-15 minutes. Ensure your node groups are configured correctly and have appropriate IAM roles and network connectivity. Use kubectl get nodes to check node status.
Datadog Agent Pods Failing: Check pod logs with kubectl logs <datadog-agent-pod> -n default. Verify your Datadog API and APP keys are correct and that the agent has network access to Datadog endpoints.
PagerDuty Alerts Not Triggering: Double-check the Datadog-PagerDuty integration key. Ensure the Datadog monitor's notification message correctly references the PagerDuty integration (e.g., @pagerduty-YourIntegrationName).

Best Practices:

State Management: Always use an S3 backend with DynamoDB locking for Terraform state to prevent concurrent modifications and ensure state integrity in team environments.
IAM Least Privilege: Implement the principle of least privilege for all IAM roles associated with EKS and its components.
Cost Optimization: Monitor EKS node group sizing. Consider using Karpenter or Cluster Autoscaler for dynamic scaling and Spot Instances for non-critical workloads.
Security Hardening: Integrate security scanning tools, manage Kubernetes network policies, and regularly update EKS versions and node AMI images.
Observability from Day 1: Deploy monitoring agents (like Datadog) immediately after cluster provisioning to gain instant visibility into performance and health.

Conclusion

Automating AWS EKS cluster deployment with Terraform, Datadog, and PagerDuty establishes a robust foundation for modern cloud-native applications. This integrated approach ensures that your infrastructure is provisioned consistently, your applications are continuously monitored, and critical incidents are managed effectively, leading to improved reliability, faster issue resolution, and a more efficient DevOps workflow. By investing in this automation, organizations can focus more on innovation and less on operational overhead, driving significant business value.

CloudStack YAML