Terraform AWS EKS Cluster with Datadog Observability and PagerDuty Alerting

Architecture Pro-Tip: Always design your EKS infrastructure with modularity, security, and scalability in mind. Utilize Terraform modules for reusable components, implement least-privilege IAM roles, and configure autoscaling from day one. Consistent naming conventions and environment separation are crucial for maintainability in production systems.

Building a Production-Ready AWS EKS Cluster with Terraform, Datadog, and PagerDuty

In today's cloud-native landscape, Kubernetes has become the de-facto standard for container orchestration. AWS EKS provides a robust, managed Kubernetes service, but deploying and maintaining a production-grade cluster requires careful orchestration of infrastructure, observability, and incident management. This guide provides a comprehensive, technical walkthrough on how to provision an AWS EKS cluster using Terraform, integrate it with Datadog for unparalleled observability, and configure PagerDuty for critical incident alerting.

Why Terraform, Datadog, and PagerDuty for EKS?

Terraform (Infrastructure as Code): Automates the provisioning and management of your entire AWS infrastructure, including VPCs, EKS clusters, IAM roles, and associated resources, ensuring consistency, reproducibility, and version control.
AWS EKS (Managed Kubernetes): Simplifies the deployment, management, and scaling of Kubernetes applications in the AWS cloud, offloading operational burdens of the control plane.
Datadog (Unified Observability): Provides end-to-end monitoring for your EKS cluster, applications, and AWS resources. Collects metrics, logs, traces, and events in a single platform, offering deep insights into performance and health.
PagerDuty (Reliable Incident Response): Integrates seamlessly with Datadog to transform critical alerts into actionable incidents, ensuring the right team members are notified immediately and efficiently, reducing mean time to resolution (MTTR).

Prerequisites

Before you begin, ensure you have the following tools and accounts configured:

An AWS Account with administrative access.
Terraform CLI (v1.0.0+) installed.
AWS CLI configured with appropriate credentials.
kubectl CLI installed and configured.
A Datadog account with API and Application keys.
A PagerDuty account with an Integration Key for a new or existing service.
Helm CLI (v3.0.0+) installed (for Datadog Agent deployment).

Terraform for AWS EKS Cluster Provisioning

We'll use Terraform to provision the foundational AWS infrastructure, including a dedicated Virtual Private Cloud (VPC), the EKS control plane, and associated node groups. For a production environment, modularity is key. We recommend using existing Terraform modules where appropriate to reduce boilerplate and leverage community best practices.

1. AWS Provider Configuration

Define your AWS provider and specify the region.

2. VPC and Networking Setup

A dedicated VPC with public and private subnets is crucial for EKS. Public subnets are for load balancers and NAT gateways, while private subnets host your EKS worker nodes for enhanced security.

3. EKS Cluster Control Plane

The aws_eks_cluster resource will provision the Kubernetes control plane. It requires IAM roles for the EKS service and network configuration.

4. EKS Node Groups

You can use either Managed Node Groups or Fargate profiles. Managed Node Groups provide auto-scaling and auto-updating EC2 instances. Fargate offers serverless compute for pods, reducing operational overhead.

Ready-to-Use Terraform Configuration

Below is a simplified, yet functional, Terraform configuration demonstrating the setup for an AWS EKS cluster. For production use, consider breaking this into separate modules (e.g., vpc, eks, iam) and managing state with an S3 backend.

        // main.tf

        terraform {
            required_providers {
                aws = {
                    source  = "hashicorp/aws"
                    version = "~> 5.0"
                }
                kubernetes = {
                    source  = "hashicorp/kubernetes"
                    version = "~> 2.23"
                }
                helm = {
                    source  = "hashicorp/helm"
                    version = "~> 2.11"
                }
                datadog = {
                    source  = "DataDog/datadog"
                    version = "~> 3.0"
                }
            }
        }

        provider "aws" {
            region = "us-east-1"
        }

        data "aws_availability_zones" "available" {}

        // VPC Module
        module "vpc" {
            source  = "terraform-aws-modules/vpc/aws"
            version = "~> 5.0"

            name = "eks-vpc"
            cidr = "10.0.0.0/16"

            azs             = data.aws_availability_zones.available.names
            private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
            public_subnets  = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]

            enable_nat_gateway = true
            single_nat_gateway = true

            tags = {
                Environment = "production"
                ManagedBy   = "Terraform"
            }
        }

        // EKS Cluster Module
        module "eks" {
            source  = "terraform-aws-modules/eks/aws"
            version = "~> 19.0"

            cluster_name    = "my-production-eks"
            cluster_version = "1.28"

            vpc_id                   = module.vpc.vpc_id
            subnet_ids               = module.vpc.private_subnets
            control_plane_subnet_ids = module.vpc.public_subnets

            enable_irsa = true

            eks_managed_node_groups = {
                general = {
                    min_size     = 2
                    max_size     = 5
                    desired_size = 2
                    instance_types = ["t3.medium"]
                    labels = {
                        role = "general"
                    }
                    tags = {
                        Environment = "production"
                    }
                }
            }

            tags = {
                Environment = "production"
                ManagedBy   = "Terraform"
            }
        }

        // Kubeconfig generation for kubectl access
        resource "local_file" "kubeconfig" {
            content  = module.eks.kubeconfig
            filename = "${path.module}/kubeconfig_${module.eks.cluster_name}"
        }

        provider "kubernetes" {
            host                   = module.eks.cluster_endpoint
            cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
            exec {
                api_version = "client.authentication.k8s.io/v1beta1"
                command     = "aws"
                args        = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
            }
        }

        provider "helm" {
            kubernetes {
                host                   = module.eks.cluster_endpoint
                cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
                exec {
                    api_version = "client.authentication.k8s.io/v1beta1"
                    command     = "aws"
                    args        = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
                }
            }
        }

        provider "datadog" {
            api_key = var.datadog_api_key
            app_key = var.datadog_app_key
        }

        // Deploy Datadog Agent via Helm
        resource "helm_release" "datadog_agent" {
            name       = "datadog"
            repository = "https://helm.datadoghq.com"
            chart      = "datadog"
            namespace  = "datadog"
            create_namespace = true

            set {
                name  = "datadog.apiKey"
                value = var.datadog_api_key
            }
            set {
                name  = "datadog.appKey"
                value = var.datadog_app_key
            }
            set {
                name  = "clusterAgent.enabled"
                value = "true"
            }
            set {
                name  = "clusterAgent.metricsProvider.enabled"
                value = "true"
            }
            set {
                name  = "targetSystem"
                value = "linux"
            }
            set {
                name  = "kubeStateMetricsExternal.enabled"
                value = "true"
            }
            set {
                name  = "processAgent.enabled"
                value = "true"
            }
            set {
                name  = "logs.enabled"
                value = "true"
            }
            set {
                name  = "logs.containerCollectAll"
                value = "true"
            }
            set {
                name  = "apm.enabled"
                value = "true"
            }
            set {
                name  = "confd.kube_state_metrics_core.enabled"
                value = "true"
            }
            set {
                name  = "clusterAgent.rbac.create"
                value = "true"
            }
            set {
                name  = "datadog.site"
                value = "datadoghq.com" // Or datadoghq.eu, etc.
            }
            set {
                name = "rbac.create"
                value = "true"
            }
        }

        // Datadog Monitor for EKS Node CPU Utilization (example)
        resource "datadog_monitor" "eks_node_cpu" {
            name = "EKS Node CPU Utilization High on my-production-eks"
            type = "metric alert"
            query = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:my-production-eks} by {host} > 80"

            message = <
            @webhook-pagerduty @slack-devops
            EKS Node {{host.name}} CPU utilization is at {{value}}%. This indicates potential capacity issues.
            Please investigate the pods running on this node.
            EOT

            tags = ["environment:production", "service:eks", "alert-type:cpu"]
            priority = 3
            renotify_interval = 60

            notify_no_data = false
            new_host_delay = 300
            require_full_window = true
            include_tags = true
            escalation_message = "CPU utilization remains high. Escalating to on-call."
        }

        // variables.tf
        variable "datadog_api_key" {
            description = "Datadog API Key"
            type        = string
            sensitive   = true
        }

        variable "datadog_app_key" {
            description = "Datadog Application Key"
            type        = string
            sensitive   = true
        }
    

Integrating Datadog Observability

Once your EKS cluster is operational, the next critical step is to deploy the Datadog Agent to collect comprehensive observability data. The Datadog Agent runs as a DaemonSet on your EKS nodes and a Deployment for the Cluster Agent, collecting metrics, logs, traces, and events from Kubernetes, containers, and applications.

The Terraform configuration above includes a helm_release resource to deploy the Datadog Agent. This approach ensures that the Datadog Agent itself is part of your Infrastructure as Code, making its deployment consistent and auditable.

Metrics: The Agent collects host-level metrics (CPU, memory, disk), Kubernetes metrics (pod, deployment, service status), and custom application metrics via integrations or DogStatsD.
Logs: Container logs are automatically collected, parsed, and tagged, providing centralized log management.
APM Traces: Datadog APM enables distributed tracing for your applications, helping you pinpoint performance bottlenecks across services.
Network Performance Monitoring: Visualize network traffic and dependencies between services and pods.

After applying the Terraform, verify the Datadog Agent pods are running correctly in your EKS cluster:

        kubectl get pods -n datadog
        kubectl get svc -n datadog
    

You should then navigate to your Datadog dashboard to see your EKS cluster, nodes, and running containers appearing in the Infrastructure List and various dashboards.

Setting Up PagerDuty Alerting

Observability is incomplete without timely and reliable alerting. PagerDuty is an industry leader in incident management, and integrating it with Datadog allows for robust alert escalation. The integration typically involves creating a Datadog-PagerDuty integration in Datadog, which generates a webhook URL or service key.

The Terraform configuration includes an example of a datadog_monitor resource. This resource defines an alert condition within Datadog that, when met, can trigger notifications to various endpoints, including PagerDuty. The message field of the monitor is crucial for directing alerts:

@webhook-pagerduty: This special mention in Datadog directs the alert to your pre-configured PagerDuty integration. Ensure you have set up a PagerDuty integration within Datadog's integrations page and named it 'pagerduty'. Alternatively, you can directly use @pagerduty-your-service-name if you've configured a service-specific integration.
Clear Message: The message should contain context, affected resources (e.g., {{host.name}}, {{value}}), and actionable advice.
Priority and Tags: Assign a priority to help PagerDuty determine the urgency, and use tags for easy filtering and organization.

This monitor will automatically be created and managed by Terraform, ensuring your alert definitions are version-controlled and deployed consistently.

Deployment and Validation

With your Terraform configuration ready, follow these steps to deploy and validate your setup:

Initialize Terraform: Navigate to your Terraform project directory and run terraform init to download providers and modules.
Plan the Deployment: Execute terraform plan -var="datadog_api_key=YOUR_DD_API_KEY" -var="datadog_app_key=YOUR_DD_APP_KEY" to review the resources Terraform will create. Replace placeholders with your actual Datadog keys.
Apply the Configuration: Run terraform apply -var="datadog_api_key=YOUR_DD_API_KEY" -var="datadog_app_key=YOUR_DD_APP_KEY". Type yes when prompted to proceed with the creation of resources. This process can take 15-20 minutes.
Verify EKS Cluster: Once applied, configure kubectl to connect to your new EKS cluster:

            aws eks update-kubeconfig --name my-production-eks --region us-east-1
            kubectl get nodes
            kubectl get pods -A
        

Validate Datadog Integration:
- Log in to your Datadog account.
- Navigate to Infrastructure > Infrastructure List. You should see your EKS nodes reporting.
- Check Metrics > Explorer for Kubernetes metrics (e.g., kubernetes.cpu.usage.total).
- Confirm your Datadog monitor (EKS Node CPU Utilization High...) is active under Monitors > Manage Monitors.
Test PagerDuty Alerting: You can simulate an alert by manually adjusting the monitor threshold in Datadog to a value lower than current usage, or by intentionally stressing a node (though be cautious in production). Verify an incident is created in PagerDuty.

Advanced Considerations

For a truly production-ready setup, consider these enhancements:

IAM Roles for Service Accounts (IRSA): Configure specific IAM roles for Kubernetes service accounts to provide fine-grained AWS permissions to pods, improving security posture.
Cluster Autoscaler & Horizontal Pod Autoscaler (HPA): Implement both to automatically scale your EKS nodes and pods based on demand.
ExternalDNS & Cert-Manager: Automate DNS record management and TLS certificate provisioning within your cluster.
VPC CNI Tuning: Optimize the AWS VPC CNI for IP address management and network performance.
Security Hardening: Implement Network Policies, Pod Security Standards (PSS), and regularly review IAM roles.
GitOps with FluxCD/ArgoCD: Manage your Kubernetes deployments and configurations declaratively from Git.
Cost Optimization: Utilize Karpenter for intelligent node provisioning, spot instances where appropriate, and right-size your node groups.

Troubleshooting & Best Practices

Terraform Apply Errors: Check AWS account limits, IAM permissions, and ensure network CIDR blocks don't overlap. Debug with terraform apply -auto-approve -no-color | grep -E "Error:|Fail:".
EKS Node Issues: Verify worker nodes can join the cluster by checking their security groups, IAM instance profiles, and network connectivity to the EKS control plane. Look at logs on the worker nodes (/var/log/kubelet.log, /var/log/cloud-init-output.log).
Datadog Agent Not Reporting: Ensure the Helm chart deployed successfully (kubectl get pods -n datadog), API/APP keys are correct, and network policies aren't blocking outbound traffic to Datadog endpoints. Check agent logs: kubectl logs -f -n datadog <datadog-agent-pod-name>.
PagerDuty Alerts Not Firing: Confirm the Datadog monitor is correctly configured to mention @webhook-pagerduty (or your specific PagerDuty integration name). Verify the Datadog-PagerDuty integration itself is healthy in Datadog's integrations settings.
Alert Fatigue: Design your monitors carefully. Focus on actionable alerts, use composite monitors for complex conditions, and define clear escalation policies in PagerDuty.
State Management: Always use a remote backend (like S3 with DynamoDB locking) for Terraform state in production to prevent state corruption and enable team collaboration.

Conclusion

By leveraging Terraform for Infrastructure as Code, AWS EKS for managed Kubernetes, Datadog for comprehensive observability, and PagerDuty for reliable incident management, you can build a highly resilient, scalable, and observable cloud-native platform. This guide provides the foundation for a production-ready EKS environment, empowering your teams to deploy, monitor, and maintain applications with confidence and efficiency.

CloudStack YAML