Terraform AWS EKS Provisioning with Datadog Observability: A Comprehensive Guide

In the fast-evolving landscape of cloud-native development, deploying and managing Kubernetes clusters efficiently is paramount. AWS EKS (Elastic Kubernetes Service) provides a robust, managed Kubernetes environment, while Terraform offers powerful Infrastructure as Code (IaC) capabilities for automated provisioning. Integrating Datadog ensures comprehensive observability, giving you deep insights into your EKS cluster's performance, health, and security.

This guide will walk you through setting up an AWS EKS cluster using Terraform, and subsequently integrating Datadog for end-to-end monitoring, logging, and tracing. By the end, you'll have a fully provisioned and observable Kubernetes environment ready for your containerized applications.

Architecture Pro-Tip:

Always separate your Terraform code into logical modules (e.g., VPC, EKS Cluster, EKS Node Group, Datadog Agent) for better organization, reusability, and maintainability. This modular approach significantly improves readability and simplifies complex deployments, especially in multi-environment setups.

Prerequisites

AWS Account: With programmatic access and sufficient permissions to create EKS clusters, VPCs, IAM roles, etc.
Terraform CLI: Version 1.0 or higher installed.
AWS CLI: Configured with your AWS credentials.
kubectl CLI: Installed for interacting with the Kubernetes cluster.
Helm CLI: (Optional, but recommended for Datadog Agent deployment).
Datadog Account: With API and Application keys.

Core Concepts Explained

AWS EKS (Elastic Kubernetes Service)

EKS is a managed Kubernetes service that simplifies running Kubernetes on AWS without needing to install, operate, and maintain your own Kubernetes control plane. It integrates with various AWS services like EC2 for worker nodes, IAM for authentication, and VPC for networking.

Terraform Infrastructure as Code (IaC)

Terraform, by HashiCorp, allows you to define and provision cloud infrastructure using a declarative configuration language (HCL). It supports numerous cloud providers, including AWS, and enables consistent, repeatable, and version-controlled infrastructure deployments.

Datadog Observability

Datadog provides a unified platform for monitoring, logging, and tracing applications and infrastructure. For Kubernetes, it offers deep visibility into cluster health, pod performance, node resource utilization, and application-level metrics through its agent, seamlessly integrating with EKS.

Terraform Project Structure

A typical Terraform project for EKS provisioning includes:

main.tf: Defines the core resources (VPC, EKS cluster, Node Groups).
variables.tf: Declares input variables for customization.
outputs.tf: Defines output values from the provisioned infrastructure (e.g., cluster endpoint, ARN).
versions.tf: Specifies required Terraform and provider versions.
providers.tf: Configures cloud provider (AWS) and any other providers.

Step-by-Step EKS Provisioning with Terraform

1. Setup AWS Provider and Backend

First, configure the AWS provider and optionally a remote backend for state management (e.g., S3).

versions.tf:

terraform {
  required_version = ">= 1.0.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
  }
}

provider "aws" {
  region = var.aws_region
}
    

2. Define VPC and Subnets

EKS requires a dedicated VPC with public and private subnets. We'll use the terraform-aws-modules/vpc/aws module for simplicity.

vpc.tf (or part of main.tf):

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "${var.cluster_name}-vpc"
  cidr = "10.0.0.0/16"

  azs             = data.aws_availability_zones.available.names
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]

  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true

  tags = {
    Environment = var.environment
    Project     = var.cluster_name
  }
}

data "aws_availability_zones" "available" {}
    

3. Create EKS Cluster

Provision the EKS control plane and associated IAM roles. We recommend using the terraform-aws-modules/eks/aws module.

eks.tf (or part of main.tf):

module "eks_cluster" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0"

  cluster_name    = var.cluster_name
  cluster_version = "1.28"

  vpc_id                   = module.vpc.vpc_id
  subnet_ids               = module.vpc.private_subnets
  control_plane_subnet_ids = module.vpc.public_subnets # Public for load balancers

  # EKS Addons - crucial for cluster stability
  enable_cluster_creator_admin_permissions = true
  cluster_addons = {
    coredns = {
      resolve_conflicts = "OVERWRITE"
    }
    kube-proxy = {
      resolve_conflicts = "OVERWRITE"
    }
    vpc-cni = {
      resolve_conflicts = "OVERWRITE"
    }
  }

  # EKS Managed Node Group
  eks_managed_node_groups = {
    default = {
      name           = "${var.cluster_name}-node-group"
      instance_types = ["t3.medium"]
      min_size       = 2
      max_size       = 5
      desired_size   = 2
      subnet_ids     = module.vpc.private_subnets
    }
  }

  tags = {
    Environment = var.environment
    Project     = var.cluster_name
  }
}
    

Datadog Observability Integration

To integrate Datadog, we'll deploy the Datadog Agent to your EKS cluster. The agent collects metrics, logs, and traces from your nodes, pods, and applications.

1. Configure Kubernetes Provider

Terraform needs to authenticate with the EKS cluster. The Kubernetes provider uses the kubeconfig generated by AWS CLI.

providers.tf:

data "aws_eks_cluster" "cluster" {
  name = module.eks_cluster.cluster_id
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks_cluster.cluster_id
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  token                  = data.aws_eks_cluster_auth.cluster.token
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    token                  = data.aws_eks_cluster_auth.cluster.token
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
  }
}
    

2. Deploy Datadog Agent using Helm

The most common and recommended way to deploy the Datadog Agent on Kubernetes is via its Helm chart. You'll need your Datadog API and APP keys.

Ready-to-Use Configuration Example (`main.tf`)

Below is a consolidated main.tf, variables.tf, and outputs.tf example for setting up the basic EKS cluster and Datadog agent.

`main.tf`:

# main.tf

# Configure AWS provider
terraform {
  required_version = ">= 1.0.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

# Fetch available AZs
data "aws_availability_zones" "available" {}

# Create VPC for EKS
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "${var.cluster_name}-vpc"
  cidr = "10.0.0.0/16"

  azs             = data.aws_availability_zones.available.names
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]

  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true

  tags = {
    Environment = var.environment
    Project     = var.cluster_name
  }
}

# Create EKS Cluster
module "eks_cluster" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0"

  cluster_name    = var.cluster_name
  cluster_version = "1.28"

  vpc_id                   = module.vpc.vpc_id
  subnet_ids               = module.vpc.private_subnets
  control_plane_subnet_ids = module.vpc.public_subnets

  enable_cluster_creator_admin_permissions = true
  cluster_addons = {
    coredns = {
      resolve_conflicts = "OVERWRITE"
    }
    kube-proxy = {
      resolve_conflicts = "OVERWRITE"
    }
    vpc-cni = {
      resolve_conflicts = "OVERWRITE"
    }
  }

  eks_managed_node_groups = {
    default = {
      name           = "${var.cluster_name}-node-group"
      instance_types = ["t3.medium"]
      min_size       = 2
      max_size       = 5
      desired_size   = 2
      subnet_ids     = module.vpc.private_subnets
    }
  }

  tags = {
    Environment = var.environment
    Project     = var.cluster_name
  }
}

# Configure Kubernetes provider for kubectl/helm interaction
data "aws_eks_cluster" "cluster" {
  name = module.eks_cluster.cluster_id
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks_cluster.cluster_id
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  token                  = data.aws_eks_cluster_auth.cluster.token
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    token                  = data.aws_eks_cluster_auth.cluster.token
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
  }
}

# Deploy Datadog Agent using Helm
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.site"
    value = "datadoghq.com" # Or your specific Datadog site (e.g., eu.datadoghq.com)
  }
  set_sensitive {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }
  set_sensitive {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }
  set {
    name  = "kubeStateMetricsExternal.enabled"
    value = "true"
  }
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "processAgent.enabled"
    value = "true"
  }
  set {
    name  = "networkMonitoring.enabled"
    value = "true"
  }
  set {
    name  = "datadog.kubelet.tlsVerify"
    value = "false" # Set to true with proper cert setup
  }
}
    

`variables.tf`:

# variables.tf

variable "aws_region" {
  description = "AWS region for the deployment"
  type        = string
  default     = "us-east-1"
}

variable "cluster_name" {
  description = "Name for the EKS cluster"
  type        = string
  default     = "my-eks-cluster"
}

variable "environment" {
  description = "Deployment environment tag"
  type        = string
  default     = "dev"
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}
    

`outputs.tf`:

# outputs.tf

output "eks_cluster_id" {
  description = "The name/ID of the EKS cluster"
  value       = module.eks_cluster.cluster_id
}

output "eks_cluster_endpoint" {
  description = "The endpoint for the EKS cluster"
  value       = module.eks_cluster.cluster_endpoint
}

output "kubeconfig" {
  description = "Generated kubeconfig for the EKS cluster"
  value       = module.eks_cluster.kubeconfig
  sensitive   = true
}
    

Deployment Steps

Follow these steps to deploy your EKS cluster with Datadog:

Save Files: Place the above main.tf, variables.tf, and outputs.tf into a new directory (e.g., ./eks-datadog/).
Initialize Terraform: Navigate to your directory and run:
terraform init
Create terraform.tfvars: Create a file named terraform.tfvars in the same directory and provide your Datadog keys:
datadog_api_key = "YOUR_DATADOG_API_KEY" datadog_app_key = "YOUR_DATADOG_APP_KEY" # Optional: customize cluster_name and environment # cluster_name = "my-production-eks" # environment = "prod"

Note: For production, consider using environment variables or a secrets manager for sensitive keys.
Review Plan:
terraform plan

Review the proposed changes to ensure they align with your expectations.
Apply Changes:
terraform apply --auto-approve

This will take 15-20 minutes as EKS cluster creation is time-consuming.
Configure kubectl: After successful deployment, configure your kubectl to connect to the new cluster:
aws eks update-kubeconfig --region $(terraform output -raw aws_region) --name $(terraform output -raw eks_cluster_id)
Verify Datadog Agent: Check if the Datadog Agent pods are running:
kubectl get pods -n datadog

You should see datadog-agent pods in a Running state.

Verifying Observability in Datadog

Once the Datadog Agent is deployed and running, navigate to your Datadog dashboard:

Infrastructure List: You should see your EKS worker nodes reporting in.
Kubernetes Dashboard: Datadog automatically populates a comprehensive Kubernetes dashboard with cluster health, resource utilization, and pod status.
Container Map: Visualize your services, deployments, and pods.
Logs: If enabled, pod logs will be flowing into Datadog Log Explorer.
APM (Traces): If your applications are instrumented, traces will appear in the APM section.

Troubleshooting and Best Practices

Common Troubleshooting

EKS Cluster Creation Failure: Check IAM roles, subnet configurations, and security group rules. Ensure the EKS service role has necessary permissions.
Node Group Issues: Verify the instance type, IAM instance profile attached to the node group, and the availability of subnets. Look at CloudWatch logs for Auto Scaling Group or EC2 instance launch failures.
Datadog Agent Not Reporting:
- Check kubectl get events -n datadog for pod creation errors.
- Inspect agent logs: kubectl logs <datadog-agent-pod> -n datadog.
- Ensure datadog_api_key and datadog_app_key are correct and sensitive.
- Verify network connectivity from worker nodes to Datadog endpoints.
kubeconfig Issues: Ensure your AWS CLI is configured with the correct region and credentials.

Best Practices

Modularize Terraform: Break down your configuration into reusable modules (e.g., VPC, EKS, Datadog) for maintainability.
State Management: Always use a remote backend (like S3 with DynamoDB locking) for Terraform state in collaborative environments.
IAM Least Privilege: Grant only the necessary permissions to your EKS roles and node groups.
Security Groups: Configure EKS security groups to restrict traffic appropriately.
Resource Naming: Use consistent naming conventions with tags for easier resource identification and cost allocation.
Regular Updates: Keep your Terraform providers, EKS cluster, and Datadog Agent versions up-to-date for security patches and new features.
Cost Management: Monitor EKS costs using AWS Cost Explorer and Datadog's cost management features. Consider Karpenter for intelligent node scaling.

Conclusion

You've successfully provisioned an AWS EKS cluster using Terraform and integrated Datadog for comprehensive observability. This robust setup forms the foundation for deploying highly available, scalable, and observable containerized applications. By leveraging Infrastructure as Code and a powerful monitoring solution, you gain unparalleled control and insight into your cloud-native infrastructure, streamlining operations and accelerating your development cycles.

CloudStack YAML