Production Terraform IaC Template for AWS EKS Cluster with Datadog Monitoring

In today's dynamic cloud landscape, building scalable, resilient, and observable infrastructure is paramount for any successful application. This guide provides a comprehensive, production-ready Terraform IaC (Infrastructure as Code) template for deploying an AWS EKS (Elastic Kubernetes Service) cluster, seamlessly integrated with Datadog for robust monitoring and observability.

Architecture Pro-Tip

Design your infrastructure and monitoring layers concurrently. Integrating observability tools like Datadog from the initial IaC phase ensures that every component is inherently observable, reducing technical debt and enabling faster incident resolution. Prioritize modularity in your Terraform code, separating concerns like VPC, EKS cluster, and monitoring agents into distinct modules for reusability and maintainability across environments.

Why Terraform, AWS EKS, and Datadog?

The combination of Terraform, AWS EKS, and Datadog represents a powerful trifecta for modern cloud-native deployments:

Terraform: Enables declarative, version-controlled infrastructure provisioning across multiple cloud providers. It streamlines the creation and management of complex resources like EKS clusters, ensuring consistency and repeatability.
AWS EKS: Provides a fully managed Kubernetes control plane, offloading operational overhead and allowing teams to focus on application development. It offers high availability, scalability, and deep integration with other AWS services.
Datadog: An end-to-end monitoring, security, and analytics platform. For EKS, Datadog offers unparalleled visibility into cluster health, container performance, application logs, network traffic, and security events, all from a unified dashboard.

Solution Architecture Overview

Our production template for AWS EKS with Datadog monitoring will typically involve:

1. Core AWS Networking with VPC

A dedicated Virtual Private Cloud (VPC) with public and private subnets, NAT Gateways, Internet Gateways, and appropriate route tables and security groups to ensure secure and isolated network infrastructure for the EKS cluster.

2. AWS EKS Cluster and Node Groups

The EKS control plane, deployed across multiple Availability Zones for high availability. Managed node groups will be provisioned to run the Kubernetes worker nodes, offering various instance types and scaling options.

3. AWS IAM Roles and Policies

Granular IAM roles for the EKS control plane and node groups, ensuring the principle of least privilege. This includes roles for Kubernetes service accounts (IRSA) to securely access AWS resources.

4. Datadog Agent Deployment

The Datadog Agent, deployed as a DaemonSet across all EKS worker nodes, collecting metrics, logs, and traces. The Datadog Cluster Agent will be deployed for cluster-level metrics and to centralize communication with the Datadog API.

Prerequisites

Before you begin, ensure you have the following tools installed and configured:

Terraform CLI: Version 1.0 or higher.
AWS CLI: Configured with appropriate credentials to manage resources in your AWS account.
kubectl: For interacting with the Kubernetes cluster post-deployment.
Datadog Account: With an API Key and Application Key.
Helm CLI: For deploying the Datadog agent via Helm charts (recommended).

Terraform Module Structure for Production

A robust Terraform setup for production environments typically follows a modular approach. Here's a suggested directory structure:

.
├── environments/
│   ├── dev/
│   │   └── main.tf
│   └── prod/
│       └── main.tf
├── modules/
│   ├── datadog_agent/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── outputs.tf
│   ├── eks_cluster/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── outputs.tf
│   └── vpc/
│       ├── main.tf
│       ├── variables.tf
│       └── outputs.tf
└── README.md
    

This structure promotes reusability and allows for environment-specific configurations while leveraging shared modules.

Step-by-Step Implementation with Terraform

1. Define your AWS VPC

Start by defining your network infrastructure. We recommend using the terraform-aws-modules/vpc/aws module for a robust and production-ready VPC setup.

modules/vpc/main.tf:

resource "aws_vpc" "this" {
  cidr_block           = var.vpc_cidr
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = merge(
    var.tags,
    {
      Name = "${var.project_name}-vpc"
    }
  )
}

resource "aws_internet_gateway" "this" {
  vpc_id = aws_vpc.this.id
  tags   = var.tags
}

# ... other resources like subnets, route tables, NAT Gateways ...
    

2. Deploy the AWS EKS Cluster

Next, leverage the terraform-aws-modules/eks/aws module to provision your EKS cluster and node groups.

modules/eks_cluster/main.tf:

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0"

  cluster_name    = var.cluster_name
  cluster_version = var.kubernetes_version

  vpc_id                   = var.vpc_id
  subnet_ids               = var.private_subnet_ids
  control_plane_subnet_ids = var.public_subnet_ids # For public endpoint access if needed

  # EKS Managed Node Group
  eks_managed_node_groups = {
    default = {
      min_size       = var.node_group_min_size
      max_size       = var.node_group_max_size
      desired_size   = var.node_group_desired_size
      instance_types = var.node_instance_types
      capacity_type  = "ON_DEMAND" # or "SPOT" for cost savings
      labels = {
        "app.kubernetes.io/name" = var.cluster_name
      }
      tags = var.tags
    }
  }

  # EKS Addons (VPC CNI, CoreDNS, Kube-proxy)
  enable_irsa = true
  addons = {
    vpc_cni = {
      resolve_conflicts = "OVERWRITE"
    }
    coredns = {
      resolve_conflicts = "OVERWRITE"
    }
    kube_proxy = {
      resolve_conflicts = "OVERWRITE"
    }
  }

  tags = var.tags
}
    

3. Integrate Datadog Monitoring

To deploy the Datadog Agent, we'll use the Helm provider within Terraform. This allows us to manage Helm releases declaratively.

modules/datadog_agent/main.tf:

resource "helm_release" "datadog_agent" {
  name             = "datadog"
  repository       = "https://helm.datadoghq.com"
  chart            = "datadog"
  namespace        = var.namespace
  create_namespace = true
  version          = var.chart_version # e.g., "2.33.0"

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }
  set {
    name  = "datadog.site"
    value = var.datadog_site # e.g., "us5.datadoghq.com"
  }
  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
  }
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "clusterAgent.kubeStateMetricsEnabled"
    value = "true"
  }
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }
  set {
    name  = "apm.enabled"
    value = "true"
  }
  set {
    name  = "processAgent.enabled"
    value = "true"
  }
  set {
    name  = "networkMonitoring.enabled"
    value = "true"
  }
  set {
    name  = "securityAgent.compliance.enabled"
    value = "true"
  }
  set {
    name  = "securityAgent.runtime.enabled"
    value = "true"
  }
}
    

Ready-to-Use Configuration Example (`environments/prod/main.tf`)

This example brings together the modules in a production environment configuration. Remember to replace placeholder values with your actual configuration and sensitive data should be managed securely (e.g., using AWS Secrets Manager or Vault).

`environments/prod/main.tf`

# Configure AWS Provider
provider "aws" {
  region = "us-east-1"
}

# Configure Kubernetes Provider to interact with the EKS cluster
data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_id
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_id
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

# Configure Helm Provider
provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}

# -----------------------------------------------------------
# CORE VPC
# -----------------------------------------------------------
module "vpc" {
  source  = "../../modules/vpc"

  project_name = "production-eks"
  vpc_cidr     = "10.0.0.0/16"
  azs          = ["us-east-1a", "us-east-1b", "us-east-1c"]

  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway     = true
  single_nat_gateway     = true
  enable_dns_hostnames   = true
  enable_dns_support     = true
  map_public_ip_on_launch = false

  tags = {
    Environment = "production"
    ManagedBy   = "Terraform"
  }
}

# -----------------------------------------------------------
# EKS CLUSTER
# -----------------------------------------------------------
module "eks" {
  source  = "../../modules/eks_cluster"

  cluster_name       = "prod-my-app-eks"
  kubernetes_version = "1.27"

  vpc_id           = module.vpc.vpc_id
  private_subnet_ids = module.vpc.private_subnets
  public_subnet_ids  = module.vpc.public_subnets # For worker nodes to pull images from public repos or for public endpoint access if enabled

  node_group_min_size     = 3
  node_group_max_size     = 10
  node_group_desired_size = 3
  node_instance_types     = ["t3.medium"] # Choose appropriate instance types

  tags = {
    Environment = "production"
    ManagedBy   = "Terraform"
  }
}

# -----------------------------------------------------------
# DATADOG MONITORING
# -----------------------------------------------------------
module "datadog_agent" {
  source  = "../../modules/datadog_agent"

  # IMPORTANT: Use secure methods for secrets management in production
  # e.g., using data "aws_secretsmanager_secret_version" to fetch keys
  datadog_api_key = var.datadog_api_key
  datadog_app_key = var.datadog_app_key
  datadog_site    = "us5.datadoghq.com" # Adjust based on your Datadog region
  namespace       = "datadog"
  chart_version   = "2.33.0" # Always pin to a specific chart version
}

# -----------------------------------------------------------
# OUTPUTS
# -----------------------------------------------------------
output "eks_cluster_name" {
  description = "Name of the EKS cluster"
  value       = module.eks.cluster_id
}

output "kubeconfig" {
  description = "Kubeconfig for the EKS cluster"
  value       = module.eks.kubeconfig
  sensitive   = true
}
    

Variables (`environments/prod/variables.tf`)

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

# Add other variables as needed for cluster size, instance types, etc.
    

Deployment Steps:

Navigate to your environment directory: cd environments/prod
Initialize Terraform: terraform init
Review the plan: terraform plan -var="datadog_api_key=YOUR_DD_API_KEY" -var="datadog_app_key=YOUR_DD_APP_KEY" (For production, use environment variables or a secure secret management solution for keys).
Apply the configuration: terraform apply -var="datadog_api_key=YOUR_DD_API_KEY" -var="datadog_app_key=YOUR_DD_APP_KEY"

Post-Deployment Verification

After a successful terraform apply, verify your deployment:

Configure kubectl: Use the outputted kubeconfig or aws eks update-kubeconfig --name prod-my-app-eks --region us-east-1.
Check EKS Nodes: kubectl get nodes. You should see your worker nodes in a Ready state.
Verify Datadog Agents: kubectl get pods -n datadog. Ensure all Datadog Agent and Cluster Agent pods are running.
Check Datadog UI: Log in to your Datadog account. Navigate to Infrastructure List and Log Explorer to confirm metrics, logs, and host data are flowing in from your EKS cluster.

Best Practices for Production Environments

Terraform State Management: Always use a remote backend like AWS S3 with DynamoDB locking for Terraform state. This is crucial for collaborative environments and preventing state corruption.
Secrets Management: Avoid hardcoding sensitive values like Datadog API keys. Use AWS Secrets Manager, AWS Parameter Store, or HashiCorp Vault, and fetch them dynamically in Terraform.
IAM Roles for Service Accounts (IRSA): Leverage IRSA for fine-grained AWS permissions for your Kubernetes workloads, rather than granting broad permissions to EC2 instances.
Network Policies: Implement Kubernetes Network Policies to control traffic flow between pods for enhanced security.
Cost Optimization: Explore using AWS Spot Instances for non-critical workloads or leverage Karpenter for intelligent and cost-effective node provisioning.
Continuous Integration/Continuous Deployment (CI/CD): Integrate your Terraform code into a CI/CD pipeline (e.g., GitLab CI, GitHub Actions, Jenkins) to automate deployments and enforce best practices.
Monitoring Alerts: Configure critical alerts in Datadog for EKS cluster health, node group capacity, and application-specific metrics.
GitOps for Kubernetes: Consider adopting GitOps practices with tools like Argo CD or Flux to manage your Kubernetes applications and configurations declaratively.

Troubleshooting & FAQ

Q: Datadog Agent pods are stuck in `Pending`.

A: This often indicates a lack of resources (CPU/Memory) or issues with node affinity/taints. Check node resources (kubectl describe node <node-name>) and ensure nodes are healthy. Also, check pod events: kubectl describe pod <datadog-agent-pod-name> -n datadog.

Q: I'm getting "Access Denied" errors when Terraform tries to create EKS resources.

A: Ensure your AWS CLI credentials (used by Terraform) have sufficient IAM permissions to create/manage EKS clusters, VPCs, IAM roles, and EC2 instances. The AWS managed policies AmazonEKSClusterPolicy and AmazonEKSVPCResourceController are a good starting point, but refine for least privilege in production.

Q: Datadog metrics are not appearing in the UI.

A: Verify your datadog_api_key and datadog_app_key are correct and match your Datadog organization's region (datadog_site). Check Datadog Agent logs (kubectl logs <datadog-agent-pod-name> -n datadog) for any connection errors or misconfigurations.

Conclusion

Deploying an AWS EKS cluster with integrated Datadog monitoring using Terraform is a powerful strategy for managing modern, observable cloud-native applications. This guide provides a solid foundation for a production-grade setup, emphasizing modularity, security, and best practices. By automating your infrastructure and embedding comprehensive monitoring from day one, you empower your teams to build faster, operate more reliably, and respond proactively to challenges.

Continuously refine your Terraform modules, explore advanced Datadog features, and adapt your deployment strategy to meet the evolving needs of your applications and business. The journey to a fully automated, observable, and resilient cloud infrastructure is ongoing.

CloudStack YAML