Architecture Pro-Tip: Always isolate your EKS cluster's VPC, subnets, and security groups from other workloads for enhanced security and simplified network management. Leverage a dedicated Terraform module for your VPC to ensure reusability and maintain consistency across environments. Implement a robust IAM strategy, granting the least privilege necessary for EKS service roles and worker node instance profiles, and integrate Datadog for comprehensive visibility from day one to catch issues early.

Terraform Provisioning of AWS EKS with Datadog APM and Infrastructure Monitoring

Modern cloud-native applications demand robust, scalable, and observable infrastructure. AWS Elastic Kubernetes Service (EKS) offers a powerful platform for deploying containerized workloads, but managing its provisioning and ensuring comprehensive monitoring requires sophisticated tools. This guide provides a comprehensive, technical walkthrough on how to leverage Terraform for declarative provisioning of an AWS EKS cluster, seamlessly integrating Datadog for end-to-end Application Performance Monitoring (APM) and infrastructure observability.

Introduction to IaC, EKS, and Datadog

Infrastructure as Code (IaC) with Terraform

Infrastructure as Code (IaC) is a paradigm that manages and provisions computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools. Terraform, developed by HashiCorp, is an open-source IaC tool that enables you to define and provision datacenter infrastructure using a declarative configuration language. It supports a multitude of cloud providers, including AWS, making it an ideal choice for automating EKS deployments.

AWS Elastic Kubernetes Service (EKS)

AWS EKS is a managed Kubernetes service that makes it easy to deploy, manage, and scale containerized applications using Kubernetes on AWS. EKS runs the Kubernetes control plane across multiple Availability Zones to eliminate a single point of failure and provides a highly available and scalable architecture. While EKS simplifies Kubernetes management, its underlying infrastructure components (VPC, subnets, EC2 instances, IAM roles) still require careful provisioning.

Datadog for APM and Infrastructure Monitoring

Datadog is a leading monitoring and analytics platform for cloud-scale applications. It integrates and automates infrastructure monitoring, application performance monitoring (APM), log management, and more. For Kubernetes environments like EKS, Datadog offers deep visibility into container metrics, pod health, node resource utilization, and application traces, providing a unified view of your entire stack.

Prerequisites

AWS Account: With necessary permissions to create EKS clusters, VPCs, IAM roles, and EC2 instances.
Terraform CLI: Installed locally (version 1.0+ recommended).
AWS CLI: Configured with credentials and a default region.
Kubectl: Installed locally for interacting with the EKS cluster.
Helm CLI: Installed locally (version 3+ recommended) for deploying Datadog Agent.
Datadog Account: With an API key and Application key. You can find these in your Datadog organization settings.

Project Structure and Core Components

We'll organize our Terraform configuration into modules for better reusability and maintainability. A typical structure might look like this:

main.tf: Orchestrates modules and defines the EKS cluster.
variables.tf: Defines input variables.
outputs.tf: Defines output values.
providers.tf: Configures AWS and Kubernetes providers.
vpc.tf: Defines the VPC, subnets, and networking components.
iam.tf: Manages IAM roles for EKS and worker nodes.
eks.tf: Defines the EKS cluster and node groups.
datadog.tf: Deploys the Datadog Agent using the Helm provider.

Step-by-Step Terraform Configuration

1. Configure AWS and Kubernetes Providers

Start by defining your AWS provider and ensure Terraform can interact with your AWS account. We will also define the Kubernetes provider, which will be dynamically configured after the EKS cluster is created.

Create a providers.tf file:

provider "aws" {
  region = var.aws_region
}

# The Kubernetes provider configuration will be dynamic,
# depending on the EKS cluster output.
# It requires `host`, `cluster_ca_certificate`, and `token`.
# We'll configure this later with outputs from the EKS module.
provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}

2. Define Networking (VPC)

EKS requires a robust network configuration. We'll use the official Terraform AWS VPC module for simplicity and best practices.

Create a vpc.tf file:

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "${var.cluster_name}-vpc"
  cidr = "10.0.0.0/16"

  azs             = var.vpc_azs
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true

  tags = {
    Environment = var.environment
    Project     = var.cluster_name
  }
}

3. Set up IAM Roles for EKS

EKS requires specific IAM roles for the cluster itself and for the worker nodes. These roles grant necessary permissions for EKS to manage AWS resources.

Create an iam.tf file:

# IAM role for EKS Cluster
resource "aws_iam_role" "eks_cluster_role" {
  name = "${var.cluster_name}-eks-cluster-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "eks.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eks_cluster_role.name
}

resource "aws_iam_role_policy_attachment" "eks_service_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
  role       = aws_iam_role.eks_cluster_role.name
}

# IAM role for EKS Node Group (Worker Nodes)
resource "aws_iam_role" "eks_node_group_role" {
  name = "${var.cluster_name}-eks-node-group-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_worker_node_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.eks_node_group_role.name
}

resource "aws_iam_role_policy_attachment" "eks_cni_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.eks_node_group_role.name
}

resource "aws_iam_role_policy_attachment" "ec2_container_registry_readonly" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.eks_node_group_role.name
}

4. Provision the EKS Cluster and Node Group

Now, let's define the EKS cluster and an associated managed node group using the terraform-aws-modules/eks/aws module.

Create an eks.tf file:

module "eks_cluster" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = var.cluster_name
  cluster_version = var.kubernetes_version
  vpc_id          = module.vpc.vpc_id
  subnet_ids      = module.vpc.private_subnets
  cluster_endpoint_private_access = true
  cluster_endpoint_public_access  = true
  cluster_security_group_additional_rules = {
    ingress_self_all = {
      description                = "Allow traffic from cluster to itself"
      protocol                   = "all"
      from_port                  = 0
      to_port                    = 0
      type                       = "ingress"
      self                       = true
    }
  }

  enable_irsa = true # Enable IAM Roles for Service Accounts

  # EKS Cluster Role
  cluster_primary_security_group_id = module.vpc.default_security_group_id
  cluster_service_ipv4_cidr         = "172.20.0.0/16"
  create_vpc_endpoints              = true

  # Worker Node Group
  eks_managed_node_groups = {
    default = {
      name            = "${var.cluster_name}-node-group"
      instance_types  = [var.instance_type]
      min_size        = var.min_size
      max_size        = var.max_size
      desired_size    = var.desired_size
      disk_size       = 20
      ami_type        = "AL2_x86_64" # Amazon Linux 2
      capacity_type   = "ON_DEMAND"
      subnet_ids      = module.vpc.private_subnets # Private subnets for nodes
      iam_role_arn    = aws_iam_role.eks_node_group_role.arn

      tags = {
        Environment = var.environment
        Project     = var.cluster_name
      }
    }
  }

  tags = {
    Environment = var.environment
    Project     = var.cluster_name
  }
}

# Data sources to configure Kubernetes/Helm providers
data "aws_eks_cluster" "cluster" {
  name = module.eks_cluster.cluster_id
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks_cluster.cluster_id
}

5. Deploy Datadog Agent with Helm

Finally, we'll deploy the Datadog Agent to your EKS cluster using the Terraform Helm provider. This will automatically install the Datadog Agent, APM Agent, and Node Agent on your Kubernetes nodes.

Create a datadog.tf file:

resource "helm_release" "datadog_agent" {
  name             = "datadog"
  repository       = "https://helm.datadoghq.com"
  chart            = "datadog"
  namespace        = "datadog"
  create_namespace = true
  version          = "2.33.10" # Use a stable and recent version

  values = [
    templatefile("${path.module}/datadog-values.yaml", {
      datadog_api_key      = var.datadog_api_key
      datadog_app_key      = var.datadog_app_key
      datadog_site         = var.datadog_site # e.g., datadoghq.com, eu.datadoghq.com
      cluster_name         = var.cluster_name
      kube_state_metrics_url = "http://kube-state-metrics.kube-system.svc.cluster.local:8080/metrics" # Adjust if KSM is not in kube-system
    })
  ]

  set {
    name  = "datadog.kubeStateMetricsCore.enabled"
    value = "true"
  }

  set {
    name  = "datadog.apm.enabled"
    value = "true"
  }

  set {
    name  = "datadog.logCollection.enabled"
    value = "true"
  }

  set {
    name  = "datadog.containerExclude"
    value = "name:datadog-agent"
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "clusterAgent.metricsProvider.enabled"
    value = "true"
  }

  # Ensure the Helm provider is explicitly configured to depend on EKS cluster creation
  depends_on = [module.eks_cluster]
}

You'll also need a datadog-values.yaml template file to pass sensitive keys and dynamic values:

# datadog-values.yaml
datadog:
  apiKey: ${datadog_api_key}
  appKey: ${datadog_app_key}
  site: ${datadog_site}
  clusterName: ${cluster_name}
  kubeStateMetricsCore:
    enabled: true
    kubeStateMetricsUrl: ${kube_state_metrics_url}
  processAgent:
    enabled: true
  apm:
    enabled: true
  logs:
    enabled: true
    containerCollectAll: true
    config: |
      - type: docker
        container_collect_all: true

6. Variables and Outputs

Define your input variables in variables.tf:

# variables.tf
variable "aws_region" {
  description = "AWS region for the EKS cluster"
  type        = string
  default     = "us-east-1"
}

variable "cluster_name" {
  description = "Name of the EKS cluster"
  type        = string
  default     = "my-datadog-eks-cluster"
}

variable "environment" {
  description = "Environment tag for resources"
  type        = string
  default     = "development"
}

variable "kubernetes_version" {
  description = "Kubernetes version for the EKS cluster"
  type        = string
  default     = "1.28" # Check AWS EKS supported versions
}

variable "instance_type" {
  description = "Instance type for EKS worker nodes"
  type        = string
  default     = "t3.medium"
}

variable "min_size" {
  description = "Minimum size of the EKS node group"
  type        = number
  default     = 2
}

variable "max_size" {
  description = "Maximum size of the EKS node group"
  type        = number
  default     = 4
}

variable "desired_size" {
  description = "Desired size of the EKS node group"
  type        = number
  default     = 2
}

variable "vpc_azs" {
  description = "List of Availability Zones for the VPC"
  type        = list(string)
  default     = ["us-east-1a", "us-east-1b", "us-east-1c"]
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "datadog_site" {
  description = "Datadog site (e.g., datadoghq.com)"
  type        = string
  default     = "datadoghq.com"
}

And useful outputs in outputs.tf:

# outputs.tf
output "kubeconfig" {
  description = "Kubeconfig snippet to connect to the EKS cluster"
  value       = module.eks_cluster.kubeconfig
  sensitive   = true
}

output "cluster_endpoint" {
  description = "Endpoint for EKS control plane."
  value       = module.eks_cluster.cluster_endpoint
}

output "cluster_arn" {
  description = "ARN of the EKS cluster."
  value       = module.eks_cluster.cluster_arn
}

output "datadog_helm_release_status" {
  description = "Status of the Datadog Helm release"
  value       = helm_release.datadog_agent.status
}

Deployment Steps

With your Terraform configuration files set up, follow these steps to deploy your EKS cluster with Datadog monitoring:

Initialize Terraform: Navigate to your project directory in the terminal and run:
terraform init
This command initializes the working directory, downloads provider plugins, and sets up the backend.
Review the Plan: Generate an execution plan to see what Terraform will do. This is a crucial step to avoid unexpected changes.
terraform plan -var "datadog_api_key=YOUR_DD_API_KEY" -var "datadog_app_key=YOUR_DD_APP_KEY"
Replace YOUR_DD_API_KEY and YOUR_DD_APP_KEY with your actual Datadog keys. Consider using environment variables (TF_VAR_datadog_api_key) or a terraform.tfvars file (with appropriate security measures for sensitive data) for production environments.
Apply the Configuration: If the plan looks good, apply the changes to provision your infrastructure.
terraform apply -var "datadog_api_key=YOUR_DD_API_KEY" -var "datadog_app_key=YOUR_DD_APP_KEY"
Type yes when prompted to confirm the operation. This process can take 15-20 minutes as AWS provisions the EKS cluster and worker nodes.
Configure Kubectl: After successful deployment, update your kubeconfig file to interact with the new EKS cluster.
aws eks update-kubeconfig --region ${var.aws_region} --name ${var.cluster_name}
Alternatively, you can use the output directly:
terraform output kubeconfig > ~/.kube/config-eks && KUBECONFIG=~/.kube/config-eks kubectl get nodes

Verifying Datadog Integration

Once Terraform apply completes and your kubeconfig is updated, you can verify that Datadog is properly deployed and collecting data:

Check Datadog Pods:
kubectl get pods -n datadog
You should see datadog-agent pods running across your nodes, along with a datadog-cluster-agent pod.
View Datadog Agent Logs:
kubectl logs -f -n datadog <datadog-agent-pod-name>
Look for successful connections to the Datadog API and reports of collected metrics/traces.
Access Datadog Dashboard: Log in to your Datadog account.
- Navigate to Infrastructure -> Host Map to see your EKS worker nodes reporting.
- Go to Infrastructure -> Containers to see a detailed view of your Kubernetes pods and containers.
- Check APM -> Traces if you have an application deployed with Datadog APM instrumentation.
- Explore Logs -> Search to see logs collected from your EKS cluster.

Best Practices and Advanced Considerations

Environment Segregation: Use separate AWS accounts or distinct VPCs for development, staging, and production environments.
Secrets Management: Avoid hardcoding sensitive values like Datadog API keys. Use AWS Secrets Manager, AWS Parameter Store, or HashiCorp Vault with Terraform to manage secrets securely.
EKS Add-ons: Consider integrating other essential EKS add-ons like AWS Load Balancer Controller, Cluster Autoscaler, and ExternalDNS via Terraform for a fully automated setup.
Fine-grained IAM for Datadog: Instead of broad permissions, create an IAM Role for Service Accounts (IRSA) for the Datadog Agent to allow more granular AWS API access if needed (e.g., for EC2 tags, CloudWatch metrics).
Cost Optimization: Utilize AWS Spot Instances for your EKS worker node groups in non-production environments to reduce costs. Terraform can easily configure this.
Version Pinning: Always pin Terraform provider versions (e.g., ~> 5.0), module versions, and Helm chart versions to ensure consistent and reproducible deployments.
Continuous Integration/Continuous Delivery (CI/CD): Integrate your Terraform deployment into a CI/CD pipeline (e.g., GitLab CI, GitHub Actions, AWS CodePipeline) to automate provisioning and updates.
Observability Beyond Datadog: While Datadog is comprehensive, consider complementary tools for specific needs, such as Prometheus for custom metrics or Fluent Bit for advanced log forwarding.

Troubleshooting and FAQ

Q: Why is my Datadog Agent not reporting?

A: Common issues include incorrect Datadog API/App keys, network connectivity issues (firewall, security groups blocking outbound traffic to Datadog endpoints), or insufficient IAM permissions for the EKS worker nodes. Check Datadog Agent pod logs for errors and verify your network configuration.

Q: Terraform apply fails with "Access Denied" for EKS resources.

A: Ensure the AWS credentials used by Terraform have adequate IAM permissions to create/manage EKS clusters, IAM roles, VPCs, and EC2 instances. Review the policies attached to your AWS user or role.

Q: My EKS worker nodes are not joining the cluster.

A: Verify the IAM role attached to the worker nodes (aws_iam_role.eks_node_group_role) has the correct policies (AmazonEKSWorkerNodePolicy, AmazonEKS_CNI_Policy, AmazonEC2ContainerRegistryReadOnly). Also, check security group rules to ensure communication between the control plane and worker nodes is allowed. The EKS module handles most of this, but custom changes can break it.

Q: How do I upgrade my EKS cluster version with Terraform?

A: Update the kubernetes_version variable in your variables.tf file and run terraform apply. EKS supports in-place upgrades. For managed node groups, you might need to update the AMI type or initiate a rolling update through the EKS console or AWS CLI after the control plane upgrade.

Conclusion

Provisioning AWS EKS with Terraform and integrating Datadog for comprehensive monitoring provides a robust, scalable, and observable foundation for your containerized applications. By adopting Infrastructure as Code, you gain repeatability, version control, and auditability for your cloud infrastructure, while Datadog ensures you have critical insights into the performance and health of your services from the get-go. This guide empowers DevOps teams to build resilient and well-monitored cloud-native platforms efficiently.

CloudStack YAML