Terraform Configuration for AWS EKS with Datadog Observability Integration

In today's dynamic cloud landscape, managing Kubernetes clusters efficiently and ensuring robust observability are paramount. This guide delves into configuring an AWS Elastic Kubernetes Service (EKS) cluster using Terraform, an industry-standard Infrastructure as Code (IaC) tool, and seamlessly integrating Datadog for comprehensive monitoring, logging, and tracing. By the end of this tutorial, you'll have a fully deployed EKS cluster with Datadog agents reporting vital operational data.

Architecture Pro-Tip: Always separate your Terraform state for different environments (development, staging, production) using S3 backend buckets and DynamoDB for state locking. For EKS, design your VPC with sufficient CIDR blocks for future growth and ensure your IAM roles follow the principle of least privilege. Consider implementing a GitOps workflow with tools like ArgoCD or Flux for deploying applications on your EKS cluster, leveraging the foundational infrastructure defined by Terraform.

Prerequisites

Before embarking on this configuration journey, ensure you have the following prerequisites in place:

AWS Account: With programmatic access and appropriate permissions to create EKS clusters, VPCs, IAM roles, and other AWS resources.
Terraform CLI: Version 1.0 or higher installed on your local machine.
AWS CLI: Configured with your credentials.
Kubectl CLI: Installed to interact with your EKS cluster.
Datadog Account: With an API key and an Application key. You can sign up for a free trial.
Helm CLI: (Optional, but recommended for Datadog Agent deployment).

Core Concepts: Terraform, AWS EKS, and Datadog

Understanding the roles of each technology is crucial for a successful integration.

Terraform: An open-source IaC tool that allows you to define and provision cloud and on-premises resources in human-readable configuration files. It supports a vast ecosystem of providers, including AWS, making it ideal for managing EKS.
AWS EKS: A managed Kubernetes service that makes it easy to deploy, manage, and scale containerized applications on AWS. EKS handles the Kubernetes control plane, allowing you to focus on your applications.
Datadog: A comprehensive SaaS platform for monitoring cloud applications, servers, and databases. It provides end-to-end visibility into performance metrics, logs, traces, and user experience, critical for operating Kubernetes at scale.

Step-by-Step Terraform Configuration

1. Project Structure

Organize your Terraform files for clarity and maintainability. A common structure involves separate files for providers, VPC, EKS, IAM, and variables.

main.tf: Main configuration, calls modules.
variables.tf: Input variables.
outputs.tf: Output values from the deployment.
providers.tf: AWS and Kubernetes provider configuration.
vpc.tf: VPC, subnets, and routing.
iam.tf: IAM roles for EKS.
eks.tf: EKS cluster and node groups.
datadog.tf: Datadog specific resources (e.g., secrets for API keys).

2. Initialize Terraform Providers and Backend

First, define your AWS and Kubernetes providers. It's best practice to configure a remote backend (e.g., S3) for your Terraform state.

provider "aws" {
  region = var.aws_region
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
  # Avoid sensitive data in state by using an exec plugin, or ensure proper state encryption.
  # For local testing, you might need to run `aws eks update-kubeconfig --name ${cluster_name} --region ${aws_region}`
  # and then rely on the default kubeconfig for the provider to pick up context.
  # The token method above requires `aws_eks_cluster_auth` data source.
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
  }

  backend "s3" {
    bucket         = "your-terraform-state-bucket-unique-name" # Replace with your S3 bucket name
    key            = "eks/terraform.tfstate"
    region         = "us-east-1" # Replace with your desired region
    encrypt        = true
    dynamodb_table = "your-terraform-state-lock-table" # Replace with your DynamoDB table name
  }
}

3. Define VPC for EKS

A well-structured VPC is fundamental for EKS. Use a dedicated module for this.

# vpc.tf
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "eks-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["${var.aws_region}a", "${var.aws_region}b", "${var.aws_region}c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway = true
  single_nat_gateway = true
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Environment = var.environment
    Project     = "EKS-Datadog"
  }
}

4. Configure IAM Roles for EKS

EKS requires specific IAM roles for the cluster itself and for the node groups.

# iam.tf
resource "aws_iam_role" "eks_cluster_role" {
  name = "eks-cluster-role-${var.environment}"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "eks.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eks_cluster_role.name
}

resource "aws_iam_role_policy_attachment" "eks_service_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
  role       = aws_iam_role.eks_cluster_role.name
}

resource "aws_iam_role" "eks_node_group_role" {
  name = "eks-node-group-role-${var.environment}"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_worker_node_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.eks_node_group_role.name
}

resource "aws_iam_role_policy_attachment" "eks_cni_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.eks_node_group_role.name
}

resource "aws_iam_role_policy_attachment" "ec2_container_registry_readonly" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.eks_node_group_role.name
}

5. Create the EKS Cluster and Node Group

With VPC and IAM roles ready, define the EKS cluster and its managed node groups.

# eks.tf
module "eks_cluster" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0"

  cluster_name    = "my-datadog-eks-cluster-${var.environment}"
  cluster_version = "1.28"
  vpc_id          = module.vpc.vpc_id
  subnet_ids      = module.vpc.private_subnets

  cluster_endpoint_private_access = false # Set to true for private-only endpoint
  cluster_endpoint_public_access  = true  # Set to false to disable public access

  enable_irsa = true # Enable IAM Roles for Service Accounts

  eks_cluster_role_arn = aws_iam_role.eks_cluster_role.arn

  # EKS Managed Node Group
  eks_managed_node_groups = {
    default = {
      name           = "default-node-group"
      instance_types = ["t3.medium"]
      min_size       = 2
      max_size       = 5
      desired_size   = 2
      tags = {
        "Name" = "eks-node-group-${var.environment}"
        "k8s.io/cluster-autoscaler/enabled" = "true"
        "k8s.io/cluster-autoscaler/my-datadog-eks-cluster-${var.environment}" = "owned"
      }
      iam_role_arn = aws_iam_role.eks_node_group_role.arn
    }
  }

  tags = {
    Environment = var.environment
    Project     = "EKS-Datadog"
  }
}

# Data sources to configure Kubernetes provider
data "aws_eks_cluster" "cluster" {
  name = module.eks_cluster.cluster_id
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks_cluster.cluster_id
}

6. Datadog Observability Integration

Integrating Datadog requires deploying agents to your EKS cluster. We'll use the Helm provider for this, along with Kubernetes secrets for API keys.

6.1. Store Datadog API Keys Securely

It's crucial to manage your Datadog API and Application keys securely. For production, consider AWS Secrets Manager or HashiCorp Vault. For this guide, we'll demonstrate using Kubernetes secrets directly.

# datadog.tf
resource "kubernetes_secret" "datadog_api_key" {
  metadata {
    name      = "datadog-api-key"
    namespace = "default" # Or a dedicated monitoring namespace
  }

  data = {
    "api-key" = var.datadog_api_key
    "app-key" = var.datadog_app_key # Optional, but good for some Datadog features
  }
  type = "Opaque"
}

Remember to pass your Datadog API and Application keys via environment variables or a terraform.tfvars file, and never hardcode them.

6.2. Deploy Datadog Agent via Helm

The Datadog Agent collects metrics, logs, and traces from your cluster. Deploy it using the Helm provider.

7. Complete Terraform Configuration

Here's a consolidated view of the main.tf, variables.tf, and outputs.tf files, incorporating the Datadog Agent deployment.

# main.tf
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "eks-vpc-${var.environment}"
  cidr = var.vpc_cidr

  azs             = data.aws_availability_zones.available.names
  private_subnets = var.private_subnets
  public_subnets  = var.public_subnets

  enable_nat_gateway = true
  single_nat_gateway = true
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Environment = var.environment
    Project     = "EKS-Datadog"
  }
}

resource "aws_iam_role" "eks_cluster_role" {
  name = "eks-cluster-role-${var.environment}"
  assume_role_policy = jsonencode({
    Version = "2012-10-17", Statement = [{ Effect = "Allow", Principal = { Service = "eks.amazonaws.com" }, Action = "sts:AssumeRole" }]
  })
}
resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eks_cluster_role.name
}
resource "aws_iam_role_policy_attachment" "eks_service_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
  role       = aws_iam_role.eks_cluster_role.name
}

resource "aws_iam_role" "eks_node_group_role" {
  name = "eks-node-group-role-${var.environment}"
  assume_role_policy = jsonencode({
    Version = "2012-10-17", Statement = [{ Effect = "Allow", Principal = { Service = "ec2.amazonaws.com" }, Action = "sts:AssumeRole" }]
  })
}
resource "aws_iam_role_policy_attachment" "eks_worker_node_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.eks_node_group_role.name
}
resource "aws_iam_role_policy_attachment" "eks_cni_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.eks_node_group_role.name
}
resource "aws_iam_role_policy_attachment" "ec2_container_registry_readonly" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.eks_node_group_role.name
}


module "eks_cluster" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0"

  cluster_name    = "my-datadog-eks-cluster-${var.environment}"
  cluster_version = var.eks_cluster_version
  vpc_id          = module.vpc.vpc_id
  subnet_ids      = module.vpc.private_subnets

  cluster_endpoint_private_access = false
  cluster_endpoint_public_access  = true

  enable_irsa = true

  eks_cluster_role_arn = aws_iam_role.eks_cluster_role.arn

  eks_managed_node_groups = {
    default = {
      name           = "default-node-group"
      instance_types = ["t3.medium"]
      min_size       = var.node_group_min_size
      max_size       = var.node_group_max_size
      desired_size   = var.node_group_desired_size
      tags = {
        "Name" = "eks-node-group-${var.environment}"
        "k8s.io/cluster-autoscaler/enabled" = "true"
        "k8s.io/cluster-autoscaler/my-datadog-eks-cluster-${var.environment}" = "owned"
      }
      iam_role_arn = aws_iam_role.eks_node_group_role.arn
    }
  }

  tags = {
    Environment = var.environment
    Project     = "EKS-Datadog"
  }
}

data "aws_eks_cluster" "cluster" {
  name = module.eks_cluster.cluster_id
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks_cluster.cluster_id
}

data "aws_availability_zones" "available" {
  state = "available"
}


# Datadog Integration
resource "kubernetes_secret" "datadog_api_key" {
  metadata {
    name      = "datadog-api-key"
    namespace = "default"
  }

  data = {
    "api-key" = var.datadog_api_key
    "app-key" = var.datadog_app_key
  }
  type = "Opaque"
}

resource "helm_release" "datadog" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Or a dedicated monitoring namespace
  version    = "2.33.0" # Use a specific stable version

  set {
    name  = "datadog.site"
    value = "datadoghq.com" # Or eu.datadoghq.com, us3.datadoghq.com, etc.
  }

  set_sensitive {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }

  set_sensitive {
    name  = "datadog.appKey" # Needed for some Datadog features like events API
    value = var.datadog_app_key
  }

  set {
    name  = "clusterAgent.enabled"
    value = true
  }

  set {
    name  = "agents.tolerations[0].key"
    value = "CriticalAddonsOnly"
  }
  set {
    name  = "agents.tolerations[0].operator"
    value = "Exists"
  }

  set {
    name  = "datadog.kubeStateMetricsCore.enabled"
    value = true
  }
  set {
    name  = "datadog.logCollection.enabled"
    value = true
  }
  set {
    name  = "datadog.logCollection.logsConfigContainerCollectAll"
    value = true
  }
  set {
    name  = "datadog.apm.enabled"
    value = true
  }
  set {
    name  = "datadog.apm.hostPort"
    value = "8126"
  }
  set {
    name  = "datadog.networkMonitoring.enabled"
    value = true
  }
  set {
    name = "datadog.processAgent.enabled"
    value = true
  }
  set {
    name = "datadog.securityAgent.runtime.enabled"
    value = true
  }
}

# variables.tf
variable "aws_region" {
  description = "The AWS region to deploy resources."
  type        = string
  default     = "us-east-1"
}

variable "environment" {
  description = "The deployment environment (e.g., dev, staging, prod)."
  type        = string
  default     = "dev"
}

variable "vpc_cidr" {
  description = "The CIDR block for the VPC."
  type        = string
  default     = "10.0.0.0/16"
}

variable "public_subnets" {
  description = "List of public subnets CIDR blocks."
  type        = list(string)
  default     = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]
}

variable "private_subnets" {
  description = "List of private subnets CIDR blocks."
  type        = list(string)
  default     = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
}

variable "eks_cluster_version" {
  description = "The Kubernetes version for the EKS cluster."
  type        = string
  default     = "1.28"
}

variable "node_group_min_size" {
  description = "Minimum size of the EKS node group."
  type        = number
  default     = 2
}

variable "node_group_max_size" {
  description = "Maximum size of the EKS node group."
  type        = number
  default     = 5
}

variable "node_group_desired_size" {
  description = "Desired size of the EKS node group."
  type        = number
  default     = 2
}

variable "datadog_api_key" {
  description = "Your Datadog API Key. KEEP THIS SECURE!"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Your Datadog Application Key. KEEP THIS SECURE!"
  type        = string
  sensitive   = true
}

# outputs.tf
output "eks_cluster_name" {
  description = "The name of the EKS cluster."
  value       = module.eks_cluster.cluster_id
}

output "eks_cluster_endpoint" {
  description = "The endpoint for the EKS cluster."
  value       = module.eks_cluster.cluster_endpoint
}

output "kubeconfig_command" {
  description = "Command to update your kubeconfig for EKS."
  value       = "aws eks update-kubeconfig --name ${module.eks_cluster.cluster_id} --region ${var.aws_region}"
}

Deployment Steps

Follow these steps to deploy your EKS cluster with Datadog integration:

1. Create terraform.tfvars:
Create a file named terraform.tfvars in your project root and populate it with your specific values, including sensitive Datadog keys:

aws_region = "us-east-1" environment = "dev" datadog_api_key = "YOUR_DATADOG_API_KEY" datadog_app_key = "YOUR_DATADOG_APPLICATION_KEY" # ... other variable overrides
2. Initialize Terraform:
Navigate to your project directory in the terminal and run:

terraform init

This command initializes the working directory, downloads provider plugins, and sets up the S3 backend.
3. Plan the Deployment:
Review the changes Terraform plans to make before applying them:

terraform plan
4. Apply the Configuration:
Execute the plan to provision the resources:

terraform apply --auto-approve

This will take some time as EKS cluster creation can be lengthy.
5. Configure Kubectl:
Once the cluster is up, configure kubectl to interact with it using the output command:

aws eks update-kubeconfig --name my-datadog-eks-cluster-dev --region us-east-1

Verification

Confirm your EKS cluster and Datadog integration are operational:

Check EKS Nodes:
kubectl get nodes

You should see your EKS worker nodes in a 'Ready' state.
Check Datadog Agents:
kubectl get pods -n default -l app=datadog

Verify that Datadog Agent pods (datadog-agent-* and datadog-cluster-agent-*) are running.
Check Datadog UI:
Log in to your Datadog account. Navigate to Infrastructure -> Hosts or Kubernetes -> Clusters. You should see your EKS nodes and Kubernetes metrics appearing.

Troubleshooting and Best Practices

Common Issues

Terraform Apply Failure: Review the error messages carefully. Often, it's an IAM permission issue, an incorrect VPC CIDR, or an AWS service limit.
Datadog Agents Not Reporting:
- Check Datadog API and Application keys in your Kubernetes secret and Helm release values.
- Inspect Datadog Agent logs: kubectl logs datadog-agent-<pod-id> -n default.
- Ensure network connectivity from EKS nodes to Datadog endpoints (e.g., app.datadoghq.com).
Kubectl Connection Issues: Ensure your kubeconfig is correctly updated and your AWS CLI credentials are valid.

Best Practices

State Management: Always use a remote backend (like S3 with DynamoDB locking) for Terraform state in collaborative environments.
Modularity: Break down your Terraform configuration into reusable modules (e.g., a VPC module, an EKS module).
Version Pinning: Pin Terraform provider and module versions to prevent unexpected breaking changes.
Security: Use AWS Secrets Manager or other secrets management tools for sensitive data like Datadog API keys, instead of passing them directly as variables or storing them in plain text.
Tagging: Implement a consistent tagging strategy across all AWS resources for cost allocation and resource identification.
Datadog Customizations: Explore Datadog's extensive configuration options within the Helm chart for specific needs, such as custom metrics collection, specific log integrations, or APM instrumentation for your applications.

Conclusion

By following this comprehensive guide, you've successfully deployed an AWS EKS cluster using Terraform, a powerful IaC tool, and integrated it with Datadog for robust observability. This foundational setup provides a scalable, observable, and automated environment for your containerized applications, empowering your DevOps teams with the insights needed to maintain high performance and reliability. Continue to explore Datadog's capabilities and fine-tune your EKS configurations to meet the evolving demands of your infrastructure.

CloudStack YAML