Terraform for AWS EKS: Automated Datadog Monitoring and Alerting Configuration

Architecture Pro-Tip

For robust production environments, always decouple your EKS cluster definition from your monitoring setup. While this guide integrates them for clarity, consider using separate Terraform root modules for EKS infrastructure and Datadog configurations. This enhances modularity, allows independent deployment, and improves team collaboration by clearly delineating responsibilities (e.g., Platform Team for EKS, SRE/Observability Team for Datadog).

In the dynamic world of cloud-native applications, maintaining peak performance and ensuring high availability for Kubernetes clusters is paramount. AWS Elastic Kubernetes Service (EKS) provides a robust platform for orchestrating containers, but without comprehensive monitoring, operational blind spots can quickly emerge. This technical guide will walk you through leveraging Terraform to automate the deployment of Datadog for monitoring and alerting on your AWS EKS clusters, ensuring complete observability with Infrastructure as Code (IaC).

Automating your monitoring infrastructure with Terraform brings significant benefits:

Consistency: Ensure identical monitoring configurations across all your EKS environments (development, staging, production).
Version Control: Track changes to your monitoring setup, enabling easy rollbacks and collaboration.
Scalability: Easily apply monitoring to new EKS clusters as your infrastructure grows.
Auditability: Have a clear, declarative record of your monitoring and alerting policies.

Prerequisites

Before we dive into the configuration, ensure you have the following:

An active AWS Account with administrative access.
Terraform CLI (v1.0+) installed.
An active Datadog Account.
Basic understanding of AWS EKS and Kubernetes concepts.
kubectl configured to connect to your EKS cluster.
helm CLI installed (used by Terraform for the Datadog agent).

Understanding the Observability Stack

AWS EKS: Foundation for Containerized Workloads

AWS EKS simplifies the deployment, management, and scaling of Kubernetes applications in the AWS cloud. It handles the Kubernetes control plane, providing a highly available and resilient environment for your workloads. While EKS offers some integration with AWS monitoring tools like CloudWatch, a dedicated observability platform like Datadog provides a more comprehensive, unified view.

Datadog: Comprehensive Monitoring & Alerting

Datadog is an all-in-one monitoring and analytics platform for cloud applications. It integrates metrics, traces, and logs to give you full visibility into your EKS clusters, applications, and underlying AWS infrastructure. Key features for EKS include:

Unified Dashboarding: Visualize EKS health, pod states, resource utilization, and application performance in one place.
Real-time Metrics: Collects thousands of metrics from Kubernetes, Docker, and underlying EC2 instances.
Log Management: Ingests and analyzes logs from all Kubernetes components and applications.
Distributed Tracing (APM): End-to-end visibility into application requests.
Intelligent Alerting: Proactive notifications on anomalies and critical events with flexible notification channels.

Terraform: Infrastructure as Code for Automation

Terraform, by HashiCorp, allows you to define and provision infrastructure using a declarative configuration language. For EKS and Datadog, Terraform enables you to:

Provision and manage your EKS cluster resources (if not already existing).
Deploy the Datadog Agent using the Kubernetes provider.
Configure Datadog monitors, dashboards, and integrations using the Datadog provider.

Step-by-Step Implementation Guide

1. Initialize Your Terraform Project

Create a new directory for your Terraform configuration. Inside, you'll define your providers and variables. We'll use the AWS, Kubernetes, Helm, and Datadog providers.

2. Secure Datadog API & Application Keys

Your Datadog API Key and Application Key are crucial for Terraform to authenticate with Datadog. Never hardcode these keys in your Terraform files. For demonstration, we'll use Terraform variables. In production, consider using AWS Secrets Manager or environment variables.

Datadog API Key: Used by the Datadog Agent to send metrics and by the Datadog provider to create resources. Found under Integrations -> APIs.
Datadog Application Key: Used by the Datadog provider to create and manage resources. Found under Integrations -> APIs.

3. Deploy the Datadog Agent to EKS

The Datadog Agent collects metrics, logs, and traces from your Kubernetes cluster. We'll deploy it using the official Datadog Helm chart via Terraform's Helm provider. The Kubernetes provider will automatically detect your EKS cluster context if kubectl is configured correctly.

4. Configure Datadog Monitors and Dashboards

Once the agent is deployed, you can start defining your monitoring and alerting logic directly in Terraform using the datadog_monitor and datadog_dashboard resources. This ensures that your observability configuration is version-controlled and deployed alongside your infrastructure.

Ready-to-Use Terraform Configuration

The following comprehensive Terraform configuration (`main.tf` and `variables.tf` combined for simplicity) demonstrates how to deploy the Datadog Agent, configure a basic EKS overview dashboard, and set up critical alerts for your EKS cluster. Replace placeholder values like <YOUR_EKS_CLUSTER_NAME>, <YOUR_AWS_REGION>, and Datadog API keys with your actual environment details.

variable "aws_region" {
  description = "AWS region for the EKS cluster"
  type        = string
  default     = "us-east-1" # Change to your region
}

variable "eks_cluster_name" {
  description = "Name of the existing EKS cluster"
  type        = string
  default     = "my-production-eks-cluster" # Change to your EKS cluster name
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "datadog_site" {
  description = "The Datadog site to use (e.g., datadoghq.com, eu.datadoghq.com)"
  type        = string
  default     = "datadoghq.com"
}

variable "datadog_alert_recipients" {
  description = "Comma-separated list of email addresses or Datadog @handles for alerts"
  type        = string
  default     = "@pagerduty-prod" # Example: @slack-channel, email@example.com
}

# --- Providers Configuration ---
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.23"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

# Data source to retrieve EKS cluster details
data "aws_eks_cluster" "main" {
  name = var.eks_cluster_name
}

# Data source to retrieve EKS cluster authentication
data "aws_eks_cluster_auth" "main" {
  name = var.eks_cluster_name
}

# Kubernetes Provider Configuration
# Connects to the EKS cluster using the cluster endpoint and auth token
provider "kubernetes" {
  host                   = data.aws_eks_cluster.main.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.main.token
}

# Helm Provider Configuration
# Uses the Kubernetes provider for cluster access
provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.main.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.main.token
  }
}

# Datadog Provider Configuration
provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
  api_url = "https://api.${var.datadog_site}"
}

# --- Datadog Agent Deployment (Helm Chart) ---
resource "helm_release" "datadog_agent" {
  name       = "datadog-agent"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "datadog.site"
    value = var.datadog_site
  }

  set {
    name  = "kubeStateMetricsNetworkPolicy.enabled"
    value = "true"
  }

  set {
    name  = "agents.tolerations[0].operator"
    value = "Exists"
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "clusterChecksRunner.enabled"
    value = "true"
  }

  set {
    name  = "targetSystem"
    value = "linux"
  }

  # Enable APM for distributed tracing
  set {
    name  = "apm.enabled"
    value = "true"
  }

  # Enable Log Collection
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }

  # Integrations (optional, enable as needed)
  set {
    name  = "datadog.integrations.kubernetes_state.enabled"
    value = "true"
  }
  set {
    name  = "datadog.integrations.kube_proxy.enabled"
    value = "true"
  }

  # Add any additional values here as needed for your specific setup
  # For example, to enable admission controller:
  # set {
  #   name  = "datadog.admissionController.enabled"
  #   value = "true"
  # }
  # set {
  #   name  = "datadog.admissionController.mutateUnlabelled"
  #   value = "true"
  # }
}

# --- Datadog Monitors for EKS Health ---

# EKS Node Not Ready Monitor
resource "datadog_monitor" "eks_node_not_ready" {
  name                = "EKS: Node Not Ready - {{kube_cluster_name}}"
  type                = "metric alert"
  query               = "avg(last_5m):sum:kubernetes.node.ready{kube_cluster_name:\"${var.eks_cluster_name}\"} by {host} < 1"
  message             = "EKS Node {{host.name}} in cluster ${var.eks_cluster_name} is not ready! @webhook-ops ${var.datadog_alert_recipients}"
  monitor_threshold_windows {
    recovery_window = "15m"
  }
  monitor_thresholds {
    critical = "1"
    warning  = "0.5" # Can be tuned
  }
  renotify_interval   = 60
  notify_no_data      = true
  no_data_timeframe   = 30
  tags                = ["environment:production", "service:eks", "alert-type:critical"]
}

# EKS Pod Pending Monitor
resource "datadog_monitor" "eks_pod_pending" {
  name                = "EKS: Pods Pending - {{kube_cluster_name}}"
  type                = "metric alert"
  query               = "avg(last_5m):sum:kubernetes.pod.status{kube_cluster_name:\"${var.eks_cluster_name}\",status:pending} by {kube_namespace} > 0"
  message             = "There are {{value}} pending pods in namespace {{kube_namespace}} in EKS cluster ${var.eks_cluster_name}! @webhook-devops ${var.datadog_alert_recipients}"
  monitor_thresholds {
    critical = "1"
  }
  renotify_interval   = 60
  notify_no_data      = false
  tags                = ["environment:production", "service:eks", "alert-type:warning"]
}

# EKS Node CPU Utilization Monitor
resource "datadog_monitor" "eks_node_cpu_utilization" {
  name                = "EKS: High Node CPU Utilization - {{host.name}} in {{kube_cluster_name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.cpu.usage.total{kube_cluster_name:\"${var.eks_cluster_name}\"} by {host} > 80"
  message             = "CPU utilization on EKS node {{host.name}} in cluster ${var.eks_cluster_name} is {{value}}% and is above 80%! Consider scaling. ${var.datadog_alert_recipients}"
  monitor_thresholds {
    critical = "80"
    warning  = "70"
  }
  renotify_interval   = 120
  notify_no_data      = false
  tags                = ["environment:production", "service:eks", "alert-type:performance"]
}

# EKS Pod Restarts Monitor
resource "datadog_monitor" "eks_pod_restarts" {
  name                = "EKS: High Pod Restarts - {{kube_cluster_name}} / {{kube_namespace}}"
  type                = "metric alert"
  query               = "sum(last_5m):sum:kubernetes.pod.restarts{kube_cluster_name:\"${var.eks_cluster_name}\"} by {kube_namespace,kube_pod_name} > 3"
  message             = "Pod {{kube_pod_name}} in namespace {{kube_namespace}} (cluster ${var.eks_cluster_name}) has restarted {{value}} times in the last 5 minutes! Investigate. ${var.datadog_alert_recipients}"
  monitor_thresholds {
    critical = "3"
  }
  renotify_interval   = 30
  notify_no_data      = false
  tags                = ["environment:production", "service:eks", "alert-type:critical", "application:all"]
}


# --- Datadog Dashboard for EKS Overview ---
resource "datadog_dashboard" "eks_overview_dashboard" {
  title        = "EKS Cluster Overview - ${var.eks_cluster_name}"
  description  = "A comprehensive dashboard for monitoring EKS cluster health and performance."
  layout_type  = "ordered"
  is_read_only = true

  widget {
    type = "group"
    id   = 123456789 # Placeholder, Datadog assigns ID on creation. Remove for first apply.
    layout {
      x      = 0
      y      = 0
      width  = 48
      height = 5
    }
    title = "EKS Cluster Health - ${var.eks_cluster_name}"
    definition {
      layout_type = "ordered"
      background_color = "white"

      widget {
        type = "timeseries"
        id   = 123 # Placeholder
        definition {
          title = "Node CPU Utilization"
          show_legend = true
          request {
            q     = "avg:kubernetes.cpu.usage.total{kube_cluster_name:\"${var.eks_cluster_name}\"} by {host}"
            display_type = "area"
          }
        }
        layout {
          x = 0
          y = 0
          width = 12
          height = 4
        }
      }

      widget {
        type = "timeseries"
        id   = 124 # Placeholder
        definition {
          title = "Node Memory Utilization"
          show_legend = true
          request {
            q     = "avg:kubernetes.memory.usage.total{kube_cluster_name:\"${var.eks_cluster_name}\"} by {host}"
            display_type = "area"
          }
        }
        layout {
          x = 12
          y = 0
          width = 12
          height = 4
        }
      }

      widget {
        type = "query_value"
        id   = 125 # Placeholder
        definition {
          title = "Total Nodes Ready"
          autoscale = true
          request {
            q = "sum:kubernetes.node.ready{kube_cluster_name:\"${var.eks_cluster_name}\"}"
          }
          text_align = "left"
          timeseries_background = false
        }
        layout {
          x = 24
          y = 0
          width = 6
          height = 2
        }
      }

      widget {
        type = "query_value"
        id   = 126 # Placeholder
        definition {
          title = "Total Pods Running"
          autoscale = true
          request {
            q = "sum:kubernetes.pod.status{kube_cluster_name:\"${var.eks_cluster_name}\",status:running}"
          }
          text_align = "left"
          timeseries_background = false
        }
        layout {
          x = 30
          y = 0
          width = 6
          height = 2
        }
      }

      widget {
        type = "query_value"
        id   = 127 # Placeholder
        definition {
          title = "Total Pods Pending"
          autoscale = true
          request {
            q = "sum:kubernetes.pod.status{kube_cluster_name:\"${var.eks_cluster_name}\",status:pending}"
          }
          text_align = "left"
          timeseries_background = false
        }
        layout {
          x = 36
          y = 0
          width = 6
          height = 2
        }
      }

      widget {
        type = "query_value"
        id   = 128 # Placeholder
        definition {
          title = "Total Pod Restarts (5m)"
          autoscale = true
          request {
            q = "sum(last_5m):kubernetes.pod.restarts{kube_cluster_name:\"${var.eks_cluster_name}\"}"
          }
          text_align = "left"
          timeseries_background = false
        }
        layout {
          x = 42
          y = 0
          width = 6
          height = 2
        }
      }

      # Add more widgets as needed for a comprehensive overview
      # For example, per-namespace metrics, network I/O, disk usage, etc.
    }
  }

  tags = ["environment:production", "service:eks", "cluster:${var.eks_cluster_name}"]
}

output "datadog_agent_helm_status" {
  description = "Status of the Datadog Agent Helm release"
  value       = helm_release.datadog_agent.status
}

output "datadog_dashboard_url" {
  description = "URL to the created Datadog dashboard"
  value       = "https://app.${var.datadog_site}/dashboard/${datadog_dashboard.eks_overview_dashboard.id}"
}

output "datadog_monitor_urls" {
  description = "URLs to the created Datadog monitors"
  value = {
    node_not_ready      = "https://app.${var.datadog_site}/monitors/${datadog_monitor.eks_node_not_ready.id}"
    pod_pending         = "https://app.${var.datadog_site}/monitors/${datadog_monitor.eks_pod_pending.id}"
    node_cpu_util       = "https://app.${var.datadog_site}/monitors/${datadog_monitor.eks_node_cpu_utilization.id}"
    pod_restarts        = "https://app.${var.datadog_site}/monitors/${datadog_monitor.eks_pod_restarts.id}"
  }
}

How to Use This Configuration:

Save the above code as main.tf in an empty directory.
Replace placeholder values (e.g., my-production-eks-cluster, us-east-1) with your actual EKS cluster name and AWS region.
Set your DATADOG_API_KEY and DATADOG_APP_KEY as environment variables or pass them via the command line for security:
export TF_VAR_datadog_api_key="<YOUR_DATADOG_API_KEY>" export TF_VAR_datadog_app_key="<YOUR_DATADOG_APP_KEY>"
Initialize Terraform: terraform init
Review the planned changes: terraform plan
Apply the configuration: terraform apply (type yes to confirm)

After a successful apply, the Datadog Agent will be deployed to your EKS cluster, and the specified monitors and dashboard will be visible in your Datadog account. You can navigate to the outputted URLs to see your new resources.

Best Practices for Production Environments

While the above configuration provides a solid foundation, consider these best practices for production-grade deployments:

Secrets Management: Utilize AWS Secrets Manager or HashiCorp Vault to securely store and retrieve your Datadog API/App keys, rather than relying on environment variables for automated deployments.
Terraform Modules: Break down your configuration into reusable modules (e.g., an eks-datadog-agent module, an eks-datadog-monitors module) for better organization, reusability, and maintainability.
CI/CD Integration: Integrate your Terraform workflows into a CI/CD pipeline (e.g., GitLab CI/CD, GitHub Actions, AWS CodePipeline) to automate `plan` and `apply` operations upon code changes, ensuring GitOps principles.
Granular IAM Permissions: Ensure the IAM role/user executing Terraform has only the necessary permissions (least privilege) to interact with EKS and Datadog.
Tagging: Implement a consistent tagging strategy across all your AWS and Datadog resources. This improves cost allocation, resource identification, and filtering in Datadog.
Advanced Datadog Configuration: Explore advanced Datadog agent configurations (e.g., custom checks, log processing pipelines, APM instrumentation) and additional monitors/dashboards tailored to your specific applications and business needs.

Troubleshooting Common Issues

Datadog Agent Pods Not Running:
- Check kubectl get pods -n datadog for pod status.
- Examine pod logs: kubectl logs <datadog-agent-pod-name> -n datadog. Look for API key issues or connectivity problems.
- Verify EKS worker node connectivity to Datadog endpoints.
No Data in Datadog:
- Ensure the correct datadog.apiKey and datadog.site are configured in the Helm release.
- Confirm the Datadog Agent is running and healthy (kubectl describe pod <pod-name> -n datadog).
- Check Datadog's Agent Status page for your cluster.
Terraform EKS Provider Authentication Errors:
- Ensure your AWS CLI is configured with credentials that have permission to run eks:DescribeCluster and eks:ListClusters.
- Confirm kubectl is configured and can access the EKS cluster. Terraform leverages this context.
Datadog Monitor/Dashboard Errors:
- Verify TF_VAR_datadog_api_key and TF_VAR_datadog_app_key are correctly set for the Datadog provider.
- Double-check monitor query syntax in Datadog itself. Sometimes subtle typos cause issues.

Conclusion

Automating Datadog monitoring and alerting for your AWS EKS clusters with Terraform empowers your DevOps teams with unparalleled visibility, consistency, and efficiency. By treating your observability configuration as code, you gain the benefits of version control, CI/CD integration, and a repeatable process that scales effortlessly with your cloud-native infrastructure. This guide provides a robust foundation, enabling you to build upon it with more specific metrics, advanced alerts, and tailored dashboards to meet the unique demands of your applications.

Embrace Infrastructure as Code for monitoring, and take a significant step towards a fully observable and resilient EKS environment.

CloudStack YAML