Automating AWS EKS Observability with Terraform and Datadog

In the dynamic world of cloud-native applications, maintaining robust observability for Kubernetes clusters is paramount. AWS Elastic Kubernetes Service (EKS) provides a managed Kubernetes experience, but the complexity of monitoring distributed microservices still presents a significant challenge. This technical guide explores how to leverage the power of Infrastructure as Code (IaC) with Terraform and the comprehensive monitoring capabilities of Datadog to automate EKS observability, ensuring consistent, scalable, and efficient operations.

Architecture Pro-Tip:

For production environments, always establish a dedicated AWS account or VPC for your observability stack, separate from your application workloads. Implement strict IAM policies following the principle of least privilege, especially for the Datadog Agent's Service Account. Utilize AWS Tagging heavily for cost allocation, resource identification, and filtering within Datadog, ensuring consistency across your Terraform configurations.

The Observability Challenge in EKS Environments

Kubernetes, by design, is a highly distributed system. Applications running on EKS generate vast amounts of telemetry data across various layers:

• Cluster Level: Node health, pod scheduling, API server performance.
• Workload Level: Pod CPU/memory, network I/O, application logs, request latency.
• Infrastructure Level: Underlying EC2 instances, EBS volumes, VPC network components.

Manually configuring monitoring for each component is time-consuming, error-prone, and difficult to scale. Automation is key to achieving a proactive observability posture.

Why Terraform for EKS Observability?

Terraform, an open-source IaC tool, allows you to define and provision entire infrastructure stacks using a declarative configuration language. When applied to observability, Terraform offers several compelling advantages:

• Consistency and Repeatability: Deploy the same observability setup across multiple EKS clusters (development, staging, production) with identical configurations.
• Version Control: Manage observability configurations like any other code, enabling collaboration, peer review, and rollbacks.
• Drift Detection: Identify and rectify configuration drift where manual changes deviate from the desired state.
• Automation: Automate the entire lifecycle of your observability infrastructure, from deployment to updates and decommissioning.
• Multi-Cloud/Provider Support: Terraform's provider model allows it to manage resources across AWS, Kubernetes, and Datadog seamlessly.

Why Datadog for EKS Observability?

Datadog provides a unified monitoring and analytics platform that consolidates metrics, logs, and traces into a single pane of glass, making it an ideal choice for complex EKS environments:

• Comprehensive Kubernetes Integration: Out-of-the-box dashboards, monitors, and integrations for EKS, including cluster health, node performance, pod metrics, and daemonset status.
• Unified Data Platform: Correlate metrics, logs, and traces from applications, containers, and underlying infrastructure, simplifying root cause analysis.
• Application Performance Monitoring (APM): End-to-end distributed tracing for microservices running on EKS, identifying bottlenecks and performance issues.
• Network Performance Monitoring (NPM): Visualize and troubleshoot network communication between pods and services.
• Alerting and Incident Management: Powerful alerting capabilities with integrations to communication tools, reducing MTTR.
• Security Monitoring: Datadog Cloud Security Platform (CSPM/CSAM/CIEM) can monitor security posture across your EKS clusters.

Prerequisites

Before you begin, ensure you have the following:

• An active AWS Account with administrative privileges or appropriate IAM permissions to manage EKS, IAM roles, and EC2 instances.
• An existing AWS EKS Cluster. This guide assumes you have an EKS cluster already provisioned.
• Terraform CLI installed (v1.0.0 or later recommended).
• kubectl CLI installed and configured to connect to your EKS cluster.
• A Datadog account with API and Application keys.

Terraform Configuration for Datadog Agent Deployment

The core of EKS observability with Datadog relies on the Datadog Agent, deployed as a DaemonSet within your Kubernetes cluster. For secure and best practice deployment, we'll use IAM Roles for Service Accounts (IRSA) to grant the Datadog Agent the necessary AWS permissions without managing long-lived AWS credentials.

1. Provider Configuration

We'll need to configure the AWS, Kubernetes, and Datadog providers. Ensure your AWS provider is configured for the correct region and authentication.

provider "aws" {
  region = var.aws_region
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.example.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.example.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.example.token
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}
    

2. EKS Cluster Data Source

Retrieve information about your existing EKS cluster.

data "aws_eks_cluster" "example" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "example" {
  name = var.eks_cluster_name
}
    

3. IAM Role for Service Account (IRSA)

Create an IAM role and attach policies that grant the Datadog Agent permissions to collect metrics from AWS services (e.g., EC2, S3, CloudWatch). This role will be assumed by the Kubernetes Service Account.

resource "aws_iam_policy" "datadog_agent" {
  name        = "${var.eks_cluster_name}-datadog-agent-policy"
  description = "IAM policy for Datadog Agent to access AWS resources"

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action = [
          "ec2:DescribeInstances",
          "ec2:DescribeRegions",
          "tag:GetResources",
          "autoscaling:DescribeAutoScalingGroups",
          "ec2:DescribeVpcs",
          "ec2:DescribeSubnets",
          "s3:GetBucketLocation",
          "s3:ListAllMyBuckets",
          "s3:GetLifecycleConfiguration",
          "s3:GetBucketTagging",
          "s3:GetBucketAcl",
          "s3:GetBucketPolicyStatus",
          "s3:GetBucketPublicAccessBlock",
          "s3:GetBucketRequestPayment",
          "s3:GetEncryptionConfiguration",
          "s3:GetBucketVersioning",
          "s3:GetReplicationConfiguration",
          "s3:GetAccelerateConfiguration",
          "cloudwatch:ListMetrics",
          "cloudwatch:GetMetricStatistics",
          "cloudwatch:GetMetricData",
          "logs:DescribeLogGroups",
          "logs:DescribeLogStreams",
          "logs:FilterLogEvents"
        ],
        Effect   = "Allow",
        Resource = "*"
      },
    ]
  })
}

resource "aws_iam_role" "datadog_agent" {
  name               = "${var.eks_cluster_name}-datadog-agent-role"
  assume_role_policy = data.aws_iam_policy_document.datadog_agent_assume_role.json
}

resource "aws_iam_role_policy_attachment" "datadog_agent" {
  role       = aws_iam_role.datadog_agent.name
  policy_arn = aws_iam_policy.datadog_agent.arn
}

data "aws_iam_policy_document" "datadog_agent_assume_role" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    effect  = "Allow"

    principals {
      type        = "Federated"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${replace(data.aws_eks_cluster.example.identity.0.oidc.0.issuer, "https://", "")}"]
    }

    condition {
      test     = "StringEquals"
      variable = "${replace(data.aws_eks_cluster.example.identity.0.oidc.0.issuer, "https://", "")}:sub"
      values   = ["system:serviceaccount:datadog:datadog-agent"]
    }
  }
}

data "aws_caller_identity" "current" {}
    

4. Kubernetes Manifests for Datadog Agent

Now we'll define the Kubernetes resources for the Datadog Agent using the Terraform Kubernetes provider. This includes a Service Account, ClusterRole, ClusterRoleBinding, ConfigMap, and the DaemonSet itself.

# Namespace for Datadog Agent
resource "kubernetes_namespace" "datadog" {
  metadata {
    name = "datadog"
  }
}

# Service Account for Datadog Agent with IRSA annotation
resource "kubernetes_service_account" "datadog_agent" {
  metadata {
    name      = "datadog-agent"
    namespace = kubernetes_namespace.datadog.metadata.0.name
    annotations = {
      "eks.amazonaws.com/role-arn" = aws_iam_role.datadog_agent.arn
    }
  }
}

# Cluster Role for Datadog Agent
resource "kubernetes_cluster_role" "datadog_agent" {
  metadata {
    name = "datadog-agent"
  }

  rule {
    api_groups = [""]
    resources = [
      "nodes",
      "nodes/metrics",
      "nodes/spec",
      "nodes/proxy",
      "pods",
      "services"
    ]
    verbs = ["get", "list", "watch"]
  }

  rule {
    api_groups = ["extensions", "apps"]
    resources = [
      "replicasets",
      "daemonsets"
    ]
    verbs = ["get", "list", "watch"]
  }

  rule {
    api_groups = ["batch"]
    resources = [
      "jobs"
    ]
    verbs = ["get", "list", "watch"]
  }

  rule {
    api_groups = [""]
    resources = [
      "events"
    ]
    verbs = ["get", "list", "watch"]
  }

  rule {
    api_groups = [""]
    resources = [
      "configmaps"
    ]
    verbs = ["get"]
  }

  rule {
    api_groups = ["autoscaling"]
    resources = [
      "horizontalpodautoscalers"
    ]
    verbs = ["get", "list", "watch"]
  }
}

# Cluster Role Binding for Datadog Agent
resource "kubernetes_cluster_role_binding" "datadog_agent" {
  metadata {
    name = "datadog-agent"
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = kubernetes_cluster_role.datadog_agent.metadata.0.name
  }
  subject {
    kind      = "ServiceAccount"
    name      = kubernetes_service_account.datadog_agent.metadata.0.name
    namespace = kubernetes_namespace.datadog.metadata.0.name
  }
}

# ConfigMap for Datadog Agent (minimal example)
resource "kubernetes_config_map" "datadog_agent_config" {
  metadata {
    name      = "datadog-agent"
    namespace = kubernetes_namespace.datadog.metadata.0.name
  }

  data = {
    "datadog.yaml" = <<-EOT
      api_key: ${var.datadog_api_key}
      tags:
        - environment:${var.environment}
        - cluster_name:${var.eks_cluster_name}
      logs_enabled: true
      log_level: INFO
      process_config:
        enabled: "true"
      apm_config:
        enabled: "true"
    EOT
  }
}

# Datadog Agent DaemonSet
resource "kubernetes_daemon_set" "datadog_agent" {
  metadata {
    name      = "datadog-agent"
    namespace = kubernetes_namespace.datadog.metadata.0.name
    labels = {
      "app" = "datadog-agent"
    }
  }

  spec {
    selector {
      match_labels = {
        "app" = "datadog-agent"
      }
    }
    template {
      metadata {
        labels = {
          "app" = "datadog-agent"
        }
      }
      spec {
        service_account_name = kubernetes_service_account.datadog_agent.metadata.0.name
        toleration {
          operator = "Exists"
        }
        container {
          name  = "agent"
          image = "gcr.io/datadog/agent:7.48.0" # Use a specific stable version

          env {
            name  = "DD_API_KEY"
            value_from {
              secret_key_ref {
                name = "datadog-secret"
                key  = "api-key"
              }
            }
          }
          env {
            name  = "DD_APP_KEY"
            value_from {
              secret_key_ref {
                name = "datadog-secret"
                key  = "app-key"
              }
            }
          }
          env {
            name  = "DD_SITE"
            value = "datadoghq.com" # or eu.datadoghq.com, etc.
          }
          env {
            name  = "DD_LOGS_ENABLED"
            value = "true"
          }
          env {
            name  = "DD_PROCESS_AGENT_ENABLED"
            value = "true"
          }
          env {
            name  = "DD_APM_ENABLED"
            value = "true"
          }
          env {
            name  = "DD_EKS_FARGATE"
            value = var.eks_fargate_cluster ? "true" : "false" # Set if using Fargate
          }
          env {
            name  = "DD_KUBERNETES_KUBELET_HOST"
            value_from {
              field_ref {
                field_path = "status.hostIP"
              }
            }
          }
          env {
            name  = "DD_ORCHESTRATOR_EXPLORER_ENABLED"
            value = "true" # For Datadog Cluster Agent and Kubernetes Explorer
          }
          env {
            name  = "DD_CLUSTER_AGENT_ENABLED"
            value = "true" # Enable if deploying Cluster Agent separately
          }

          port {
            name           = "dogstatsd"
            container_port = 8125
            host_port      = 8125
            protocol       = "UDP"
          }
          port {
            name           = "traceport"
            container_port = 8126
            host_port      = 8126
            protocol       = "TCP"
          }

          volume_mount {
            name       = "proc"
            mount_path = "/host/proc"
            read_only  = true
          }
          volume_mount {
            name       = "cgroup"
            mount_path = "/host/sys/fs/cgroup"
            read_only  = true
          }
          volume_mount {
            name       = "var-run"
            mount_path = "/var/run"
            read_only  = true
          }
          volume_mount {
            name       = "config"
            mount_path = "/etc/datadog-agent/datadog.yaml"
            sub_path   = "datadog.yaml"
          }
        }
        volume {
          name = "proc"
          host_path {
            path = "/proc"
          }
        }
        volume {
          name = "cgroup"
          host_path {
            path = "/sys/fs/cgroup"
          }
        }
        volume {
          name = "var-run"
          host_path {
            path = "/var/run"
          }
        }
        volume {
          name = "config"
          config_map {
            name = kubernetes_config_map.datadog_agent_config.metadata.0.name
          }
        }
      }
    }
  }
}

# Secret for Datadog API and APP keys
resource "kubernetes_secret" "datadog_secret" {
  metadata {
    name      = "datadog-secret"
    namespace = kubernetes_namespace.datadog.metadata.0.name
  }

  data = {
    "api-key" = var.datadog_api_key
    "app-key" = var.datadog_app_key
  }
  type = "Opaque"
}
    

5. Datadog Monitors and Dashboards (Optional but Recommended)

The Datadog Terraform provider allows you to manage Datadog monitors, dashboards, and other configurations as code. This ensures your alerting and visualization setup is consistent and version-controlled alongside your agent deployment.

# Example: Datadog Monitor for EKS Node CPU Utilization
resource "datadog_monitor" "eks_node_cpu_high" {
  name               = "[EKS] High CPU utilization on {{host.name}}"
  type               = "metric alert"
  message            = "CPU utilization is above 80% for node {{host.name}}. Investigate capacity. @webhook-devops"
  query              = "avg(last_5m):avg:system.cpu.idle{cluster_name:${var.eks_cluster_name}} by {host} < 20"
  monitor_thresholds {
    critical = 80
    warning  = 70
  }
  notify_no_data       = false
  new_group_delay      = 60
  no_data_timeframe    = 20
  include_tags         = true
  require_full_window  = false
  evaluation_delay     = 90
  renotify_interval    = 0
  locked               = false
  timeout_h            = 0
  restricted_roles     = []
  force_delete         = false
  notify_audit         = false
  enable_logs_sample   = false
  escalation_message   = ""
  tags                 = ["eks", "cpu", "alert"]
}

# Example: Datadog Dashboard (simplified)
resource "datadog_dashboard" "eks_overview" {
  title        = "[Terraform] EKS Cluster Overview - ${var.eks_cluster_name}"
  description  = "Overview dashboard for EKS cluster ${var.eks_cluster_name}"
  layout_type  = "ordered"
  is_read_only = false
  widget {
    definition {
      type          = "timeseries"
      title         = "EKS Node CPU Utilization"
      live_span     = "1h"
      request {
        q = "avg:system.cpu.usage{cluster_name:${var.eks_cluster_name}} by {host}"
        display_type = "area"
        style {
          palette = "dog_classic"
          type    = "solid"
          width   = "normal"
        }
      }
    }
  }
  widget {
    definition {
      type          = "timeseries"
      title         = "EKS Node Memory Utilization"
      live_span     = "1h"
      request {
        q = "avg:system.mem.used{cluster_name:${var.eks_cluster_name}} by {host}"
        display_type = "area"
        style {
          palette = "dog_classic"
          type    = "solid"
          width   = "normal"
        }
      }
    }
  }
  tags = ["eks", "dashboard", "terraform"]
}
    

Ready-to-Use Terraform Configuration (main.tf and variables.tf)

Here's a consolidated example of the Terraform configuration (`main.tf`) and its corresponding variables (`variables.tf`). Remember to replace placeholder values and adapt the configuration to your specific needs.

main.tf

provider "aws" { region = var.aws_region } provider "kubernetes" { host = data.aws_eks_cluster.example.endpoint cluster_ca_certificate = base64decode(data.aws_eks_cluster.example.certificate_authority.0.data) token = data.aws_eks_cluster_auth.example.token } provider "datadog" { api_key = var.datadog_api_key app_key = var.datadog_app_key } data "aws_eks_cluster" "example" { name = var.eks_cluster_name } data "aws_eks_cluster_auth" "example" { name = var.eks_cluster_name } data "aws_caller_identity" "current" {} # --- IAM Role and Policy for Datadog Agent (IRSA) --- resource "aws_iam_policy" "datadog_agent_policy" { name = "${var.eks_cluster_name}-datadog-agent-policy" description = "IAM policy for Datadog Agent to access AWS resources" policy = jsonencode({ Version = "2012-10-17", Statement = [ { Action = [ "ec2:DescribeInstances", "ec2:DescribeRegions", "tag:GetResources", "autoscaling:DescribeAutoScalingGroups", "ec2:DescribeVpcs", "ec2:DescribeSubnets", "s3:GetBucketLocation", "s3:ListAllMyBuckets", "s3:GetLifecycleConfiguration", "cloudwatch:ListMetrics", "cloudwatch:GetMetricStatistics", "cloudwatch:GetMetricData", "logs:DescribeLogGroups", "logs:DescribeLogStreams", "logs:FilterLogEvents" ], Effect = "Allow", Resource = "*" }, ] }) } resource "aws_iam_role" "datadog_agent_role" { name = "${var.eks_cluster_name}-datadog-agent-role" assume_role_policy = data.aws_iam_policy_document.datadog_agent_assume_role.json } resource "aws_iam_role_policy_attachment" "datadog_agent_attach" { role = aws_iam_role.datadog_agent_role.name policy_arn = aws_iam_policy.datadog_agent_policy.arn } data "aws_iam_policy_document" "datadog_agent_assume_role" { statement { actions = ["sts:AssumeRoleWithWebIdentity"] effect = "Allow" principals { type = "Federated" identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${replace(data.aws_eks_cluster.example.identity.0.oidc.0.issuer, "https://", "")}"] } condition { test = "StringEquals" variable = "${replace(data.aws_eks_cluster.example.identity.0.oidc.0.issuer, "https://", "")}:sub" values = ["system:serviceaccount:datadog:datadog-agent"] } } } # --- Kubernetes Resources for Datadog Agent --- resource "kubernetes_namespace" "datadog" { metadata { name = "datadog" } } resource "kubernetes_service_account" "datadog_agent" { metadata { name = "datadog-agent" namespace = kubernetes_namespace.datadog.metadata.0.name annotations = { "eks.amazonaws.com/role-arn" = aws_iam_role.datadog_agent_role.arn } } } resource "kubernetes_cluster_role" "datadog_agent" { metadata { name = "datadog-agent" } rule { api_groups = [""] resources = ["nodes", "nodes/metrics", "nodes/spec", "nodes/proxy", "pods", "services", "events", "configmaps"] verbs = ["get", "list", "watch"] } rule { api_groups = ["extensions", "apps"] resources = ["replicasets", "daemonsets"] verbs = ["get", "list", "watch"] } rule { api_groups = ["batch"] resources = ["jobs"] verbs = ["get", "list", "watch"] } rule { api_groups = ["autoscaling"] resources = ["horizontalpodautoscalers"] verbs = ["get", "list", "watch"] } } resource "kubernetes_cluster_role_binding" "datadog_agent" { metadata { name = "datadog-agent" } role_ref { api_group = "rbac.authorization.k8s.io" kind = "ClusterRole" name = kubernetes_cluster_role.datadog_agent.metadata.0.name } subject { kind = "ServiceAccount" name = kubernetes_service_account.datadog_agent.metadata.0.name namespace = kubernetes_namespace.datadog.metadata.0.name } } resource "kubernetes_config_map" "datadog_agent_config" { metadata { name = "datadog-agent" namespace = kubernetes_namespace.datadog.metadata.0.name } data = { "datadog.yaml" = <<-EOT api_key: ${var.datadog_api_key} tags: - environment:${var.environment} - cluster_name:${var.eks_cluster_name} logs_enabled: true log_level: INFO process_config: enabled: "true" apm_config: enabled: "true" kubelet_tls_verify: false # Set to true if kubelet has valid certs EOT } } resource "kubernetes_secret" "datadog_secret" { metadata { name = "datadog-secret" namespace = kubernetes_namespace.datadog.metadata.0.name } data = { "api-key" = var.datadog_api_key "app-key" = var.datadog_app_key } type = "Opaque" } resource "kubernetes_daemon_set" "datadog_agent" { metadata { name = "datadog-agent" namespace = kubernetes_namespace.datadog.metadata.0.name labels = { "app" = "datadog-agent" } } spec { selector { match_labels = { "app" = "datadog-agent" } } template { metadata { labels = { "app" = "datadog-agent" } } spec { service_account_name = kubernetes_service_account.datadog_agent.metadata.0.name toleration { operator = "Exists" } container { name = "agent" image = "gcr.io/datadog/agent:7.48.0" # Always pin to a specific version env { name = "DD_API_KEY" value_from { secret_key_ref { name = kubernetes_secret.datadog_secret.metadata.0.name key = "api-key" } } } env { name = "DD_APP_KEY" value_from { secret_key_ref { name = kubernetes_secret.datadog_secret.metadata.0.name key = "app-key" } } } env { name = "DD_SITE"; value = var.datadog_site } env { name = "DD_LOGS_ENABLED"; value = "true" } env { name = "DD_PROCESS_AGENT_ENABLED"; value = "true" } env { name = "DD_APM_ENABLED"; value = "true" } env { name = "DD_EKS_FARGATE"; value = var.eks_fargate_cluster ? "true" : "false" } env { name = "DD_KUBERNETES_KUBELET_HOST" value_from { field_ref { field_path = "status.hostIP" } } } env { name = "DD_ORCHESTRATOR_EXPLORER_ENABLED"; value = "true" } env { name = "DD_CLUSTER_AGENT_ENABLED"; value = "true" } env { name = "DD_KUBERNETES_STATE_METRICS_ENABLED"; value = "true" } port { name = "dogstatsd" container_port = 8125 host_port = 8125 protocol = "UDP" } port { name = "traceport" container_port = 8126 host_port = 8126 protocol = "TCP" } volume_mount { name = "proc"; mount_path = "/host/proc"; read_only = true } volume_mount { name = "cgroup"; mount_path = "/host/sys/fs/cgroup"; read_only = true } volume_mount { name = "var-run"; mount_path = "/var/run"; read_only = true } volume_mount { name = "config"; mount_path = "/etc/datadog-agent/datadog.yaml"; sub_path = "datadog.yaml" } volume_mount { name = "host-id"; mount_path = "/etc/machine-id"; read_only = true } # For host_aliases } volume { name = "proc"; host_path { path = "/proc" } } volume { name = "cgroup"; host_path { path = "/sys/fs/cgroup" } } volume { name = "var-run"; host_path { path = "/var/run" } } volume { name = "config" config_map { name = kubernetes_config_map.datadog_agent_config.metadata.0.name } } volume { name = "host-id"; host_path { path = "/etc/machine-id" } } # For host_aliases } } } } # --- Datadog Monitors & Dashboards (optional) --- resource "datadog_monitor" "eks_node_cpu_high" { name = "[EKS] High CPU utilization on {{host.name}}" type = "metric alert" message = "CPU utilization is above 80% for node {{host.name}}. Investigate capacity. @webhook-devops" query = "avg(last_5m):avg:system.cpu.idle{cluster_name:${var.eks_cluster_name}} by {host} < 20" monitor_thresholds { critical = 80 warning = 70 } notify_no_data = false tags = ["eks", "cpu", "alert", "env:${var.environment}"] } resource "datadog_dashboard" "eks_overview" { title = "[Terraform] EKS Cluster Overview - ${var.eks_cluster_name} (${var.environment})" description = "Overview dashboard for EKS cluster ${var.eks_cluster_name}" layout_type = "ordered" is_read_only = false widget { definition { type = "timeseries" title = "EKS Node CPU Utilization" live_span = "1h" request { q = "avg:system.cpu.usage{cluster_name:${var.eks_cluster_name}} by {host}" display_type = "area" } } } widget { definition { type = "timeseries" title = "EKS Node Memory Utilization" live_span = "1h" request { q = "avg:system.mem.used{cluster_name:${var.eks_cluster_name}} by {host}" display_type = "area" } } } tags = ["eks", "dashboard", "terraform", "env:${var.environment}"] }

variables.tf

variable "aws_region" { description = "AWS region where your EKS cluster is deployed." type = string default = "us-east-1" } variable "eks_cluster_name" { description = "The name of your existing EKS cluster." type = string } variable "datadog_api_key" { description = "Datadog API Key (sensitive, do not commit directly)." type = string sensitive = true } variable "datadog_app_key" { description = "Datadog Application Key (sensitive, do not commit directly)." type = string sensitive = true } variable "datadog_site" { description = "Datadog site (e.g., datadoghq.com, eu.datadoghq.com)." type = string default = "datadoghq.com" } variable "environment" { description = "Environment tag for Datadog resources (e.g., 'dev', 'prod')." type = string default = "dev" } variable "eks_fargate_cluster" { description = "Set to true if your EKS cluster uses Fargate for some or all pods." type = bool default = false }

Deployment and Verification

1. Initialize Terraform

Navigate to your Terraform project directory and initialize the providers:

terraform init

2. Plan the Deployment

Review the changes Terraform plans to make. This step is crucial for understanding what resources will be created or modified.

terraform plan -var "eks_cluster_name=my-production-eks" -var "datadog_api_key=YOUR_DD_API_KEY" -var "datadog_app_key=YOUR_DD_APP_KEY"

It's recommended to pass sensitive variables via environment variables (e.g., TF_VAR_datadog_api_key) or a terraform.tfvars file that is properly excluded from version control.

3. Apply the Configuration

If the plan looks correct, apply the configuration:

terraform apply -var "eks_cluster_name=my-production-eks" -var "datadog_api_key=YOUR_DD_API_KEY" -var "datadog_app_key=YOUR_DD_APP_KEY"

Confirm by typing yes when prompted.

4. Verify in Kubernetes

Check if the Datadog Agent pods are running in your EKS cluster:

kubectl get pods -n datadog

You should see datadog-agent pods running (one per node).

5. Verify in Datadog

Log in to your Datadog account and navigate to:

• Infrastructure > Host Map: You should see your EKS nodes appearing.
• Metrics > Explorer: Query for kubernetes.*, aws.*, and system.* metrics.
• Logs > Search: Filter by source:kubernetes or service:datadog-agent.
• Monitors > Monitor Status: Your Terraform-created monitors should be listed.
• Dashboards > Dashboards List: Your EKS overview dashboard should be present.

Advanced Observability Patterns

Beyond basic cluster monitoring, Datadog and Terraform can enable sophisticated observability:

• Datadog Cluster Agent: Deploy the Cluster Agent for cluster-level metrics (e.g., Kubernetes API server, scheduler), Horizontal Pod Autoscaler (HPA) recommendations, and event collection. This offloads some work from the node-level agents.
• APM and Distributed Tracing: Instrument your applications with Datadog APM libraries to get end-to-end visibility into requests flowing through your EKS microservices.
• Custom Metrics: Collect application-specific metrics using DogStatsD, pushing them from your applications to the Datadog Agent.
• Synthetics Monitoring: Use Datadog Synthetics to proactively monitor the availability and performance of your EKS-hosted applications from various geographic locations.
• Security Monitoring: Integrate Datadog Security Monitoring for threat detection, compliance, and auditing within your EKS environment.

Troubleshooting Common Issues

• Datadog Agent Pods Not Running:
  • Check kubectl describe pod datadog-agent-xyz -n datadog for events and errors.
  • Inspect logs: kubectl logs datadog-agent-xyz -n datadog.
  • Verify resource limits if pods are in CrashLoopBackOff.
• Missing Metrics in Datadog:
  • Ensure DD_API_KEY is correctly set in the Kubernetes Secret and referenced by the DaemonSet.
  • Check if the IAM Role attached via IRSA has the necessary permissions (e.g., CloudWatch read access).
  • Verify network connectivity from EKS nodes to Datadog endpoints.
  • Check Datadog Agent status page: kubectl exec -it datadog-agent-xyz -n datadog -- agent status.
• IRSA Configuration Issues:
  • Confirm your EKS cluster has an OIDC provider associated.
  • Double-check the assume_role_policy document for the correct OIDC provider URL and service account name.
  • Ensure the Kubernetes Service Account has the eks.amazonaws.com/role-arn annotation pointing to the correct IAM Role ARN.

Conclusion

Automating AWS EKS observability with Terraform and Datadog provides a powerful, scalable, and consistent approach to understanding the health and performance of your containerized applications. By codifying your monitoring infrastructure, you gain the benefits of version control, repeatability, and efficient management, freeing your teams to focus on innovation rather than manual configurations. Embracing this strategy is a critical step towards achieving true DevOps maturity and operational excellence in your cloud-native journey.