Terraform AWS EKS Cluster Observability with Datadog Log and Metric Integration

Architecture Pro-Tip:

Implement observability from day one, not as an afterthought. Integrating Datadog with your AWS EKS cluster via Terraform ensures that monitoring, logging, and tracing are intrinsically linked to your infrastructure deployment, promoting consistency, auditability, and proactive issue detection across your entire Kubernetes environment. This approach significantly reduces operational overhead and accelerates troubleshooting cycles.

Unlocking Full Observability for AWS EKS Clusters with Terraform and Datadog

In today's dynamic cloud-native landscape, running Kubernetes clusters on AWS EKS (Elastic Kubernetes Service) demands robust observability. To maintain optimal performance, security, and reliability, DevOps teams need comprehensive visibility into every layer of their EKS infrastructure, from control plane logs to application metrics. This guide provides a detailed, technical walkthrough on how to achieve end-to-end observability for your AWS EKS clusters using Terraform for infrastructure as code (IaC) and Datadog for unified monitoring, logging, and metrics integration.

We will cover the integration of Datadog's powerful monitoring capabilities for collecting metrics and logs directly from your EKS cluster nodes, pods, and the EKS control plane itself, all managed and automated through Terraform.

Why Datadog for AWS EKS Observability?

Datadog offers a unified platform that simplifies the complexity of monitoring distributed systems like Kubernetes. Its key advantages for EKS include:

Unified View: Consolidates metrics, logs, and traces from EKS nodes, pods, applications, and AWS services into a single pane of glass.
Deep Kubernetes Integration: Provides out-of-the-box dashboards and monitors for EKS health, resource utilization, and pod performance.
Log Management: Collects, processes, and analyzes logs from all components within your EKS cluster, enhancing troubleshooting and compliance.
APM & Distributed Tracing: Offers end-to-end visibility into application performance across microservices, crucial for modern EKS workloads.
Alerting & Automation: Configurable alerts based on anomalies and thresholds, with integrations for various notification channels.

Prerequisites

Before you begin, ensure you have the following:

An AWS Account with administrative access.
Terraform CLI (v1.0.0+) installed.
AWS CLI configured with appropriate credentials.
Kubectl CLI installed and configured to interact with your EKS cluster.
A Datadog Account with an API key and Application key.
An existing AWS EKS Cluster deployed via Terraform. If not, you'll need to deploy one first.

Core Components of Datadog Integration for EKS

To achieve full observability, we'll integrate several Datadog components:

Datadog Agent: Deployed as a DaemonSet on each EKS worker node, it collects infrastructure metrics, application logs, and traces.
Datadog Cluster Agent: Deployed as a Deployment, it provides cluster-level visibility, handles leader election, and reduces API calls to the Kubernetes API server.
AWS Integration: Connects your Datadog account directly to your AWS account to pull metrics and logs from various AWS services, including EKS control plane logs (via CloudWatch).

Terraform Configuration for Datadog Integration

We will structure our Terraform code to manage the Datadog Agent deployment within the EKS cluster and configure AWS integration.

1. Configure Terraform Providers

Ensure your Terraform configuration includes the necessary providers: aws, kubernetes, helm, and optionally datadog.

provider "aws" {
  region = var.aws_region
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args        = ["eks", "get-token", "--cluster-name", data.aws_eks_cluster.this.name, "--region", var.aws_region]
  }
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.this.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args        = ["eks", "get-token", "--cluster-name", data.aws_eks_cluster.this.name, "--region", var.aws_region]
    }
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
  api_url = var.datadog_site == "us" ? "https://api.datadoghq.com/" : "https://api.${var.datadog_site}.datadoghq.com/"
}

# Data source to fetch existing EKS cluster details
data "aws_eks_cluster" "this" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "this" {
  name = var.eks_cluster_name
}
    

2. Deploy Datadog Agent using Helm with Terraform

The Datadog Agent is best deployed via its official Helm chart. This Terraform configuration deploys both the Datadog Agent (as a DaemonSet) and the Cluster Agent (as a Deployment) to your EKS cluster.

You'll need your Datadog API Key and Application Key. It's recommended to manage these as Terraform variables and potentially fetch them from a secure secret store (e.g., AWS Secrets Manager).

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "datadog.site"
    value = var.datadog_site # e.g., "us", "eu", "us3", "us5", "ap1"
  }

  values = [
    yamlencode({
      # Global agent settings
      clusterName = var.eks_cluster_name
      kubelet = {
        host = {
          # Automatically detect kubelet host for EKS
          # If using custom EKS setup or Fargate, adjust accordingly
          detectIP = true
        }
      }
      
      # Enable Datadog Cluster Agent
      clusterAgent = {
        enabled = true
        metricsProvider = {
          enabled = true
          service = {
            type = "ClusterIP"
          }
        }
      }

      # EKS specific integration
      datadog = {
        kubeEKS = {
          # Recommended for EKS to collect control plane metrics
          # Make sure to set aws_integration to true for logs/metrics from CloudWatch
          enabled = true
        }
        # Enable logs collection
        logs = {
          enabled = true
          # Recommended for EKS for log collection
          containerCollectAll = true
          # Auto-discovery for logs
          autoDiscovery = {
            enabled = true
          }
          # Optional: additional log processing rules
          processingRules = [
            {
              type = "exclude_at_match"
              name = "exclude_datadog_logs"
              pattern = "container_name:(datadog-agent|datadog-cluster-agent)"
            }
          ]
        }
        # Enable APM and Process monitoring
        apm = {
          enabled = true
        }
        processAgent = {
          enabled = true
          # For container-level process visibility
          containerCollection = true
        }
        # Enable network performance monitoring
        networkMonitoring = {
          enabled = true
        }
      }

      # RBAC for Datadog Agent
      rbac = {
        create = true
        # Required for Cluster Agent Metrics Provider to access Kubernetes metrics
        metricsServer = {
          create = true
        }
      }

      # Service account for Datadog Agent (if custom)
      serviceAccount = {
        create = true
        name = "datadog-agent"
      }
    })
  ]
}
    

This configuration enables:

Cluster-wide monitoring: Metrics from nodes, pods, deployments, and services.
Log collection: Automatically collects logs from all containers.
APM and Process Monitoring: Essential for application-level insights.
Network Performance Monitoring: Visibility into network traffic within the cluster.
EKS-specific integrations: Optimized collection for AWS EKS.

3. Enable EKS Control Plane Logs to CloudWatch

To get full visibility into your EKS cluster's health, it's critical to capture control plane logs (API Server, Controller Manager, Scheduler, etc.). AWS allows you to publish these logs directly to CloudWatch Logs.

# Configure a CloudWatch Log Group for EKS Control Plane logs
resource "aws_cloudwatch_log_group" "eks_control_plane_logs" {
  name              = "/aws/eks/${var.eks_cluster_name}/cluster"
  retention_in_days = 90 # Adjust as needed
  tags = {
    Environment = var.environment
    Service     = "EKS"
  }
}

# Update your EKS cluster to enable control plane logs
resource "aws_eks_cluster" "this" {
  # ... existing EKS cluster configuration ...
  name     = var.eks_cluster_name
  role_arn = var.eks_cluster_iam_role_arn
  vpc_config {
    subnet_ids         = var.eks_subnet_ids
    security_group_ids = var.eks_security_group_ids
  }
  
  enabled_cluster_log_types = ["api", "audit", "authenticator", "controllerManager", "scheduler"]

  version = var.eks_cluster_version # e.g., "1.28"

  depends_on = [aws_cloudwatch_log_group.eks_control_plane_logs]
  # ... other EKS cluster settings ...
}
    

Note: The aws_eks_cluster resource shown above is a simplified representation. You would integrate the enabled_cluster_log_types into your existing EKS cluster definition.

4. Integrate AWS Account with Datadog for CloudWatch Logs and Metrics

To pull metrics and logs from AWS services (including EKS control plane logs from CloudWatch) into Datadog, you need to set up the AWS integration. This involves creating an IAM Role that Datadog can assume, and then configuring the integration within Datadog. The datadog_integration_aws resource allows you to manage this via Terraform.

# Create IAM Role for Datadog integration
resource "aws_iam_role" "datadog_integration" {
  name = "DatadogIntegrationRole-${var.environment}"

  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect = "Allow",
        Principal = {
          AWS = "arn:aws:iam::464622532012:root" # Datadog AWS integration account ID
        },
        Action = "sts:AssumeRole",
        Condition = {
          StringEquals = {
            "sts:ExternalId" = var.datadog_external_id # Use a unique external ID for security
          }
        }
      }
    ]
  })
}

# Attach policies for Datadog to collect metrics and logs
# Datadog AWS Integration Policy (Read-only access for core metrics/logs)
resource "aws_iam_policy" "datadog_integration_policy" {
  name        = "DatadogIntegrationPolicy-${var.environment}"
  description = "Allows Datadog to read metrics and logs from AWS services"
  policy      = file("datadog-iam-policy.json") # Reference an external JSON policy file
}

# Attach the policy to the role
resource "aws_iam_role_policy_attachment" "datadog_policy_attach" {
  role       = aws_iam_role.datadog_integration.name
  policy_arn = aws_iam_policy.datadog_integration_policy.arn
}

# Optionally, add a separate policy for CloudWatch Logs specifically for EKS
resource "aws_iam_policy" "datadog_eks_logs_policy" {
  name        = "DatadogEKSLogsPolicy-${var.environment}"
  description = "Allows Datadog to read EKS control plane logs from CloudWatch"
  policy      = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect   = "Allow",
        Action   = [
          "logs:DescribeLogGroups",
          "logs:FilterLogEvents",
          "logs:GetLogEvents",
          "logs:GetLogGroupFields",
          "logs:GetQueryResults",
          "logs:StartQuery",
          "logs:StopQuery"
        ],
        Resource = "${aws_cloudwatch_log_group.eks_control_plane_logs.arn}:*"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "datadog_eks_logs_attach" {
  role       = aws_iam_role.datadog_integration.name
  policy_arn = aws_iam_policy.datadog_eks_logs_policy.arn
}


# Configure Datadog AWS integration
resource "datadog_integration_aws" "this" {
  account_id = data.aws_caller_identity.current.account_id
  role_name  = aws_iam_role.datadog_integration.name
  host_tags  = ["environment:${var.environment}", "source:terraform"]

  # Enable logs collection from specific CloudWatch log groups
  logs {
    enabled = true
    # Filter only EKS control plane logs to avoid ingesting all CloudWatch logs
    lambda_arn = "" # Leave empty if not using Datadog's Lambda forwarder
    log_group_name_filter = [
      aws_cloudwatch_log_group.eks_control_plane_logs.name
    ]
    # Optional: exclude other log groups if you have a broader filter
    # log_group_name_filter_blacklist = []
  }

  # Enable host metric collection
  host_tags_regex = ["^eks-"] # Example to tag hosts starting with 'eks-'
  # Enable specific service collection (e.g., EKS, EC2, EBS, ELB etc.)
  # This list can be extensive, start with critical services.
  filter_tags_diff = false # Use host_tags_regex for filtering
  
  # Services to collect metrics from
  excluded_regions = ["ap-northeast-3"] # Example: exclude a region
  
  # Explicitly enable EKS service in Datadog integration
  # This typically happens automatically when the EKS Cluster Agent sends data.
  # However, for broader AWS EKS service metrics, you can list it here if needed.
  # Otherwise, Datadog typically collects based on attached policies and tags.
}

# Example datadog-iam-policy.json (basic read-only access for Datadog)
# {
#   "Version": "2012-10-17",
#   "Statement": [
#     {
#       "Action": [
#         "tag:GetResources",
#         "ec2:Describe*",
#         "autoscaling:Describe*",
#         "cloudwatch:Describe*",
#         "cloudwatch:Get*",
#         "cloudwatch:List*",
#         "rds:Describe*",
#         "elasticloadbalancing:Describe*",
#         "route53:List*",
#         "s3:GetBucketTagging",
#         "s3:ListAllMyBuckets",
#         "s3:GetBucketLocation",
#         "lambda:List*",
#         "lambda:Get*",
#         "apigateway:GET",
#         "dynamodb:List*",
#         "dynamodb:Describe*",
#         "sqs:ListQueues",
#         "sqs:GetQueueAttributes",
#         "sns:ListTopics",
#         "sns:GetTopicAttributes",
#         "eks:DescribeCluster",
#         "eks:ListClusters",
#         "kafka:List*",
#         "kafka:Describe*",
#         "firehose:List*",
#         "firehose:Describe*",
#         "wafv2:List*",
#         "wafv2:Get*",
#         "events:List*",
#         "events:Describe*",
#         "states:List*",
#         "states:Describe*",
#         "logs:Describe*",
#         "logs:Get*",
#         "logs:FilterLogEvents"
#       ],
#       "Effect": "Allow",
#       "Resource": "*"
#     }
#   ]
# }
    

Important Security Note: Always use a strong, unique external_id for your Datadog AWS integration. This enhances security by preventing the "confused deputy" problem. The provided datadog-iam-policy.json is a starting point; review and restrict permissions according to the principle of least privilege for your specific needs.

Verifying Datadog Integration

After applying your Terraform configuration, it's crucial to verify that data is flowing correctly into Datadog.

Datadog Infrastructure List: Navigate to "Infrastructure" -> "Hosts" or "Containers" in Datadog. You should see your EKS nodes and pods appearing.
Kubernetes Dashboard: Check the "Kubernetes" dashboard in Datadog for cluster-level metrics.
Log Explorer: Go to "Logs" -> "Log Explorer". Filter by source:kubernetes or source:aws.eks. You should see logs from your pods, containers, and EKS control plane.
Integrations Page: In Datadog, go to "Integrations" -> "AWS". Ensure your integrated AWS account shows as "Connected".

Advanced Observability and Best Practices

Once your basic integration is complete, consider these advanced steps and best practices:

APM and Tracing: Instrument your applications with Datadog APM libraries to get distributed tracing and deeper code-level insights.
Synthetic Monitoring: Simulate user journeys to proactively detect availability and performance issues from different geographic locations.
Security Monitoring: Leverage Datadog Cloud Security Posture Management (CSPM) and Cloud Workload Security (CWS) for threat detection and compliance.
Custom Metrics: Push custom metrics from your applications using Datadog's client libraries for highly specific monitoring needs.
Resource Limits & Requests: Properly configure resource limits and requests for your pods in Kubernetes to optimize performance and prevent resource starvation, which Datadog can help identify.
Tagging Strategy: Implement a consistent tagging strategy across your AWS resources and Kubernetes manifests. Datadog leverages tags extensively for filtering, grouping, and correlation.

Troubleshooting and Common Issues

Encountering issues during setup is common. Here are some troubleshooting tips:

Datadog Agent Not Reporting:
- Check Datadog Agent pod logs (kubectl logs -f -l app.kubernetes.io/name=datadog).
- Ensure datadog.apiKey is correct in the Helm values.
- Verify the Datadog Agent DaemonSet is running on all nodes (kubectl get ds -n datadog).
- Check network connectivity from EKS nodes to Datadog endpoints.
Missing EKS Control Plane Logs:
- Confirm EKS cluster log types are enabled in AWS (check aws_eks_cluster resource or AWS console).
- Verify the IAM role for Datadog has appropriate permissions for CloudWatch Logs (logs:FilterLogEvents, logs:GetLogEvents on the correct log group ARN).
- Ensure the datadog_integration_aws resource is correctly configured with logs.enabled = true and log_group_name_filter.
Terraform Apply Issues:
- Verify your AWS and Kubernetes provider configurations are correct and authenticated.
- Check for any typos in Helm chart values or resource names.

Conclusion

By leveraging Terraform for IaC and Datadog for comprehensive monitoring, you can establish a robust, automated, and scalable observability solution for your AWS EKS clusters. This approach not only streamlines deployment but also ensures that your critical Kubernetes workloads are continuously monitored, providing the insights necessary to maintain high performance, diagnose issues rapidly, and support the agile demands of modern cloud-native applications. Embracing Infrastructure as Code for observability is a fundamental step towards operational excellence in your DevOps practice.

CloudStack YAML