Implementing Advanced Datadog Observability for AWS EKS Workloads with Terraform

In today's dynamic cloud-native landscape, achieving comprehensive observability for Kubernetes workloads running on AWS EKS is paramount for maintaining application performance, ensuring reliability, and optimizing resource utilization. This guide provides a detailed, technical walkthrough on leveraging Datadog, a leading monitoring and analytics platform, to gain deep insights into your EKS environments, all orchestrated efficiently and repeatably using Terraform for Infrastructure as Code (IaC).

By the end of this guide, you will understand how to deploy and configure the Datadog Agent, integrate essential services, and harness advanced Datadog features to monitor your EKS clusters and applications with precision and scale.

Architecture Pro-Tip:

Always standardize your observability stack deployment using Infrastructure as Code (IaC). For EKS and Datadog, this means managing IAM roles, Kubernetes service accounts, RBAC, and the Datadog Agent Helm chart configuration entirely via Terraform. This approach ensures consistency, auditability, and facilitates rapid recovery or scaling, making your monitoring infrastructure as robust and agile as your applications.

Why Datadog for AWS EKS?

Datadog offers a unified platform that brings together metrics, logs, traces, and synthetic monitoring, providing a holistic view of your EKS clusters and the applications running within them. Its native integrations with AWS services and Kubernetes make it an ideal choice for complex cloud-native environments.

Comprehensive Visibility: Monitor node health, pod performance, container metrics, and application-level traces.
Troubleshooting Efficiency: Correlate logs and traces with infrastructure metrics for faster root cause analysis.
Scalability: Designed to handle the dynamic and ephemeral nature of Kubernetes workloads.
Security Monitoring: Identify and respond to security threats within your EKS environment.

Prerequisites

Before you begin, ensure you have the following in place:

An active AWS account with administrative privileges.
An existing AWS EKS cluster.
A Datadog account with API and Application keys.
Terraform (v1.0+) installed on your local machine.
kubectl configured to connect to your EKS cluster.
AWS CLI configured with appropriate credentials.

Core Components for Datadog on EKS

Datadog's observability for EKS relies on several key components:

The Datadog Agent

The Datadog Agent is an open-source software that runs on your hosts (EKS nodes) and collects events and metrics. It’s deployed as a DaemonSet in Kubernetes, ensuring an instance runs on every node.

The Datadog Cluster Agent

Deployed as a single replica, the Cluster Agent performs cluster-wide tasks, reduces resource consumption by the node Agents, and centralizes certain operations like admission control and external metrics for HPA.

IAM Roles for Service Accounts (IRSA)

IRSA allows Kubernetes service accounts to assume IAM roles, providing fine-grained permissions to pods. This is crucial for securely granting the Datadog Agent access to AWS services (e.g., CloudWatch, EC2 metadata) without exposing AWS credentials directly to pods or nodes.

Kubernetes RBAC

Role-Based Access Control (RBAC) in Kubernetes dictates what actions the Datadog Agent service account can perform within the cluster, such as reading pod metadata or listing nodes.

Step-by-Step Implementation with Terraform

1. Configure Datadog API and Application Keys

First, ensure you have your Datadog API and Application keys. These will be used by Terraform to configure the Datadog Agent. It's best practice to manage these sensitive values using a secure secrets manager like AWS Secrets Manager or environment variables when running Terraform. For this guide, we'll assume they are provided as Terraform variables.

2. Set up IAM Role for Datadog Agent Service Account (IRSA)

The Datadog Agent requires permissions to collect metadata and metrics from AWS services. We'll create an IAM role and associate it with a Kubernetes service account using IRSA. This Terraform configuration assumes your EKS cluster already has an OIDC provider.

Create a file named iam.tf:

resource "aws_iam_policy" "datadog_agent_policy" {
  name        = "datadog-agent-policy-eks-${var.cluster_name}"
  description = "IAM policy for Datadog Agent on EKS"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "ec2:DescribeInstances",
          "ec2:DescribeTags",
          "ec2:DescribeVpcs",
          "ec2:DescribeSubnets",
          "ec2:DescribeSecurityGroups",
          "ec2:DescribeNatGateways",
          "autoscaling:DescribeAutoScalingGroups",
          "ecs:DescribeClusters",
          "ecs:ListClusters",
          "ecs:DescribeContainerInstances",
          "ecs:ListContainerInstances",
          "ecs:DescribeServices",
          "ecs:ListServices",
          "ecs:ListTasks",
          "ecs:DescribeTasks",
          "logs:DescribeLogGroups",
          "logs:GetLogEvents",
          "logs:FilterLogEvents",
          "tag:GetResources",
          "xray:PutTraceSegments",
          "xray:PutTelemetryRecords",
          "xray:GetSamplingRules",
          "xray:GetSamplingTargets",
          "xray:GetSamplingStatisticSummaries"
        ]
        Effect   = "Allow"
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_role" "datadog_agent_role" {
  name                   = "datadog-agent-role-eks-${var.cluster_name}"
  assume_role_policy     = data.aws_iam_policy_document.datadog_agent_assume_role_policy.json
  managed_policy_arns    = [aws_iam_policy.datadog_agent_policy.arn]
}

data "aws_iam_policy_document" "datadog_agent_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    effect  = "Allow"
    principals {
      type        = "Federated"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${replace(var.oidc_provider_arn, "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/", "")}"]
    }
    condition {
      test     = "StringEquals"
      variable = "${replace(var.oidc_provider_arn, "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/", "")}:sub"
      values   = ["system:serviceaccount:datadog:datadog-agent"]
    }
  }
}

data "aws_caller_identity" "current" {}
    

3. Deploy Datadog Agent with Helm via Terraform

Now, we'll use Terraform's Helm provider to deploy the Datadog Agent. This will involve configuring the Kubernetes provider to interact with your EKS cluster and then using the helm_release resource.

Create a file named main.tf:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

data "aws_eks_cluster" "cluster" {
  name = var.cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = var.cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}

resource "kubernetes_namespace" "datadog" {
  metadata {
    name = "datadog"
  }
}

resource "helm_release" "datadog_agent" {
  depends_on = [
    aws_iam_role.datadog_agent_role,
    kubernetes_namespace.datadog
  ]

  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = kubernetes_namespace.datadog.metadata[0].name
  version    = "2.37.0" # Use a recent stable version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "datadog.site"
    value = var.datadog_site # e.g., "datadoghq.com" or "eu.datadoghq.com"
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "clusterChecksRunner.enabled"
    value = "true"
  }

  set {
    name  = "agents.enabled"
    value = "true"
  }

  set {
    name  = "clusterAgent.metricsProvider.enabled"
    value = "true"
  }

  set {
    name  = "datadog.kubelet.host"
    value = "%%NODE_IP%%" # Required for some metric collection
  }

  set {
    name  = "datadog.tags[0]"
    value = "env:${var.environment}"
  }
  set {
    name  = "datadog.tags[1]"
    value = "cluster:${var.cluster_name}"
  }

  # Enable APM and Distributed Tracing
  set {
    name  = "datadog.apm.enabled"
    value = "true"
  }
  set {
    name  = "datadog.apm.hostPortEnabled"
    value = "true"
  }
  set {
    name  = "datadog.apm.env[0].name"
    value = "DD_APM_ENABLED"
  }
  set {
    name  = "datadog.apm.env[0].value"
    value = "true"
  }

  # Enable Log Collection
  set {
    name  = "datadog.logs.enabled"
    value = "true"
  }
  set {
    name  = "datadog.logs.containerCollectAll"
    value = "true"
  }

  # Enable Network Performance Monitoring (NPM)
  set {
    name  = "datadog.networkMonitoring.enabled"
    value = "true"
  }

  # Enable Live Process Monitoring
  set {
    name  = "datadog.processAgent.enabled"
    value = "true"
  }
  set {
    name  = "datadog.processAgent.processCollection"
    value = "true"
  }

  # Configure IRSA for the Datadog Agent
  set {
    name  = "serviceAccount.create"
    value = "true" # Let Helm create the service account
  }
  set {
    name  = "serviceAccount.name"
    value = "datadog-agent"
  }
  set {
    name  = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
    value = aws_iam_role.datadog_agent_role.arn
  }

  # Configure Cluster Agent for IRSA
  set {
    name  = "clusterAgent.serviceAccount.create"
    value = "true"
  }
  set {
    name  = "clusterAgent.serviceAccount.name"
    value = "datadog-cluster-agent"
  }
  set {
    name  = "clusterAgent.serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
    value = aws_iam_role.datadog_agent_role.arn
  }

  # For security monitoring and Cloud Workload Security (CWS)
  set {
    name  = "datadog.securityAgent.runtime.enabled"
    value = "false" # Set to "true" for CWS
  }
  set {
    name  = "datadog.securityAgent.compliance.enabled"
    value = "false" # Set to "true" for Compliance Monitoring
  }
}
    

Create a variables.tf file:

variable "aws_region" {
  description = "AWS region for the EKS cluster."
  type        = string
}

variable "cluster_name" {
  description = "The name of the EKS cluster."
  type        = string
}

variable "oidc_provider_arn" {
  description = "The ARN of the EKS OIDC provider."
  type        = string
}

variable "datadog_api_key" {
  description = "Datadog API Key."
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key."
  type        = string
  sensitive   = true
}

variable "datadog_site" {
  description = "The Datadog site URL (e.g., 'datadoghq.com' or 'eu.datadoghq.com')."
  type        = string
  default     = "datadoghq.com"
}

variable "environment" {
  description = "The environment tag for Datadog."
  type        = string
  default     = "dev"
}
    

Finally, run Terraform:

terraform init
terraform plan
terraform apply
    

Advanced Datadog Features for EKS

Distributed Tracing (APM)

With APM enabled in the Helm chart, ensure your applications are instrumented with Datadog's tracing libraries. This allows you to visualize end-to-end request flows, identify bottlenecks, and monitor service health.

Log Management

Datadog's log collection with containerCollectAll: true will automatically gather logs from all containers. Further configuration can be done to parse and enrich logs for specific applications, enabling powerful log analytics and alerting.

Network Performance Monitoring (NPM)

NPM provides deep insights into network traffic within your EKS cluster, between services, and to external endpoints. It helps in identifying network bottlenecks, understanding service dependencies, and troubleshooting connectivity issues.

Live Process Monitoring

Gain real-time visibility into every process running across your EKS nodes and containers. This is invaluable for identifying runaway processes, resource hogs, and security anomalies.

Universal Service Monitoring (USM)

USM builds upon NPM and APM to provide comprehensive service-level visibility, mapping service dependencies and showing performance metrics from the perspective of each service.

Container Security Monitoring (CSM)

By enabling the Datadog Security Agent, you can detect and respond to security threats within your EKS workloads, including suspicious process activity, file integrity monitoring, and network anomalies.

Validation and Next Steps

Once Terraform has successfully applied the configuration, verify the deployment:

Kubernetes Pods: Run kubectl get pods -n datadog to ensure the datadog-agent DaemonSet and datadog-cluster-agent Deployment pods are running.
Datadog UI: Log into your Datadog account. Navigate to the "Infrastructure" section to see your EKS nodes and pods reporting data. Check the "Metrics Explorer," "Logs," and "APM" sections for incoming data.
Default Dashboards: Datadog automatically provides out-of-the-box dashboards for Kubernetes and EKS. Explore these to get immediate insights.

Next, consider:

Creating custom dashboards tailored to your applications' KPIs.
Setting up monitors and alerts based on critical metrics and log patterns.
Integrating with other AWS services (e.g., RDS, Lambda) using Datadog's extensive integrations.

Troubleshooting & Best Practices

Common Issues:

Pods not starting: Check kubectl describe pod <pod-name> -n datadog and kubectl logs <pod-name> -n datadog for errors, especially related to image pulling, resource limits, or incorrect API keys.
No data in Datadog: Verify the API and Application keys are correct. Ensure the IAM role for the service account has the necessary permissions. Check agent logs for connection errors to Datadog endpoints.
IRSA configuration: Double-check the OIDC provider ARN and the IAM policy document's `condition` block for exact string matches.

Best Practices:

Centralized Secrets: Always use a secrets manager for Datadog API/APP keys in production environments.
Resource Limits: Set appropriate CPU and memory limits for Datadog Agent pods to prevent resource contention on your EKS nodes.
Tagging Strategy: Implement a consistent tagging strategy across your AWS resources and Kubernetes objects. Datadog leverages these tags for powerful filtering and aggregation.
Regular Updates: Keep the Datadog Agent Helm chart updated to benefit from the latest features, performance improvements, and security patches.

Conclusion

By following this comprehensive guide, you've successfully deployed an advanced Datadog observability solution for your AWS EKS workloads using Terraform. This not only streamlines the deployment process through Infrastructure as Code but also ensures consistent, repeatable, and scalable monitoring for your critical cloud-native applications. With Datadog's rich feature set, you are now equipped to gain unparalleled insights into your EKS environments, proactively identify issues, and drive continuous improvement.

CloudStack YAML