Terraform-Managed AWS EKS Observability with Datadog and PagerDuty Alerting

In the dynamic landscape of cloud-native applications, managing and monitoring complex systems like AWS EKS (Elastic Kubernetes Service) is paramount for maintaining reliability and performance. This comprehensive guide details how to establish robust observability for your Terraform-managed AWS EKS clusters using Datadog for unified monitoring and PagerDuty for efficient incident response and alerting.

Architecture Pro-Tip:

Always deploy your observability agents, such as the Datadog Agent, as a DaemonSet within your Kubernetes cluster. This ensures that an instance of the agent runs on every node, providing comprehensive coverage for metrics, logs, and traces. Leverage Kubernetes Service Accounts (KSA) associated with IAM Roles for Service Accounts (IRSA) to securely grant the Datadog Agent the necessary AWS permissions, adhering to the principle of least privilege without exposing sensitive credentials.

Introduction to Comprehensive EKS Observability

AWS EKS offers a powerful platform for deploying and scaling containerized applications, but its inherent complexity necessitates sophisticated monitoring. A robust observability strategy provides deep insights into the health, performance, and operational state of your cluster and its workloads. By integrating Terraform for infrastructure as code (IaC), Datadog for a unified observability platform, and PagerDuty for streamlined incident management, you can achieve proactive monitoring and rapid incident resolution for your EKS environments.

Why Terraform, Datadog, and PagerDuty for EKS?

Terraform: Enables reproducible, version-controlled deployment and management of EKS clusters, observability agents, and monitoring configurations. This ensures consistency and reduces manual errors.
Datadog: Provides a single pane of glass for metrics, logs, traces, network performance, and user experience monitoring across your entire EKS ecosystem, including pods, nodes, services, and AWS infrastructure.
PagerDuty: Offers advanced incident routing, on-call scheduling, and automated escalation policies, ensuring critical alerts from Datadog reach the right team members promptly.

Prerequisites

Before diving into the configuration, ensure you have the following:

An active AWS Account with necessary permissions to create EKS clusters, IAM roles, and secrets.
Terraform CLI installed (v1.0.0 or higher).
AWS CLI installed and configured.
kubectl CLI installed and configured to interact with your EKS cluster.
A Datadog Account with API and Application keys.
A PagerDuty Account with an API key and service integration capabilities.
An existing AWS EKS cluster managed by Terraform (or the ability to create one).
Helm CLI installed (for Datadog Agent deployment via Helm).

Terraform Setup for Datadog Agent on EKS

The Datadog Agent is the cornerstone of collecting metrics, logs, and traces from your EKS cluster. Deploying it via Helm and managing its configuration with Terraform provides a robust and scalable solution.

1. Datadog API Keys as Kubernetes Secrets

For security, store your Datadog API and Application keys as Kubernetes Secrets. This example uses the kubernetes_secret resource.

2. IAM Role for Service Accounts (IRSA) for Datadog Agent

To enable the Datadog Agent to collect enhanced metrics and interact with AWS services securely (e.g., pulling EC2 tags, CloudWatch metrics), configure IRSA. This involves creating an IAM Role and associating it with the Kubernetes Service Account used by the Datadog Agent.

3. Deploying Datadog Agent with Helm via Terraform

The helm_release resource is ideal for deploying the Datadog Agent. You'll configure it to use the created secrets and IRSA.

Example Terraform Configuration: Datadog Agent & PagerDuty Integration

Here's a comprehensive Terraform configuration block demonstrating how to set up the Datadog Agent on EKS with IRSA, a basic Datadog monitor, and its integration with PagerDuty.

resource "kubernetes_secret" "datadog_api_key" {
  metadata {
    name      = "datadog-api-key"
    namespace = "datadog" # Ensure this namespace exists or is created
  }

  data = {
    "api-key" = var.datadog_api_key
    "app-key" = var.datadog_app_key
  }
  type = "Opaque"
}

resource "aws_iam_role" "datadog_agent" {
  name_prefix = "datadog-agent-irsa-"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Federated = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${replace(aws_eks_cluster.this.identity[0].oidc[0].issuer, "https://", "")}"
        }
        Action = "sts:AssumeRoleWithWebIdentity"
        Condition = {
          StringEquals = {
            "${replace(aws_eks_cluster.this.identity[0].oidc[0].issuer, "https://", "")}:sub" = "system:serviceaccount:datadog:datadog-agent"
            "${replace(aws_eks_cluster.this.identity[0].oidc[0].issuer, "https://", "")}:aud" = "sts.amazonaws.com"
          }
        }
      }
    ]
  })

  tags = {
    Env = var.environment
  }
}

resource "aws_iam_policy" "datadog_agent_policy" {
  name_prefix = "datadog-agent-policy-"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = [
          "ec2:DescribeTags",
          "ec2:DescribeInstances",
          "ec2:DescribeVolumes",
          "ec2:DescribeLocalGatewayRouteTablesVPCAssociations",
          "autoscaling:DescribeAutoScalingGroups",
          "lambda:ListFunctions",
          "lambda:ListTags",
          "logs:DescribeLogGroups",
          "logs:DescribeLogStreams",
          "logs:FilterLogEvents",
          "sts:GetServiceBearerToken",
        ]
        Resource = "*"
      },
      # Add more permissions as needed for specific Datadog integrations
    ]
  })
}

resource "aws_iam_role_policy_attachment" "datadog_agent_policy_attach" {
  role       = aws_iam_role.datadog_agent.name
  policy_arn = aws_iam_policy.datadog_agent_policy.arn
}

resource "kubernetes_service_account" "datadog_agent" {
  metadata {
    name        = "datadog-agent"
    namespace   = "datadog"
    annotations = {
      "eks.amazonaws.com/role-arn" = aws_iam_role.datadog_agent.arn
    }
  }
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  version    = "2.33.2" # Use the latest stable version

  set {
    name  = "datadog.site"
    value = "us5.datadoghq.com" # Replace with your Datadog site (e.g., us.datadoghq.com, eu.datadoghq.com)
  }

  set_sensitive {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }

  set_sensitive {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "clusterAgent.metricsProvider.enabled"
    value = "true"
  }

  set {
    name  = "agents.tolerations[0].operator"
    value = "Exists"
  }
  
  set {
    name  = "agents.tolerations[0].effect"
    value = "NoSchedule"
  }

  set {
    name  = "agents.tolerations[1].operator"
    value = "Exists"
  }

  set {
    name  = "agents.tolerations[1].effect"
    value = "NoExecute"
  }

  set {
    name  = "clusterAgent.rbac.create"
    value = "true"
  }

  set {
    name  = "datadog.createPodDisruptionBudget"
    value = "true"
  }

  set {
    name  = "datadog.dogstatsd.useHostPort"
    value = "true"
  }

  set {
    name  = "datadog.kubelet.host"
    value = "true"
  }

  # Enable IRSA for the Datadog Agent
  set {
    name  = "serviceAccount.create"
    value = "false" # We manage SA explicitly with `kubernetes_service_account`
  }
  set {
    name  = "serviceAccount.name"
    value = kubernetes_service_account.datadog_agent.metadata[0].name
  }
  set {
    name  = "clusterAgent.serviceAccount.create"
    value = "false"
  }
  set {
    name  = "clusterAgent.serviceAccount.name"
    value = kubernetes_service_account.datadog_agent.metadata[0].name
  }
  set {
    name  = "targetSystem"
    value = "linux"
  }

  # Example log collection for Kubernetes
  set {
    name  = "datadog.logs.enabled"
    value = "true"
  }
  set {
    name  = "datadog.logs.containerCollectAll"
    value = "true"
  }
  set {
    name  = "datadog.logs.autoMultiLineLogDetection"
    value = "true"
  }

  # APM configuration
  set {
    name  = "datadog.apm.enabled"
    value = "true"
  }
  set {
    name  = "datadog.apm.portEnabled"
    value = "true"
  }
}

# PagerDuty Service
resource "pagerduty_service" "eks_observability_service" {
  name        = "${var.environment}-EKS-Observability"
  description = "PagerDuty service for critical alerts from EKS Observability via Datadog"
  escalation_policy = data.pagerduty_escalation_policy.primary.id
}

# PagerDuty Service Integration for Datadog
resource "pagerduty_service_integration" "datadog_integration" {
  name    = "Datadog-Integration"
  type    = "datadog_inbound_integration"
  service = pagerduty_service.eks_observability_service.id
}

# Datadog Monitor for EKS Node CPU Utilization
resource "datadog_monitor" "high_node_cpu_utilization" {
  name    = "[${var.environment}-EKS] High Node CPU Utilization"
  type    = "metric alert"
  query   = "avg(last_5m):avg:kubernetes.cpu.usage.total{*} by {host} > 80"
  message = <}}
  High CPU utilization detected on EKS node {{host.name}} ({{value}}%).
  This could indicate a bottleneck or over-provisioned workload.
  Please investigate node and pod resource usage.
  EOT
  
  tags = [
    "environment:${var.environment}",
    "service:eks-observability",
    "severity:high",
    "team:devops",
  ]

  options {
    thresholds = {
      warning  = 70
      critical = 80
    }
    
    notify_no_data = false
    renotify_interval = 60 # minutes
    
    # Optional: Configure automatic resolution
    # require_full_object = true 
    # new_group_delay = 300 # seconds
  }
}

# Variables (to be defined in variables.tf)
/*
variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "environment" {
  description = "Deployment environment (e.g., dev, staging, prod)"
  type        = string
}

data "aws_caller_identity" "current" {}

data "aws_eks_cluster" "this" {
  name = var.eks_cluster_name # Replace with your EKS cluster name
}

data "pagerduty_escalation_policy" "primary" {
  name = "Primary On-Call" # Replace with your escalation policy name
}
*/
    

Important Notes:

Replace var.datadog_api_key, var.datadog_app_key, var.environment, and var.eks_cluster_name with your actual values or Terraform variables.
The @pagerduty-{{<your_pagerduty_integration_name_from_datadog>}} in the monitor message refers to the PagerDuty integration you configure within Datadog's UI, under "Integrations" -> "PagerDuty". The name needs to match exactly.
Ensure the datadog Kubernetes namespace exists or create it: kubectl create namespace datadog.
The IAM policy for the Datadog Agent should follow the principle of least privilege. Adjust the actions based on the specific Datadog integrations you plan to enable.

Terraform Setup for Datadog Monitors with PagerDuty

Defining your Datadog monitors as Terraform resources (`datadog_monitor`) allows for version control, automated deployment, and consistency. The key is to include the PagerDuty notification within the monitor's message.

1. PagerDuty Service and Integration

First, define your PagerDuty service and a Datadog-specific integration using pagerduty_service and pagerduty_service_integration.

The data.pagerduty_escalation_policy resource fetches an existing escalation policy by name, which is then assigned to the service.

2. Datadog Monitors

The datadog_monitor resource allows you to define various alert types (metric, anomaly, outlier, forecast). Crucially, the message field is where you specify the PagerDuty integration token for notifications. This token is configured in the Datadog UI under "Integrations" -> "PagerDuty".

Testing and Validation

After applying your Terraform configuration, it's essential to validate the setup:

Verify Datadog Agents: Run kubectl get pods -n datadog to ensure Datadog agents and cluster agents are running. Check their logs for any errors.
Check Datadog UI: Navigate to your Datadog account. Verify that hosts, metrics, logs, and traces from your EKS cluster are flowing in. Confirm the deployed monitors are visible.
Trigger a Test Alert: To test the PagerDuty integration, you can deliberately trigger an alert. For instance, scale down a critical deployment to zero pods, or introduce a resource-intensive workload on a node to exceed the CPU threshold. Observe if an incident is created in PagerDuty and if the correct escalation policy is followed.
Review PagerDuty: Confirm that the new PagerDuty service and integration are correctly configured and visible in your PagerDuty dashboard.

Troubleshooting and Best Practices

Common Troubleshooting Steps:

Datadog API/APP Keys: Double-check that your keys are correct and the Kubernetes secret is properly mounted.
IAM Permissions: Ensure the IRSA role attached to the Datadog Agent's service account has all necessary AWS permissions. Use CloudTrail to debug `Access Denied` errors.
Network Policies: If you have strict network policies in your EKS cluster, ensure they allow egress from Datadog Agent pods to Datadog's ingest endpoints.
Datadog Integration Name in Monitor: The @pagerduty-{{<your_pagerduty_integration_name_from_datadog>}} in the monitor message must exactly match the integration name configured in Datadog's PagerDuty integration settings.
Helm Chart Version: Always refer to the official Datadog Helm chart documentation for the latest versions and configuration options.

Best Practices:

Tag Everything: Utilize Datadog's robust tagging capabilities (e.g., environment, service, team) for better filtering, dashboard organization, and monitor scoping.
Granular Monitors: Beyond basic metrics, create monitors for application-specific KPIs, log patterns, and trace anomalies.
Alert Fatigue Management: Configure PagerDuty escalation policies carefully. Use Datadog's composite monitors and anomaly detection to reduce alert noise.
Cost Optimization: Monitor Datadog usage (hosts, custom metrics, logs) and optimize configurations to manage costs effectively.
Version Control Your Observability: Treat your Datadog monitors and PagerDuty services as code, checked into your Git repository alongside your infrastructure.

Conclusion

Achieving comprehensive observability for AWS EKS is no longer optional but a critical requirement for modern cloud-native operations. By leveraging the power of Terraform for declarative infrastructure management, Datadog for unified monitoring, and PagerDuty for intelligent incident response, you empower your DevOps and SRE teams with the tools needed to maintain high availability, optimize performance, and swiftly resolve issues in your Kubernetes environments. This integrated approach ensures that your EKS clusters are not only well-managed but also proactively monitored, leading to a more resilient and efficient operational posture.

CloudStack YAML