Terraform Provisioning for AWS EKS Observability with Datadog and PagerDuty

In today's dynamic cloud-native landscape, ensuring robust observability for Kubernetes clusters is paramount. AWS Elastic Kubernetes Service (EKS) provides a powerful platform for deploying containerized applications, but effective monitoring and incident response require specialized tools. This comprehensive guide details how to leverage Terraform Infrastructure as Code (IaC) to provision and configure a world-class observability stack for AWS EKS using Datadog for monitoring, logging, and tracing, and PagerDuty for streamlined incident management. By automating this setup, organizations can achieve consistent, scalable, and resilient operational visibility.

Architecture Pro-Tip:

Always encapsulate your observability stack configuration within dedicated Terraform modules. This promotes reusability, ensures consistency across multiple EKS clusters or environments, and simplifies maintenance. Parameterize sensitive information like API keys using Terraform variables and secure them with tools like AWS Secrets Manager or HashiCorp Vault.

Understanding the Observability Challenge in AWS EKS

Kubernetes, while incredibly powerful, introduces significant complexity to monitoring. Applications run in ephemeral pods, services scale dynamically, and issues can arise at various layers: node, kubelet, pod, container, or application. A holistic observability strategy for EKS must encompass:

Metrics: CPU, memory, network, disk I/O at node, pod, and container levels, along with application-specific metrics.
Logs: Centralized collection and analysis of logs from all applications, Kubernetes components, and underlying infrastructure.
Traces: Distributed tracing to understand request flow across microservices and pinpoint latency issues.
Events: Kubernetes events for insights into cluster state changes, scheduling, and resource issues.

Datadog excels at consolidating these data types into a unified platform, offering powerful visualizations, alerting, and AI-driven insights. When critical issues are detected, PagerDuty steps in to provide reliable incident notification, on-call scheduling, and escalation policies, ensuring prompt resolution and minimizing downtime.

Prerequisites

Before proceeding, ensure you have the following:

An active AWS Account with necessary permissions to create EKS clusters, IAM roles, and policies.
Terraform CLI installed (version 1.0+ recommended).
An existing AWS EKS Cluster. This guide assumes your EKS cluster is already provisioned.
A Datadog Account with API and APP keys.
A PagerDuty Account with necessary API access to create services and integration keys.
kubectl configured to interact with your EKS cluster.

Core Terraform Setup for AWS EKS Integration

Your Terraform configuration should start by defining the necessary providers. We'll need the AWS provider for IAM roles, the Kubernetes provider to deploy Datadog agents, and the Datadog and PagerDuty providers for their respective configurations.

variable "aws_region" {
  description = "AWS region for resource deployment"
  type        = string
  default     = "us-east-1"
}

variable "cluster_name" {
  description = "Name of the EKS cluster"
  type        = string
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog APP Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_auth_token" {
  description = "PagerDuty Auth Token"
  type        = string
  sensitive   = true
}

provider "aws" {
  region = var.aws_region
}

data "aws_eks_cluster" "cluster" {
  name = var.cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = var.cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_auth_token
}

Explanation:

We declare variables for sensitive keys and common configurations. Remember to manage these securely (e.g., using Terraform Cloud, AWS Secrets Manager, or environment variables).
The AWS provider handles interaction with AWS services.
The Kubernetes provider fetches cluster details and authentication from AWS EKS to deploy Kubernetes resources.
The Datadog and PagerDuty providers are configured with their respective API/APP keys and tokens.

Datadog Integration with AWS EKS via Terraform

Datadog's integration with EKS involves two main parts: deploying the Datadog Agent within your cluster to collect metrics, logs, and traces from pods and nodes, and configuring the Datadog AWS integration to pull cloud service metrics and events directly from AWS.

Deploying the Datadog Agent as a DaemonSet

The Datadog Agent runs as a DaemonSet on your EKS worker nodes, ensuring an agent is present on every node to collect host-level metrics, container metrics, and application logs. We'll use the Kubernetes provider to deploy the agent, referencing Datadog's official Helm chart values for common configurations.

Configuring Datadog AWS Integration

To get full visibility into your AWS infrastructure (EC2, RDS, Lambda, etc.) and EKS control plane metrics, Datadog needs read-only access to your AWS account. This is typically achieved by creating an IAM role in your AWS account and granting Datadog permission to assume it.

Setting Up Datadog Agent and AWS Integration

# Create a Kubernetes Namespace for Datadog
resource "kubernetes_namespace" "datadog" {
  metadata {
    name = "datadog"
  }
}

# Deploy Datadog Agent (using a simplified manifest or referencing Helm chart values)
# For a full production deployment, consider using the official Datadog Helm chart
# with `helm_release` resource, or generate YAML from Helm and apply.
# Below is a simplified example for clarity.
resource "kubernetes_deployment" "datadog_agent" {
  metadata {
    name      = "datadog-agent"
    namespace = kubernetes_namespace.datadog.metadata[0].name
    labels = {
      app = "datadog-agent"
    }
  }

  spec {
    replicas = 1 # DaemonSet is more common, but Deployment works for single node test
    selector {
      match_labels = {
        app = "datadog-agent"
      }
    }
    template {
      metadata {
        labels = {
          app = "datadog-agent"
        }
      }
      spec {
        service_account_name = kubernetes_service_account.datadog_sa.metadata[0].name
        container {
          name  = "agent"
          image = "gcr.io/datadog-prod/agent:7.50.0" # Use a specific version
          env {
            name  = "DD_API_KEY"
            value = var.datadog_api_key
          }
          env {
            name  = "DD_APP_KEY"
            value = var.datadog_app_key
          }
          env {
            name  = "DD_KUBERNETES_COLLECT_EVENTS"
            value = "true"
          }
          env {
            name  = "DD_LOGS_ENABLED"
            value = "true"
          }
          env {
            name  = "DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL"
            value = "true"
          }
          port {
            container_port = 8125
            name           = "dogstatsd"
            protocol       = "UDP"
          }
          # Add hostPath volumes for full node visibility (e.g., /var/run/docker.sock, /sys/fs/cgroup)
        }
      }
    }
  }
}

resource "kubernetes_service_account" "datadog_sa" {
  metadata {
    name      = "datadog-sa"
    namespace = kubernetes_namespace.datadog.metadata[0].name
  }
}

resource "kubernetes_cluster_role" "datadog_cluster_role" {
  metadata {
    name = "datadog-cluster-role"
  }
  rule {
    api_groups = [""]
    resources  = ["pods", "nodes", "events", "services", "namespaces", "endpoints"]
    verbs      = ["get", "list", "watch"]
  }
  rule {
    api_groups = ["apps"]
    resources  = ["deployments", "replicasets", "daemonsets", "statefulsets"]
    verbs      = ["get", "list", "watch"]
  }
  # Add more rules as per Datadog's official documentation for full capabilities
}

resource "kubernetes_cluster_role_binding" "datadog_cluster_role_binding" {
  metadata {
    name = "datadog-cluster-role-binding"
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = kubernetes_cluster_role.datadog_cluster_role.metadata[0].name
  }
  subject {
    kind      = "ServiceAccount"
    name      = kubernetes_service_account.datadog_sa.metadata[0].name
    namespace = kubernetes_namespace.datadog.metadata[0].name
  }
}

# IAM Role for Datadog AWS Integration
resource "aws_iam_role" "datadog_integration_role" {
  name = "${var.cluster_name}-datadog-integration-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect = "Allow",
        Principal = {
          Federated = "arn:aws:iam::464622532012:root" # Datadog AWS account ID
        },
        Action = "sts:AssumeRole",
        Condition = {
          StringEquals = {
            "sts:ExternalId" = "your-datadog-external-id" # Replace with your actual External ID from Datadog
          }
        }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "datadog_readonly_policy" {
  role       = aws_iam_role.datadog_integration_role.name
  policy_arn = "arn:aws:iam::aws:policy/ReadOnlyAccess" # For simplicity, use a custom policy for production
}

# Datadog AWS Integration resource
resource "datadog_integration_aws" "aws_integration" {
  account_id = data.aws_caller_identity.current.account_id # Your AWS Account ID
  role_name  = aws_iam_role.datadog_integration_role.name
  host_tags  = ["env:${var.cluster_name}", "provider:terraform"]
  # filter_tags {
  #   namespace = "ec2"
  #   key       = "environment"
  #   value     = "production"
  # }
}
data "aws_caller_identity" "current" {}

Note: The Datadog Agent deployment example above is simplified. For production, refer to Datadog's official Helm chart documentation, which provides comprehensive configuration options for DaemonSets, log collection, APM, and security features. You would typically use the helm_release resource from the Helm provider for this.

Also, for aws_iam_role_policy_attachment, ReadOnlyAccess is broad. In production, create a custom IAM policy with minimal required permissions for Datadog to follow the principle of least privilege.

PagerDuty Integration for EKS Incident Management

PagerDuty acts as your central nervous system for incident response. We'll use Terraform to provision a PagerDuty service, which represents a component or application you want to monitor, and integrate it with Datadog so that alerts can automatically trigger incidents.

Automating PagerDuty Integration

# PagerDuty Team (optional, but good for organization)
resource "pagerduty_team" "devops_team" {
  name        = "DevOps EKS Team"
  description = "Team responsible for EKS operations and observability"
}

# PagerDuty Service for EKS Cluster
resource "pagerduty_service" "eks_observability_service" {
  name                        = "${var.cluster_name}-EKS-Observability"
  auto_resolve_timeout_enabled = true
  auto_resolve_timeout         = 14400 # 4 hours
  acknowledgement_timeout      = 600   # 10 minutes
  escalation_policy            = pagerduty_escalation_policy.eks_ep.id # Assuming an EP exists or is created
  description                 = "Monitoring and incident response for AWS EKS cluster: ${var.cluster_name}"
  # team                        = pagerduty_team.devops_team.id # Uncomment if using teams
}

# PagerDuty Escalation Policy (simplified example)
resource "pagerduty_escalation_policy" "eks_ep" {
  name      = "${var.cluster_name}-EKS-EP"
  num_loops = 2

  rule {
    # Send to primary on-call schedule immediately
    escalation_delay_in_minutes = 0
    target {
      type = "user"
      id   = "PD_USER_ID" # Replace with a PagerDuty user ID
    }
  }

  rule {
    # If not acknowledged in 30 minutes, escalate to another user/team
    escalation_delay_in_minutes = 30
    target {
      type = "user"
      id   = "ANOTHER_PD_USER_ID" # Replace with another PagerDuty user ID
    }
  }
}


# PagerDuty Integration for Datadog
resource "pagerduty_extension" "datadog_to_pagerduty" {
  name     = "${var.cluster_name}-Datadog-Integration"
  endpoint = "https://events.pagerduty.com/integration/YOUR_DATADOG_INTEGRATION_KEY/enqueue" # PagerDuty will provide this URL
  type     = "datadog_v2" # Using Datadog's official PagerDuty integration type
  service  = pagerduty_service.eks_observability_service.id
}

Explanation:

We define a pagerduty_service that represents our EKS observability domain.
An escalation_policy dictates how incidents are escalated through on-call schedules or users.
The pagerduty_extension resource links Datadog to PagerDuty. You'll need to create a Datadog integration within your PagerDuty service manually to get the specific integration URL, or use a pagerduty_service_integration resource if you know the integration type details.

Automating Alerting and Dashboards

Once Datadog is collecting data and PagerDuty is set up for incident management, the next step is to define alerts and dashboards. Terraform can manage these resources within Datadog.

Creating Datadog Monitors

Datadog monitors are critical for proactive issue detection. You can define conditions for metrics, logs, or traces that, when breached, trigger an alert. These alerts can then be routed to PagerDuty.

# Example: EKS Node CPU Utilization Monitor
resource "datadog_monitor" "node_cpu_utilization" {
  name               = "[EKS] High Node CPU Utilization on ${var.cluster_name}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.cluster_name}} by {host} > 90"
  message            = "CPU utilization for {{host.name}} is above 90% for 5 minutes. @webhook-pagerduty" # Use the PagerDuty integration webhook
  tags               = ["environment:${var.cluster_name}", "severity:high", "eks"]
  escalation_message = "CPU still high after 15 minutes. Investigating further. @pagerduty"
  thresholds {
    critical = 90
    warning  = 75
  }
  notify_no_data    = false
  renotify_interval = 30
}

# Example: EKS Pod Restarts Monitor
resource "datadog_monitor" "pod_restarts" {
  name    = "[EKS] Frequent Pod Restarts on ${var.cluster_name}"
  type    = "metric alert"
  query   = "sum(last_15m):kubernetes.pod.restarts.count{cluster_name:${var.cluster_name}} > 5"
  message = "Pod {{kube_namespace}}/{{kube_pod_name}} has restarted more than 5 times in the last 15 minutes. @webhook-pagerduty"
  tags    = ["environment:${var.cluster_name}", "severity:medium", "eks"]
  thresholds {
    critical = 5
  }
}

Important: The @webhook-pagerduty in the message is a placeholder. After setting up the Datadog integration within PagerDuty, Datadog will automatically configure a webhook integration, which you'll reference in your monitor message to send alerts to the correct PagerDuty service.

Building Datadog Dashboards

Dashboards provide real-time visualizations of your EKS cluster's health and performance. While verbose to create in Terraform directly, it ensures consistency and version control.

# Example: EKS Cluster Overview Dashboard
resource "datadog_dashboard" "eks_overview" {
  title        = "${var.cluster_name} EKS Cluster Overview"
  description  = "High-level overview of EKS cluster health."
  layout_type  = "ordered"
  is_read_only = false
  widget {
    # Host Map Widget
    hostmap_definition {
      title    = "EKS Host Map"
      group_by = ["cluster_name", "host"]
      query    = "avg:kubernetes.cpu.usage.total{cluster_name:${var.cluster_name}}"
      node_type = "host"
      no_group_hosts = true
      no_metric_hosts = false
      fill_by {
        metric_name = "kubernetes.cpu.usage.total"
        palette     = "green_to_red"
        type        = "heat"
      }
      scope = ["cluster_name:${var.cluster_name}"]
    }
  }
  widget {
    # Timeseries Widget - Cluster CPU Usage
    timeseries_definition {
      title = "Cluster CPU Usage"
      query {
        query_string = "avg:kubernetes.cpu.usage.total{cluster_name:${var.cluster_name}}"
        display_type = "line"
      }
      legend_layout = "auto"
    }
  }
  # Add more widgets for Memory, Network, Pods, Deployments, Logs etc.
}

Deployment and Validation

With your Terraform configuration complete, deploy it:

Initialize Terraform: terraform init
Review the plan: terraform plan
Apply the configuration: terraform apply (and confirm with 'yes')

Validation Steps:

Datadog UI: Log into Datadog. Verify that your EKS hosts and pods are reporting metrics, logs are being ingested, and the deployed monitors and dashboards are visible.
Kubernetes: Run kubectl get pods -n datadog to ensure the Datadog Agents are running and healthy.
PagerDuty: Trigger a test alert (e.g., manually push a metric that exceeds a threshold or use a Datadog test event) to confirm an incident is created in PagerDuty and on-call teams are notified.

Best Practices and Advanced Considerations

Secret Management

Never hardcode API keys. Use Terraform variables with the sensitive = true flag and retrieve values from secure sources:

AWS Secrets Manager: For AWS-native solutions.
HashiCorp Vault: For multi-cloud or more complex secret management.
Terraform Cloud/Enterprise: Securely store variables.

Module Reusability

Encapsulate the Datadog and PagerDuty configurations into separate Terraform modules. This allows you to easily apply the same observability stack to multiple EKS clusters or environments with minimal changes.

Granular IAM Permissions

Instead of ReadOnlyAccess for Datadog AWS integration, define a custom IAM policy that grants only the specific permissions Datadog requires to collect metrics and logs. This significantly enhances your security posture.

Cost Optimization

Monitor your Datadog usage, especially for logs and custom metrics, as these can incur significant costs. Implement appropriate filtering at the agent or integration level to avoid ingesting unnecessary data.

Troubleshooting Common Issues

Datadog Agent Not Reporting

Check Pod Status: kubectl get pods -n datadog. Ensure agents are running.
Logs: kubectl logs <datadog-agent-pod-name> -n datadog. Look for API key errors or connection issues.
Network: Ensure your EKS nodes have outbound internet access to Datadog endpoints (e.g., app.datadoghq.com for metrics/events, agent-intake.datadoghq.com for logs).
RBAC: Verify the Service Account, ClusterRole, and ClusterRoleBinding for the Datadog Agent have the necessary permissions.

Alerts Not Firing / PagerDuty Integration Failures

Datadog Monitor Status: In Datadog, check the monitor's status page. Has the query threshold been met? Is it in an alert state?
Integration Webhook: Ensure the PagerDuty integration webhook in Datadog is correctly configured and used in the monitor's message.
PagerDuty Service: Verify the PagerDuty service is healthy and its integration key is active. Check the service's event log in PagerDuty for incoming events.
On-Call Schedules: Confirm the PagerDuty escalation policy targets valid users or schedules that are currently on-call.

Conclusion

By leveraging Terraform to provision Datadog for AWS EKS observability and PagerDuty for incident management, you create a robust, automated, and scalable operational framework. This IaC approach not only streamlines deployment but also ensures consistency, reduces manual errors, and provides a clear, version-controlled audit trail for your observability infrastructure. With a well-configured system, your teams can gain deeper insights into their EKS applications, respond faster to incidents, and ultimately deliver a more reliable service to your customers.

CloudStack YAML