Terraform for AWS EKS: Automated Datadog Observability and PagerDuty Incident Management

In today's fast-paced cloud-native environment, managing Amazon Elastic Kubernetes Service (EKS) clusters demands not just robust infrastructure, but also proactive observability and swift incident response. This comprehensive guide details how to leverage Terraform to automate the setup of Datadog for deep observability into your EKS workloads and integrate PagerDuty for streamlined incident management. By codifying these critical components, you ensure consistency, reduce manual errors, and accelerate your time to detection and resolution.

Architecture Pro-Tip

For a highly resilient and scalable setup, always modularize your Terraform configurations. Separate concerns like AWS infrastructure (VPC, EKS), Datadog agents and integrations, and PagerDuty services into distinct modules. Implement a GitOps workflow where all changes to your EKS configuration, Datadog monitors, and PagerDuty services are committed to version control and applied automatically, ensuring traceability and auditability. Utilize IAM Roles for Service Accounts (IRSA) for fine-grained permissions for your Datadog Agent within EKS, enhancing security posture.

Why Automate Observability and Incident Management?

Automating your observability and incident response stack with Terraform offers significant advantages:

Consistency: Ensure all EKS clusters have the same monitoring and alerting configurations.
Speed: Provision and update complex setups quickly and efficiently.
Auditability: Track every change to your monitoring and alerting infrastructure through version control.
Scalability: Easily replicate your setup across multiple environments or new clusters.
Reduced Toil: Minimize manual configuration, freeing up DevOps teams for more strategic work.

Key Components of Our Stack

We'll be integrating three powerful tools:

Terraform: An open-source Infrastructure as Code (IaC) tool by HashiCorp for provisioning and managing cloud infrastructure.
AWS EKS: Amazon's managed Kubernetes service, simplifying the deployment, management, and scaling of containerized applications.
Datadog: A comprehensive monitoring and analytics platform that brings together data from servers, databases, tools, and services to present a unified view of your entire stack.
PagerDuty: An incident management platform that provides reliable notifications, automatic escalations, and on-call scheduling to resolve critical incidents quickly.

Prerequisites

Before you begin, ensure you have the following in place:

An active AWS Account with administrative access.
Terraform CLI installed (v1.0.0 or higher recommended).
AWS CLI installed and configured with appropriate credentials.
A Datadog Account with an API key and Application key.
A PagerDuty Account with an API token.
Basic understanding of Kubernetes and EKS.

Step-by-Step Implementation

1. Setting up Terraform Providers

First, configure the necessary Terraform providers in your versions.tf file:

provider "aws" {
  region = "us-east-1"
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
    pagerduty = {
      source  = "PagerDuty/pagerduty"
      version = "~> 2.0"
    }
  }
  required_version = "~> 1.0"
}
    

2. EKS Cluster Setup (Pre-existing or via Terraform)

While the full EKS cluster setup is beyond the scope of this guide, assume you have an EKS cluster running. If you're building one with Terraform, ensure it has necessary IAM roles for node groups and OIDC provider enabled. You will typically use modules like terraform-aws-modules/eks/aws.

For Datadog Agent deployment, you'll generally use its Helm chart. We'll focus on the Datadog platform configuration (monitors, dashboards) and PagerDuty integration using Terraform. The Datadog Agent requires an IAM role for Kubernetes Service Accounts (IRSA) to collect metrics securely from AWS services.

3. Datadog Integration with AWS and EKS

To allow Datadog to pull metrics from your AWS account and EKS, you need to set up an IAM role that Datadog can assume. This is crucial for collecting metrics from EC2, RDS, Lambda, and more, as well as EKS control plane metrics.

3.1. AWS IAM Role for Datadog Integration

Create an IAM role that Datadog can assume. This role needs specific permissions to read metrics, logs, and configurations from AWS services.

The datadog_integration_aws resource then links this role to your Datadog account.

3.2. Datadog Agent Deployment (Helm Chart reference)

While the Datadog Agent is usually deployed using a Helm chart directly into your EKS cluster, you would configure its values to use IRSA and your Datadog API key. This part is typically managed by a Kubernetes manifest or Helm release resource after the EKS cluster is provisioned. For example, using the kubernetes_helm_release resource if you choose to deploy it via Terraform:

# Example for deploying Datadog Agent via Helm using Terraform (requires Kubernetes provider config)
/*
resource "kubernetes_helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  # Enable APM and Log collection
  set {
    name  = "apm.enabled"
    value = "true"
  }
  set {
    name  = "logs.enabled"
    value = "true"
    type  = "string" # Important for boolean values in Helm sets
  }
  set {
    name  = "logs.containerCollectAll"
    value = "true"
    type  = "string"
  }

  # Enable EKS specific integrations and IRSA
  set {
    name  = "clusterAgent.enabled"
    value = "true"
    type  = "string"
  }
  set {
    name  = "clusterAgent.createRbac"
    value = "true"
    type  = "string"
  }
  set {
    name  = "clusterAgent.externalMetrics.enabled"
    value = "true"
    type  = "string"
  }
  set {
    name  = "agents.podLabelsAsTags"
    value = "{}" # Add any custom labels here
  }
  set {
    name  = "agents.containerExclude"
    value = "name:kube-proxy" # Example exclusion
  }

  # IRSA configuration for the Datadog Agent
  # This assumes you have an IAM role and service account created via Terraform.
  # Example:
  # set {
  #   name  = "clusterAgent.env[0].name"
  #   value = "DD_KUBERNETES_KUBELET_TLS_VERIFY"
  # }
  # set {
  #   name  = "clusterAgent.env[0].value"
  #   value = "false"
  # }
  # set {
  #   name  = "clusterAgent.rbac.serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
  #   value = aws_iam_role.datadog_agent_irsa_role.arn
  # }
}
*/
    

The above commented-out block demonstrates how you *would* deploy the Datadog agent using Terraform's Helm provider if you manage your Kubernetes manifests directly with Terraform. For simplicity and focus on Datadog/PagerDuty platform automation, we will proceed assuming the Datadog agent is already deployed and configured to report to your Datadog account.

4. Automated Datadog Monitors and Dashboards

Now, let's define observability rules. Terraform allows you to codify Datadog monitors, which trigger alerts based on specific metrics or log patterns, and dashboards for visual insights.

4.1. Datadog Monitors for EKS Health

Example monitors for common EKS issues:

High CPU Usage: Alert when a node or pod consistently exceeds a CPU threshold.
Memory Exceeded: Warn when a pod is approaching its memory limit.
OOMKilled Pods: Detect when pods are being killed due to out-of-memory conditions.
Pod Not Ready: Alert on pods that are not reaching a "Ready" state.
High HTTP Error Rates: For your application services.

4.2. Datadog Dashboards for EKS Overview

Create custom dashboards to visualize key EKS metrics:

Node and Pod resource utilization (CPU, Memory, Disk).
Kubernetes events and logs.
Network performance (e.g., ingress/egress bytes).
Application-specific metrics.

5. PagerDuty Integration for Incident Management

Integrate PagerDuty to ensure that critical Datadog alerts escalate to the right team members at the right time.

5.1. PagerDuty Service and Integration

Define a PagerDuty service, which represents a component your team is responsible for, and an integration, which is how events get into PagerDuty (e.g., a Datadog integration).

5.2. Connecting Datadog Monitors to PagerDuty

In your Datadog monitor definitions, you can specify PagerDuty as a notification channel. When a monitor's alert condition is met, Datadog will trigger an incident in PagerDuty.

Full Terraform Configuration Example

Below is a comprehensive Terraform example demonstrating the integration of Datadog and PagerDuty for your AWS EKS environment. This example includes an IAM role for Datadog to assume, a Datadog monitor for EKS node CPU utilization, a Datadog dashboard, and a PagerDuty service with a Datadog integration. Remember to replace placeholder values with your actual cluster and account details.

Terraform Code for Datadog and PagerDuty Automation

# main.tf

# Define variables for sensitive information
variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

variable "aws_account_id" {
  description = "Your AWS Account ID"
  type        = string
}

variable "eks_cluster_name" {
  description = "Name of your EKS cluster"
  type        = string
}

variable "datadog_external_id" {
  description = "Datadog external ID for AWS integration"
  type        = string
  default     = "123456789012" # Replace with your actual Datadog External ID
}

# --- AWS IAM Role for Datadog Integration ---
resource "aws_iam_role" "datadog_integration_role" {
  name = "DatadogIntegrationRole-${var.eks_cluster_name}"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          AWS = "arn:aws:iam::464622532012:root" # Datadog's AWS account ID
        }
        Condition = {
          StringEquals = {
            "sts:ExternalId" = var.datadog_external_id
          }
        }
      },
    ]
  })
}

resource "aws_iam_policy" "datadog_read_only_policy" {
  name        = "DatadogReadOnlyPolicy-${var.eks_cluster_name}"
  description = "Grants Datadog read-only access for monitoring"

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action = [
          "apigateway:GET",
          "autoscaling:Describe*",
          "cloudtrail:DescribeTrails",
          "cloudwatch:Describe*",
          "cloudwatch:Get*",
          "cloudwatch:List*",
          "ec2:Describe*",
          "ecs:Describe*",
          "ecs:List*",
          "elasticache:DescribeCacheClusters",
          "elasticache:ListTagsForResource",
          "elb:Describe*",
          "health:DescribeEvents",
          "health:DescribeEventDetails",
          "health:DescribeAffectedEntities",
          "kinesis:ListShards",
          "lambda:List*",
          "rds:Describe*",
          "redshift:Describe*",
          "route53:ListHostedZones",
          "s3:GetBucketLocation",
          "s3:ListAllMyBuckets",
          "sagemaker:ListEndpoints",
          "sagemaker:ListTrainingJobs",
          "sns:ListTopics",
          "sqs:ListQueues",
          "tag:GetResources",
          "tag:GetTagKeys",
          "tag:GetTagValues"
        ],
        Effect   = "Allow",
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "datadog_policy_attach" {
  role       = aws_iam_role.datadog_integration_role.name
  policy_arn = aws_iam_policy.datadog_read_only_policy.arn
}

# --- Datadog AWS Integration ---
resource "datadog_integration_aws" "aws_integration" {
  account_id = var.aws_account_id
  role_name  = aws_iam_role.datadog_integration_role.name
  host_tags  = ["env:production", "eks_cluster:${var.eks_cluster_name}"]
  # Set `filter_tags` if you want to limit what Datadog monitors
  # filter_tags = ["region:us-east-1", "env:prod"]
  # `excluded_regions` if you want to exclude specific regions
  # excluded_regions = ["us-west-1"]
}


# --- PagerDuty Service for EKS Incidents ---
resource "pagerduty_user" "oncall_user" {
  name  = "John Doe"
  email = "john.doe@example.com"
  # Set to your actual user details, or retrieve an existing user ID
}

resource "pagerduty_team" "devops_team" {
  name        = "DevOps Team"
  description = "Team responsible for EKS operations and incident response."
}

resource "pagerduty_escalation_policy" "eks_escalation_policy" {
  name      = "EKS Cluster Escalation Policy"
  num_loops = 2
  team      = pagerduty_team.devops_team.id

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "user_reference"
      id   = pagerduty_user.oncall_user.id
    }
  }
}

resource "pagerduty_service" "eks_monitoring_service" {
  name                    = "EKS Cluster Monitoring - ${var.eks_cluster_name}"
  auto_resolve_timeout    = "14400" # 4 hours
  acknowledgement_timeout = "600"   # 10 minutes
  escalation_policy       = pagerduty_escalation_policy.eks_escalation_policy.id
  team                    = pagerduty_team.devops_team.id
}

# PagerDuty integration for Datadog
resource "pagerduty_extension" "datadog_integration" {
  name        = "Datadog Integration for EKS"
  endpoint_url = "https://events.pagerduty.com/integration/YOUR_DATADOG_INTEGRATION_KEY/enqueue" # Replace with actual Datadog integration key from PagerDuty
  extension_objects = [pagerduty_service.eks_monitoring_service.id]
  type        = "datadog_extension" # This type is specific to PagerDuty's Datadog integration type
  # NOTE: The above endpoint_url for PagerDuty integration should be generated in PagerDuty under
  # 'Service Integrations' for a 'Datadog' integration type.
  # Terraform PagerDuty provider doesn't directly create the Datadog integration key for you,
  # it expects you to provide the URL from a pre-configured PagerDuty service integration.
  # For direct Datadog integration setup, you'd typically create a 'generic_events_api_v2' integration.
  # Let's adjust for a more direct Terraform-managed integration:
}

resource "pagerduty_service_integration" "datadog_integration_v2" {
  name    = "Datadog Monitoring EKS"
  service = pagerduty_service.eks_monitoring_service.id
  type    = "generic_events_api_v2" # Using Generic Events API V2 for broader compatibility
}


# --- Datadog Monitors ---

resource "datadog_monitor" "eks_node_cpu_utilization" {
  name               = "[EKS] High Node CPU Utilization on {{host.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:system.cpu.idle{eks_cluster_name:${var.eks_cluster_name}} by {host} < 10" # Alert if avg idle CPU is < 10% (i.e., usage > 90%)
  message            = "CPU utilization on host {{host.name}} is {{value}}%. @slack-devops @${pagerduty_service.eks_monitoring_service.name}"
  monitor_thresholds {
    critical = 10
    warning  = 20
  }
  notify_no_data       = false
  new_group_delay      = 150 # 2.5 minutes
  no_data_timeframe    = 20 # 20 minutes
  include_tags         = true
  renotify_interval    = 60
  force_delete         = false
  tags                 = ["env:production", "eks", "cpu"]
  escalation_message = "CPU utilization on host {{host.name}} is still critical. Escalating to PagerDuty."
  # Integrating with PagerDuty: use the integration URL obtained from PagerDuty
  # or simply mention the PagerDuty service name if Datadog is already configured
  # to send alerts to PagerDuty by default via its global PagerDuty integration.
  # For direct integration via a custom webhook/integration, you'd use a webhook notification.
  # A simpler approach is to mention the PagerDuty service via @pagerduty-
  # provided Datadog's global PagerDuty integration is configured.
  # For this example, we assume Datadog's global PagerDuty integration is set up
  # or the specific service is mentioned.
}

resource "datadog_monitor" "eks_pod_oomkilled" {
  name               = "[EKS] Pod OOMKilled detected in {{host.name}}"
  type               = "log alert"
  query              = "logs(\"service:${var.eks_cluster_name} status:error @kubernetes.reason:OOMKilled\").index(\"main\").rollup(\"count\").last(\"5m\") > 0"
  message            = "A pod was OOMKilled in cluster ${var.eks_cluster_name} on host {{host.name}}. Investigate memory limits. @slack-devops @${pagerduty_service.eks_monitoring_service.name}"
  monitor_thresholds {
    critical = 1
  }
  notify_no_data       = false
  include_tags         = true
  renotify_interval    = 0
  tags                 = ["env:production", "eks", "oomkilled", "memory"]
}

# --- Datadog Dashboard for EKS Overview ---
resource "datadog_dashboard" "eks_overview_dashboard" {
  title        = "EKS Cluster Overview - ${var.eks_cluster_name}"
  description  = "Overview of key metrics for EKS Cluster: ${var.eks_cluster_name}"
  layout_type  = "ordered"
  is_read_only = false
  widget {
    definition {
      title = "Node CPU Utilization"
      type  = "timeseries"
      request {
        query {
          q = "avg:kubernetes.cpu.usage.total{eks_cluster_name:${var.eks_cluster_name}} by {host}"
        }
      }
    }
  }

  widget {
    definition {
      title = "Node Memory Utilization"
      type  = "timeseries"
      request {
        query {
          q = "avg:kubernetes.memory.usage.total{eks_cluster_name:${var.eks_cluster_name}} by {host}"
        }
      }
    }
  }

  widget {
    definition {
      title = "Pod Count by Status"
      type  = "toplist"
      request {
        query {
          q = "sum:kubernetes.pod.status{eks_cluster_name:${var.eks_cluster_name}} by {kube_state}"
        }
      }
    }
  }

  widget {
    definition {
      title = "Kubernetes Events Stream"
      type  = "event_stream"
      query = "tags:kubernetes,eks_cluster_name:${var.eks_cluster_name}"
    }
  }

  tags = ["env:production", "eks", "observability", "dashboard"]
}
    

Running the Terraform Configuration

Once you have placed the above code in your .tf files (e.g., main.tf, variables.tf), follow these steps:

Initialize Terraform: Open your terminal in the directory containing your Terraform files and run terraform init. This downloads the necessary providers.
Plan Changes: Execute terraform plan -var="datadog_api_key=..." -var="datadog_app_key=..." -var="pagerduty_api_token=..." -var="aws_account_id=..." -var="eks_cluster_name=...". Replace ... with your actual keys and IDs. Terraform will show you what resources will be created.
Apply Changes: If the plan looks correct, run terraform apply -var="datadog_api_key=..." -var="datadog_app_key=..." -var="pagerduty_api_token=..." -var="aws_account_id=..." -var="eks_cluster_name=...". Confirm with yes when prompted.

Your AWS IAM role, Datadog integration, monitors, dashboards, and PagerDuty service should now be provisioned.

Verification

After applying the Terraform configuration:

Datadog UI: Navigate to your Datadog account.
- Check Integrations -> AWS to confirm your AWS account is connected and the role is assumed correctly.
- Go to Monitors -> Manage Monitors to see the newly created EKS monitors.
- Visit Dashboards -> Dashboard List to find your EKS Overview dashboard and verify data is populating.
PagerDuty UI: Log in to your PagerDuty account.
- Go to Services -> Service Directory to find your new EKS Cluster Monitoring service.
- Under the service, check Integrations to ensure the Datadog integration is present.
AWS IAM: Verify the DatadogIntegrationRole exists in your AWS IAM console with the correct trust policy and permissions.

Advanced Considerations and Best Practices

Terraform Modules: Encapsulate common configurations into reusable modules (e.g., an eks-datadog-module).
Secret Management: Use AWS Secrets Manager, HashiCorp Vault, or environment variables to manage sensitive API keys and tokens instead of hardcoding or passing them directly via command line.
State Management: Always use a remote backend for your Terraform state (e.g., S3 with DynamoDB locking) to enable team collaboration and prevent data loss.
Granular Permissions: Refine IAM policies to follow the principle of least privilege, granting Datadog only the necessary permissions.
PagerDuty On-Call Schedules: Define comprehensive on-call schedules and user rosters in PagerDuty to ensure alerts always reach an available responder.
Alert Fatigue: Design your Datadog monitors carefully to avoid alert storms. Use composite monitors, anomaly detection, and machine learning features where appropriate.
GitOps Workflow: Integrate your Terraform code into a GitOps pipeline (e.g., using AWS CodePipeline, GitHub Actions, GitLab CI, Argo CD) for automated deployments and rollbacks.

Troubleshooting Common Issues

Datadog not showing AWS data:
- Verify the IAM role's trust policy allows Datadog's AWS account ID and that the external ID matches.
- Check if the IAM policy attached to the role has the necessary read-only permissions.
- Ensure the Datadog API and Application keys are correct and active.
Datadog not showing EKS metrics/logs:
- Confirm the Datadog Agent is successfully deployed on your EKS cluster and its pods are running healthy.
- Check Datadog Agent logs for any errors (kubectl logs -f datadog-agent-<pod-id>).
- Verify the Datadog Agent's IAM Role for Service Account (IRSA) has permissions to scrape Kubernetes APIs and collect metrics.
PagerDuty incidents not triggering:
- Ensure the Datadog monitor's message explicitly includes the PagerDuty service (e.g., @pagerduty-eks-monitoring-service) or that your Datadog global PagerDuty integration is configured.
- Check the PagerDuty integration key/endpoint URL is correct.
- Verify the PagerDuty service has an on-call schedule configured and active users.
Terraform provider errors:
- Ensure your AWS, Datadog, and PagerDuty credentials (API keys/tokens) are correctly passed as variables or configured in your environment.
- Check the provider versions in versions.tf match what's expected.

Conclusion

Automating Datadog observability and PagerDuty incident management for AWS EKS with Terraform transforms your operational capabilities. It shifts your team from reactive firefighting to proactive, efficient incident resolution, all while ensuring your infrastructure is monitored with consistency and confidence. By embracing IaC for your entire observability and incident response stack, you lay the groundwork for a truly resilient and scalable cloud-native environment.

Start implementing these practices today to gain unparalleled visibility and control over your EKS clusters.

CloudStack YAML