Terraform for AWS EKS: Integrated Datadog Monitoring and PagerDuty Incident Response

In the dynamic world of cloud-native applications, managing highly available and observable Kubernetes clusters is paramount. This comprehensive guide details how to leverage Terraform to provision and maintain an AWS Elastic Kubernetes Service (EKS) cluster, seamlessly integrate Datadog for robust monitoring, and establish automated incident response through PagerDuty. By codifying your infrastructure and observability stack, you achieve consistency, repeatability, and an accelerated path to resolving critical issues.

Architecture Pro-Tip

Always design your EKS clusters with an "observability first" mindset. Before deploying any application workloads, ensure your core monitoring agents (like Datadog Agent) are in place, configured with appropriate IAM roles (IRSA), and reporting essential cluster metrics. This proactive approach prevents blind spots and facilitates faster root cause analysis when issues inevitably arise in production environments.

Why Terraform, Datadog, and PagerDuty?

Each of these tools plays a critical role in building a resilient and manageable cloud infrastructure:

Terraform: Infrastructure as Code (IaC)
Terraform enables you to define your entire infrastructure – including AWS EKS clusters, networking, security groups, and even application deployments via Helm charts – as code. This provides version control, auditability, and the ability to spin up identical environments across development, staging, and production.
Datadog: Comprehensive Monitoring and Observability
Datadog offers a unified platform for metrics, logs, traces, and synthetic monitoring. Its Kubernetes integration provides deep insights into cluster health, pod performance, container logs, and application-specific metrics, crucial for proactive issue detection.
PagerDuty: Automated Incident Response and On-Call Management
When issues are detected, PagerDuty acts as the central hub for incident response. It integrates with Datadog to ingest alerts, trigger on-call rotations, escalate incidents, and facilitate communication, ensuring critical problems are never missed and are addressed promptly by the right team members.

Prerequisites

Before you begin, ensure you have the following:

An AWS Account with appropriate IAM permissions to create EKS clusters, EC2 instances, and IAM roles.
Terraform CLI installed (version 1.0+ recommended).
AWS CLI configured with credentials.
kubectl CLI installed and configured.
Helm CLI installed (version 3+ recommended).
A Datadog Account with API and Application Keys.
A PagerDuty Account with a Service API Key.

Core Concepts and Integration Points

1. Terraform for AWS EKS Provisioning

We'll use Terraform's AWS provider to define the EKS cluster, node groups, VPC, subnets, and security groups. Key components include:

aws_eks_cluster: The EKS control plane.
aws_eks_node_group or aws_launch_template with aws_autoscaling_group: For worker nodes.
IAM Roles for Service Accounts (IRSA): Critical for allowing Kubernetes service accounts (like the Datadog Agent) to assume AWS IAM roles, providing fine-grained permissions without distributing AWS credentials directly to pods.

2. Datadog Agent Deployment on EKS via Helm and Terraform

The Datadog Agent collects metrics, logs, and traces from your cluster. We deploy it using its official Helm chart, managed by Terraform's Helm provider.

Helm Provider: Manages Helm releases within Terraform.
Datadog API & Application Keys: Passed as Helm chart values for authentication.
Cluster Agent & Node Agent: The Helm chart deploys both for comprehensive coverage.

3. Datadog Monitors Integrated with PagerDuty via Terraform

Terraform can define Datadog monitors directly, linking them to PagerDuty services for automated alerting.

datadog_integration_pagerduty: Establishes the integration between Datadog and a PagerDuty service.
datadog_monitor: Defines alert conditions. The monitor's message field will reference the PagerDuty integration.

Terraform Implementation Guide

This section provides a structured approach to setting up the integration.

Step 1: Project Structure and Providers

Organize your Terraform code into logical files. We'll assume you have a pre-existing EKS cluster or are defining one in separate modules.

.
├── main.tf
├── variables.tf
└── outputs.tf

In main.tf, configure your providers:

provider "aws" {
  region = var.aws_region
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.eks_cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.eks_cluster.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.eks_cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.eks_cluster.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

# Data sources for EKS cluster details (assuming cluster already exists or defined elsewhere)
data "aws_eks_cluster" "eks_cluster" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "eks_cluster" {
  name = var.eks_cluster_name
}
    

Step 2: IAM Role for Datadog Agent Service Account (IRSA)

Create an IAM role that the Datadog Agent's Kubernetes Service Account will assume. This is crucial for securely accessing AWS services (e.g., CloudWatch, EC2 metadata) without exposing AWS credentials directly.

The Trust Policy for this role must allow the EKS OIDC provider to assume it. Replace <YOUR_EKS_OIDC_PROVIDER_URL> and <KUBERNETES_NAMESPACE> with your specific values.

resource "aws_iam_role" "datadog_agent_role" {
  name_prefix        = "datadog-agent-eks"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Federated = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${replace(data.aws_eks_cluster.eks_cluster.identity[0].oidc[0].issuer, "https://", "")}"
        }
        Action = "sts:AssumeRoleWithWebIdentity"
        Condition = {
          StringEquals = {
            "${replace(data.aws_eks_cluster.eks_cluster.identity[0].oidc[0].issuer, "https://", "")}:sub" = "system:serviceaccount:datadog:datadog-agent"
          }
        }
      }
    ]
  })
}

resource "aws_iam_policy" "datadog_agent_policy" {
  name_prefix = "datadog-agent-eks"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "ec2:DescribeInstances",
          "ec2:DescribeTags",
          "ec2:DescribeVpcs",
          "autoscaling:DescribeAutoScalingGroups",
          "logs:DescribeLogGroups",
          "logs:DescribeLogStreams",
          "logs:GetLogEvents",
          "tag:GetResources",
          "xray:PutTraceSegments",
          "xray:PutTelemetryRecords",
          "xray:GetSamplingTargets",
        ]
        Effect   = "Allow"
        Resource = "*"
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "datadog_agent_attachment" {
  policy_arn = aws_iam_policy.datadog_agent_policy.arn
  role       = aws_iam_role.datadog_agent_role.name
}

# Required to get current AWS Account ID for OIDC provider ARN
data "aws_caller_identity" "current" {}
    

Step 3: Deploy Datadog Agent via Helm Chart

Utilize the helm_release resource to deploy the Datadog Agent, passing your API/App keys and referencing the IRSA role.

Ready-to-Use Configuration: Datadog Agent & Monitoring Setup

# Variables (in variables.tf)
variable "aws_region" {
  description = "AWS region"
  type        = string
  default     = "us-east-1"
}

variable "eks_cluster_name" {
  description = "Name of the EKS cluster"
  type        = string
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_service_name" {
  description = "Name of the PagerDuty service to integrate with"
  type        = string
}

variable "pagerduty_integration_key" {
  description = "PagerDuty Integration Key for Datadog"
  type        = string
  sensitive   = true
}


# main.tf - Datadog Agent Deployment and PagerDuty Integration

# Deploy Datadog Agent using Helm
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }
  set {
    name  = "datadog.clusterName"
    value = var.eks_cluster_name
  }
  set {
    name  = "datadog.kubeStateMetricsCore.enabled"
    value = "true"
  }
  set {
    name  = "datadog.logs.enabled"
    value = "true"
  }
  set {
    name  = "datadog.logs.containerCollectAll"
    value = "true"
  }
  set {
    name  = "datadog.apm.enabled"
    value = "true"
  }
  set {
    name  = "datadog.processAgent.enabled"
    value = "true"
  }
  set {
    name  = "datadog.rbac.create"
    value = "true"
  }
  set {
    name  = "datadog.serviceAccount.create"
    value = "true"
  }
  set {
    name  = "datadog.serviceAccount.name"
    value = "datadog-agent"
  }
  set {
    name  = "datadog.serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
    value = aws_iam_role.datadog_agent_role.arn
    type  = "string"
  }
}


# Configure PagerDuty Integration in Datadog
resource "datadog_integration_pagerduty" "pagerduty_integration" {
  api_key = var.pagerduty_integration_key
  name    = var.pagerduty_service_name
}

# Example Datadog Monitor for High EKS Node CPU Utilization
resource "datadog_monitor" "high_node_cpu" {
  name               = "[EKS] High Node CPU Utilization on ${var.eks_cluster_name}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.eks_cluster_name}} by {host} > 80"
  message            = "High CPU utilization detected on EKS node {{host.name}} in cluster ${var.eks_cluster_name}. Please investigate. @pagerduty-${datadog_integration_pagerduty.pagerduty_integration.name}"
  tags               = ["environment:production", "service:eks", "alert-type:performance"]
  renotify_interval  = 60
  no_data_timeframe  = 20
  timeout_h          = 0
  escalation_message = "CPU utilization remains high after 1 hour. Escalating to engineering lead."
  priority           = 1 # P1 for critical alerts
}

# Example Datadog Monitor for EKS Pod Restarts
resource "datadog_monitor" "pod_restarts" {
  name               = "[EKS] High Pod Restart Rate in ${var.eks_cluster_name}"
  type               = "metric alert"
  query              = "sum(last_5m):sum:kubernetes.pod.restarts{cluster_name:${var.eks_cluster_name}} > 5"
  message            = "Multiple pod restarts detected in cluster ${var.eks_cluster_name}. Investigate problematic deployments. @pagerduty-${datadog_integration_pagerduty.pagerduty_integration.name}"
  tags               = ["environment:production", "service:eks", "alert-type:stability"]
  renotify_interval  = 60
  no_data_timeframe  = 10
  timeout_h          = 0
  priority           = 2 # P2 for important alerts
}
    

Important: Replace placeholder values like var.eks_cluster_name, var.datadog_api_key, var.datadog_app_key, var.pagerduty_service_name, and var.pagerduty_integration_key with your actual values, preferably loaded from environment variables or a secure secret management system.

Deployment and Verification

1. Deploy Terraform Configuration

Initialize Terraform:
```
terraform init
```
Review the plan:
```
terraform plan
```
Apply the changes:
```
terraform apply
```

2. Verify Datadog Agent Deployment

Check Kubernetes pods:
```
kubectl get pods -n datadog
```
You should see datadog-agent-* pods running.
In Datadog, navigate to Infrastructure -> Kubernetes. Your EKS cluster and nodes should appear, reporting metrics, logs, and traces.

3. Test Datadog Monitors and PagerDuty Integration

In Datadog, go to Monitors -> Manage Monitors. You should see the monitors created by Terraform.
To test PagerDuty, you can manually trigger an incident from a Datadog monitor or intentionally create a scenario that violates your alert conditions (e.g., scale down nodes to increase CPU utilization on remaining nodes).
Verify that an incident is created in PagerDuty and the correct on-call team is notified according to your escalation policies.

Advanced Considerations

Custom Metrics and Autodiscovery

Datadog's Autodiscovery feature allows the agent to automatically apply monitoring configurations based on pod annotations or container images. For custom application metrics, integrate the Datadog client libraries into your application code, and the agent will collect them.

Security Best Practices

Least Privilege: Ensure the IAM role for the Datadog Agent has only the necessary permissions. Regularly review and prune policies.
Secrets Management: Never hardcode API/App keys. Use AWS Secrets Manager, HashiCorp Vault, or environment variables in CI/CD pipelines.
Network Policies: Implement Kubernetes Network Policies to restrict traffic to/from the Datadog Agent pods.

Cost Optimization

Selective Logging/Metrics: Configure Datadog to ingest only necessary logs and metrics to control costs.
Resource Limits: Set appropriate CPU and memory limits for the Datadog Agent pods to prevent resource overconsumption.
Monitor Tuning: Optimize alert queries and thresholds to reduce false positives, minimizing unnecessary PagerDuty notifications and on-call disruptions.

Troubleshooting and FAQ

Q: Datadog Agent pods are not running or in CrashLoopBackOff.

A: Check pod logs: kubectl logs <datadog-agent-pod> -n datadog. Common issues include incorrect API/App keys, insufficient IAM permissions (verify the IRSA role and policy), or resource constraints on the node.

Q: Metrics are not appearing in Datadog.

A: Ensure the Datadog Agent pods are healthy. Verify the datadog.apiKey and datadog.appKey values in your Helm release are correct. Check the agent status: kubectl exec -it <datadog-agent-pod> -n datadog -- agent status.

Q: PagerDuty incidents are not being created from Datadog alerts.

Verify the datadog_integration_pagerduty resource was applied successfully.
Ensure the @pagerduty-<INTEGRATION_NAME> tag in your Datadog monitor message exactly matches the name you gave your PagerDuty integration in Datadog.
Check Datadog's event stream for the monitor to see if it's firing and if there are any errors related to PagerDuty integration.
In PagerDuty, check the integration status for the relevant service.

Conclusion

By leveraging Terraform, you can provision and manage your AWS EKS infrastructure with the same rigor and automation as your application code. Integrating Datadog provides unparalleled observability into your Kubernetes clusters, while PagerDuty ensures that critical issues are handled efficiently and effectively, minimizing downtime and operational overhead. This codified approach to infrastructure, monitoring, and incident response is a cornerstone of modern, high-performing DevOps teams, enabling greater reliability and faster innovation.

CloudStack YAML