Terraform AWS EKS Provisioning, Datadog Observability, PagerDuty Incident Management, Infrastructure as Code, Cloud DevOps Solutions [CONTENT]

Terraform Provisioning for AWS EKS: Datadog Observability & PagerDuty Incident Management

In the dynamic landscape of cloud-native applications, establishing robust, observable, and resilient Kubernetes clusters is paramount. This guide provides a comprehensive, technical walkthrough on how to leverage Terraform for provisioning an AWS Elastic Kubernetes Service (EKS) cluster, integrating it seamlessly with Datadog for full-stack observability, and empowering incident response with PagerDuty. By the end of this guide, you will have a fully automated, production-ready EKS environment with powerful monitoring and incident management capabilities, all defined as Infrastructure as Code (IaC).

Architecture Pro-Tip: Embracing Modular IaC

Always design your Terraform configurations with modularity in mind. Utilize modules for EKS, VPC, and even Datadog agent deployments. This approach promotes reusability, reduces cognitive load, and enables easier management of complex cloud infrastructure. Furthermore, integrate secrets management (e.g., AWS Secrets Manager, HashiCorp Vault) for sensitive API keys (Datadog, PagerDuty) rather than hardcoding them.

Why Terraform, AWS EKS, Datadog, and PagerDuty?

Terraform: The IaC Standard

Terraform is the industry-leading open-source IaC tool that allows you to define and provision cloud and on-prem resources using a declarative configuration language. Its provider ecosystem supports a vast array of services, making it ideal for orchestrating complex cloud environments like EKS with third-party integrations.

AWS EKS: Managed Kubernetes at Scale

Amazon EKS simplifies the deployment, management, and scaling of Kubernetes applications on AWS. It integrates with other AWS services, providing a highly available and secure control plane. Leveraging EKS frees development teams from managing Kubernetes infrastructure, allowing them to focus on application logic.

Datadog: Unified Observability for Cloud-Native

Datadog offers a comprehensive SaaS-based monitoring and analytics platform for cloud-scale applications. It provides end-to-end visibility across your EKS cluster, applications, and underlying AWS infrastructure, consolidating metrics, logs, traces, and user experience data into a single pane of glass. This holistic view is critical for understanding performance bottlenecks and debugging issues.

Metrics & APM: Real-time performance monitoring for EKS pods, nodes, and applications.
Log Management: Centralized log collection and analysis from all EKS components.
Synthetic Monitoring: Proactive testing of application availability and performance.
Security Monitoring: Detection of security threats and misconfigurations.

PagerDuty: Incident Management Automation

PagerDuty is a leading digital operations management platform that provides intelligent incident response, on-call management, and automated resolution workflows. Integrating Datadog with PagerDuty ensures that critical alerts from your EKS cluster are immediately routed to the right teams, enabling rapid response and minimizing downtime.

Automated On-Call Schedules: Ensure the right person is always notified.
Rich Incident Context: PagerDuty incidents are enriched with Datadog alert details.
Response Automation: Trigger runbooks and automated actions directly from incidents.

Prerequisites

Before you begin, ensure you have the following installed and configured:

AWS CLI: Configured with appropriate credentials and permissions (e.g., AdministratorAccess for simplicity, or more granular IAM policies for production).
Terraform CLI: Version 1.0 or higher.
kubectl: For interacting with the EKS cluster post-provisioning.
Helm CLI: For deploying the Datadog Agent, although Terraform will manage the release.
Datadog Account: With your Datadog API Key and Application Key.
PagerDuty Account: With an integration key for a service you want to connect.

Terraform Setup for EKS, Datadog, and PagerDuty

We will structure our Terraform project to provision the EKS cluster, deploy the Datadog Agent using the Helm provider, and define a Datadog monitor that uses PagerDuty for incident alerting. Create a directory (e.g., `eks-observability`) and place your `.tf` files within it.

1. Provider Configuration (`providers.tf`)

Define the AWS, Kubernetes, Helm, and Datadog providers. Ensure the AWS region is set correctly.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.30"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

provider "kubernetes" {
  host                   = aws_eks_cluster.eks_cluster.endpoint
  cluster_ca_certificate = base64decode(aws_eks_cluster.eks_cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.eks_auth.token
  # Explicitly disable client_certificate/key for EKS for better compatibility
  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args        = ["eks", "get-token", "--cluster-name", aws_eks_cluster.eks_cluster.name]
  }
}

provider "helm" {
  kubernetes {
    host                   = aws_eks_cluster.eks_cluster.endpoint
    cluster_ca_certificate = base64decode(aws_eks_cluster.eks_cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.eks_auth.token
    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args        = ["eks", "get-token", "--cluster-name", aws_eks_cluster.eks_cluster.name]
    }
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}
    

2. Variables Definition (`variables.tf`)

Define variables for your AWS region, EKS cluster name, instance types, and sensitive API keys. Use environment variables or a `terraform.tfvars` file to provide values for sensitive data.

variable "aws_region" {
  description = "AWS region for the EKS cluster."
  type        = string
  default     = "us-east-1"
}

variable "cluster_name" {
  description = "Name of the EKS cluster."
  type        = string
  default     = "my-eks-cluster"
}

variable "instance_type" {
  description = "EC2 instance type for EKS worker nodes."
  type        = string
  default     = "t3.medium"
}

variable "desired_size" {
  description = "Desired number of worker nodes."
  type        = number
  default     = 2
}

variable "max_size" {
  description = "Maximum number of worker nodes."
  type        = number
  default     = 3
}

variable "min_size" {
  description = "Minimum number of worker nodes."
  type        = number
  default     = 1
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_service_integration_key" {
  description = "PagerDuty Service Integration Key (from Datadog integration)"
  type        = string
  sensitive   = true
}
    

3. Main Configuration (`main.tf`)

This file will contain the core resources: VPC, EKS cluster, Node Group, Datadog Helm release, and a sample Datadog monitor.

Note: For production environments, consider using the official `terraform-aws-modules/eks/aws` module for a more robust and feature-rich EKS setup. This example uses direct resource definitions for clarity.

# --- VPC and Subnets ---
resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
  tags = {
    Name = "${var.cluster_name}-vpc"
  }
}

resource "aws_subnet" "public" {
  count             = 2
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.0.${count.index}.0/24"
  availability_zone = "${var.aws_region}${element(["a", "b"], count.index)}"
  tags = {
    Name = "${var.cluster_name}-public-subnet-${count.index}"
    "kubernetes.io/cluster/${var.cluster_name}" = "owned"
    "kubernetes.io/role/elb"                  = "1" # Required for AWS Load Balancer Controller
  }
}

resource "aws_internet_gateway" "gw" {
  vpc_id = aws_vpc.main.id
  tags = {
    Name = "${var.cluster_name}-igw"
  }
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.gw.id
  }
  tags = {
    Name = "${var.cluster_name}-public-rt"
  }
}

resource "aws_route_table_association" "public" {
  count          = 2
  subnet_id      = aws_subnet.public[count.index].id
  route_table_id = aws_route_table.public.id
}

# --- EKS Cluster ---
resource "aws_iam_role" "eks_cluster_role" {
  name = "${var.cluster_name}-eks-cluster-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect    = "Allow",
        Principal = { Service = "eks.amazonaws.com" },
        Action    = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eks_cluster_role.name
}

resource "aws_iam_role_policy_attachment" "eks_vpc_cni_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy" # Required for EKS to use CNI
  role       = aws_iam_role.eks_cluster_role.name
}

resource "aws_eks_cluster" "eks_cluster" {
  name     = var.cluster_name
  role_arn = aws_iam_role.eks_cluster_role.arn
  vpc_config {
    subnet_ids = [for s in aws_subnet.public : s.id]
    security_group_ids = [] # EKS creates a security group by default
  }
  depends_on = [
    aws_iam_role_policy_attachment.eks_cluster_policy,
    aws_iam_role_policy_attachment.eks_vpc_cni_policy,
  ]
}

data "aws_eks_cluster_auth" "eks_auth" {
  name = aws_eks_cluster.eks_cluster.name
}

# --- EKS Node Group ---
resource "aws_iam_role" "eks_node_role" {
  name = "${var.cluster_name}-eks-node-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect    = "Allow",
        Principal = { Service = "ec2.amazonaws.com" },
        Action    = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_worker_node_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.eks_node_role.name
}

resource "aws_iam_role_policy_attachment" "eks_cni_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.eks_node_role.name
}

resource "aws_iam_role_policy_attachment" "eks_ec2_container_registry_readonly_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.eks_node_role.name
}

resource "aws_eks_node_group" "private_nodes" {
  cluster_name    = aws_eks_cluster.eks_cluster.name
  node_group_name = "${var.cluster_name}-private-nodes"
  node_role_arn   = aws_iam_role.eks_node_role.arn
  subnet_ids      = [for s in aws_subnet.public : s.id] # Using public subnets for simplicity in this example

  instance_types = [var.instance_type]

  scaling_config {
    desired_size = var.desired_size
    max_size     = var.max_size
    min_size     = var.min_size
  }

  depends_on = [
    aws_iam_role_policy_attachment.eks_worker_node_policy,
    aws_iam_role_policy_attachment.eks_cni_policy,
    aws_iam_role_policy_attachment.eks_ec2_container_registry_readonly_policy,
  ]
}

# --- Datadog Agent Deployment via Helm ---
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "clusterChecksRunner.enabled"
    value = "true"
  }

  set {
    name  = "kubeStateMetricsExternal.enabled"
    value = "false" # Datadog can collect KSM metrics internally
  }

  set {
    name  = "datadog.site"
    value = "datadoghq.com" # Or eu.datadoghq.com, etc.
  }

  set {
    name  = "datadog.confd.kube_proxy_metrics.yaml"
    value = <<-EOF
      ad_identifiers:
        - kube-proxy
      init_config:
      instances:
        - prometheus_url: http://%%host%%:10249/metrics
          labels_as_tags:
            kubernetes_pod_name: pod_name
    EOF
  }

  depends_on = [
    aws_eks_cluster.eks_cluster,
    aws_eks_node_group.private_nodes # Ensure nodes are ready to receive agent
  ]
}

# --- Datadog PagerDuty Integration (via Datadog Monitor) ---
# First, ensure PagerDuty integration is set up in Datadog UI (Configuration -> Integrations -> PagerDuty).
# This key is usually obtained from Datadog after setting up the PagerDuty integration.
resource "datadog_integration_pagerduty" "pagerduty_integration" {
  # This resource can manage the integration itself.
  # If you prefer to manually configure in Datadog UI,
  # ensure 'pagerduty_service_integration_key' is the specific key
  # for the service you configured in Datadog which maps to a PagerDuty service.
  # For direct PagerDuty service integration, you might need a different API.
  # For simplicity, we assume the integration is set up in Datadog UI
  # and we are using a variable to reference the service key known to Datadog.

  # In a real scenario, you'd likely create this integration once via the UI
  # or a dedicated script, and then just reference its ID in monitors.
  # The datadog_integration_pagerduty resource primarily manages
  # the global Datadog <> PagerDuty account-level integration.
  # For specific service integrations, you refer to them in monitors.
  # The variable `pagerduty_service_integration_key` would be the key
  # for a specific PagerDuty service integration *configured within Datadog*.
  # Example: data "datadog_integration_pagerduty" "by_name" { service_name = "My PagerDuty Service" }
  # Then use data.datadog_integration_pagerduty.by_name.service_key in the monitor.
}


resource "datadog_monitor" "eks_high_cpu_monitor" {
  name                = "[EKS] High CPU Utilization on Node: {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.cpu.usage.total{*} by {host} > 80"
  message             = "EKS Node {{host.name}} CPU utilization is over 80%! Investigate immediately. @pagerduty-service-${var.pagerduty_service_integration_key}"
  tags                = ["environment:production", "service:eks", "alert:cpu"]
  new_group_delay     = 60
  no_data_timeframe   = 20
  renotify_interval   = 0
  escalation_message  = "CPU still high after 10 minutes. @slack-channel-critical"

  # Define the PagerDuty notification for this monitor
  # Ensure the PagerDuty integration is set up in Datadog
  # The @pagerduty-service-YOUR_INTEGRATION_KEY syntax is how Datadog maps to PagerDuty services
  # `pagerduty_service_integration_key` here refers to the key obtained from Datadog's PagerDuty integration setup.
}
    

4. Outputs (`outputs.tf`)

Define outputs to easily retrieve important information about your deployed infrastructure.

output "eks_cluster_name" {
  description = "The name of the EKS cluster."
  value       = aws_eks_cluster.eks_cluster.name
}

output "eks_cluster_endpoint" {
  description = "The endpoint for the EKS cluster."
  value       = aws_eks_cluster.eks_cluster.endpoint
}

output "kubeconfig_command" {
  description = "Command to update your local kubeconfig to connect to the EKS cluster."
  value       = "aws eks update-kubeconfig --name ${aws_eks_cluster.eks_cluster.name} --region ${var.aws_region}"
}
    

Deployment Steps

Follow these steps to deploy your EKS cluster with Datadog and PagerDuty integration:

Initialize Terraform: Navigate to your project directory and run `terraform init`. This downloads the necessary providers.
Set Variables: Create a `terraform.tfvars` file or use environment variables to provide values for `datadog_api_key`, `datadog_app_key`, and `pagerduty_service_integration_key`.
datadog_api_key = "YOUR_DATADOG_API_KEY" datadog_app_key = "YOUR_DATADOG_APP_KEY" pagerduty_service_integration_key = "YOUR_PAGERDUTY_INTEGRATION_KEY_FROM_DATADOG"

Important: The `pagerduty_service_integration_key` should be the unique identifier Datadog provides for a PagerDuty service integration configured within the Datadog UI.
Review Plan: Run `terraform plan` to see the infrastructure changes Terraform will make. Carefully review the output.
Apply Changes: Execute `terraform apply` and type `yes` when prompted to provision the resources. This process can take 15-20 minutes as AWS provisions the EKS cluster and nodes.

Verification

After `terraform apply` completes successfully:

Configure kubectl: Use the `kubeconfig_command` outputted by Terraform to update your local kubeconfig:
aws eks update-kubeconfig --name my-eks-cluster --region us-east-1
Verify EKS Nodes:
kubectl get nodes

You should see your worker nodes in a `Ready` state.
Verify Datadog Agent:
kubectl get pods -n datadog

All Datadog pods (Agent, Cluster Agent, Cluster Checks Runner) should be running.
Check Datadog Dashboard: Log in to your Datadog account. You should see your EKS nodes, pods, and container metrics populating in the infrastructure list and Kubernetes dashboards. The monitor you created should also be visible under Monitors > Monitor Status.
Test PagerDuty Integration: To test the PagerDuty integration, you could intentionally cause a high CPU load on one of your EKS nodes (e.g., run a busybox container with CPU-intensive tasks) and observe if the Datadog monitor triggers an alert in PagerDuty.

Troubleshooting and Best Practices

Common Issues

IAM Permissions: Most issues with EKS provisioning stem from incorrect IAM roles or policies. Ensure your AWS CLI user, EKS cluster role, and EKS node role have all required permissions.
Networking: Verify your VPC, subnets, and security groups are correctly configured to allow communication between EKS components and the internet (for Datadog agent outbound calls).
Datadog Agent Connectivity: If Datadog data isn't showing up, check the Datadog Agent logs (`kubectl logs -f -n datadog`). Ensure your API and APP keys are correct and there are no network restrictions.
PagerDuty Integration Key: Double-check that the `pagerduty_service_integration_key` used in the Datadog monitor matches an active PagerDuty integration within Datadog that points to the correct PagerDuty service.

Advanced Considerations & Best Practices

Secrets Management: Never hardcode API keys. Use AWS Secrets Manager, AWS Parameter Store, or HashiCorp Vault to securely store and retrieve `datadog_api_key`, `datadog_app_key`, and `pagerduty_service_integration_key`. You can then dynamically inject these into Terraform or the Helm release.
Terraform Modules: For production-grade deployments, leverage community or custom Terraform modules for EKS (e.g., `terraform-aws-modules/eks/aws`) to handle networking, IAM roles, and node groups more robustly.
Dedicated Subnets: Use private subnets for EKS worker nodes and control plane endpoints, leveraging NAT Gateways for outbound internet access to enhance security.
Service Accounts for Datadog: For granular permissions, configure the Datadog Agent to run with an IAM Role for Service Accounts (IRSA) for specific AWS permissions (e.g., for collecting metrics from EC2, S3, RDS).
Comprehensive Monitoring: Beyond basic CPU alerts, implement a full suite of Datadog monitors for pod health, memory usage, network I/O, application-specific metrics, and security events.
Alert Fatigue: Design your Datadog monitors to be actionable. Use composites, groups, and clear thresholds to avoid alert fatigue. PagerDuty's incident grouping features can also help.
CI/CD Integration: Automate `terraform plan` and `terraform apply` within your CI/CD pipeline for true GitOps practices, ensuring all infrastructure changes are version-controlled and reviewed.

Conclusion

By following this guide, you have successfully provisioned an AWS EKS cluster with Terraform, integrated Datadog for comprehensive observability, and established PagerDuty for effective incident management. This robust setup provides the foundation for deploying and operating highly available, performant, and observable containerized applications. Embracing Infrastructure as Code practices ensures consistency, repeatability, and scalability in your cloud operations, making your DevOps workflow more efficient and reliable.

CloudStack YAML