Automating AWS EKS Cluster Configuration with Terraform, Datadog Observability, and PagerDuty Alerting

Architecture Pro-Tip: Modular Design & Security First

For production-grade EKS deployments, adopt a modular Terraform structure (e.g., using `modules/vpc`, `modules/eks`, `modules/observability`). Implement strict IAM policies with the principle of least privilege for all components (EKS service role, node group role, Datadog IRSA). Leverage AWS Organizations and a multi-account strategy for environment separation (dev, staging, prod) to enhance security and isolation. Always encrypt sensitive data at rest and in transit.

Automating AWS EKS Cluster Configuration with Terraform, Datadog Observability, and PagerDuty Alerting

In today's dynamic cloud-native landscape, managing Kubernetes clusters manually is no longer sustainable. This comprehensive guide will walk you through automating the deployment and configuration of an AWS Elastic Kubernetes Service (EKS) cluster using Terraform, integrating robust observability with Datadog, and establishing critical incident alerting with PagerDuty. By the end, you'll have a production-ready, fully automated EKS environment with powerful monitoring and on-call capabilities.

Why Automate Your EKS Infrastructure?

Automating your EKS setup brings immense benefits:

  • Consistency: Eliminate configuration drift and ensure identical environments across development, staging, and production.
  • Speed: Rapidly provision new EKS clusters or replicate existing ones in minutes.
  • Reliability: Reduce human error and ensure deployments adhere to best practices.
  • Auditability: Track infrastructure changes through version control systems.
  • Disaster Recovery: Rebuild your entire EKS infrastructure quickly and reliably in case of an incident.

Prerequisites

Before you begin, ensure you have the following tools and accounts set up:

  • An AWS Account with Administrator access.
  • AWS CLI configured with appropriate credentials.
  • Terraform installed (v1.0+ recommended).
  • kubectl installed.
  • Helm CLI installed (v3+ recommended).
  • A Datadog Account with API and Application Keys.
  • A PagerDuty Account with an API Token.

Core Terraform EKS Configuration

We'll start by defining the fundamental AWS resources required for an EKS cluster: a Virtual Private Cloud (VPC), subnets, the EKS cluster itself, and its associated node groups.

1. Provider Configuration

Set up the AWS, Kubernetes, Helm, Datadog, and PagerDuty providers.

provider "aws" {
  region = "us-east-1"
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.main.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.main.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.main.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.main.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

data "aws_eks_cluster" "main" {
  name = aws_eks_cluster.main.name
}

data "aws_eks_cluster_auth" "main" {
  name = aws_eks_cluster.main.name
}

2. VPC and Networking

EKS requires a well-configured VPC with public and private subnets. For brevity, we'll assume a simplified setup or an existing VPC. In production, use dedicated modules for this.

# Example: Using an existing VPC and subnets (recommended for modularity)
data "aws_vpc" "selected" {
  tags = {
    Name = "my-eks-vpc"
  }
}

data "aws_subnets" "private" {
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.selected.id]
  }
  tags = {
    Tier = "private"
  }
}

data "aws_subnets" "public" {
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.selected.id]
  }
  tags = {
    Tier = "public"
  }
}

3. EKS Cluster Definition

Define the EKS cluster and its IAM role. The role needs permissions for EKS to manage cluster resources.

resource "aws_iam_role" "eks_cluster" {
  name = "eks-cluster-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "eks.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eks_cluster.name
}

resource "aws_eks_cluster" "main" {
  name     = "my-automated-eks-cluster"
  role_arn = aws_iam_role.eks_cluster.arn
  version  = "1.28" # Specify your desired Kubernetes version

  vpc_config {
    subnet_ids         = data.aws_subnets.private.ids # EKS uses private subnets for control plane ENIs
    security_group_ids = [aws_security_group.eks_cluster.id]
    endpoint_private_access = true
    endpoint_public_access  = true
  }

  tags = {
    Environment = "production"
    ManagedBy   = "Terraform"
  }
}

4. EKS Node Groups

Node groups host your Kubernetes pods. Define their IAM role and attach necessary policies.

resource "aws_iam_role" "eks_nodes" {
  name = "eks-node-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_nodes_amazon_eks_worker_node_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.eks_nodes.name
}

resource "aws_iam_role_policy_attachment" "eks_nodes_amazon_eks_cni_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.eks_nodes.name
}

resource "aws_iam_role_policy_attachment" "eks_nodes_amazon_ec2_container_registry_read_only" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.eks_nodes.name
}

resource "aws_eks_node_group" "private" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "private-nodes"
  node_role_arn   = aws_iam_role.eks_nodes.arn
  subnet_ids      = data.aws_subnets.private.ids
  instance_types  = ["t3.medium"] # Choose appropriate instance types
  desired_size    = 2
  max_size        = 4
  min_size        = 1

  scaling_config {
    desired_size = 2
    max_size     = 4
    min_size     = 1
  }

  remote_access {
    ec2_ssh_key = "your-ssh-key-name" # Optional: for direct SSH access
  }

  labels = {
    "node.kubernetes.io/lifecycle" = "on-demand"
  }

  tags = {
    Environment = "production"
    ManagedBy   = "Terraform"
  }

  depends_on = [
    aws_iam_role_policy_attachment.eks_nodes_amazon_eks_worker_node_policy,
    aws_iam_role_policy_attachment.eks_nodes_amazon_eks_cni_policy,
    aws_iam_role_policy_attachment.eks_nodes_amazon_ec2_container_registry_read_only,
  ]
}

Integrating Datadog for Observability

Datadog provides comprehensive monitoring for your EKS cluster, applications, and infrastructure. We'll deploy the Datadog Agent using Helm and configure IAM Roles for Service Accounts (IRSA) for secure access.

1. Set up IAM Role for Service Accounts (IRSA) for Datadog

IRSA allows Kubernetes service accounts to assume IAM roles, providing granular permissions to the Datadog Agent.

resource "aws_iam_openid_connect_provider" "main" {
  url = aws_eks_cluster.main.identity[0].oidc[0].issuer
  client_id_list = ["sts.amazonaws.com"]
  thumbprint_list = [data.tls_certificate.main.certificates[0].sha1_fingerprint]
}

data "tls_certificate" "main" {
  url = aws_eks_cluster.main.identity[0].oidc[0].issuer
}

resource "aws_iam_policy" "datadog_agent_policy" {
  name        = "DatadogAgentPolicy-${aws_eks_cluster.main.name}"
  description = "IAM Policy for Datadog Agent on EKS"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "ec2:DescribeInstances",
          "ec2:DescribeRegions",
          "ec2:DescribeTags",
          "eks:ListTagsForResource",
          "eks:DescribeCluster",
          "autoscaling:DescribeAutoScalingGroups",
          "logs:DescribeLogGroups",
          "logs:DescribeLogStreams",
          "logs:GetLogEvents",
          "sqs:ListQueues",
          "lambda:ListFunctions",
          "lambda:GetFunctionConfiguration",
          "tag:GetResources",
        ]
        Effect   = "Allow"
        Resource = "*"
      },
    ]
  })
}

resource "aws_iam_role" "datadog_agent_role" {
  name = "DatadogAgentRole-${aws_eks_cluster.main.name}"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Federated = aws_iam_openid_connect_provider.main.arn
        }
        Action = "sts:AssumeRoleWithWebIdentity"
        Condition = {
          StringEquals = {
            "${replace(aws_iam_openid_connect_provider.main.url, "https://", "")}:sub" = "system:serviceaccount:datadog:datadog-agent"
          }
        }
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "datadog_agent_policy_attach" {
  policy_arn = aws_iam_policy.datadog_agent_policy.arn
  role       = aws_iam_role.datadog_agent_role.name
}

2. Deploy Datadog Agent with Helm

Use the Helm provider to deploy the Datadog Agent, configured to use the IRSA role.

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }

  set {
    name  = "datadog.site"
    value = "datadoghq.com" # Or eu.datadoghq.com, etc.
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "kubeStateMetricsExternal.enabled"
    value = "true"
  }

  # Enable IRSA
  set {
    name  = "datadog.aws.enableEKSFargate"
    value = "false" # Set to true if using Fargate profiles
  }
  set {
    name  = "datadog.aws.eks.podIdentity.enabled"
    value = "true"
  }
  set {
    name  = "datadog.aws.eks.podIdentity.awsIamServiceAccount"
    value = "datadog-agent"
  }
  set {
    name  = "datadog.aws.eks.podIdentity.awsIamRoleArn"
    value = aws_iam_role.datadog_agent_role.arn
  }

  set {
    name  = "agents.podLabels.datadog\\.com/exclude"
    value = "false"
  }
  set {
    name  = "clusterAgent.container.env[0].name"
    value = "DD_KUBERNETES_KUBELET_TLS_VERIFY"
  }
  set {
    name  = "clusterAgent.container.env[0].value"
    value = "false"
  }

  # Example: Enable host and process monitoring
  set {
    name  = "agents.enabled"
    value = "true"
  }
  set {
    name  = "processAgent.enabled"
    value = "true"
  }
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }
  set {
    name  = "apm.enabled"
    value = "true"
  }
  set {
    name  = "networkMonitoring.enabled"
    value = "true"
  }

  depends_on = [
    aws_eks_cluster.main,
    aws_eks_node_group.private,
    aws_iam_role_policy_attachment.datadog_agent_policy_attach
  ]
}

3. Datadog Monitors (Example)

Create Datadog monitors to alert on critical EKS metrics. These can later be linked to PagerDuty.

resource "datadog_monitor" "eks_node_cpu_utilization" {
  name                = "[EKS] High Node CPU Utilization on {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:my-automated-eks-cluster} by {host} > 80"
  message             = "EKS Node {{host.name}} CPU utilization is high! Current: {{value}}%. Check for runaway processes or scale nodes. @webhook-pagerduty"
  monitor_thresholds {
    critical = 80
    warning  = 70
  }
  notify_no_data      = false
  renotify_interval   = 0
  tags                = ["environment:production", "service:eks", "alert-type:performance"]
  require_full_window = false
  timeout_h           = 0
}

resource "datadog_monitor" "eks_node_memory_utilization" {
  name                = "[EKS] High Node Memory Utilization on {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.memory.usage.total{cluster_name:my-automated-eks-cluster} by {host} > 75"
  message             = "EKS Node {{host.name}} memory utilization is high! Current: {{value}}%. Check for memory leaks or scale nodes. @webhook-pagerduty"
  monitor_thresholds {
    critical = 75
    warning  = 65
  }
  notify_no_data      = false
  renotify_interval   = 0
  tags                = ["environment:production", "service:eks", "alert-type:performance"]
  require_full_window = false
  timeout_h           = 0
}

PagerDuty for Alerting

PagerDuty provides robust incident management and on-call scheduling. We'll define a PagerDuty service and link Datadog monitors to it.

1. PagerDuty Escalation Policy and Service

First, define an escalation policy and then a service that uses it. This ensures incidents are routed to the correct teams.

# Example: PagerDuty User (replace with your actual user IDs)
data "pagerduty_user" "admin" {
  email = "admin@example.com"
}

# Example: PagerDuty Escalation Policy
resource "pagerduty_escalation_policy" "eks_escalation_policy" {
  name      = "EKS Critical Escalation Policy"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "user"
      id   = data.pagerduty_user.admin.id # Or use team/schedule IDs
    }
  }

  # Add more rules for additional escalation steps
}

# PagerDuty Service for EKS alerts
resource "pagerduty_service" "eks_service" {
  name                      = "AWS EKS Cluster Alerts"
  auto_resolve_timeout      = 14400 # 4 hours
  acknowledgement_timeout   = 600   # 10 minutes
  escalation_policy         = pagerduty_escalation_policy.eks_escalation_policy.id
  alert_creation            = "create_alerts_and_incidents"
  alert_grouping            = "time"
  alert_grouping_timeout    = 5
  description               = "Handles critical alerts for the automated EKS cluster."

  incident_urgency_rule {
    type = "constant"
    urgency = "high"
  }
}

2. Connecting Datadog Monitors to PagerDuty

Integrate Datadog with PagerDuty. The `datadog_integration_pagerduty` resource creates the integration within Datadog, which then generates a webhook URL. You'll update your Datadog monitors to use this integration.

resource "datadog_integration_pagerduty" "eks_integration" {
  services {
    service_name = pagerduty_service.eks_service.name
    service_key  = pagerduty_service.eks_service.integration[0].integration_key
  }

  # Make sure this is linked to your Datadog monitors
  # The 'service_key' is dynamically created when a PagerDuty integration is added to a service.
  # The datadog_monitor message field then references '@webhook-pagerduty'
  # and Datadog's PagerDuty integration handles the routing.
  # Alternatively, you can use `datadog_monitor_pagerduty_integration` resource for direct linking.
}

# Update the Datadog monitors to include the PagerDuty integration
# We already did this in the Datadog Monitors section by adding `@webhook-pagerduty`
# Datadog will automatically route alerts to the PagerDuty service associated with the configured webhook.

Copy & Paste Configuration Example

This simplified example combines the core components into a single `main.tf` for quick setup. Remember to replace placeholders like API keys, SSH key names, and emails with your actual values. For a production environment, structure this into separate modules.

resource "aws_iam_role" "eks_cluster" {
  name = "eks-cluster-role-example"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Service = "eks.amazonaws.com"
      }
      Action = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eks_cluster.name
}

resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true
  tags = { Name = "eks-vpc-example" }
}

resource "aws_subnet" "private" {
  count             = 2
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.0.${count.index + 1}.0/24"
  availability_zone = data.aws_availability_zones.available.names[count.index]
  tags = { Name = "eks-private-subnet-${count.index}", Tier = "private" }
}

data "aws_availability_zones" "available" {
  state = "available"
}

resource "aws_eks_cluster" "main" {
  name     = "my-automated-eks-cluster-example"
  role_arn = aws_iam_role.eks_cluster.arn
  version  = "1.28"
  vpc_config {
    subnet_ids = aws_subnet.private[*].id
    endpoint_private_access = true
    endpoint_public_access  = true
  }
  tags = { Environment = "production" }
}

resource "aws_iam_role" "eks_nodes" {
  name = "eks-node-role-example"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Service = "ec2.amazonaws.com"
      }
      Action = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy_attachment" "eks_nodes_amazon_eks_worker_node_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.eks_nodes.name
}

resource "aws_iam_role_policy_attachment" "eks_nodes_amazon_eks_cni_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.eks_nodes.name
}

resource "aws_iam_role_policy_attachment" "eks_nodes_amazon_ec2_container_registry_read_only" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.eks_nodes.name
}

resource "aws_eks_node_group" "private" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "private-nodes-example"
  node_role_arn   = aws_iam_role.eks_nodes.arn
  subnet_ids      = aws_subnet.private[*].id
  instance_types  = ["t3.medium"]
  desired_size    = 2
  max_size        = 4
  min_size        = 1
  scaling_config {
    desired_size = 2
    max_size     = 4
    min_size     = 1
  }
  remote_access {
    ec2_ssh_key = "your-ssh-key-name" # Replace with your key
  }
  tags = { Environment = "production" }
  depends_on = [
    aws_iam_role_policy_attachment.eks_nodes_amazon_eks_worker_node_policy,
    aws_iam_role_policy_attachment.eks_nodes_amazon_eks_cni_policy,
    aws_iam_role_policy_attachment.eks_nodes_amazon_ec2_container_registry_read_only,
  ]
}

data "aws_eks_cluster" "main" {
  name = aws_eks_cluster.main.name
}

data "aws_eks_cluster_auth" "main" {
  name = aws_eks_cluster.main.name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.main.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.main.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.main.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.main.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

# Datadog IRSA Setup
resource "aws_iam_openid_connect_provider" "main" {
  url            = aws_eks_cluster.main.identity[0].oidc[0].issuer
  client_id_list = ["sts.amazonaws.com"]
  thumbprint_list = [data.tls_certificate.main.certificates[0].sha1_fingerprint]
}

data "tls_certificate" "main" {
  url = aws_eks_cluster.main.identity[0].oidc[0].issuer
}

resource "aws_iam_policy" "datadog_agent_policy" {
  name        = "DatadogAgentPolicy-${aws_eks_cluster.main.name}"
  description = "IAM Policy for Datadog Agent on EKS"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = [
        "ec2:DescribeInstances", "ec2:DescribeRegions", "ec2:DescribeTags",
        "eks:ListTagsForResource", "eks:DescribeCluster",
        "autoscaling:DescribeAutoScalingGroups",
        "logs:DescribeLogGroups", "logs:DescribeLogStreams", "logs:GetLogEvents",
        "sqs:ListQueues", "lambda:ListFunctions", "lambda:GetFunctionConfiguration",
        "tag:GetResources",
      ]
      Effect   = "Allow"
      Resource = "*"
    }]
  })
}

resource "aws_iam_role" "datadog_agent_role" {
  name = "DatadogAgentRole-${aws_eks_cluster.main.name}"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Federated = aws_iam_openid_connect_provider.main.arn
      }
      Action = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "${replace(aws_iam_openid_connect_provider.main.url, "https://", "")}:sub" = "system:serviceaccount:datadog:datadog-agent"
        }
      }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "datadog_agent_policy_attach" {
  policy_arn = aws_iam_policy.datadog_agent_policy.arn
  role       = aws_iam_role.datadog_agent_role.name
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true
  set { name = "datadog.apiKey"; value = var.datadog_api_key }
  set { name = "datadog.appKey"; value = var.datadog_app_key }
  set { name = "datadog.site"; value = "datadoghq.com" }
  set { name = "clusterAgent.enabled"; value = "true" }
  set { name = "kubeStateMetricsExternal.enabled"; value = "true" }
  set { name = "datadog.aws.eks.podIdentity.enabled"; value = "true" }
  set { name = "datadog.aws.eks.podIdentity.awsIamServiceAccount"; value = "datadog-agent" }
  set { name = "datadog.aws.eks.podIdentity.awsIamRoleArn"; value = aws_iam_role.datadog_agent_role.arn }
  set { name = "logs.enabled"; value = "true" }
  set { name = "logs.containerCollectAll"; value = "true" }
  set { name = "apm.enabled"; value = "true" }
  depends_on = [
    aws_eks_cluster.main,
    aws_eks_node_group.private,
    aws_iam_role_policy_attachment.datadog_agent_policy_attach
  ]
}

# PagerDuty Setup
resource "pagerduty_escalation_policy" "eks_escalation_policy" {
  name      = "EKS Critical Escalation Policy"
  num_loops = 2
  rule {
    escalation_delay_in_minutes = 10
    target { type = "user"; id = "YOUR_PAGERDUTY_USER_ID" } # REPLACE THIS
  }
}

resource "pagerduty_service" "eks_service" {
  name                      = "AWS EKS Cluster Alerts"
  escalation_policy         = pagerduty_escalation_policy.eks_escalation_policy.id
  alert_creation            = "create_alerts_and_incidents"
  description               = "Handles critical alerts for the automated EKS cluster."
}

resource "datadog_integration_pagerduty" "eks_integration" {
  services {
    service_name = pagerduty_service.eks_service.name
    service_key  = pagerduty_service.eks_service.integration[0].integration_key
  }
}

resource "datadog_monitor" "eks_node_cpu_utilization" {
  name                = "[EKS] High Node CPU Utilization on {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:my-automated-eks-cluster-example} by {host} > 80"
  message             = "EKS Node {{host.name}} CPU utilization is high! Current: {{value}}%. @webhook-pagerduty"
  monitor_thresholds { critical = 80; warning = 70 }
  tags                = ["environment:production", "service:eks"]
  notify_no_data      = false
}

resource "datadog_monitor" "eks_node_memory_utilization" {
  name                = "[EKS] High Node Memory Utilization on {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.memory.usage.total{cluster_name:my-automated-eks-cluster-example} by {host} > 75"
  message             = "EKS Node {{host.name}} memory utilization is high! Current: {{value}}%. @webhook-pagerduty"
  monitor_thresholds { critical = 75; warning = 65 }
  tags                = ["environment:production", "service:eks"]
  notify_no_data      = false
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}

Deployment Steps

Follow these steps to deploy your EKS cluster with integrated observability and alerting:

  1. Save the Terraform configuration (e.g., in `main.tf`) and define your sensitive variables in `terraform.tfvars` or using environment variables.
  2. Initialize your Terraform workspace: `terraform init`
  3. Review the plan: `terraform plan`
  4. Apply the configuration: `terraform apply -auto-approve`
  5. Once applied, configure `kubectl` to interact with your new EKS cluster: 
    aws eks update-kubeconfig --name my-automated-eks-cluster-example --region us-east-1
  6. Verify Datadog Agent pods are running: `kubectl get pods -n datadog`
  7. Check your Datadog dashboard for EKS metrics and confirm PagerDuty integration status.

Best Practices for Production EKS Deployments

  • Modularity: Break your Terraform configuration into logical modules (VPC, EKS, Datadog, PagerDuty) for reusability and manageability.
  • State Management: Use a remote backend like S3 and DynamoDB for Terraform state to enable collaboration and prevent state corruption.
  • Security: Implement network policies (e.g., Calico), restrict public access where possible, use KMS for encryption, and regularly audit IAM roles.
  • Cost Optimization: Utilize Cluster Autoscaler and Karpenter for efficient node scaling, choose appropriate instance types, and consider Spot Instances for fault-tolerant workloads.
  • CI/CD Integration: Integrate your Terraform deployment into a CI/CD pipeline (e.g., GitHub Actions, GitLab CI, AWS CodePipeline) for automated and consistent deployments.
  • Logging: Beyond Datadog metrics, ensure centralized logging (e.g., Fluent Bit to CloudWatch, S3, or Datadog Logs) is configured for all cluster components and applications.
  • Version Control: Keep all your Terraform code under version control (Git) for traceability and collaborative development.

Conclusion

Automating your AWS EKS cluster configuration with Terraform, Datadog, and PagerDuty is a foundational step towards building resilient, scalable, and observable cloud-native applications. This guide provides a robust framework to get started, enabling your teams to focus on innovation rather than infrastructure management. Continuously refine your configurations, monitoring, and alerting strategies to adapt to evolving application needs and maintain optimal operational efficiency.

Comments

Post a Comment

Popular posts from this blog

Terraform Configuration for Datadog-PagerDuty Incident Management on AWS EKS

Terraform Configuration for Datadog-PagerDuty Incident Management on AWS EKS In the fast-paced world of cloud-native operations, ensuring high availability and rapid incident response for applications running on AWS EKS is paramount. This comprehensive guide details how to leverage Terraform to seamlessly integrate Datadog for robust monitoring and PagerDuty for efficient incident management, creating a resilient and automated incident response pipeline for your Kubernetes clusters. Architecture Pro-Tip: Modular Design for Scalability When building your incident management infrastructure with Terraform, consider a modular design. Separate your Datadog monitors, PagerDuty services, and AWS EKS resources into distinct Terraform modules. This approach enhances reusability, improves readability, and makes it significantly easier to manage configurations across multiple environments (development, staging, production) or different EKS clusters. Always priori...
Read more

Terraform-Managed AWS EKS Observability and Incident Response with Datadog and PagerDuty

Terraform-Managed AWS EKS Observability and Incident Response with Datadog and PagerDuty In the dynamic world of cloud-native applications, managing AWS Elastic Kubernetes Service (EKS) requires robust observability and a streamlined incident response strategy. This guide delves into integrating Datadog for comprehensive monitoring and PagerDuty for efficient incident management, all orchestrated declaratively using Terraform. We'll explore how to establish a resilient system that not only detects issues promptly but also ensures critical alerts reach the right teams for rapid resolution, maintaining high availability and operational excellence. Architecture Pro-Tip: Always centralize your observability configuration within your Infrastructure as Code (IaC) repository. Treat your Datadog monitors, dashboards, and PagerDuty services as code alongside your EKS clusters. This ensures version control, simplifies rollbacks, an...
Read more

Terraform for Production AWS EKS Observability with Datadog, Prometheus, and PagerDuty Integration

Terraform for Production AWS EKS Observability with Datadog, Prometheus, and PagerDuty Integration In the dynamic world of cloud-native applications, maintaining peak performance and rapid incident response for AWS Elastic Kubernetes Service (EKS) is paramount. Observability, often misunderstood as mere monitoring, extends to understanding the internal state of a system from its external outputs. This guide provides a comprehensive, technical walkthrough on leveraging Terraform to provision and manage a robust observability stack for your production EKS clusters, integrating the power of Datadog for holistic monitoring, Prometheus for deep Kubernetes-native metrics, and PagerDuty for automated incident management. Architecture Pro-Tip: For a truly resilient and scalable production observability strategy on AWS EKS, aim for a layered approach. Use Datadog as your primary pane of glass for aggregated metrics, logs, and traces, leveraging its native integrat...
Read more