Terraform-Driven AWS EKS Operations with Datadog Monitoring and PagerDuty Incident Response: A Comprehensive Guide

In the fast-evolving landscape of cloud-native development, managing complex Kubernetes environments efficiently is paramount. This guide provides a detailed walkthrough for establishing a robust, automated, and observable AWS Elastic Kubernetes Service (EKS) infrastructure using Terraform for infrastructure as code, integrated with Datadog for comprehensive monitoring, and PagerDuty for streamlined incident response. This powerful trifecta ensures high availability, operational excellence, and rapid problem resolution.

Architecture Pro-Tip: Modular Design is Key

When building your EKS infrastructure with Terraform, prioritize a modular design. Separate your core VPC, EKS cluster, node groups, and addon deployments into distinct Terraform modules. This enhances reusability, simplifies updates, and enforces clearer separation of concerns, crucial for maintaining complex cloud-native environments. Always implement least-privilege IAM roles and network policies from the outset to bolster security.

Why This Modern DevOps Stack?

Adopting a cohesive stack of Terraform, AWS EKS, Datadog, and PagerDuty delivers significant operational advantages:

Terraform for Infrastructure as Code (IaC): Enables declarative provisioning and management of cloud resources. It ensures environment consistency, reduces manual errors, and facilitates version control and collaboration.
AWS EKS for Kubernetes Orchestration: A fully managed Kubernetes service that simplifies the deployment, management, and scaling of containerized applications. EKS integrates seamlessly with other AWS services, offering high availability and robust security.
Datadog for Unified Observability: Provides end-to-end visibility across your entire stack – from infrastructure and applications to logs and traces. Its comprehensive dashboards, powerful analytics, and intelligent alerting capabilities are indispensable for proactive monitoring.
PagerDuty for Incident Response Automation: Transforms monitoring alerts into actionable incidents, routing them to the right teams via on-call schedules and escalation policies. This minimizes Mean Time To Resolution (MTTR) and improves operational efficiency.

Core Components Overview

Terraform-Managed AWS EKS Deployment

Terraform will be used to define and provision every aspect of your EKS environment, including:

VPC and Networking: Subnets, route tables, security groups, and NAT gateways essential for EKS operation.
EKS Cluster: The control plane, version, and associated IAM roles.
Node Groups: EC2 instances (managed or unmanaged) that act as worker nodes, scaling policies, and their respective IAM instance profiles.
AWS IAM: Roles and policies for EKS to interact with other AWS services, and for Kubernetes service accounts.
Addons: Core Kubernetes components like the AWS Load Balancer Controller, EKS Pod Identity, and Cluster Autoscaler.

Datadog for Comprehensive Monitoring

Datadog will collect, aggregate, and visualize metrics, logs, and traces from your EKS cluster:

Datadog Agent: Deployed as a DaemonSet on your EKS cluster to collect host-level metrics, container metrics, Kubernetes events, and logs.
APM and Distributed Tracing: Gain deep insights into application performance and identify bottlenecks.
Custom Dashboards: Create tailored dashboards to monitor critical KPIs, resource utilization, and application health.
Alerting: Set up intelligent monitors with conditional logic to trigger alerts based on anomalies or thresholds.

PagerDuty for Actionable Incident Response

PagerDuty acts as the central hub for incident management:

Service Directory: Define services corresponding to your applications or infrastructure components.
Escalation Policies: Configure rules for how incidents escalate through different team members or roles.
On-Call Schedules: Manage rotating on-call shifts to ensure continuous coverage.
Datadog Integration: Connect Datadog monitors directly to PagerDuty services to automatically trigger incidents.

Implementing the Solution

Prerequisites

Before you begin, ensure you have the following:

AWS Account with appropriate permissions.
AWS CLI configured.
Terraform (v1.0+) installed.
kubectl installed and configured.
Datadog API and APP keys.
PagerDuty API key and a configured service.

Terraform Setup for AWS EKS

We'll use a simplified structure for demonstrating the core EKS setup. In a production environment, consider using the official terraform-aws-modules/eks/aws module for a more comprehensive and robust solution.

Datadog Agent Deployment on EKS

Once your EKS cluster is provisioned, you'll deploy the Datadog Agent as a DaemonSet across all worker nodes. This ensures comprehensive data collection.

Datadog-PagerDuty Integration

Integrate Datadog with PagerDuty to automatically trigger incidents from your Datadog monitors:

In PagerDuty: Create a new service (e.g., "EKS Critical Alerts") and add a "Datadog" integration. Copy the generated Integration Key.
In Datadog: Navigate to Integrations -> PagerDuty. Add your PagerDuty Integration Key to establish the connection.
Configure Monitors: When creating or editing a Datadog monitor, in the "Notify your team" section, select @pagerduty and specify the service you created (e.g., @pagerduty-eks-critical-alerts).

Ready-to-Use Configuration Snippets

Here are simplified examples to get you started. Remember to replace placeholders (e.g., <YOUR_REGION>, <YOUR_DATADOG_API_KEY>) with your actual values.

Terraform for a Basic EKS Cluster and Node Group

provider "aws" {
  region = "<YOUR_REGION>"
}

resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name = "eks-vpc"
  }
}

resource "aws_subnet" "public" {
  count             = 2
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.0.${count.index}.0/24"
  availability_zone = data.aws_availability_zones.available.names[count.index]
  map_public_ip_on_launch = true

  tags = {
    Name                                       = "eks-public-subnet-${count.index}"
    "kubernetes.io/cluster/eks-cluster-example" = "shared"
    "kubernetes.io/role/elb"                   = "1"
  }
}

resource "aws_subnet" "private" {
  count             = 2
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.0.${count.index + 100}.0/24"
  availability_zone = data.aws_availability_zones.available.names[count.index]

  tags = {
    Name                                       = "eks-private-subnet-${count.index}"
    "kubernetes.io/cluster/eks-cluster-example" = "shared"
    "kubernetes.io/role/internal-elb"          = "1"
  }
}

resource "aws_internet_gateway" "gw" {
  vpc_id = aws_vpc.main.id

  tags = {
    Name = "eks-igw"
  }
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.gw.id
  }

  tags = {
    Name = "eks-public-rt"
  }
}

resource "aws_route_table_association" "public" {
  count          = length(aws_subnet.public)
  subnet_id      = aws_subnet.public[count.index].id
  route_table_id = aws_route_table.public.id
}

resource "aws_eip" "nat" {
  count  = length(aws_subnet.public)
  vpc    = true
}

resource "aws_nat_gateway" "nat" {
  count         = length(aws_subnet.public)
  allocation_id = aws_eip.nat[count.index].id
  subnet_id     = aws_subnet.public[count.index].id

  tags = {
    Name = "eks-nat-gw-${count.index}"
  }
}

resource "aws_route_table" "private" {
  count  = length(aws_subnet.private)
  vpc_id = aws_vpc.main.id

  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.nat[count.index].id
  }

  tags = {
    Name = "eks-private-rt-${count.index}"
  }
}

resource "aws_route_table_association" "private" {
  count          = length(aws_subnet.private)
  subnet_id      = aws_subnet.private[count.index].id
  route_table_id = aws_route_table.private[count.index].id
}

data "aws_availability_zones" "available" {
  state = "available"
}

resource "aws_iam_role" "eks_cluster_role" {
  name = "eks-cluster-role-example"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "eks.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
  role       = aws_iam_role.eks_cluster_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
}

resource "aws_iam_role_policy_attachment" "eks_service_policy" {
  role       = aws_iam_role.eks_cluster_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
}

resource "aws_eks_cluster" "example" {
  name     = "eks-cluster-example"
  role_arn = aws_iam_role.eks_cluster_role.arn
  version  = "1.28" # Specify your desired Kubernetes version

  vpc_config {
    subnet_ids = concat(aws_subnet.public.*.id, aws_subnet.private.*.id)
    security_group_ids = []
    endpoint_private_access = false
    endpoint_public_access  = true
  }

  tags = {
    Environment = "development"
    Project     = "EKS-Guide"
  }

  # Ensure that the EKS cluster is created before managing worker nodes
  depends_on = [
    aws_iam_role_policy_attachment.eks_cluster_policy,
    aws_iam_role_policy_attachment.eks_service_policy,
  ]
}

resource "aws_iam_role" "eks_node_role" {
  name = "eks-node-role-example"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_worker_node_policy" {
  role       = aws_iam_role.eks_node_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
}

resource "aws_iam_role_policy_attachment" "eks_cni_policy" {
  role       = aws_iam_role.eks_node_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
}

resource "aws_iam_role_policy_attachment" "ec2_container_registry_readonly" {
  role       = aws_iam_role.eks_node_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
}

resource "aws_eks_node_group" "example" {
  cluster_name    = aws_eks_cluster.example.name
  node_group_name = "eks-node-group-example"
  node_role_arn   = aws_iam_role.eks_node_role.arn
  subnet_ids      = aws_subnet.private.*.id
  instance_types  = ["t3.medium"] # Choose appropriate instance type
  disk_size       = 20
  capacity_type   = "ON_DEMAND"

  scaling_config {
    desired_size = 2
    max_size     = 3
    min_size     = 1
  }

  update_config {
    max_unavailable = 1
  }

  labels = {
    "node-type" = "general"
  }

  tags = {
    Environment = "development"
    Project     = "EKS-Guide"
  }

  depends_on = [
    aws_iam_role_policy_attachment.eks_worker_node_policy,
    aws_iam_role_policy_attachment.eks_cni_policy,
    aws_iam_role_policy_attachment.ec2_container_registry_readonly,
  ]
}

output "kubeconfig_command" {
  value = "aws eks update-kubeconfig --name ${aws_eks_cluster.example.name} --region ${var.aws_region}"
  description = "Command to update your local kubeconfig to connect to the EKS cluster."
}

Kubernetes Manifest for Datadog Agent (DaemonSet)

apiVersion: v1
kind: Namespace
metadata:
  name: datadog
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: datadog-agent
rules:
  - apiGroups: [""]
    resources: ["pods", "nodes", "services", "endpoints"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["replicasets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["extensions"]
    resources: ["replicasets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["autoscaling"]
    resources: ["horizontalpodautoscalers"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["policy"]
    resources: ["poddisruptionbudgets"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: datadog-agent
subjects:
  - kind: ServiceAccount
    name: datadog-agent
    namespace: datadog
roleRef:
  kind: ClusterRole
  name: datadog-agent
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: datadog-agent
  namespace: datadog
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: datadog-agent
  namespace: datadog
spec:
  selector:
    matchLabels:
      app: datadog-agent
  template:
    metadata:
      labels:
        app: datadog-agent
      name: datadog-agent
    spec:
      serviceAccountName: datadog-agent
      containers:
        - name: datadog-agent
          image: "gcr.io/datadog-prod/agent:latest" # Use the latest stable agent image
          env:
            - name: DD_API_KEY
              valueFrom:
                secretKeyRef:
                  name: datadog-secret
                  key: api-key
            - name: DD_SITE
              value: "datadoghq.com" # Or "eu.datadoghq.com" for Europe
            - name: DD_KUBERNETES_KUBELET_HOST
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: DD_KUBERNETES_KUBELET_PORT
              value: "10250"
            - name: DD_COLLECT_KUBERNETES_EVENTS
              value: "true"
            - name: DD_LOGS_ENABLED
              value: "true"
            - name: DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL
              value: "true"
            - name: DD_PROCESS_AGENT_ENABLED
              value: "true"
            - name: DD_APM_ENABLED
              value: "true"
            - name: DD_TAGS
              value: "env:development,cluster:eks-cluster-example" # Add relevant tags
          resources:
            requests:
              memory: "256Mi"
              cpu: "200m"
            limits:
              memory: "512Mi"
              cpu: "500m"
          volumeMounts:
            - name: proc
              mountPath: /host/proc
              readOnly: true
            - name: cgroup
              mountPath: /host/sys/fs/cgroup
              readOnly: true
            - name: run
              mountPath: /var/run/docker.sock
              readOnly: true
            - name: logpodpath
              mountPath: /var/log/pods
              readOnly: true
            - name: logcontainerpath
              mountPath: /var/lib/docker/containers
              readOnly: true
      volumes:
        - name: proc
          hostPath:
            path: /proc
        - name: cgroup
          hostPath:
            path: /sys/fs/cgroup
        - name: run
          hostPath:
            path: /var/run/docker.sock
        - name: logpodpath
          hostPath:
            path: /var/log/pods
        - name: logcontainerpath
          hostPath:
            path: /var/lib/docker/containers
---
apiVersion: v1
kind: Secret
metadata:
  name: datadog-secret
  namespace: datadog
type: Opaque
stringData:
  api-key: "<YOUR_DATADOG_API_KEY>"

Advanced Monitoring & Alerting Strategies

To maximize the value of your monitoring and incident response stack:

Service-Level Objectives (SLOs): Define SLOs in Datadog to track the reliability of your services. PagerDuty can then be triggered if SLOs are at risk.
Anomaly Detection: Leverage Datadog's machine learning capabilities to detect unusual patterns in your metrics, providing early warnings before issues escalate.
Log Management & Analysis: Centralize all EKS and application logs in Datadog. Use log patterns and attributes to create robust monitors and accelerate troubleshooting.
Synthetic Monitoring: Implement synthetic checks in Datadog to simulate user journeys and proactively test the availability and performance of your applications.
Runbooks in PagerDuty: Attach automated or manual runbooks to PagerDuty services to guide on-call responders through initial diagnostic and resolution steps.

Best Practices for Production Environments

Terraform State Management: Use remote state (e.g., S3 backend with DynamoDB locking) and implement state locking to prevent concurrent modifications.
IAM Least Privilege: Configure IAM roles with the absolute minimum permissions required for each component (EKS cluster, node groups, service accounts).
Network Policies: Implement Kubernetes Network Policies to control traffic flow between pods, enhancing security posture.
Regular Updates: Keep your EKS cluster, worker nodes, and Datadog Agent up-to-date with the latest versions and security patches.
Cost Optimization: Utilize EC2 Spot Instances for non-critical workloads in separate node groups to reduce costs. Monitor resource usage in Datadog to right-size your nodes.
Incident Post-Mortems: Conduct regular post-mortems for critical incidents to identify root causes, implement preventative measures, and refine your monitoring and response strategies.

Troubleshooting & Conclusion

Common Troubleshooting Areas

EKS Cluster Creation Issues: Often related to IAM permissions for the EKS service role or incorrect VPC/subnet configurations. Check CloudTrail logs and EKS console events.
Node Group Joining Issues: Verify the node IAM role has the correct policies (AmazonEKSWorkerNodePolicy, AmazonEKS_CNI_Policy, AmazonEC2ContainerRegistryReadOnly) and that the Security Group allows necessary control plane communication.
Datadog Agent Not Reporting: Ensure DD_API_KEY is correctly set in the secret and the agent's DaemonSet. Check Datadog Agent logs (kubectl logs -n datadog <datadog-agent-pod>) for errors.
PagerDuty Alerts Not Triggering: Confirm the Datadog-PagerDuty integration is active and correctly configured in Datadog. Test your monitors with a low threshold to ensure they trigger.

Conclusion

By leveraging Terraform for declarative infrastructure, AWS EKS for scalable container orchestration, Datadog for unified observability, and PagerDuty for intelligent incident response, organizations can achieve a highly automated, resilient, and efficient cloud-native operational model. This integrated approach not only streamlines deployment and management but also significantly improves your team's ability to monitor performance, detect issues proactively, and resolve incidents with speed and confidence. Embrace this powerful stack to elevate your DevOps practices and deliver superior service reliability.

CloudStack YAML