Terraform AWS EKS Datadog and PagerDuty for Enterprise Observability

In the dynamic landscape of modern enterprise IT, maintaining high availability and optimal performance of mission-critical applications deployed on Kubernetes is paramount. AWS Elastic Kubernetes Service (EKS) provides a robust platform for running containerized workloads, but its complexity necessitates a sophisticated observability strategy. This guide details how to leverage Terraform for Infrastructure as Code (IaC) to provision and manage AWS EKS, integrate Datadog for comprehensive monitoring and unified observability, and hook into PagerDuty for streamlined incident response. This holistic approach empowers enterprise DevOps teams to achieve proactive monitoring, rapid incident resolution, and ultimately, superior operational excellence.

Architecture Pro-Tip: Layered Observability Strategy

For true enterprise observability, adopt a layered approach: instrument your application code (APM), monitor your Kubernetes infrastructure (EKS metrics, logs, events), and integrate cloud provider services (AWS EKS logs, CloudWatch). Datadog excels at consolidating these layers, but ensuring proper tagging and naming conventions in Terraform from the outset is crucial for meaningful dashboards and alerts across all levels.

The Pillars of Enterprise Observability

Achieving a resilient and observable EKS environment relies on the synergistic integration of best-of-breed tools:

Terraform (Infrastructure as Code): Automates the provisioning and management of AWS EKS clusters, associated networking, IAM roles, and even the deployment of observability agents. IaC ensures consistency, repeatability, and version control for your infrastructure.
AWS EKS (Container Orchestration): A managed Kubernetes service that simplifies the deployment, management, and scaling of containerized applications in the AWS cloud. It forms the backbone for modern microservices architectures.
Datadog (Unified Observability Platform): Provides end-to-end visibility across your entire technology stack. It collects metrics, logs, traces, and events from EKS, applications, and AWS services, offering powerful dashboards, alerting, and APM capabilities.
PagerDuty (Incident Management): Integrates with Datadog to provide real-time incident alerting, on-call scheduling, escalation policies, and post-incident analysis, ensuring critical issues are addressed promptly by the right team members.

Prerequisites for Implementation

Before diving into the configuration, ensure you have the following:

AWS Account: With necessary permissions to create EKS clusters, IAM roles, VPCs, and EC2 instances.
Terraform CLI: Installed and configured on your local machine or CI/CD environment.
Kubernetes CLI (kubectl): For interacting with your EKS cluster.
Helm CLI: For deploying the Datadog Agent.
Datadog Account: With an API key and Application key.
PagerDuty Account: With a service created and an integration key ready (typically obtained from a Datadog integration within PagerDuty or directly from a PagerDuty service).

Terraform Configuration for Enterprise Observability

This section provides a structured approach to building your EKS, Datadog, and PagerDuty integration using Terraform. We'll cover the core components and their interconnections.

1. AWS Provider and EKS Cluster Setup

Start by defining your AWS provider and setting up a modular EKS cluster. For simplicity, we'll use a basic EKS setup.

2. Datadog Integration for AWS and EKS

Configure the Datadog AWS integration to pull metrics from AWS services, including EKS. Then, deploy the Datadog Agent onto your EKS cluster using a Helm chart via Terraform's helm_release resource.

3. Datadog Monitor and PagerDuty Integration

Define a Datadog monitor that triggers based on EKS metrics (e.g., node CPU utilization). This monitor will then send alerts to PagerDuty.

Note: Ensure you have created the PagerDuty integration in Datadog's integrations settings first. This will provide you with the `@pagerduty` handle or service name to use in your monitor message.

# main.tf

provider "aws" {
  region = "us-east-1"
}

provider "kubernetes" {
  host                   = aws_eks_cluster.main.endpoint
  cluster_ca_certificate = base64decode(aws_eks_cluster.main.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.main.token
}

provider "helm" {
  kubernetes {
    host                   = aws_eks_cluster.main.endpoint
    cluster_ca_certificate = base64decode(aws_eks_cluster.main.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.main.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

# EKS Cluster
resource "aws_eks_cluster" "main" {
  name     = "enterprise-eks"
  role_arn = aws_iam_role.eks_master.arn
  vpc_config {
    subnet_ids         = ["subnet-0abcdef1234567890", "subnet-0fedcba9876543210"] # Replace with your actual subnet IDs
    security_group_ids = [aws_security_group.eks_cluster_sg.id]
  }

  tags = {
    Name        = "enterprise-eks-cluster"
    Environment = "production"
  }
}

resource "aws_security_group" "eks_cluster_sg" {
  name        = "eks-cluster-sg"
  description = "Security group for EKS cluster"
  vpc_id      = "vpc-0123456789abcdef0" # Replace with your actual VPC ID

  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_iam_role" "eks_master" {
  name = "eks-master-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "eks.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eks_master.name
}

resource "aws_iam_role_policy_attachment" "eks_service_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
  role       = aws_iam_role.eks_master.name
}

data "aws_eks_cluster_auth" "main" {
  name = aws_eks_cluster.main.name
}

# EKS Node Group (Example, use a proper module for production)
resource "aws_iam_role" "eks_nodes" {
  name = "eks-node-role"

  assume_role_policy = jsonencode({
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "ec2.amazonaws.com"
      }
    }]
    Version = "2012-10-17"
  })
}

resource "aws_iam_role_policy_attachment" "eks_worker_node_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.eks_nodes.name
}

resource "aws_iam_role_policy_attachment" "eks_cni_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.eks_nodes.name
}

resource "aws_iam_role_policy_attachment" "eks_ec2_container_registry_readonly" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.eks_nodes.name
}

resource "aws_eks_node_group" "main_nodes" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "main-nodes"
  node_role_arn   = aws_iam_role.eks_nodes.arn
  subnet_ids      = ["subnet-0abcdef1234567890", "subnet-0fedcba9876543210"] # Replace with your actual subnet IDs
  instance_types  = ["t3.medium"]
  disk_size       = 20

  scaling_config {
    desired_size = 2
    max_size     = 3
    min_size     = 1
  }

  depends_on = [
    aws_iam_role_policy_attachment.eks_worker_node_policy,
    aws_iam_role_policy_attachment.eks_cni_policy,
    aws_iam_role_policy_attachment.eks_ec2_container_registry_readonly,
  ]
}

# Datadog Agent Deployment via Helm
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Or create a dedicated namespace
  version    = "2.33.0"  # Use a specific version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }

  set {
    name  = "datadog.clusterName"
    value = aws_eks_cluster.main.name
  }

  set {
    name  = "datadog.site"
    value = "datadoghq.com" # Or eu.datadoghq.com etc.
  }

  set {
    name  = "datadog.kubelet.host"
    value = "$${HOST_IP}" # Standard way for Datadog to discover Kubelet
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "agents.tolerations[0].operator"
    value = "Exists"
  }

  # Enable APM, Log Collection, Network Performance Monitoring (NPM)
  set {
    name  = "apm.enabled"
    value = "true"
  }
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }
  set {
    name  = "networkMonitoring.enabled"
    value = "true"
  }

  # For pulling metrics from AWS directly (requires IAM role)
  # This assumes you have set up a separate IAM role for Datadog's AWS integration
  # resource "datadog_integration_aws" and attach policy for read access
}

# Datadog Monitor for EKS Node CPU Utilization
resource "datadog_monitor" "eks_node_cpu_alert" {
  name               = "EKS Node CPU High Utilization"
  type               = "metric alert"
  message            = "High CPU utilization detected on EKS node in {{host.name}}! Team, please investigate immediately. @pagerduty-your-service-name" # Replace 'pagerduty-your-service-name' with your actual Datadog PagerDuty integration handle
  query              = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:enterprise-eks} by {host} > 80"
  monitor_thresholds {
    critical = 80
    warning  = 70
  }
  notify_no_data = false
  renotify_interval = 60 # minutes
  tags = ["environment:production", "service:eks", "team:devops"]
}

# variables.tf

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

Implementing a Comprehensive Observability Strategy

Once your infrastructure is provisioned and agents are deployed, focus on leveraging the collected data:

Metrics Collection: Utilize Datadog to gather granular metrics from EKS nodes, pods, containers, and applications. Create custom dashboards for key performance indicators (KPIs) relevant to your services.
Log Management: Centralize all Kubernetes logs (control plane, node, pod) in Datadog. Use log processing pipelines to extract meaningful attributes, create facets, and generate metrics from logs for advanced analytics and alerting.
APM and Tracing: Instrument your application code with Datadog APM to gain deep visibility into service performance, latency, error rates, and distributed traces, crucial for debugging microservices.
Synthetic Monitoring: Implement synthetic checks (e.g., uptime monitors, API tests) from various geographical locations to proactively detect issues before they impact end-users.
Alerting and Incident Response: Configure sophisticated alerts in Datadog with dynamic thresholds. Ensure these alerts are routed to the correct PagerDuty services, triggering incidents, notifying on-call teams, and orchestrating response workflows based on severity and team ownership.

Best Practices for Enterprise Deployment

To maximize the benefits of this setup:

Modularity: Break down your Terraform configuration into logical modules (e.g., EKS cluster module, Datadog agent module) for better organization, reusability, and maintainability.
State Management: Always use remote state storage (e.g., AWS S3 with DynamoDB locking) for Terraform to enable team collaboration and prevent state corruption.
Secrets Management: Never hardcode sensitive information like API keys. Utilize AWS Secrets Manager, HashiCorp Vault, or environment variables in conjunction with your CI/CD pipelines.
Tagging Strategy: Implement a consistent tagging strategy across all AWS resources. This is invaluable for cost allocation, resource identification, and filtering in Datadog.
Granular IAM Roles: Adhere to the principle of least privilege for all IAM roles, especially for EKS and the Datadog AWS integration.
CI/CD Integration: Integrate Terraform, Helm, and your Datadog/PagerDuty configurations into your CI/CD pipelines to automate deployments and enforce governance.

Conclusion and Next Steps

By meticulously integrating Terraform, AWS EKS, Datadog, and PagerDuty, enterprises can build a robust, observable, and resilient cloud-native infrastructure. This approach not only automates deployment but also empowers DevOps teams with deep insights and efficient incident management capabilities, reducing MTTR (Mean Time To Resolution) and improving overall service reliability.

Your next steps should involve:

Refining Terraform: Expand upon the provided code with proper modules, variables, and outputs tailored to your organization's standards.
Advanced Datadog Configuration: Explore advanced Datadog features like Service Maps, SLOs (Service Level Objectives), RUM (Real User Monitoring), and security monitoring for a more comprehensive view.
PagerDuty Automation: Investigate PagerDuty's automation capabilities, such as runbooks and automatic diagnostic actions triggered by incident types.
Regular Audits: Periodically review your monitoring alerts, dashboards, and incident response procedures to ensure they remain relevant and effective as your system evolves.

CloudStack YAML