Terraform for AWS EKS Observability with Datadog and PagerDuty Incident Management

Architecture Pro-Tip: Always isolate your observability stack from your core application infrastructure. Deploy Datadog agents and related resources within a dedicated observability VPC or namespace, and use AWS IAM roles for service accounts (IRSA) for fine-grained permissions. This enhances security, simplifies management, and ensures your monitoring remains resilient even if application components face issues.

Terraform for AWS EKS Observability with Datadog and PagerDuty Incident Management

In the dynamic landscape of modern cloud-native applications, maintaining robust observability and efficient incident management is paramount. AWS EKS (Elastic Kubernetes Service) provides a powerful platform for deploying containerized workloads, but it demands sophisticated tooling to gain deep insights into its health, performance, and security. This technical guide explores how to leverage Terraform to provision and manage a comprehensive observability stack for AWS EKS, integrating Datadog for real-time monitoring and PagerDuty for streamlined incident response.

Why Terraform for EKS Observability?

Terraform, as an Infrastructure as Code (IaC) tool, offers significant advantages for managing your observability infrastructure:

Reproducibility: Define your entire EKS, Datadog, and PagerDuty configuration in code, ensuring consistent deployments across environments.
Version Control: Track changes, revert to previous states, and collaborate on infrastructure definitions using standard Git workflows.
Automation: Automate the provisioning and configuration of agents, monitors, dashboards, and alerting rules, reducing manual effort and human error.
Scalability: Easily scale your observability solution alongside your EKS clusters and application growth.

Core Components

AWS EKS

Amazon EKS is a managed Kubernetes service that makes it easy to run Kubernetes on AWS without needing to install, operate, and maintain your own Kubernetes control plane. It integrates with AWS services for networking, load balancing, and IAM for robust security and scalability.

Datadog

Datadog is a leading monitoring and analytics platform for cloud applications. It provides end-to-end visibility across infrastructure, applications, logs, and security, with powerful features like custom dashboards, machine learning-driven alerts, APM, and log management. For Kubernetes, Datadog offers a dedicated agent to collect metrics, events, and logs from nodes, pods, and containers.

PagerDuty

PagerDuty is an incident management platform that orchestrates real-time response to critical incidents. It ingests alerts from monitoring tools like Datadog, intelligently routes them to the right on-call teams, and facilitates incident communication and resolution. Integrating Datadog with PagerDuty ensures that critical EKS issues trigger immediate, actionable alerts to your operations teams.

Prerequisites

Before you begin, ensure you have the following:

An AWS account with appropriate IAM permissions to create EKS clusters, VPCs, and related resources.
Terraform CLI installed (version 1.0+ recommended).
AWS CLI configured with credentials.
A Datadog account with an API key and Application key.
A PagerDuty account with an integration key (service key) for Datadog.
kubectl and helm CLIs installed.

Step-by-Step Implementation Guide

1. AWS EKS Cluster Setup with Terraform

First, we'll provision an AWS EKS cluster, its associated VPC, subnets, and node groups using Terraform. This example provides a basic EKS setup.

        
provider "aws" {
  region = "us-east-1"
}

locals {
  cluster_name = "eks-observability-cluster"
  tags = {
    Project     = "EKS Observability"
    Environment = "Dev"
  }
}

# --- VPC Module ---
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "3.1.0"

  name = "${local.cluster_name}-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true

  tags = local.tags
}

# --- EKS Module ---
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "17.2.0"

  cluster_name    = local.cluster_name
  cluster_version = "1.28"
  vpc_id          = module.vpc.vpc_id
  subnet_ids      = module.vpc.private_subnets

  # EKS Cluster managed node group
  eks_managed_node_groups = {
    default = {
      instance_types = ["t3.medium"]
      min_size       = 2
      max_size       = 4
      desired_size   = 2
      vpc_security_group_ids = [module.vpc.default_security_group_id]
      tags = local.tags
    }
  }

  tags = local.tags
}

output "eks_kubeconfig" {
  description = "Kubernetes config for EKS cluster"
  value       = module.eks.kubeconfig
  sensitive   = true
}

output "cluster_endpoint" {
  description = "The endpoint for the EKS cluster."
  value       = module.eks.cluster_endpoint
}
        
    

Run terraform init, terraform plan, and terraform apply to provision your EKS cluster. After creation, configure kubectl to connect to your cluster.

2. Deploying Datadog Agent to EKS

Datadog provides an official Helm chart for deploying its agent to Kubernetes. We can use Terraform's Helm provider to automate this deployment. You'll need your Datadog API key and (optionally) application key.

        
provider "helm" {
  kubernetes {
    host                   = module.eks.cluster_endpoint
    cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args        = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
    }
  }
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }

  # Recommended for advanced features like APM, RUM, etc.
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }

  # Enable APM and Live Process monitoring
  set {
    name  = "apm.enabled"
    value = true
  }
  set {
    name  = "processAgent.enabled"
    value = true
  }
  set {
    name  = "liveContainers.enabled"
    value = true
  }

  # Enable log collection
  set {
    name  = "logs.enabled"
    value = true
  }
  set {
    name  = "logs.containerCollectAll"
    value = true
  }

  # Kube-state-metrics integration
  set {
    name  = "kubeStateMetrics.enabled"
    value = true
  }

  # RBAC for Datadog Agent
  set {
    name  = "rbac.create"
    value = true
  }

  values = [
    templatefile("${path.module}/datadog-values.yaml", {
      cluster_name = local.cluster_name
    })
  ]
}

# Variable for Datadog API Key
variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

# Variable for Datadog Application Key
variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}
        
    

Create a datadog-values.yaml file in the same directory for any specific custom configurations:

        
# datadog-values.yaml
clusterAgent:
  metricsProvider:
    enabled: true
clusterName: ${cluster_name}
        
    

After applying this Terraform configuration, Datadog agents will be deployed to your EKS cluster, and you should start seeing metrics, logs, and traces appearing in your Datadog account.

3. Configuring Datadog Monitors and Dashboards

Once data flows into Datadog, you can use Terraform to define monitors and dashboards. This ensures critical alerts and visualizations are consistently applied across your environments.

        
provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

# Example Datadog Monitor for EKS Node CPU Utilization
resource "datadog_monitor" "eks_node_cpu_utilization" {
  name               = "[EKS] High Node CPU Utilization on ${local.cluster_name}"
  type               = "metric alert"
  message            = "CPU utilization on an EKS node in ${local.cluster_name} is high. @slack-devops @pagerduty"
  query              = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster_name:${local.cluster_name}} by {host} < 10" # Alert if idle CPU < 10% (i.e., usage > 90%)
  
  monitor_thresholds {
    critical = 10
    warning  = 20
  }

  tags               = ["environment:dev", "service:eks", "alert-type:performance", "cluster:${local.cluster_name}"]
  notify_no_data     = false
  new_host_delay     = 300
  renotify_interval  = 0
  escalation_message = "CPU utilization remains critical after 15 minutes."

  # Reference PagerDuty integration from PagerDuty section
  # This assumes a PagerDuty service is already set up in Datadog integration settings
  # or you define it via Datadog provider (not covered in this specific example)
  # For simplicity, we assume an existing integration or use generic @pagerduty notation.
}

# Example Datadog Dashboard for EKS Overview
resource "datadog_dashboard" "eks_overview_dashboard" {
  title        = "[EKS] Cluster Overview - ${local.cluster_name}"
  description  = "Overview of EKS Cluster health and performance"
  layout_type  = "ordered"
  is_read_only = false

  widget {
    # Node CPU Usage graph
    widget_template = <
    


    Apply these Terraform configurations to provision your monitors and dashboards in Datadog.

    4. Integrating Datadog with PagerDuty
    To ensure critical Datadog alerts are routed to your on-call teams, we integrate with PagerDuty. The datadog_integration_pagerduty resource in the Datadog provider allows you to configure this integration.
    
        
# Provider configuration for Datadog remains the same as above.

# Example PagerDuty Service Integration (assuming it exists in PagerDuty)
# In PagerDuty, create a new service and add an "Integrations" section with "Datadog".
# Get the Integration Key (or Routing Key if using Event Intelligence).

resource "datadog_integration_pagerduty" "main_pagerduty_integration" {
  # This resource configures the *connection* between Datadog and PagerDuty.
  # The `api_key` here is the PagerDuty "Integration Key" or "Service API Key"
  # obtained from your PagerDuty service integration.
  api_key = var.pagerduty_integration_key
  
  # The service_name is often the PagerDuty service name you're integrating with
  # This is for Datadog's internal reference of this integration.
  # If you have multiple PagerDuty integrations, give it a unique name.
  service_name = "EKS Observability Service"

  # Optionally, enable auto_resolve for Datadog events
  auto_resolve = true
}

# Variable for PagerDuty Integration Key
variable "pagerduty_integration_key" {
  description = "PagerDuty Integration Key for Datadog service"
  type        = string
  sensitive   = true
}

# To trigger PagerDuty from a Datadog monitor, you reference the PagerDuty
# integration in the monitor's message or via notification preferences.
# For example, in the `datadog_monitor` resource above, you would include:
# message = "CPU utilization on an EKS node is high. @pagerduty-EKS Observability Service"
# (replace "EKS Observability Service" with the actual service_name configured above)
        
    
    After applying this, alerts from Datadog monitors configured with the @pagerduty-<service_name> notation will automatically create incidents in PagerDuty, routing them to the correct on-call teams.

    Advanced Observability Strategies
    To further enhance your EKS observability:
    
        APM and Distributed Tracing: Instrument your applications to collect traces using Datadog APM. This provides deep visibility into application performance, service dependencies, and error rates within your EKS microservices.
        Security Monitoring: Leverage Datadog Security Monitoring to detect threats and suspicious activities in your EKS clusters, including audit logs, container runtime security, and network activity.
        Cost Optimization: Utilize Datadog's cloud cost management features and integrate with AWS Cost Explorer to monitor and optimize your EKS spending based on usage and performance.
        Synthetics Monitoring: Deploy Datadog synthetic tests to proactively monitor the availability and performance of your EKS-hosted applications from an end-user perspective.
    

    



    Troubleshooting and Best Practices

    Common Issues
    
        Datadog Agent Not Reporting: Check the Datadog agent's pod logs (kubectl logs -n datadog -l app=datadog) for API key issues, network connectivity problems, or permission errors. Ensure your EKS nodes can reach the Datadog API endpoints.
        Missing Metrics/Logs: Verify that the Datadog agent's configuration (via Helm values) correctly enables relevant integrations (e.g., Kube-state-metrics, log collection). Check Kubernetes pod annotations for specific application-level collection.
        PagerDuty Incidents Not Triggering: Confirm that the Datadog monitor's message correctly references the PagerDuty integration (e.g., @pagerduty-MyService). Ensure the PagerDuty integration key in Datadog is valid and the corresponding service is set up correctly in PagerDuty.
        Terraform State Drift: Regularly run terraform plan to detect and rectify any manual changes made outside of Terraform. Implement CI/CD pipelines to enforce IaC principles.
    

    Best Practices
    
        Modularize Terraform: Break down your Terraform configuration into reusable modules (e.g., EKS module, Datadog module, PagerDuty module) for better organization and scalability.
        Secrets Management: Use AWS Secrets Manager or HashiCorp Vault to securely store sensitive data like Datadog API keys and PagerDuty integration keys, retrieving them dynamically in Terraform.
        Least Privilege: Apply the principle of least privilege to all IAM roles, Kubernetes RBAC, and Datadog/PagerDuty credentials.
        Tagging Strategy: Implement a consistent AWS tagging strategy. Datadog automatically ingests these tags, allowing for powerful filtering and grouping in dashboards and monitors.
        Alert Fatigue: Fine-tune your Datadog monitors to prevent alert fatigue. Focus on actionable alerts, use composite monitors for complex conditions, and leverage Datadog's machine learning capabilities for anomaly detection.
    

    Conclusion
    By adopting Terraform to manage your AWS EKS cluster along with Datadog for comprehensive observability and PagerDuty for efficient incident management, you establish a robust, automated, and scalable cloud-native operations framework. This approach empowers your teams with deep insights into your Kubernetes workloads, streamlines incident response, and ultimately contributes to the reliability and performance of your applications.

CloudStack YAML