Terraform AWS EKS Datadog PagerDuty Production Observability

Mastering Production Observability on AWS EKS with Terraform, Datadog, and PagerDuty

Architecture Pro-Tip: Always design your observability stack alongside your infrastructure. Integrating monitoring and alerting from the initial Terraform deployment ensures that every new service or component automatically inherits your observability standards, preventing blind spots in production and streamlining incident response.

In the fast-paced world of cloud-native applications, maintaining robust production observability is paramount. This comprehensive guide details how to leverage Terraform for declarative infrastructure provisioning on AWS Elastic Kubernetes Service (EKS), integrate powerful monitoring with Datadog, and establish an efficient incident response pipeline with PagerDuty. This synergy empowers DevOps teams to build resilient, self-healing systems with confidence.

The Pillars of Production Observability: Terraform, EKS, Datadog, PagerDuty

Achieving true production observability requires a cohesive strategy involving infrastructure as code (IaC), a scalable container orchestration platform, comprehensive monitoring, and a robust incident management system. Each component plays a critical role:

Terraform: Automates the provisioning and management of AWS EKS clusters, IAM roles, networking, and even Datadog/PagerDuty resources, ensuring consistency and repeatability across environments.
AWS EKS: Provides a managed Kubernetes service on AWS, offering high availability, scalability, and security for your containerized applications.
Datadog: A unified observability platform that collects metrics, logs, and traces from your EKS clusters, applications, and AWS infrastructure, providing real-time insights and AI-powered anomaly detection.
PagerDuty: An incident management system that integrates with Datadog to transform alerts into actionable incidents, ensuring the right teams are notified immediately and efficiently manage critical outages.

Setting Up AWS EKS with Terraform

Provisioning an EKS cluster with Terraform ensures that your infrastructure is version-controlled, auditable, and easily reproducible. Key components include the EKS cluster itself, Node Groups, VPC configuration, and necessary IAM roles.

Core EKS Cluster Configuration

The aws_eks_cluster resource defines your Kubernetes control plane, while aws_eks_node_group provisions the worker nodes that run your containerized applications. It's crucial to define appropriate IAM roles for both the EKS service and the node groups to ensure proper permissions for interacting with AWS services.

Integrating Datadog for Comprehensive Monitoring

Datadog provides deep visibility into your EKS environment, collecting everything from cluster-level metrics to application traces. The Datadog Agent, deployed as a DaemonSet on your EKS cluster, is the primary mechanism for data collection.

Deploying the Datadog Agent on EKS

The Datadog Agent can be deployed using a Helm chart, which simplifies configuration and management. You'll need to provide your Datadog API and APP keys for the agent to authenticate and send data to your Datadog organization. These keys should be securely managed, ideally through AWS Secrets Manager or similar vault solutions.

Terraform for Datadog Monitors and Dashboards

Beyond agent deployment, Terraform can manage Datadog resources directly. This includes creating monitors for critical metrics (e.g., CPU utilization, memory pressure, pod restarts), defining synthetic tests, and building custom dashboards to visualize the health and performance of your EKS applications.

Connecting PagerDuty for Incident Response

PagerDuty acts as the central hub for incident management, ensuring that alerts from Datadog are escalated to the right teams promptly. Integrating Datadog with PagerDuty is straightforward and crucial for minimizing downtime.

Setting Up PagerDuty Services and Integrations

In PagerDuty, you'll create services that represent the components or applications you want to monitor. Each service will have an integration with Datadog (or other monitoring tools) and an escalation policy that defines who gets notified and when. Terraform can manage PagerDuty services, escalation policies, and user schedules, ensuring your incident response structure is fully automated.

Automating Alerting from Datadog to PagerDuty

Once the Datadog-PagerDuty integration is configured, any Datadog monitor that triggers an alert can be configured to send an incident to a specific PagerDuty service. This automates the notification process, reducing manual intervention and accelerating incident resolution.

Comprehensive Terraform Configuration Example

Below is a simplified, ready-to-use Terraform configuration demonstrating how to set up an EKS cluster, deploy the Datadog Agent via Helm, and configure a basic Datadog monitor that alerts PagerDuty.

Terraform for EKS, Datadog Agent, and Datadog Monitor with PagerDuty Integration

resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
  tags = { Name = "eks-vpc" }
}

resource "aws_subnet" "public" {
  count             = 2
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.0.${count.index}.0/24"
  availability_zone = data.aws_availability_zones.available.names[count.index]
  tags = { Name = "eks-public-subnet-${count.index}" }
}

resource "aws_eks_cluster" "main" {
  name     = "my-eks-cluster"
  role_arn = aws_iam_role.eks_master.arn
  vpc_config {
    subnet_ids = aws_subnet.public[*].id
  }
  depends_on = [
    aws_iam_role_policy_attachment.eks_cluster_policy,
    aws_iam_role_policy_attachment.eks_vpc_cni_policy,
  ]
}

resource "aws_iam_role" "eks_master" {
  name = "eks-cluster-master-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "eks.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eks_master.name
}

resource "aws_iam_role_policy_attachment" "eks_vpc_cni_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSVPCResourceController"
  role       = aws_iam_role.eks_master.name
}

# Add aws_eks_node_group, required IAM roles/policies for nodes, and k8s provider config here.
# For brevity, these are omitted but essential for a functional cluster.

# Datadog Agent Deployment via Helm
resource "kubernetes_secret" "datadog_api_key" {
  metadata {
    name = "datadog-api-key"
  }
  data = {
    api_key = var.datadog_api_key
  }
  type = "Opaque"
}

resource "helm_release" "datadog" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default"
  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }
  set {
    name  = "agents.enabled"
    value = true
  }
  set {
    name  = "clusterAgent.enabled"
    value = true
  }
  set {
    name  = "kubeStateMetrics.enabled"
    value = true
  }
  # Ensure your Kubernetes provider is configured correctly to deploy this
}

# PagerDuty Service (pre-existing or create with PagerDuty provider)
resource "pagerduty_service" "eks_critical_service" {
  name                    = "EKS Critical Applications"
  escalation_policy       = data.pagerduty_escalation_policy.main.id # Reference an existing policy
  alert_creation          = "create_alerts_and_incidents"
}

resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog Integration"
  type        = "datadog_inbound_integration"
  service_id  = pagerduty_service.eks_critical_service.id
}

# Datadog Monitor for EKS Node CPU Utilization
resource "datadog_monitor" "eks_node_cpu_high" {
  name                = "EKS Node CPU Utilization High on {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:kubernetes.cpu.usage.total{*} by {host} > 80"
  message             = "EKS Node CPU usage is high on {{host.name}}. Check for runaway processes or scale out node groups. @webhook-${pagerduty_service_integration.datadog_integration.id}" # Use PagerDuty Webhook Integration
  tags                = ["environment:production", "team:devops", "eks"]
  require_full_window = false
  notify_no_data      = false
  new_group_delay     = 60
  no_data_timeframe   = 20
  renotify_interval   = 0
  escalation_message  = "CPU still high after 15 minutes, escalating to on-call."
  priority            = 1
  silenced            = false
}

# Variables for API keys (manage securely)
variable "datadog_api_key" {
  type        = string
  description = "Your Datadog API Key"
  sensitive   = true
}

variable "datadog_app_key" {
  type        = string
  description = "Your Datadog APP Key"
  sensitive   = true
}

This example provides the foundation. In a production environment, you would expand on this by:

Configuring private subnets and NAT Gateways for secure EKS worker nodes.
Setting up Karpenter or Cluster Autoscaler for dynamic node scaling.
Integrating AWS Load Balancer Controller for ingress.
More granular Datadog monitors for application-specific metrics, logs, and traces.
Defining comprehensive PagerDuty escalation policies and schedules.

Best Practices for Production Observability

Beyond initial setup, continuous refinement of your observability strategy is key to operational excellence:

Define SLOs and SLIs: Clearly define what good performance looks like for your services using Service Level Objectives (SLOs) and Service Level Indicators (SLIs). Datadog can help track these.
Alert Fatigue Management: Be judicious with alerts. Focus on actionable alerts that indicate a genuine issue requiring human intervention. Use Datadog's anomaly detection and composite monitors to reduce noise.
Runbook Automation: For recurring incidents, document troubleshooting steps in runbooks linked to PagerDuty services. Automate remediation steps where possible.
Cost Optimization: Monitor your Datadog usage and EKS resource consumption. Use rightsizing recommendations and optimize log retention policies to manage costs effectively.
Security Best Practices: Ensure all API keys and sensitive credentials are stored securely (e.g., AWS Secrets Manager, HashiCorp Vault) and accessed via IAM roles with least privilege.

Troubleshooting and Common FAQs

Datadog Agent not reporting data?

Ensure your datadog_api_key and datadog_app_key are correctly configured in the Helm chart. Check the agent's logs (kubectl logs -f datadog-agent-...) for connectivity issues or authentication failures. Verify network reachability to Datadog endpoints.

PagerDuty incidents not triggering?

Confirm the Datadog monitor's message contains the correct PagerDuty webhook integration ID (e.g., @webhook-YOUR_INTEGRATION_ID). Check the Datadog event stream for monitor triggers and any associated errors. Ensure the PagerDuty service integration is active and correctly configured.

Terraform apply errors on EKS cluster creation?

Most EKS creation failures relate to IAM permissions or VPC configuration. Double-check that the IAM role assumed by EKS has the required policies (AmazonEKSClusterPolicy, AmazonEKSVPCResourceController). Ensure subnets have sufficient IP addresses and are correctly tagged.

Conclusion

Building a robust production observability stack for AWS EKS with Terraform, Datadog, and PagerDuty is a critical investment in the reliability and stability of your cloud-native applications. By automating infrastructure provisioning, centralizing monitoring, and streamlining incident response, DevOps teams can achieve higher operational efficiency, reduce downtime, and focus on delivering value. Embrace these tools to transform your production environment into a truly observable and resilient system.

CloudStack YAML