Terraform for AWS EKS Production: Datadog Observability and PagerDuty Incident Management

In the dynamic world of cloud-native applications, maintaining robust, scalable, and highly available Kubernetes clusters is paramount. AWS Elastic Kubernetes Service (EKS) offers a managed control plane, but ensuring optimal performance, security, and rapid incident response in a production environment requires a sophisticated observability and incident management strategy. This guide delves into leveraging Terraform for provisioning and managing your EKS infrastructure, integrating Datadog for comprehensive observability, and orchestrating incident response with PagerDuty, all through Infrastructure as Code (IaC).

Architecture Pro-Tip:

Always encapsulate your EKS cluster, node groups, and essential add-ons (like VPC CNI, CoreDNS) within a dedicated Terraform module. This promotes reusability, consistency, and easier management across different environments. For production, ensure your EKS security groups are meticulously configured, allowing minimal ingress and egress, and integrate with AWS IAM roles for service accounts (IRSA) for fine-grained access control.

The Pillars of Production Readiness: Terraform, Datadog, and PagerDuty

Terraform: Declarative Infrastructure for EKS

Terraform, HashiCorp's open-source IaC tool, enables you to define and provision your entire AWS EKS infrastructure using a declarative configuration language. This includes the EKS cluster itself, node groups, networking (VPC, subnets, security groups), IAM roles, and even the deployment of Kubernetes resources like the Datadog Agent via Helm charts. Benefits include:

Version Control: Treat your infrastructure like application code.
Consistency: Eliminate configuration drift and ensure identical environments.
Automation: Automate deployment, updates, and scaling of your EKS clusters.
Reproducibility: Quickly spin up or tear down environments with confidence.

Datadog: Unified Observability for Kubernetes

Datadog provides an all-in-one monitoring and analytics platform that brings together metrics, traces, and logs from your entire application stack. For EKS production, Datadog offers:

Comprehensive Kubernetes Monitoring: Collect metrics from nodes, pods, containers, and services.
Application Performance Monitoring (APM): Trace requests across distributed services.
Log Management: Aggregate, search, and analyze logs from all EKS components and applications.
Network Performance Monitoring (NPM): Visualize and troubleshoot network traffic.
Security Monitoring: Detect threats and misconfigurations within your EKS cluster.
Custom Dashboards and Alerts: Create real-time operational visibility and proactive notifications.

PagerDuty: Streamlined Incident Management

PagerDuty is a leading incident management platform that transforms any signal into an actionable incident. Integrated with Datadog, it ensures that critical alerts from your EKS cluster reach the right on-call personnel immediately, facilitating swift response and resolution. Key capabilities include:

Automated On-Call Scheduling: Manage rotations, escalations, and overrides.
Intelligent Alert Grouping: Reduce alert fatigue by consolidating related events.
Incident Collaboration: Facilitate communication and coordination during critical events.
Post-Mortem Analysis: Capture data for continuous improvement.

Integrating Datadog Observability with AWS EKS via Terraform

The Datadog Agent is the primary component for collecting data from your EKS cluster. It runs as a DaemonSet on your worker nodes and as a Deployment for cluster-level checks. Deploying it via Terraform with the Helm provider is the recommended approach for production.

Prerequisites for Datadog Integration

An active Datadog account.
Your Datadog API Key and Application Key (securely stored, e.g., AWS Secrets Manager).
A running AWS EKS cluster, preferably provisioned with Terraform.
Configured kubectl access to your EKS cluster.
Terraform setup with AWS, Kubernetes, and Helm providers.

Configuring Datadog Monitors and PagerDuty Integration with Terraform

After deploying the Datadog Agent, the next step is to define monitors for critical EKS metrics and integrate with PagerDuty for incident alerting. Datadog provides a Terraform provider that allows you to manage dashboards, monitors, and integrations programmatically.

Terraform Configuration Example: Datadog Agent, Monitor, and PagerDuty Integration

Here’s a conceptual Terraform configuration demonstrating how to deploy the Datadog Agent, define a critical EKS monitor, and link it to PagerDuty. Replace placeholders with your actual values.

        provider "aws" {
          region = "us-east-1" # Or your desired region
        }

        # Assume EKS cluster is already provisioned or referenced
        # data "aws_eks_cluster" "eks_cluster" {
        #   name = "my-production-eks-cluster"
        # }
        #
        # data "aws_eks_cluster_auth" "eks_cluster_auth" {
        #   name = "my-production-eks-cluster"
        # }

        provider "kubernetes" {
          host                   = aws_eks_cluster.main.endpoint # Replace with your EKS endpoint
          token                  = data.aws_eks_cluster_auth.main.token
          cluster_ca_certificate = base64decode(aws_eks_cluster.main.certificate_authority[0].data)
        }

        provider "helm" {
          kubernetes {
            host                   = aws_eks_cluster.main.endpoint # Replace with your EKS endpoint
            token                  = data.aws_eks_cluster_auth.main.token
            cluster_ca_certificate = base64decode(aws_eks_cluster.main.certificate_authority[0].data)
          }
        }

        provider "datadog" {
          api_key = var.datadog_api_key
          app_key = var.datadog_app_key
        }

        variable "datadog_api_key" {
          description = "Datadog API Key"
          type        = string
          sensitive   = true
        }

        variable "datadog_app_key" {
          description = "Datadog Application Key"
          type        = string
          sensitive   = true
        }

        variable "pagerduty_service_key" {
          description = "PagerDuty integration key for the Datadog service"
          type        = string
          sensitive   = true
        }

        resource "helm_release" "datadog_agent" {
          name             = "datadog"
          repository       = "https://helm.datadoghq.com"
          chart            = "datadog"
          namespace        = "datadog"
          create_namespace = true

          set {
            name  = "datadog.apiKey"
            value = var.datadog_api_key
            sensitive = true
          }

          set {
            name  = "datadog.appKey"
            value = var.datadog_app_key
            sensitive = true
          }

          # Enable APM, Log Collection, and Process Agent
          set {
            name  = "agents.apm.enabled"
            value = "true"
          }
          set {
            name  = "agents.logs.enabled"
            value = "true"
          }
          set {
            name  = "agents.logs.containerCollectAll"
            value = "true"
          }
          set {
            name  = "agents.processAgent.enabled"
            value = "true"
          }

          # Enable Kubernetes specific checks
          set {
            name  = "clusterAgent.enabled"
            value = "true"
          }
          set {
            name  = "kubeStateMetrics.enabled"
            value = "true"
          }
          set {
            name  = "targetSystem"
            value = "linux"
          }
          set {
            name  = "env.DD_EKS_FARGATE"
            value = "false" # Set to "true" if using Fargate profiles
          }

          # Add any other required configurations, e.g., pod labels, node selectors
          values = [
            file("datadog-values.yaml") # Optional: external values file for complex configs
          ]
        }

        resource "datadog_integration_pagerduty" "pagerduty_integration" {
          api_token = var.pagerduty_service_key # This is the service integration key from PagerDuty
        }

        resource "datadog_monitor" "eks_node_cpu_utilization_critical" {
          name            = "[EKS] High Node CPU Utilization on Production Cluster: {{host.name}}"
          type            = "metric alert"
          query           = "avg(last_5m):avg:system.cpu.idle{environment:production,cluster-name:my-production-eks-cluster} by {host} < 10" # Alert if avg idle CPU < 10% for 5 mins
          message         = "CPU utilization on node {{host.name}} is critically high! Investigate immediate causes (e.g., rogue pods, resource limits). @pagerduty-my-service"
          tags            = ["team:devops", "env:production", "severity:critical", "eks", "cpu"]
          escalation_message = "Still high CPU after 10 minutes. Escalating to core EKS team. @pagerduty-my-service-escalation"
          no_data_timeframe = 10 # minutes
          notify_no_data    = true
          renotify_interval = 60 # minutes
          new_host_delay    = 300 # seconds
          timeout_h         = 0
          include_tags      = true
          # Referencing the PagerDuty integration in the message
          # The service key configured above will be linked to the Datadog PagerDuty integration
          # and then you can use @pagerduty-<service_name> in your monitor messages.
          # Ensure 'my-service' corresponds to the name given to the PagerDuty integration in Datadog UI
          # or configured through the provider if that becomes available.
          # For direct PagerDuty notifications, you typically configure PagerDuty services in Datadog UI
          # and then reference them by their integration name.
        }

        resource "datadog_monitor" "eks_deployment_replica_mismatch" {
          name            = "[EKS] Production Deployment Replica Mismatch: {{kube_deployment}}"
          type            = "metric alert"
          query           = "min(last_5m):avg:kubernetes.deployment.replicas.unavailable{environment:production,cluster-name:my-production-eks-cluster} by {kube_deployment} > 0"
          message         = "Deployment {{kube_deployment}} has unavailable replicas in production! @pagerduty-my-service"
          tags            = ["team:devops", "env:production", "severity:high", "eks", "deployment"]
          no_data_timeframe = 10
          notify_no_data    = true
          renotify_interval = 30
          include_tags      = true
        }

        # Example of a PagerDuty service resource if you manage PagerDuty services directly via Terraform
        # This requires the PagerDuty Terraform provider, which is separate from Datadog's.
        # provider "pagerduty" {
        #   token = var.pagerduty_api_token # PagerDuty API token, not integration key
        # }
        #
        # resource "pagerduty_user" "example_user" {
        #   name  = "John Doe"
        #   email = "john.doe@example.com"
        # }
        #
        # resource "pagerduty_escalation_policy" "example_policy" {
        #   name      = "EKS Production Critical Policy"
        #   num_loops = 2
        #
        #   rule {
        #     escalation_delay_in_minutes = 10
        #     target {
        #       type = "user"
        #       id   = pagerduty_user.example_user.id
        #     }
        #   }
        # }
        #
        # resource "pagerduty_service" "eks_production_service" {
        #   name                    = "EKS Production Monitoring"
        #   auto_resolve_timeout    = "14400" # 4 hours
        #   acknowledgement_timeout = "600"   # 10 minutes
        #   escalation_policy       = pagerduty_escalation_policy.example_policy.id
        # }
        #
        # resource "pagerduty_service_integration" "datadog_eks_integration" {
        #   name    = "Datadog Integration"
        #   type    = "generic_events_api_v2"
        #   service = pagerduty_service.eks_production_service.id
        # }
        # Output the integration key to be used in Datadog for linking
        # output "pagerduty_datadog_integration_key" {
        #   value     = pagerduty_service_integration.datadog_eks_integration.integration_key
        #   sensitive = true
        # }
    

Best Practices for Production EKS Observability and Incident Management

To truly succeed with EKS in production, go beyond basic setup:

Granular Monitoring: Don't just monitor nodes; drill down to deployments, pods, containers, and even specific application metrics using custom checks and APM.
Alert Fatigue Prevention: Tune your Datadog monitors to be actionable. Use composite monitors, anomaly detection, and machine learning-driven alerts to reduce noise. Prioritize alerts by severity.
Runbook Automation: For common incidents triggered by PagerDuty, provide clear runbooks in your alert messages or link to internal documentation (e.g., Confluence, Notion) to guide responders.
Automated Remediation: For certain predictable issues (e.g., scaling up a deployment due to high CPU), consider integrating automated remediation actions (e.g., using AWS Lambda or Kubernetes operators) that can be triggered by Datadog webhooks or PagerDuty automation actions.
Cost Optimization Monitoring: Monitor AWS costs related to EKS (EC2 instances, EBS, data transfer) within Datadog to identify potential savings.
Regular Reviews: Periodically review your Terraform configurations, Datadog monitors, and PagerDuty escalation policies to ensure they align with evolving application needs and organizational structure.
Security Observability: Leverage Datadog Security Monitoring to detect unusual activities, compliance violations, and potential threats within your EKS cluster and underlying AWS infrastructure.

Troubleshooting and FAQ

Datadog Agent not reporting data?

Check the Datadog Agent pod logs for errors (kubectl logs -n datadog -l app=datadog --tail=100). Ensure your DD_API_KEY and DD_APP_KEY are correct and have the necessary permissions. Verify network connectivity from your EKS nodes to Datadog endpoints.

PagerDuty alerts not firing?

Confirm the Datadog-PagerDuty integration is correctly set up in both platforms. In Datadog, check the integration status under "Integrations" -> "PagerDuty". Ensure your Datadog monitor's message correctly references the PagerDuty service (e.g., @pagerduty-<YOUR_SERVICE_NAME>).

Terraform apply errors for Kubernetes/Helm resources?

Ensure your Kubernetes provider is correctly configured to connect to your EKS cluster. This typically involves sourcing the EKS cluster endpoint, token, and CA certificate. Also, verify that the IAM role associated with your Terraform execution environment has permissions to interact with EKS.

Conclusion

Achieving production readiness for AWS EKS demands a robust strategy for infrastructure management, observability, and incident response. By standardizing your EKS provisioning with Terraform, gaining deep insights with Datadog's unified observability platform, and orchestrating rapid incident resolution with PagerDuty, your organization can build resilient, high-performing, and easily maintainable cloud-native applications. Embracing these tools as part of a comprehensive IaC and DevOps culture is key to operational excellence in the modern cloud landscape.

CloudStack YAML