Terraform-driven Observability for AWS EKS using Datadog and PagerDuty

Terraform-Driven Observability for AWS EKS with Datadog and PagerDuty

In the dynamic world of cloud-native applications, managing complex Kubernetes clusters like AWS EKS demands robust observability and incident management. This comprehensive guide delves into leveraging Terraform for infrastructure as code (IaC) to seamlessly integrate Datadog for unified monitoring and PagerDuty for intelligent incident response within your AWS EKS environment. By codifying your observability stack, you achieve consistency, repeatability, and agility, ensuring your critical applications remain performant and resilient.

Architecture Pro-Tip:

Always design your observability stack with modularity in mind. Separate your Terraform configurations for core EKS infrastructure, Datadog agents/monitors, and PagerDuty services. This approach enhances maintainability, reduces blast radius during changes, and promotes reusability across different EKS clusters or environments. Utilize Terraform workspaces or distinct directories for environment-specific configurations.

The Pillars of Modern EKS Observability

Terraform: Infrastructure as Code for Consistency

Terraform, by HashiCorp, allows you to define and provision your entire infrastructure using declarative configuration files. For EKS, this extends beyond just the cluster itself to include crucial observability components. Managing Datadog agents, monitors, dashboards, and PagerDuty services through Terraform ensures that your monitoring and alerting configurations are version-controlled, auditable, and consistently applied across all your environments.

Datadog: Unified Monitoring and Analytics

Datadog provides an end-to-end observability platform that consolidates metrics, logs, and traces from your EKS clusters, applications, and underlying AWS infrastructure. With its rich integrations and powerful dashboarding capabilities, Datadog gives you a real-time view of your system's health, performance, and user experience. Its robust alerting engine allows for sophisticated thresholding and anomaly detection.

PagerDuty: Intelligent Incident Response

When issues arise, prompt and effective incident response is paramount. PagerDuty takes alerts from monitoring tools like Datadog and intelligently routes them to the right on-call teams based on schedules, escalation policies, and urgency. It streamlines the incident lifecycle from detection to resolution, minimizing downtime and reducing operational toil.

Prerequisites

Before we begin, ensure you have the following:

An active AWS Account with permissions to manage EKS clusters.
An existing AWS EKS Cluster. If not, Terraform can also provision this.
Terraform CLI installed (v1.0.0+ recommended).
A Datadog Account with API and Application Keys.
A PagerDuty Account with an API Token and an Event Integration Key (if using the V2 Events API).
kubectl configured to connect to your EKS cluster.
Helm CLI installed (for Datadog Agent deployment).

Architectural Overview: Seamless Integration

The integration workflow can be visualized as follows:

AWS EKS Cluster: Your Kubernetes workloads run here.
Datadog Agent (on EKS): Deployed as a DaemonSet/Deployment, it collects metrics, logs, and traces from your pods, nodes, and applications.
Datadog Platform: Ingests and processes the collected data, allowing for visualization, analysis, and alert evaluation based on configured monitors.
Datadog Integrations: When a monitor's alert conditions are met, Datadog triggers an event via its PagerDuty integration.
PagerDuty: Receives the event, creates an incident, and initiates the defined escalation policy to notify the appropriate on-call personnel.
Terraform: Manages the deployment of the Datadog Agent, the definition of Datadog monitors, and the configuration of PagerDuty services and integrations, ensuring everything is codified.

Step-by-Step Implementation with Terraform

1. Configure Terraform Providers

Begin by defining the necessary providers in your Terraform configuration. This includes AWS (to interact with EKS), Datadog, and PagerDuty.

provider "aws" {
  region = "us-east-1"
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.main.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.main.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.main.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.main.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_token
}

data "aws_eks_cluster" "main" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "main" {
  name = var.eks_cluster_name
}

variable "eks_cluster_name" {
  description = "Name of the EKS cluster"
  type        = string
}

variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

variable "pagerduty_api_token" {
  description = "PagerDuty API Token"
  type        = string
  sensitive   = true
}
    

2. Deploy Datadog Agent to AWS EKS

The Datadog Agent is deployed as a Helm chart. Terraform's Helm provider can manage this deployment, passing necessary configurations like your API key and enabling EKS-specific integrations.

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
    sensitive = true
  }
  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
    sensitive = true
  }
  set {
    name  = "datadog.clusterName"
    value = var.eks_cluster_name
  }
  set {
    name  = "agents.config.logLevel"
    value = "INFO"
  }
  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "kubeStateMetricsNetworkPolicy.enabled"
    value = "true"
  }
  set {
    name  = "targetSystem"
    value = "linux"
  }
  # Enable APM, Log Collection, and Process Agent
  set {
    name  = "datadog.apmEnabled"
    value = "true"
  }
  set {
    name  = "datadog.logs.enabled"
    value = "true"
  }
  set {
    name  = "datadog.processAgent.enabled"
    value = "true"
  }
}
    

3. Set Up PagerDuty Services and Integrations

Before Datadog can send alerts, PagerDuty needs a service to receive them and an integration to process them. We'll define a basic service and a generic email integration for simplicity, though Datadog has a native integration as well.

resource "pagerduty_user" "on_call_engineer" {
  name  = "On-Call Engineer"
  email = "oncall@example.com"
  # Replace with actual user details
}

resource "pagerduty_team" "devops_team" {
  name        = "DevOps Team"
  description = "Team responsible for EKS operations and observability"
}

resource "pagerduty_service" "eks_observability_service" {
  name        = "EKS Observability"
  description = "Service for critical EKS alerts from Datadog"
  team        = pagerduty_team.devops_team.id
  auto_resolve_timeout = "14400" # 4 hours
  acknowledgement_timeout = "600" # 10 minutes
}

resource "pagerduty_service_integration" "datadog_integration" {
  name    = "Datadog Integration"
  service = pagerduty_service.eks_observability_service.id
  type    = "generic_events_api_v2"
  # This creates an Events API v2 integration key within PagerDuty
}
    

4. Define Datadog Monitors with Terraform

Now, let's create some critical Datadog monitors. We'll define a monitor for high EKS node CPU utilization and link it to the PagerDuty service we just created. The message field is crucial for sending actionable information to PagerDuty.

Ready-to-Use Terraform Configuration

resource "datadog_monitor" "high_node_cpu" {
  name               = "EKS Node CPU Utilization High on {{host.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster:${var.eks_cluster_name}} by {host} < 10" # Alert if CPU idle is less than 10%
  message            = "High CPU utilization detected on EKS node {{host.name}} in cluster ${var.eks_cluster_name}. Current idle is {{value}}%. @pagerduty-${pagerduty_service.eks_observability_service.name}"
  tags               = ["environment:production", "service:eks-control-plane", "team:devops"]
  priority           = 1
  no_data_timeframe  = 20
  notify_no_data     = false
  renotify_interval  = 0
  escalation_message = "Still high CPU after 15 mins: @pagerduty-${pagerduty_service.eks_observability_service.name}"

  monitor_thresholds {
    critical = 10
    warning  = 20
  }

  notify_audit = false
  timeout_h    = 0
  require_full_window = false
  locked       = false
}

resource "datadog_monitor" "pod_restarts" {
  name               = "Excessive Pod Restarts in EKS cluster ${var.eks_cluster_name}"
  type               = "metric alert"
  query              = "sum(last_5m):sum:kubernetes.containers.restarts{kubernetes_cluster:${var.eks_cluster_name}} > 5" # More than 5 restarts in 5 minutes
  message            = "Multiple pod restarts detected in EKS cluster ${var.eks_cluster_name}. Investigate recent deployments or resource constraints. @pagerduty-${pagerduty_service.eks_observability_service.name}"
  tags               = ["environment:production", "service:eks-workloads", "team:devops"]
  priority           = 2
  no_data_timeframe  = 10
  notify_no_data     = false
  renotify_interval  = 0

  monitor_thresholds {
    critical = 5
    warning  = 3
  }
  notify_audit = false
  timeout_h    = 0
  require_full_window = false
  locked       = false
}
    

To apply this configuration:

Save the above code into main.tf files in a dedicated directory.
Set your sensitive variables (datadog_api_key, datadog_app_key, pagerduty_api_token) either via environment variables (e.g., TF_VAR_datadog_api_key="your_key") or a terraform.tfvars file (ensure it's excluded from version control).
Run terraform init
Run terraform plan to review the changes.
Run terraform apply to provision the resources.

Testing and Validation

After applying your Terraform configuration, it's crucial to validate the setup:

Datadog Agent: Verify that the Datadog Agent pods are running correctly in your EKS cluster: kubectl get pods -n datadog. Check Datadog's Infrastructure List to ensure your EKS nodes are reporting.
Datadog Monitors: Navigate to Datadog's "Monitors" section to confirm your newly created monitors are present and configured as expected.
PagerDuty Service: Log into PagerDuty, confirm the "EKS Observability" service exists, and the Datadog integration key is present.
Simulate an Incident: The most robust test is to trigger an alert. For a CPU monitor, you could run a temporary, CPU-intensive workload on an EKS node (e.g., stress-ng --cpu 4 --timeout 60s within a busybox pod). Observe Datadog triggering the alert and PagerDuty creating an incident.

Best Practices for Terraform-Driven Observability

Modularity and Reusability

Organize your Terraform code into logical modules (e.g., modules/datadog-monitors, modules/pagerduty-services). This promotes reusability across different projects and environments.

State Management

Always use a remote backend for your Terraform state (e.g., AWS S3 with DynamoDB locking). This is critical for team collaboration and prevents data loss.

Granular Permissions

Adhere to the principle of least privilege for AWS IAM roles used by Terraform, as well as for Datadog and PagerDuty API keys. Rotate keys regularly.

Alert Fatigue Prevention

Be judicious with your monitor thresholds and notification settings. Too many alerts lead to fatigue and missed critical incidents. Leverage Datadog's composite monitors and anomaly detection features.

Continuous Improvement

Observability is not a one-time setup. Regularly review your monitors, dashboards, and PagerDuty escalation policies. Update them as your EKS applications evolve and new operational insights emerge.

Troubleshooting Common Issues / FAQ

Why isn't my Datadog Agent reporting data?

Check Agent Logs: kubectl logs -n datadog -l app=datadog --tail 100. Look for API key errors, connectivity issues, or permission problems.
API/App Keys: Ensure datadog.apiKey and datadog.appKey are correctly set in the Helm release and are valid. Double-check for typos or leading/trailing spaces.
Network Connectivity: Verify that EKS nodes can reach Datadog's endpoints (e.g., app.datadoghq.com, api.datadoghq.com) on port 443. Check Security Groups and Network ACLs.

PagerDuty incidents not firing?

Datadog Monitor Status: Is the Datadog monitor actually triggering an alert state? Check the monitor history in Datadog.
Integration Name: Ensure the @pagerduty-YOUR_SERVICE_NAME in your Datadog monitor message exactly matches the PagerDuty service name configured.
PagerDuty Integration Key: For generic event integrations, verify the key embedded in the webhook URL (if using webhooks directly, which the Terraform provider abstracts for you in this example). Ensure the service integration is enabled.

How to manage API keys securely?

Store sensitive API keys in a secure secrets manager (e.g., AWS Secrets Manager, HashiCorp Vault) and retrieve them dynamically in your Terraform configuration or provide them via environment variables when running Terraform (TF_VAR_...). Avoid hardcoding them or committing them to version control.

Conclusion

By embracing Terraform for your AWS EKS observability stack, you transform a complex, manual process into an automated, version-controlled, and highly reliable workflow. Integrating Datadog for comprehensive monitoring and PagerDuty for intelligent incident response ensures that your cloud-native applications are not only performant but also resilient to unforeseen issues. This approach empowers your DevOps teams to operate with greater confidence, faster resolution times, and a deeper understanding of their EKS environments, ultimately driving business continuity and innovation.

CloudStack YAML