Terraform Deployment of Datadog and Prometheus for AWS EKS Observability and PagerDuty Alerting

In the dynamic landscape of cloud-native applications, maintaining robust observability for Kubernetes clusters is paramount. AWS Elastic Kubernetes Service (EKS) offers a powerful platform, but understanding its internal workings and application performance requires sophisticated monitoring tools. This guide delves into deploying a comprehensive observability stack – Datadog and Prometheus – on AWS EKS, fully automated with Terraform, and integrating with PagerDuty for proactive incident management.

Architecture Pro-Tip:

For large-scale EKS environments, consider separating your observability tools into their own dedicated namespaces and potentially even dedicated node groups (e.g., using taints and tolerations) to ensure they are not impacted by resource contention from application workloads. Additionally, leverage AWS IAM Roles for Service Accounts (IRSA) for fine-grained permissions for your monitoring agents, enhancing security by avoiding long-lived AWS access keys.

Why a Hybrid Observability Stack?

While Datadog provides a unified platform for metrics, logs, and traces, Prometheus excels in specific areas, especially for custom metric exposition and integration with the CNCF ecosystem. Combining them offers a resilient and flexible observability solution:

Datadog: Offers a holistic view, combining infrastructure metrics, application performance monitoring (APM), log management, real user monitoring (RUM), and security monitoring. Its intuitive dashboards and powerful alerting capabilities streamline operations.
Prometheus: An open-source monitoring system with a powerful query language (PromQL) and a vast ecosystem of exporters. It's ideal for deep dives into Kubernetes cluster components and applications exposing Prometheus-compatible metrics.
PagerDuty: The industry standard for incident response. Integrating monitoring alerts with PagerDuty ensures critical issues are escalated to the right teams promptly, minimizing downtime and business impact.
Terraform: Enables Infrastructure as Code (IaC), allowing you to define, provision, and manage your entire observability stack using declarative configuration files. This ensures consistency, reproducibility, and version control.

Prerequisites

Before you begin, ensure you have the following:

An active AWS Account with appropriate permissions to create EKS clusters, IAM roles, and other AWS resources.
An existing AWS EKS Cluster. This guide assumes you have one ready.
Terraform CLI installed (v1.0+ recommended).
AWS CLI installed and configured.
kubectl CLI installed and configured to connect to your EKS cluster.
A Datadog account with an API Key and Application Key.
A PagerDuty account with an API Key for integration.
Helm CLI installed (Terraform will manage Helm releases, but understanding Helm is beneficial).

Terraform Project Structure

A recommended project structure for clarity and maintainability:

.
├── main.tf
├── variables.tf
├── outputs.tf
├── providers.tf
├── modules/
│   ├── datadog/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   └── prometheus/
│       ├── main.tf
│       ├── variables.tf
│       └── versions.tf
└── backend.tf

Step-by-Step Deployment with Terraform

1. Configure Terraform Providers

Your providers.tf file will include AWS, Kubernetes, Helm, Datadog, and PagerDuty providers.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
    pagerduty = {
      source  = "PagerDuty/pagerduty"
      version = "~> 2.0"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

provider "pagerduty" {
  token = var.pagerduty_api_key
}

data "aws_eks_cluster" "cluster" {
  name = var.eks_cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = var.eks_cluster_name
}
    

2. Deploy Datadog Agent to EKS via Helm

The Datadog Agent is deployed as a DaemonSet and a Cluster Agent in your EKS cluster. We'll use the helm_release resource in Terraform.

# main.tf or modules/datadog/main.tf
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }

  set {
    name  = "datadog.site"
    value = "datadoghq.com" # Or your specific Datadog site (e.g., eu.datadoghq.com)
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "clusterAgent.metricsProvider.enabled"
    value = "true"
  }

  set {
    name  = "datadog.hostLabels.eks_cluster_name"
    value = var.eks_cluster_name
  }

  # Enable EKS specific integrations
  set {
    name  = "datadog.confd.kube_proxy.yaml"
    value = <<-EOT
instances:
  -
    EOT
  }

  set {
    name  = "datadog.confd.kubelet.yaml"
    value = <<-EOT
instances:
  -
    EOT
  }

  # For IRSA (recommended)
  set {
    name  = "rbac.create"
    value = "true"
  }
  set {
    name  = "serviceAccount.create"
    value = "true"
  }
  set {
    name  = "clusterAgent.serviceAccount.create"
    value = "true"
  }
  set {
    name  = "clusterAgent.serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
    value = aws_iam_role.datadog_agent_role.arn
  }
  set {
    name  = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
    value = aws_iam_role.datadog_agent_role.arn
  }
  # ... other Datadog agent configurations (APM, Logs, etc.)
}

# IAM Role for Service Account (IRSA) for Datadog Agent
resource "aws_iam_role" "datadog_agent_role" {
  name = "${var.eks_cluster_name}-datadog-agent-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Federated = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${replace(data.aws_eks_cluster.cluster.identity.0.oidc.0.issuer, "https://", "")}"
        }
        Action = "sts:AssumeRoleWithWebIdentity"
        Condition = {
          StringEquals = {
            "${replace(data.aws_eks_cluster.cluster.identity.0.oidc.0.issuer, "https://", "")}:sub" = "system:serviceaccount:datadog:datadog-agent"
          }
        }
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "datadog_agent_policy_attachment" {
  role       = aws_iam_role.datadog_agent_role.name
  policy_arn = "arn:aws:iam::aws:policy/ReadOnlyAccess" # Adjust with specific policies required by Datadog for EKS
}

data "aws_caller_identity" "current" {}
    

3. Configure Datadog Monitors and PagerDuty Integration

Use the Datadog provider to define monitors that alert on critical EKS metrics and integrate directly with PagerDuty.

# main.tf or modules/datadog/main.tf
resource "datadog_integration_pagerduty" "pagerduty_integration" {
  api_token = var.pagerduty_api_key
}

resource "datadog_monitor" "eks_node_cpu_utilization" {
  name               = "[EKS] Node CPU Utilization High (Cluster: ${var.eks_cluster_name})"
  type               = "metric alert"
  query              = "avg(last_5m):avg:kubernetes.cpu.usage.total{kube_cluster_name:${var.eks_cluster_name}} by {host} > 80"
  message            = "CPU utilization on node {{host.name}} is {{value}}%. @pagerduty-YourPagerDutyServiceIntegration"
  escalation_message = "CPU utilization on node {{host.name}} is still high after 15 minutes. @pagerduty-YourPagerDutyServiceIntegration"
  tags               = ["environment:${var.environment}", "service:eks", "alert:cpu"]

  notify_no_data    = false
  new_group_delay   = 60
  no_data_timeframe = 20

  threshold_windows {
    recovery_window = "last_15m"
    trigger_window  = "last_5m"
  }

  thresholds {
    critical = 80
    warning  = 70
  }

  # PagerDuty integration via Datadog's built-in mechanism
  # Ensure 'YourPagerDutyServiceIntegration' matches the service key/integration name configured in Datadog's PagerDuty integration settings.
  # This typically corresponds to the 'integration_key' from a PagerDuty service.
  # The datadog_integration_pagerduty resource does not create a *service* in PagerDuty,
  # but rather configures Datadog to push to PagerDuty services.
  # For new PagerDuty services, you'd define them using the pagerduty provider.
}

resource "pagerduty_service" "eks_alerts_service" {
  name                    = "${var.environment}-EKS-Alerts"
  auto_resolve_timeout    = "14400" # 4 hours
  acknowledgement_timeout = "600"   # 10 minutes
  escalation_policy       = pagerduty_escalation_policy.devops_ep.id
  description             = "Service for critical EKS alerts managed via Terraform."
}

resource "pagerduty_escalation_policy" "devops_ep" {
  name      = "${var.environment}-DevOps-Escalation-Policy"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "user"
      id   = var.pagerduty_devops_user_id # PagerDuty user ID
    }
  }

  rule {
    escalation_delay_in_minutes = 20
    target {
      type = "schedule"
      id   = var.pagerduty_oncall_schedule_id # PagerDuty schedule ID
    }
  }
}

# The actual PagerDuty integration key for Datadog is obtained from the service.
# This key is what you'd use in the Datadog monitor message.
# You typically grab this manually or use a data source if it was created outside Terraform.
# For Terraform-managed services, you can create a PagerDuty service integration directly.
resource "pagerduty_service_integration" "datadog_integration" {
  name    = "Datadog Integration"
  service = pagerduty_service.eks_alerts_service.id
  type    = "generic_events_api_inbound_integration"
}

# Output the integration key to use in Datadog monitor messages
output "datadog_pagerduty_integration_key" {
  value       = pagerduty_service_integration.datadog_integration.integration_key
  description = "The PagerDuty integration key for Datadog alerts."
  sensitive   = true
}

# Update the Datadog monitor message to use the dynamic integration key
# For this, you would need to reference the output in the message or use a local variable
# Example:
# message = "CPU utilization on node {{host.name}} is {{value}}%. @pagerduty-${pagerduty_service_integration.datadog_integration.integration_key}"
# Note: Direct interpolation of integration_key might require careful handling of its availability
# when the Datadog monitor is created. Often, it's simpler to set the integration manually
# once the service is created, or pass the key as a variable.
    

4. Deploy Prometheus Operator and Alertmanager to EKS

Prometheus Operator simplifies the deployment and management of Prometheus, Alertmanager, and related components in Kubernetes. Alertmanager handles routing alerts to PagerDuty.

# main.tf or modules/prometheus/main.tf
resource "helm_release" "prometheus_operator" {
  name       = "kube-prometheus-stack"
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  namespace  = "monitoring"
  create_namespace = true

  version = "54.1.0" # Use a stable version

  values = [
    "${file("${path.module}/values/prometheus-values.yaml")}"
  ]
}

# prometheus-values.yaml (inside modules/prometheus/values/)
# This YAML file would configure Alertmanager to send alerts to PagerDuty.
# You'd typically use a Kubernetes Secret to store the PagerDuty routing key.
    

Example prometheus-values.yaml snippet for Alertmanager:

alertmanager:
  enabled: true
  config:
    global:
      # PagerDuty routing key stored in a Kubernetes secret
      pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'
    receivers:
      - name: 'pagerduty'
        pagerduty_configs:
          - routing_key: {{ .alertmanager.pagerduty_routing_key }} # This would come from a secret
            send_resolved: true
    route:
      group_by: ['alertname', 'cluster']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      receiver: 'pagerduty'
      routes:
        - match:
            severity: critical
          receiver: 'pagerduty'
    

To securely pass the PagerDuty routing key, use a Kubernetes Secret managed by Terraform:

# main.tf or modules/prometheus/main.tf
resource "kubernetes_secret" "alertmanager_pagerduty_secret" {
  metadata {
    name      = "alertmanager-pagerduty-secret"
    namespace = "monitoring"
  }

  data = {
    # This routing key is specific to a PagerDuty integration created for Prometheus
    "PAGERDUTY_ROUTING_KEY" = var.pagerduty_prometheus_routing_key
  }
}

# The prometheus-values.yaml would then reference this secret:
# alertmanager:
#   config:
#     global:
#       pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'
#     receivers:
#       - name: 'pagerduty'
#         pagerduty_configs:
#           - routing_key: "{{ .alertmanager.secret.PAGERDUTY_ROUTING_KEY }}" # Reference the secret
#             send_resolved: true
# ... or set via `set` block in helm_release if the chart supports it.
# For `kube-prometheus-stack`, you'd configure it in the `alertmanager.config` section
# via values directly or referencing a secret volumeMount.
# A common approach is to provide the full Alertmanager config directly in `alertmanager.config`
# using a template or file content, where the routing key is interpolated.
    

5. Apply Terraform Configuration

Once all your .tf files are configured, initialize and apply Terraform.

terraform init
terraform plan
terraform apply

Verification and Post-Deployment Steps

After applying your Terraform configuration:

Datadog:
- Log into your Datadog account.
- Navigate to "Infrastructure" -> "Hosts" to see your EKS nodes.
- Check "Kubernetes" -> "EKS" dashboard for cluster-wide metrics.
- Verify your configured monitors under "Monitors" -> "Manage Monitors".
- Test an alert to confirm PagerDuty integration.
Prometheus:
- Port-forward to the Prometheus UI: kubectl port-forward svc/kube-prometheus-stack-prometheus -n monitoring 9090:9090
- Access http://localhost:9090 in your browser to explore metrics.
- Port-forward to Alertmanager UI: kubectl port-forward svc/kube-prometheus-stack-alertmanager -n monitoring 9093:9093
- Access http://localhost:9093 to see Alertmanager status and configured receivers.
PagerDuty:
- Log into PagerDuty.
- Verify that the new service and escalation policy exist.
- Once an alert is triggered (e.g., by simulating a critical condition or testing a Datadog monitor), an incident should appear in PagerDuty.

Best Practices

IAM Roles for Service Accounts (IRSA): Always use IRSA for your Datadog and Prometheus agents to grant them least-privilege access to AWS resources.
Secrets Management: Store API keys and sensitive information in a secure vault like AWS Secrets Manager or HashiCorp Vault, and fetch them using Terraform data sources or external providers, rather than hardcoding in variables.
Version Control: Keep your Terraform configurations in a Git repository and follow branching strategies for changes.
Observability of Observability: Monitor your monitoring stack itself. Ensure Datadog agents are healthy, Prometheus scrape targets are up, and Alertmanager is correctly routing.
Cost Optimization: Monitor the resource consumption of your monitoring agents. Datadog costs scale with metrics, logs, and hosts, while Prometheus storage can grow large.
Custom Metrics: Leverage Prometheus's flexibility for application-specific custom metrics, complementing Datadog's broader coverage.

Troubleshooting and FAQ

Q: Datadog Agent is not reporting any data.

A: Check the Datadog Agent pods logs (kubectl logs -n datadog -l app=datadog). Ensure the API key and application key are correct. Verify that the agent has the necessary IAM permissions if using IRSA. Also, check network connectivity from EKS nodes to Datadog endpoints.

Q: Prometheus is not scraping metrics.

A: Confirm that your ServiceMonitors or PodMonitors are correctly configured and targeting the right labels. Check the Prometheus UI under "Status" -> "Targets" for any scrape errors. Ensure your applications are exposing metrics at the expected /metrics endpoint.

Q: PagerDuty incidents are not being created.

For Datadog: Verify the Datadog monitor's message contains the correct @pagerduty-YourIntegrationKey. Ensure the PagerDuty integration is properly configured in Datadog.
For Prometheus Alertmanager: Check Alertmanager logs. Confirm the pagerduty_routing_key in Alertmanager's configuration is correct and points to a valid PagerDuty integration key.
Verify network reachability from where alerts are sent (Datadog's servers or Alertmanager pods) to PagerDuty's event API.

Conclusion

By leveraging Terraform, you can seamlessly deploy a robust and scalable observability solution for your AWS EKS clusters using Datadog, Prometheus, and PagerDuty. This Infrastructure as Code approach ensures that your monitoring and alerting infrastructure is version-controlled, repeatable, and easily manageable, empowering your DevOps teams to maintain high availability and performance for your critical applications.

Embrace automation to build a resilient and highly observable cloud-native environment, transforming reactive incident response into proactive problem-solving.

CloudStack YAML