Terraform-driven AWS EKS Observability: Datadog APM, Prometheus Metrics, and PagerDuty Alerts

In the rapidly evolving landscape of cloud-native applications, maintaining robust observability for Kubernetes clusters is not just a best practice—it's a critical operational necessity. AWS Elastic Kubernetes Service (EKS) provides a powerful, managed foundation for containerized workloads, but its dynamic nature demands sophisticated monitoring, logging, and alerting strategies to ensure reliability and performance. This technical guide delves into building a comprehensive, Terraform-driven observability stack for AWS EKS, integrating industry-leading tools: Datadog for deep application performance monitoring (APM) and infrastructure insights, Prometheus for powerful, open-source metrics collection and analysis, and PagerDuty for streamlined incident management and on-call alerting. By leveraging Infrastructure as Code (IaC) with Terraform, organizations can provision, configure, and manage this entire observability ecosystem with unparalleled consistency, scalability, and auditability, transforming reactive troubleshooting into proactive operational excellence.

Architecture Pro-Tip: For optimal EKS observability, always consider a multi-faceted approach. While Prometheus excels at metric collection and custom instrumentation, a commercial solution like Datadog offers a unified platform for APM, logs, traces, and infrastructure health. Integrate these tools at the data ingestion layer and centralize incident response with PagerDuty to create a resilient, end-to-end monitoring and alerting framework managed entirely through Infrastructure as Code.

Establishing the AWS EKS Foundation with Terraform

The first step in building a robust observability stack is to provision a stable and secure AWS EKS cluster. Terraform is the undisputed champion for managing cloud infrastructure as code, ensuring that your EKS environment is provisioned consistently, repeatably, and version-controlled. This approach minimizes configuration drift and enhances collaboration across engineering teams. When setting up EKS, key considerations include networking (VPC, subnets, security groups), IAM roles for the EKS control plane and worker nodes, and the Kubernetes version. We'll leverage the official Terraform AWS EKS module, which abstracts away much of the complexity, allowing us to focus on higher-level configurations and integrations.

The following Terraform configuration snippet illustrates how to define a basic EKS cluster. It includes defining a VPC, associated subnets, and an IAM role for the EKS service itself, which grants permissions to manage AWS resources on your behalf. This foundational setup is crucial before deploying any observability agents or applications, ensuring that your cluster has the necessary networking and security context to operate effectively. Remember to customize the region, name, and desired node group configurations to align with your specific architectural requirements and performance demands. Proper planning at this stage sets the groundwork for a scalable and observable Kubernetes environment.

resource "aws_vpc" "eks_vpc" {
  cidr_block = "10.0.0.0/16"
  tags = { Name = "eks-observability-vpc" }
}

resource "aws_subnet" "eks_public_subnets" {
  count             = 2
  vpc_id            = aws_vpc.eks_vpc.id
  cidr_block        = "10.0.${count.index}.0/24"
  availability_zone = data.aws_availability_zones.available.names[count.index]
  tags              = { Name = "eks-public-subnet-${count.index}" }
}

resource "aws_iam_role" "eks_master_role" {
  name = "eks-master-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "eks.amazonaws.com"
        }
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_master_AmazonEKSClusterPolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eks_master_role.name
}

module "eks_cluster" {
  source          = "terraform-aws-modules/eks/aws"
  version         = "~> 19.0"

  cluster_name    = "observability-eks-cluster"
  cluster_version = "1.28"
  vpc_id          = aws_vpc.eks_vpc.id
  subnet_ids      = aws_subnet.eks_public_subnets[*].id
  eks_role_arn    = aws_iam_role.eks_master_role.arn

  # EKS Managed Node Group
  eks_managed_node_groups = {
    default = {
      instance_types = ["t3.medium"]
      desired_capacity = 2
      max_capacity     = 3
      min_capacity     = 1
    }
  }
}

Integrating Datadog for Comprehensive APM and Infrastructure Monitoring

Datadog provides a unified platform for monitoring, logging, and tracing, making it an invaluable tool for EKS observability. Its capabilities span infrastructure monitoring, application performance monitoring (APM), log management, network performance monitoring, and security monitoring, all presented in intuitive dashboards. For EKS, the Datadog Agent is deployed as a DaemonSet across your worker nodes, collecting metrics, logs, and traces directly from your Kubernetes pods and the underlying infrastructure. This agent is crucial for gaining deep insights into resource utilization, container health, and application behavior within your cluster. By integrating Datadog, you empower your teams with real-time visibility into the entire stack, from the host kernel to individual microservices, facilitating faster incident resolution and performance optimization. The APM features are particularly strong, offering distributed tracing capabilities that help pinpoint bottlenecks in complex service architectures.

Deploying the Datadog Agent into EKS is typically managed using Helm charts, but Terraform can orchestrate this deployment by utilizing the Kubernetes provider to apply Helm releases. This ensures that the Datadog Agent's lifecycle is managed alongside your infrastructure and application deployments, maintaining consistency. The following Terraform code block demonstrates how to deploy the Datadog Agent using the Helm provider within Terraform, configuring it with your Datadog API key and specifying the EKS cluster context. This setup enables Datadog to begin collecting vital operational data almost immediately after deployment, providing crucial visibility into your new EKS environment.

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }

  set {
    name  = "datadog.site"
    value = "datadoghq.com" # or eu.datadoghq.com etc.
  }

  set {
    name  = "kubeStateMetrics.enabled"
    value = "true"
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "clusterAgent.metricsProvider.enabled"
    value = "true"
  }

  set {
    name  = "apm.enabled"
    value = "true"
  }

  set {
    name  = "logAgent.enabled"
    value = "true"
  }

  set {
    name  = "logAgent.containerCollectAll"
    value = "true"
  }

  depends_on = [module.eks_cluster] # Ensure EKS cluster is ready
}

# Example variables, replace with actual sensitive data management (e.g., AWS Secrets Manager)
variable "datadog_api_key" {
  description = "Datadog API Key"
  type        = string
  sensitive   = true
}

variable "datadog_app_key" {
  description = "Datadog Application Key"
  type        = string
  sensitive   = true
}

Leveraging Prometheus for Custom Metrics and Service Monitoring

Prometheus has become the de facto standard for open-source metric monitoring in cloud-native environments. Its powerful multi-dimensional data model, flexible query language (PromQL), and pull-based metric collection mechanism make it ideal for scraping metrics from Kubernetes components, custom applications, and various exporters. While Datadog offers comprehensive monitoring, Prometheus shines particularly in scenarios requiring fine-grained control over metric collection, custom instrumentation, and integration with the wider CNCF ecosystem. It excels at collecting high-cardinality data and providing real-time insights into the performance and health of individual services, often complementing commercial solutions by offering a deeper, application-centric view of specific metrics that might not be available out-of-the-box elsewhere. By deploying Prometheus, you gain the flexibility to define precise scraping targets and leverage its robust alerting capabilities.

Deploying Prometheus on EKS is often simplified by using the kube-prometheus-stack Helm chart, which bundles Prometheus, Grafana, Alertmanager, and various Kubernetes exporters. This provides a powerful, ready-to-use monitoring solution for your cluster. Terraform, again using the Helm provider, can manage the deployment of this stack, ensuring that Prometheus, its dependencies, and its configurations are deployed consistently across your environments. The configuration below sets up the kube-prometheus-stack in a dedicated namespace, enabling Prometheus to scrape metrics from your EKS cluster and any instrumented applications. This forms the backbone for gathering detailed, time-series data crucial for performance analysis and problem diagnosis within your Kubernetes workloads.

resource "kubernetes_namespace" "monitoring_ns" {
  metadata {
    name = "monitoring"
  }
}

resource "helm_release" "kube_prometheus_stack" {
  name       = "kube-prometheus-stack"
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  namespace  = kubernetes_namespace.monitoring_ns.metadata[0].name

  set {
    name  = "grafana.enabled"
    value = "false" # Often prefer Datadog dashboards, or external Grafana
  }

  set {
    name  = "prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues"
    value = "false"
  }

  set {
    name  = "prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues"
    value = "false"
  }

  depends_on = [
    module.eks_cluster,
    kubernetes_namespace.monitoring_ns
  ]
}

Bridging Metrics: Datadog-Prometheus Integration for Unified Observability

While both Datadog and Prometheus offer powerful metric collection capabilities, the true strength of this observability stack lies in their integration. Datadog can be configured to scrape Prometheus metrics endpoints, allowing you to consolidate metrics from both sources into a single pane of glass within the Datadog platform. This unification is incredibly valuable for teams that might have existing Prometheus deployments or applications already instrumented with Prometheus exporters, but also desire the extensive APM, logging, and tracing features of Datadog. By bringing Prometheus metrics into Datadog, you eliminate context switching, simplify dashboarding, and centralize alerting, providing a more coherent and comprehensive view of your application and infrastructure health. This integration ensures that no critical performance indicator is missed, regardless of its origin, and empowers a unified approach to monitoring across diverse technology stacks.

To achieve this, the Datadog Agent needs to be configured to discover and scrape Prometheus endpoints. This is typically done through Datadog Autodiscovery, where the agent identifies services annotated for Prometheus scraping. Alternatively, you can explicitly configure the Datadog Agent's `prometheus.yaml` file. The following Terraform snippet, building on the Datadog Agent Helm release, illustrates how to enable Prometheus scraping and configure a basic service discovery rule. This configuration tells the Datadog Agent to look for pods with specific annotations that expose Prometheus metrics, and then scrape those metrics. The integration ensures that all your critical metrics, whether from native Datadog integrations or custom Prometheus exporters, are available for analysis and alerting within your Datadog account.

resource "helm_release" "datadog_agent_with_prometheus_scrape" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  create_namespace = true
  # ... (other datadog agent settings from previous block) ...

  set {
    name  = "agents.config.prometheus_scrape.enabled"
    value = "true"
  }

  set {
    name  = "agents.config.prometheus_scrape.service_endpoints"
    value = "true" # This enables scraping based on service annotations
  }

  # Example of an explicit Prometheus check configuration (optional, typically prefer Autodiscovery)
  # set {
  #   name  = "agents.config.integrations.prometheus_check.init_config"
  #   value = "{}"
  #   type  = "yaml"
  # }
  # set {
  #   name  = "agents.config.integrations.prometheus_check.instances"
  #   value = <PagerDuty for Incident Response and Alert ManagementEffective observability is incomplete without a robust incident management system. PagerDuty is a leading platform for orchestrating incident response, providing on-call scheduling, automated escalations, and actionable alerts. Integrating PagerDuty with your EKS observability stack ensures that critical issues detected by Datadog or Prometheus (via Alertmanager) are immediately routed to the right teams, reducing MTTR (Mean Time To Resolution). Instead of alerts getting lost in a sea of notifications, PagerDuty intelligently routes them based on severity, service impact, and on-call schedules, ensuring timely acknowledgment and resolution. This crucial component transforms raw data and alerts into actionable incidents, fostering a culture of accountability and efficiency in your operational workflows. The ability to define service dependencies, escalation policies, and incident workflows within PagerDuty is paramount for maintaining high availability and reliability for your EKS-hosted applications.
Terraform can be used to manage PagerDuty resources, including services, escalation policies, and users, bringing your incident response configuration under infrastructure as code. This approach ensures consistency, auditability, and ease of modification for your on-call rotations and alerting rules. The following Terraform configuration demonstrates how to define a PagerDuty service with an associated escalation policy. This service acts as the entry point for alerts originating from Datadog or Prometheus. By managing PagerDuty configurations through Terraform, you can seamlessly integrate your incident response strategy with your EKS infrastructure and monitoring deployments, creating a fully automated and resilient operational pipeline.

resource "pagerduty_user" "devops_engineer" {
  name  = "DevOps Engineer"
  email = "devops-engineer@example.com"
}

resource "pagerduty_escalation_policy" "eks_critical_ep" {
  name      = "EKS Critical Escalation Policy"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 15
    target {
      type = "user"
      id   = pagerduty_user.devops_engineer.id
    }
  }

  rule {
    escalation_delay_in_minutes = 30
    target {
      type = "user"
      id   = pagerduty_user.devops_engineer.id # Could be another user or a team
    }
  }
}

resource "pagerduty_service" "eks_observability_service" {
  name                      = "EKS Observability Service"
  auto_resolve_timeout_days = 1
  acknowledgement_timeout_minutes = 10
  escalation_policy         = pagerduty_escalation_policy.eks_critical_ep.id
}
Orchestrating Alerts: Terraform-Driven Notification ChainsThe final piece of our EKS observability puzzle is to automate the alerting mechanism, connecting the detected anomalies from Datadog and Prometheus directly to PagerDuty. Terraform allows us to define Datadog monitors and Prometheus Alertmanager configurations as code, specifying thresholds, query logic, and most importantly, notification channels. This creates a fully automated feedback loop: metrics are collected, analyzed for deviations, and if a critical threshold is breached, an incident is automatically triggered in PagerDuty, alerting the appropriate on-call personnel. This entire process, from infrastructure provisioning to alert definition and incident routing, is managed through version-controlled Terraform code, guaranteeing consistency and enabling rapid iteration on your observability strategy. Establishing these automated notification chains is paramount for reducing mean time to detection (MTTD) and mean time to recovery (MTTR), thereby enhancing the overall reliability of your EKS applications.
For Datadog, we can define monitors using the `datadog_monitor` resource. These monitors can evaluate metrics from EKS, applications, or even Prometheus-ingested data, triggering alerts based on predefined conditions. For Prometheus, alerts are defined within Alertmanager configuration, which can also be managed via Terraform (though often directly via a ConfigMap in Kubernetes). The Datadog integration with PagerDuty is straightforward, simply requiring the PagerDuty service integration key. The following Terraform example shows how to create a Datadog monitor that alerts on high CPU utilization in EKS nodes and routes these alerts to the previously defined PagerDuty service. This demonstrates the power of consolidating monitoring, alerting, and incident response into a coherent, automated, and code-driven system, ensuring your EKS cluster maintains optimal performance and reliability.

# Retrieve the PagerDuty service integration key
data "pagerduty_service_integration" "eks_observability_integration" {
  service_id = pagerduty_service.eks_observability_service.id
  type       = "generic_events_api_v2" # or "datadog_v2" if specific
}

resource "datadog_monitor" "eks_high_cpu_alert" {
  name                = "[EKS] High CPU Utilization on Node: {{host.name}}"
  type                = "metric alert"
  query               = "avg(last_5m):avg:system.cpu.idle{kubernetes_cluster_name:observability-eks-cluster} by {host} < 10" # less than 10% idle = high usage
  message             = <Advanced Observability Practices and Best RecommendationsBeyond the foundational setup, achieving truly advanced observability for AWS EKS requires continuous refinement and adherence to best practices. This includes implementing robust logging strategies where all application and system logs are centralized (e.g., into Datadog Logs or AWS CloudWatch Logs, then forwarded), enabling distributed tracing for microservices (Datadog APM excels here), and regularly reviewing and optimizing monitoring dashboards. Consider implementing chaos engineering experiments to test the resilience of your observability stack and incident response workflows under adverse conditions. Leverage advanced features like Datadog's Watchdog for anomaly detection and Prometheus's recording rules for pre-aggregating frequently queried metrics, improving dashboard performance and reducing alert latency. Furthermore, ensure that your Terraform configurations for observability components are part of a CI/CD pipeline, allowing for automated testing and deployment of changes.
Cost optimization is another critical aspect. While comprehensive observability is crucial, it's also important to manage the volume of metrics, logs, and traces ingested by Datadog and stored by Prometheus. Regularly audit your metric cardinality, prune unnecessary logs, and implement sampling for traces where appropriate. Utilize Datadog's cost management features and consider optimizing Prometheus retention policies. Finally, educate your development and operations teams on how to effectively use these tools, interpret dashboards, and respond to alerts. A powerful observability stack is only as effective as the teams using it. By combining Terraform's declarative power with the comprehensive capabilities of Datadog, Prometheus, and PagerDuty, organizations can build a resilient, scalable, and highly observable AWS EKS environment that drives operational excellence and fosters a proactive approach to application reliability.

CloudStack YAML