Terraform-Managed AWS EKS Observability: Datadog Monitoring and PagerDuty Alerting

Architecture Pro-Tip: Always establish your observability baseline (logging, metrics, traces) as an integral part of your Infrastructure as Code (IaC) from day one. This proactive approach ensures consistent monitoring, faster incident response, and prevents critical blind spots in your cloud-native environments, especially with dynamic platforms like AWS EKS.

Terraform-Managed AWS EKS Observability: Datadog Monitoring and PagerDuty Alerting

In the dynamic landscape of cloud-native applications, managing and monitoring AWS Elastic Kubernetes Service (EKS) clusters effectively is paramount. As organizations scale, ensuring robust observability—encompassing metrics, logs, and traces—becomes a critical challenge. This guide provides a comprehensive, technical walkthrough on establishing a state-of-the-art observability stack for your AWS EKS environment, fully managed with Terraform, leveraging the power of Datadog for monitoring and PagerDuty for incident alerting.

By treating your observability infrastructure as code (IaC), you gain benefits like version control, repeatability, and disaster recovery for your monitoring and alerting configurations. This article will detail how to deploy and configure the Datadog Agent on EKS, set up critical monitors and dashboards, and integrate with PagerDuty for streamlined incident response, all defined and managed through Terraform.

Why Terraform for EKS Observability?

Terraform, HashiCorp's open-source IaC tool, enables you to define and provision infrastructure using a declarative configuration language. Extending this paradigm to observability offers significant advantages:

Consistency: Ensure identical monitoring setups across development, staging, and production environments.
Version Control: Track changes to your monitoring configurations, revert to previous states, and collaborate effectively.
Automation: Automate the deployment and scaling of observability agents, monitors, and dashboards alongside your EKS cluster and applications.
Reduced Manual Error: Eliminate human error associated with manual configuration through UI clicks.
Auditability: Maintain a clear audit trail of all changes to your observability stack.

Core Components of Our Observability Stack

AWS EKS: The Foundation

AWS EKS provides a managed Kubernetes service, abstracting away the complexities of Kubernetes control plane management. Our focus here is on ensuring the applications and infrastructure running within EKS are fully observable.

Datadog: Comprehensive Monitoring

Datadog is a leading monitoring and analytics platform that offers full-stack visibility. For EKS, Datadog collects metrics, logs, and traces from your cluster nodes, pods, containers, and applications. Its powerful dashboards, synthetic monitoring, and AI-driven alerts provide deep insights into your cluster's health and performance.

PagerDuty: Actionable Alerting

PagerDuty is an incident management platform that integrates with monitoring tools like Datadog to provide reliable, real-time alerts and on-call scheduling. It ensures that critical incidents are escalated to the right teams immediately, minimizing downtime and business impact.

Prerequisites

Before you begin, ensure you have the following:

AWS Account and CLI: Configured with appropriate permissions to manage EKS and IAM resources.
Terraform CLI: Installed and configured on your local machine (version 1.0+ recommended).
Kubectl: Installed and configured to interact with your EKS cluster.
Helm CLI: Installed (version 3.x+ recommended) for deploying the Datadog Agent.
Datadog Account: With your Datadog API Key and Application Key.
PagerDuty Account: With an Integration Key or Service ID for Datadog integration.

Step-by-Step Implementation with Terraform

1. Setting Up Your Terraform Project

Start by creating a new directory for your Terraform project. You'll need to define providers for AWS, Kubernetes, Helm, and Datadog.

Ensure your AWS EKS cluster is already provisioned. This guide assumes you have an existing EKS cluster and its configuration (e.g., Kubeconfig, OIDC provider ARN) is accessible. If you're provisioning EKS with Terraform, you'd typically use the terraform-aws-modules/eks/aws module.

2. Deploying the Datadog Agent on EKS

The Datadog Agent is deployed as a DaemonSet within your EKS cluster, ensuring an agent runs on every node to collect metrics, logs, and traces. We'll use the Helm provider for this, as Datadog provides a robust Helm chart.

For enhanced security and best practices, it's recommended to configure IAM Roles for Service Accounts (IRSA) for the Datadog Agent. This allows the Agent to assume an AWS IAM role with specific permissions (e.g., for EC2, CloudWatch, EKS APIs) without storing AWS credentials directly in Kubernetes secrets.

resource "aws_iam_policy" "datadog_agent_policy" {
  name        = "datadog-agent-policy-eks-observability"
  description = "IAM policy for Datadog Agent in EKS for AWS integration."

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action = [
          "ec2:DescribeInstances",
          "ec2:DescribeVolumes",
          "ec2:DescribeTags",
          "tag:GetResources",
          "logs:DescribeLogGroups",
          "logs:DescribeLogStreams",
          "logs:FilterLogEvents",
          "logs:GetLogEvents",
          "logs:PutLogEvents",
          "s3:GetBucketTagging",
          "s3:GetLifecycleConfiguration",
          "s3:GetBucketLocation",
          "s3:ListAllMyBuckets",
          "s3:GetBucketNotification",
          "s3:GetBucketLogging",
          "s3:GetBucketPolicy",
          "s3:GetBucketCORS",
          "s3:GetBucketAcl",
          "s3:GetBucketWebsite",
          "s3:GetBucketVersioning",
          "s3:GetBucketRequestPayment",
          "s3:GetBucketPublicAccessBlock",
          "s3:GetEncryptionConfiguration",
          "s3:GetBucketObjectLockConfiguration",
          "s3:GetReplicationConfiguration",
          "s3:ListBucket",
          "s3:GetObject" # If you need to read logs from S3 directly
          # Add more AWS services as needed for specific Datadog integrations
        ],
        Effect   = "Allow",
        Resource = "*"
      },
      {
        Action = [
          "eks:ListClusters",
          "eks:DescribeCluster",
          "eks:AccessKubernetesApi",
          "eks:ListFargateProfiles",
          "eks:DescribeFargateProfile"
        ],
        Effect   = "Allow",
        Resource = "arn:aws:eks:${var.aws_region}:${data.aws_caller_identity.current.account_id}:cluster/${var.eks_cluster_name}"
      },
      {
        Action = [
          "iam:CreateServiceLinkedRole",
          "iam:ListAttachedRolePolicies",
          "iam:ListPolicies",
          "iam:ListRoles",
          "iam:ListUsers"
        ],
        Effect   = "Allow",
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_role" "datadog_agent_role" {
  name = "datadog-agent-role-eks-observability"

  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect = "Allow",
        Principal = {
          Federated = data.aws_iam_openid_connect_provider.eks_oidc_provider.arn
        },
        Action = "sts:AssumeRoleWithWebIdentity",
        Condition = {
          StringEquals = {
            "${data.aws_iam_openid_connect_provider.eks_oidc_provider.url}:sub" : "system:serviceaccount:default:datadog-agent", # Adjust namespace and service account name if different
            "${data.aws_iam_openid_connect_provider.eks_oidc_provider.url}:aud" : "sts.amazonaws.com"
          }
        }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "datadog_agent_attachment" {
  role       = aws_iam_role.datadog_agent_role.name
  policy_arn = aws_iam_policy.datadog_agent_policy.arn
}

resource "kubernetes_service_account" "datadog_agent" {
  metadata {
    name      = "datadog-agent"
    namespace = "default" # Or dedicated datadog namespace
    annotations = {
      "eks.amazonaws.com/role-arn" = aws_iam_role.datadog_agent_role.arn
    }
  }
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default" # Or dedicated datadog namespace
  version    = "2.35.0" # Use a specific chart version

  values = [
    templatefile("${path.module}/datadog_agent_values.yaml", {
      datadog_api_key        = var.datadog_api_key
      datadog_app_key        = var.datadog_app_key
      cluster_name           = var.eks_cluster_name
      kube_state_metrics_url = "http://kube-state-metrics.kube-system.svc.cluster.local:8080/metrics" # Adjust if kube-state-metrics is in a different namespace/service name
    })
  ]

  depends_on = [
    aws_iam_role_policy_attachment.datadog_agent_attachment,
    kubernetes_service_account.datadog_agent
  ]
}

# Example datadog_agent_values.yaml template:
# agents:
#   image:
#     repository: gcr.io/datadoghq/agent
#     tag: 7.51.0
# clusterAgent:
#   enabled: true
#   image:
#     repository: gcr.io/datadoghq/cluster-agent
#     tag: 1.34.0
# datadog:
#   apiKey: ${datadog_api_key}
#   appKey: ${datadog_app_key}
#   site: "datadoghq.com" # or eu.datadoghq.com, us3.datadoghq.com, etc.
#   clusterName: ${cluster_name}
#   env:
#     - name: DD_KUBERNETES_KUBE_STATE_METRICS_URL
#       value: ${kube_state_metrics_url}
#   kubelet:
#     host:
#       collectReporting: true
# logs:
#   enabled: true
#   containerCollectAll: true
#   autoMultiLine: true
# apm:
#   enabled: true
#   hostPort: 8126
# processAgent:
#   enabled: true
# networkMonitoring:
#   enabled: true
# systemProbe:
#   enabled: true
# serviceAccount:
#   create: false # We create it with Terraform
#   name: datadog-agent
# rbac:
#   create: true
# clusterAgent:
#   rbac:
#     create: true
#   serviceAccount:
#     create: false # We create it with Terraform
#     name: datadog-agent
    

3. Configuring Datadog Monitors and Dashboards with Terraform

Once the Datadog Agent is collecting data, you can define monitors to alert on specific conditions and create dashboards for visualization. Terraform allows you to manage these resources declaratively using the Datadog provider.

Datadog Monitor Example: EKS Node CPU Utilization

Let's create a monitor that alerts when any EKS node's CPU utilization exceeds a certain threshold.

resource "datadog_monitor" "eks_node_cpu_high" {
  name               = "[EKS] High Node CPU Utilization on {{host.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:system.cpu.idle{eks_cluster_name:${var.eks_cluster_name}} by {host} < 10" # Alert if idle CPU is < 10%
  message            = "High CPU utilization detected on EKS node {{host.name}} ({{system.cpu.idle}}% idle). Investigate potential resource contention or rogue processes."
  escalation_message = "CPU utilization remains high on {{host.name}}. Escalating to on-call."
  tags               = ["eks", "kubernetes", "cpu", "critical"]
  thresholds = {
    warning  = 20
    critical = 10
  }
  notify_no_data      = false
  new_group_delay     = 60
  new_host_delay      = 300
  renotify_interval   = 120
  timeout_h           = 0
  no_data_timeframe   = 20
  require_full_window = true
  notify_audit        = false
  force_delete        = false

  # Optional: Integrate with PagerDuty for critical alerts
  # Add this if you have a PagerDuty integration configured in Datadog
  # See next section for PagerDuty integration.
  # If you set `query` to target PagerDuty directly, replace the example with:
  # `query = "avg(last_5m):avg:system.cpu.idle{eks_cluster_name:${var.eks_cluster_name}} by {host}.rollup(avg, 60) < 10 and ${datadog_integration_pagerduty.my_pagerduty_integration.query_tag}"`
}

resource "datadog_dashboard" "eks_overview_dashboard" {
  title        = "EKS Cluster Overview - ${var.eks_cluster_name}"
  description  = "High-level overview of EKS cluster health and performance."
  layout_type  = "ordered"
  is_read_only = false
  tags         = ["eks", "kubernetes", "overview"]

  widget {
    # Node CPU Usage Graph
    timeseries_definition {
      title = "EKS Node CPU Utilization"
      requests {
        q     = "avg:system.cpu.usage{eks_cluster_name:${var.eks_cluster_name}} by {host}"
        display_type = "line"
        style {
          palette = "dog_classic"
          type    = "solid"
          width   = "normal"
        }
      }
    }
    layout {
      x      = 0
      y      = 0
      width  = 6
      height = 4
    }
  }

  widget {
    # Node Memory Usage Graph
    timeseries_definition {
      title = "EKS Node Memory Utilization"
      requests {
        q     = "avg:system.mem.used{eks_cluster_name:${var.eks_cluster_name}} by {host}"
        display_type = "line"
        style {
          palette = "dog_classic"
          type    = "solid"
          width   = "normal"
        }
      }
    }
    layout {
      x      = 6
      y      = 0
      width  = 6
      height = 4
    }
  }

  widget {
    # Pod Count by Status
    query_value_definition {
      title = "Running Pods"
      requests {
        q = "sum:kubernetes.pods.running{eks_cluster_name:${var.eks_cluster_name}}"
      }
    }
    layout {
      x      = 0
      y      = 4
      width  = 3
      height = 2
    }
  }

  widget {
    # Pending Pods
    query_value_definition {
      title = "Pending Pods"
      requests {
        q = "sum:kubernetes.pods.pending{eks_cluster_name:${var.eks_cluster_name}}"
      }
    }
    layout {
      x      = 3
      y      = 4
      width  = 3
      height = 2
    }
  }
  # Add more widgets as needed for a comprehensive EKS dashboard
}
    

4. Integrating Datadog with PagerDuty for Incident Response

To ensure critical alerts are actionable, integrate Datadog with PagerDuty. This typically involves setting up the integration in Datadog and then referencing it in your monitors. The `datadog_integration_pagerduty` resource can manage the integration itself.

# First, create the PagerDuty integration in Datadog via Terraform
resource "datadog_integration_pagerduty" "my_pagerduty_integration" {
  services {
    service_key = var.pagerduty_integration_key # PagerDuty integration key for a specific service
    service_name = "EKS Critical Alerts"
  }
}

# Now, update your Datadog monitor to notify PagerDuty
resource "datadog_monitor" "eks_node_cpu_critical_pagerduty" {
  name               = "[EKS CRITICAL] High Node CPU Utilization on {{host.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:system.cpu.idle{eks_cluster_name:${var.eks_cluster_name}} by {host} < 10"
  message            = "@pagerduty-EKS Critical Alerts High CPU utilization detected on EKS node {{host.name}}. Investigate immediately!" # Use @pagerduty-[Service Name]
  tags               = ["eks", "kubernetes", "cpu", "critical", "pagerduty"]
  thresholds = {
    critical = 10
  }
  notify_no_data      = false
  new_group_delay     = 60
  renotify_interval   = 120
  timeout_h           = 0
  no_data_timeframe   = 20
  require_full_window = true
  notify_audit        = false
  force_delete        = false

  # You can also use query_tag from the integration resource if more complex routing is needed
  # e.g., query = "... and ${datadog_integration_pagerduty.my_pagerduty_integration.query_tag}"
}
    

Best Practices for Production-Grade EKS Observability

To maximize the effectiveness and efficiency of your EKS observability strategy:

Granular RBAC for Datadog Agent: Implement the principle of least privilege for the Datadog Agent's Kubernetes Service Account and associated IAM Role. Only grant the necessary permissions.
Consistent Tagging Strategy: Leverage AWS and Kubernetes tags (which Datadog can ingest) consistently across your resources. This allows for powerful filtering, aggregation, and segmentation in Datadog dashboards and monitors.
Cost Optimization: While Datadog is powerful, it can be costly at scale. Be strategic about what you monitor. Use sampling for traces, configure log retention, and only collect metrics essential for operational visibility.
Custom Metrics and APM Integration: Beyond infrastructure metrics, ensure your applications expose custom metrics (e.g., Prometheus format) that Datadog can scrape. Integrate Datadog APM for distributed tracing to gain end-to-end visibility into application performance.
Proactive Alerting & Runbooks: Focus on creating proactive alerts that notify you *before* an issue impacts users. Couple critical alerts with well-documented runbooks in PagerDuty to guide on-call engineers through initial troubleshooting and resolution steps.
Secrets Management: Use AWS Secrets Manager or HashiCorp Vault to securely manage your Datadog API/App keys and PagerDuty integration keys, referencing them in Terraform using data sources.

Troubleshooting Common Issues

Datadog Agent Not Reporting

If your Datadog Agent isn't reporting data:

Check Pod Status: Use kubectl get pods -n <datadog-namespace> to ensure all Datadog Agent pods are running.
Review Agent Logs: Use kubectl logs <datadog-agent-pod-name> -n <datadog-namespace> to check for API key issues, connectivity problems, or misconfigurations.
Verify API/App Keys: Ensure var.datadog_api_key and var.datadog_app_key are correct and active in your Datadog account.
IRSA Permissions: Confirm the IAM role attached to the Datadog Service Account has the necessary permissions and the OIDC provider is correctly configured.
Network Connectivity: Ensure your EKS nodes can reach Datadog endpoints (e.g., https://api.datadoghq.com).

Missing Metrics or Logs

If specific data types are missing:

Helm Chart Values: Double-check the datadog_agent_values.yaml for correct settings like logs.enabled, apm.enabled, and specific integration configurations.
Container/Pod Annotations: For auto-discovery, ensure your application pods have the correct Datadog annotations.
EKS Cluster Agent: Ensure the Cluster Agent is enabled and working correctly for Kubernetes-specific metrics and events.

Alerts Not Firing

If your Datadog monitors aren't triggering PagerDuty incidents:

Monitor Query: Verify the monitor's query is correct and actually detecting the condition you expect. Use the Datadog UI to test the query against historical data.
Notification Message: Confirm the @pagerduty-[Service Name] syntax is correct in the monitor's message field, matching the integration name.
PagerDuty Integration: Ensure the datadog_integration_pagerduty resource successfully created the integration and the service_key is valid in PagerDuty.
PagerDuty Service Status: Check the PagerDuty service linked to the integration for any issues (e.g., maintenance mode, disabled).

Conclusion

Establishing robust observability for AWS EKS is not just a best practice; it's a necessity for maintaining healthy, high-performing, and resilient cloud-native applications. By harnessing the declarative power of Terraform, coupled with Datadog's comprehensive monitoring capabilities and PagerDuty's incident response efficiency, you can build a highly automated, scalable, and reliable observability stack.

This guide provides the foundational steps and Terraform configurations to get you started. Remember to adapt the examples to your specific requirements, explore Datadog's vast array of integrations, and continually refine your monitoring and alerting strategies to meet the evolving demands of your EKS environments.

CloudStack YAML