Terraform for AWS EKS: Automated Datadog Observability and PagerDuty Integration
Architecture Pro-Tip: Always treat your observability infrastructure as code. This ensures consistency, simplifies disaster recovery, and allows for seamless integration into your CI/CD pipelines, mirroring the agility of your application deployments. Version control your Datadog dashboards, monitors, and PagerDuty services alongside your EKS cluster definitions for a truly unified operations model.
Terraform for AWS EKS: Automated Datadog Observability and PagerDuty Integration
In the dynamic world of cloud-native applications, maintaining robust observability for Kubernetes clusters running on AWS EKS is paramount. Manually configuring monitoring agents, dashboards, and alerting systems is not only time-consuming but also prone to errors and inconsistencies. This guide demonstrates how to leverage Terraform to automate the deployment of Datadog for comprehensive EKS observability and integrate it seamlessly with PagerDuty for incident management, ensuring your critical services are always monitored and actionable alerts are delivered.
Why Automate Observability with Terraform?
Terraform, as an Infrastructure as Code (IaC) tool, offers significant advantages for managing observability components:
- Consistency: Ensure all EKS clusters have uniform monitoring and alerting configurations.
- Version Control: Track changes to your observability setup, enabling rollbacks and audits.
- Efficiency: Rapidly provision and update Datadog agents, monitors, dashboards, and PagerDuty services across environments.
- Reduced Manual Error: Eliminate human error associated with manual configuration.
- Scalability: Easily extend observability to new clusters or services without additional manual effort.
Prerequisites
Before diving into the Terraform configuration, ensure you have the following:
- An active AWS account with an existing EKS cluster.
- AWS CLI configured with appropriate permissions.
- Terraform installed (version 1.0+ recommended).
- A Datadog account with API and Application keys.
- A PagerDuty account with an API key and the intent to create a service integration.
kubectlconfigured to interact with your EKS cluster.
Datadog for EKS Observability
Datadog offers a unified platform for monitoring, logging, and tracing. For EKS, the Datadog Agent is deployed as a DaemonSet to collect metrics, logs, and events from all nodes and pods. The Datadog Terraform provider allows us to programmatically define monitors, dashboards, and notification channels.
Key Datadog Components for EKS
- Datadog Agent: A lightweight agent deployed on each EKS node to collect infrastructure metrics, application metrics, logs, and traces.
- Monitors: Rules that trigger alerts based on specific metric thresholds, log patterns, or event occurrences.
- Dashboards: Visualizations to track the health and performance of your EKS cluster and applications.
- Notification Channels: Integrations with services like PagerDuty to route alerts to the right teams.
PagerDuty for Incident Management
PagerDuty acts as the central hub for incident response. When Datadog detects an issue that requires human intervention, it can trigger an incident in PagerDuty, routing it to the appropriate on-call team based on predefined schedules and escalation policies. The PagerDuty Terraform provider enables the creation of services, escalation policies, and integration keys.
Terraform Configuration Walkthrough
This section guides you through the Terraform setup to deploy the Datadog Agent, configure basic monitors, and integrate with PagerDuty.
1. Provider Configuration
First, define the necessary providers for AWS, Datadog, and PagerDuty. Remember to keep sensitive API keys out of your code; use environment variables or a secrets manager.
provider "aws" {
region = "us-east-1"
}
provider "datadog" {
api_key = var.datadog_api_key
app_key = var.datadog_app_key
}
provider "pagerduty" {
token = var.pagerduty_api_token
}
variable "datadog_api_key" {
description = "Datadog API Key"
type = string
sensitive = true
}
variable "datadog_app_key" {
description = "Datadog Application Key"
type = string
sensitive = true
}
variable "pagerduty_api_token" {
description = "PagerDuty API Token"
type = string
sensitive = true
}
variable "eks_cluster_name" {
description = "Name of the existing EKS cluster"
type = string
}
variable "eks_cluster_region" {
description = "Region of the existing EKS cluster"
type = string
default = "us-east-1"
}
2. Reference Existing EKS Cluster
We'll use a data source to fetch information about your pre-existing EKS cluster. This is crucial for configuring kubectl and the Datadog Agent.
data "aws_eks_cluster" "eks_cluster" {
name = var.eks_cluster_name
}
data "aws_eks_cluster_auth" "eks_cluster_auth" {
name = var.eks_cluster_name
}
# Used to connect kubectl locally for the Datadog agent
provider "kubernetes" {
host = data.aws_eks_cluster.eks_cluster.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority.0.data)
token = data.aws_eks_cluster_auth.eks_cluster_auth.token
}
3. Deploy Datadog Agent to EKS
The Datadog Agent is best deployed using its official Helm chart. However, for simplicity and demonstration within a single Terraform file, we'll embed the Kubernetes manifest (DaemonSet, RBAC) directly, assuming a basic setup. In a production scenario, using the helm_release resource from the Helm provider is recommended.
This manifest creates a ServiceAccount, ClusterRole, ClusterRoleBinding, and the DaemonSet for the Datadog Agent. Remember to replace <YOUR_DATADOG_API_KEY> with your actual Datadog API key, though using environment variables is more secure.
Ready-to-Use Configuration: Full Terraform Example
# main.tf
# --- Providers ---
provider "aws" {
region = var.eks_cluster_region
}
provider "kubernetes" {
host = data.aws_eks_cluster.eks_cluster.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority.0.data)
token = data.aws_eks_cluster_auth.eks_cluster_auth.token
}
provider "datadog" {
api_key = var.datadog_api_key
app_key = var.datadog_app_key
}
provider "pagerduty" {
token = var.pagerduty_api_token
}
# --- Variables (Define in a variables.tf or pass via -var) ---
variable "datadog_api_key" {
description = "Datadog API Key"
type = string
sensitive = true
}
variable "datadog_app_key" {
description = "Datadog Application Key"
type = string
sensitive = true
}
variable "pagerduty_api_token" {
description = "PagerDuty API Token"
type = string
sensitive = true
}
variable "eks_cluster_name" {
description = "Name of the existing EKS cluster"
type = string
}
variable "eks_cluster_region" {
description = "Region of the existing EKS cluster"
type = string
default = "us-east-1"
}
# --- Data Sources ---
data "aws_eks_cluster" "eks_cluster" {
name = var.eks_cluster_name
}
data "aws_eks_cluster_auth" "eks_cluster_auth" {
name = var.eks_cluster_name
}
# --- Kubernetes Manifests for Datadog Agent ---
resource "kubernetes_namespace" "datadog_agent_ns" {
metadata {
name = "datadog"
}
}
resource "kubernetes_service_account" "datadog_sa" {
metadata {
name = "datadog-agent"
namespace = kubernetes_namespace.datadog_agent_ns.metadata[0].name
}
}
resource "kubernetes_cluster_role" "datadog_cr" {
metadata {
name = "datadog-agent"
}
rule {
api_groups = ["", "apps", "extensions", "batch", "events.k8s.io", "admissionregistration.k8s.io"]
resources = ["pods", "nodes", "endpoints", "services", "componentstatuses", "events", "configmaps", "replicationcontrollers", "replicasets", "statefulsets", "daemonsets", "deployments", "jobs", "cronjobs"]
verbs = ["get", "list", "watch"]
}
rule {
api_groups = [""]
resources = ["secrets"]
verbs = ["get"]
}
rule {
api_groups = ["policy"]
resources = ["podsecuritypolicies"]
verbs = ["use"]
resource_names = ["datadog-agent"]
}
}
resource "kubernetes_cluster_role_binding" "datadog_crb" {
metadata {
name = "datadog-agent"
}
role_ref {
api_group = "rbac.authorization.k8s.io"
kind = "ClusterRole"
name = kubernetes_cluster_role.datadog_cr.metadata[0].name
}
subject {
kind = "ServiceAccount"
name = kubernetes_service_account.datadog_sa.metadata[0].name
namespace = kubernetes_namespace.datadog_agent_ns.metadata[0].name
}
}
resource "kubernetes_daemonset" "datadog_agent_ds" {
metadata {
name = "datadog-agent"
namespace = kubernetes_namespace.datadog_agent_ns.metadata[0].name
labels = {
app = "datadog-agent"
}
}
spec {
selector {
match_labels = {
app = "datadog-agent"
}
}
template {
metadata {
labels = {
app = "datadog-agent"
}
}
spec {
service_account_name = kubernetes_service_account.datadog_sa.metadata[0].name
containers {
name = "datadog-agent"
image = "gcr.io/datadog-prod/agent:7.50.0" # Use a stable, recent version
env {
name = "DD_API_KEY"
value = var.datadog_api_key
}
env {
name = "DD_SITE"
value = "datadoghq.com" # Or datadoghq.eu, etc.
}
env {
name = "DD_KUBERNETES_KUBELET_HOST"
value_from {
field_ref {
field_path = "status.hostIP"
}
}
}
env {
name = "DD_EKS_FARGATE"
value = "false" # Set to true if using Fargate nodes
}
env {
name = "DD_LOGS_ENABLED"
value = "true"
}
env {
name = "DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL"
value = "true"
}
env {
name = "DD_PROCESS_AGENT_ENABLED"
value = "true"
}
env {
name = "DD_APM_ENABLED"
value = "true"
}
resources {
limits = {
cpu = "200m"
memory = "256Mi"
}
requests = {
cpu = "100m"
memory = "128Mi"
}
}
volume_mounts {
name = "proc"
mount_path = "/host/proc"
read_only = true
}
volume_mounts {
name = "cgroups"
mount_path = "/host/sys/fs/cgroup"
read_only = true
}
volume_mounts {
name = "docker-sock"
mount_path = "/var/run/docker.sock"
}
volume_mounts {
name = "cri-sock"
mount_path = "/var/run/crio/crio.sock"
read_only = true
}
volume_mounts {
name = "log-path"
mount_path = "/var/log"
read_only = true
}
volume_mounts {
name = "run-path"
mount_path = "/opt/datadog-agent/run"
}
}
volumes {
name = "proc"
host_path {
path = "/proc"
}
}
volumes {
name = "cgroups"
host_path {
path = "/sys/fs/cgroup"
}
}
volumes {
name = "docker-sock"
host_path {
path = "/var/run/docker.sock"
}
}
volumes {
name = "cri-sock"
host_path {
path = "/var/run/crio/crio.sock"
}
}
volumes {
name = "log-path"
host_path {
path = "/var/log"
}
}
volumes {
name = "run-path"
host_path {
path = "/opt/datadog-agent/run"
}
}
toleration {
operator = "Exists"
}
}
}
}
}
# --- PagerDuty Service and Integration ---
resource "pagerduty_user" "oncall_devops" {
name = "DevOps On-Call User" # Replace with actual user if needed
email = "devops@example.com" # Replace with actual email
}
resource "pagerduty_escalation_policy" "devops_escalation_policy" {
name = "DevOps Escalation Policy"
num_loops = 2
rule {
delay_in_minutes = 5
target {
id = pagerduty_user.oncall_devops.id
type = "user"
}
}
}
resource "pagerduty_service" "eks_observability_service" {
name = "${var.eks_cluster_name}-Observability"
auto_resolve_timeout_s = 14400 # 4 hours
acknowledgement_timeout_s = 600 # 10 minutes
escalation_policy = pagerduty_escalation_policy.devops_escalation_policy.id
}
resource "pagerduty_service_integration" "datadog_integration" {
name = "Datadog Integration"
type = "datadog_api_inbound_integration"
service_id = pagerduty_service.eks_observability_service.id
}
# --- Datadog Monitors ---
resource "datadog_monitor" "high_cpu_utilization" {
name = "[EKS-${var.eks_cluster_name}] High Node CPU Utilization"
type = "metric alert"
query = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:${var.eks_cluster_name}} by {host} > 80"
message = "Node {{host.name}} in EKS cluster ${var.eks_cluster_name} has high CPU utilization ({{value}}%). @pagerduty-EKS-Observability"
escalation_message = "CPU utilization remains high. Escalating!"
monitor_thresholds {
critical = 80
warning = 70
}
notify_no_data = false
renotify_interval = 60
tags = ["environment:production", "service:eks", "cluster:${var.eks_cluster_name}"]
new_group_delay = 60
include_tags = true
require_full_window = false
# Integrates with PagerDuty via the service_integration_key
# The name 'pagerduty-EKS-Observability' is constructed from 'pagerduty-' + the service name (eks_observability_service.name)
# This relies on Datadog's automatic PagerDuty integration name convention.
}
resource "datadog_monitor" "pod_restarts_alert" {
name = "[EKS-${var.eks_cluster_name}] Frequent Pod Restarts"
type = "metric alert"
query = "sum(last_5m):sum:kubernetes.containers.restarts{cluster_name:${var.eks_cluster_name}} by {kube_deployment} > 5"
message = "Deployment {{kube_deployment.name}} in EKS cluster ${var.eks_cluster_name} is experiencing frequent pod restarts ({{value}} restarts in last 5 minutes). @pagerduty-EKS-Observability"
escalation_message = "Pod restarts still occurring. Investigating."
monitor_thresholds {
critical = 5
warning = 2
}
notify_no_data = false
renotify_interval = 60
tags = ["environment:production", "service:eks", "cluster:${var.eks_cluster_name}"]
new_group_delay = 60
include_tags = true
require_full_window = false
}
# Example: Datadog Dashboard (optional but recommended)
resource "datadog_dashboard" "eks_overview_dashboard" {
title = "[EKS - ${var.eks_cluster_name}] Overview"
description = "Overview of EKS Cluster Metrics for ${var.eks_cluster_name}"
layout_type = "ordered"
is_read_only = true
widget {
definition {
title = "Node CPU Utilization"
type = "timeseries"
requests {
q = "avg:kubernetes.cpu.usage.total{cluster_name:${var.eks_cluster_name}} by {host}"
display_type = "area"
}
}
}
widget {
definition {
title = "Node Memory Utilization"
type = "timeseries"
requests {
q = "avg:kubernetes.memory.usage.total{cluster_name:${var.eks_cluster_name}} by {host}"
display_type = "area"
}
}
}
widget {
definition {
title = "Pod Restarts"
type = "timeseries"
requests {
q = "sum:kubernetes.containers.restarts{cluster_name:${var.eks_cluster_name}} by {kube_deployment}"
display_type = "line"
}
}
}
}
4. Deployment Steps
To deploy this configuration:
- Save the code above as
main.tfin a new directory. - Set your Datadog API Key, Application Key, and PagerDuty API Token as environment variables (e.g.,
TF_VAR_datadog_api_key,TF_VAR_pagerduty_api_token) or create aterraform.tfvarsfile (though environment variables are recommended for sensitive data). - Initialize your Terraform workspace:
terraform init
- Review the planned changes:
terraform plan -var="eks_cluster_name=your-eks-cluster-name" -var="eks_cluster_region=your-aws-region"
- Apply the configuration:
terraform apply -var="eks_cluster_name=your-eks-cluster-name" -var="eks_cluster_region=your-aws-region"
5. Verification and Validation
After applying the Terraform configuration:
- Datadog Agent: Verify that the Datadog Agent pods are running on all your EKS nodes:
kubectl get pods -n datadog
- Datadog UI: Log into your Datadog account. You should see metrics flowing from your EKS cluster nodes and pods, the newly created dashboard, and the monitors.
- PagerDuty: Check your PagerDuty account. A new service and integration should be visible, ready to receive alerts from Datadog.
- Test Alert: To test the full flow, you might intentionally create a scenario that triggers one of your Datadog monitors (e.g., scale down a deployment to a very low CPU-intensive pod on a node with other high CPU pods, or induce pod restarts). Observe if a PagerDuty incident is created.
Advanced Considerations
This guide provides a foundational setup. For production environments, consider:
- Helm Provider: Use the Terraform Helm provider to manage the Datadog Agent via its official Helm chart, which offers more configuration options and easier upgrades.
- Module-based Approach: Structure your Terraform code into reusable modules for EKS, Datadog configuration, and PagerDuty services.
- Secrets Management: Store Datadog and PagerDuty API keys securely using AWS Secrets Manager or HashiCorp Vault, accessed by Terraform.
- Comprehensive Monitoring: Expand your Datadog monitors to cover critical application metrics, network performance, security events, and cost optimization.
- Datadog Tracing & APM: Integrate Datadog APM agents within your applications for distributed tracing.
- Log Management: Configure Datadog Agent's log collection with advanced parsing rules.
Troubleshooting and FAQ
Q: Datadog Agent pods are not starting. What should I check?
A:
kubectl describe pod <pod-name> -n datadog: Look for events indicating issues like image pull errors, insufficient resources, or failed volume mounts.- RBAC Permissions: Ensure the
ClusterRoleandClusterRoleBindingprovide sufficient permissions for the Datadog Agent Service Account. - API Key: Double-check that the
DD_API_KEYenvironment variable is correctly set in the DaemonSet. - Resource Limits: Ensure your nodes have enough CPU/memory for the agent.
Q: My Datadog monitors are not triggering PagerDuty incidents.
A:
- Datadog Event Stream: Verify that your Datadog monitor is indeed triggering alerts by checking the Datadog Event Stream.
- Integration Name: Ensure the
@pagerduty-<YOUR_PAGERDUTY_SERVICE_NAME>tag in your Datadog monitor's message matches the PagerDuty service name exactly (case-sensitive) as it appears in Datadog's integrations. - PagerDuty Service Configuration: Check the PagerDuty service integration type (should be Datadog) and that the service is enabled.
- API Token: Confirm your PagerDuty API token used by Terraform is valid and has sufficient permissions to create services and integrations.
Q: How can I manage different Datadog configurations for multiple EKS clusters?
A: Use Terraform workspaces or a multi-folder structure. Each workspace/folder can have its own .tfvars file specifying unique cluster names, Datadog tags, and PagerDuty service names, allowing you to manage configurations declaratively per environment.
Conclusion
Automating observability for AWS EKS with Terraform, Datadog, and PagerDuty significantly enhances operational efficiency and incident response capabilities. By codifying your monitoring and alerting infrastructure, you gain consistency, version control, and scalability, allowing your DevOps teams to focus on innovation rather than manual configurations. This setup provides a robust foundation for proactive monitoring and efficient incident resolution, crucial for maintaining high availability in your cloud-native environments.
Comments
Post a Comment