Terraform-Driven Datadog Observability and PagerDuty Integration for AWS EKS Production Environments
Terraform-Driven Datadog Observability and PagerDuty Integration for AWS EKS Production Environments
In modern cloud-native architectures, ensuring robust observability and incident response is paramount, especially for critical production workloads running on Kubernetes. AWS EKS provides a powerful, managed Kubernetes service, but gaining deep insights into its health and performance, coupled with effective incident management, requires sophisticated tooling. This guide details how to leverage Terraform to programmatically establish comprehensive Datadog observability and seamlessly integrate it with PagerDuty for incident alerting and management within your AWS EKS production environments, ensuring consistency, auditability, and rapid response capabilities.
Architecture Pro-Tip: Always treat your observability and incident response configurations as code. By managing Datadog monitors, dashboards, and PagerDuty services via Terraform, you embed these critical operational components directly into your CI/CD pipelines. This not only enforces consistency across environments but also provides a clear audit trail for all changes, significantly reducing configuration drift and improving the reliability of your monitoring and alerting systems. Implement separate Terraform modules for Datadog and PagerDuty resources to promote reusability and maintainability.
Why Terraform for Datadog & PagerDuty Integration?
The dynamic nature of cloud environments and Kubernetes clusters necessitates an Infrastructure as Code (IaC) approach for managing all components, including observability tools. Terraform excels in orchestrating complex multi-cloud and multi-service integrations, making it an ideal choice for this setup.
The Power of Infrastructure as Code (IaC)
- Consistency: Deploy identical monitoring and alerting configurations across development, staging, and production environments, eliminating manual errors.
- Version Control: Manage configurations through Git, enabling collaborative development, change tracking, and rollbacks.
- Automation: Automate the provisioning and modification of Datadog monitors, dashboards, and PagerDuty services as part of your application deployment pipelines.
- Auditability: Every change to your observability setup is recorded in version control, providing a clear history and accountability.
Standardized Observability Deployments
With Terraform, you can define a standardized set of metrics, logs, and traces to collect from your EKS cluster, along with predefined thresholds and alert recipients. This ensures that all critical components are monitored consistently, and incidents are routed to the correct teams via PagerDuty without manual intervention.
Prerequisites
Before diving into the Terraform configuration, ensure you have the following prerequisites in place:
- AWS Account & CLI: Configured with appropriate permissions to manage EKS, IAM, and other AWS resources.
- AWS EKS Cluster: A running EKS cluster where your production workloads are deployed.
- Datadog Account: An active Datadog account with API and Application keys.
- PagerDuty Account: An active PagerDuty account with a Service and an Integration Key (e.g., Global Event Routing or Datadog Integration).
- Terraform CLI: Version 1.0 or higher installed.
- Kubectl CLI: Configured to interact with your EKS cluster (for validating Datadog Agent deployment).
Step-by-Step Implementation Guide
1. Configure AWS EKS Cluster for Datadog
Ensure your EKS cluster nodes have the necessary IAM permissions for the Datadog Agent to collect metrics from AWS services (e.g., CloudWatch, EC2 metadata). The Datadog Agent itself will run within the EKS cluster, requiring permissions to read Kubernetes API objects.
For integrating Datadog with AWS CloudWatch and other AWS services, you'll typically set up an IAM role that Datadog can assume. This is achieved using the datadog_integration_aws Terraform resource.
2. Set Up Datadog API and Application Keys
Log in to your Datadog account and navigate to Organization Settings > API Keys to retrieve or generate your API Key and Application Key. These keys are crucial for Terraform to authenticate with the Datadog API and manage resources.
- Datadog API Key: Used for data submission.
- Datadog Application Key: Used for interacting with the Datadog API to create resources (monitors, dashboards).
3. Configure PagerDuty Service and API Key
In PagerDuty, you'll need a Service where incidents related to EKS will be routed. Within this service, create a "Datadog" integration. This will provide you with an Integration Key. Additionally, you'll need a PagerDuty API Token (Admin or Manager role) for Terraform to interact with PagerDuty.
- PagerDuty Integration Key: For the Datadog integration within a specific PagerDuty Service.
- PagerDuty API Token: Used by the Terraform PagerDuty provider for authentication.
4. Terraform Provider Configuration
Your Terraform configuration will need to declare the AWS, Datadog, and PagerDuty providers. Store your API keys securely, preferably using environment variables or a secrets manager like AWS Secrets Manager or HashiCorp Vault.
variable "aws_region" {
description = "AWS region for EKS."
type = string
default = "us-east-1"
}
variable "datadog_api_key" {
description = "Datadog API Key"
type = string
sensitive = true
}
variable "datadog_app_key" {
description = "Datadog Application Key"
type = string
sensitive = true
}
variable "pagerduty_token" {
description = "PagerDuty API Token"
type = string
sensitive = true
}
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
datadog = {
source = "DataDog/datadog"
version = "~> 3.0"
}
pagerduty = {
source = "PagerDuty/pagerduty"
version = "~> 1.0"
}
# For deploying Datadog Agent via Helm if desired
helm = {
source = "hashicorp/helm"
version = "~> 2.0"
}
kubernetes = {
source = "hashicorp/kubernetes"
version = "~> 2.0"
}
}
}
provider "aws" {
region = var.aws_region
}
provider "datadog" {
api_key = var.datadog_api_key
app_key = var.datadog_app_key
}
provider "pagerduty" {
token = var.pagerduty_token
}
# Configure Kubernetes and Helm providers to deploy Datadog Agent
data "aws_eks_cluster" "eks_cluster" {
name = "your-eks-cluster-name" # Replace with your EKS cluster name
}
data "aws_eks_cluster_auth" "eks_cluster_auth" {
name = "your-eks-cluster-name" # Replace with your EKS cluster name
}
provider "kubernetes" {
host = data.aws_eks_cluster.eks_cluster.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)
token = data.aws_eks_cluster_auth.eks_cluster_auth.token
}
provider "helm" {
kubernetes {
host = data.aws_eks_cluster.eks_cluster.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)
token = data.aws_eks_cluster_auth.eks_cluster_auth.token
}
}
5. Deploying Datadog Agent to EKS & AWS Integration
The Datadog Agent is typically deployed to EKS using its official Helm chart. While the Helm deployment itself is managed via helm_release, the core of integrating Datadog with AWS services and setting up monitors is done via the Datadog Terraform provider.
# Datadog Agent Helm Chart Deployment (Optional, but recommended for EKS)
resource "helm_release" "datadog_agent" {
name = "datadog"
repository = "https://helm.datadoghq.com"
chart = "datadog"
namespace = "datadog" # Ensure this namespace exists or create it
create_namespace = true
set {
name = "datadog.apiKey"
value = var.datadog_api_key
sensitive = true
}
set {
name = "datadog.appKey"
value = var.datadog_app_key
sensitive = true
}
set {
name = "datadog.site"
value = "datadoghq.com" # Or eu.datadoghq.com, etc.
}
set {
name = "clusterAgent.enabled"
value = "true"
}
set {
name = "clusterAgent.metricsProvider.enabled"
value = "true"
}
set {
name = "kubeStateMetricsExternal.enabled"
value = "true"
}
set {
name = "datadog.leaderElection"
value = "true"
}
# Enable APM, Log Collection, Process Monitoring
set {
name = "apm.enabled"
value = "true"
}
set {
name = "logs.enabled"
value = "true"
}
set {
name = "logs.containerCollectAll"
value = "true"
}
set {
name = "processAgent.enabled"
value = "true"
}
set {
name = "targetSystem"
value = "linux"
}
# Add other specific EKS configurations as needed
}
# Integrate Datadog with AWS for CloudWatch metrics, EC2 metadata, etc.
resource "datadog_integration_aws" "aws_integration" {
account_id = data.aws_caller_identity.current.account_id
host_tags = ["environment:production", "eks:true"]
# Optional: Filter namespaces or regions if needed
# filter_tags_diff = ["key:value"]
# filter_tags_include = ["key:value"]
# filter_tags_exclude = ["key:value"]
# excluded_regions = ["us-west-1"]
}
data "aws_caller_identity" "current" {}
# Create an IAM Role for Datadog to assume (best practice)
resource "aws_iam_role" "datadog_integration_role" {
name = "DatadogAWSIntegrationRole"
assume_role_policy = jsonencode({
Version = "2012-10-17",
Statement = [
{
Effect = "Allow",
Principal = {
AWS = "464622532012" # Datadog AWS Integration Account ID
},
Action = "sts:AssumeRole",
Condition = {
StringEquals = {
"sts:ExternalId" = var.datadog_external_id # Use a securely generated external ID
}
}
}
]
})
}
resource "aws_iam_role_policy_attachment" "datadog_readonly_policy" {
role = aws_iam_role.datadog_integration_role.name
policy_arn = "arn:aws:iam::aws:policy/ReadOnlyAccess" # Or more granular policies
}
# Update the datadog_integration_aws to use the assumed role
resource "datadog_integration_aws_lambda_arn" "datadog_arn" {
account_id = data.aws_caller_identity.current.account_id
lambda_arn = aws_iam_role.datadog_integration_role.arn
}
Note: The `datadog_integration_aws` resource registers your AWS account with Datadog. For best security practices, Datadog recommends configuring an IAM role that Datadog can assume (Role Delegation) rather than providing direct API keys. The example above shows how to create such a role.
6. Integrating Datadog with PagerDuty
The integration between Datadog and PagerDuty is established using the datadog_integration_pagerduty resource. Once configured, you can then specify PagerDuty as a notification recipient in your Datadog monitors.
# Integrate Datadog with PagerDuty
resource "datadog_integration_pagerduty" "pagerduty_integration" {
services {
service_name = "My EKS Production Service" # Name of your PagerDuty service
service_key = var.pagerduty_integration_key # PagerDuty integration key for the service
}
}
# Example Datadog Monitor for EKS Node CPU Utilization that alerts PagerDuty
resource "datadog_monitor" "eks_node_cpu_high" {
name = "[EKS Production] High CPU Utilization on EKS Node - {{host.name}}"
type = "metric alert"
query = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:your-eks-cluster-name} by {host} > 80" # Replace with your EKS cluster name
message = "High CPU utilization detected on EKS node {{host.name}} ({{kubernetes.pod.name}}). @pagerduty-My-EKS-Production-Service"
tags = ["environment:production", "severity:high", "team:sre", "source:terraform"]
priority = 1
monitor_thresholds {
critical = 80
warning = 70
}
notify_no_data = false
renotify_interval = 30
timeout_h = 0
# Specify PagerDuty as a recipient
# The @pagerduty-My-EKS-Production-Service tag corresponds to the service_name in datadog_integration_pagerduty
# If using the PagerDuty integration, Datadog automatically maps this.
# If you need to specify a PagerDuty service directly, you might need to use its service ID or name in the message.
# The format @pagerduty- is the most common for direct integration.
}
# Example Datadog Monitor for EKS Deployment failures
resource "datadog_monitor" "eks_deployment_failure" {
name = "[EKS Production] Deployment failure detected"
type = "log alert"
query = "logs(\"status:error service:kubernetes @kubernetes.event.reason:FailedScheduling\").index(\"main\").last(\"5m\") > 0"
message = "A Kubernetes deployment has failed to schedule pods in EKS. Investigate pod events and resource availability. @pagerduty-My-EKS-Production-Service"
tags = ["environment:production", "severity:high", "team:devops", "source:terraform", "alert_type:deployment"]
priority = 2
monitor_thresholds {
critical = 0
}
}
# Example Datadog Dashboard for EKS
resource "datadog_dashboard" "eks_overview_dashboard" {
title = "[EKS Production] Cluster Overview"
description = "Overview of EKS Cluster Metrics"
layout_type = "ordered"
is_read_only = true
tags = ["environment:production", "eks", "terraform"]
widget {
definition {
type = "timeseries"
title = "EKS Node CPU Usage"
time_frame = "1h"
request {
q = "avg:kubernetes.cpu.usage.total{cluster_name:your-eks-cluster-name} by {host}"
display_type = "line"
}
}
}
widget {
definition {
type = "query_value"
title = "EKS Node Count"
time_frame = "1h"
request {
q = "count_not_null(avg:kubernetes.node.uptime{cluster_name:your-eks-cluster-name} by {host})"
}
}
}
widget {
definition {
type = "log_stream"
title = "Recent EKS Logs"
query = "service:kubernetes env:production"
}
}
# Add more widgets as needed for memory, network, pod status, etc.
}
7. Terraform Code Structure for Production EKS
For a production environment, it's best to organize your Terraform code into a modular structure:
main.tf: Main configuration, calls modules.variables.tf: Input variables (e.g., EKS cluster name, sensitive API keys).outputs.tf: Output values from your deployment.providers.tf: Provider declarations.modules/datadog/: Contains Datadog-specific resources (monitors, dashboards, integrations).modules/pagerduty/: Contains PagerDuty-specific resources (services, users, teams).
Ready-to-Use Terraform Configuration Example
Below is a consolidated example of the Terraform configuration. Remember to replace placeholder values like "your-eks-cluster-name" and ensure your API/APP/Integration keys are passed securely.
# main.tf
# This example assumes you have a running EKS cluster named "my-production-eks"
# and required API/App/Integration keys are provided via environment variables or tfvars.
# --- Providers Configuration ---
variable "aws_region" {
description = "AWS region for EKS."
type = string
default = "us-east-1"
}
variable "eks_cluster_name" {
description = "Name of the existing EKS cluster."
type = string
}
variable "datadog_api_key" {
description = "Datadog API Key"
type = string
sensitive = true
}
variable "datadog_app_key" {
description = "Datadog Application Key"
type = string
sensitive = true
}
variable "pagerduty_token" {
description = "PagerDuty API Token"
type = string
sensitive = true
}
variable "pagerduty_service_name" {
description = "Name of the PagerDuty service to integrate with"
type = string
default = "My EKS Production Service"
}
variable "pagerduty_integration_key" {
description = "PagerDuty integration key for the Datadog integration within the service"
type = string
sensitive = true
}
variable "datadog_external_id" {
description = "External ID for Datadog AWS IAM role assumption"
type = string
sensitive = true
}
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
datadog = {
source = "DataDog/datadog"
version = "~> 3.0"
}
pagerduty = {
source = "PagerDuty/pagerduty"
version = "~> 1.0"
}
helm = {
source = "hashicorp/helm"
version = "~> 2.0"
}
kubernetes = {
source = "hashicorp/kubernetes"
version = "~> 2.0"
}
}
}
provider "aws" {
region = var.aws_region
}
provider "datadog" {
api_key = var.datadog_api_key
app_key = var.datadog_app_key
}
provider "pagerduty" {
token = var.pagerduty_token
}
data "aws_eks_cluster" "eks_cluster" {
name = var.eks_cluster_name
}
data "aws_eks_cluster_auth" "eks_cluster_auth" {
name = var.eks_cluster_name
}
provider "kubernetes" {
host = data.aws_eks_cluster.eks_cluster.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)
token = data.aws_eks_cluster_auth.eks_cluster_auth.token
}
provider "helm" {
kubernetes {
host = data.aws_eks_cluster.eks_cluster.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)
token = data.aws_eks_cluster_auth.eks_cluster_auth.token
}
}
# --- Datadog AWS Integration ---
data "aws_caller_identity" "current" {}
resource "aws_iam_role" "datadog_integration_role" {
name = "DatadogAWSIntegrationRole-${var.eks_cluster_name}"
assume_role_policy = jsonencode({
Version = "2012-10-17",
Statement = [
{
Effect = "Allow",
Principal = {
AWS = "464622532012" # Datadog AWS Integration Account ID
},
Action = "sts:AssumeRole",
Condition = {
StringEquals = {
"sts:ExternalId" = var.datadog_external_id
}
}
}
]
})
}
resource "aws_iam_role_policy_attachment" "datadog_readonly_policy_attachment" {
role = aws_iam_role.datadog_integration_role.name
policy_arn = "arn:aws:iam::aws:policy/ReadOnlyAccess" # For basic monitoring, consider more granular policies
}
resource "datadog_integration_aws" "aws_integration" {
account_id = data.aws_caller_identity.current.account_id
host_tags = ["environment:production", "eks_cluster:${var.eks_cluster_name}"]
# Assuming the IAM role is now managed directly by Datadog's integration setup
# For role delegation, you register the role ARN with Datadog:
# This resource creates the integration within Datadog that uses the role.
# No direct `lambda_arn` attribute on `datadog_integration_aws` itself,
# but rather it uses the Role ARN generated.
}
# --- Datadog Agent Deployment (Helm) ---
resource "kubernetes_namespace" "datadog_namespace" {
metadata {
name = "datadog"
}
}
resource "helm_release" "datadog_agent" {
name = "datadog"
repository = "https://helm.datadoghq.com"
chart = "datadog"
namespace = kubernetes_namespace.datadog_namespace.metadata[0].name
set {
name = "datadog.apiKey"
value = var.datadog_api_key
sensitive = true
}
set {
name = "datadog.appKey"
value = var.datadog_app_key
sensitive = true
}
set {
name = "datadog.site"
value = "datadoghq.com"
}
set {
name = "clusterAgent.enabled"
value = "true"
}
set {
name = "kubeStateMetricsExternal.enabled"
value = "true"
}
set {
name = "logs.enabled"
value = "true"
}
set {
name = "logs.containerCollectAll"
value = "true"
}
set {
name = "processAgent.enabled"
value = "true"
}
set {
name = "systemProbe.enabled"
value = "true"
}
set {
name = "tags"
value = "{environment:production,eks_cluster:${var.eks_cluster_name}}"
}
# Add specific EKS configuration for Datadog Agent, e.g., IAM roles for service accounts if used.
}
# --- PagerDuty Integration ---
resource "datadog_integration_pagerduty" "pagerduty_integration" {
services {
service_name = var.pagerduty_service_name
service_key = var.pagerduty_integration_key
}
}
# --- Datadog Monitors ---
resource "datadog_monitor" "eks_node_cpu_high" {
name = "[EKS Production] High CPU Utilization on EKS Node - {{host.name}}"
type = "metric alert"
query = "avg(last_5m):avg:kubernetes.cpu.usage.total{eks_cluster:${var.eks_cluster_name}} by {host} > 80"
message = "High CPU utilization detected on EKS node {{host.name}}. Investigate processes and workloads. @pagerduty-${var.pagerduty_service_name}"
tags = ["environment:production", "severity:high", "team:sre", "source:terraform", "alert_type:cpu"]
priority = 1
monitor_thresholds {
critical = 80
warning = 70
}
notify_no_data = false
renotify_interval = 30
}
resource "datadog_monitor" "eks_memory_usage_high" {
name = "[EKS Production] High Memory Usage on EKS Node - {{host.name}}"
type = "metric alert"
query = "avg(last_5m):avg:kubernetes.memory.usage.total{eks_cluster:${var.eks_cluster_name}} by {host} > 85"
message = "High memory usage detected on EKS node {{host.name}}. Check for memory leaks or excessive workloads. @pagerduty-${var.pagerduty_service_name}"
tags = ["environment:production", "severity:high", "team:sre", "source:terraform", "alert_type:memory"]
priority = 1
monitor_thresholds {
critical = 85
warning = 75
}
notify_no_data = false
renotify_interval = 30
}
resource "datadog_monitor" "eks_deployment_failed" {
name = "[EKS Production] Kubernetes Deployment Failed Event"
type = "log alert"
query = "logs(\"status:error service:kubernetes kubernetes.event.reason:(FailedScheduling OR FailedAttachVolume OR FailedMount OR FailedSync)\").index(\"main\").rollup(\"count\").last(\"5m\") > 0"
message = "Critical Kubernetes event detected: Deployment failure in cluster ${var.eks_cluster_name}. Investigate pod/volume status. @pagerduty-${var.pagerduty_service_name}"
tags = ["environment:production", "severity:critical", "team:devops", "source:terraform", "alert_type:kubernetes_event"]
priority = 0
monitor_thresholds {
critical = 0
}
notify_no_data = false
}
# --- Datadog Dashboard ---
resource "datadog_dashboard" "eks_production_overview" {
title = "[EKS Production] Cluster Health Overview - ${var.eks_cluster_name}"
description = "A comprehensive dashboard for EKS production cluster health."
layout_type = "ordered"
is_read_only = true
tags = ["environment:production", "eks", "terraform", "overview"]
widget {
definition {
type = "timeseries"
title = "Cluster CPU Utilization"
time_frame = "1h"
request {
q = "avg:kubernetes.cpu.usage.total{eks_cluster:${var.eks_cluster_name}}"
display_type = "line"
}
}
}
widget {
definition {
type = "timeseries"
title = "Cluster Memory Utilization"
time_frame = "1h"
request {
q = "avg:kubernetes.memory.usage.total{eks_cluster:${var.eks_cluster_name}}"
display_type = "line"
}
}
}
widget {
definition {
type = "query_value"
title = "Node Count"
time_frame = "1h"
request {
q = "count_not_null(avg:kubernetes.node.uptime{eks_cluster:${var.eks_cluster_name}} by {host})"
}
}
}
widget {
definition {
type = "log_stream"
title = "Recent EKS Cluster Logs"
query = "service:kubernetes eks_cluster:${var.eks_cluster_name} status:error"
message_display = "expanded-with-timestamp"
}
}
}
Benefits of This Approach
Implementing Datadog and PagerDuty integration via Terraform offers significant advantages:
- Consistency and Reliability: Ensure all production EKS clusters are monitored uniformly with defined alerts.
- Auditability and Compliance: All changes to monitoring and alerting are tracked in Git, simplifying compliance audits.
- Accelerated Incident Response: Automated PagerDuty integration means critical alerts are routed immediately to the correct on-call teams.
- Scalability: Easily replicate or modify monitoring configurations for new clusters or services with minimal effort.
- Reduced Toil: Automate repetitive configuration tasks, freeing up DevOps teams to focus on more strategic initiatives.
Troubleshooting and Best Practices
Common Troubleshooting Steps
- API Keys/Tokens: Double-check that all Datadog and PagerDuty keys/tokens are correctly set and have the necessary permissions. Invalid keys are a common cause of Terraform errors.
- IAM Permissions: Ensure the IAM role created for Datadog integration has sufficient
ReadOnlyAccessor more granular permissions to fetch metrics from AWS services. - Datadog Agent Connectivity: Verify the Datadog Agent pods are running correctly in EKS (
kubectl get pods -n datadog) and can communicate with the Datadog intake endpoint. Check agent logs for errors. - Terraform State: Address any state discrepancies immediately. Consider using remote state storage (e.g., S3 backend).
- PagerDuty Service Key: Ensure the
service_keyused indatadog_integration_pagerdutymatches the integration key from the specific PagerDuty service intended for Datadog alerts.
Best Practices
- Secrets Management: Never hardcode API keys or tokens. Use environment variables (e.g.,
DD_API_KEY,PD_TOKEN), Terraform Cloud variables, or a dedicated secrets manager (AWS Secrets Manager, HashiCorp Vault) with a Terraform data source. - Modular Design: Break down your Terraform configuration into reusable modules (e.g.,
datadog-monitors,pagerduty-services) for easier management and scalability. - Granular IAM Policies: While
ReadOnlyAccessis quick, for production, create specific IAM policies for Datadog to follow the principle of least privilege. - Tagging Strategy: Implement a consistent tagging strategy for your AWS resources and Kubernetes objects. These tags are crucial for filtering and organizing data in Datadog.
- Version Control: Store all Terraform configurations in Git and utilize pull requests for review and approval of changes.
- Review Monitoring Queries: Regularly review and optimize your Datadog monitoring queries to ensure accuracy and reduce alert fatigue.
Conclusion
Achieving comprehensive observability and efficient incident management in AWS EKS production environments is a critical undertaking. By adopting a Terraform-driven approach for Datadog monitoring and PagerDuty integration, organizations can ensure their critical systems are consistently monitored, alerts are intelligently routed, and incidents are resolved swiftly. This IaC methodology not only streamlines deployment but also enhances the reliability, auditability, and scalability of your entire operational stack, laying a solid foundation for robust cloud-native operations.
Comments
Post a Comment