Terraform AWS EKS Provisioning with Datadog Observability and PagerDuty Integration

Terraform AWS EKS Provisioning, Datadog EKS Observability, PagerDuty DevOps Integration, Cloud Infrastructure Automation, Kubernetes Infrastructure as Code [CONTENT]

In the rapidly evolving landscape of cloud-native applications, managing highly available and scalable Kubernetes clusters is paramount for modern DevOps teams. Amazon Elastic Kubernetes Service (EKS) stands out as a robust platform for deploying, managing, and scaling containerized applications on AWS. However, the sheer complexity of provisioning EKS, along with its associated networking, security, and observability components, necessitates a powerful Infrastructure as Code (IaC) tool. Terraform, from HashiCorp, provides the declarative language needed to define and provision a complete EKS ecosystem reliably and repeatedly. This guide delves into a comprehensive approach to provisioning an AWS EKS cluster using Terraform, extending beyond basic setup to integrate crucial observability with Datadog and proactive incident management with PagerDuty. By following this methodology, organizations can achieve a fully automated, observable, and resilient Kubernetes infrastructure, minimizing manual intervention and maximizing operational efficiency, ensuring that issues are detected and resolved swiftly before they impact end-users significantly.

Architecture Pro-Tip: Always segregate your Terraform configurations into logical modules (e.g., VPC, EKS, Datadog, PagerDuty). This enhances reusability, simplifies maintenance, and promotes team collaboration. Utilize Terraform workspaces for managing different environments (dev, staging, prod) efficiently, ensuring state isolation and reducing the risk of unintended changes across environments. Implement a robust backend for Terraform state, such as AWS S3 with DynamoDB locking, to prevent concurrency issues and ensure state durability.

Setting Up Your AWS and Terraform Environment for EKS

Before diving into the EKS provisioning, it's crucial to ensure your local environment is correctly configured to interact with AWS and execute Terraform commands. This foundational step involves installing the necessary tools and configuring authentication credentials. You'll need the AWS Command Line Interface (CLI) installed and configured with appropriate programmatic access keys that have permissions to create and manage EKS clusters, VPCs, IAM roles, and other dependent AWS resources. Additionally, Terraform must be installed on your system. Once these prerequisites are met, the first Terraform configuration step is to declare the AWS provider, specifying the region where your EKS cluster will be deployed. It's recommended to use a specific version constraint for the provider to ensure consistent behavior across deployments and prevent unexpected changes from newer provider versions. This ensures that your infrastructure deployments are predictable and repeatable, reducing the chances of environment drift. Establishing a secure and reliable connection to your AWS account is the cornerstone of any cloud infrastructure automation project, particularly when dealing with complex services like EKS.

provider "aws" {
  region = "us-east-1"
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
    pagerduty = {
      source  = "PagerDuty/pagerduty"
      version = "~> 2.0"
    }
  }
  backend "s3" {
    bucket         = "your-terraform-state-bucket"
    key            = "eks/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "your-terraform-lock-table"
  }
}
    

Core EKS Cluster and Networking Provisioning with Terraform

The foundation of any EKS deployment is a well-designed network infrastructure. This typically involves creating a Virtual Private Cloud (VPC) with public and private subnets across multiple Availability Zones for high availability, along with NAT Gateways for private subnet outbound internet access. While you could provision these resources manually, Terraform's aws_vpc and aws_subnet resources, or preferably the dedicated terraform-aws-modules/vpc/aws module, streamline this process significantly. Once the networking is in place, the EKS cluster itself can be provisioned. The terraform-aws-modules/eks/aws module is highly recommended as it abstracts away much of the complexity, handling the creation of the EKS control plane, IAM roles, security groups, and optionally managed node groups or Fargate profiles. This modular approach ensures that best practices are followed regarding EKS cluster setup, including appropriate IAM roles for the EKS service and node groups, along with necessary security group rules to allow communication between the control plane and worker nodes. Defining these elements declaratively ensures consistency and repeatability, which are critical for robust cloud deployments.

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "eks-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true

  tags = {
    "kubernetes.io/cluster/eks-cluster" = "owned"
  }
}

module "eks_cluster" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0"

  cluster_name    = "my-observability-eks"
  cluster_version = "1.27"
  vpc_id          = module.vpc.vpc_id
  subnet_ids      = module.vpc.private_subnets

  enable_irsa = true

  eks_managed_node_groups = {
    default = {
      instance_types = ["t3.medium"]
      desired_size   = 2
      max_size       = 3
      min_size       = 1
    }
  }

  tags = {
    Environment = "Dev"
    Project     = "EKS Observability"
  }
}
    

Integrating Datadog for Comprehensive EKS Observability

Once the EKS cluster is operational, the next critical step is to establish robust observability. Datadog provides a powerful, unified platform for monitoring, logging, and tracing across your entire infrastructure and applications, making it an excellent choice for EKS. Integrating Datadog involves several key steps using Terraform. First, you'll need to configure the Datadog provider with your API and Application keys, which should be stored securely, ideally in a secret management service like AWS Secrets Manager or HashiCorp Vault. After the provider is configured, the Datadog Agent needs to be deployed onto your EKS cluster. This is typically done via a Kubernetes DaemonSet, ensuring an agent runs on every node to collect metrics, logs, and traces. The official Datadog Helm chart is the recommended way to deploy the agent, and Terraform can manage Helm chart deployments using the helm_release resource. This setup will automatically discover and monitor your EKS nodes, pods, services, and various AWS services linked to your EKS cluster, giving you deep insights into performance and health.

resource "kubernetes_secret" "datadog_api_key" {
  metadata {
    name      = "datadog-secret"
    namespace = "datadog"
  }

  data = {
    "api-key" = var.datadog_api_key
    "app-key" = var.datadog_app_key
  }
  type = "Opaque"
}

resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "datadog"
  version    = "3.3.16" # Use a specific version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }

  set {
    name  = "datadog.site"
    value = "datadoghq.com"
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }

  set {
    name  = "clusterAgent.metricsProvider.enabled"
    value = "true"
  }

  set {
    name  = "kubeStateMetricsCore.enabled"
    value = "true"
  }

  set {
    name  = "aws.enabled"
    value = "true"
  }

  set {
    name  = "processAgent.enabled"
    value = "true"
  }

  set {
    name  = "containerRuntime.cri.enabled"
    value = "true"
  }

  set {
    name  = "rbac.create"
    value = "true"
  }

  depends_on = [
    kubernetes_secret.datadog_api_key,
    module.eks_cluster # Ensure EKS is ready before deploying agent
  ]
}
    

Configuring PagerDuty for Incident Management Automation

While Datadog provides excellent observability, detecting issues is only half the battle; responding to them effectively is equally important. PagerDuty excels in incident management, ensuring that critical alerts reach the right on-call personnel promptly. Integrating PagerDuty with your Terraform-managed EKS environment allows you to automate the creation of services, escalation policies, and integration keys, which can then be used by Datadog to trigger incidents. The PagerDuty Terraform provider enables you to define these resources declaratively. You'll typically start by creating a PagerDuty service, which represents a component or application you want to monitor, and associate it with an escalation policy that defines who gets notified and in what order. For the integration with Datadog, you'll create a PagerDuty service integration of type datadog_events_api_v2. This setup ensures that any high-severity alerts generated by Datadog monitors automatically create an incident in PagerDuty, streamlining your incident response workflow and minimizing potential downtime.

resource "pagerduty_user" "devops_engineer" {
  name  = "DevOps Engineer"
  email = "devops@example.com"
}

resource "pagerduty_escalation_policy" "eks_primary_ep" {
  name      = "EKS Primary Escalation Policy"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "user_reference"
      id   = pagerduty_user.devops_engineer.id
    }
  }
}

resource "pagerduty_service" "eks_observability_service" {
  name                      = "EKS Observability Service"
  auto_resolve_timeout_days = 1
  acknowledgement_timeout_minutes = 10
  escalation_policy         = pagerduty_escalation_policy.eks_primary_ep.id
  incident_urgency_rule {
    type = "constant"
    urgency = "high"
  }
}

resource "pagerduty_service_integration" "datadog_integration" {
  name    = "Datadog Integration"
  service = pagerduty_service.eks_observability_service.id
  type    = "datadog_events_api_v2"
}
    

Defining Datadog Monitors and PagerDuty Alerts

With Datadog collecting metrics and PagerDuty ready to manage incidents, the crucial final step is to define the monitors in Datadog that will trigger alerts and, consequently, PagerDuty incidents. Terraform's Datadog provider allows you to declare these monitors as code, ensuring consistency, version control, and easy replication across environments. A Datadog monitor defines a specific condition to watch (e.g., CPU utilization exceeding a threshold, a pod crash loop) and what action to take when that condition is met. When creating monitors for EKS, you'll typically focus on key metrics such as node CPU/memory utilization, pod restarts, deployment availability, and network errors. The integration with PagerDuty is achieved by including the PagerDuty service integration key in the monitor's message or a specific notification field. This tells Datadog to route the alert to the configured PagerDuty service, initiating the incident response process. Automating monitor creation through Terraform is a powerful practice that minimizes human error and ensures that your critical infrastructure is always under vigilant observation, triggering immediate alerts when anomalies occur.

resource "datadog_monitor" "eks_high_cpu_node" {
  name               = "[EKS] High Node CPU Utilization on {{host.name}}"
  type               = "metric alert"
  query              = "avg(last_5m):avg:kubernetes.cpu.usage.total{cluster_name:my-observability-eks} by {host} > 80"
  message            = "EKS Node {{host.name}} CPU utilization is over 80%! @pagerduty-${pagerduty_service_integration.datadog_integration.integration_key}"
  tags               = ["environment:dev", "eks", "cpu"]
  alert_well_know_text = "Node CPU utilization is consistently high, investigate workload scaling or node capacity."
  # Set thresholds
  thresholds = {
    warning = 70.0
    critical = 80.0
  }
  # Notification options
  notify_no_data = false
  renotify_interval = 60
  no_data_timeframe = 10
  include_tags = true
  force_delete   = false
}

resource "datadog_monitor" "eks_pod_crashloop" {
  name               = "[EKS] Pod CrashLoopBackOff Detected in {{kube_namespace}}/{{kube_deployment}}"
  type               = "log alert"
  query              = "logs(\"status:error service:kube_state_metrics message:Pod \\\"CrashLoopBackOff\\\" detected @cluster_name:my-observability-eks\").by(\"kube_namespace\",\"kube_deployment\").last(\"5m\") > 0"
  message            = "A Pod in {{kube_namespace}}/{{kube_deployment}} is in CrashLoopBackOff state. Investigate application logs. @pagerduty-${pagerduty_service_integration.datadog_integration.integration_key}"
  tags               = ["environment:dev", "eks", "pod_health"]
  alert_well_know_text = "A pod is repeatedly crashing. Check application logs and resource limits."
  thresholds = {
    critical = 0.0 # Any occurrence is critical
  }
  notify_no_data = false
  renotify_interval = 30
  no_data_timeframe = 10
  include_tags = true
  force_delete   = false
}
    

Deployment and Validation of Your EKS Observability Stack

With all Terraform configurations meticulously defined, the next crucial phase involves deploying your EKS cluster with Datadog observability and PagerDuty integration, followed by thorough validation. The deployment process begins with initializing your Terraform working directory using terraform init, which downloads the necessary providers and modules. Subsequently, terraform plan allows you to preview the changes Terraform will make to your infrastructure, providing a critical opportunity to review and confirm that the intended resources will be created, updated, or destroyed. Once satisfied with the plan, execute terraform apply --auto-approve to provision the infrastructure. This command will create your VPC, EKS cluster, deploy the Datadog Agent via Helm, set up PagerDuty services, and configure Datadog monitors. After the apply completes, it's essential to validate the deployment. Verify the EKS cluster status via kubectl get nodes and kubectl get pods -n datadog. Log into your Datadog account to confirm that metrics and logs are flowing in from your EKS cluster and that the created monitors are visible. Finally, test the PagerDuty integration by intentionally triggering an alert condition (e.g., simulating high CPU) or manually triggering a test incident from Datadog to ensure PagerDuty receives it and notifies the correct on-call team.

# Initialize Terraform
terraform init

# Review the execution plan
terraform plan

# Apply the configuration to provision resources
terraform apply --auto-approve

# Verify EKS cluster nodes
kubectl get nodes

# Verify Datadog Agent pods
kubectl get pods -n datadog

# Once deployed, login to Datadog and PagerDuty to verify integration and monitor status
    

Best Practices and Future Enhancements for EKS Observability

While this guide provides a comprehensive foundation, the journey to a fully optimized EKS observability and incident management system is continuous. Adhering to best practices and planning for future enhancements will ensure long-term stability and efficiency. For Terraform, always modularize your code, use remote state with locking (as demonstrated), and implement CI/CD pipelines to automate deployments and enforce policy checks. Securely manage sensitive information like Datadog API keys and PagerDuty tokens using AWS Secrets Manager or other vault solutions, referencing them in Terraform using data sources instead of hardcoding. For Datadog, explore advanced features such as service maps, APM for distributed tracing, log management with robust parsing rules, and custom dashboards tailored to your application's specific KPIs. Leverage Datadog's Watchdog for anomaly detection and forecasting to proactively identify issues before they become critical. In PagerDuty, refine escalation policies, integrate with collaboration tools like Slack or Microsoft Teams for immediate communication, and regularly review incident post-mortems to learn and improve. Consider incorporating Chaos Engineering principles to test the resilience of your EKS cluster and the effectiveness of your monitoring and alerting systems under stress. Regularly update Terraform providers and modules to benefit from new features and security patches, ensuring your infrastructure remains cutting-edge and secure.

CloudStack YAML