Terraform for AWS EKS Blue/Green Deployments with Datadog and PagerDuty Observability

In the dynamic world of cloud-native applications, ensuring high availability, minimizing downtime, and achieving seamless deployments are paramount. This comprehensive guide delves into implementing a robust Blue/Green deployment strategy for AWS EKS using Terraform, integrating powerful observability tools like Datadog for monitoring and PagerDuty for incident management. By following this guide, DevOps engineers and SREs will learn to automate their deployment pipelines, enhance system reliability, and improve incident response capabilities.

Architecture Pro-Tip: Leveraging Route 53 and ALB for Seamless Traffic Shifting

For truly seamless Blue/Green deployments on AWS EKS, consider using AWS Application Load Balancers (ALB) to manage traffic to your Kubernetes services. Combine this with AWS Route 53 weighted routing or listener rule modifications to precisely control traffic flow between your "Blue" and "Green" environments. This approach minimizes DNS propagation delays and allows for immediate rollback, offering superior control and faster recovery times.

Understanding Blue/Green Deployments

Blue/Green deployment is a release strategy that reduces downtime and risk by running two identical production environments, "Blue" and "Green". At any time, only one environment is live, serving all production traffic. When a new version of the application is released, it is deployed to the inactive environment (e.g., Green). Once tested and validated, traffic is switched from Blue to Green. If any issues arise, traffic can be instantly rolled back to the stable Blue environment.

Why Blue/Green for AWS EKS?

Zero Downtime: Users experience no interruption during deployments.
Instant Rollback: Quick recovery from problematic deployments.
Reduced Risk: New versions are thoroughly tested in a production-like environment before going live.
Enhanced Reliability: Contributes to a more stable and resilient application architecture.

Prerequisites and Core Technologies

Before diving into the implementation, ensure you have the following:

AWS Account: With necessary IAM permissions to create EKS clusters, VPCs, ALBs, etc.
Terraform CLI: Installed and configured.
Kubectl: Installed for interacting with Kubernetes clusters.
Datadog Account: An active account for monitoring.
PagerDuty Account: An active account for incident management.
Basic Kubernetes Knowledge: Understanding of Deployments, Services, Ingress.

Core Technologies Overview:

AWS EKS: Amazon's managed Kubernetes service, simplifying Kubernetes cluster operations.
Terraform: Infrastructure as Code (IaC) tool for provisioning and managing cloud resources.
Datadog: A comprehensive monitoring and analytics platform for cloud applications and infrastructure.
PagerDuty: An incident management platform that aggregates alerts, orchestrates on-call rotations, and facilitates rapid incident resolution.

Architecting Blue/Green on AWS EKS with Terraform

The core idea is to have two distinct sets of Kubernetes deployments and services, one for "Blue" and one for "Green". An AWS Application Load Balancer (ALB) will sit in front, dynamically routing traffic to either the Blue or Green services based on listener rules.

Architectural Components:

VPC and Subnets: Network foundation for EKS.
EKS Cluster: The Kubernetes control plane.
EKS Node Groups: Worker nodes for your pods.
Kubernetes Deployments (Blue & Green): Two distinct deployments for your application pods.
Kubernetes Services (Blue & Green): Expose the deployments internally.
AWS Application Load Balancer (ALB): External entry point.
ALB Target Groups (Blue & Green): Associated with EKS services.
ALB Listener Rules: Used to switch traffic between Blue and Green target groups.
Datadog Agent (DaemonSet): For collecting metrics, logs, and traces from EKS.
Datadog Monitors: Alerting rules based on collected data.
PagerDuty Service & Integration: For incident routing and on-call management.

Terraform Configuration for Blue/Green EKS

Here, we'll outline key Terraform configurations. Note that a complete, runnable setup would involve many more files (VPC, IAM roles, etc.). This focuses on the Blue/Green and observability components.

1. EKS Cluster and Kubernetes Provider Setup

Ensure your EKS cluster is provisioned. The Kubernetes provider needs to be configured to interact with it.

resource "aws_eks_cluster" "main" {
  name     = "blue-green-eks"
  role_arn = aws_iam_role.eks_cluster_role.arn
  vpc_config {
    subnet_ids = [aws_subnet.private_a.id, aws_subnet.private_b.id]
  }
  depends_on = [
    aws_iam_role_policy_attachment.eks_cluster_policy,
  ]
}

data "aws_eks_cluster_auth" "main" {
  name = aws_eks_cluster.main.name
}

provider "kubernetes" {
  host                   = aws_eks_cluster.main.endpoint
  cluster_ca_certificate = base64decode(aws_eks_cluster.main.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.main.token
}

provider "helm" {
  kubernetes {
    host                   = aws_eks_cluster.main.endpoint
    cluster_ca_certificate = base64decode(aws_eks_cluster.main.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.main.token
  }
}

2. Application Deployments and Services (Blue/Green)

Define separate Kubernetes deployments and services for your Blue and Green application versions. The key is distinguishing them, often with labels and suffixes.

# Blue Application Deployment
resource "kubernetes_deployment" "app_blue" {
  metadata {
    name      = "my-app-blue"
    namespace = "default"
    labels = {
      app  = "my-app"
      env  = "blue"
      version = "1.0.0"
    }
  }
  spec {
    replicas = 2
    selector {
      match_labels = {
        app = "my-app"
        env = "blue"
      }
    }
    template {
      metadata {
        labels = {
          app = "my-app"
          env = "blue"
        }
      }
      spec {
        container {
          name  = "my-app"
          image = "myrepo/my-app:1.0.0" # Blue version image
          port {
            container_port = 8080
          }
        }
      }
    }
  }
}

# Blue Application Service
resource "kubernetes_service" "app_blue" {
  metadata {
    name      = "my-app-blue-service"
    namespace = "default"
    annotations = {
      "service.beta.kubernetes.io/aws-load-balancer-type" = "external"
      "service.beta.kubernetes.io/aws-load-balancer-backend-protocol" = "http"
    }
  }
  spec {
    selector = {
      app = kubernetes_deployment.app_blue.metadata.0.labels.app
      env = kubernetes_deployment.app_blue.metadata.0.labels.env
    }
    port {
      port        = 80
      target_port = 8080
      protocol    = "TCP"
    }
    type = "NodePort" # Or ClusterIP if using ALB Ingress Controller
  }
}

# Green Application Deployment (for new version)
resource "kubernetes_deployment" "app_green" {
  metadata {
    name      = "my-app-green"
    namespace = "default"
    labels = {
      app  = "my-app"
      env  = "green"
      version = "1.1.0" # New version
    }
  }
  spec {
    replicas = 2
    selector {
      match_labels = {
        app = "my-app"
        env = "green"
      }
    }
    template {
      metadata {
        labels = {
          app = "my-app"
          env = "green"
        }
      }
      spec {
        container {
          name  = "my-app"
          image = "myrepo/my-app:1.1.0" # Green version image
          port {
            container_port = 8080
          }
        }
      }
    }
  }
}

# Green Application Service
resource "kubernetes_service" "app_green" {
  metadata {
    name      = "my-app-green-service"
    namespace = "default"
    annotations = {
      "service.beta.kubernetes.io/aws-load-balancer-type" = "external"
      "service.beta.kubernetes.io/aws-load-balancer-backend-protocol" = "http"
    }
  }
  spec {
    selector = {
      app = kubernetes_deployment.app_green.metadata.0.labels.app
      env = kubernetes_deployment.app_green.metadata.0.labels.env
    }
    port {
      port        = 80
      target_port = 8080
      protocol    = "TCP"
    }
    type = "NodePort" # Or ClusterIP if using ALB Ingress Controller
  }
}

3. AWS Application Load Balancer (ALB) Configuration

The ALB will manage traffic. We'll create two target groups (one for Blue, one for Green) and listener rules to direct traffic.

resource "aws_lb" "main" {
  name               = "blue-green-lb"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [aws_security_group.lb_sg.id]
  subnets            = [aws_subnet.public_a.id, aws_subnet.public_b.id]

  enable_deletion_protection = false
  tags = {
    Environment = "production"
  }
}

resource "aws_lb_target_group" "blue_tg" {
  name        = "my-app-blue-tg"
  port        = 80
  protocol    = "HTTP"
  vpc_id      = aws_vpc.main.id
  target_type = "instance" # Or 'ip' if using ALB Ingress Controller and IP mode
  health_check {
    path = "/health"
    port = "traffic-port"
  }
  tags = {
    Environment = "blue"
  }
}

resource "aws_lb_target_group" "green_tg" {
  name        = "my-app-green-tg"
  port        = 80
  protocol    = "HTTP"
  vpc_id      = aws_vpc.main.id
  target_type = "instance" # Or 'ip' if using ALB Ingress Controller and IP mode
  health_check {
    path = "/health"
    port = "traffic-port"
  }
  tags = {
    Environment = "green"
  }
}

resource "aws_lb_listener" "http_listener" {
  load_balancer_arn = aws_lb.main.arn
  port              = 80
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.blue_tg.arn # Initially direct traffic to Blue
  }
}

# To switch to Green:
# Update aws_lb_listener.http_listener.default_action.target_group_arn to aws_lb_target_group.green_tg.arn
# And vice-versa for rollback. This can be controlled via a CI/CD pipeline.

4. Datadog Observability Integration

Deploy the Datadog Agent to your EKS cluster using Helm. Then, define a Datadog monitor with Terraform.

# Datadog Agent Helm Chart Deployment
resource "helm_release" "datadog_agent" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = "default"
  version    = "2.33.0" # Use a stable, recent version

  set {
    name  = "datadog.apiKey"
    value = var.datadog_api_key
  }

  set {
    name  = "datadog.appKey"
    value = var.datadog_app_key
  }

  set {
    name  = "clusterAgent.enabled"
    value = "true"
  }
  set {
    name  = "kubeStateMetricsExternal.enabled"
    value = "true"
  }
  set {
    name  = "networkMonitoring.enabled"
    value = "true"
  }
  # Enable APM, Log Collection, etc. as needed
  set {
    name  = "apm.enabled"
    value = "true"
  }
  set {
    name  = "logs.enabled"
    value = "true"
  }
  set {
    name  = "logs.containerCollectAll"
    value = "true"
  }
}

# Datadog Monitor for Application Latency
resource "datadog_monitor" "app_latency_monitor" {
  name               = "[EKS Blue/Green] Application Latency Alert - {{env}}"
  type               = "query alert"
  query              = "avg(last_5m):avg:aws.applicationelb.target_response_time{env:blue} > 0.5 || avg(last_5m):avg:aws.applicationelb.target_response_time{env:green} > 0.5"
  message            = "High application latency detected on EKS Blue/Green deployment for {{env}}.\n{{< DD_EXTERNAL_LINK url=\"${var.datadog_dashboard_url}\" text=\"View Dashboard\" >}}"
  tags               = ["environment:production", "service:my-app", "team:devops"]
  priority           = 2
  notify_no_data     = false
  renotify_interval  = 0
  escalation_message = "Application latency remains high. Paging on-call team!"

  # PagerDuty integration through Datadog notification
  # Ensure your PagerDuty integration is set up in Datadog UI
  # You might use @pagerduty-service-name or a webhook as notification channel
  notify_audit     = false
  timeout_h        = 0
  new_group_delay  = 60
  no_data_timeframe = 10
  require_full_window = false
  include_tags = true

  # Set specific `notification_webhook_name` if using webhook
  # Otherwise, Datadog's built-in PagerDuty integration will use `message` with @pagerduty tag
}

5. PagerDuty Integration

Configure a PagerDuty service and its integration with Datadog using Terraform.

# PagerDuty Provider Setup (requires PAGERDUTY_TOKEN env var)
provider "pagerduty" {
  token = var.pagerduty_token
}

# PagerDuty Escalation Policy
resource "pagerduty_escalation_policy" "devops_escalation" {
  name      = "DevOps Primary Escalation"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 15
    target {
      type = "user"
      id   = var.pagerduty_devops_user_id # ID of your DevOps user in PagerDuty
    }
  }

  rule {
    escalation_delay_in_minutes = 30
    target {
      type = "schedule"
      id   = var.pagerduty_devops_schedule_id # ID of your DevOps on-call schedule
    }
  }
}

# PagerDuty Service for My-App
resource "pagerduty_service" "my_app_service" {
  name                      = "My-App EKS Service"
  auto_resolve_timeout      = "14400" # 4 hours
  acknowledgement_timeout   = "300"   # 5 minutes
  escalation_policy         = pagerduty_escalation_policy.devops_escalation.id
  description               = "Handles incidents for the main application running on EKS."
  alert_creation_parameters {
    type = "create_incidents_and_alert_on_all_alerts"
  }
}

# PagerDuty Service Integration for Datadog
resource "pagerduty_service_integration" "datadog_integration" {
  name        = "Datadog"
  service_id  = pagerduty_service.my_app_service.id
  type        = "datadog_inbound_integration"
}

# In Datadog Monitor message, use @pagerduty-service-name (e.g., @pagerduty-my_app_service)
# Or, if using the integration_key directly:
# "message": "@pagerduty-YOUR_INTEGRATION_KEY Application latency high for {{env}}!"
# The pagerduty_service_integration resource will provide an integration key.
# For the datadog_monitor, you'd typically reference the service name in the message field
# and ensure the Datadog-PagerDuty integration is configured in Datadog UI to route based on service names.
}

Blue/Green Deployment Workflow

With the infrastructure in place, here's a typical workflow for deploying a new application version:

Initial State: Traffic is routed to the Blue environment (e.g., `aws_lb_listener.http_listener` points to `aws_lb_target_group.blue_tg`).
Deploy Green: Update `kubernetes_deployment.app_green` with the new application image and apply your Terraform. This provisions the new version in the Green environment without affecting Blue.
Smoke Testing: Perform thorough testing on the Green environment. You can use a separate ALB listener rule with a specific host header or path to access Green directly for internal testing.
Traffic Shift: Update the `aws_lb_listener.http_listener` default action to point to `aws_lb_target_group.green_tg`. Apply this Terraform change. This instantly switches all traffic to Green.
Monitor and Validate: Closely watch Datadog dashboards and logs for any anomalies. If Datadog monitors trigger, PagerDuty will alert the on-call team.
Rollback (if needed): If issues are detected, revert the `aws_lb_listener.http_listener` default action back to `aws_lb_target_group.blue_tg` and apply Terraform. This switches traffic back to the stable Blue environment.
Teardown (optional): Once Green is stable and proven, the old Blue deployment and service can be scaled down or removed if no longer needed. Alternatively, Blue becomes the new 'inactive' environment for the next release.

Datadog and PagerDuty for Enhanced Observability

The power of Blue/Green deployments is amplified with robust observability. Datadog provides end-to-end visibility across your EKS cluster, applications, and AWS infrastructure, while PagerDuty ensures critical issues are never missed.

Key Datadog Features for EKS Blue/Green:

Unified Metrics: Collects metrics from EKS, EC2 instances, ALB, and custom application metrics.
Log Management: Aggregates logs from all containers, helping identify issues specific to Blue or Green.
APM & Tracing: Provides deep insights into application performance, allowing you to compare Blue vs. Green performance.
Dashboards: Create custom dashboards to visualize Blue and Green metrics side-by-side, making it easy to spot divergences after a traffic switch.
Monitors & Alerts: Set up thresholds for key metrics (latency, error rates, resource utilization) that trigger alerts if breached.

Leveraging PagerDuty for Incident Response:

Automated Alerting: Datadog monitors automatically trigger incidents in PagerDuty.
On-Call Management: PagerDuty ensures the right person is notified at the right time, based on schedules and escalation policies.
Incident Orchestration: Facilitates timely communication and collaboration during critical incidents, speeding up mean time to resolution (MTTR).
Post-Mortem Analysis: Detailed incident timelines aid in root cause analysis and continuous improvement.

Troubleshooting and Best Practices

Common Troubleshooting Scenarios:

Traffic Not Shifting: Double-check ALB listener rules and target group registrations. Ensure EKS services are correctly exposing pods to the ALB. Verify security groups allow traffic.
Application Errors in Green: Check application logs in Datadog. Verify container image, environment variables, and Kubernetes manifest differences between Blue and Green.
Datadog Agent Not Reporting: Verify the Datadog Agent DaemonSet is running in EKS. Check agent logs for API key issues or connectivity problems. Ensure necessary IAM permissions for Datadog integration.
PagerDuty Alerts Not Firing: Confirm Datadog monitor thresholds are met. Verify the Datadog-PagerDuty integration is configured correctly in Datadog settings and the monitor message includes the correct PagerDuty service tag.

Best Practices for Production Environments:

Automate Everything: Integrate Terraform into your CI/CD pipeline to automate Blue/Green traffic shifts.
Idempotency: Ensure your Terraform configurations are idempotent, meaning applying them multiple times yields the same result without errors.
State Management: Use remote Terraform state (e.g., S3 backend with DynamoDB locking) to prevent state corruption.
Health Checks: Implement robust Kubernetes liveness and readiness probes for your application pods and configure thorough health checks on your ALB target groups.
Canary Releases: For more cautious rollouts, consider a staged traffic shift (e.g., 10% to Green, then 50%, then 100%) by manipulating ALB listener rules or using tools like Flagger with Istio.
Cost Management: Be mindful of running two full environments. Scale down the inactive environment's pods if resources are constrained, or tear it down after a successful deployment if an immediate rollback isn't a primary concern.
Testing: Implement automated tests (unit, integration, end-to-end) that run against the Green environment before the traffic switch.

Conclusion

Implementing Blue/Green deployments on AWS EKS with Terraform offers a powerful approach to minimize deployment risks and achieve near-zero downtime. When coupled with the comprehensive observability provided by Datadog and the efficient incident management capabilities of PagerDuty, your DevOps teams gain unparalleled control and insight into your production systems. By adopting these strategies, organizations can significantly improve their software delivery lifecycle, enhance system reliability, and foster a more proactive incident response posture.

This guide serves as a foundational blueprint. Real-world implementations may require further customization, advanced routing techniques, and more sophisticated monitoring tailored to specific application needs. However, the principles outlined here provide a solid starting point for building highly resilient cloud-native applications.

CloudStack YAML