Capstone Part 3 — Deployment Strategy

This is the third of six parts building one continuous project: shipping checkout-svc safely. Part 1 gave it a pipeline that gates every push; Part 2 gave it three real environments, each behind its own Application Load Balancer, provisioned from one reusable Terraform module. Every deploy since Part 2 has meant one thing: point prod's single target group at a new image tag and let the Auto Scaling Group cycle through it, all at once, whether or not the new build was actually safe. Today that changes. This page adds a second target group to prod's existing ALB, wires a weighted rollout into the exact pipeline Part 1 built, and gates every traffic shift behind a CloudWatch alarm that watches only the build under test — not the blended, fleet-wide number. By the end, a bad build never receives more than 10% of prod's traffic before it is rolled back automatically, with nobody watching a dashboard in real time to catch it.

What this part assumes, and what it produces

☺ Like you're 10: The store, the registers, and the two-lane trick — nothing new to install, just the AWS account and the repo Parts 1 and 2 already built.

This part assumes Part 2's three environments already exist and are healthy — specifically prod, running checkout-svc:7f3a9c2 behind module.checkout's ALB and Auto Scaling Group. You need the same AWS CLI and Terraform 1.9+ from Part 2, plus jq for the JSON the script below leans on. Nothing here touches dev or staging — a deployment strategy is a prod-specific concern, because dev and staging's whole job is catching a bad build before it ever reaches an environment where blast radius matters; see deployment strategies for why the strategy you pick is really a statement about how much exposure you can tolerate at once, and how fast you can undo it. What you're about to build is deliberately a hybrid of the two named strategies on that page, not a pure pick of one: a canary-style weighted, gradual traffic shift is the day-to-day promotion mechanism, riding on top of a blue-green-style parked idle fallback — the old fleet stays fully provisioned and one API call from serving 100% again, exactly like blue-green promises, for as long as the new fleet hasn't been proven under a full bake.

The world model this part adds

Building on Part 2's infrastructure, here's what gets born on this page, all of it scoped to prod alone:

Keep that table in mind for the rest of this capstone: Part 4 replaces today's one narrow alarm with real dashboards on both target groups, and Part 6 comes back to tighten the canary launch template the same way it tightens blue's.

Standing up the canary fleet: a second target group behind the same ALB

☺ Like you're 10: A second lane, built next to the first one, borrowing the same building and the same guard rules — not a whole new store.

Part 2's module never had to hand out its listener or its own security group — nothing needed to attach to them from outside the module. Today's second fleet does, so the module gets three small, purely additive outputs before anything else changes. Add these to the bottom of infra/modules/checkout-service/outputs.tf; nothing already in the module is touched:

## infra/modules/checkout-service/outputs.tf — three lines added in Part 3

output "listener_arn"          { value = aws_lb_listener.checkout.arn }
output "app_security_group_id" { value = aws_security_group.app.id }
output "alb_arn_suffix"        { value = aws_lb.checkout.arn_suffix }

With those in place, add a new file — infra/envs/prod/canary.tf — that stands up the canary fleet beside Part 2's existing prod resources, reusing its ALB, its app security group, and its AMI lookup rather than duplicating a whole new environment:

## infra/envs/prod/canary.tf — new in Part 3; nothing above this file changes

variable "canary_image_tag" { type = string }   # e.g. "a18e4f0" — the build under test

data "aws_vpc" "default" { default = true }
data "aws_subnets" "default" {
  filter { name = "vpc-id", values = [data.aws_vpc.default.id] }
}
data "aws_ami" "al2023" {
  most_recent = true
  owners      = ["amazon"]
  filter { name = "name", values = ["al2023-ami-*-x86_64"] }
}

resource "aws_lb_target_group" "checkout_canary" {
  name     = "checkout-prod-canary"
  port     = 8080
  protocol = "HTTP"
  vpc_id   = data.aws_vpc.default.id
  health_check { path = "/healthz", healthy_threshold = 2, unhealthy_threshold = 3, interval = 15 }
}

resource "aws_launch_template" "checkout_canary" {
  name_prefix             = "checkout-prod-canary-"
  image_id                = data.aws_ami.al2023.id
  instance_type           = "m5.large"                              # match blue's size, not dev's t3.micro
  vpc_security_group_ids  = [module.checkout.app_security_group_id] # reuse blue's app SG — same ALB, same rules
  user_data = base64encode(<<-EOT
    #!/bin/bash
    dnf install -y docker && systemctl enable --now docker
    docker run -d --restart unless-stopped -p 8080:8080 \
      registry.internal/checkout-svc:${var.canary_image_tag}
  EOT
  )
  tag_specifications {
    resource_type = "instance"
    tags          = { Name = "checkout-prod-canary", Environment = "prod-canary" }
  }
}

resource "aws_autoscaling_group" "checkout_canary" {
  name                = "checkout-prod-canary"
  vpc_zone_identifier = data.aws_subnets.default.ids
  target_group_arns   = [aws_lb_target_group.checkout_canary.arn]
  health_check_type   = "ELB"
  min_size            = 0
  max_size            = 6                # room to match blue's full size on a successful promotion
  desired_capacity    = 0                # idle between releases — the pipeline wakes it when a rollout starts
  launch_template { id = aws_launch_template.checkout_canary.id, version = "$Latest" }
  tag { key = "Name", value = "checkout-prod-canary", propagate_at_launch = true }
}

Run it the same reviewed way Part 2 taught — terraform plan -out=tfplan -var-file=prod.tfvars -var canary_image_tag=a18e4f0, read every line, then terraform apply tfplan. Notice what this apply does not do: it never touches module.checkout's block, blue's ASG, or blue's target group at all — the diff is additive only, exactly the property infrastructure as code promises when a module boundary is drawn correctly.

The weighted listener rule, and the alarm that watches only the canary

☺ Like you're 10: A dial that decides what percentage of customers walk into which lane, plus a guard whose whole job is watching the new lane and nobody else's.

Two more resources, in the same canary.tf file, finish the picture: the rule that splits live traffic by percentage, and the alarm that decides whether that percentage is allowed to grow.

resource "aws_lb_listener_rule" "canary_split" {
  listener_arn = module.checkout.listener_arn
  priority     = 100
  action {
    type = "forward"
    forward {
      target_group { arn = module.checkout.target_group_arn,        weight = 100 }
      target_group { arn = aws_lb_target_group.checkout_canary.arn, weight = 0   }
      stickiness { enabled = false, duration = 1 }   # weighted split, not session-pinned
    }
  }
  condition { path_pattern { values = ["/*"] } }
}

resource "aws_cloudwatch_metric_alarm" "canary_5xx" {
  alarm_name          = "checkout-prod-canary-5xx"
  namespace           = "AWS/ApplicationELB"
  metric_name         = "HTTPCode_Target_5XX_Count"
  statistic           = "Sum"
  period              = 60
  evaluation_periods  = 2
  threshold           = 5
  comparison_operator = "GreaterThanThreshold"
  treat_missing_data  = "notBreaching"
  dimensions = {
    LoadBalancer = module.checkout.alb_arn_suffix
    TargetGroup  = aws_lb_target_group.checkout_canary.arn_suffix
  }
}

output "canary_rule_arn"         { value = aws_lb_listener_rule.canary_split.arn }
output "canary_target_group_arn" { value = aws_lb_target_group.checkout_canary.arn }
output "blue_target_group_arn"   { value = module.checkout.target_group_arn }

The rule starts at weight = 100 for blue and weight = 0 for canary the moment it's created — it exists, but sends nobody to the new fleet until a rollout deliberately says otherwise. The alarm's dimensions block is the detail worth slowing down on: it names both LoadBalancer and TargetGroup, which scopes HTTPCode_Target_5XX_Count to the canary's own instances only. Leave off the TargetGroup dimension and you get the ALB's blended, fleet-wide 5xx count instead — a canary erroring on 100% of its own small slice of traffic barely moves a number that's 90%+ healthy blue underneath it, and the alarm that's supposed to catch a bad build quietly never fires.

Wiring the rollout into Part 1's pipeline

☺ Like you're 10: The same inspector from Part 1, now also holding the dial that decides how many customers walk into the new lane.

Everything above is infrastructure — reviewed, applied once, sitting idle until a release actually starts. The rollout itself is a runtime operation, not an infrastructure change, so it belongs in a script the pipeline calls, not in another terraform apply. Add a job to the end of Part 1's .github/workflows/ci.yml:

## .github/workflows/ci.yml — deploy-canary job, appended in Part 3
  deploy-canary:
    needs: build-and-test
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest
    environment: prod              # requires a reviewer's approval before this job even starts
    permissions:
      id-token: write              # OIDC — no long-lived AWS keys stored in the repo
      contents: read
    steps:
      - uses: actions/checkout@v4

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/checkout-svc-deploy
          aws-region: us-east-1

      - uses: hashicorp/setup-terraform@v3
        with: { terraform_version: "1.9.8" }

      - name: Provision this build's canary launch template
        working-directory: infra/envs/prod
        run: |
          terraform init
          terraform apply -auto-approve -var-file=prod.tfvars \
            -var canary_image_tag=${{ github.sha }}

      - name: Run the weighted rollout
        run: bash scripts/promote-canary.sh

The environment: prod line uses GitHub's own environment protection rules — a named reviewer has to approve the job before it starts, the same manual gate Part 2 put in front of a raw terraform apply against prod. Once approved, this job first applies only the canary launch template for ${{ github.sha }} — blue's variables are untouched, so that apply's diff is exactly the one resource change it needs to be — then hands off to the script that actually moves traffic:

#!/usr/bin/env bash
## scripts/promote-canary.sh — the runtime half; no terraform apply anywhere below this line
set -euo pipefail

RULE_ARN=$(terraform -chdir=infra/envs/prod output -raw canary_rule_arn)
BLUE_TG=$(terraform -chdir=infra/envs/prod output -raw blue_target_group_arn)
GREEN_TG=$(terraform -chdir=infra/envs/prod output -raw canary_target_group_arn)
GREEN_ASG=checkout-prod-canary
BLUE_ASG=checkout-prod
ALARM=checkout-prod-canary-5xx
BAKE=180
STEPS=(10 50 100)

set_weight() {   # $1 = canary's percentage
  local green=$1 blue=$((100 - green))
  aws elbv2 modify-rule --rule-arn "$RULE_ARN" --actions '[{
    "Type":"forward","ForwardConfig":{"TargetGroups":[
      {"TargetGroupArn":"'"$BLUE_TG"'","Weight":'"$blue"'},
      {"TargetGroupArn":"'"$GREEN_TG"'","Weight":'"$green"'}]}}]' >/dev/null
  echo "traffic split -> blue ${blue}% / canary ${green}%"
}

alarm_in_alarm() {
  [ "$(aws cloudwatch describe-alarms --alarm-names "$ALARM" \
       --query 'MetricAlarms[0].StateValue' --output text)" = "ALARM" ]
}

rollback() {
  echo "canary alarm tripped -- rolling back to 100% blue"
  set_weight 0
  aws autoscaling update-auto-scaling-group --auto-scaling-group-name "$GREEN_ASG" --desired-capacity 0
  exit 1   # fail the job -- a red deploy-canary run is the point, not a bug
}

echo "waking the canary fleet..."
aws autoscaling update-auto-scaling-group --auto-scaling-group-name "$GREEN_ASG" --desired-capacity 1
until aws elbv2 describe-target-health --target-group-arn "$GREEN_TG" \
      --query 'TargetHealthDescriptions[0].TargetHealth.State' --output text 2>/dev/null | grep -q healthy; do
  sleep 10
done

for pct in "${STEPS[@]}"; do
  set_weight "$pct"
  echo "baking ${BAKE}s at canary=${pct}%..."
  sleep "$BAKE"
  alarm_in_alarm && rollback
done

echo "held through every step -- canary is now serving 100% of prod traffic"
echo "parking blue as the instant-rollback fallback, not deleting it"
aws autoscaling update-auto-scaling-group --auto-scaling-group-name "$BLUE_ASG" \
  --min-size 0 --desired-capacity 0

Proving it: a build that passes its health check and still gets caught

☺ Like you're 10: The new register turns on fine and says "ready" — the bug is only in how it rings up a real sale, and only real customers find that out.

The most convincing bad build for this test isn't one that crashes outright — a crashed process fails its target-group health check and never receives a single request, canary or not. The build worth proving against is the one that passes /healthz because the process itself is alive and listening, while a real request path — a downstream call that got misconfigured in the same change — starts returning 5xx. Ship that build as a18e4f0 and watch the rollout:

git push origin main   # deploy-canary job starts once a reviewer approves the prod environment gate

# ...a few minutes in:
# traffic split -> blue 90% / canary 10%
# baking 180s at canary=10%...
# canary alarm tripped -- rolling back to 100% blue

aws elbv2 describe-rules --listener-arn "$(terraform -chdir=infra/envs/prod output -raw ...)" \
  --query 'Rules[?Priority==`100`].Actions[0].ForwardConfig.TargetGroups'
# blue back at Weight 100, canary at Weight 0 -- confirmed independently of the script's own log line

aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names checkout-prod-canary \
  --query 'AutoScalingGroups[0].DesiredCapacity'
# 0 -- the bad instance is gone, not just untrafficked

The whole point is in that transcript: checkout-svc:a18e4f0 never received more than 10% of prod's real traffic, for no longer than one bake window, and nobody pressed a rollback button — the alarm crossing its threshold is what triggered rollback(). Compare that against what today's old deploy process (a straight cutover of the single prod target group) would have done with the exact same bad build: 100% of prod traffic, immediately, with a human's own attention as the only detection mechanism. That gap — bounded, automatic exposure versus unbounded, human-detected exposure — is the entire value this page adds.

What "done" looks like for Part 3, and where Part 4 picks up

☺ Like you're 10: A second lane that turns itself on, watches itself, and turns itself back off if it has to — proven, not assumed.

At the end of this part: prod has a second target group and a canary ASG sitting idle behind the same ALB Part 2 built; a weighted listener rule and a canary-scoped CloudWatch alarm both exist as reviewed, applied Terraform; a deploy-canary job in Part 1's pipeline shifts real traffic in steps and can roll itself back with no human pressing a button; and you have watched, with your own commands, a build that passed its health check still get caught and confined to 10% of traffic before it was reversed. Nothing here gets thrown away:

Part 3 gave checkout-svc a real deployment strategy — a rollout that promotes on its own and rolls itself back on its own, gated by an alarm that watches only the fleet actually under test. Continue to Capstone Part 4 — Observability, where this page's one narrow alarm becomes real dashboards on real golden signals. Or step back to Ship It — Start Here for how this capstone's six parts fit together, revisit deployment strategies and infrastructure as code for the concepts behind what you just built, and drill this exact failure mode under timed conditions in Drill — Roll Back a Bad Deploy.