How I Cut a Three-Day Infrastructure Migration to Under Ten Minutes

The migration ran on a wiki page. Forty steps, a few warnings in bold, and a note at the bottom that said do not skip step nineteen. It took three days, mostly because a human had to wait, watch, and re-check. It failed at least once per run, always at a different step.

A runbook that humans follow by hand is just a script nobody has written yet. So I wrote it.

Idempotency first, speed second

The temptation is to optimise for speed. That is the wrong first move. The first move is making every step safe to run twice. If a step already did its work, it should detect that and move on quietly instead of erroring or duplicating.

bash
ensure_bucket() {
  local bucket="$1"
  if aws s3api head-bucket --bucket "$bucket" 2>/dev/null; then
    echo "bucket $bucket already exists, skipping"
    return 0
  fi
  aws s3api create-bucket --bucket "$bucket" \
    --create-bucket-configuration LocationConstraint="$AWS_REGION"
}

Every function in the script follows that pattern. Check the desired state, act only if reality does not match it. Once every step is idempotent, a failure stops being a crisis. You fix the cause and run the whole thing again.

Fail loud, fail early

The script runs with strict shell options so an unset variable or a failed command halts execution instead of limping forward into a worse state.

bash
set -euo pipefail
trap 'echo "failed at line $LINENO" >&2' ERR

• Every AWS call is checked, not assumed to succeed.
• The script logs what it is about to do before it does it, so a failed run reads like a story.
• Dry-run mode prints the plan without touching anything, which is what you actually demo to stakeholders.

The result

The three-day process became a five to ten minute run. The more important number is the second one: human error went to zero, because there was no human in the loop to make one. The runbook is now code, it lives in version control, and the next migration starts from something that already works.