← All writing
Singy Songbook · 3 of 12
May 3, 2026 · 4 min read

The Outage That Turned into a Memory File

Merged a PR that added analytics events. Both Fly machines crash-looped within a minute. About 15 minutes of downtime.

The PR itself was clean — nothing to do with database migrations. It just changed startup timing enough to expose a latent bug.

What the logs showed

Will assume transactional DDL.

Then nothing. Machine kill. Restart. Same line. Nothing.

Alembic was printing its banner and hanging. Migration wasn't running, uvicorn wasn't starting, health checks were timing out.

What was actually wrong

alembic upgrade head was running inside FastAPI's lifespan context manager. The migration was sharing the event loop with uvicorn — calling a sync Alembic API from inside an async context. It had been working by accident: just enough daylight in startup timing for the migration to finish before the loop got busy. The analytics PR added enough to startup to close the window. Migration blocked the loop; loop blocked the migration. Deadlock.

Reverting the analytics PR would have hidden the bug, not fixed it. Same failure would resurface on the next PR that changed startup timing.

The fix

Move migrations out of the lifespan entirely. Dockerfile CMD now runs them in a separate process before uvicorn starts:

CMD ["sh", "-c", "alembic upgrade head && uvicorn app.main:app --host 0.0.0.0 --port 8080"]

Migrations run to completion before uvicorn exists. They can't block an event loop that doesn't exist yet. If they fail, the container crashes and Fly surfaces the error. The FastAPI lifespan became empty.

Deploy was healthy about 15 minutes after the first crash.

The wrong path first

First attempt was a Python wrapper script that called Alembic programmatically and then imported uvicorn. Same hang, different wrapper. About 15 minutes of digging before Claude said "I've been spelunking too long here, let me sidestep by moving migrations out of Python entirely."

What got written down

Two memory files after prod was healthy.

feedback_migrations_outside_lifespan.md: migrations run via the Dockerfile CMD, not inside FastAPI's lifespan. Push back on any "just run it in lifespan" suggestion.

feedback_smoke_test_before_merge.md: boot uvicorn for real (not just unit tests) before merging changes that touch startup. For PRs touching main.py, lifespan, Dockerfile, fly.toml, or any import-time side effect, run uvicorn locally and confirm it reaches "application startup complete" before merging.

Both files load into context on every new conversation. No deploy hangs since.

Takeaways

  • Migrations don't belong in an async event loop. Run them as a separate process before the server starts (Dockerfile CMD, not FastAPI lifespan).
  • A latent bug surfaced by an unrelated change isn't fixed by reverting the unrelated change. The right fix removes the condition that made the bug possible.
  • Compile-time checks (tests, types, imports) don't catch startup-shape regressions. Boot the real server before merging.
  • Encode architectural decisions with their reasons in memory files. The reason is what lets future sessions tell when the rule applies and when it doesn't.

Terms

Pull request (PR) — A proposed batch of code changes. On GitHub, opening one creates a review surface — a list of what changed, comments, and a Merge button.

Fly — Fly.io. Hosts backend services in containers. The "machines" are the running instances of my backend.

Crash-loop — A container or process that fails on startup, gets restarted by the platform, fails again, and so on.

Migration — A script that changes the database schema (adds a table, adds a column, etc). Run when the app deploys so the running code matches the database shape.

Alembic — The Python tool that runs migrations. Reads a folder of migration scripts and applies the ones that haven't been applied yet.

Dockerfile — A text file describing how to build a container image: what to install, what files to copy, what command to run.

CMD — The Dockerfile directive that specifies what command runs when the container starts.

FastAPI lifespan — FastAPI is the Python web framework the backend is built with. The "lifespan" is its startup-and-shutdown hooks: code that runs when the server boots and when it stops.

Uvicorn — The web server that runs FastAPI. Handles incoming HTTP requests.

Async / event loop — Python's mechanism for handling many requests concurrently in a single process. Code that's blocking (doesn't yield control) prevents other work from running. A migration that calls a sync database driver from within the async loop blocks it.

Deadlock — Two operations each waiting for the other to finish. Neither makes progress.

Health check — A request the hosting platform sends to the app every few seconds to confirm it's responding. If it fails enough times in a row, the platform restarts the container.

Memory file — Plain text file Claude Code automatically reads on every new session. Records preferences and lessons so they persist across conversations.