Production Surfaces Bugs Your Tests Can't
First time I clicked "Study 10" in newly-built Study mode, the celebration animation fired immediately, before any flashcard appeared. Confetti. "Nice work!" banner. Tried to navigate me back to the home screen.
I had not studied anything. I had clicked the button and been congratulated for finishing.
Tests passed. Types checked. Code reviewed clean. On my phone, the feature was a clown show.
Why tests didn't catch it
The feature: a deck that cycles through unmastered lyrics, removing mastered cards, shuffling missed ones back, celebrating when the deck is empty.
The test for "celebrate when the deck is empty" checked that when the deck was empty, celebration logic fired. The test passed.
The bug was in the timing. On first render, the data hadn't loaded yet. The list of cards was [] — empty, because the server hadn't responded. The component checked: deck empty? Yes. Fire celebration. A fraction of a second later, real data arrived. By then the celebration had already fired.
The test couldn't see this because the test gave the component the real data up front. In the test's universe, data was always present. The test never simulated "render the page, wait for data, render again." In the real-world sequence, there's a brief window where the page thinks the deck is empty because there's no deck at all yet.
The fix
About twenty lines: add a flag for "data hasn't loaded yet." Only check whether the deck is empty after data has arrived. Until then, show a loading state.
Took five minutes to write. The bug took ten seconds to notice — because I actually clicked the button.
Categories of bug tests in this project have missed
- Timing bugs in loading sequences. The Study mode bug. Anything with "wait for data, then show something" has a race condition tests rarely exercise.
- User-visible regressions in unrelated features. I fixed a scoring bug and accidentally broke the audio replay button on a completely different page — both used a shared hook. No test failed. Noticed because I replayed a recording.
- Copy that's technically correct but terrible. "Card hinted, will not count as mastered this round" is precise. On a phone screen at 7am with a toddler tugging my sleeve, it's incomprehensible. No test can tell you that.
- Accessibility issues. The Study page had an outcome banner I could see but a screen reader couldn't. Tests didn't know what a screen reader was.
- Performance feels. Recording upload worked, scientifically. It also took four seconds with no indication anything was happening. "Worked" is not the same as "felt good."
All bugs. None failed a test. All visible within ten seconds of actually using the product.
The rule
Nothing is done until I have used it on my phone, on a preview link, as an end user would.
I've blocked my own PRs with it. Feature passes tests, types, lint, but I haven't played with it: not done. Back it goes.
The reason this matters more with an AI assistant: when a feature takes 30 minutes to build instead of 8 hours, 20 minutes of clicking around feels like almost doubling the cost. You start cutting corners. You reason: "the tests passed." You ship. The confetti fires immediately.
What I ask Claude to do now
The shape of "finish this feature" requests now includes: "Before we merge, let's make sure it actually works in a browser. Start the dev servers, walk me through the things to click."
Claude turns out to be good at this. It starts the backend, starts the frontend, gives me a URL, and narrates: "Open the home page. You should see a song list. Click on the first song. You should see a Study mode button. Click it. You should see a flashcard." I do the clicking, report what I see.
Tests catch categories of bug that repeat. The human catches the bugs that are new this week. Both.
Takeaways
- Tests assert what the author anticipated. New bugs — the ones the author didn't anticipate — are invisible to them.
- Categories tests routinely miss: timing/race conditions during loading, cross-feature regressions through shared code, technically-correct-but-bad copy, accessibility, performance feel.
- "Done" includes using the change on a phone, on a preview link, as an end user would. Block your own PRs on it.
- The faster the AI builds, the stronger the temptation to skip the clicking. The cost of skipping doesn't go down; it goes up.
Terms
Test suite — A collection of automated checks that run when code changes. Each test asserts that some piece of the code does what's expected ("when this function is called, it returns X"). If any test fails, the build is blocked.
Type check — A separate kind of automated check that confirms data is being passed around in the right shapes (a function expecting a number doesn't receive a string). For TypeScript / Python type-annotated codebases.
Lint — A third kind of automated check that flags style issues and common mistakes (unused variables, unreachable code).
Race condition — A bug where two things happen in an unexpected order. The Study mode bug was a race between "page renders" and "data loads" — sometimes the page rendered first and saw an empty deck.
Hook — In React (the framework the frontend uses), a reusable piece of state-and-behavior that multiple components can share. If two unrelated components import the same hook, changing it can affect both — easy to forget.
Screen reader — Software that reads page content aloud for visually impaired users. Tests usually don't check what a screen reader sees, so accessibility regressions slip through silently.
Preview link — Vercel auto-builds a temporary URL for every pull request branch (sing-with-me2-git-<branch>-...vercel.app). Lets you click around the change as it would feel in production before merging.
Dev server — The local processes that run the frontend and backend on your laptop while you're working on the code. npm run dev for the frontend; uvicorn for the backend.