A launch-day bug, and why the hotfix was the conservative call, not the brave one.
We had a launch-day bug on an early-access release this week. A segment of customers couldn’t see their appointments. It’s the kind of thing that makes your stomach drop: visible, customer-facing, on something you just shipped.
The reflex in that moment is to roll back. Rollback feels safe because it returns everyone to a known good state. But rollback has a blast radius of its own. It unships the release for everyone, including the customers it’s working perfectly well for, and it turns a contained problem into a broad disruption. The safe-looking move isn’t always the one that harms fewer people.
The call
We went with a hotfix instead. Three things made that the conservative choice, not the risky one. The fix was a one-line change. The resolution window was roughly two hours. And the affected set was bounded, on the order of a hundred and twenty accounts, with high confidence in the fix. When impact is contained, the fix is well understood, and the turnaround is short, a hotfix disrupts fewer people than a full revert.
There was a second, quieter call that’s easy to miss: we told customer support to hold on outreach. A fix landing within hours meant that proactive comms would create two touches, the apology and then the all-clear, when the situation only warranted one. Every notification is a touch, and an unnecessary one costs a little trust even when it’s well-intentioned.
The teachable part
Rollback versus hotfix isn’t a courage question, it’s a blast-radius question. Three variables decide it: how bounded the impact is, how confident you are in the fix, and how fast you can ship it. When all three are favorable, reverting actually disrupts more people than fixing forward. The comms hold is the same logic pointed at the customer’s inbox.
So write down the blast radius of your response, not just the bug. The cost of the cure is a variable you get to design. Caution is about minimizing harm, and sometimes the smaller, faster fix is the one that harms the fewest people.