Reverting Commits in Client Apps

When faced with a critical bug in a client app that’s already shipped to customers, one of the key first steps in the debugging process is finding the earliest commit in which the bug is present. Tools like git bisect can make finding this commit much easier. But what do you do once you’ve found the offending change?

The obvious answer is to revert the commit, verify that the bug is resolved, and ship a new version of the app. Sometimes this is all it takes - you found the commit that introduced the bug and removing it took the bug away. But sometimes reverting a commit can cause even bigger problems. Rolling back code without understanding why it fixes the problem can be dangerous.

Discovering that the reversion resolved the issue is only one piece of understanding the effects of reverting that commit. There are multiple potential results when you roll back a commit, such as:

That last case is obviously quite bad. But how can this happen? After all, you’re reverting back to code that you’ve already shipped to your customers in the past, right?

Unfortunately, that’s not entirely true. Yes, the specific lines of code you’re reverting to have shipped before, but in many cases other parts of the app it interacts with will have changed since then. Identical code is only guaranteed to behave the same if all of the code around it and the data it processes is also the same.

There’s an important distinction here that comes up a lot when talking with our friends on the server side of things. Client rollbacks typically involve reverting a commit, while server rollbacks typically involve reverting to a commit. Reverting to a commit allows the project to be redeployed with all of the code in the same state it was when that (hopefully stable) commit was originally shipped. There’s still a danger of the data being different, but this makes the risk analysis much easier.

So what questions can you ask to determine whether it’s safe to revert a commit? Here’s some to get started:

This is not meant to be a complete checklist, but it illustrates some of the ways that reverting a commit can have unintended consequences. When faced with a crash and a potential commit to revert, your job is to perform the risk analysis to determine how likely you are to hit any of those consequences. When I’m unsure of the risk to revert, I find it best to think about a revert commit as if it were a newly written fix. Understanding the change you’re reverting is just as important as understanding new code you write.

With the time pressure of a crash in the wild, it can feel like a waste of time to do this extra analysis before shipping a potential fix. There’s a saying that I think applies really well to this situation though: slow is smooth, smooth is fast. Iterating on mobile releases can be a somewhat slow process given the delays caused by the app review process. Taking the time to increase confidence in your first fix can make the entire process faster. If your team has the resources available, it can be helpful to prepare a new version and submit it for review, but wait to start rolling out to customers until you’re confident in the fix.

I realize this probably paints a fairly bleak picture of the debugging process. Unfortunately there is no quick and easy answer for what to do when you have a serious bug out in the wild that will work for all situations. Reverting is a great solution when it works, but needs to be considered based on the current situation. The good news is that there are steps you can take to increase the likelihood of being able to safely revert changes in the future. Things like:

Reverting an old commit is a great tool in the bug-fighting toolbox, but it isn’t always the right tool for the job. It’s important to understand when it’s safe to revert a commit, so you can keep the option available while avoiding a potential disaster.