The Hidden Cost of Updating Your Backend Without a Rollback Plan
Photo by charlesdeluvio on Unsplash
Imagine owning an expensive speaker (connected through an app, naturally) and one day the alarm stops working correctly.
That’s right, Sonos have managed to upset their customers by breaking a host of functionality users have gotten used to for years. Breaking functionality when a system costs thousands of dollars is a problem.
When software engineers can’t roll back the change, you’re in a world of pain.
Welcome to today’s cautionary tale.
The App Release
The Sonos front-end engineers decided to revamp their app.
On launch Sonos CEO was bullish.
“We felt now was the time to reimagine our app experience. After thorough development and testing, we are confident this redesigned app is easier, faster and better.”
It seems clear that he hadn’t tried the app? Also he seemed to have blind faith in the work of his software engineers. A misplaced faith.
No Way Back
The business had decided to release the app in a state where even basic features missed the day one release.
Customers were furious that their expensive home speaker system stopped working in the way they had for years. Product owners looked at a JIRA board and said that features were “coming soon”, which did nothing to placate customers.
As alarm bells at Sonos rang, forcing the CEO to step up and make an announcement that was barely heard amongst the calls for the old app.
“There are clearly people who are having an experience that is subpar”
Yeah. And he had this to say about rolling back to the old app (S2).
“The trick of course is that Sonos is not just the mobile app, but software that runs on your speakers and in the cloud too. In the months since the new mobile app launched, we’ve been updating the software that runs on our speakers and in the cloud to the point where today S2 is less reliable & less stable than you remember. After doing extensive testing we’ve reluctantly concluded that re-releasing S2 would make the problems worse, not better. I’m sure this is disappointing. It was disappointing to me.”
The Lessons
Rollback
I think most readers of this blog are familiar with the idea that you need a rollback plan. If something goes wrong, what are you going to do?
Each and every software upgrade that I’ve worked on has had a rollback plan. The fact that Sonos has been unable to revert changes when something has gone wrong with their core set of products is amazing.
Tight Hardware and Software Coupling
Leaving customers stuck on broken software, leaving their hardware at least partly non-functional is a massive problem.
It’s not just about keeping the app and backend aligned; the tight coupling of hardware and software introduces its own set of challenges. Sonos’ speakers rely on cloud-based services, meaning every update to the backend has to be reflected across the entire ecosystem. If the app changes but the backend or speakers aren’t properly synced, you’re not going to hear those banging beats.
From a development perspective, it’s clear that the company was trying to juggle multiple moving parts. Hardware like Sonos speakers often require updates not just in the software users interact with but also in the underlying systems that manage communication between the app, speakers, and cloud services. It’s a delicate balance — and when one piece of the puzzle goes out of sync, everything goes sideways. Worse, you can’t simply roll back a backend change without the front end (in this case the speaker) changes.
What Sonos Should Have Done Differently
Sonos could have avoided much of the backlash by doing a few key things differently:
Communicate Early and Often
If users had been warned that some features would be temporarily unavailable due to backend updates, the backlash could have been softened. Transparency is key when rolling out major updates, especially when legacy features are affected.
Better Beta Testing
Sonos admitted that they missed some of the app’s flaws during testing. More comprehensive beta testing, including regression testing that covers all the hardware-software interactions, might have uncovered these issues earlier.
A Rollback Plan
How about that?
Conclusion
The Sonos debacle is an opportunity for us to learn from their mistakes. The poor customers were not just annoyed, they were heartbroken as the product they loved simply stopped performing how they would want.
The solution? Thorough testing, clear communication, and a deep understanding that your most loyal users depend on every piece of your system working seamlessly together.
We will see if Sonos can survive this long-term, or whether they might join Blackberry.