The Digital Parasite
Photo by Sangga Rima Roman Selia on Unsplash
Software developers have enough issues to deal with without taking part in an arms race where nobody ever wins. Product owners might ask for impossible deadlines to be met. Designers might think of a UX that has never been seen before, and nobody can use. There might be bugs that take 10x the expected time to solve.
Software engineering can be very stressful and intense.
So, when you find out that users want to use your product yet do not want to pay for it you are well within your rights to think “Why don’t we just block it”? That’s when the can of worms starts, and you’re a couple of lines of code away from damaging your paying user’s experience.
Lower the Paywall?
I write content on the web. You can tell, because you’re reading it now. If you don’t like Medium’s paywall I’m also publishing on my own website for free, because I’m nice like that.
The issue is that not all content is as great as mine (lolz), and it’s not all available for free. That makes sense, journalism and quality content costs money because people need to be paid (until recently I didn’t, having a job as a software developer).
I’m well aware that readers hate paywalls. Essentially, they hate paying. So, when people use tools like 12ft Ladder (12ft.io) to remove paywalls they are doing nobody any favors. It sets up a game of cat and mouse between developers and users, and like all such games, it leads to an arms race that nobody really wins.
The Issue
At its core, 12ft Ladder is a scraping tool. It bypasses website restrictions by pretending to be Googlebot, tricking websites into serving up unrestricted content. Users love it because it lets them read paywalled content for free. Developers, on the other hand, see their source of income undermined and essentially are forced to create countermeasures to protect their work.
The more effort developers put into stopping scrapers, the more effort scrapers put into getting around those blocks. It’s an infinite loop of wasted energy.
Developer’s Nightmare
A Waste of Time
As developers nothing makes us happier than creating and pushing new features. I’m sure some cyber security people are out there who love the battle with people who try to circumvent protections, but I’m speaking to the majority of developers here.
Whether it’s bot mitigation, ad-blocker detection, or scraping prevention, it all amounts to a giant time sink that stops us delivering great experiences to users. Every hour spent adding bot detection and fingerprinting techniques is an hour not spent improving the actual product.
Bad UX Decisions
Because of tools like 12ft Ladder, some websites have responded by making life worse for all users. You’ve seen it before, and I’ve been in meetings where product owners have spoken about being forced to make changes that are to the detriment of users.
I worked on an app where we used a nest of anti-security measures that made the user experience worse. You needed a user id to login, a username, password and passcode. The reason for this? Legal said that it was more secure.
Don’t get me started on company policies which make you change your password every 3 months. If you aren’t writing down those passwords somewhere I really don’t know how you cope.
If we can’t keep a high-quality developer experience within our companies what hope is there for the end user experience?
I’m also going to say it. None of these measures ultimately help. They create annoying experiences for real users while doing little to nothing to prevent the most determined circumventing of the measures that you have lovingly implemented.
The Unwinnable Game
Blocking scrapers is like playing chess against a cheater and while you may think you’ve won, but they’ll always find a way to move the pieces in their favor whether it is against the rules or not.
Developers who fight against scrapers quickly realize the harsh truth that every countermeasure they put in place is temporary.
Here are some examples.
You block an IP range? They use proxies.
You require JavaScript? They run a headless browser.
You check for known bot user agents? They randomize them.
You fingerprint devices? They use spoofing tools.
It’s an endless arms race, and at some point, you have to ask whether this is really worth it in the end?
Who Really Loses?
Frankly it’s everybody.
Users get frustrated because legitimate browsing becomes more painful.
Companies lose revenue, which means inevitable layoffs or they stop viewing their customers as anything other than an inconvenience.
Developers waste their time battling a moving target rather than building great features.
The web becomes a mess of artificial barriers and dark patterns, where every site assumes you’re trying to steal from them.
12ft Ladder is a symptom of wider problems. I don’t blame the people developing or using it, but it’s escalating the conflict between developers and end users. Piracy for paywalls feels like a win for users in the short term, but in a longer time horizon it is just making everything worse.
What’s the Solution?
So, what can be done about this?
Developers
Instead of fighting an unwinnable battle, focus on compelling features that people want to pay for. Remember when Radiohead released an album with a pay what you want pricing strategy in 2007. They had enough fans who would pay more than the minimum $0 for their work, can you make your work good enough so customers choose to pay for it? This strategy is contrary to annoying users into paying for work through annoyance.
Users
If you like content, support the people who make it. If a paywall is unreasonable, don’t consume the content. Forcing a company to rethink its approach is better than cheating the system, and of course stealing is morally dubious.
Companies
Consider alternative monetization strategies, like freemium models, subscriptions with added value, or ethical advertising. As an industry, we can do better, and it is time we did so.
Conclusion
If we keep going down the path of scrapers vs. developers, we’re just going to end up with a web full of hostile design, constant roadblocks, and an increasingly miserable experience for everyone.
But hey, maybe that’s the future we deserve.