Replies (1)

• @lunareclipse@snug.moe 2026-08-29 15:10

  @pastelfluffyfox@meow.social see the problem is that these scrapers will literally take down your website in some cases, and making it hard enough to open that you need a full web browser that can execute code in order to load the page does stop the scrapers. This is especially true for websites that run a bunch of calculations on the server to respond, like wikis and git forges. Static websites usually don't care about serving more requests as much, and have these aggressive anti scraper measures much more rarely. It is infuriating that this is a thing, I've been blocked from accessing pages on my phone in some instances at this point, but this is entirely a result of AI scrapers ignoring and bypassing every single efficient mechanism we've had for decades for handling crawlers. Well behaved crawlers respect robots.txt, crappy crawlers report a user agent, hostile crawlers actively lie about what they are. But it used to be that if they pretended to be a real user, they would also try to stay under the radar by not causing issues like excessive load for the website owner. AI scrapers don't care and will use every possible measure to evade blocks, including using "residential proxies" ("free VPNs", malware, etc, will sell the victims internet connection as a proxy) to hide their origin. This is why we're at PoW challenges, because nothing else works, nothing else is viable unless you want to stare at server logs all day every day just for your little wiki or personal git forge to work. It's actually that bad. Even if the scrapers still go for it, if they have to run an entire Chromium instance to crawl a page, that will slow them down enough to stop the website from going down. So this is an issue entirely manufactured by the AI industry's ravenous hunger for data, and complete disregard for ethics. I mean, if they kill a website a competitor can't get the juicy Real Human training data right? Just like it is with destroying rare books after digitizing them into a top secret private dataset.

  Open ##5142607