2026-09-02 09:54 UTC
@gabrielesvelto@mas.to @simonzerafa@infosec.exchange I don't see how doing country based filtering helps with AI crawlers. A lot of them are using botnets (also called "residential proxy") to get around any IP based filtering.
There are other techniques to block AI crawlers that are less disruptive for your audience.
Replies (1)
-
@gabrielesvelto@mas.to 2026-09-02 10:35
@anthony@indieweb.social @simonzerafa@infosec.exchange residential proxies are scattered around the world. If your audience comes largely from one country it's very effective to filter traffic from other countries given the vast majority of it will be bots. Heavy levels of proof-of-work filtering are very disruptive to users but seem to have become the norm if you want to keep the bots out: https://people.kernel.org/monsieuricon/creepy-crawlies I can see why people would chose a simpler solution that doesn't really affect their audience much.