@paxed – Same here, for https://roter-kreis.de/. They ignore robots.txt anyway, so I gave up and let them crawl. Meanwhile, AI crawlers produce most traffic. Indeed, the only effective defense would be the removal of content from public access, but then you become invisible to classic search engines.