Managing which bots are allowed access to your site is almost a full time job, even with Cloudflare. The line is blurry between AI crawler/Search Crawler. But the line is pretty clear on who respects the crawl delay and who doesn't 😡