@anthropy @greyduck Well, originally it made a lot of sense to make 'crawling ok unless told not'. Search engines were how whatever you wrote could be reached, and presumably you wanted people to reach your site. Nowadays is another situation.