robots.txt is a published preference, not a fence. Rate limits are an engineering constraint. Terms of servic…