robots.txt is a published preference, not a fence. Rate limits are an engineering constraint. Terms of servic…

A site decides you're a bot before it shows you anything. How IP and ASN reputation, TLS fingerprints, header…