What robots.txt, Rate Limits and Terms Actually Require

Most automation teams treat compliance as a paragraph in a README. It is actually three separate systems with three separate failure modes, and confusing them is how well-intentioned engineers end up on a block list — or somewhere worse.

The takeaway up front: robots.txt is a published preference in a machine-readable file, rate limits are a hard engineering constraint, and terms of service are a contract. None of the three is the law, and the law does not go away when all three are satisfied. Ignore robots.txt and you lose the operator's goodwill. Ignore rate limits and you degrade someone's service and your own success rate. Ignore terms and you have a contract problem. Satisfy all three and you are still answerable for what you collect and what you do with it.

Three rulebooks, and the law underneath them

Nothing on that list substitutes for anything else on it. A site with no robots.txt has not granted permission; a site whose terms permit scraping has not raised its rate limit.

What robots.txt actually is

The Robots Exclusion Protocol was folklore for twenty-five years before it was standardised as RFC 9309 in 2022. The specified parts:

• The file lives at /robots.txt on the origin — scheme, host and port. https://example.com and https://api.example.com are separate files, as are HTTP and HTTPS on the same host.
• Rules are organised into groups keyed by user-agent product token, matched case-insensitively. A crawler uses the group matching its own token; only if none matches does it fall back to the * group.
• Within the applicable group, the longest matching path rule wins, and where an Allow and a Disallow match equal length, Allow wins. * and $ are the only supported wildcards.
• A cached copy should not be reused for more than 24 hours.

The parts that are convention, not spec

Crawl-delay, Request-rate and Sitemap are not in RFC 9309. Sitemap comes from sitemaps.org and is independent of user-agent grouping; Crawl-delay is non-standard but widely honoured. Obey it when present — an operator who wrote a number down has told you their tolerance — but never read its absence as "no limit".

The parts people get wrong

A group for your agent replaces the wildcard group; it does not add to it. If the file has a User-agent: * block with ten Disallow lines and a User-agent: AcmeBot block with one Allow, AcmeBot is governed by that one line and the ten do not apply. Teams mis-read this in both directions.

Status codes are part of the protocol. A 404 means unrestricted. A 401 or 403 means treat the whole origin as disallowed. A 5xx means unreachable, not permissive — assume full disallow until it recovers.

Your parser probably does not implement the spec. CPython's urllib.robotparser is a reasonable default but it applies rules in file order and takes the first match, not the longest. Verified on 3.12:

>>> p.parse(["User-agent: *", "Disallow: /private", "Allow: /private/ok"])
>>> p.can_fetch("AcmeBot", "http://x/private/ok")
False                     # RFC 9309 says True — longest match wins

It also has no 5xx handling: a robots.txt returning 503 leaves the ruleset empty and every can_fetch answers True. Wrap it and add the missing cases yourself.

robots.txt governs fetching, not use. It says nothing about copyright, licensing, personal data or redistribution. A path being crawlable is not a licence.

Rate limits are an engineering constraint

A published limit is not a request for politeness. It is the operator naming the capacity they provisioned for you, and there are only two ways to learn where it sits: read what they published, or discover it by hurting them.

Read the response, in this order of authority:

1. Retry-After on a 429 or 503 — genuinely specified (RFC 9110), carrying delay-seconds or an HTTP date. It is the server naming its own limit; obey it exactly and never apply a shorter backoff on top.
2. RateLimit-Limit / RateLimit-Remaining / RateLimit-Reset, or the older X-RateLimit-* family. Convention rather than settled standard, so parse defensively, but when present they let you throttle before being refused.
3. The documented limit in the target's docs, usually enforced per key, per IP and per endpoint at once. The binding one is whichever is tightest.

The limit you were never told about matters most: concurrency. Plenty of services publish a requests-per-second figure and quietly enforce a parallel-connection ceiling too. Treat 429, 503 and a rising p99 as one signal — back off multiplicatively, recover additively, and cap concurrency per host rather than globally.

There is a self-interested reason to pace yourself too. Volumetric behaviour is the cheapest bot signal there is, and it feeds the risk score that decides whether you are challenged at all — the challenge is the output of that score, not the check itself. Hammering a host does not merely risk a block; it manufactures the CAPTCHAs you will then pay to solve.

Terms of service are a contract, not a statute

This is engineering guidance, not legal advice, and the details vary by jurisdiction — for anything consequential, ask a lawyer. But two categories are worth keeping straight:

• Terms of service are an agreement with the operator. Breaching them is generally a contractual matter: termination, account closure, a demand to stop. They can forbid things the law permits, and cannot authorise things the law forbids.
• Law runs independently. Computer-misuse statutes tend to turn on whether you accessed something you were not authorised to access, which is why anything behind a login, paywall or payment step is categorically different from a public URL. Copyright and database rights govern what you may store and republish, and data-protection regimes apply to personal data whether or not the page was public — "it was on the open web" is not a lawful basis.

The practical bright line is authentication. Public page, no credential, robots and rate limits honoured: ordinary engineering. Anything requiring you to be someone you are not — a credential you were not issued, an account created in bulk, a paywall routed around — is off the table, and this blog will not help with it.

Building the gate in code

Compliance that lives in a README is not compliance. Make it a module every fetch passes through: one gate per origin, holding a robots decision, a pace and a penalty box.

import threading, time, urllib.robotparser

UA = "AcmeQABot/1.2 (+https://acme.example/bot; ops@acme.example)"

class HostGate:
    """One per origin: robots decision, minimum interval, 429 penalty box."""

    def __init__(self, origin, default_interval=2.0):
        self.rp = urllib.robotparser.RobotFileParser()
        self.rp.set_url(origin + "/robots.txt")
        self.rp.read()                       # wrap this: add the 5xx case
        delay = self.rp.crawl_delay(UA)
        self.interval = float(delay) if delay else default_interval
        self.next_slot = 0.0
        self.hold_until = 0.0
        self.lock = threading.Lock()

    def allowed(self, url):
        return self.rp.can_fetch(UA, url)

    def wait_turn(self):
        with self.lock:
            start = max(time.monotonic(), self.next_slot, self.hold_until)
            self.next_slot = start + self.interval
        time.sleep(max(0.0, start - time.monotonic()))

    def back_off(self, seconds):
        """429/503 with Retry-After: the server just published its real limit."""
        with self.lock:
            self.hold_until = time.monotonic() + seconds

The shared hold_until is the part worth copying: Retry-After moves one deadline for every worker on that host, so they back off together instead of each discovering the 429 separately. The user agent matters too — descriptive, with a contact URL, because being identifiable is what lets an operator allow-list you rather than block you.

Where CAPTCHA solving legitimately sits

Solving is downstream of the gate, never a substitute for it. If allowed() returns False, there is nothing to solve — you were asked not to fetch that path, and a token does not change the answer.

The lanes where a solver is straightforwardly legitimate:

• QA automation against an application you own or are contracted to test. Prefer the test site keys Google and Cloudflare publish for exactly this; reach for a solver when a staging environment sits behind a real challenge you cannot reconfigure.
• Accessibility testing — CAPTCHAs are a documented accessibility barrier, and audio fallbacks fail often enough that verifying your own flows is real work.
• Synthetic monitoring of services you operate or are authorised to check, where a challenge turns a health check into a false alarm.
• Lawful data work — public data collected within the target's terms, robots and rate limits, for research, price or compliance monitoring, or archival.

The lanes it is not for: authentication of any kind, mass account creation, credential stuffing, payment fraud, spam, paywall circumvention. Those are not edge cases handled carefully — they are outside what this product and this blog exist for.

When the gate passes and the challenge is still there, CaptchaAI uses the legacy 2Captcha-shaped protocol: submit to in.php, poll res.php.

curl -s -X POST "https://ocr.captchaai.com/in.php" \
  -d "key=YOUR_API_KEY" \
  -d "method=userrecaptcha" \
  -d "googlekey=6Lc..." \
  -d "pageurl=https://example.com/public-report" \
  -d "proxy=user:pass@203.0.113.10:8080" \
  -d "proxytype=HTTP" \
  -d "json=1"
# {"status":1,"request":"2122988149"}

# 2. Poll roughly every 5s until it stops saying not-ready.
curl -s "https://ocr.captchaai.com/res.php?key=YOUR_API_KEY&action=get&id=2122988149&json=1"
# {"status":0,"request":"CAPCHA_NOT_READY"}   -> keep waiting
# {"status":1,"request":"03AGdBq26..."}       -> token; submit it with the form

CAPCHA_NOT_READY is the only response meaning keep polling; ERROR_UNSOLVABLE and ERROR_ZERO_BALANCE are terminal, so surface them rather than retrying into them. Poll to a deadline, not a fixed duration — CaptchaAI publishes under half a second for image CAPTCHAs and under 15 seconds for a Cloudflare Challenge.

Two details tie the solver back to the gate. The poll loop talks to ocr.captchaai.com, not to the target, so it costs nothing against the target's budget — but the retried request afterwards does, and a challenged request is still a request. And because CaptchaAI prices concurrent threads rather than solves — published tiers run BASIC at $15/month for 5 threads, ADVANCE at $90/month for 50, ENTERPRISE at $300/month for 200 — there is no per-CAPTCHA fee and no daily cap to slow you down. Cost will not police your pace; your gate has to. Chosen deliberately, the thread count doubles as a ceiling on challenged requests in flight.

Pre-flight checklist

Before a new target enters your pipeline: read robots.txt by hand once and find the group that applies to your token; look for a published API or bulk export, which beats scraping every time; halve the documented rate limit; set a descriptive user agent with a contact address; confirm nothing in the flow requires a credential; and name which of the four lanes this job is in. If you cannot name the lane, do not build the job.

FAQ

Does robots.txt apply to my QA suite hitting my own staging site?

No — the protocol governs automated fetching of someone else's origin, and on your own property you are the operator. It applies the moment your suite touches a third party: a partner sandbox, a vendor API, an external dependency in an end-to-end test.

Is ignoring robots.txt illegal?

The file carries no enforcement of its own; it is a convention, and this is not legal advice. Exposure arrives from the neighbouring layers — terms, computer-misuse statutes, copyright, data protection. Practically, ignoring a published Disallow also reads as intent and is the fastest route to a permanent block.

Is using a CAPTCHA solver against a site's terms of service?

Sometimes — it depends entirely on the target, so read their terms rather than generalising. Where it is prohibited, the answer is to ask permission, use a published API, or not do the job. On your own application, an accessibility audit, authorised monitoring, or collection the target permits, solving a challenge is a normal engineering step.

How do I choose a rate when nothing is published?

Start at one request per second per host with a concurrency of one or two. Watch 429s, 5xxs, latency percentiles and challenge rate together: increase in small additive steps while all four stay flat, back off multiplicatively the moment any moves, then cap it. A discovered limit is not an invitation to sit on it.

Next step

Pick your busiest target and answer three questions in writing: which robots group applies to you, what the published rate limit is, and which lane the job belongs to. Most teams find at least one answer they did not have. Once the gate exists and the self-inflicted challenges are gone, the residue is what a solver is for — see how CaptchaAI handles it, sized against peak concurrent solves rather than daily volume.