Why Your Solve Rate Dropped Overnight

Yesterday you were at 98%. This morning you are at 61%, and git says nobody touched the pipeline in nine days. The instinct is to open a ticket with the solver — which usually comes back saying success rates look normal on their side, an answer that is both true and useless.

Key takeaway: "solve rate" is two rates that have been quietly added together. One is whether the solver returned an artifact. The other is whether the target accepted it. They fail for completely different reasons, and until you separate them you are guessing. Once they are separate, the shape of the drop — uniform or dispersed, step or ramp — tells you which of three things changed on the target.

First, split the number in two

A solve has two verdicts, and only the first belongs to the solver.

1. Did a token or answer come back? That is the solver's job, and the number its dashboard shows.
2. Did the target accept it? That happens in the request you used the token in, minutes later, somewhere else entirely in your code.

Most teams instrument stage one because it is local and easy. Stage two is where the interesting failures live: a valid reCAPTCHA v3 token can still be scored and rejected by the site's backend after the solver did everything right. If your dashboard cannot tell those apart, the drop is unattributable by construction.

Plumb the target's verdict back onto the solve record, so every solve carries one outcome label:

def outcome(status, request, accepted, elapsed_s, deadline_s):
    """status/request come from res.php with json=1.
    `accepted` is fed back from the downstream request that used the token."""
    if status == 1:                          # an artifact came back
        return "accepted" if accepted else "rejected_by_target"
    if request == "CAPCHA_NOT_READY":
        return "deadline_expired" if elapsed_s >= deadline_s else "pending"
    return f"solver_error:{request}"         # ERROR_UNSOLVABLE, ERROR_ZERO_BALANCE, ...

Tag every record with target, route, egress_pool and method. Those four labels are the entire diagnosis below; without them you have one scalar going down and no way to interrogate it.

The sixty-second checks

Two causes produce a clean overnight cliff and neither is a detection change.

ERROR_ZERO_BALANCE. It is terminal for every task, not just the one that saw it, and it turns a healthy pipeline into a wall of failures at a precise timestamp. A classifier that lumps solver errors together reads this as a mysterious total failure. It should page someone, never retry.

Your own throughput ceiling. A thread is one in-flight task. If your workload grew or a target got slower, tasks queue behind your semaphore and start hitting your client-side deadline. The signature is a spike in deadline_expired with solver_error and acceptance both flat. That is congestion, not detection, and no proxy tuning will move it.

Then check whether you really shipped nothing. A proxy pool that rotated on the vendor's side, a base image rebuild that bumped your TLS library, an expired allow-list entry — none appear in your git log, and all look exactly like the target changing.

Three ways the target changes

Each has a distinct fingerprint across your labels.

A new challenge type

The site swapped or added a challenge: a rotated site key, Turnstile placed in front of a form that had reCAPTCHA v2, v2 migrated to v3, an interstitial added at the edge.

Failures move upstream, to submission and parsing rather than acceptance. Your scraper cannot find the widget, or in.php accepts a task whose parameters no longer describe the page and ERROR_UNSOLVABLE climbs. Solve latency often falls while success collapses, because failing is faster than succeeding.

Don't infer this one. Fetch the challenge page and diff it against last week's copy — a changed site key, or a cf-turnstile div that was not there before, is direct evidence and takes two minutes.

A tightened score threshold

reCAPTCHA v3 returns a score to the site's backend and takes no action itself; the site chooses what counts as a pass. Someone moving that cut-off from 0.3 to 0.5 changes nothing you can see from outside, and everything about your outcomes.

The fingerprint: solver-side success stays exactly where it was, rejected_by_target absorbs the whole loss, latency is unchanged, and the drop is uniform across every egress pool. This is the one people misfile as a solver regression, because the solver's own numbers really are fine.

Fresh IP reputation weighting

The target started weighting network origin harder, or your ranges aged into a worse classification. Same surface symptom as a threshold move — tokens come back, the site says no — with one critical difference.

Dispersion. Reputation is a property of addresses, so its effect is uneven. Pool A holds at 96%, pool B at 94%, pool C falls to 41%. That spread is the discriminator, and it is why egress_pool belongs on every metric you emit. A threshold change lands on all pools equally; a reputation change almost never does. It also tends to track your own concurrency peaks, since session density per address is itself a scored signal.

The mechanics underneath all three — how origin, fingerprint and behaviour become one risk score, and why the challenge you get is downstream policy — are covered in how anti-bot detection actually decides you're a bot.

Check the denominator before you trust the drop

One failure mode produces a real drop with no cause at all. If you added routes or targets this week, your aggregate is a weighted average over a mix that changed: every route can hold steady while the total falls, purely because a harder target now carries more of the traffic. Group by target and route and confirm at least one group actually moved — if none did, your pipeline is fine and your chart is lying. Count retries as retries, too, or one failing route inflates the denominator out of proportion to what broke.

Run the experiment instead of arguing about it

Uniform-versus-dispersed is a measurement, so measure it. Hold everything constant except egress and submit a controlled batch through each pool, against a target you are permitted to automate and well inside its published rate limits:

curl -s -X POST "https://ocr.captchaai.com/in.php" \
  -d "key=YOUR_API_KEY" \
  -d "method=userrecaptcha" \
  -d "googlekey=6Lc_SITE_KEY_FROM_THE_PAGE" \
  -d "pageurl=https://example.com/status" \
  -d "version=v3" -d "action=verify" -d "min_score=0.3" \
  -d "proxy=user:pass@POOL_A_HOST:8080" -d "proxytype=HTTP" \
  -d "json=1"
# {"status":1,"request":"2122988149"}

# Poll roughly every 5s until CAPCHA_NOT_READY stops coming back.
curl -s "https://ocr.captchaai.com/res.php?key=YOUR_API_KEY&action=get&id=2122988149&json=1"
# {"status":0,"request":"CAPCHA_NOT_READY"}
# {"status":1,"request":"03AGdBq26..."}

Fifty tasks per pool usually separates 95% from 45%. Score the batches on rejected_by_target, not on whether a token came back — both pools will return tokens.

Two parameters decide whether the test is valid. action must match the string the site's own JavaScript passes; without it, a v3 task returns a token the site scores badly, which looks exactly like a threshold change you did not cause. And per-task proxying matters because score-based challenges fold network origin into the score — testing reputation means solving from the network you will submit from.

What each diagnosis actually fixes

• New challenge type — update method and its parameters to the type actually on the page. A Cloudflare Challenge is the special case: the response carries a clearance cookie and a user agent, and later requests must use that exact user agent, because the pairing is what makes the cookie valid.
• Tighter threshold — verify action and min_score, then improve the context the token is earned in: correct page URL, per-task proxy on the egress you will submit from, a real browser engine where the target expects one. You cannot lower someone else's threshold; you can stop handing it easy reasons to fire.
• Reputation — move the affected pool, reduce session density per address, and re-run the A/B weekly rather than after every incident. Reputation drifts, so this is a standing measurement.
• Congestion — raise concurrency, or lower it and accept a queue. The ceiling is roughly threads divided by average solve time, and mix matters: CaptchaAI publishes >99.5% in under 60 seconds for reCAPTCHA v2 and >99% in under 4 seconds for v3, so identical thread counts buy very different throughput.

Because pricing is thread-based — no per-CAPTCHA fee, no daily cap, no surcharge by type — a diagnostic batch costs nothing beyond the concurrency you already bought: run the experiment before opening a support ticket.

Keep the target legitimate

All of this assumes you are permitted to be there: your own staging environment, a QA suite you own, synthetic monitoring, accessibility testing, or data collection within a site's terms, robots.txt and published rate limits. A diagnostic batch is still traffic — run it inside your rate limiter, not around it. And a sudden drop is sometimes the target withdrawing consent: if the cause is a new terms clause or a robots.txt change, the fix is to stop, not to tune.

FAQ

My provider says their success rate is normal. Are they lying? Probably not. They report stage one, an artifact came back; your drop is likely stage two, the target rejecting a valid token. Both are true at once. Ask for their success rate by challenge type and compare it with your own rejected_by_target count.

How do I tell a threshold change from an IP reputation change? Dispersion across egress pools. A threshold move hits every pool equally; reputation hits some far harder than others. Fifty controlled tasks per pool separates them in under an hour.

Why did solve latency drop at the same time as success? Because failing is fast. A rising share of ERROR_UNSOLVABLE pulls the average down. Falling latency alongside falling success usually means a parameter or challenge-type mismatch, not a slow provider.

Should I retry harder when the rate falls? No. Only CAPCHA_NOT_READY means keep waiting. ERROR_UNSOLVABLE is terminal for that task — resubmit at most once — and ERROR_ZERO_BALANCE is terminal for all of them. Retrying into a threshold change multiplies load against a decision that will not change.

How long before I conclude anything? One full traffic cycle, typically a day, unless the drop is a clean step at a timestamp. Steps are configuration; ramps are reputation or volume. Alert on the derivative rather than the absolute value and you get that distinction for free.

Where to go from here

Add the four labels, split acceptance from solver success, and the next drop explains itself in minutes instead of days. The shape of the failure across egress and time is the whole diagnosis.

What remains after that is a legitimate target that challenges your server-side traffic no matter how clean it is, and that is the part a solver answers. The documented parameters for each method, the error codes and the per-type latency figures are at getcaptchaai.com — size threads against your real peak concurrency, and keep the per-pool acceptance panel running.