CAPTCHA (in web scraping)
A challenge-response test sites use to distinguish human users from automated bots - frequently triggered by scraping activity.
CAPTCHA stands for Completely Automated Public Turing test to tell Computers and Humans Apart. In practice, it is a gate that sites place in front of users they suspect are bots - requiring image recognition, puzzle solving, or checkbox interaction.
CAPTCHAs trigger during scraping for several reasons: too many requests from one IP, datacenter IP address ranges, missing browser headers, inconsistent TLS fingerprints, and unusual request timing patterns.
The most effective way to reduce CAPTCHA rates is using residential or mobile proxies instead of datacenter IPs. The ASN type of the IP is one of the first signals CAPTCHA systems check. Residential IPs from real ISPs trigger CAPTCHAs far less often than datacenter IPs.
When CAPTCHAs do appear, there are two main approaches: CAPTCHA solving services (2Captcha, Anti-Captcha) that use humans or ML models to solve them programmatically, and behavior-based avoidance that mimics human browsing patterns closely enough to avoid triggering the challenge.
Google's reCAPTCHA v3 scores requests invisibly without showing a challenge - it assigns a risk score (0-1) based on behavior. A score below 0.5 may trigger additional checks. Residential proxies combined with realistic browser fingerprints keep reCAPTCHA v3 scores high.
A challenge is a score, not a verdict
A CAPTCHA appears when a site is uncertain. Systems like reCAPTCHA and Turnstile score every visit on address reputation, TLS fingerprint, browser consistency and behaviour, then challenge the requests that land in the middle band. Confident blocks return 403 instead.
That makes a CAPTCHA useful information: you are close to passing. Something in your setup is off, but not so far off that the site refuses outright.
Modern challenges are frequently invisible. A score high enough passes without any interaction, which is why the same page shows a puzzle to one client and nothing to another. Solving services address the visible ones and do nothing about the score that produced them.
Moving the score instead of solving the puzzle
Most CAPTCHA problems in scraping come from one of three places, and only one of them is the address:
What to change first
Datacenter address on a protected site -> switch to residential or ISP
Python TLS fingerprint -> use a real browser or curl-impersonate
Too many requests too fast -> slow down, spread across addresses
# Verify the address is not the problem:
curl -x login_c_US:password@proxy.sotaproxy.com:10000 https://ipinfo.io/json- Test with a real browser through the same address. If the browser passes and your script does not, the address was never the issue.
- Residential and mobile addresses score better than hosting ranges, which is often enough to move a borderline case under the threshold.
- Solving services are a cost per challenge and do nothing to stop the next one. Fixing the score reduces how often you meet them.
- Persistent profiles help: an account with history and cookies scores better than a fresh visitor every time.
CAPTCHA misconceptions
A CAPTCHA is not a ban
It is an uncertain score. The same setup often passes after a small change in pacing or fingerprint.
Better proxies do not always help
If your client announces itself as a script in the handshake, the address cannot rescue it.
Solving them is not a strategy
It treats the symptom at a per-challenge price and leaves the cause untouched.
Invisible challenges exist
Not seeing a puzzle does not mean you were not scored. Silent failures often mean you scored badly and got reduced content.
Related terms
See this in practice
Ready to use captcha (in web scraping)?
SotaProxy gives you access to rotating residential, mobile, datacenter, and ISP proxies. No minimum commitment.
Get started