Slow responses and timeouts
Most timeout reports turn out to be a client configured for datacenter latency talking to a residential route.
Set the timeout to thirty seconds
Building a route through a household connection takes longer than reaching a server in a datacenter. A five second timeout discards requests that would have completed, and the run looks unreliable when it was only impatient.
The first request of a sticky session is the slowest, because that is when the route is established. Subsequent requests through the same session are faster.
Timeouts that work
requests.get(url, proxies=proxies, timeout=30)
page.goto(url, timeout=30000) # Playwright, milliseconds
DOWNLOAD_TIMEOUT = 30 # ScrapyTelling the layers apart
If a static address is slow, the route is short and the destination is the likely cause. If a residential exit is slow, the exit device may simply be on a poor connection, and a new session will pick a different one.
A useful check is to fetch a lightweight endpoint through the same session. If that returns quickly and your target does not, the proxy is fine.
Separating proxy from target
# fast? then the route is healthy
curl -w '%{time_total}\n' -o /dev/null -s \
-x login_c_US_s_5_ttl_15m:password@proxy.sotaproxy.com:10000 \
https://api.ipify.orgThroughput is not latency
Raising concurrency does not make individual requests faster, and past the destination tolerance it makes everything slower by producing rate limits. Spread work across more addresses or more sessions instead of more threads on one.
Reusing connections helps more than most tuning: a pooled client removes a handshake from every request.
A budget that works in practice
Timeout values are usually copied from a datacenter setup and never revisited. These are the numbers that hold up against residential routes without leaving workers stuck for minutes.
Sensible defaults
Connect timeout 10 seconds
Read timeout 30 seconds
Total per request 30 seconds
Retries 2, each on a different address
Backoff 1s, 3s, then give up on that target for a while
# Static addresses tolerate tighter values: a 10 second total
# is reasonable there and will surface real problems faster.