Scrapy Proxy Setup
Add rotating proxies to your Scrapy spider in two config lines.
Scrapy is the most popular Python scraping framework. SotaProxy integrates as a standard HTTP proxy through Scrapy's built-in proxy middleware. Set it in settings.py and every request your spider makes will route through the proxy automatically.
Setup guide
Enable proxy middleware in settings.py
# settings.py
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
}
HTTP_PROXY = "http://YOUR_USERNAME:YOUR_PASSWORD@proxy.sotaproxy.com:10000"Add these two settings to your Scrapy project's settings.py. All spider requests will route through SotaProxy automatically.
Per-request proxy (optional)
# In your spider
def start_requests(self):
for url in self.start_urls:
yield scrapy.Request(
url,
meta={'proxy': 'http://YOUR_USERNAME:YOUR_PASSWORD@proxy.sotaproxy.com:10000'},
)Set the proxy per-request in the meta dict if you need different proxy configurations for different URLs.
Geo-targeting in Scrapy
# Target US IPs for all requests
HTTP_PROXY = "http://YOUR_USERNAME_c_US:YOUR_PASSWORD@proxy.sotaproxy.com:10000"
# Or per-request targeting
meta={'proxy': 'http://YOUR_USERNAME_c_DE:YOUR_PASSWORD@proxy.sotaproxy.com:10000'}Append _c_XX to the username. Use per-request targeting when different URLs need different geos.
Automate it with the SotaProxy API
Everything above works without opening the dashboard. With an API key, Scrapy can pull fresh proxy credentials, buy new proxies, and renew expiring ones - straight from code.
import requests
API = "https://api.sotaproxy.com/api/v1"
H = {"Authorization": "Bearer sk_live_your_key"}
# Every active proxy on your account - no copy-pasting credentials
proxies = requests.get(f"{API}/proxies", headers=H).json()["proxies"]
p = proxies[0]
proxy_url = f"http://{p['login']}:{p['password']}@{p['ip']}:{p['portHttp']}"
# Buy more when you scale (price-check first with POST /quote)
requests.post(
f"{API}/orders",
headers={**H, "Idempotency-Key": "my-unique-order-id"},
json={"product": "ipv4", "countryId": 565, "periodId": "1m", "quantity": 5},
)Proxy credentials in a Scrapy project
Scrapy ships the middleware you need. Point request.meta at our endpoint and let HttpProxyMiddleware handle the rest:
Put the login and password in the proxy URL itself. Scrapy reads credentials from the URL and turns them into the Proxy-Authorization header for you.
One address per request, or one per spider
Set the proxy in meta and you control rotation per request without any third-party middleware:
Python: rotate inside start_requests
import itertools, scrapy
BASE = "proxy.sotaproxy.com:10000"
class ShopSpider(scrapy.Spider):
name = "shop"
custom_settings = {
"CONCURRENT_REQUESTS": 32,
"CONCURRENT_REQUESTS_PER_DOMAIN": 8,
"DOWNLOAD_TIMEOUT": 30,
"RETRY_TIMES": 2,
}
def start_requests(self):
sessions = itertools.cycle(range(1, 11))
for url in self.urls:
login = f"login_c_US_s_{next(sessions)}_ttl_15m"
yield scrapy.Request(
url,
meta={"proxy": f"http://{login}:password@{BASE}"},
headers={"Accept-Encoding": "gzip"},
)- Ten session ids give you ten stable addresses that survive redirects. Drop the _s_ part entirely and every request lands on a new one.
- Raise CONCURRENT_REQUESTS rather than hammering one address. We do not cap concurrent connections, target sites do.
- DOWNLOAD_TIMEOUT defaults to 180 seconds. On a residential route 30 is a better tradeoff between patience and throughput.
- AutoThrottle plays well with proxies. Leave it on for unfamiliar targets and it will find a rate that does not trip rate limits.
Scrapy details that cost you data
Retry middleware reuses the same proxy
A retried request carries the original meta, including the address that just failed. Set a fresh proxy in the retry or write a tiny middleware that does.
Cookies crossing sessions
Scrapy keeps one cookiejar by default. When addresses rotate under it, the site sees one session hopping between countries. Use cookiejar per session id.
robots.txt fetched without the proxy
ROBOTSTXT_OBEY sends its own request. It goes through the same middleware, but the response is cached per domain, so a blocked fetch silently stops the crawl.
Assuming meta persists across redirects
It does, which is the point, but it means a redirect chain stays on one address. That is what you want for logins and not what you want for wide crawls.
Notes & tips
- •Set CONCURRENT_REQUESTS to control parallelism. High concurrency from a single proxy endpoint is fine - rotation is server-side.
- •Enable RETRY_ENABLED = True and RETRY_HTTP_CODES = [403, 429, 503] to automatically retry on blocks.
- •For per-request random sessions, generate a unique session ID per request and append it to the username.
FAQ
Does Scrapy rotate IPs automatically with SotaProxy?
Yes. With rotating residential proxies, each request through the same endpoint gets a different IP. You do not need a proxy rotation middleware - rotation is server-side.
How do I handle 403 blocks in Scrapy?
Add 403 to RETRY_HTTP_CODES. For sites that block on user agent, rotate user agents using the scrapy-fake-useragent middleware.
Related integrations
Get your proxy credentials
Sign up, top up your balance, and copy your endpoint into Scrapy. Takes under 5 minutes.
Get started