Налаштування проксі у Scrapy

Proxy credentials in a Scrapy project

Scrapy ships the middleware you need. Point request.meta at our endpoint and let HttpProxyMiddleware handle the rest:

Put the login and password in the proxy URL itself. Scrapy reads credentials from the URL and turns them into the Proxy-Authorization header for you.

One address per request, or one per spider

Set the proxy in meta and you control rotation per request without any third-party middleware:

Python: rotate inside start_requests

import itertools, scrapy

BASE = "proxy.sotaproxy.com:10000"

class ShopSpider(scrapy.Spider):
    name = "shop"
    custom_settings = {
        "CONCURRENT_REQUESTS": 32,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 8,
        "DOWNLOAD_TIMEOUT": 30,
        "RETRY_TIMES": 2,
    }

    def start_requests(self):
        sessions = itertools.cycle(range(1, 11))
        for url in self.urls:
            login = f"login_c_US_s_{next(sessions)}_ttl_15m"
            yield scrapy.Request(
                url,
                meta={"proxy": f"http://{login}:password@{BASE}"},
                headers={"Accept-Encoding": "gzip"},
            )

• Ten session ids give you ten stable addresses that survive redirects. Drop the _s_ part entirely and every request lands on a new one.
• Raise CONCURRENT_REQUESTS rather than hammering one address. We do not cap concurrent connections, target sites do.
• DOWNLOAD_TIMEOUT defaults to 180 seconds. On a residential route 30 is a better tradeoff between patience and throughput.
• AutoThrottle plays well with proxies. Leave it on for unfamiliar targets and it will find a rate that does not trip rate limits.

Scrapy details that cost you data