Скрапте будь-який сайт. Без блокувань і CAPTCHA.

What a million pages actually costs

Proxy choice is an arithmetic problem before it is a technical one. Here is the same job priced both ways, using our list prices and a page that weighs 80 KB as gzipped HTML:

Datacenter wins on price whenever the target lets it through, and the gap grows with volume. Residential earns its price on sites that reject hosting ranges outright, and on those the choice is not between $27 and $76, it is between $76 and no data at all.

A configuration that survives contact with a real site

Start on datacenter, measure your block rate, and move only the targets that fail. Rotating residential needs no session id at all, which is what makes it cheap to run:

Python: datacenter pool with a residential fallback

import requests
from itertools import cycle

# Datacenter pool: rotate yourself
pool = cycle([
    "http://login:password@198.51.100.20:50100",
    "http://login:password@198.51.100.21:50100",
])

# Residential fallback: no session id means a new IP per request
RESI = "http://login_c_US:password@proxy.sotaproxy.com:10000"

headers = {"Accept-Encoding": "gzip"}

def fetch(url, use_resi=False):
    proxy = RESI if use_resi else next(pool)
    r = requests.get(url, proxies={"http": proxy, "https": proxy},
                     headers=headers, timeout=30)
    if r.status_code == 403 and not use_resi:
        return fetch(url, use_resi=True)
    return r

• Ask for gzip. It cuts an HTML page three to four times, and on residential that is the difference between $76 and $250 for the same run.
• Block images, fonts and video in your headless browser. They carry most of the weight and none of the data.
• Thirty second timeout. A residential route takes longer to build than a datacenter one, and a five second timeout throws away requests that would have succeeded.
• Spread load across addresses instead of speeding up one. One IP behaving like fifty users is the exact pattern rate limiters catch.

Four ways people burn money here