Gather data from any market. No restrictions.

Twenty markets on a monthly budget

Research pulls a wide but shallow sample: many countries, moderate volume in each. Content pages sit around 80 KB gzipped:

Residential is the practical answer because coverage matters more than cost here. Twenty markets from one login beats twenty purchases, and the countries where static stock does not exist are often exactly the ones a research brief asks about.

Country in the login, country in the record

The rule that saves the dataset: whatever you fetch, store the market it came from next to it. Rotate freely, but never lose the provenance:

Python: fetch and keep provenance

def fetch_market(url, country):
    login = f"login_c_{country}"
    proxy = f"http://{login}:password@proxy.sotaproxy.com:10000"
    r = requests.get(url, proxies={"http": proxy, "https": proxy},
                     headers={"Accept-Language": LOCALES[country]}, timeout=30)
    return {
        "market": country,
        "fetched_at": datetime.utcnow().isoformat(),
        "status": r.status_code,
        "html": r.text,
    }

• Send the locale header that a real visitor from that country would send. Half of what makes a page local is the header, not the address.
• Sample the same weekdays and hours across markets. Comparing a Monday in Brazil against a Saturday in Japan produces a difference that has nothing to do with the market.
• Where the brief is city-scoped, use _city_. Retail assortment and pricing in a capital rarely represent the country.
• Keep the raw HTML, not just your parsed fields. Six weeks later you will want a field you did not think to extract.

How a research dataset quietly becomes useless