Pull real fares. Not inflated ones.

Hourly fare collection across markets

Fares change through the day and differ by point of sale, so this job is defined by frequency rather than page count:

The cost of doing this properly is trivial. The cost of doing it from a single geography is a fare table that does not match what customers are quoted, which is the only failure mode that matters in this business.

One session per point of sale

Airline and OTA sessions bind currency, residency and sometimes the fare bucket. Keep one address per market for the whole sweep:

Python: a session per point of sale

def pos_proxy(pos_index, country):
    login = f"login_c_{country}_s_{pos_index}_ttl_15m"
    return f"http://{login}:password@proxy.sotaproxy.com:10000"

for i, pos in enumerate(points_of_sale, start=1):
    proxy = pos_proxy(i, pos.country)
    for route in routes:
        fare = search(route, proxy=proxy, currency=pos.currency)
        store(fare, market=pos.country, address_session=i)

• Set the currency explicitly and record it with every fare. A number without a currency is not data.
• Fifteen minutes of sticky covers a search plus the follow-up calls that price the itinerary. Rotating between them resets the search.
• Airlines rate-limit aggressively. Spread the hourly sweep across the hour instead of firing all routes at the top of it.
• Store the session id alongside the fare so you can reproduce a quote when someone disputes it.

Where fare data goes wrong