robots.txt
A file at the root of a site describing what its owner would prefer automated clients not to fetch. It is a convention, not an access control.
The file lists user agents and paths, and asks crawlers to stay out of some of them. It is honoured voluntarily. Nothing in a web server enforces it, which is why it is a statement of preference rather than a permission system.
That makes it neither meaningless nor binding on its own. Search engines follow it because it is in their interest. A commercial scraper is in a different position: ignoring it carries no technical penalty, and it may still matter in a dispute about terms of service, which is a legal question rather than a technical one.
Two directives carry practical weight for collection. Crawl-delay, where present, tells you the pace the site considers acceptable, and honouring it is usually cheaper than discovering the same number through rate limits. Sitemap entries point at the structured index of the site, which is often the fastest way to enumerate what exists.
Our position is the same as everywhere else on this site: we sell addresses and we do not tell you to ignore a site’s stated preferences. Reading the file before a crawl costs one request and frequently saves an argument.
What the file can and cannot say
The format is a list of blocks, each naming a user agent and then allowing or disallowing paths. Matching is by prefix, and the most specific block for your agent is the one that applies.
Crawl-delay is not part of the original standard and is interpreted differently by different crawlers, but where a site publishes it, it is the clearest statement you will get about acceptable pace.
Sitemap lines are the useful part almost nobody reads. They point at an XML index that lists URLs the owner wants discovered, which beats crawling a category tree page by page.
Reading it before you plan
One request, three things worth knowing before the first crawl:
What the site says about itself
# the file itself
curl -s -x login_c_US:password@proxy.sotaproxy.com:10000 https://target.example/robots.txt
# the pace it suggests, if any
curl -s -x login_c_US:password@proxy.sotaproxy.com:10000 https://target.example/robots.txt | grep -i "crawl-delay"
# the structured index, which usually saves a lot of crawling
curl -s -x login_c_US:password@proxy.sotaproxy.com:10000 https://target.example/robots.txt | grep -i "sitemap"- A published Crawl-delay is a free hint about the rate limit you are going to meet anyway.
- The sitemap is often the cheapest enumeration available and it is public by design.
- The file can differ per country on sites that localise everything, so read it through the exit you will collect from.
What it is not
It is not a security control
A path listed as disallowed is still publicly reachable. Listing it tells the world it exists.
It is not the terms of service
Terms are a separate document with separate consequences. The two often disagree.
It is not a law
It is a convention. Whether ignoring it matters legally depends on jurisdiction, on the data and on what you do next, which is a question for a lawyer.
It is not a promise about defences
A permissive robots.txt says nothing about whether the site will rate limit you or refuse your address class.
Дивись на практиці
Готовий використовувати robots.txt?
SotaProxy надає доступ до ротуючих резидентських, мобільних, дата-центр та ISP проксі. Без мінімальних платежів.
Почати