URL Canonicalization
Reducing the many addresses that lead to one page down to a single form, so a dataset counts pages rather than links.
The same page is reachable through a surprising number of addresses. A tracking parameter, a trailing slash, an uppercase letter in the path, a session id, a sort order that changes nothing visible: each produces a distinct string, and a crawler that keys on the string collects the same page repeatedly.
Sites publish the answer themselves. A canonical link element in the head names the address the owner considers authoritative, and it is the cheapest deduplication signal available. It costs nothing to read and it is right far more often than any rule you would write yourself.
Where it is missing, the work is to decide which parameters matter. Pagination and filters usually do; campaign tags, referrers and session identifiers usually do not. That decision belongs in the pipeline once, not in every consumer of the data afterwards.
The failure mode is quiet and expensive. Nothing errors, the crawl completes, and the dataset simply reports more listings, more products or more vacancies than exist. Every number computed from it is inflated by an amount nobody can estimate after the fact.
Where duplicates come from
Parameters are the largest source. Campaign tags, affiliate identifiers, internal analytics and sort orders all change the string without changing the page.
Path variations come next: a trailing slash, differing case, an index file spelled out, the same article under a category prefix and again at the root.
Protocol and host variations complete the set: http and https, with and without a leading www, plus country subdomains that serve identical content to the same visitor.
Reading the answer off the page
Check the canonical first, and only build rules where there is none:
What the page says about itself
# the address the site considers authoritative
curl -s -x login_c_US:password@proxy.sotaproxy.com:10000 https://target.example/product?utm_source=x \
| grep -o '<link[^>]*rel="canonical"[^>]*>'
# compare it with what you requested before storing anything- Store the canonical as the key and the requested address as a separate field. You need both: one to deduplicate, one to reproduce the fetch.
- A canonical pointing somewhere unexpected is information, not an error. Some sites canonicalise variants onto a parent deliberately.
- Deduplicate at intake. Doing it in reporting means every consumer of the table repeats the same work and reaches different answers.
Where it goes wrong
A canonical is a hint, not a rule
It expresses the owner’s intent. Some sites set it wrong, and a sanity check on a sample is worth the minute.
Not every parameter is noise
Page numbers and filters change the content. Stripping them merges pages that are genuinely different.
Deduplication is not a reporting step
Done late, it is done differently by every consumer, and the numbers stop agreeing with each other.
Redirects are not canonicalisation
A redirect moves you; a canonical tells you where the content belongs. A page can have one without the other.
Términos relacionados
Ver esto en práctica
¿Listo para usar url canonicalization?
SotaProxy te da acceso a proxies residenciales rotativos, móviles, de centro de datos e ISP. Sin compromiso mínimo.
Empezar