JavaScript Rendering
Executing a page’s scripts so the content they build exists. Without it many sites return a valid, successful and completely empty page.
A growing share of pages ship a shell and assemble the content in the browser. Fetch such a page with a plain HTTP client and you get status 200, a few kilobytes of markup and none of the data you came for.
This is the single most misdiagnosed situation in scraping. The response was successful, so nothing in the logs says block, yet the parser finds nothing and the natural conclusion is that the site refused. Changing address type at that point spends money on a problem that was never about addresses.
The test is quick: fetch the page and look at its size and at whether the values you want appear in the raw bytes. If the shell is there and the numbers are not, you need rendering rather than a different exit.
Rendering has a price, and on per-gigabyte billing it is the largest single multiplier in this business. A headless browser pulls scripts, fonts, images and trackers unless told otherwise, which is why blocking media at the request level is part of the setup rather than an optimisation for later.
Why the page arrives empty
Server-rendered pages contain their content in the HTML. Client-rendered ones contain a container and a script that fills it after a second round of requests to an internal API.
That second round is the interesting part. Often the data you want arrives as JSON from an endpoint the page calls, and hitting that endpoint directly is cheaper and steadier than rendering the page at all.
Rendering is required when the endpoint is signed, when the payload is assembled from several calls, or when the site checks that a real browser environment executed the code.
Deciding whether you need a browser
Three requests answer it, and two of the three outcomes save you a lot of traffic:
Empty page or refused page
# 1. how big is the response and does it contain the value
curl -s -x login_c_US:password@proxy.sotaproxy.com:10000 https://target.example/product | wc -c
curl -s -x login_c_US:password@proxy.sotaproxy.com:10000 https://target.example/product | grep -c "price"
# 2. is there an API behind the page: look for the call in the markup
curl -s -x login_c_US:password@proxy.sotaproxy.com:10000 https://target.example/product | grep -oE "/api/[a-z0-9/_-]+" | head
# 3. only if both fail, render, with media blocked at the request level- An internal API returning JSON is the best outcome: no browser, a fraction of the weight and a stable shape to parse.
- If you must render, block images, fonts, video and analytics. On residential billing this is the difference between tens and hundreds of gigabytes.
- Rendering makes each page slower as well as heavier, so concurrency planning changes with it, not just the invoice.
What an empty 200 means
It is usually not a block
A block arrives as 403, as a challenge page or as a redirect. An empty shell with a 200 is a page that has not been built yet.
A better address will not fill it
Residential returns the same empty shell as datacenter, because nothing was refused.
Rendering is not always required
Most client-rendered sites call an API you can call yourself, which is cheaper in both traffic and time.
Rendering is not free
It is the largest multiplier on a per-gigabyte bill, and it is invisible until the first month closes.
Пов'язані терміни
Дивись на практиці
Готовий використовувати javascript rendering?
SotaProxy надає доступ до ротуючих резидентських, мобільних, дата-центр та ISP проксі. Без мінімальних платежів.
Почати