The webpage I'm trying to scrape haven't rendered completely

If you don't use JavaScript rendering:

If you're not using JavaScript rendering (render_js=False) and your page doesn't render completely, chances are that you are scraping a webpage that needs javascript rendering to work correctly.

Some websites need to be rendered inside a real browser to work and load all information correctly. To enable this using our API you need to use render_js=True on your API call (documentation)

If you are already using JavaScript rendering:

If you are already using JavaScript rendering and some parts of the page are not loading completely here is what you can try:

Disabling block_ressources

By default, and to speed up requests, ScrapingBee blocks all images and CSS in the scraped page. Sometimes, blocking CSS and images can break how the website works.
To disable this behavior use: block_resources=false.

Use wait_for and wait

Sometimes, we will return you the HTML of the page you want to scrape before some elements are rendered.

In this case, you should use the wait_for parameter. It will make us wait for a particular CSS selector to be present in the page before returning results.

So, for example, if you are scraping an e-commerce page and want to be sure that the div with id="price" is present in the page pass wait_for=#price to your API call.

If you don't know what CSS selector to use, you can always pass a fixed duration to wait for using the wait parameter. The ScrapingBee headless browsers will then wait the duration of the time set in milliseconds before returning the page's HTML.

You can read more about it in the full documentation.

Updated on: 21/09/2021