The webpage I'm trying to scrape haven't rendered completely
Some websites need to be rendered inside a real browser to work and load all information correctly. To enable this using our API you need to use render_js=True on your API call (documentation)
By default, and to speed up requests, ScrapingBee blocks all images and CSS in the scraped page. Sometimes, blocking CSS and images can break how the website works.
To disable this behavior use: block_resources=false.
Use wait_for and wait
Sometimes, we will return you the HTML of the page you want to scrape before some elements are rendered.
In this case, you should use the wait_for parameter. It will make us wait for a particular CSS selector to be present in the page before returning results.
So, for example, if you are scraping an e-commerce page and want to be sure that the div with id="price" is present in the page pass wait_for=#price to your API call.
If you don't know what CSS selector to use, you can always pass a fixed duration to wait for using the wait parameter. The ScrapingBee headless browsers will then wait the duration of the time set in milliseconds before returning the page's HTML.
You can read more about it in the full documentation.
Updated on: 21/09/2021