Question 1

Getting started

Accepted Answer

ScrapingBee is a really simple API that allows you to extract HTML from every website in one single API call

If you need more to use our custom options such as JS rendering or our Premium Proxy take a look at our full documentation.

To get your API Key, you just need to create an account here.

Of course, don't forget to replace "YOUR_URL" by the URL of the page you want to scrape.

| Curl

curl "https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR-URL"

| Python

# Install the Python ScrapingBee library:
# pip install scrapingbee

from scrapingbee import ScrapingBeeClient

client = ScrapingBeeClient(api_key='YOUR-API-KEY')
response = client.get('YOUR-URL')

print('Response HTTP Status Code: ', response.status_code)
print('Response HTTP Response Body: ', response.content)

| NodeJS

// request Axios
const axios = require('axios');

axios.get('https://app.scrapingbee.com/api/v1', {
    params: {
        'api_key': 'YOUR-API-KEY',
        'url': 'YOUR-URL',
    }
}).then(function (response) {
    // handle success
    console.log(response);
})

| Java ``` import java.io.IOException; import org.apache.http.client.fluent.*;

public class SendRequest { public static void main(String[] args) { sendRequest(); }

private static void sendRequest() {

// Classic (GET )

try {

  // Create request
  Content content = Request.Get("https:/

Question 2

URL encoding

Accepted Answer

When you are making an API call to ScrapingBee, you can pass different parameters as query strings. Here is an example:

https://app.scrapingbee.com/api/v1/?url=https://example.com/&premium_proxies=True&api_key=YOUR-API-KEY

Now what happens when the URL you want to scrape also contains a query parameter named param1?

curl 'https://app.scrapingbee.com/api/v1/?url=https://example.com?param1=value1&param2=value2&premium_proxy=True&api_key=YOUR-API-KEY'

In this case, the API will return an error, because it will parse these param1 and param2, and since ScrapingBee doesn't use these parameters you will get this error:

{"message": "Unknown arguments: param2"}

In order to pass your URL including the query parameters, you have to encode it. Most HTTP clients in most programming languages will do it for you.

If your client doesn't (like cURL …) you can use a third-party tool to encode the URL like this one.

There are also many ways to encode a URL directly in your favorite language:

| Curl

sudo apt-get install gridsite-clients
urlencode "YOUR URL"

| Python

import urllib.parse
encoded_url = urllib.parse.quote("YOUR URL")

| NodeJS

encoded_url = encodeURIComponent("YOUR URL")

| Java

String encoded_url = URLEncoder.encode("YOUR URL", "UTF-8");

| Ruby

require 'uri'
encoded_url = URI::encode("YOUR URL")

| Php

$url_encoded = urlencode("YOUR URL");

| Go ``` pa

Question 3

What to do if my request fails?

Accepted Answer

Please find below the most commons ways to fix your API call.

Check that your URL is correctly encoded.

Most of the errors you will encounter will be because you haven't correctly encoded your URL.

To do this quickly you can go on this website, and click on the encode button.

If you need to do this programmatically, learn here how to do it.

Disable `block_resources`.

To speed up your request, we block the downloading of images and css by default. Sometimes this break some websites. You can disable this behaviour by using block_resources=False

This can only change something when using render_js=True.

Wait for the JavaScript to render

Some websites use a lot of JavaScript, for those websites, you will have to use the wait parameter with the number of milliseconds you want to wait. (documentation)

Example: wait=2000 if you want to wait 2 seconds or 2000 milliseconds

Use premium proxy

For very difficult websites, you should use the premium_proxy=True parameter. Every call using premium proxies will cost 25 API credits if you use JavaScript rendering, or 10 API credits if JavaScript rendering is disabled. You can read more about this parameter here: (Premium Proxy - Documentation)

Use stealth proxy

For some websites, even premium proxy wo

Question 4

The webpage I'm trying to scrape haven't rendered completely

Accepted Answer

If you don't use JavaScript rendering:

If you're not using JavaScript rendering (render_js=False) and your page doesn't render completely, chances are that you are scraping a webpage that needs javascript rendering to work correctly.

Some websites need to be rendered inside a real browser to work and load all information correctly. To enable this using our API you need to use render_js=True on your API call (documentation)

If you are already using JavaScript rendering:

If you are already using JavaScript rendering and some parts of the page are not loading completely here is what you can try:

Disabling block_ressources

By default, and to speed up requests, ScrapingBee blocks all images and CSS in the scraped page. Sometimes, blocking CSS and images can break how the website works. To disable this behavior use: block_resources=false.

Use wait_for and wait

Sometimes, we will return you the HTML of the page you want to scrape before some elements are rendered.

In this case, you should use the wait_for parameter. It will make us wait for a particular CSS selector to be present in the page before returning results.

So, for example, if you are scraping an e-commerce page and want to be sure that the div with id="price" is present in the page pass wait_for=#price to your API call.

If you don't know what CSS selector to use, you can always pass a fixed duration to wait fo

Question 5

How to use concurrency?

Accepted Answer

According to the plan you chose, you will have access to a specific number of concurrent request. This means that you'll be able to only do a specific number of request at the same time.

For example, if you need to make 100 requests and have an allowed concurrency of 5, it means that you can send 5 requests at the same time. The simplest way for you to take advantage of this concurrency is to set up 5 workers / threads and having each of them send 20 requests.

Below you'll find some resources that can help you doing that.

Python

import requests
from multiprocessing.dummy import Pool as ThreadPool

def request_scrapingbee(url):
    r = requests.get(
      url="https://app.scrapingbee.com/api/v1/",
      params={
        "api_key": "",
        "url": url,
      },
    )
    response = {
        "statusCode": r.status_code,
        "body": r.text,
        "url": url,
    }
    return response

concurrency = 2
pool = ThreadPool(concurrency)

urls = ["", ""]
results = pool.map(request_scrapingbee, urls)
pool.close()
pool.join()

for result in results:
  print(result)

Question 6

How can I bypass Google’s cookie consent page?

Accepted Answer

According to Google's EU User Consent Policy, the company must make certain disclosures to users in the European Economic Area (EEA) and the UK, and obtain their consent for the use of cookies or other local storage where legally required.

And this is why you get that popup agreement page whenever you try to scrape Google. So to fix this issue and bypass it, we need to accept Google's terms, and we have to approaches to do that:

1. Using a CONSENT Cookie:

Whenever you accept Google's terms, a cookie with the name CONSENT is saved on your computer, it usually contains a YES+ text with a timestamp, and the user's country code. So to mimic this behavior, we can visit the page we want to scrape with a cookie that contains a similar value:

from scrapingbee import ScrapingBeeClient
client = ScrapingBeeClient(api_key='API_KEY')

response = client.get(
'https://www.google.com/search?q=Best+Laptops+in+Europe&tbm=shop',
params={
'custom_google': 'true',
'premium_proxy': 'true',
'country_code':'gb',
'wait': '1500' # Waiting for the content to load (1.5 seconds)
},
cookies= {'CONSENT': 'YES+'} # Sending a cookie with the request
)

if response.ok:
    print(response.content)
    # Do whatever you want with the request!
else:
    print(response.content)

2. Using a JavaScript Scenario:

With this method, we mimic the exact behavior of a typical EU user: We click on the [I agree] button, and we continue our search!

``` from scrapingbee import ScrapingBeeClient cli

Frequently Read Articles

Browse All Categories