How can I bypass Google’s cookie consent page?

According to Google's EU User Consent Policy, the company must make certain disclosures to users in the European Economic Area (EEA) and the UK, and obtain their consent for the use of cookies or other local storage where legally required.

And this is why you get that popup agreement page whenever you try to scrape Google. So to fix this issue and bypass it, we need to accept Google's terms, and we have to approaches to do that:

1. Using a CONSENT Cookie:

Whenever you accept Google's terms, a cookie with the name CONSENT is saved on your computer, it usually contains a YES+ text with a timestamp, and the user's country code. So to mimic this behavior, we can visit the page we want to scrape with a cookie that contains a similar value:

from scrapingbee import ScrapingBeeClient
client = ScrapingBeeClient(api_key='API_KEY')

response = client.get(
'https://www.google.com/search?q=Best+Laptops+in+Europe&tbm=shop',
params={
'custom_google': 'true',
'premium_proxy': 'true',
'country_code':'gb',
'wait': '1500' # Waiting for the content to load (1.5 seconds)
},
cookies= {'CONSENT': 'YES+'} # Sending a cookie with the request
)

if response.ok:
    print(response.content)
	# Do whatever you want with the request!
else:
    print(response.content)

2. Using a JavaScript Scenario:

With this method, we mimic the exact behavior of a typical EU user: We click on the [I agree] button, and we continue our search!

from scrapingbee import ScrapingBeeClient
client = ScrapingBeeClient(api_key='API_KEY')

response = client.get(
'https://www.google.com/search?q=Best+Laptops+in+Europe&tbm=shop',
params={
'custom_google': 'true',
'premium_proxy': 'true',
'country_code':'gb',
'js_scenario': {"instructions": [{"click": "form > div > div > button"}]}, # Clicking the agreement button
'wait': '1500' # Waiting for the content to load (1.5 seconds)
}
)

if response.ok:
    print(response.content)
	# Do whatever you want with the request!
else:
    print(response.content)

Happy Google scraping!

Updated on: 04/03/2022