Scraping an E-commerce product page

Don't hesitate to take a look at our Python web scraping 101 to read a detailed introduction to these libraries. We also have a lot of tutorials in different languages in our web scraping blog.

We are going to extract the price, image URL, and product name for this product: https://clever-lichterman-044f16.netlify.app/products/taba-cream.1/



We are going to use some basic CSS selectors to extract the price, title, and image URL. Then we simply print the result into the console.

import requests
from bs4 import BeautifulSoup

# Extract product data from a (dummy) E-commerce product page
# https://www.scrapingbee.com/

# Replace with your ScrapingBee API
SCRAPINGBEE_API_KEY = ""
endpoint = "https://app.scrapingbee.com/api/v1"
rows = []

params = {
    'api_key': SCRAPINGBEE_API_KEY,
    'url': 'https://clever-lichterman-044f16.netlify.app/products/taba-cream.1/',
}

response = requests.get(endpoint, params=params)
if response.status_code != 200:
    print('Error with your request: ' + str(response.status_code))
    print(response.content)
else:
    #import pdb;pdb.set_trace()
    soup = BeautifulSoup(response.content, 'html.parser')
    product = {
        'price': soup.select('.my-4 s')[0].text,
        'image_url': soup.select('.slick-track img')[0]['src'],
        'product_title': soup.select('.col-12 h2')[0].text
    }
    print(product)

Updated on: 19/01/2022