Website's internal requests

Firstly, you need to identify the internal requests. On the page you want to scrape, right-click and select Inspect (or Inspect Element). Then, open the Network tab and refresh the page. You should see a large number of requests appear.

Next, review these requests to determine where the data you need is being loaded from.

For example, Best Buy loads most of its product information through multiple GraphQL requests:

That’s why it can be challenging to load all the information on this page, as most APIs time out before returning all of the data.

You can also check where the information you need is actually stored. If you are unable to fully reach or load the page, you may be able to extract the relevant data by targeting an internal request instead. These requests usually have less security than the main page.

Let’s use G2 to see what information might be relevant for a request. Right-click on the request that contains the data you need and select Copy as cURL. You can then paste this request into HTTPie or Postman to inspect which headers or cookies are required.

This allows you to confirm whether any specific information needs to be added or removed for the request to work properly.

If you disable all headers and cookies the request fails:

If you enable the cookies and user-agent header then it works:

Now you can apply this information to your API requests!

Updated on: 01/01/2026