How to intercept XHR / Ajax requests?
Some web pages will load data dynamically using Ajax requests and not load them on the main webpage.
So if you are requesting those web pages using our API, you might not be able to find the data you're interested in in the returned HTML.
But you can use the response_json=True parameter (documentation).
If you use this parameter, your API requests will return a JSON response.
In that JSON response, you will be able to access all the XHR / Ajax requests sent by the browser while scraping the webpage.
Below you can find an example of such a response:
So if you are requesting those web pages using our API, you might not be able to find the data you're interested in in the returned HTML.
But you can use the response_json=True parameter (documentation).
If you use this parameter, your API requests will return a JSON response.
In that JSON response, you will be able to access all the XHR / Ajax requests sent by the browser while scraping the webpage.
Below you can find an example of such a response:
{
# Headers sent by the server
"headers": {
"Date": "Fri, 16 Apr 2021 15:03:54 GMT",
...
"Access-Control-Allow-Credentials": "true"
},
# Credit cost of your request
"cost": 1,
# Initial status code of the server
"initial-status-code": 200,
# Resolved URL (following redirection)
"resolved-url": "https://httpbin.org/",
# Type of the response "html" of "json"
"type": "html",
# Content of the answer
"body": "<html>... </body>"
# XHR / Ajax requests sent by the browser
"xhr": [
{
# URL
"url": "https://",
# status code of the server
"status_code": 200,
# Method of the request
"method": "POST",
# Headers of the XHR / Ajax request
"headers": {
"pragma": "no-cache",
...
},
# Response of the XHR / Ajax request
"body": "2d,x"
},
...
]
}Updated on: 15/09/2021
Thank you!