Articles on: API

URL encoding

When you are making an API call to ScrapingBee, you can pass different parameters as query strings. Here is an example:

https://app.scrapingbee.com/api/v1/?url=https://example.com/&premium_proxies=True&api_key=YOUR-API-KEY


Now what happens when the URL you want to scrape also contains a query parameter named param1?

curl 'https://app.scrapingbee.com/api/v1/?url=https://example.com?param1=value1&param2=value2&premium_proxy=True&api_key=YOUR-API-KEY'


In this case, the API will return an error, because it will parse these param1 and param2, and since ScrapingBee doesn't use these parameters you will get this error:

{"message": "Unknown arguments: param2"}


In order to pass your URL including the query parameters, you have to encode it. Most HTTP clients in most programming languages will do it for you.

If your client doesn't (like cURL ...) you can use a third-party tool to encode the URL like this one.

There are also many ways to encode a URL directly in your favorite language:

Curl
sudo apt-get install gridsite-clients
urlencode "YOUR URL"

Python
import urllib.parse
encoded_url = urllib.parse.quote("YOUR URL")

NodeJS
encoded_url = encodeURIComponent("YOUR URL")

Java
String encoded_url = URLEncoder.encode("YOUR URL", "UTF-8");

Ruby
require 'uri'
encoded_url = URI::encode("YOUR URL")

Php
$url_encoded = urlencode("YOUR URL");

Go
package main

import (
	"net/url"
)

func main() {
	encoded_url := url.QueryEscape("YOUR URL")
}

Updated on: 19/01/2022

Was this article helpful?

Share your feedback

Cancel

Thank you!