URL encoding

When you are making an API call to ScrapingBee, you can pass different parameters as query strings. Here is an example:

https://app.scrapingbee.com/api/v1/?url=https://example.com/&premium_proxies=True&api_key=YOUR-API-KEY

Now what happens when the URL you want to scrape also contains a query parameter named param1?

curl 'https://app.scrapingbee.com/api/v1/?url=https://example.com?param1=value1&param2=value2&premium_proxy=True&api_key=YOUR-API-KEY'

In this case, the API will return an error, because it will parse these param1 and param2, and since ScrapingBee doesn't use these parameters you will get this error:

{"message": "Unknown arguments: param2"}

In order to pass your URL including the query parameters, you have to encode it. Most HTTP clients in most programming languages will do it for you.

If your client doesn't (like cURL ...) you can use a third-party tool to encode the URL like this one.

There are also many ways to encode a URL directly in your favorite language:

Curl

sudo apt-get install gridsite-clients
urlencode "YOUR URL"

Python

import urllib.parse
encoded_url = urllib.parse.quote("YOUR URL")

NodeJS

encoded_url = encodeURIComponent("YOUR URL")

Java

String encoded_url = URLEncoder.encode("YOUR URL", "UTF-8");

Ruby

require 'uri'
encoded_url = URI::encode("YOUR URL")

Php

$url_encoded = urlencode("YOUR URL");

Go

package main

import (
	"net/url"
)

func main() {
	encoded_url := url.QueryEscape("YOUR URL")
}

Updated on: 19/01/2022