### Perform Basic HTTP GET Request in Ruby

Source: https://crawlbase.com/docs/fr/smart-proxy/parameters

This Ruby snippet demonstrates a very basic HTTP GET request using an assumed `http` client. It prints the response code, message, and body. This example does not include proxy configuration and assumes `http`, `uri`, and `headers` are defined elsewhere.

```Ruby
res = http.get(uri.request_uri, headers)
puts res.code, res.message, res.body
```

--------------------------------

### Perform Basic HTTP Request and Print Response in Ruby

Source: https://crawlbase.com/docs/de/smart-proxy/parameters

This Ruby snippet demonstrates how to execute a simple HTTP GET request using an 'http' object and then print the response's status code, message, and body. It's a fundamental example of handling HTTP responses.

```Ruby
res = http.get(uri.request_uri, headers)
puts res.code, res.message, res.body
```

--------------------------------

### Configure Crawlbase Smart Proxy for Ruby GET Requests

Source: https://crawlbase.com/docs/smart-proxy/get

Illustrates the initial setup for making a GET request using the Crawlbase Smart Proxy in Ruby with the Typhoeus gem. This snippet includes the gem installation command and the definition of proxy URLs for both HTTPS (recommended) and HTTP connections. Note: The actual request execution code is truncated in the provided text.

```ruby
# Installation: gem install typhoeus
require 'typhoeus'

# HTTPS proxy (recommended)
proxy_url = "https://[email protected]:8013"
# HTTP proxy alternative:
```

--------------------------------

### Make GET Request to Crawlbase API

Source: https://crawlbase.com/docs/zh-cn/crawling-api/headless-browsers

This snippet demonstrates how to make a GET request to the Crawlbase API endpoint. It fetches content from a specified URL using a provided token. The examples cover various programming languages, showing how to construct the URL with parameters and retrieve the response body.

```curl
curl "https://api.crawlbase.com/?token=_JS_TOKEN_&url=https://www.missguidedau.com/petite-black-msgd-insert-mesh-cut-out-gym-leggings-10202499"
```

```Ruby
require 'net/http'

uri = URI('https://api.crawlbase.com')
uri.query = URI.encode_www_form({
  token: '_JS_TOKEN_',
  url: 'https://www.missguidedau.com/petite-black-msgd-insert-mesh-cut-out-gym-leggings-10202499'})

res = Net::HTTP.get_response(uri)
puts "Response HTTP Status Code: #{res.code}"
puts "Response HTTP Header Original Status: #{res['original_status']}"
puts "Response HTTP Response Body: #{res.body}"
```

```Node.js
const https = require('https');

const url = encodeURIComponent('https://www.missguidedau.com/petite-black-msgd-insert-mesh-cut-out-gym-leggings-10202499');
const options = {
  hostname: 'api.crawlbase.com',
  path: '/?token=_JS_TOKEN_&url=' + url
};

https.request(options, (response) => {
  let body = '';
  response.on('data', chunk => body += chunk).on('end', () => console.log(body));
}).end();
```

```PHP
$url = urlencode('https://www.missguidedau.com/petite-black-msgd-insert-mesh-cut-out-gym-leggings-10202499');

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://api.crawlbase.com/?token=_JS_TOKEN_&url=' . $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

$response = curl_exec($ch);
curl_close($ch);

var_dump($response);
```

```Python
import requests
from urllib.parse import quote_plus

url = quote_plus('https://github.com/crawlbase?tab=repositories')

response = requests.get(f'https://api.crawlbase.com/?token=_JS_TOKEN_&url={url}')

print(response.text)
```

```Go
package main

import (
  "net/http"
  "fmt"
  "io/ioutil"
  "net/url"
)

func main() {
  url := url.QueryEscape("https://www.missguidedau.com/petite-black-msgd-insert-mesh-cut-out-gym-leggings-10202499")
  resp, _ := http.Get("https://api.crawlbase.com/?token=_JS_TOKEN_&url=" + url)

  defer resp.Body.Close()
  body, _ := ioutil.ReadAll(resp.Body)

  fmt.Println("response Body: ", string(body))
}
```

--------------------------------

### Make HTTP Request via Crawlbase Smart Proxy (Go)

Source: https://crawlbase.com/docs/ru/smart-proxy/parameters

This Go example illustrates how to perform an HTTP GET request through the Crawlbase Smart Proxy. It sets up a custom HTTP client with a proxy configuration and disables SSL certificate verification. Custom 'CrawlbaseAPI-Parameters' are added to the request headers.

```Go
package main

import (
  "crypto/tls"
  "fmt"
  "io/ioutil"
  "net/http"
  "net/url"
)

func main() {
  // Create the proxy URL (use https:// and port 8013 for HTTPS)
  proxyStr := "http://[email protected]:8012"
  proxyURL, _ := url.Parse(proxyStr)
  transport := &http.Transport{
    Proxy: http.ProxyURL(proxyURL),
    TLSClientConfig: &tls.Config{InsecureSkipVerify: true}
  }

  client := &http.Client{
    Transport: transport
  }

  req, _ := http.NewRequest("GET", "http://httpbin.org/ip", nil)
  req.Header.Add("CrawlbaseAPI-Parameters", "country=US&get_headers=true")

  resp, err := client.Do(req)
  if err != nil {
    fmt.Println(err)
    return
  }
  defer resp.Body.Close()

  body, _ := ioutil.ReadAll(resp.Body)
  fmt.Println("response Body : ", string(body))
}
```

--------------------------------

### Make Proxied HTTP GET Request with Crawlbase Smart Proxy

Source: https://crawlbase.com/docs/fr/smart-proxy/parameters

These code examples illustrate how to send HTTP GET requests through the Crawlbase Smart Proxy. They include setting up proxy authentication using a user token, specifying the proxy host and port, and adding `CrawlbaseAPI-Parameters` headers for country targeting and header retrieval. Both HTTP and HTTPS proxy configurations are supported.

```JavaScript
const http = require('http');
const https = require('https');
const crawlbaseToken = '_USER_TOKEN_';
const auth = `Basic ${Buffer.from(crawlbaseToken).toString('base64')}`;

// HTTP request
const httpRequest = http.request({
    host: 'smartproxy.crawlbase.com',
    port: 8012,  // Use 8013 for HTTPS
    path: 'http://httpbin.org/ip',
    headers: {
      'Proxy-Authorization': auth,
      'CrawlbaseAPI-Parameters': 'country=US&get_headers=true',
    },
}, res => {
    res.setEncoding('utf8');
    res.on('data', console.log);
});

hTTPRequest.on('error', console.error);
hTTPRequest.end();
```

```PHP
<?php
$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, 'http://httpbin.org/ip');
curl_setopt($ch, CURLOPT_PROXY, "http://_USER_TOKEN_:@smartproxy.crawlbase.com:8012");  // Use https:// and port 8013 for HTTPS
curl_setopt($ch, CURLOPT_HTTPHEADER, array('CrawlbaseAPI-Parameters: country=US&get_headers=true'));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);

$response = curl_exec($ch);
if(curl_errno($ch)) {
    echo 'Error: ' . curl_error($ch);
}
curl_close($ch);
?>
```

```Python
# pip install requests
import requests
from urllib3.exceptions import InsecureRequestWarning

# Suppress only the single warning from urllib3 needed.
requests.packages.urllib3.disable_warnings(category=InsecureRequestWarning)

# Choose either HTTP or HTTPS endpoint
proxy_url = "http://_USER_TOKEN_:@smartproxy.crawlbase.com:8012"  # Use https:// and port 8013 for HTTPS
proxies = {
    "http": proxy_url,
    "https": proxy_url
}

try:
    response = requests.get(
        url="http://httpbin.org/ip",
        headers={"CrawlbaseAPI-Parameters": "country=US&get_headers=true"},
        proxies=proxies,
        verify=False,
        timeout=30
    )
    response.raise_for_status()  # Raise an exception for bad status codes

    print('Response Code:', response.status_code)
    print('Response Body:', response.text)
except requests.exceptions.RequestException as e:
    print(f"An error occurred: {e}")
```

```Go
package main

import (
  "crypto/tls"
  "fmt"
  "io/ioutil"
  "net/http"
  "net/url"
)

func main() {
  // Create the proxy URL (use https:// and port 8013 for HTTPS)
  proxyStr := "http://_USER_TOKEN_:@smartproxy.crawlbase.com:8012"
  proxyURL, _ := url.Parse(proxyStr)
  transport := &http.Transport{
    Proxy: http.ProxyURL(proxyURL),
    TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
  }

  client := &http.Client{
    Transport: transport,
  }

  req, _ := http.NewRequest("GET", "http://httpbin.org/ip", nil)
  req.Header.Add("CrawlbaseAPI-Parameters", "country=US&get_headers=true")

  resp, err := client.Do(req)
  if err != nil {
    fmt.Println(err)
    return
  }
  defer resp.Body.Close()

  body, _ := ioutil.ReadAll(resp.Body)
  fmt.Println("response Body : ", string(body))
}
```

--------------------------------

### Make Crawlbase API Request with Go

Source: https://crawlbase.com/docs/crawling-api/headless-browsers

Shows how to perform an HTTP GET request to the Crawlbase API in Go, utilizing `net/http` and `net/url` for URL encoding and reading the response body.

```go
package main

import (
  "net/http"
  "fmt"
  "io/ioutil"
  "net/url"
)

func main() {
  url := url.QueryEscape("https://www.missguidedau.com/petite-black-msgd-insert-mesh-cut-out-gym-leggings-10202499")
  resp, _ := http.Get("https://api.crawlbase.com/?token=_JS_TOKEN_&url=" + url)

  defer resp.Body.Close()
  body, _ := ioutil.ReadAll(resp.Body)

  fmt.Println("response Body: ", string(body))
}
```

--------------------------------

### Example Crawlbase Leads API Request and Response

Source: https://crawlbase.com/docs/leads-api/response

Illustrates a GET request to the Crawlbase Leads API with a placeholder token and domain, followed by a sample JSON response showing successful data retrieval, remaining request count, the queried domain, and an array of leads.

```HTTP
GET 'https://api.crawlbase.com/leads?token=_USER_TOKEN_&domain=target-domain.com'
```

```JSON
{
  "success": true,
  "remaining_requests": 4560,
  "domain": "target-domain.com",
  "leads": [{"email": "[email\u00a0protected]", "sources": ['https://target-domain.com/contact']}, ...]
}
```

--------------------------------

### Make HTTP Request with Proxy in Ruby using Typhoeus

Source: https://crawlbase.com/docs/fr/smart-proxy/get

This Ruby snippet demonstrates how to make an HTTP GET request through a proxy using the Typhoeus gem. It configures the proxy URL and explicitly disables SSL verification for both peer and host, which can be useful for testing or specific proxy setups where certificate validation is not required.

```Ruby
request = Typhoeus::Request.new(
  "https://httpbin.org/ip",
  proxy: proxy_url,
  ssl_verifypeer: false,
  ssl_verifyhost: 0
)

response = request.run
puts response.code, response.body
```

--------------------------------

### Make Your First User Agents API Call with cURL

Source: https://crawlbase.com/docs/fr/user-agents-api

This example demonstrates how to make a basic GET request to the Crawlbase User Agents API using cURL, including the required authentication token.

```bash
curl 'https://api.crawlbase.com/user_agents?token=_USER_TOKEN_'
```

--------------------------------

### Make Crawlbase API Request with Python

Source: https://crawlbase.com/docs/crawling-api/headless-browsers

Demonstrates how to use the `requests` library in Python to make a GET request to the Crawlbase API, handling URL encoding and printing the response text.

```python
import requests
from urllib.parse import quote_plus

url = quote_plus('https://github.com/crawlbase?tab=repositories')

response = requests.get(f'https://api.crawlbase.com/?token=_JS_TOKEN_&url={url}')

print(response.text)
```

--------------------------------

### Make HTTP Request with Proxy in Python using Requests

Source: https://crawlbase.com/docs/fr/smart-proxy/get

This Python example demonstrates making an HTTP GET request through a proxy using the popular `requests` library. It includes suppressing `InsecureRequestWarning` for cleaner output when `verify=False` is used, and implements error handling for network issues, supporting both HTTP and HTTPS proxy URLs.

```Python
# pip install requests
import requests
from urllib3.exceptions import InsecureRequestWarning

# Suppress only the single warning from urllib3 needed.
requests.packages.urllib3.disable_warnings(category=InsecureRequestWarning)

# Choose either HTTP or HTTPS endpoint
proxy_url = "http://_USER_TOKEN_:@smartproxy.crawlbase.com:8012"
proxies = {
    "http": proxy_url,
    "https": proxy_url
}

try:
    response = requests.get(
        url="http://httpbin.org/ip",
        proxies=proxies,
        verify=False,
        timeout=30
    )
    response.raise_for_status()

    print('Response Code:', response.status_code)
    print('Response Body:', response.text)
except requests.exceptions.RequestException as e:
    print(f"An error occurred: {e}")
```

--------------------------------

### Make a Basic GET Request to Crawlbase API

Source: https://crawlbase.com/docs/crawling-api/parameters

Demonstrates how to construct and execute a basic GET request to the Crawlbase API using various programming languages. These examples illustrate the inclusion of the authentication token and a URL-encoded target URL as query parameters.

```curl
curl "https://api.crawlbase.com/?token=_USER_TOKEN_&url=https%3A%2F%2Fgithub.com%2Fcrawlbase%3Ftab%3Drepositories"
```

```ruby
require 'net/http'

uri = URI('https://api.crawlbase.com')
uri.query = URI.encode_www_form({
  token: '_USER_TOKEN_',
  url: 'https://github.com/crawlbase?tab=repositories'})

res = Net::HTTP.get_response(uri)
puts "Response HTTP Status Code: #{res.code}"
puts "Response HTTP Header Original Status: #{res['original_status']}"
puts "Response HTTP Response Body: #{res.body}"
```

```node
const https = require('https');

const url = encodeURIComponent('https://github.com/crawlbase?tab=repositories');
const options = {
  hostname: 'api.crawlbase.com',
  path: '/?token=_USER_TOKEN_&url=' + url
};

https.request(options, (response) => {
  let body = '';
  response.on('data', chunk => body += chunk).on('end', () => console.log(body));
}).end();
```

```php
$url = urlencode('https://github.com/crawlbase?tab=repositories');

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://api.crawlbase.com/?token=_USER_TOKEN_&url=' . $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

$response = curl_exec($ch);
curl_close($ch);

var_dump($response);
```

```python
import requests
from urllib.parse import quote_plus

url = quote_plus('https://github.com/crawlbase?tab=repositories')

response = requests.get(f'https://api.crawlbase.com/?token=_USER_TOKEN_&url={url}')

print(response.text)
```

```go
package main

import (
  "net/http"
  "fmt"
  "io/ioutil"
  "net/url"
)

func main() {
  url := url.QueryEscape("https://github.com/crawlbase?tab=repositories")
  resp, _ := http.Get("https://api.crawlbase.com/?token=_USER_TOKEN_&url=" + url)

  defer resp.Body.Close()
  body, _ := ioutil.ReadAll(resp.Body)

  fmt.Println("response Body: ", string(body))
}
```

--------------------------------

### Make your first Crawlbase Screenshots API call with cURL

Source: https://crawlbase.com/docs/screenshots-api

This example demonstrates how to make a basic GET request to the Crawlbase Screenshots API using cURL. It shows how to include your user token and the target URL as query parameters to capture a screenshot of a specified website.

```curl
curl 'https://api.crawlbase.com/screenshots?token=_USER_TOKEN_&url=https%3A%2F%2Fapple.com'
```

--------------------------------

### Make Proxied HTTP Request with Python Requests

Source: https://crawlbase.com/docs/de/smart-proxy/parameters

This Python example utilizes the popular 'requests' library to perform an HTTP GET request through the Crawlbase Smart Proxy. It shows how to define proxy settings, include custom headers, disable SSL warnings and verification, and implement basic error handling for network requests.

```Python
# pip install requests
import requests
from urllib3.exceptions import InsecureRequestWarning

# Suppress only the single warning from urllib3 needed.
requests.packages.urllib3.disable_warnings(category=InsecureRequestWarning)

# Choose either HTTP or HTTPS endpoint
proxy_url = "http://_USER_TOKEN_:@smartproxy.crawlbase.com:8012"  # Use https:// and port 8013 for HTTPS
proxies = {
    "http": proxy_url,
    "https": proxy_url
}

try:
    response = requests.get(
        url="http://httpbin.org/ip",
        headers={"CrawlbaseAPI-Parameters": "country=US&get_headers=true"},
        proxies=proxies,
        verify=False,
        timeout=30
    )
    response.raise_for_status()  # Raise an exception for bad status codes

    print('Response Code:', response.status_code)
    print('Response Body:', response.text)
except requests.exceptions.RequestException as e:
    print(f"An error occurred: {e}")
```

--------------------------------

### Make Crawlbase API Request with cURL

Source: https://crawlbase.com/docs/crawling-api/headless-browsers

Demonstrates how to make a GET request to the Crawlbase API using the cURL command-line tool, including token and URL parameters.

```curl
curl "https://api.crawlbase.com/?token=_JS_TOKEN_&url=https://www.missguidedau.com/petite-black-msgd-insert-mesh-cut-out-gym-leggings-10202499"
```

--------------------------------

### Make HTTP Request with Crawlbase Smart Proxy in Go

Source: https://crawlbase.com/docs/zh-cn/smart-proxy/parameters

This Go example illustrates how to make an HTTP GET request through the Crawlbase Smart Proxy. It configures a custom `http.Transport` with the proxy URL and disables SSL certificate verification. It also shows how to add custom `CrawlbaseAPI-Parameters` headers and read the response body.

```go
package main

import (
  "crypto/tls"
  "fmt"
  "io/ioutil"
  "net/http"
  "net/url"
)

func main() {
  // Create the proxy URL (use https:// and port 8013 for HTTPS)
  proxyStr := "http://[email\u00a0protected]:8012"
  proxyURL, _ := url.Parse(proxyStr)
  transport := &http.Transport{
    Proxy: http.ProxyURL(proxyURL),
    TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
  }

  client := &http.Client{
    Transport: transport,
  }

  req, _ := http.NewRequest("GET", "http://httpbin.org/ip", nil)
  req.Header.Add("CrawlbaseAPI-Parameters", "country=US&get_headers=true")

  resp, err := client.Do(req)
  if err != nil {
    fmt.Println(err)
    return
  }
  defer resp.Body.Close()

  body, _ := ioutil.ReadAll(resp.Body)
  fmt.Println("response Body : ", string(body))
}
```

--------------------------------

### Make HTTPS Request with Proxy in Go

Source: https://crawlbase.com/docs/fr/smart-proxy/get

This Go snippet illustrates how to make an HTTPS GET request through a proxy. It configures an `http.Transport` with the proxy URL and sets `InsecureSkipVerify` to `true` in the `TLSClientConfig` to bypass TLS certificate validation, which is useful for testing or specific proxy setups.

```Go
package main

import (
  "fmt"
  "crypto/tls"
  "io/ioutil"
  "net/http"
  "net/url"
)

func main() {
  // Create the proxy URL (use https:// and port 8013 for HTTPS)
  proxyURL, _ := url.Parse("http://[email protected]:8012")

  // Create a Transport with the proxy and insecure TLS configuration
  transport := &http.Transport{
    Proxy: http.ProxyURL(proxyURL),
    TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
  }

  // Create an HTTP client with the Transport
  client := &http.Client{Transport: transport}

  // Send the request
  resp, err := client.Get("https://httpbin.org/ip")
  if err != nil {
    fmt.Println("Failure : ", err)
    return
  }
  defer resp.Body.Close()

  // Read the response body
  respBody, _ := ioutil.ReadAll(resp.Body)

  // Display results
  fmt.Println("response Status : ", resp.Status)
  fmt.Println("response Headers : ", resp.Header)
  fmt.Println("response Body : ", string(respBody))
}
```

--------------------------------

### Making HTTP/HTTPS Requests via Crawlbase Smart Proxy

Source: https://crawlbase.com/docs/de/smart-proxy/get

These code snippets demonstrate how to configure and send HTTP and HTTPS requests through the Crawlbase Smart Proxy using different programming languages. Each example shows how to set up the proxy URL, handle proxy authentication (where applicable), and manage SSL/TLS verification for successful communication.

```Ruby
request = Typhoeus::Request.new(
  "https://httpbin.org/ip",
  proxy: "http://_USER_TOKEN_:@smartproxy.crawlbase.com:8012",
  ssl_verifypeer: false,
  ssl_verifyhost: 0
)

response = request.run
puts response.code, response.body
```

```JavaScript
const http = require('http');
const https = require('https');
const crawlbaseToken = '_USER_TOKEN_';
const auth = `Basic ${Buffer.from(crawlbaseToken).toString('base64')}`;

// HTTP request
const httpRequest = http.request({
    host: 'smartproxy.crawlbase.com',
    port: 8012,
    path: 'http://httpbin.org/ip',
    headers: {'Proxy-Authorization': auth}
}, res => {
    res.setEncoding('utf8');
    res.on('data', console.log);
});

httpRequest.on('error', console.error);
httpRequest.end();

// HTTPS request
process.env['NODE_TLS_REJECT_UNAUTHORIZED'] = 0;

const httpsRequest = http.request({
    host: 'smartproxy.crawlbase.com',
    port: 8013,
    method: 'CONNECT',
    path: 'httpbin.org:443',
    headers: {'Proxy-Authorization': auth}
}, (res, socket) => {
    if (res.statusCode === 200) {
        https.get({
            host: 'httpbin.org',
            path: '/ip',
            socket: socket,
            agent: false
        }, res => {
            res.setEncoding('utf8');
            res.on('data', console.log);
        });
    }
});

httpsRequest.on('error', console.error);
httpsRequest.end();
```

```PHP
<?php

// cURL initialization
$ch = curl_init('http://httpbin.org/ip');

// Configuring cURL options
$curlOptions = [
    CURLOPT_PROXY => "http://_USER_TOKEN_:@smartproxy.crawlbase.com:8012",
    CURLOPT_CUSTOMREQUEST => 'GET',
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_SSL_VERIFYHOST => false,
    CURLOPT_SSL_VERIFYPEER => false
];

// Setting cURL options
curl_setopt_array($ch, $curlOptions);

// Execute request and capture response
$response = curl_exec($ch);

// Handle any error
if (curl_error($ch)) {
    die('Error: "'.curl_error($ch).'" - Code: '.curl_errno($ch));
}

// Output HTTP Status Code and Response Body
echo 'HTTP Status Code: '.curl_getinfo($ch, CURLINFO_HTTP_CODE) . PHP_EOL;
echo 'Response Body: '.$response . PHP_EOL;

// Close cURL session
curl_close($ch);

?>
```

```Python
import requests
from urllib3.exceptions import InsecureRequestWarning

# Suppress only the single warning from urllib3 needed.
requests.packages.urllib3.disable_warnings(category=InsecureRequestWarning)

# Choose either HTTP or HTTPS endpoint
proxy_url = "http://_USER_TOKEN_:@smartproxy.crawlbase.com:8012"
proxies = {
    "http": proxy_url,
    "https": proxy_url
}

try:
    response = requests.get(
        url="http://httpbin.org/ip",
        proxies=proxies,
        verify=False,
        timeout=30
    )
    response.raise_for_status()

    print('Response Code:', response.status_code)
    print('Response Body:', response.text)
except requests.exceptions.RequestException as e:
    print(f"An error occurred: {e}")
```

```Go
package main

import (
  "fmt"
  "crypto/tls"
  "io/ioutil"
  "net/http"
  "net/url"
)

func main() {
  // Create the proxy URL (use https:// and port 8013 for HTTPS)
  proxyURL, _ := url.Parse("http://_USER_TOKEN_:@smartproxy.crawlbase.com:8012")

  // Create a Transport with the proxy and insecure TLS configuration
  transport := &http.Transport{
    Proxy: http.ProxyURL(proxyURL),
    TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
  }

  // Create an HTTP client with the Transport
  client := &http.Client{Transport: transport}

  // Send the request
  resp, err := client.Get("https://httpbin.org/ip")
  if err != nil {
    fmt.Println("Failure : ", err)
    return
  }
  defer resp.Body.Close()

  // Read the response body
  respBody, _ := ioutil.ReadAll(resp.Body)

  // Display results
  fmt.Println("response Status : ", resp.Status)
  fmt.Println("response Headers : ", resp.Header)
  fmt.Println("response Body : ", string(respBody))
}
```

--------------------------------

### Make HTTP Request via Crawlbase Smart Proxy in Python

Source: https://crawlbase.com/docs/smart-proxy/get

Provides an example of making an HTTP GET request through the Crawlbase Smart Proxy using the `requests` library in Python. It demonstrates how to configure proxy settings, suppress SSL warnings, and handle potential request exceptions.

```Python
# pip install requests
import requests
from urllib3.exceptions import InsecureRequestWarning

# Suppress only the single warning from urllib3 needed.
requests.packages.urllib3.disable_warnings(category=InsecureRequestWarning)

# Choose either HTTP or HTTPS endpoint
proxy_url = "http://_USER_TOKEN_:@smartproxy.crawlbase.com:8012"  # Use https:// and port 8013 for HTTPS
proxies = {
    "http": proxy_url,
    "https": proxy_url
}

try:
    response = requests.get(
        url="http://httpbin.org/ip",
        proxies=proxies,
        verify=False,
        timeout=30
    )
    response.raise_for_status()  # Raise an exception for bad status codes

    print('Response Code:', response.status_code)
    print('Response Body:', response.text)
except requests.exceptions.RequestException as e:
    print(f"An error occurred: {e}")
```

--------------------------------

### Make Crawlbase API Request with Node.js

Source: https://crawlbase.com/docs/crawling-api/headless-browsers

Illustrates how to send an HTTP GET request to the Crawlbase API using Node.js's `https` module, encoding the target URL and logging the response.

```javascript
const https = require('https');

const url = encodeURIComponent('https://www.missguidedau.com/petite-black-msgd-insert-mesh-cut-out-gym-leggings-10202499');
const options = {
  hostname: 'api.crawlbase.com',
  path: '/?token=_JS_TOKEN_&url=' + url
};

https.request(options, (response) => {
  let body = '';
  response.on('data', chunk => body += chunk).on('end', () => console.log(body));
}).end();
```

--------------------------------

### Make Proxied HTTP Request with Go

Source: https://crawlbase.com/docs/de/smart-proxy/parameters

This Go language example demonstrates how to make an HTTP request through the Crawlbase Smart Proxy. It involves setting up a custom HTTP client with a transport layer configured for proxy usage and skipping SSL certificate verification. The code also shows how to add custom headers and read the response body.

```Go
package main

import (
  "crypto/tls"
  "fmt"
  "io/ioutil"
  "net/http"
  "net/url"
)

func main() {
  // Create the proxy URL (use https:// and port 8013 for HTTPS)
  proxyStr := "http://[email protected]:8012"
  proxyURL, _ := url.Parse(proxyStr)
  transport := &http.Transport{
    Proxy: http.ProxyURL(proxyURL),
    TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
  }

  client := &http.Client{
    Transport: transport,
  }

  req, _ := http.NewRequest("GET", "http://httpbin.org/ip", nil)
  req.Header.Add("CrawlbaseAPI-Parameters", "country=US&get_headers=true")

  resp, err := client.Do(req)
  if err != nil {
    fmt.Println(err)
    return
  }
  defer resp.Body.Close()

  body, _ := ioutil.ReadAll(resp.Body)
  fmt.Println("response Body : ", string(body))
}
```

--------------------------------

### Make a Crawlbase API Call with cURL

Source: https://crawlbase.com/docs/crawling-api

This example demonstrates how to make a basic GET request to the Crawlbase Crawling API using cURL. It shows how to use both a standard user token and a JavaScript token for pages requiring client-side rendering, targeting a GitHub repository page.

```bash
curl 'https://api.crawlbase.com/?token=_USER_TOKEN_&url=https%3A%2F%2Fgithub.com%2Fcrawlbase%3Ftab%3Drepositories'
```

```bash
curl 'https://api.crawlbase.com/?token=_JS_TOKEN_&url=https%3A%2F%2Fgithub.com%2Fcrawlbase%3Ftab%3Drepositories'
```

--------------------------------

### Make HTTP Request with Crawlbase Smart Proxy in Python

Source: https://crawlbase.com/docs/zh-cn/smart-proxy/parameters

This Python example demonstrates how to configure and use the Crawlbase Smart Proxy with the `requests` library. It includes disabling SSL warnings, setting up proxy URLs for both HTTP and HTTPS, adding custom `CrawlbaseAPI-Parameters` headers, and handling potential request exceptions.

```python
# pip install requests
import requests
from urllib3.exceptions import InsecureRequestWarning

# Suppress only the single warning from urllib3 needed.
requests.packages.urllib3.disable_warnings(category=InsecureRequestWarning)

# Choose either HTTP or HTTPS endpoint
proxy_url = "http://_USER_TOKEN_:@smartproxy.crawlbase.com:8012"  # Use https:// and port 8013 for HTTPS
proxies = {
    "http": proxy_url,
    "https": proxy_url
}

try:
    response = requests.get(
        url="http://httpbin.org/ip",
        headers={"CrawlbaseAPI-Parameters": "country=US&get_headers=true"},
        proxies=proxies,
        verify=False,
        timeout=30
    )
    response.raise_for_status()  # Raise an exception for bad status codes

    print('Response Code:', response.status_code)
    print('Response Body:', response.text)
except requests.exceptions.RequestException as e:
    print(f"An error occurred: {e}")
```

--------------------------------

### Making a Crawlbase API Screenshot Request

Source: https://crawlbase.com/docs/crawling-api/parameters

Illustrates how to programmatically send an HTTP GET request to the Crawlbase API to trigger a screenshot capture of a specified URL. Examples are provided in Curl, Ruby, Node.js, PHP, Python, and Go, demonstrating URL encoding and basic request handling.

```curl
curl "https://api.crawlbase.com/?token=_JS_TOKEN_&screenshot=true&url=https%3A%2F%2Fgithub.com%2Fcrawlbase%3Ftab%3Drepositories"
```

```ruby
require 'net/http'

uri = URI('https://api.crawlbase.com')
uri.query = URI.encode_www_form({
  token: '_JS_TOKEN_',
  screenshot: true,
  url: 'https://github.com/crawlbase?tab=repositories'})

res = Net::HTTP.get_response(uri)
puts "Response HTTP Status Code: #{res.code}"
puts "Response HTTP Header Original Status: #{res['original_status']}"
puts "Response HTTP Response Body: #{res.body}"
puts "Response Screenshot URL: #{res['screenshot_url']}"
```

```node
const https = require('https');

const url = encodeURIComponent('https://github.com/crawlbase?tab=repositories');
const options = {
  hostname: 'api.crawlbase.com',
  path: '/?token=_JS_TOKEN_&screenshot=true&url=' + url
};

https.request(options, (response) => {
  let body = '';
  response.on('data', chunk => body += chunk).on('end', () => console.log(body));
}).end();
```

```php
$url = urlencode('https://github.com/crawlbase?tab=repositories');

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://api.crawlbase.com/?token=_JS_TOKEN_&screenshot=true&url=' . $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

$response = curl_exec($ch);
curl_close($ch);

var_dump($response);
```

```python
import requests
from urllib.parse import quote_plus

url = quote_plus('https://github.com/crawlbase?tab=repositories')

response = requests.get(f'https://api.crawlbase.com/?token=_JS_TOKEN_&screenshot=true&url={url}')

print(response.text)
```

```go
package main

import (
  "net/http"
  "fmt"
  "io/ioutil"
  "net/url"
)

func main() {
  url := url.QueryEscape("https://github.com/crawlbase?tab=repositories")
  resp, _ := http.Get("https://api.crawlbase.com/?token=_JS_TOKEN_&screenshot=true&url=" + url)

  defer resp.Body.Close()
  body, _ := ioutil.ReadAll(resp.Body)

  fmt.Println("response Body: ", string(body))
}
```

--------------------------------

### Make Crawlbase API Request with Ruby

Source: https://crawlbase.com/docs/crawling-api/headless-browsers

Shows how to fetch data from the Crawlbase API in Ruby using `Net::HTTP`, handling URL encoding and printing the response body and headers.

```ruby
require 'net/http'

uri = URI('https://api.crawlbase.com')
uri.query = URI.encode_www_form({
  token: '_JS_TOKEN_',
  url: 'https://www.missguidedau.com/petite-black-msgd-insert-mesh-cut-out-gym-leggings-10202499'})

res = Net::HTTP.get_response(uri)
puts "Response HTTP Status Code: #{res.code}"
puts "Response HTTP Header Original Status: #{res['original_status']}"
puts "Response HTTP Response Body: #{res.body}"
```

--------------------------------

### Perform HTTP Request via Proxy in Go

Source: https://crawlbase.com/docs/zh-cn/smart-proxy/get

This Go example demonstrates how to send an HTTP GET request through a proxy. It utilizes the `net/http` package to create a custom transport with proxy settings and disables TLS certificate verification for insecure connections.

```Go
package main

import (
  "fmt"
  "crypto/tls"
  "io/ioutil"
  "net/http"
  "net/url"
)

func main() {
  // Create the proxy URL (use https:// and port 8013 for HTTPS)
  proxyURL, _ := url.Parse("http://[email protected]:8012")

  // Create a Transport with the proxy and insecure TLS configuration
  transport := &http.Transport{
    Proxy: http.ProxyURL(proxyURL),
    TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
  }

  // Create an HTTP client with the Transport
  client := &http.Client{Transport: transport}

  // Send the request
  resp, err := client.Get("https://httpbin.org/ip")
  if err != nil {
    fmt.Println("Failure : ", err)
    return
  }
  defer resp.Body.Close()

  // Read the response body
  respBody, _ := ioutil.ReadAll(resp.Body)

  // Display results
  fmt.Println("response Status : ", resp.Status)
  fmt.Println("response Headers : ", resp.Header)
  fmt.Println("response Body : ", string(respBody))
}
```

--------------------------------

### Make Crawlbase API Request with pretty=true

Source: https://crawlbase.com/docs/crawling-api/parameters

Demonstrates how to make an HTTP GET request to the Crawlbase API, including the `pretty=true` parameter to format the JSON response for improved readability. Examples cover constructing the URL with query parameters and processing the API response in various programming languages.

```curl
curl "https://api.crawlbase.com/?token=_USER_TOKEN_&url=https%3A%2F%2Fgithub.com%2Fcrawlbase%3Ftab%3Drepositories&format=json&pretty=true"
```

```ruby
require 'net/http'
require 'json'

uri = URI('https://api.crawlbase.com')
uri.query = URI.encode_www_form({
  token: '_USER_TOKEN_',
  format: 'json',
  pretty: true,
  url: 'https://github.com/crawlbase?tab=repositories'})

res = Net::HTTP.get_response(uri)
json = JSON.parse(res)
puts "Response HTTP Status Code: #{res.code}"
puts "Response HTTP Response JSON Body: #{res.body}"
puts "Response Body: #{json['body']}"
puts "Response Original Status: #{json['original_status']}"
```

```javascript
const https = require('https');

const url = encodeURIComponent('https://github.com/crawlbase?tab=repositories');
const options = {
  hostname: 'api.crawlbase.com',
  path: '/?token=_USER_TOKEN_&format=json&pretty=true&url=' + url
};

https.request(options, (response) => {
  let body = '';
  response.on('data', chunk => body += chunk).on('end', () => {
    const json = JSON.parse(body);
    console.log(json.original_status);
    console.log(json.body);
  });
}).end();
```

```php
$url = urlencode('https://github.com/crawlbase?tab=repositories');

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://api.crawlbase.com/?token=_USER_TOKEN_&format=json&pretty=true&url=' . $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

$response = curl_exec($ch);
curl_close($ch);

$json = json_decode($response);
var_dump($json['body']);
var_dump($json['original_status']);
```

```python
import requests
from urllib.parse import quote_plus

url = quote_plus('https://github.com/crawlbase?tab=repositories')

response = requests.get(f'https://api.crawlbase.com/?token=_USER_TOKEN_&format=json&pretty=true&url={url}')

print(response.text)
```

```go
package main

import (
  "net/http"
  "fmt"
  "io/ioutil"
  "net/url"
  "encoding/json"
)

func main() {
  url := url.QueryEscape("https://github.com/crawlbase?tab=repositories")
  resp, _ := http.Get("https://api.crawlbase.com/?token=_USER_TOKEN_&format=json&pretty=true&url=" + url)

  defer resp.Body.Close()
  body, _ := ioutil.ReadAll(resp.Body)

  var parsedJson map[string]interface{}
  json.Unmarshal(body, &parsedJson)

  fmt.Println("response: ", parsedJson["original_status"].(float64), parsedJson["body"].(string))
}
```

--------------------------------

### Perform Basic HTTP GET Request in Ruby

Source: https://crawlbase.com/docs/smart-proxy/parameters

Demonstrates how to make a simple HTTP GET request and print the response status, message, and body using a generic HTTP client in Ruby. This snippet illustrates fundamental HTTP client interaction.

```Ruby
res = http.get(uri.request_uri, headers)
puts res.code, res.message, res.body
```

--------------------------------

### Make a GET Request to Crawlbase Crawling API

Source: https://crawlbase.com/docs/crawling-api/try-crawling-api

This snippet demonstrates how to make a basic GET request to the Crawlbase Crawling API using various programming languages. It fetches headers from `httpbin.org` through the API, showcasing the core functionality.

```curl
curl "https://api.crawlbase.com/?token=_USER_TOKEN_&url=https://httpbin.org/headers"
```

```ruby
require 'net/http'

uri = URI('https://api.crawlbase.com')
uri.query = URI.encode_www_form({
  token: '_USER_TOKEN_',
  url: 'https://httpbin.org/headers'})

res = Net::HTTP.get_response(uri)
puts "Response HTTP Status Code: #{res.code}"
puts "Response HTTP Header Original Status: #{res['original_status']}"
puts "Response HTTP Response Body: #{res.body}"
```

```node
const https = require('https');

const url = encodeURIComponent('https://httpbin.org/headers');
const options = {
  hostname: 'api.crawlbase.com',
  path: '/?token=_USER_TOKEN_&url=' + url,
};

https
  .request(options, (response) => {
    let body = '';
    response.on('data', (chunk) => (body += chunk)).on('end', () => console.log(body));
  })
  .end();
```

```php
$url = urlencode('https://httpbin.org/headers');

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://api.crawlbase.com/?token=_USER_TOKEN_&url=' . $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

$response = curl_exec($ch);
curl_close($ch);

var_dump($response);
```

```python
import requests
from urllib.parse import quote_plus

url = quote_plus('https://httpbin.org/headers')

response = requests.get(f'https://api.crawlbase.com/?token=_USER_TOKEN_&url={url}')

print(response.text)
```

```go
package main

import (
  "net/http"
  "fmt"
  "io/ioutil"
  "net/url"
)

func main() {
  url := url.QueryEscape("https://httpbin.org/headers")
  resp, _ := http.Get("https://api.crawlbase.com/?token=_USER_TOKEN_&url=" + url)

  defer resp.Body.Close()
  body, _ := ioutil.ReadAll(resp.Body)

  fmt.Println("response Body: ", string(body))
}
```

--------------------------------

### Make Crawlbase API Request with PHP

Source: https://crawlbase.com/docs/crawling-api/headless-browsers

Provides an example of making a Crawlbase API request in PHP using cURL functions, including URL encoding and retrieving the response.

```php
$url = urlencode('https://www.missguidedau.com/petite-black-msgd-insert-mesh-cut-out-gym-leggings-10202499');

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://api.crawlbase.com/?token=_JS_TOKEN_&url=' . $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

$response = curl_exec($ch);
curl_close($ch);

var_dump($response);
```

--------------------------------

### Making Crawlbase API Requests with Automated Scrolling

Source: https://crawlbase.com/docs/crawling-api/parameters

Examples demonstrating how to make API requests to Crawlbase using the `scroll` and `scroll_interval` parameters to enable automated page scrolling for dynamic content. These examples show how to construct the API URL and handle the response in various programming languages.

```curl
curl "https://api.crawlbase.com/?token=_JS_TOKEN_&scroll=true&scroll_interval=20&url=https%3A%2F%2Fwww.reddit.com%2Fsearch%2F%3Fq%3Dcrawlbase"
```

```ruby
require 'net/http'

uri = URI('https://api.crawlbase.com')
uri.query = URI.encode_www_form({
  token: '_JS_TOKEN_',
  scroll: true,
  scroll_interval: 20,
  url: 'https://www.reddit.com/search/?q=crawlbase'})

res = Net::HTTP.get_response(uri)
puts "Response HTTP Status Code: #{res.code}"
puts "Response HTTP Header Original Status: #{res['original_status']}"
puts "Response HTTP Response Body: #{res.body}"
```

```node
const https = require('https');

const url = encodeURIComponent('https://www.reddit.com/search/?q=crawlbase');
const options = {
  hostname: 'api.crawlbase.com',
  path: '/?token=_JS_TOKEN_&scroll=true&scroll_interval=20&url=' + url
};

https.request(options, (response) => {
  let body = '';
  response.on('data', chunk => body += chunk).on('end', () => console.log(body));
}).end();
```

```php
$url = urlencode('https://www.reddit.com/search/?q=crawlbase');

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://api.crawlbase.com/?token=_JS_TOKEN_&scroll=true&scroll_interval=20&url=' . $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

$response = curl_exec($ch);
curl_close($ch);

var_dump($response);
```

```python
import requests
from urllib.parse import quote_plus

url = quote_plus('https://www.reddit.com/search/?q=crawlbase')

response = requests.get(f'https://api.crawlbase.com/?token=_JS_TOKEN_&scroll=true&scroll_interval=20&url={url}')

print(response.text)
```

```go
package main

import (
  "net/http"
  "fmt"
  "io/ioutil"
  "net/url"
)

func main() {
  url := url.QueryEscape("https://www.reddit.com/search/?q=crawlbase")
  resp, _ := http.Get("https://api.crawlbase.com/?token=_JS_TOKEN_&scroll=true&scroll_interval=20&url=" + url)

  defer resp.Body.Close()
  body, _ := ioutil.ReadAll(resp.Body)

  fmt.Println("response Body: ", string(body))
}
```

--------------------------------

### Make HTTP Request with Proxy in PHP using cURL

Source: https://crawlbase.com/docs/fr/smart-proxy/get

This PHP snippet demonstrates how to perform an HTTP GET request through a proxy using the cURL library. It configures `CURLOPT_PROXY` and disables SSL verification (`CURLOPT_SSL_VERIFYHOST`, `CURLOPT_SSL_VERIFYPEER`) for flexible proxy integration, handling potential errors and displaying the response.

```PHP
<?php

// cURL initialization
$ch = curl_init('http://httpbin.org/ip');

// Configuring cURL options
$curlOptions = [
    CURLOPT_PROXY => "http://[email protected]:8012",
    CURLOPT_CUSTOMREQUEST => 'GET',
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_SSL_VERIFYHOST => false,
    CURLOPT_SSL_VERIFYPEER => false
];

// Setting cURL options
curl_setopt_array($ch, $curlOptions);

// Execute request and capture response
$response = curl_exec($ch);

// Handle any error
if (curl_error($ch)) {
    die('Error: "'.curl_error($ch).'" - Code: '.curl_errno($ch));
}

// Output HTTP Status Code and Response Body
echo 'HTTP Status Code: '.curl_getinfo($ch, CURLINFO_HTTP_CODE) . PHP_EOL;
echo 'Response Body: '.$response . PHP_EOL;

// Close cURL session
curl_close($ch);

?>
```

--------------------------------

### Send POST Request with Form Data via Crawlbase Proxy (Go)

Source: https://crawlbase.com/docs/zh-cn/smart-proxy/post

This Go example illustrates how to make a POST request with URL-encoded form data using the Crawlbase Smart Proxy. It sets up a custom HTTP transport with proxy configuration and disables TLS certificate verification for the connection.

```Go
package main

import (
  "net/http"
  "net/url"
  "strings"
  "fmt"
  "crypto/tls"
)

func main() {
  // Create the proxy URL (use https:// and port 8013 for HTTPS)
  proxyURL, _ := url.Parse("http://_USER_TOKEN_:@smartproxy.crawlbase.com:8012")

  // Create a Transport with the proxy and insecure TLS configuration
  transport := &http.Transport{
    Proxy: http.ProxyURL(proxyURL),
    TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
  }

  client := &http.Client{Transport: transport}

  data := url.Values{}
  data.Set("param", "value")
  req, _ := http.NewRequest("POST", "http://httpbin.org/anything", strings.NewReader(data.Encode()))
  req.Header.Set("Content-Type", "application/x-www-form-urlencoded")

  resp, err := client.Do(req)
  if err != nil {
    fmt.Println(err)
  } else {
    fmt.Println("POST request sent successfully")
  }
  resp.Body.Close()
}
```

--------------------------------

### Make Proxied HTTP Request with Node.js

Source: https://crawlbase.com/docs/de/smart-proxy/parameters

This Node.js example shows how to configure and send an HTTP request through the Crawlbase Smart Proxy. It includes setting proxy authorization using a base64 encoded token and adding custom 'CrawlbaseAPI-Parameters' headers. The snippet handles response data streaming and error logging.

```JavaScript
const http = require('http');
const https = require('https');
const crawlbaseToken = '_USER_TOKEN_';
const auth = `Basic ${Buffer.from(crawlbaseToken).toString('base64')}`;

// HTTP request
const httpRequest = http.request({
    host: 'smartproxy.crawlbase.com',
    port: 8012,  // Use 8013 for HTTPS
    path: 'http://httpbin.org/ip',
    headers: {
      'Proxy-Authorization': auth,
      'CrawlbaseAPI-Parameters': 'country=US&get_headers=true',
    },
}, res => {
    res.setEncoding('utf8');
    res.on('data', console.log);
});

httpRequest.on('error', console.error);
httpRequest.end();
```

--------------------------------

### Crawlbase Leads API Reference

Source: https://crawlbase.com/docs/ru/leads-api

Comprehensive documentation for the Crawlbase Leads API, detailing its base URL, authentication requirements, and a basic usage example. Includes an important note regarding the API's deprecation for new registrations.

```APIDOC
Crawlbase Leads API:

Base URL: https://api.crawlbase.com/leads

Authentication:
  All requests to the Leads API require a 'token' parameter.
  This token is your personal user token.
  Example: ?token=_USER_TOKEN_

Deprecation Notice:
  As of October 1, 2024, the Leads API is no longer available for new registrations.
  Existing Leads API clients will continue to have access to the product.

First API Call Example (cURL):
  curl 'https://api.crawlbase.com/leads?token=_USER_TOKEN_&domain=slack.com'
```

--------------------------------

### Make HTTP and HTTPS Requests with Proxy in Node.js

Source: https://crawlbase.com/docs/fr/smart-proxy/get

This Node.js example illustrates how to make both HTTP and HTTPS requests via a proxy using the built-in `http` and `https` modules. It sets up `Proxy-Authorization` for authentication and includes a method to disable TLS/SSL verification for HTTPS connections, often used in development or specific proxy environments.

```Node.js
const http = require('http');
const https = require('https');
const crawlbaseToken = '_USER_TOKEN_';
const auth = `Basic ${Buffer.from(crawlbaseToken).toString('base64')}`;

// HTTP request
const httpRequest = http.request({
    host: 'smartproxy.crawlbase.com',
    port: 8012,
    path: 'http://httpbin.org/ip',
    headers: {'Proxy-Authorization': auth},
}, res => {
    res.setEncoding('utf8');
    res.on('data', console.log);
});

httpRequest.on('error', console.error);
httpRequest.end();

// HTTPS request
process.env['NODE_TLS_REJECT_UNAUTHORIZED'] = 0;

const httpsRequest = http.request({
    host: 'smartproxy.crawlbase.com',
    port: 8013,
    method: 'CONNECT',
    path: 'httpbin.org:443',
    headers: {'Proxy-Authorization': auth},
}, (res, socket) => {
    if (res.statusCode === 200) {
        https.get({
            host: 'httpbin.org',
            path: '/ip',
            socket: socket,
            agent: false,
        }, res => {
            res.setEncoding('utf8');
            res.on('data', console.log);
        });
    }
});

httpsRequest.on('error', console.error);
httpsRequest.end();
```

--------------------------------

### Send POST Request with Form Data via Crawlbase Proxy (Python)

Source: https://crawlbase.com/docs/zh-cn/smart-proxy/post

This Python example demonstrates how to send a POST request with URL-encoded form data through the Crawlbase Smart Proxy. It configures proxy settings, disables SSL verification for convenience, and includes error handling for network requests.

```Python
# Suppress only the single warning from urllib3 needed.
requests.packages.urllib3.disable_warnings(category=InsecureRequestWarning)

# Choose either HTTP or HTTPS endpoint
proxy_url = "http://_USER_TOKEN_:@smartproxy.crawlbase.com:8012"  # Use https:// and port 8013 for HTTPS
proxies = {
    "http": proxy_url,
    "https": proxy_url
}

# Form data
data = {'param': 'value'}

try:
    response = requests.post(
        'http://httpbin.org/anything',
        data=data,
        proxies=proxies,
        verify=False,
        timeout=30
    )
    response.raise_for_status()  # Raise an exception for bad status codes

    print('Response Code:', response.status_code)
    print('Response Body:', response.text)
except requests.exceptions.RequestException as e:
    print(f"An error occurred: {e}")
```

--------------------------------

### Send POST Request with JSON Data via Crawlbase Proxy (Multi-language)

Source: https://crawlbase.com/docs/zh-cn/smart-proxy/post

This collection of examples demonstrates how to send POST requests containing JSON data through the Crawlbase Smart Proxy using various programming languages. It covers both HTTP and HTTPS proxy configurations and proper JSON payload handling for each language.

```cURL
curl -H "accept: application/json" \
--data '{"key1":"value1","key2":"value2"}' \
-X POST \
-x "http://_USER_TOKEN_:@smartproxy.crawlbase.com:8012" \
-k "http://httpbin.org/anything"
```

```cURL
curl -H "accept: application/json" \
--data '{"key1":"value1","key2":"value2"}' \
-X POST \
-x "https://_USER_TOKEN_:@smartproxy.crawlbase.com:8013" \
-k "http://httpbin.org/anything"
```

```Ruby
require 'net/http'
require 'openssl'
require 'uri'
require 'json'

# Choose either HTTP or HTTPS endpoint
proxy_host = 'smartproxy.crawlbase.com'
proxy_port = 8012  # Use 8013 for HTTPS
proxy_user = '_USER_TOKEN_'
uri = URI('http://httpbin.org/anything')

http = Net::HTTP.new(uri.host, uri.port, proxy_host, proxy_port, proxy_user)
http.use_ssl = uri.scheme == 'https'
http.verify_mode = OpenSSL::SSL::VERIFY_NONE if http.use_ssl?

# JSON data
json_data = { 'key1' => 'value1', 'key2' => 'value2' }

request = Net::HTTP::Post.new(uri.request_uri, 'Content-Type' => 'application/json')
request.body = json_data.to_json

res = http.request(request)
puts res.code, res.message, res.body
```

```Node.js
const http = require('http');
const https = require('https');

// JSON data
const postData = JSON.stringify({
  'key1': 'value1',
  'key2': 'value2'
});

const options = {
  hostname: 'httpbin.org',
  port: 80,
  path: '/anything',
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': Buffer.byteLength(postData)
  }
};

const proxy = {
  host: 'smartproxy.crawlbase.com',
  port: 8012,  // Use 8013 for HTTPS
  auth: '_USER_TOKEN_'
};

const req = http.request(options, (res) => {
  res.setEncoding('utf8');
  res.on('data', (chunk) => {
    console.log(`BODY: ${chunk}`);
  });
});

req.on('error', (e) => {
  console.error(`problem with request: ${e.message}`);
});

req.write(postData);
req.end();
```

```PHP
<?php
$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, 'http://httpbin.org/anything');
curl_setopt($ch, CURLOPT_PROXY, 'http://_USER_TOKEN_:@smartproxy.crawlbase.com:8012');  // Use https:// and port 8013 for HTTPS
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode(['key1' => 'value1', 'key2' => 'value2']));
curl_setopt($ch, CURLOPT_HTTPHEADER, ['Content-Type: application/json']);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);

$response = curl_exec($ch);
if(curl_errno($ch)) {
    echo 'Error: ' . curl_error($ch);
}
curl_close($ch);
?>
```

--------------------------------

### Send JSON POST Request with Go net/http

Source: https://crawlbase.com/docs/fr/smart-proxy/post

This Go example demonstrates how to send a POST request with JSON data using the `net/http` package. It includes setting up a custom `http.Transport` for proxy configuration and handling insecure TLS connections. The example marshals a Go map into JSON for the request body.

```Go
package main

import (
  "bytes"
  "encoding/json"
  "net/http"
  "net/url"
  "fmt"
  "crypto/tls"
)

func main() {
  // Create the proxy URL (use https:// and port 8013 for HTTPS)
  proxyURL, _ := url.Parse("http://[email protected]:8012")

  // Create a Transport with the proxy and insecure TLS configuration
  transport := &http.Transport{
    Proxy: http.ProxyURL(proxyURL),
    TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
  }

  client := &http.Client{Transport: transport}

  // JSON data
  jsonData := map[string]string{
    "key1": "value1",
    "key2": "value2",
  }

  jsonValue, _ := json.Marshal(jsonData)

  req, _ := http.NewRequest("POST", "http://httpbin.org/anything", bytes.NewBuffer(jsonValue))
  req.Header.Set("Content-Type", "application/json")

  resp, err := client.Do(req)
  if err != nil {
    fmt.Println(err)
    return
  }
  defer resp.Body.Close()

  fmt.Println("POST request sent successfully")
}
```

--------------------------------

### Making Requests to Crawlbase Leads API

Source: https://crawlbase.com/docs/zh-cn/leads-api/parameters

Demonstrates how to construct and send requests to the Crawlbase Leads API using various programming languages and cURL, providing practical examples for integrating lead extraction into different applications.

```curl
curl "https://api.crawlbase.com/leads?token=_USER_TOKEN_&domain=twitter.com"
```

```ruby
require 'net/http'

uri = URI('https://api.crawlbase.com/leads')
uri.query = URI.encode_www_form({
  token: '_USER_TOKEN_',
  domain: 'dropbox.com'})

res = Net::HTTP.get_response(uri)
puts "Response HTTP Status Code: #{res.code}"
puts "Response HTTP Response Body: #{res.body}"
```

```node
const https = require('https');

const domain = 'dropbox.com';
const options = {
  hostname: 'api.crawlbase.com',
  path: '/leads?token=_USER_TOKEN_&domain=' + domain
};

https.request(options, (response) => {
  let body = '';
  response.on('data', chunk => body += chunk).on('end', () => console.log(body));
}).end();
```

```php
$domain = 'instagram.com';

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://api.crawlbase.com/leads?token=_USER_TOKEN_&domain=' . $domain);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

$response = curl_exec($ch);
curl_close($ch);

var_dump($response);
```

```python
import requests

domain = 'paddle.com'

response = requests.get(f'https://api.crawlbase.com/leads?token=_USER_TOKEN_&domain={domain}')
print(response.text)
```

```go
package main

import (
  "net/http"
  "fmt"
  "io/ioutil"
)

func main() {
  domain := "stackoverflow.com"
  resp, _ := http.Get("https://api.crawlbase.com/leads?token=_USER_TOKEN_&domain=" + domain)

  defer resp.Body.Close()
  body, _ := ioutil.ReadAll(resp.Body)

  fmt.Println("response Body: ", string(body))
}
```

--------------------------------

### Make Crawlbase API Request with Custom User Agent

Source: https://crawlbase.com/docs/fr/crawling-api/parameters

These code examples demonstrate how to make a request to the Crawlbase API, specifically utilizing the 'user_agent' parameter to send a custom user agent string. Each example targets the 'https://postman-echo.com/headers' URL to show the effect of the custom user agent.

```curl
curl "https://api.crawlbase.com/?token=_USER_TOKEN_&user_agent=Mozilla%2F5.0+%28Macintosh%3B+Intel+Mac+OS+X+10_12_5%29+AppleWebKit%2F603.2.4+%28KHTML%2C+like+Gecko%29+Version%2F10.1.1+Safari%2F603.2.4&url=https%3A%2F%2Fpostman-echo.com%2Fheaders"
```

```ruby
require 'net/http'

uri = URI('https://api.crawlbase.com')
uri.query = URI.encode_www_form({
  token: '_USER_TOKEN_',
  user_agent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1.1 Safari/605.1.15',
  url: 'https://postman-echo.com/headers'})

res = Net::HTTP.get_response(uri)
puts "Response HTTP Status Code: #{res.code}"
puts "Response HTTP Header Original Status: #{res['original_status']}"
puts "Response HTTP Response Body: #{res.body}"
```

```node
const https = require('https');

const url = encodeURIComponent('https://postman-echo.com/headers');
const userAgent = encodeURIComponent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1.1 Safari/605.1.15');
const options = {
  hostname: 'api.crawlbase.com',
  path: '/?token=_USER_TOKEN_&user_agent=' + userAgent + '&url=' + url
};

https.request(options, (response) => {
  let body = '';
  response.on('data', chunk => body += chunk).on('end', () => console.log(body));
}).end();
```

```php
$url = urlencode('https://postman-echo.com/headers');
$userAgent = urlencode('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1.1 Safari/605.1.15');

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://api.crawlbase.com/?token=_USER_TOKEN_&user_agent=' . $userAgent . '&url=' . $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

$response = curl_exec($ch);
curl_close($ch);

var_dump($response);
```

```python
import requests
from urllib.parse import quote_plus

url = quote_plus('https://postman-echo.com/headers')
user_agent = quote_plus('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1.1 Safari/605.1.15')

response = requests.get(f'https://api.crawlbase.com/?token=_USER_TOKEN_&user_agent={user_agent}&url={url}')

print(response.text)
```

```go
package main

import (
  "net/http"
  "fmt"
  "io/ioutil"
  "net/url"
)

func main() {
  urlString := url.QueryEscape("https://postman-echo.com/headers")
  userAgent := url.QueryEscape("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1.1 Safari/605.1.15")
  resp, _ := http.Get("https://api.crawlbase.com/?token=_USER_TOKEN_&user_agent=" + userAgent + "&url=" + urlString)

  defer resp.Body.Close()
  body, _ := ioutil.ReadAll(resp.Body)

  fmt.Println("response Body: ", string(body))
}
```

--------------------------------

### Make HTTP Request via Crawlbase Smart Proxy (PHP)

Source: https://crawlbase.com/docs/ru/smart-proxy/parameters

This PHP example uses cURL to send an HTTP request through the Crawlbase Smart Proxy. It sets the proxy address, includes custom 'CrawlbaseAPI-Parameters' in the headers, and handles potential cURL errors. SSL verification is disabled for simplicity.

```PHP
<?php
$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, 'http://httpbin.org/ip');
curl_setopt($ch, CURLOPT_PROXY, "http://[email protected]:8012");  // Use https:// and port 8013 for HTTPS
curl_setopt($ch, CURLOPT_HTTPHEADER, array('CrawlbaseAPI-Parameters: country=US&get_headers=true'));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);

$response = curl_exec($ch);
if(curl_errno($ch)) {
    echo 'Error: ' . curl_error($ch);
}
curl_close($ch);
?>
```

--------------------------------

### Making Initial Requests with Crawlbase Smart Proxy

Source: https://crawlbase.com/docs/smart-proxy

Demonstrates how to make an initial request through the Crawlbase Smart Proxy using both HTTPS (recommended) and HTTP protocols, specifying the proxy address and port, and disabling SSL verification for the destination URL. Your access token is used as the proxy username.

```bash
curl -x "https://YOUR_ACCESS_TOKEN@smartproxy.crawlbase.com:8013" -k "https://httpbin.org/ip"
```

```bash
curl -x "http://YOUR_ACCESS_TOKEN@smartproxy.crawlbase.com:8012" -k "https://httpbin.org/ip"
```

--------------------------------

### Execute GET Request with Crawlbase Smart Proxy using cURL

Source: https://crawlbase.com/docs/smart-proxy/get

Demonstrates how to send a GET request through the Crawlbase Smart Proxy using cURL. It provides examples for both HTTPS (recommended) and HTTP proxy connections to fetch content from a target URL.

```curl
curl -x "https://[email protected]:8013" -k "https://httpbin.org/ip"
```

```curl
curl -x "http://[email protected]:8012" -k "https://httpbin.org/ip"
```

--------------------------------

### Crawlbase User Agents API Reference

Source: https://crawlbase.com/docs/fr/user-agents-api

Comprehensive documentation for the Crawlbase User Agents API, detailing its base URL, authentication requirements, and usage constraints.

```APIDOC
User Agents API:
  Base URL: https://api.crawlbase.com/user_agents
  Authentication: Requires 'token' parameter with your private token (?token=_USER_TOKEN_)
  Rate Limit: 1 request per second
  Cost: Free to use
```

--------------------------------

### Make Your First Leads API Call with cURL

Source: https://crawlbase.com/docs/leads-api

This cURL command demonstrates how to make a basic request to the Crawlbase Leads API to retrieve leads for a specific domain. Remember to replace `_USER_TOKEN_` with your actual private token.

```bash
curl 'https://api.crawlbase.com/leads?token=_USER_TOKEN_&domain=slack.com'
```

--------------------------------

### Crawlbase Storage API: Get Total Document Count

Source: https://crawlbase.com/docs/storage-api/total-count

Comprehensive documentation for the Crawlbase Storage API endpoint used to retrieve the total number of documents. It details the GET request, required parameters, example cURL command, and the expected JSON response structure.

```APIDOC
GET /storage/total_count
  - Description: Retrieves the total number of documents stored in the user's Crawlbase storage area.
  - Parameters:
    - token: string (required)
      - Your private Crawlbase API token.
  - Example Request:
    curl 'https://api.crawlbase.com/storage/total_count?token=_USER_TOKEN_'
  - Example Response:
    {
        "totalCount": 5491078
    }
  - Returns:
    - totalCount: integer
      - The total number of documents.
```

--------------------------------

### Perform GET Request with Headless Browser using Crawlbase Smart Proxy

Source: https://crawlbase.com/docs/smart-proxy/headless-browsers

These `curl` commands demonstrate how to make a GET request using the Crawlbase Smart Proxy with JavaScript-enabled headless browser functionality. The `javascript=true` parameter is set in the `CrawlbaseAPI-Parameters` header. Examples are provided for both HTTP and HTTPS proxy connections.

```bash
curl -H "CrawlbaseAPI-Parameters: javascript=true" \
-x "http://[email protected]:8012" \
-k "http://httpbin.org/anything"
```

```bash
curl -H "CrawlbaseAPI-Parameters: javascript=true" \
-x "https://[email protected]:8013" \
-k "http://httpbin.org/anything"
```

--------------------------------

### Make First User Agents API Call with cURL

Source: https://crawlbase.com/docs/user-agents-api

This example demonstrates how to make a basic request to the Crawlbase User Agents API using cURL, including the required authentication token. Replace `_USER_TOKEN_` with your actual API token obtained from the Crawlbase dashboard.

```bash
curl 'https://api.crawlbase.com/user_agents?token=_USER_TOKEN_'
```

--------------------------------

### Make your first Scraper API call with cURL

Source: https://crawlbase.com/docs/scraper-api

This example demonstrates how to perform a basic API call to the Crawlbase Scraper API using the cURL command-line tool. It shows the base endpoint, how to include your user token, and specify the target URL for scraping.

```bash
curl 'https://api.crawlbase.com/scraper?token=_USER_TOKEN_&url=https%3A%2F%2Fwww.amazon.com%2Fdp%2FB00JITDVD2'
```

--------------------------------

### Crawlbase Leads API: `limit` Parameter Documentation

Source: https://crawlbase.com/docs/leads-api/parameters

Comprehensive documentation for the `limit` parameter used in the Crawlbase Leads API, detailing its type, optionality, default behavior, and the credit consumption model associated with its usage.

```APIDOC
Parameter: limit
  - Type: integer
  - Optional: Yes
  - Description: Specifies the maximum number of emails to return in a single API call.
  - Default: 10 emails if not specified.
  - Credit Consumption: 1 credit is consumed for every 10 or fewer emails returned per domain. For example, if you set your limit to 100 and it returned 100 emails, that single API request will consume 10 credits.
```

--------------------------------

### Make a Crawlbase Scraper API Request

Source: https://crawlbase.com/docs/scraper-api/parameters

This snippet demonstrates how to construct and execute a basic GET request to the Crawlbase Scraper API. It showcases the use of the mandatory `token` for authentication and the `url` parameter for specifying the target webpage, with examples across multiple programming languages.

```curl
curl "https://api.crawlbase.com/scraper?token=_USER_TOKEN_&url=https%3A%2F%2Fwww.amazon.com%2Fdp%2FB00JITDVD2"
```

```ruby
require 'net/http'

uri = URI('https://api.crawlbase.com/scraper')
uri.query = URI.encode_www_form({
  token: '_USER_TOKEN_',
  url: 'https://www.amazon.com/dp/B00JITDVD2'})

res = Net::HTTP.get_response(uri)
puts "Response HTTP Status Code: #{res.code}"
puts "Response HTTP Header Original Status: #{res['original_status']}"
puts "Response HTTP Response Body: #{res.body}"
```

```javascript
const https = require('https');

const url = encodeURIComponent('https://www.amazon.com/dp/B00JITDVD2');
const options = {
  hostname: 'api.crawlbase.com',
  path: '/scraper?token=_USER_TOKEN_&url=' + url
};

https.request(options, (response) => {
  let body = '';
  response.on('data', chunk => body += chunk).on('end', () => console.log(body));
}).end();
```

```php
$url = urlencode('https://www.amazon.com/dp/B00JITDVD2');

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://api.crawlbase.com/scraper?token=_USER_TOKEN_&url=' . $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

$response = curl_exec($ch);
curl_close($ch);

var_dump($response);
```

```python
import requests
from urllib.parse import quote_plus

url = quote_plus('https://www.amazon.com/dp/B0B7CBZZ16')

response = requests.get(f'https://api.crawlbase.com/scraper?token=_USER_TOKEN_&url={url}')

print(response.text)
```

```go
package main

import (
  "net/http"
  "fmt"
  "io/ioutil"
  "net/url"
)

func main() {
  url := url.QueryEscape("https://www.amazon.com/dp/B00JITDVD2")
  resp, _ := http.Get("https://api.crawlbase.com/scraper?token=_USER_TOKEN_&url=" + url)

  defer resp.Body.Close()
  body, _ := ioutil.ReadAll(resp.Body)

  fmt.Println("response Body: ", string(body))
}
```

--------------------------------

### Crawlbase Leads API Parameters Reference

Source: https://crawlbase.com/docs/leads-api/parameters

This section provides a detailed reference for the essential parameters used when interacting with the Crawlbase Leads API. It covers authentication via the `token` and specifying the target website for lead extraction using the `domain` parameter.

```APIDOC
Leads API Parameters:

token
  - Required: Yes
  - Type: string
  - Description: Your authentication token. All requests must be authorized with this private token. Replace `_USER_TOKEN_` with your actual token.

domain
  - Required: Yes
  - Type: string
  - Description: The domain for which to retrieve email addresses as leads.
```

--------------------------------

### Send Custom Request Headers with Crawlbase API in Multiple Languages

Source: https://crawlbase.com/docs/fr/crawling-api/parameters

Demonstrates how to make a GET request to the Crawlbase API, including the `request_headers` parameter to pass custom HTTP headers to the target URL. Examples are provided for Curl, Ruby, Node.js, PHP, Python, and Go.

```curl
curl "https://api.crawlbase.com/?token=_USER_TOKEN_&request_headers=accept-language%3Aen-GB%7Caccept-encoding%3Agzip&url=https%3A%2F%2Fpostman-echo.com%2Fheaders"
```

```ruby
require 'net/http'

uri = URI('https://api.crawlbase.com')
uri.query = URI.encode_www_form({
  token: '_USER_TOKEN_',
  request_headers: 'accept-language:en-GB|accept-encoding:gzip',
  url: 'https://postman-echo.com/headers'})

res = Net::HTTP.get_response(uri)
puts "Response HTTP Status Code: \#{res.code}"
puts "Response HTTP Header Original Status: \#{res['original_status']}"
puts "Response HTTP Response Body: \#{res.body}"
```

```node
const https = require('https');

const url = encodeURIComponent('https://postman-echo.com/headers');
const options = {
  hostname: 'api.crawlbase.com',
  path: '/?token=_USER_TOKEN_&request_headers=accept-language%3Aen-GB%7Caccept-encoding%3Agzip&url=' + url
};

https.request(options, (response) => {
  let body = '';
  response.on('data', chunk => body += chunk).on('end', () => console.log(body));
}).end();
```

```php
$url = urlencode('https://postman-echo.com/headers');

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://api.crawlbase.com/?token=_USER_TOKEN_&request_headers=accept-language%3Aen-GB%7Caccept-encoding%3Agzip&url=' . $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

$response = curl_exec($ch);
curl_close($ch);

var_dump($response);
```

```python
import requests
from urllib.parse import quote_plus

url = quote_plus('https://postman-echo.com/headers')

response = requests.get(f'https://api.crawlbase.com/?token=_USER_TOKEN_&request_headers=accept-language%3Aen-GB%7Caccept-encoding%3Agzip&url={url}')

print(response.text)
```

```go
package main

import (
  "net/http"
  "fmt"
  "io/ioutil"
  "net/url"
)

func main() {
  url := url.QueryEscape("https://postman-echo.com/headers")
  resp, _ := http.Get("https://api.crawlbase.com/?token=_USER_TOKEN_&request_headers=accept-language%3Aen-GB%7Caccept-encoding%3Agzip&url=" + url)

  defer resp.Body.Close()
  body, _ := ioutil.ReadAll(resp.Body)

  fmt.Println("response Body: ", string(body))
}
```