ScrapeOps

https://scrapeops.io/docs/
Admin
ScrapeOps is a web scraping platform that provides proxy APIs, residential proxies, monitoring,...
Tokens:237,435
Snippets:692
Trust Score:8.8
Update:1 year ago
Show doc for...
Tokens:
Raw
### Getting Started with GeeLark Proxy Integration

Source: https://scrapeops.io/docs/residential-mobile-proxy-aggregator/tool-integrations/geelark-proxy-integration

This guide explains the initial steps required to set up and start using GeeLark proxies with ScrapeOps. It covers account setup and basic configuration.

```APIDOC
GeeLark Proxy Integration:
  Getting Started:
    1. Register for a ScrapeOps account.
    2. Obtain your ScrapeOps API key.
    3. Configure your GeeLark account or application to use ScrapeOps proxy endpoints.
```

--------------------------------

### ScrapeOps API Endpoint - Python Requests Example

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/getting-started/api-basics

Provides a Python example using the `requests` library to interact with the ScrapeOps Proxy API endpoint. It shows how to set up proxy parameters, including API key and target URL, and send a GET request.

```python
import requests
from urllib.parse import urlencode

proxy_params ={
'api_key':'YOUR_API_KEY',
'url':'https://httpbin.org/ip',
'render_js':True,
}

response = requests.get(
  url='https://proxy.scrapeops.io/v1/',
  params=urlencode(proxy_params),
  timeout=120,
)

print('Body: ', response.content)
```

--------------------------------

### Example Request with Request Optimization

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/getting-started/api-basics

This `curl` command demonstrates how to make a request to the ScrapeOps Proxy API with Request Optimization enabled. It includes the API key, the target URL, and the `optimize_request=true` parameter.

```curl
curl -k "https://proxy.scrapeops.io/v1/?api_key=YOUR_API_KEY&url=http://httpbin.org/anything&optimize_request=true"
```

--------------------------------

### Install Selenium Wire

Source: https://scrapeops.io/docs/residential-mobile-proxy-aggregator/integration-examples/python-selenium-example

Installs the Selenium Wire library, which is recommended for integrating proxies with Selenium. This is a prerequisite for the following examples.

```bash
pip install selenium-wire
```

--------------------------------

### ScrapeOps API Endpoint - Curl Example

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/getting-started/api-basics

Demonstrates how to make a GET request to the ScrapeOps Proxy API endpoint using curl. It shows how to include the API key and the target URL as query parameters.

```curl
curl -k "https://proxy.scrapeops.io/v1/?api_key=YOUR_API_KEY&url=http://httpbin.org/anything"
```

--------------------------------

### ScrapeOps C# Integration Example

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/csharp-code-example

This example demonstrates how to use the ScrapeOps Proxy API Aggregator with C#. It includes setting up the API key, making a request to the proxy API, and handling the response. This is a basic integration to get started with ScrapeOps in a C# environment.

```csharp
using System;
using System.Net.Http;
using System.Threading.Tasks;

public class ScrapeOpsClient
{
    private readonly string _apiKey;
    private readonly HttpClient _httpClient;

    public ScrapeOpsClient(string apiKey)
    {
        _apiKey = apiKey ?? throw new ArgumentNullException(nameof(apiKey));
        _httpClient = new HttpClient();
        _httpClient.BaseAddress = new Uri("https://proxy.scrapeops.io/v1/");
    }

    public async Task<string> GetHtmlAsync(string url, string ip = null, string country = null, string city = null, string isp = null, string userAgent = null, string geo = null, string proxyType = null, string renderJs = null, string callbackUrl = null, string callbackData = null)
    {
        var requestUrl = $"?api_key={_apiKey}&url={Uri.EscapeDataString(url)}";

        if (!string.IsNullOrEmpty(ip)) requestUrl += $"&ip={ip}";
        if (!string.IsNullOrEmpty(country)) requestUrl += $"&country={country}";
        if (!string.IsNullOrEmpty(city)) requestUrl += $"&city={city}";
        if (!string.IsNullOrEmpty(isp)) requestUrl += $"&isp={isp}";
        if (!string.IsNullOrEmpty(userAgent)) requestUrl += $"&user_agent={Uri.EscapeDataString(userAgent)}";
        if (!string.IsNullOrEmpty(geo)) requestUrl += $"&geo={geo}";
        if (!string.IsNullOrEmpty(proxyType)) requestUrl += $"&proxy_type={proxyType}";
        if (!string.IsNullOrEmpty(renderJs)) requestUrl += $"&render_js={renderJs}";
        if (!string.IsNullOrEmpty(callbackUrl)) requestUrl += $"&callback_url={Uri.EscapeDataString(callbackUrl)}";
        if (!string.IsNullOrEmpty(callbackData)) requestUrl += $"&callback_data={Uri.EscapeDataString(callbackData)}";

        try
        {
            var response = await _httpClient.GetAsync(requestUrl);
            response.EnsureSuccessStatusCode(); // Throw if HTTP status is not 2xx
            return await response.Content.ReadAsStringAsync();
        }
        catch (HttpRequestException e)
        {
            Console.WriteLine($"Error fetching URL: {e.Message}");
            return null;
        }
    }

    public static async Task Main(string[] args)
    {
        // Replace with your actual ScrapeOps API key
        string apiKey = "YOUR_SCRAPEOPS_API_KEY"; 
        var client = new ScrapeOpsClient(apiKey);

        string targetUrl = "http://httpbin.org/html"; // Example URL
        string htmlContent = await client.GetHtmlAsync(targetUrl, country: "us", renderJs: "true");

        if (htmlContent != null)
        {
            Console.WriteLine("Successfully fetched HTML content:");
            Console.WriteLine(htmlContent);
        }
        else
        {
            Console.WriteLine("Failed to fetch HTML content.");
        }
    }
}
```

--------------------------------

### Install Selenium Wire

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/python-selenium-example

Installs the Selenium Wire library, which simplifies proxy integration with Selenium.

```bash
pip install selenium-wire
```

--------------------------------

### ScrapeOps Proxy API Aggregator Documentation

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/python-selenium-example

This section details the ScrapeOps Proxy API Aggregator, including its quick start guide, getting started procedures, advanced functionalities, account management, and integration examples across various programming languages.

```APIDOC
ScrapeOps Proxy API Aggregator:
  Overview:
    A service that aggregates and manages proxies for web scraping.

  Quick Start:
    URL: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/quickstart/
    Description: Provides a rapid introduction to setting up and using the ScrapeOps Proxy API Aggregator.

  Getting Started:
    URL: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/python-selenium-example/
    Description: A comprehensive guide to begin integrating ScrapeOps with your web scraping projects.

  Advanced Functionality:
    URL: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/python-selenium-example/
    Description: Explores advanced features and configurations available with the ScrapeOps Proxy API Aggregator.

  Account Management:
    URL: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/python-selenium-example/
    Description: Information on managing your ScrapeOps account, including API keys and subscription details.

  Integration Examples:
    Python Code Examples: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/python-code-example/
    Scrapy Code Examples: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/python-scrapy-example/
    Selenium Code Examples: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/python-selenium-example/
    NodeJs Code Examples: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/nodejs-code-example/
    NodeJs Puppeteer Code Examples: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/nodejs-puppeteer-example/
    NodeJs Playwright Code Examples: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/nodejs-playwright-example/
    PHP Code Examples: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/php-code-example/
    Java Code Examples: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/java-code-example/
    Golang Colly Code Examples: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/go-colly-code-example/
    Golang Code Examples: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/golang-code-example/
    Ruby Code Examples: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/ruby-code-example/
    C# Code Examples: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/csharp-code-example/
    R Code Examples: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/r-code-example/
    Description: A collection of code examples for integrating ScrapeOps with various programming languages and scraping frameworks.
```

--------------------------------

### ScrapeOps Proxy Port - Scrapy Example

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/getting-started/api-basics

Provides an example for Scrapy users on how to integrate with the ScrapeOps proxy port. It shows how to set the proxy in the `meta` object of a Scrapy request.

```python
# ...other scrapy setup code
start_urls =['https://httpbin.org/ip']
meta ={
"proxy":"http://scrapeops:YOUR_API_KEY@proxy.scrapeops.io:5353"
}

def parse(self, response):
# ...your parsing logic here
yield scrapy.Request(url, callback=self.parse, meta=meta)
```

--------------------------------

### ScrapeOps Proxy Port - Python Requests Example

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/getting-started/api-basics

Shows how to configure the `requests` library in Python to use the ScrapeOps proxy port. It details setting up the `proxies` dictionary with the correct proxy URL and disabling SSL verification.

```python
import requests

proxies ={
"http":"http://scrapeops:YOUR_API_KEY@proxy.scrapeops.io:5353"
}
response = requests.get('https://httpbin.org/ip', proxies=proxies, verify=False)
print(response.text)
```

--------------------------------

### cURL Request with Proxy Port

Source: https://scrapeops.io/docs/residential-mobile-proxy-aggregator/getting-started/quickstart

Example of making a request using cURL with the ScrapeOps Residential Proxy Port. This demonstrates how to specify the proxy address, username, and password.

```curl
curl -x "http://scrapeops:YOUR_API_KEY@residential-proxy.scrapeops.io:8181" "https://httpbin.org/ip"
```

--------------------------------

### Install ScrapeOps Node.js SDK and Playwright

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/nodejs-playwright-example

Installs the necessary packages for ScrapeOps integration with Node.js and Playwright. This includes the ScrapeOps SDK and Playwright browser automation library.

```bash
npm install scrapeops-node-proxy-api playwright

```

--------------------------------

### Enable JavaScript Rendering

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/getting-started/api-basics

Demonstrates how to enable JavaScript rendering for a request by adding the `render_js=true` parameter to the ScrapeOps Proxy URL.

```bash
curl -k "https://proxy.scrapeops.io/v1/?api_key=YOUR_API_KEY&url=http://httpbin.org/anything&render_js=true"
```

--------------------------------

### Install httparty Gem

Source: https://scrapeops.io/docs/residential-mobile-proxy-aggregator/integration-examples/ruby-code-example

Command to install the httparty gem, a dependency for making HTTP requests in Ruby.

```bash
gem install httparty
```

--------------------------------

### Basic Request with ScrapeOps Proxy

Source: https://scrapeops.io/docs/residential-mobile-proxy-aggregator/integration-examples/python-code-example

Example Python code to send a URL to the ScrapeOps Proxy using the `requests` library. It demonstrates how to set up the proxy URL with your API key and target URL, and how to make a GET request.

```python
import requests

api_key ='YOUR_API_KEY'
target_url ='https://httpbin.org/ip'
proxy_url =f'http://scrapeops:{api_key}@residential-proxy.scrapeops.io:8181'

proxies ={
'http': proxy_url,
'https': proxy_url,
}

response = requests.get(
    url=target_url,
    proxies=proxies,
    timeout=120,
)

print('Body:', response.content)

```

--------------------------------

### R Web Scraping with ScrapeOps Proxy Aggregator

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/r-code-example

This example demonstrates how to use R to make requests through the ScrapeOps Proxy Aggregator. It includes setting up the API key and making a GET request to a target URL.

```R
library(httr)

# Replace with your ScrapeOps API Key
api_key <- "YOUR_SCRAPEOPS_API_KEY"

# Target URL to scrape
target_url <- "http://httpbin.org/ip"

# Construct the ScrapeOps API endpoint URL
scrapeops_url <- paste0("https://proxy.scrapeops.io/v1/", "?api_key=", api_key, "&url=", URLencode(target_url))

# Make the GET request
response <- GET(scrapeops_url)

# Print the response content
print(content(response, "text"))

```

--------------------------------

### Install httr Package

Source: https://scrapeops.io/docs/residential-mobile-proxy-aggregator/integration-examples/r-code-example

Installs the 'httr' package, which is required for making HTTP requests in R.

```R
install.packages("httr")
```

--------------------------------

### Python Requests with Proxy Port and Fake Browser Headers

Source: https://scrapeops.io/docs/residential-mobile-proxy-aggregator/getting-started/quickstart

An example combining the ScrapeOps Residential Proxy Port with the Fake Browser Headers API using Python Requests. It shows how to fetch a list of headers, select a random one, and use it in the request.

```python
import requests
from random import randint

# Get list of fake browser headers
response = requests.get('http://headers.scrapeops.io/v1/browser-headers?api_key=YOUR_API_KEY')
header_list = response.json().get('result',[])

# Get random header from list
defget_random_header(header_list):
    random_index = randint(0,len(header_list)-1)
return header_list[random_index]

# Configure proxy
proxies ={
"http":"http://scrapeops:YOUR_API_KEY@residential-proxy.scrapeops.io:8181",
"https":"http://scrapeops:YOUR_API_KEY@residential-proxy.scrapeops.io:8181"
}

# Make request with both proxy and fake headers
response = requests.get(
'https://httpbin.org/ip',
    proxies=proxies,
    headers=get_random_header(header_list),
    verify=False
)
print(response.text)
```

--------------------------------

### Install ScrapeOps Scrapy Proxy SDK

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/python-scrapy-example

Installs the ScrapeOps Scrapy Proxy SDK using pip. This SDK simplifies the integration of the ScrapeOps proxy into your Scrapy projects, automatically routing requests through the proxy.

```bash
pip install scrapeops-scrapy-proxy-sdk
```

--------------------------------

### Install httparty Gem

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/ruby-code-example

Command to install the httparty gem, a dependency for making HTTP requests in Ruby.

```bash
gem install httparty
```

--------------------------------

### ScrapeOps Proxy Authorization

Source: https://scrapeops.io/docs/residential-mobile-proxy-aggregator/getting-started/quickstart

This section details how to authorize requests to the ScrapeOps proxy service. It explains the necessity of an API key and how to include it in your requests to avoid authorization errors.

```APIDOC
API Key Authorization:

To use the ScrapeOps proxy, you need an API key obtained from a free account registration.

Include your API key with every request using the `password` proxy port parameter.

Example:
proxy.scrapeops.io:8012?api_key=YOUR_API_KEY

Failure to include the API key will result in a `403 Forbidden Access` status code.
```

--------------------------------

### Python Requests with Proxy Port

Source: https://scrapeops.io/docs/residential-mobile-proxy-aggregator/getting-started/quickstart

Demonstrates how to configure Python Requests to use the ScrapeOps Residential Proxy Port. It includes setting up the proxies dictionary and making a GET request. Note the requirement to disable SSL verification.

```python
import requests

proxies ={
"http":"http://scrapeops:YOUR_API_KEY@residential-proxy.scrapeops.io:8181",
"https":"http://scrapeops:YOUR_API_KEY@residential-proxy.scrapeops.io:8181"
}
response = requests.get('https://httpbin.org/ip', proxies=proxies, verify=False)
print(response.text)
```

--------------------------------

### PHP ScrapeOps Proxy Aggregator Example

Source: https://scrapeops.io/docs/residential-mobile-proxy-aggregator/integration-examples/php-code-example

This example demonstrates how to use the ScrapeOps PHP client to make requests through the Residential Mobile Proxy Aggregator. It includes setting up the client with your API key and making a basic GET request to a target URL.

```php
<?php

require 'vendor/autoload.php'; // Assuming you installed via Composer

use ScrapeOps\ProxyFetcher\ProxyFetcher;

// Replace with your ScrapeOps API Key
$apiKey = 'YOUR_API_KEY';

// Initialize the ProxyFetcher client
$proxyFetcher = new ProxyFetcher($apiKey);

// Define the target URL you want to scrape
$targetUrl = 'http://httpbin.org/ip';

// Set proxy fetcher options (optional)
$options = [
    'country' => 'us',
    'proxy_type' => 'residential',
    'return_proxy' => true
];

try {
    // Make a request using the proxy fetcher
    $response = $proxyFetcher->get($targetUrl, $options);

    // Print the response
    echo "Status Code: " . $response['status'] . "\n";
    echo "Body:\n" . $response['body'] . "\n";

    // If you requested proxy details, they will be in the 'proxy' key
    if (isset($response['proxy'])) {
        echo "Proxy Used: " . $response['proxy']['proxy'] . "\n";
    }

} catch (Exception $e) {
    echo 'Error: ' . $e->getMessage();
}

?>
```

--------------------------------

### NodeJS Puppeteer Guide

Source: https://scrapeops.io/docs/residential-mobile-proxy-aggregator/tool-integrations/puppeteer-proxy-integration

A comprehensive guide to using Puppeteer for web scraping in NodeJS. Covers setup, element selection, data extraction, and handling dynamic content.

```nodejs
const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com');
  // Further scraping logic here
  await browser.close();
})();
```

--------------------------------

### ScrapeOps Proxy API Endpoint - Python Example

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/quickstart

Provides an example of using the Python requests library to interact with the ScrapeOps Proxy API endpoint. It shows how to set parameters like API key, target URL, and render_js, and how to handle the response.

```python
import requests
from urllib.parse import urlencode

proxy_params ={
'api_key':'YOUR_API_KEY',
'url':'https://httpbin.org/ip',
'render_js':True,
}

response = requests.get(
  url='https://proxy.scrapeops.io/v1/',
  params=urlencode(proxy_params),
  timeout=120,
)

print('Body: ', response.content)
```

--------------------------------

### ScrapeOps Proxy Port - Curl with Parameters Example

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/getting-started/api-basics

Demonstrates how to pass additional parameters to the ScrapeOps proxy port using curl by appending them to the username. This example shows enabling country targeting.

```curl
curl -x "http://scrapeops.country=us:YOUR_API_KEY@proxy.scrapeops.io:5353" "https://httpbin.org/ip"
```

--------------------------------

### Ruby Code Example for ScrapeOps Proxy Aggregator

Source: https://scrapeops.io/docs/residential-mobile-proxy-aggregator/integration-examples/ruby-code-example

This Ruby code snippet demonstrates how to use the ScrapeOps Proxy Aggregator to make a web request. It includes setting up the necessary parameters like the target URL, ScrapeOps API key, and proxy settings. The example shows how to construct the request and print the response.

```ruby
require 'open-uri'

# ScrapeOps Configuration
scrapeops_api_key = 'YOUR_SCRAPEOPS_API_KEY'
scrapeops_endpoint = 'https://proxy.scrapeops.io/v1/'

# Target URL
target_url = 'https://httpbin.org/ip'

# Construct the request URL with parameters
request_url = "#{scrapeops_endpoint}?api_key=#{scrapeops_api_key}&url=#{target_url}"

begin
  # Make the request using open-uri
  response = URI.open(request_url).read
  puts "Response:"
  puts response
rescue OpenURI::HTTPError => e
  puts "HTTP Error: #{e.message}"
rescue StandardError => e
  puts "An error occurred: #{e.message}"
end

```

--------------------------------

### GeeLark Proxy Integration Examples

Source: https://scrapeops.io/docs/residential-mobile-proxy-aggregator/tool-integrations/geelark-proxy-integration

Practical examples demonstrating how to integrate GeeLark proxies with popular web scraping tools and libraries.

```APIDOC
GeeLark Proxy Integration:
  Integration Examples:
    - Python (Requests Library):
      import requests
      
      proxy_url = "http://username:password@proxy.scrapeops.io:8012?token=YOUR_API_TOKEN"
      proxies = {
          "http": proxy_url,
          "https": proxy_url
      }
      
      response = requests.get("http://httpbin.org/ip", proxies=proxies)
      print(response.json())

    - Node.js (Axios):
      const axios = require('axios');
      
      const proxyUrl = 'http://username:password@proxy.scrapeops.io:8012?token=YOUR_API_TOKEN';
      
      axios.get('http://httpbin.org/ip', {
        proxy: {
          host: 'proxy.scrapeops.io',
          port: 8012,
          auth: {
            username: 'username',
            password: 'password'
          }
        },
        headers: {
          'Proxy-Authorization': `Basic ${Buffer.from('username:password').toString('base64')}`
        }
      })
      .then(response => {
        console.log(response.data);
      })
      .catch(error => {
        console.error(error);
      });

    - Selenium:
      from selenium import webdriver
      
      proxy_host = "proxy.scrapeops.io"
      proxy_port = 8012
      proxy_username = "username"
      proxy_password = "password"
      
      webdriver_proxy = "--proxy-server=http://%s:%s@%s:%d" % (proxy_username, proxy_password, proxy_host, proxy_port)
      
      options = webdriver.ChromeOptions()
      options.add_argument(webdriver_proxy)
      
      driver = webdriver.Chrome(options=options)
      driver.get("http://httpbin.org/ip")
      print(driver.page_source)
      driver.quit()
```

--------------------------------

### ScrapeOps Proxy API Parameters

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/getting-started/api-basics

Lists and describes the various parameters that can be added to the ScrapeOps Proxy URL to enable advanced functionality. This includes parameters for response formats, request optimization, anti-bot bypass, auto-extraction, JavaScript rendering, scrolling, screenshots, premium proxies, and more.

```APIDOC
ScrapeOps Proxy API Parameters:

json_response: Return an extended JSON response with additional information about the request and response such as cookies and XHR requests/responses. Example: `json_response=true`.
optimize_request: Request with request optimization enabled. Example: `optimize_request=true`
max_request_cost: Used in conjunction with `optimize_request` to set the maximum number of API credits a request can use. Example: `max_request_cost=30`
bypass: Request with anti-bot bypass enabled. Example: `bypass=cloudflare_level_1`
auto_extract: Use maintained parsers to automatically extract data from HTML and return data in JSON format. Example: `auto_extract=amazon`
render_js: Request with Javascript rendering enabled. Example: `render_js=true`
wait: Tell headless browser to wait a specfic period of time before returning response. Example: `wait=3000`
wait_for: Tell headless browser to wait a specfic page element to appear before returning response. Example: `wait_for=.loading-done`
scroll: Tell headless browser to scroll the page down a defined number of pixels before returning the response. Example: `scroll=5000`
screenshot: Tell the headless browser to take a screenshot of the rendered page. The screenshot will be returned in a base64 encoded string. Parameters needed: `screenshot=true&render_js=true&json_response=true`.
js_scenario: Send a sequence of commands to a headless browser before returning the response.
premium: Request using premium proxy pools. Example: `premium=level_1` or `premium=level_2`.
residential: Request using residential proxy pools. Example: `residential=true`
mobile: Request using mobile proxy pools. Example: `mobile=true`
country: Make requests from specific country. Example: `country=us`
keep_headers: Use your own custom headers when making the request. Example: `keep_headers=true`
device_type: Tell API to use desktop vs mobile user-agents when making requests. Default is `desktop`. Example: `device_type=mobile`
session_number: Enable sticky sessions that use the same IP address for multiple requests by setting a `session_number`. Example: `session_number=7`
follow_redirects: Tell API to not follow redirects by setting `follow_redirects=false`.
initial_status_code: Tell API to return the inital status code the website responses with in the headers by setting `initial_status_code=true`.
final_status_code: Tell API to return the final status code the website responses with in the headers by setting `final_status_code=true`.
```

--------------------------------

### ScrapeOps Proxy API Endpoint - Curl Example

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/quickstart

Demonstrates how to make a GET request to the ScrapeOps Proxy API endpoint using curl. It includes the API key and the target URL as query parameters.

```curl
curl -k "https://proxy.scrapeops.io/v1/?api_key=YOUR_API_KEY&url=http://httpbin.org/anything"
```

--------------------------------

### ScrapeOps Proxy Port - Curl Example

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/getting-started/api-basics

Illustrates how to use the ScrapeOps proxy port with curl. It shows the format for specifying the username (scrapeops), API key as the password, the proxy address, and the target URL.

```curl
curl -x "http://scrapeops:YOUR_API_KEY@proxy.scrapeops.io:5353" "https://httpbin.org/ip"
```

--------------------------------

### Golang Web Scraping with ScrapeOps Proxy API

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/golang-code-example

This example demonstrates how to use the ScrapeOps Proxy API with Golang for web scraping. It shows basic integration and how to configure the API key and other parameters.

```go
package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
	"net/url"
	"os"
)

func main() {
	scrapeOpsApiKey := os.Getenv("SCRAPEOPS_API_KEY")
	if scrapeOpsApiKey == "" {
		fmt.Println("Error: SCRAPEOPS_API_KEY environment variable not set.")
		return
	}

	targetUrl := "http://example.com"

	// Prepare the request URL with parameters
	params := url.Values{}
	params.Add("api_key", scrapeOpsApiKey)
	params.Add("url", targetUrl)

	// You can add other ScrapeOps parameters here, e.g.:
	// params.Add("render_js", "true")
	// params.Add("country", "us")

	scrapeOpsUrl := "https://proxy.scrapeops.io/v1/?" + params.Encode()

	// Create an HTTP client
	client := &http.Client{}

	// Create a GET request
	req, err := http.NewRequest("GET", scrapeOpsUrl, nil)
	if err != nil {
		fmt.Printf("Error creating request: %v\n", err)
		return
	}

	// Send the request
	res, err := client.Do(req)
	if err != nil {
		fmt.Printf("Error sending request: %v\n", err)
		return
	}

	defer res.Body.Close()

	// Read the response body
	body, err := ioutil.ReadAll(res.Body)
	if err != nil {
		fmt.Printf("Error reading response body: %v\n", err)
		return
	}

	// Print the response status and body
	fmt.Printf("Status Code: %d\n", res.StatusCode)
	fmt.Printf("Response Body:\n%s\n", body)
}

```

--------------------------------

### Target Server Response Example

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/getting-started/api-basics

The default response from the ScrapeOps Proxy API endpoint and Proxy Port is the response returned by the target URL. This can be in various formats like HTML, JSON, or XML, depending on the target website's server response.

```html
<html>
<head>
     ...
</head>
<body>
     ...
</body>
</html>
```

--------------------------------

### Initialize npm Project

Source: https://scrapeops.io/docs/residential-mobile-proxy-aggregator/tool-integrations/puppeteer-proxy-integration

Initializes a new JavaScript project with default settings using npm.

```bash
cd puppeteer-residential
npm init --y
```

--------------------------------

### Basic ScrapeOps Proxy Request in Go

Source: https://scrapeops.io/docs/residential-mobile-proxy-aggregator/integration-examples/golang-code-example

This Go code snippet demonstrates how to send an HTTP GET request through the ScrapeOps proxy. It configures the proxy URL with an API key and target URL, sets a client timeout, and handles the response. Dependencies include the `net/http`, `net/url`, `io/ioutil`, and `time` packages.

```go
package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
	"net/url"
	"time"
)

func main() {
	apiKey := "YOUR_API_KEY"
	targetURL := "https://httpbin.org/ip"
	proxyURL := "http://scrapeops:" + apiKey + "@residential-proxy.scrapeops.io:8181"

	proxy, err := url.Parse(proxyURL)
	if err != nil {
		fmt.Println("Error parsing proxy URL:", err)
		return
	}

	client := &http.Client{
		Timeout: time.Duration(120 * time.Second),
		Transport: &http.Transport{
			Proxy: http.ProxyURL(proxy),
		},
	}

	req, err := http.NewRequest("GET", targetURL, nil)
	if err != nil {
		fmt.Println("Error creating request:", err)
		return
	}

	response, err := client.Do(req)
	if err != nil {
		fmt.Println("Error making request:", err)
		return
	}
	defer response.Body.Close()

	if response.StatusCode == http.StatusOK {
		bodyBytes, err := ioutil.ReadAll(response.Body)
		if err != nil {
			fmt.Println("Error reading response body:", err)
			return
		}
		fmt.Println("Body:", string(bodyBytes))
	} else {
		fmt.Println("Error:", response.StatusCode)
	}
}
```

--------------------------------

### Python ScrapeOps Proxy Aggregator Integration

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/python-code-example

This example demonstrates how to use the ScrapeOps Python client to send requests through the proxy aggregator. It includes setting up the client, making a request to a target URL, and handling the response. Ensure you have the `scrapeops` library installed (`pip install scrapeops`).

```python
from scrapeops.proxy import ScrapeOpsProxy

# Initialize the ScrapeOpsProxy client
# Replace 'YOUR_API_KEY' with your actual ScrapeOps API key
# Replace 'YOUR_ACCOUNT_ID' with your actual ScrapeOps Account ID
s = ScrapeOpsProxy(api_key="YOUR_API_KEY", account_id="YOUR_ACCOUNT_ID")

# Define the target URL you want to scrape
target_url = "http://httpbin.org/ip"

# Make a request through the ScrapeOps proxy
# You can specify parameters like country, city, etc.
response = s.get(target_url, country="us")

# Print the response content
print(response.text)

# You can also access other response attributes like status code, headers, etc.
print(f"Status Code: {response.status_code}")
print(f"Headers: {response.headers}")
```

--------------------------------

### Python Web Scraping Resources

Source: https://scrapeops.io/docs/residential-mobile-proxy-aggregator/tool-integrations/geelark-proxy-integration

A collection of Python web scraping guides and playbooks offered by ScrapeOps, covering beginner tutorials, Selenium usage, and more.

```python
# Creating Your First Web Scraper
# https://scrapeops.io/python-web-scraping-playbook/python-requests-beautifulsoup-beginners-guide/

# Intro To Web Scraping With Selenium
# https://scrapeops.io/selenium-web-scraping-playbook/python-selenium-web-scraping-guide/

# General Web Scraping Playbook
# https://scrapeops.io/web-scraping-playbook/

# Python Playbook
# https://scrapeops.io/python-web-scraping-playbook/

# Selenium Playbook
# https://scrapeops.io/selenium-web-scraping-playbook/
```

--------------------------------

### Install httr Package

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/r-code-example

Installs the 'httr' package, which is required for making HTTP requests in R. This is a prerequisite for using the ScrapeOps Proxy Aggregator with R.

```R
install.packages("httr")
```

--------------------------------

### Basic Golang Request to ScrapeOps Proxy

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/golang-code-example

This Go code demonstrates how to send an HTTP GET request to the ScrapeOps Proxy endpoint. It includes setting up the API key, target URL, and proxy URL, handling request creation, making the request with a timeout, and processing the response. It utilizes the `net/http` and `net/url` packages for HTTP communication and URL encoding.

```Go
package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
	"net/url"
	"time"
)

func main() {
	apiKey := "YOUR_API_KEY"
	targetURL := "https://httpbin.org/ip"
	proxyURL := "https://proxy.scrapeops.io/v1/"

	queryParams := url.Values{}
	queryParams.Add("api_key", apiKey)
	queryParams.Add("url", targetURL)

	client := &http.Client{
		Timeout: time.Duration(120 * time.Second),
	}

	req, err := http.NewRequest("GET", proxyURL+"?"+queryParams.Encode(), nil)
	if err != nil {
		fmt.Println("Error creating request:", err)
		return
	}

	response, err := client.Do(req)
	if err != nil {
		fmt.Println("Error making request:", err)
		return
	}
	defer response.Body.Close()

	if response.StatusCode == http.StatusOK {
		bodyBytes, err := ioutil.ReadAll(response.Body)
		if err != nil {
			fmt.Println("Error reading response body:", err)
			return
		}
		fmt.Println("Body:", string(bodyBytes))
	} else {
		fmt.Println("Error:", response.StatusCode)
	}
}

```

--------------------------------

### Web Scraping Without Getting Blocked

Source: https://scrapeops.io/docs/residential-mobile-proxy-aggregator/tool-integrations/capsolver-proxy-integration

This guide focuses on strategies and techniques to avoid getting blocked during web scraping activities, ensuring continuous data collection.

```text
Strategies to avoid blocking:
- Rotate IP addresses using proxies.
- Mimic real browser headers.
- Implement delays between requests.
- Handle CAPTCHAs effectively.
```

--------------------------------

### Playwright Proxy Integration Guide

Source: https://scrapeops.io/docs/residential-mobile-proxy-aggregator/tool-integrations/playwright-proxy-integration

This guide explains how to integrate ScrapeOps Residential Proxies with Playwright. It covers prerequisites, setup, testing, and configuration for geotargeting and static proxies.

```APIDOC
Playwright Proxy Integration:
  Introduction: Overview of integrating Playwright with ScrapeOps proxies.
  Prerequisites: Requirements for setting up the integration.
  Setting Up ScrapeOps Residential Proxies: Steps to configure ScrapeOps proxies.
  Testing the Integration: How to verify the proxy setup.
  Configuring Playwright for Proxy Integration:
    Geotargeting: Setting up proxies for specific geographic locations.
    Using Static Proxies: Configuring and using static proxy IPs.
  Common Issues and Troubleshooting:
    No Proxy Connection: Solutions for connection errors.
    Incorrect Location: Troubleshooting location-based issues.
    Static Proxy Issues: Resolving problems with static proxies.
  Conclusion: Summary of the integration process.
```

--------------------------------

### ScrapeOps Proxy API Aggregator - Getting Started

Source: https://scrapeops.io/docs/intro

Demonstrates how to make a request to the ScrapeOps Proxy API Aggregator using cURL. It shows how to include your API key and the target URL as query parameters. The response includes the raw HTML content and status code.

```curl
curl -k "https://proxy.scrapeops.io/v1/?api_key=YOUR_API_KEY&url=http://httpbin.org/anything"
```

--------------------------------

### Playwright Proxy Integration Example

Source: https://scrapeops.io/docs/residential-mobile-proxy-aggregator/tool-integrations/playwright-proxy-integration

This example demonstrates how to configure Playwright to use ScrapeOps Residential Proxies. It shows the necessary setup for both Python and JavaScript environments to route browser traffic through the proxy.

```python
from playwright import sync_playwright

# Replace with your ScrapeOps proxy details
proxy_host = "proxy.scrapeops.io"
proxy_port = 8080
proxy_username = "YOUR_USERNAME"
proxy_password = "YOUR_PASSWORD"

proxy_config = {
    "server": f"http://{proxy_username}:{proxy_password}@{proxy_host}:{proxy_port}"
}

with sync_playwright() as p:
    browser = p.chromium.launch(
        headless=True, 
        proxy=proxy_config
    )
    page = browser.new_page()
    page.goto("https://httpbin.org/ip")
    print(page.content())
    browser.close()
```

```javascript
const { chromium } = require('playwright');

// Replace with your ScrapeOps proxy details
const proxyHost = 'proxy.scrapeops.io';
const proxyPort = 8080;
const proxyUsername = 'YOUR_USERNAME';
const proxyPassword = 'YOUR_PASSWORD';

const proxyConfig = {
  server: `http://${proxyUsername}:${proxyPassword}@${proxyHost}:${proxyPort}`
};

(async () => {
  const browser = await chromium.launch({
    headless: true,
    proxy: proxyConfig
  });
  const page = await browser.newPage();
  await page.goto('https://httpbin.org/ip');
  console.log(await page.content());
  await browser.close();
})();
```

--------------------------------

### Golang ScrapeOps Proxy API Integration with JavaScript Rendering

Source: https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/integration-examples/golang-code-example

This Go program demonstrates how to use the ScrapeOps Proxy API to fetch data from a target URL. It specifically shows how to enable JavaScript rendering by adding the 'render_js=true' query parameter to the request. The code includes setting up an HTTP client with a timeout, constructing the request with necessary parameters (API key, URL, render_js), sending the request, and handling the response. It also includes error handling for request creation, execution, and response body reading.

```golang
package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
	"net/url"
	"time"
)

func main() {
	apiKey := "YOUR_API_KEY"
	targetURL := "https://httpbin.org/ip"
	proxyURL := "https://proxy.scrapeops.io/v1/"

	queryParams := url.Values{}
	queryParams.Add("api_key", apiKey)
	queryParams.Add("url", targetURL)
	queryParams.Add("render_js", "true")

	client := &http.Client{
		Timeout: time.Duration(120 * time.Second),
	}

	req, err := http.NewRequest("GET", proxyURL+"?"+queryParams.Encode(), nil)
	if err != nil {
		fmt.Println("Error creating request:", err)
		return
	}

	response, err := client.Do(req)
	if err != nil {
		fmt.Println("Error making request:", err)
		return
	}
	defer response.Body.Close()

	if response.StatusCode == http.StatusOK {
		bodyBytes, err := ioutil.ReadAll(response.Body)
		if err != nil {
			fmt.Println("Error reading response body:", err)
			return
		}
		fmt.Println("Body:", string(bodyBytes))
	} else {
		fmt.Println("Error:", response.StatusCode)
	}
}
```
Tokens:
Raw