Web Scraping FYI

https://webscraping.fyi
Admin
Web Scraping FYI is a comprehensive knowledge base that covers popular technologies, concepts, and...
Tokens:1,642,318
Snippets:8,358
Trust Score:5.5
Update:11 months ago
Show doc for...
Tokens:
Raw
### Make HTTP GET request with Typhoeus and Buzz

Source: https://webscraping.fyi/lib/compare/php-buzz-vs-ruby-typhoeus

Demonstrates how to perform a simple GET request using the Ruby Typhoeus library and the PHP Buzz library. Both examples initialize a client, send a GET request to a URL, and output the response content. No additional configuration is required beyond the basic client setup.

```Ruby
Typhoeus.get("www.example.com")
```

```PHP
use Buzz\Client\;
use Buzz\Message\Request;
use Buzz\Message\Response;

$client = new Curl();
// GET request
$request = new Request('GET', 'http://httpbin.org/get');
$response = new Response();
$client->send($request, $response);
echo $response->getContent();
```

--------------------------------

### Go Web Scraping with Pholcus

Source: https://webscraping.fyi/lib/compare/go-pholcus-vs-javascript-node-crawler

Illustrates a basic web scraping setup using the Pholcus library in Go. It shows how to create a new spider, define a task with a starting URL, add rules for processing routes, and start the crawling process. The example includes a placeholder for a callback function.

```go
package main

import (
    "github.com/henrylee2cn/pholcus/exec"
    _ "github.com/henrylee2cn/pholcus/spider/standard" // standard spider
)

func main() {
    // create spider object
    spider := exec.NewSpider(exec.NewTask("demo", "https://www.example.com"))
    // add a callback for URL route by regex pattern. In this case it's any route: 
    spider.AddRule(".*", "Parse")
    // Start spider
    spider.Start()
}

// define callback here
func Parse(self *exec.Spider, doc *goquery.Document) {
    // callbacks receive HTMl document reference and 
}
```

--------------------------------

### ScrapydWeb - Python Installation and Startup

Source: https://webscraping.fyi/lib/compare/go-colly-vs-python-scrapydweb

Demonstrates how to install and start ScrapydWeb, a web-based management tool for Scrapy spiders. It requires Scrapyd to be running and utilizes the Flask framework for its web interface. The installation is done via pip and the service is started via the command line.

```Python
pip install scrapydweb
scrapydweb
```

--------------------------------

### Perform HTTP GET request using Got (JavaScript) and httpx (Python)

Source: https://webscraping.fyi/lib/compare/javascript-got-vs-python-httpx

Shows how to send a simple GET request and read the response body. Both snippets require the respective library (got or httpx) to be installed. Returns status code and response content.

```JavaScript
const got = require('got');
(async () => {
  const response = await got('https://api.example.com');
  console.log(response.body);
})();
```

```Python
import httpx
response = httpx.get("http://webscraping.fyi/")
print(response.status_code)
print(response.text)
```

--------------------------------

### Make HTTP GET and POST requests using Excon (Ruby) and Buzz (PHP)

Source: https://webscraping.fyi/lib/compare/php-buzz-vs-ruby-excon

These examples illustrate how to perform GET and POST HTTP requests with Excon in Ruby and Buzz in PHP. Both snippets require the respective libraries (Excon gem for Ruby, Buzz package for PHP) and demonstrate setting request bodies, headers, and handling responses. Limitations include reliance on external services for demonstration URLs and the need for proper library installation.

```text
require 'excon'\n\n# GET requests\nresponse = Excon.get('https://jsonplaceholder.typicode.com/posts/1')\nputs response.body\nputs response.status\nputs response.headers\n\n# POST requests\nresponse = Excon.post('https://jsonplaceholder.typicode.com/posts',\n  :body => { :title => 'foo', :body => 'bar', :userId => 1 }.to_json,\n  :headers => { 'Content-Type' => 'application/json' } )\nputs response.body
```

```PHP
use Buzz\\Client\\Curl;\nuse Buzz\\Message\\Request;\nuse Buzz\\Message\\Response;\n\n$client = new Curl();\n// GET request\n$request = new Request('GET', 'http://httpbin.org/get');\n$response = new Response();\n$client->send($request, $response);\necho $response->getContent();\n\n//POST request\n$request = new Request('POST', '/api/resource', 'http://example.com');\n$request->setContent(json_encode(['name' => 'John Doe']));\n// we can also add headers or cookies:\n$request->addHeader('Content-Type: application/json');\n$request->addHeader('Cookie: name=foobar');\n$response = new Response();\n$client->send($request, $response);\necho $response->getContent();\n\n// Buzz also supports http2 (see the 2.0 parameter)\n$response = $client->sendRequest(\n  new Request('GET', 'https://http2.golang.org/serverpush', [], null, '2.0')\n);
```

--------------------------------

### Ruby Faraday HTTP Client GET and POST Examples

Source: https://webscraping.fyi/lib/compare/go-req-vs-ruby-faraday

Demonstrates basic GET requests and persistent session usage with POST requests in Faraday. Shows response handling and request customization through parameters and headers. The examples illustrate both simple one-off requests and reusable connection objects.

```ruby
# GET requests
response = Faraday.get('http://httpbingo.org')
put response.status
put response.headers
put response.body

# or use a persistent client session:
conn = Faraday.new(
  url: 'http://httpbin.org/get',
  params: {param: '1'},
  headers: {'Content-Type' => 'application/json'}
)

# POST requests
response = conn.post('/post') do |req|
  req.params['limit'] = 100
  req.body = {query: 'chunky bacon'}.to_json
end
```

--------------------------------

### HTTP Requests with Go resty (Proxies, Retries, Middleware)

Source: https://webscraping.fyi/lib/compare/go-resty-vs-python-aiohttp

This Go code snippet demonstrates making HTTP requests using the resty library. It includes examples of setting up a client with proxy, configuring retry attempts with custom wait times, and implementing request/response middleware. Ensure resty is installed (`go get github.com/go-resty/resty/v2`).

```go
package main

import (
    "errors"
    "fmt"
    "time"

    "github.com/go-resty/resty/v2"
)

func main() {
    // establish session client
    client := resty.New()
    // set proxy for the session
    client.SetProxy("http://proxyserver:8888")
    // set retries
    client.
        // Set retry count to non zero to enable retries
        SetRetryCount(3).
        // You can override initial retry wait time.
        // Default is 100 milliseconds.
        SetRetryWaitTime(5 * time.Second).
        // MaxWaitTime can be overridden as well.
        // Default is 2 seconds.
        SetRetryMaxWaitTime(20 * time.Second).
        // SetRetryAfter sets callback to calculate wait time between retries.
        // Default (nil) implies exponential backoff with jitter
        SetRetryAfter(func(client *resty.Client, resp *resty.Response) (time.Duration, error) {
            return 0, errors.New("quota exceeded")
        })

    // Make GET request
    resp, err := client.R().
        // we can set query
        SetQueryParams(map[string]string{
            "query": "foo",
        }).
        // and headers
        SetHeader("Accept", "application/json").
        Get("https://httpbin.org/get")

    if err != nil {
        fmt.Println("Error making GET request:", err)
    } else {
        fmt.Println("GET Response Status:", resp.Status())
        fmt.Println("GET Response Body:", resp.String())
    }

    // Make Post request
    resp, err = client.R().
        // JSON data
        SetHeader("Content-Type", "application/json").
        SetBody(`{"username":"testuser", "password":"testpass"}`).
        // or Form Data
        SetFormData(map[string]string{
          "username": "jeeva",
          "password": "mypass",
        }).
        Post("https://httpbin.org/post")

    if err != nil {
        fmt.Println("Error making POST request:", err)
    } else {
        fmt.Println("POST Response Status:", resp.Status())
        fmt.Println("POST Response Body:", resp.String())
    }

    // resty also support request and response middlewares
    // which allow easy modification of outgoing requests and incoming responses
    client.OnBeforeRequest(func(c *resty.Client, req *resty.Request) error {
        // Now you have access to Client and current Request object
        // manipulate it as per your need
        fmt.Println("Executing Before Request Middleware...")
        return nil  // if its success otherwise return error
      })

    // Registering Response Middleware
    client.OnAfterResponse(func(c *resty.Client, resp *resty.Response) error {
        // Now you have access to Client and current Response object
        // manipulate it as per your need
        fmt.Println("Executing After Response Middleware...")
        return nil  // if its success otherwise return error
      })

    // Example of calling with middleware registered (though not strictly necessary to show here)
    _, err = client.R().Get("https://httpbin.org/get")
    if err != nil {
        fmt.Println("Error with middleware test:", err)
    }
}
```

--------------------------------

### GET Request Examples - Excon and Httpful

Source: https://webscraping.fyi/lib/compare/php-httpful-vs-ruby-excon

Demonstrates basic GET requests using Excon Ruby library and Httpful PHP library. Shows how to fetch data from APIs and access response properties. Both libraries provide simple interfaces for HTTP GET operations with minimal configuration.

```ruby
require 'excon'

# GET requests
response = Excon.get('https://jsonplaceholder.typicode.com/posts/1')
puts response.body
puts response.status
puts response.headers
```

```php
require 'vendor/autoload.php';

use Httpful\Request;

// make GET request
$response = \Httpful\Request::get("http://httpbin.org/get")
    ->send();
echo $response->body;
```

--------------------------------

### Make HTTP requests with Got (Node.js) and Buzz (PHP)

Source: https://webscraping.fyi/lib/compare/javascript-got-vs-php-buzz

Demonstrates how to perform GET and POST requests, handle JSON bodies, cookies, and proxy settings using the Got library for Node.js and the Buzz library for PHP. Shows basic setup and response handling. Suitable for web‑scraping tasks.

```JavaScript
const got = require('got');

// GET requests are default and can be made calling the module as is:
const response = await got('https://api.example.com');
console.log(response.body);

// POST requests can send 
const response = await got.post('https://api.example.com', {
    json: { name: 'John Doe' },
});
console.log(response.body);

// handling cookies
import {CookieJar} from 'tough-cookie';

const cookieJar = new CookieJar();

await cookieJar.setCookie('foo=bar', 'https://httpbin.org');
await got('https://httpbin.org/anything', {cookieJar});

// using proxy
import got from 'got';
import {HttpsProxyAgent} from 'hpagent';

await got('https://httpbin.org/ip', {
  agent: {
    https: new HttpsProxyAgent({
      keepAlive: true,
      keepAliveMsecs: 1000,
      maxSockets: 256,
      maxFreeSockets: 256,
      scheduling: 'lifo',
      proxy: 'https://localhost:8080'
    })
  }
});
```

```PHP
use Buzz\Client\Curl;
use Buzz\Message\Request;
use Buzz\Message\Response;

$client = new Curl();
// GET request
$request = new Request('GET', 'http://httpbin.org/get');
$response = new Response();
$client->send($request, $response);
echo $response->getContent();

//POST request
$request = new Request('POST', '/api/resource', 'http://example.com');
$request->setContent(json_encode(['name' => 'John Doe']));
// we can also add headers or cookies:
$request->addHeader('Content-Type: application/json');
$request->addHeader('Cookie: name=foobar');
$response = new Response();
$client->send($request, $response);
echo $response->getContent();

// Buzz also supports http2 (see the 2.0 parameter)
$response = $client->sendRequest(
  new Request('GET', 'https://http2.golang.org/serverpush', [], null, '2.0')
);
```

--------------------------------

### Perform HTTP GET request with Ruby and Python

Source: https://webscraping.fyi/lib/compare/python-requests-vs-ruby-http

Shows how to send a simple HTTP GET request and read the response using Ruby's http gem and Python's requests library. Requires the respective libraries to be installed. Returns the response body, status code, and headers.

```Ruby
require 'http'\n\n# GET request\nresponse = HTTP.get("http://httpbin.org/get")\nputs response.body\nputs response.status\nputs response.headers
```

```Python
import requests\n\n# get request:\nresponse = requests.get("http://webscraping.fyi/")\nresponse.status_code\n200\nresponse.text\n"text"\nresponse.content\nb"bytes"
```

--------------------------------

### Node-Crawler: Basic Usage and Queueing URLs

Source: https://webscraping.fyi/lib/compare/go-geziyor-vs-javascript-node-crawler

Demonstrates the basic setup and usage of the Node-Crawler library. It shows how to initialize a crawler instance, define a callback for processing crawled pages, and queue single URLs, lists of URLs, or even raw HTML content.

```javascript
const Crawler = require('crawler');

const c = new Crawler({
    maxConnections: 10,
    // This will be called for each crawled page
    callback: (error, res, done) => {
        if (error) {
            console.log(error);
        } else {
            const $ = res.$; // $ is Cheerio by default
            console.log($('title').text());
        }
        done();
    }
});

// Queue just one URL, with default callback
c.queue('http://www.amazon.com');

// Queue a list of URLs
c.queue(['http://www.google.com/','http://www.yahoo.com']);

// Queue URLs with custom callbacks & parameters
c.queue([
    {
        uri: 'http://parishackers.org/',
        jQuery: false,
        // The global callback won't be called
        callback: (error, res, done) => {
            if (error) {
                console.log(error);
            } else {
                console.log('Grabbed', res.body.length, 'bytes');
            }
            done();
        }
    }
]);

// Queue some HTML code directly without grabbing (mostly for tests)
c.queue([
    {
        html: '<p>This is a <strong>test</strong></p>'
    }
]);
```

--------------------------------

### Go Resty HTTP Client Examples

Source: https://webscraping.fyi/lib/compare/go-resty-vs-ruby-http

Illustrates various features of the Go Resty client, including setting up a client with proxy and retries, making GET and POST requests with query parameters, headers, and JSON/form data. It also shows how to implement request and response middlewares. Dependencies: The 'resty' library.

```go
package main

import (
	"errors"
	"time"

	"github.com/go-resty/resty/v2"
)

func main() {
	// establish session client
	client := resty.New()
	// set proxy for the session
	client.SetProxy("http://proxyserver:8888")
	// set retries
	client.
		// Set retry count to non zero to enable retries
		SetRetryCount(3).
		// You can override initial retry wait time.
		// Default is 100 milliseconds.
		SetRetryWaitTime(5 * time.Second).
		// MaxWaitTime can be overridden as well.
		// Default is 2 seconds.
		SetRetryMaxWaitTime(20 * time.Second).
		// SetRetryAfter sets callback to calculate wait time between retries.
		// Default (nil) implies exponential backoff with jitter
		SetRetryAfter(func(client *resty.Client, resp *resty.Response) (time.Duration, error) {
			return 0, errors.New("quota exceeded")
		})

	// Make GET request
	resp, err := client.R().
		// we can set query
		SetQueryParams(map[string]string{
			"query": "foo",
		}).
		// and headers
		SetHeader("Accept", "application/json").
		Get("https://httpbin.org/get")

	// Make Post request
	resp, err := client.R().
		// JSON data
		SetHeader("Content-Type", "application/json").
		SetBody(`{"username":"testuser", "password":"testpass"}`).
		// or Form Data
		SetFormData(map[string]string{
		  "username": "jeeva",
		  "password": "mypass",
		}).
		Post("https://httpbin.org/post")

	// resty also support request and response middlewares
	// which allow easy modification of outgoing requests and incoming responses
	client.OnBeforeRequest(func(c *resty.Client, req *resty.Request) error {
		// Now you have access to Client and current Request object
		// manipulate it as per your need

		return nil  // if its success otherwise return error
	  })

	// Registering Response Middleware
	client.OnAfterResponse(func(c *resty.Client, resp *resty.Response) error {
		// Now you have access to Client and current Response object
		// manipulate it as per your need

		return nil  // if its success otherwise return error
	  })

}

```

--------------------------------

### Guzzle PHP HTTP Client Example

Source: https://webscraping.fyi/lib/compare/php-guzzle-vs-python-aiohttp

Illustrates sending an HTTP GET request using the Guzzle HTTP client in PHP. Guzzle simplifies making HTTP requests, handling responses, and integrating with web services. This example assumes Guzzle is installed via Composer.

```php
<?php

require 'vendor/autoload.php'; // Make sure to load Composer's autoloader

use GuzzleHttp\Client;

$client = new Client();

$response = $client->request('GET', 'https://httpbin.org/get');

echo $response->getStatusCode(); // 200
echo $response->getHeaderLine('content-type'); // 'application/json; charset=utf8'
echo $response->getBody(); // '{"args": {}, "headers": { ... }}'
```

--------------------------------

### Symfony Http Client Example: Sending GET and POST Requests

Source: https://webscraping.fyi/lib/compare/php-symfony-http-vs-python-curl-impersonate

Illustrates how to create an HTTP client using Symfony's HttpClient component in PHP. It shows how to send a GET request and a JSON POST request, and how to retrieve the status code and content from the response. This snippet requires the Symfony HttpClient component to be installed.

```php
<?php
use Symfony\Component\HttpClient\HttpClient;

// create a client object: 
$client = HttpClient::create();
// sent GET request
$response = $client->request('GET', 'https://httpbin.org/get');


// or POST request
$response = $client->request('POST', 'https://httpbin.org/post', [
    'headers' => [
        'Content-Type' => 'application/json',
    ],
    'json' => [
        'name' => 'John Doe',
        'email' => 'john.doe@example.com',
    ],
]);

// print response data:
$statusCode = $response->getStatusCode();
$content = $response->getContent();
echo "Status Code: $statusCode\n";
echo "Content: $content\n";

?>
```

--------------------------------

### Basic Web Crawling Setup with Node-Crawler

Source: https://webscraping.fyi/lib/compare/javascript-node-crawler-vs-php-php-spider

This example demonstrates initializing a crawler instance, configuring options like maxConnections, and queuing URLs for scraping. It uses the Crawler library which depends on Node.js and optionally Cheerio for jQuery-like parsing. Inputs are URLs to queue; outputs include extracted data via callbacks, with limitations on handling JavaScript-heavy sites without additional setup.

```JavaScript
const Crawler = require('crawler');

const c = new Crawler({
    maxConnections: 10,
    // This will be called for each crawled page
    callback: (error, res, done) => {
        if (error) {
            console.log(error);
        } else {
            const $ = res.$;
            // $ is Cheerio by default
            //a lean implementation of core jQuery designed specifically for the server
            console.log($('title').text());
        }
        done();
    }
});

// Queue just one URL, with default callback
c.queue('http://www.amazon.com');

// Queue a list of URLs
c.queue(['http://www.google.com/','http://www.yahoo.com']);

// Queue URLs with custom callbacks & parameters
c.queue([{
    uri: 'http://parishackers.org/',
    jQuery: false,

    // The global callback won't be called
    callback: (error, res, done) => {
        if (error) {
            console.log(error);
        } else {
            console.log('Grabbed', res.body.length, 'bytes');
        }
        done();
    }
}]);

// Queue some HTML code directly without grabbing (mostly for tests)
c.queue([{
    html: '<p>This is a <strong>test</strong></p>'
}]);
```

--------------------------------

### Command-line Download with youtube-dl

Source: https://webscraping.fyi/lib/compare/php-embed-vs-python-youtube-dl

Shows the basic command-line usage for youtube-dl to download a video from a given YouTube URL. This is a straightforward way to use the tool without any programming. Ensure youtube-dl is installed and accessible in your system's PATH.

```bash
$ youtube-dl 'https://www.youtube.com/watch?t=4&v=BaW_jenozKc'

```

--------------------------------

### JavaScript (Node.js) HTTP requests with Needle

Source: https://webscraping.fyi/lib/compare/go-req-vs-javascript-needle

Demonstrates Needle for GET and POST, async/await usage, concurrent requests via Promise.all, proxy configuration, and custom headers/cookies. Dependencies: npm install needle.

```javascript
const needle = require('needle');

// Callback-based GET
needle.get('https://httpbin.org/get', (err, res) => {
  if (err) {
    console.error(err);
    return;
  }
  console.log(res.body);
});

// Async/await GET
const response = await needle.get('https://httpbin.org/get');

// Concurrent requests using Promise.all
const results = await Promise.all([
  needle.get('http://httpbin.org/html'),
  needle.get('http://httpbin.org/html'),
  needle.get('http://httpbin.org/html'),
]);

// POST with JSON body
const data = { name: 'John Doe' };
await needle.post('https://api.example.com', data);

// Proxy
const options = {
  proxy: 'http://proxy.example.com:8080',
};
await needle.get('https://httpbin.org/ip', options);

// Headers and cookies
const options = {
  headers: {
    Cookie: 'myCookie=123',
    'X-My-Header': 'myValue',
  },
};
await needle.get('https://httpbin.org/headers', options);
```

--------------------------------

### hrequests Python HTTP Client Example

Source: https://webscraping.fyi/lib/compare/go-req-vs-python-hrequests

Demonstrates basic HTTP GET requests and advanced usage with headless browsers and asyncio concurrency using the hrequests library in Python. It requires the hrequests library to be installed.

```python
import hrequests

# perform HTTP client requests
resp = hrequests.get('https://httpbin.org/html')
print(resp.status_code)
# 200

# use headless browsers and sessions:
session = hrequests.Session('chrome', version=122, os="mac")

# supports asyncio and easy concurrency
requests = [
    hrequests.async_get('https://www.google.com/', browser='firefox'),
    hrequests.async_get('https://www.duckduckgo.com/'),
    hrequests.async_get('https://www.yahoo.com/'),
    hrequests.async_get('https://www.httpbin.org/'),
]
responses = hrequests.map(requests, size=3)  # max 3 conccurency

```

--------------------------------

### PHP Example: Basic Crawling with Geziyor

Source: https://webscraping.fyi/lib/compare/go-geziyor-vs-php-crwlr-crawler

This PHP example demonstrates how to use the Geziyor framework to crawl a website and extract data. It shows how to set up a crawler, follow links, and parse the HTML content to extract the page title and content.

```php
require_once 'vendor/autoload.php';

use Crwlr\Crawler;

$crawler = new Crawler();
$crawler->get('https://example.com', ['User-Agent' => 'webscraping.fyi']);

// more links can be followed:
$crawler->followLinks();

// and current page can be parsed:
$response = $crawler->response();
$title = $crawler->filter('title')->text();
echo $response->getContent();
```

--------------------------------

### Pholcus Go Web Crawler Example

Source: https://webscraping.fyi/lib/compare/go-pholcus-vs-r-ralger

Demonstrates a basic Pholcus web crawler in Go. It shows how to create a spider, add URL routing rules, and start the crawling process, with a placeholder for parsing callback functions.

```Go
package main

import (
    "github.com/henrylee2cn/pholcus/exec"
    _ "github.com/henrylee2cn/pholcus/spider/standard" // standard spider
)

func main() {
    // create spider object
    spider := exec.NewSpider(exec.NewTask("demo", "https://www.example.com"))
    // add a callback for URL route by regex pattern. In this case it's any route:
    spider.AddRule(".*", "Parse")
    // Start spider
    spider.Start()
}

// define callback here
func Parse(self *exec.Spider, doc *goquery.Document) {
```

--------------------------------

### Send HTTP GET and POST requests using Buzz in PHP

Source: https://webscraping.fyi/lib/compare/javascript-needle-vs-php-buzz

Shows how to use the Buzz library with the Curl client in PHP to perform GET and POST requests, set request bodies, add headers and cookies, and execute an HTTP/2 request. Requires Buzz installed via Composer and the curl extension. Responses are retrieved via getContent().

```PHP
use Buzz\\Client\\Curl;
use Buzz\\Message\\Request;
use Buzz\\Message\\Response;

$client = new Curl();
// GET request
$request = new Request('GET', 'http://httpbin.org/get');
$response = new Response();
$client->send($request, $response);
echo $response->getContent();

//POST request
$request = new Request('POST', '/api/resource', 'http://example.com');
$request->setContent(json_encode(['name' => 'John Doe']));
// we can also add headers or cookies:
$request->addHeader('Content-Type: application/json');
$request->addHeader('Cookie: name=foobar');
$response = new Response();
$client->send($request, $response);
echo $response->getContent();

// Buzz also supports http2 (see the 2.0 parameter)
$response = $client->sendRequest(
  new Request('GET', 'https://http2.golang.org/serverpush', [], null, '2.0')
);
```

--------------------------------

### Photon Python Web Scraping Examples

Source: https://webscraping.fyi/lib/compare/go-geziyor-vs-python-photon

Demonstrates how to use the Photon Python library for web scraping. It covers creating a Photon instance, extracting data from a single URL using a CSS selector, and performing asynchronous data extraction from multiple URLs. No external dependencies beyond the Photon library are explicitly mentioned for these basic examples.

```python
from photon import Photon

#Create a new Photon instance
ph = Photon()

#Extract data from a specific element of the website
url = "https://www.example.com"
selector = "div.main"
data = ph.get_data(url, selector)

#Print the extracted data
print(data)


#Extract data from multiple websites asynchronously
urls = ["https://www.example1.com", "https://www.example2.com"]
data = ph.get_data_async(urls)

```

--------------------------------

### Ruby HTTP/2 Client Example

Source: https://webscraping.fyi/lib/compare/go-req-vs-ruby-http-2

Demonstrates making GET and POST requests using the http-2 gem in Ruby. This library provides a pure Ruby implementation of the HTTP/2 protocol, supporting binary framing, stream multiplexing, flow control, and header compression.

```ruby
require 'http/2'

# GET request
client = HTTP2::Client.new
response = client.get("https://httpbin.org/get")
puts response.body

# POST reuqest
data = { name: "value" }
response = client.post("https://www.example.com", data)

```

--------------------------------

### Perform HTTP requests with hrequests (Python) and httr (R)

Source: https://webscraping.fyi/lib/compare/python-hrequests-vs-r-httr

This snippet shows how to make GET and POST calls using hrequests in Python and httr in R. It includes examples of session creation, asynchronous requests in Python, and cookie handling in R. Both libraries require their respective packages (hrequests for Python, httr for R) to be installed.

```Python
import hrequests

# perform HTTP client requests
resp = hrequests.get('https://httpbin.org/html')
print(resp.status_code)
# 200

# use headless browsers and sessions:
session = hrequests.Session('chrome', version=122, os=\"mac\")

# supports asyncio and easy concurrency
requests = [
    hrequests.async_get('https://www.google.com/', browser='firefox'),
    hrequests.async_get('https://www.duckduckgo.com/'),
    hrequests.async_get('https://www.yahoo.com/'),
    hrequests.async_get('https://www.httpbin.org/'),
]
responses = hrequests.map(requests, size=3)  # max 3 conccurency
```

```R
library(httr)

# GET requests:
resp <- GET(\"http://httpbin.org/get\")
status_code(resp)  # status code
headers(resp)  # headers
str(content(resp))  # body

# POST requests: 
# Form encoded
resp <- POST(url, body = body, encode = \"form\")
# Multipart encoded
resp <- POST(url, body = body, encode = \"multipart\")
# JSON encoded
resp <- POST(url, body = body, encode = \"json\")

# setting cookies:
resp <- GET(\"http://httpbin.org/cookies\", set_cookies(\"MeWant\" = \"cookies\"))
content(r)$cookies  # get response cookies
```

--------------------------------

### Handle JSON Responses Automatically (Python)

Source: https://webscraping.fyi/lib/compare/python-aiohttp-vs-python-requests

This example illustrates how the 'requests' library can automatically parse JSON responses into Python dictionaries. It makes a GET request to a JSON endpoint and prints the resulting dictionary. Ensure the 'requests' library is installed.

```Python
import requests

response = requests.get("http://httpbin.org/json")
print(response.json())
```

--------------------------------

### Pholcus Web Crawler Basic Usage (Go)

Source: https://webscraping.fyi/lib/compare/go-geziyor-vs-go-pholcus

This Go code snippet demonstrates the basic usage of the Pholcus web crawler library. It shows how to initialize a new spider, add a rule to process URLs matching a pattern, and start the crawling process. Dependencies include the Pholcus library and its standard spider module.

```go
package main

import (
    "github.com/henrylee2cn/pholcus/exec"
    _ "github.com/henrylee2cn/pholcus/spider/standard" // standard spider
)

func main() {
    // create spider object
    spider := exec.NewSpider(exec.NewTask("demo", "https://www.example.com"))
    // add a callback for URL route by regex pattern. In this case it's any route:
    spider.AddRule(".*", "Parse")
    // Start spider
    spider.Start()
}

// define callback here
func Parse(self *exec.Spider, doc *goquery.Document) {
    // callbacks receive HTMl document reference and 
}
```

--------------------------------

### Perform asynchronous HTTP requests with aiohttp (Python)

Source: https://webscraping.fyi/lib/compare/python-aiohttp-vs-python-curl-impersonate

Demonstrates creating an asynchronous client session, sending GET, POST (JSON and form) requests, parsing JSON responses, and handling WebSocket communication. Requires aiohttp library installed and Python 3.7+ for async/await syntax.

```python
import asyncio
from aiohttp import ClientSession, WSMsgType

async def run():
    async with ClientSession(headers={"User-Agent": "webscraping.fyi"}) as session:
        response = await session.get("http://httpbin.org/headers")
        print(response.status)
        print(await response.text())

        resp = await session.post("http://httpbin.org/post", json={"key": "value"})
        resp = await session.post("http://httpbin.org/post", data={"key": "value"})
        resp = await session.get("http://httpbin.org/json")
        data = await resp.json()
        print(data)

        async with session.ws_connect("http://example.org/ws") as ws:
            async for msg in ws:
                if msg.type == WSMsgType.TEXT:
                    if msg.data == "close cmd":
                        await ws.close()
                        break
                    else:
                        await ws.send_str(msg.data + "/answer")
                elif msg.type == WSMsgType.ERROR:
                    break

asyncio.run(run())
```

--------------------------------

### Configure and use Resty HTTP client in Go

Source: https://webscraping.fyi/lib/compare/go-req-vs-go-resty

Demonstrates creating a Resty client, setting a proxy, configuring retry behavior, making GET and POST requests, and adding request and response middlewares. Requires the Resty library and Go's standard packages. Returns response objects and errors for each request.

```Go
package main\n\n// establish session client\nclient := resty.New()\n// set proxy for the session\nclient.SetProxy(\"http://proxyserver:8888\")\n// set retries\nclient.\n    // Set retry count to non zero to enable retries\n    SetRetryCount(3).\n    // You can override initial retry wait time.\n    // Default is 100 milliseconds.\n    SetRetryWaitTime(5 * time.Second).\n    // MaxWaitTime can be overridden as well.\n    // Default is 2 seconds.\n    SetRetryMaxWaitTime(20 * time.Second).\n    // SetRetryAfter sets callback to calculate wait time between retries.\n    // Default (nil) implies exponential backoff with jitter\n    SetRetryAfter(func(client *resty.Client, resp *resty.Response) (time.Duration, error) {\n        return 0, errors.New(\"quota exceeded\")\n    })\n\n// Make GET request\nresp, err := client.R().\n    // we can set query\n    SetQueryParams(map[string]string{\n        \"query\": \"foo\",\n    }).\n    // and headers\n    SetHeader(\"Accept\", \"application/json\").\n    Get(\"https://httpbin.org/get\")\n\n// Make Post request\nresp, err = client.R().\n    // JSON data\n    SetHeader(\"Content-Type\", \"application/json\").\n    SetBody(`{\"username\":\"testuser\", \"password\":\"testpass\"}`).\n    // or Form Data\n    SetFormData(map[string]string{\n      \"username\": \"jeeva\",\n      \"password\": \"mypass\",\n    }).\n    Post(\"https://httpbin.org/post\")\n\n// resty also support request and response middlewares\n// which allow easy modification of outgoing requests and incoming responses\nclient.OnBeforeRequest(func(c *resty.Client, req *resty.Request) error {\n    // Now you have access to Client and current Request object\n    // manipulate it as per your need\n\n    return nil  // if its success otherwise return error\n  })\n\n// Registering Response Middleware\nclient.OnAfterResponse(func(c *resty.Client, resp *resty.Response) error {\n    // Now you have access to Client and current Response object\n    // manipulate it as per your need\n\n    return nil  // if its success otherwise return error\n  })
```

--------------------------------

### Make HTTP requests with httpx

Source: https://webscraping.fyi/lib/compare/python-httpx-vs-r-crul

Examples showing how to use httpx for basic HTTP operations including GET requests, JSON handling, and POST requests. Supports both sync and async operations with HTTP/2 support. Requires Python 3 and httpx library installation.

```python
import httpx

# Just like requests httpx can be used directly
response = httpx.get("http://webscraping.fyi/")
response.status_code
200
response.text
"text"
response.content
b"bytes"

# HTTP2 needs to be enabled explicitly and is recommended for web scraping:
response = httpx.get("http://webscraping.fyi/", http2=True)

# httpx can automatically convert json responses to Python dictionaries:
response = httpx.get("http://httpbin.org/json")
print(response.json())
{'slideshow': {'author': 'Yours Truly', 'date': 'date of publication', 'slides': [{'title': 'Wake up to WonderWidgets!', 'type': 'all'}, {'items': ['Why <em>WonderWidgets</em> are great', 'Who <em>buys</em> WonderWidgets'], 'title': 'Overview', 'type': 'all'}], 'title': 'Sample Slide Show'}}

# for POST request it can ingest Python's dictionaries as JSON:
response = requests.post("http://httpbin.org/post", json={"query": "hello world"})
# or form data:
response = requests.post("http://httpbin.org/post", data={"query": "hello world"})
```

--------------------------------

### Make HTTP requests with Got (JavaScript)

Source: https://webscraping.fyi/lib/compare/go-resty-vs-javascript-got

Examples showing how to make GET and POST requests using the Got HTTP client. Includes handling JSON payloads, cookies with tough-cookie, and using HTTPS proxies with hpagent. Requires got, tough-cookie, and hpagent dependencies.

```javascript
const got = require('got');

// GET requests are default and can be made calling the module as is:
const response = await got('https://api.example.com');
console.log(response.body);

// POST requests can send 
const response = await got.post('https://api.example.com', {
    json: { name: 'John Doe' },
});
console.log(response.body);

// handling cookies
import {CookieJar} from 'tough-cookie';

const cookieJar = new CookieJar();

await cookieJar.setCookie('foo=bar', 'https://httpbin.org');
await got('https://httpbin.org/anything', {cookieJar});

// using proxy
import got from 'got';
import {HttpsProxyAgent} from 'hpagent';

await got('https://httpbin.org/ip', {
  agent: {
    https: new HttpsProxyAgent({
      keepAlive: true,
      keepAliveMsecs: 1000,
      maxSockets: 256,
      maxFreeSockets: 256,
      scheduling: 'lifo',
      proxy: 'https://localhost:8080'
    })
  }
});
```

--------------------------------

### Pholcus Web Crawler Example in Go

Source: https://webscraping.fyi/lib/compare/go-colly-vs-go-pholcus

Illustrates how to set up and run a basic web crawler using the Pholcus library in Go. It involves creating a new spider instance with a task, defining a rule to specify URL patterns for parsing, and then starting the spider. A placeholder callback function `Parse` is defined to handle the scraped data.

```go
package main

import (
    "github.com/henrylee2cn/pholcus/exec"
    _ "github.com/henrylee2cn/pholcus/spider/standard" // standard spider
)

func main() {
    // create spider object
    spider := exec.NewSpider(exec.NewTask("demo", "https://www.example.com"))
    // add a callback for URL route by regex pattern. In this case it's any route:
    spider.AddRule(".*", "Parse")
    // Start spider
    spider.Start()
}

// define callback here
func Parse(self *exec.Spider, doc *goquery.Document) {
    // callbacks receive HTMl document reference and 
}

```

--------------------------------

### Perform HTTP GET and POST requests with PHP HTTP clients (Buzz & Httpful)

Source: https://webscraping.fyi/lib/compare/php-buzz-vs-php-httpful

Shows how to use the Buzz and Httpful libraries in PHP to send HTTP GET and POST requests, set JSON bodies, add custom headers, and enable HTTP/2 where supported. Requires the respective libraries installed via Composer. Returns response content as strings. Limitations include library-specific features and PHP version compatibility.

```PHP
use Buzz\Client\Curl;
use Buzz\Message\Request;
use Buzz\Message\Response;

$client = new Curl();
// GET request
$request = new Request('GET', 'http://httpbin.org/get');
$response = new Response();
$client->send($request, $response);
echo $response->getContent();

//POST request
$request = new Request('POST', '/api/resource', 'http://example.com');
$request->setContent(json_encode(['name' => 'John Doe']));
// we can also add headers or cookies:
$request->addHeader('Content-Type: application/json');
$request->addHeader('Cookie: name=foobar');
$response = new Response();
$client->send($request, $response);
echo $response->getContent();

// Buzz also supports http2 (see the 2.0 parameter)
$response = $client->sendRequest(
  new Request('GET', 'https://http2.golang.org/serverpush', [], null, '2.0')
);
```

```PHP
require 'vendor/autoload.php';

use Httpful\Request;

// make GET request
$response = \Httpful\Request::get("http://httpbin.org/get")
    ->send();
echo $response->body;

// make POST request
$data = array('name' => 'Bob', 'age' => 35);
$response = \Httpful\Request::post("http://httpbin.org/post")
    ->sendsJson()
    ->body(json_encode($data))
    ->send();
echo $response->body;

// add headers or cookies
$response = \Httpful\Request::get("http://httpbin.org/headers")
    ->addHeader("API-KEY", "mykey")
    ->addHeader("Cookie", "foo=bar")
    ->send();
echo $response->body;
```
Tokens:
Raw