### GET Request Examples

Source: https://github.com/d4vinci/scrapling/blob/main/docs/cli/extract-commands.md

Demonstrates various ways to perform a GET request, including basic downloads, custom timeouts, CSS selectors, cookies, and custom headers.

```bash
# Basic download
scrapling extract get "https://news.site.com" news.md
```

```bash
# Download with custom timeout
scrapling extract get "https://example.com" content.txt --timeout 60
```

```bash
# Extract only specific content using CSS selectors
scrapling extract get "https://blog.example.com" articles.md --css-selector "article"
```

```bash
# Send a request with cookies
scrapling extract get "https://scrapling.requestcatcher.com" content.md --cookies "session=abc123; user=john"
```

```bash
# Add user agent
scrapling extract get "https://api.site.com" data.json -H "User-Agent: MyBot 1.0"
```

```bash
# Add multiple headers
scrapling extract get "https://site.com" page.html -H "Accept: text/html" -H "Accept-Language: en-US"
```

--------------------------------

### Basic GET Request Example

Source: https://github.com/d4vinci/scrapling/blob/main/docs/fetching/static.md

Demonstrates how to perform a basic GET request using the Fetcher class. Ensure the Fetcher class is imported before use.

```python
from scrapling.fetchers import Fetcher

# Example usage (assuming 'fetcher' is an instance of Fetcher)
# fetcher.get('https://example.com')
```

--------------------------------

### API Integration and Testing Examples

Source: https://github.com/d4vinci/scrapling/blob/main/docs/cli/interactive-shell.md

Demonstrates testing API endpoints using GET, POST, and PUT requests with the `get`, `post`, and `put` functions.

```python
>>> # Test API endpoints interactively
>>> response = get('https://jsonplaceholder.typicode.com/posts/1')
>>> response.json()
{'userId': 1, 'id': 1, 'title': 'sunt aut...', 'body': 'quia et...'}

>>> # Test POST requests
>>> new_post = post('https://jsonplaceholder.typicode.com/posts', 
...                 json={'title': 'Test Post', 'body': 'Test content', 'userId': 1})
>>> new_post.json()['id']
101

>>> # Test with different data
>>> updated = put(f'https://jsonplaceholder.typicode.com/posts/{new_post.json()["id"]}',
...               json={'title': 'Updated Title'})
```

--------------------------------

### Enable uvloop for Faster Event Loop

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/spiders/advanced.md

Pass `use_uvloop=True` to the `start()` method to leverage the faster uvloop/winloop event loop implementation for potentially improved I/O-bound crawl throughput. Ensure the respective library is installed.

```python
result = MySpider().start(use_uvloop=True)
```

--------------------------------

### Scrapling extract get command examples

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/SKILL.md

Examples of using the `scrapling extract get` command to download web content in different formats. These demonstrate saving raw HTML, Markdown, or plain text.

```bash
scrapling extract get "https://blog.example.com" article.md
```

```bash
scrapling extract get "https://example.com" page.html
```

```bash
scrapling extract get "https://example.com" content.txt
```

--------------------------------

### Basic HTTP GET Request with Fetcher

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/fetching/static.md

A simple example of making a GET request using the Fetcher class and extracting data from the response. Use this for single, straightforward HTTP requests.

```python
from scrapling.fetchers import Fetcher

# Make a request
page = Fetcher.get('https://example.com')

# Check the status
if page.status == 200:
    # Extract title
    title = page.css('title::text').get()
    print(f"Page title: {title}")

    # Extract all links
    links = page.css('a::attr(href)').getall()
    print(f"Found {len(links)} links")
```

--------------------------------

### Install Scrapling

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/examples/README.md

Install Scrapling with all dependencies and force an update if necessary.

```bash
pip install "scrapling[all]>=0.4.9"
scrapling install --force
```

--------------------------------

### Install Scrapling with All Features

Source: https://github.com/d4vinci/scrapling/blob/main/docs/index.md

Install Scrapling with all available features, including fetchers, AI, and shell functionalities.

```bash
pip install "scrapling[all]"
```

--------------------------------

### Install Camoufox and Dependencies

Source: https://github.com/d4vinci/scrapling/blob/main/docs/fetching/stealthy.md

Install the Camoufox library, Playwright browser dependencies, and run the Camoufox fetch command.

```commandline
pip install camoufox
playwright install-deps firefox
camoufox fetch
```

--------------------------------

### Install Dependencies and Build Documentation

Source: https://github.com/d4vinci/scrapling/blob/main/CONTRIBUTING.md

Installs Zensical and project-specific documentation requirements, then builds the static documentation site. Ensure you are in the project root directory.

```bash
pip install zensical
pip install -r docs/requirements.txt
zensical build --clean  # Build the static site
```

--------------------------------

### Set Up Virtual Environment and Install Dependencies

Source: https://github.com/d4vinci/scrapling/blob/main/CONTRIBUTING.md

Create a Python virtual environment and install Scrapling with all development dependencies, along with specific test requirements.

```bash
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
pip install -e ".[all]"
pip install -r tests/requirements.txt
```

--------------------------------

### Install Browser Dependencies

Source: https://github.com/d4vinci/scrapling/blob/main/CONTRIBUTING.md

Install necessary browser dependencies for Scrapling to function correctly.

```bash
scrapling install
```

--------------------------------

### Synchronous StealthySession with Camoufox

Source: https://github.com/d4vinci/scrapling/blob/main/docs/fetching/stealthy.md

Inherit from StealthySession and configure Camoufox launch options within the start method for synchronous fetching. This example demonstrates setting up browser preferences for caching.

```python
from scrapling.fetchers import StealthySession
from playwright.sync_api import sync_playwright
from camoufox.utils import launch_options as generate_launch_options

class StealthySession(StealthySession):
    def start(self):
        """Create a browser for this instance and context."""
        if not self.playwright:
            self.playwright = sync_playwright().start()
            # Configure camoufox run options here
            launch_options = generate_launch_options(**{"headless": True, "user_data_dir": ''})
            # Here's an example, part of what we have been doing before v0.3.13
            launch_options = generate_launch_options(**{
                "geoip": False,
                "proxy": self._config.proxy,
                "headless": self._config.headless,
                "humanize": True if self._config.solve_cloudflare else False,  # Better enable humanize for Cloudflare, otherwise it's up to you
                "i_know_what_im_doing": True,  # To turn warnings off with the user configurations
                "allow_webgl": self._config.allow_webgl,
                "block_webrtc": self._config.block_webrtc,
                "os": None,
                "user_data_dir": self._config.user_data_dir,
                "firefox_user_prefs": {
                    # This is what enabling `enable_cache` does internally, so we do it from here instead
                    "browser.sessionhistory.max_entries": 10,
                    "browser.sessionhistory.max_total_viewers": -1,
                    "browser.cache.memory.enable": True,
                    "browser.cache.disk_cache_ssl": True,
                    "browser.cache.disk.smart_size.enabled": True,
                },
                # etc...
            })
            self.context = self.playwright.firefox.launch_persistent_context(**launch_options)
        else:
            raise RuntimeError("Session has been already started")
```

--------------------------------

### Serve Documentation Locally

Source: https://github.com/d4vinci/scrapling/blob/main/CONTRIBUTING.md

Starts a local development server to preview the documentation. This command should be run after building the documentation.

```bash
zensical serve          # Local preview
```

--------------------------------

### Install Scrapling Base Package

Source: https://github.com/d4vinci/scrapling/blob/main/docs/index.md

Use this command to install the core Scrapling parser engine and its dependencies. This installation does not include fetchers or command-line tools.

```bash
pip install scrapling
```

--------------------------------

### Install Scrapling with all dependencies

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/SKILL.md

Install Scrapling and all its optional dependencies using pip. This command ensures all features are available for use.

```bash
pip install "scrapling[all]>=0.4.9"
```

--------------------------------

### Basic Fetcher Usage

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/fetching/choosing.md

Use a fetcher class directly to fetch a webpage. This example uses StealthyFetcher to get the content of a URL.

```python
page = StealthyFetcher.fetch('https://example.com')
```

--------------------------------

### Install Scrapling with Shell Features

Source: https://github.com/d4vinci/scrapling/blob/main/docs/index.md

Install Scrapling with shell features, including the Web Scraping shell and the `extract` command.

```bash
pip install "scrapling[shell]"
```

--------------------------------

### Initialize and Serve Scrapling MCP Server in Python

Source: https://github.com/d4vinci/scrapling/blob/main/docs/api-reference/mcp-server.md

Import the ScraplingMCPServer class to programmatically start the server instance.

```python
from scrapling.core.ai import ScraplingMCPServer

server = ScraplingMCPServer()
server.serve(http=False, host="0.0.0.0", port=8000)
```

--------------------------------

### Install Scrapling with MCP Support

Source: https://github.com/d4vinci/scrapling/blob/main/docs/ai/mcp-server.md

Install Scrapling with MCP server dependencies and browser dependencies using pip and the scrapling install command.

```bash
# Install Scrapling with MCP server dependencies
pip install "scrapling[ai]"

# Install browser dependencies
scrapling install
```

--------------------------------

### Install Scrapling with Fetchers

Source: https://github.com/d4vinci/scrapling/blob/main/docs/index.md

Install Scrapling along with fetcher dependencies to enable web scraping functionalities. This command also installs necessary browser and fingerprint manipulation dependencies.

```bash
pip install "scrapling[fetchers]"
```

```bash
scrapling install
```

```bash
scrapling install  --force
```

--------------------------------

### Run Fetcher Session Example

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/examples/README.md

Execute the Python script for FetcherSession to perform persistent HTTP requests.

```bash
python examples/01_fetcher_session.py
```

--------------------------------

### Install Scrapling with AI Features

Source: https://github.com/d4vinci/scrapling/blob/main/docs/index.md

Install Scrapling with the MCP server feature, which enables AI-related functionalities.

```bash
pip install "scrapling[ai]"
```

--------------------------------

### Scraping Escalation Guide

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/examples/README.md

A guide to choosing the right Scrapling tool based on the complexity of the website and potential blocking mechanisms.

```text
get / FetcherSession
  └─ If JS required → fetch / DynamicSession
       └─ If blocked → stealthy-fetch / StealthySession
            └─ If multi-page → Spider
```

--------------------------------

### Install Beta Version from Dev Branch

Source: https://github.com/d4vinci/scrapling/blob/main/CONTRIBUTING.md

Install the beta version of Scrapling directly from the development branch using pip.

```commandline
pip3 install git+https://github.com/D4Vinci/Scrapling.git@dev
```

--------------------------------

### Start Scrapling MCP Server (STDIO Transport)

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/mcp-server.md

Use this command to start the Scrapling MCP server with standard input/output transport, which is commonly used by MCP clients.

```bash
scrapling mcp
```

--------------------------------

### GET Request with Parameters

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/fetching/static.md

Send a GET request with URL parameters. The `params` argument should be a dictionary.

```python
page = Fetcher.get('https://example.com/search', params={'q': 'query'})
```

--------------------------------

### Run a Scrapling Spider

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/spiders/getting-started.md

Instantiate a spider and call its start() method to initiate the crawling process. The start() method handles all asynchronous operations internally.

```python
result = QuotesSpider().start()
```

--------------------------------

### Scrapling Extract Delete Examples

Source: https://github.com/d4vinci/scrapling/blob/main/docs/cli/extract-commands.md

Examples demonstrating how to send data and JSON data using the delete command. The output is saved to the specified file.

```bash
# Send data
scrapling extract delete "https://scrapling.requestcatcher.com/delete" results.html
```

```bash
# Send JSON data
scrapling extract delete "https://scrapling.requestcatcher.com/" response.txt --impersonate "chrome"
```

--------------------------------

### Run a Scrapling Spider and Access Results

Source: https://github.com/d4vinci/scrapling/blob/main/docs/spiders/getting-started.md

Instantiate a spider and call its start() method to initiate the crawl. The start() method handles asynchronous operations internally. Access scraped items and crawl statistics from the returned CrawlResult object.

```python
result = QuotesSpider().start()
```

```python
result = QuotesSpider().start()

# Access scraped items
for item in result.items:
    print(item["text"], "- ", item["author"])

# Check statistics
print(f"Scraped {result.stats.items_scraped} items")
print(f"Made {result.stats.requests_count} requests")
print(f"Took {result.stats.elapsed_seconds:.1f} seconds")

# Did the crawl finish or was it paused?
print(f"Completed: {result.completed}")
```

--------------------------------

### Install Scrapling Fetchers Programmatically

Source: https://github.com/d4vinci/scrapling/blob/main/docs/index.md

Use this Python code to install fetcher dependencies programmatically. Set standalone_mode to False to integrate into existing applications.

```python
from scrapling.cli import install

install([], standalone_mode=False)          # normal install
install(["--force"], standalone_mode=False) # force reinstall
```

--------------------------------

### Run Spider Example

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/examples/README.md

Execute the Python script for Spider, which auto-crawls all pages and exports data to quotes.json.

```bash
python examples/04_spider.py           # Auto-crawls all pages, exports quotes.json
```

--------------------------------

### GET Requests

Source: https://github.com/d4vinci/scrapling/blob/main/docs/fetching/static.md

Demonstrates various ways to perform GET requests, including with parameters, custom headers, basic authentication, browser impersonation, and HTTP/3 support.

```APIDOC
## GET Requests

### Description
Perform GET requests to fetch data from a given URL. Supports various options for customization.

### Synchronous Example
```python
from scrapling.fetchers import Fetcher

# Basic GET
page = Fetcher.get('https://example.com')
page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
page = Fetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')

# With parameters
page = Fetcher.get('https://example.com/search', params={'q': 'query'})

# With headers
page = Fetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})

# Basic HTTP authentication
page = Fetcher.get("https://example.com", auth=("my_user", "password123"))

# Browser impersonation
page = Fetcher.get('https://example.com', impersonate='chrome')

# HTTP/3 support
page = Fetcher.get('https://example.com', http3=True)
```

### Asynchronous Example
```python
from scrapling.fetchers import AsyncFetcher

# Basic GET
page = await AsyncFetcher.get('https://example.com')
page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')

# With parameters
page = await AsyncFetcher.get('https://example.com/search', params={'q': 'query'})

# With headers
page = await AsyncFetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})

# Basic HTTP authentication
page = await AsyncFetcher.get("https://example.com", auth=("my_user", "password123"))

# Browser impersonation
page = await AsyncFetcher.get('https://example.com', impersonate='chrome110')

# HTTP/3 support
page = await AsyncFetcher.get('https://example.com', http3=True)
```

### Response Object
The `page` object returned is a [Response](choosing.md#response-object) object, which is also a [Selector](../parsing/main_classes.md#selector), allowing direct use of CSS selectors.

```python
page.css('.something.something')

page = Fetcher.get('https://api.github.com/events')
print(page.json())
```
```

--------------------------------

### Fetcher GET Method

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/fetching/static.md

Demonstrates various ways to use the Fetcher.get() method for making GET requests, including basic usage, with parameters, custom headers, authentication, browser impersonation, and HTTP/3 support.

```APIDOC
## GET

### Description
Fetches a resource from the specified URL using the GET HTTP method. Supports various configurations for headers, proxies, authentication, and browser impersonation.

### Method
GET

### Endpoint
/

### Parameters
#### Shared Arguments
- **url** (string) - Required - The targeted URL.
- **stealthy_headers** (boolean) - Optional - If enabled (default), it creates and adds real browser headers and sets a Google referer header.
- **follow_redirects** (string or boolean) - Optional - Controls redirect behavior. Defaults to "safe". Can be `True` to follow all redirects, or `False` to disable redirects.
- **timeout** (integer) - Optional - The number of seconds to wait for each request to be finished. Defaults to 30 seconds.
- **retries** (integer) - Optional - The number of retries for failed requests. Defaults to three retries.
- **retry_delay** (integer) - Optional - Number of seconds to wait between retry attempts. Defaults to 1 second.
- **impersonate** (string or list of strings) - Optional - Impersonate specific browsers' TLS fingerprints. Defaults to the latest available Chrome version.
- **http3** (boolean) - Optional - Use HTTP/3 protocol for requests. Defaults to False.
- **cookies** (dict or list of dicts) - Optional - Cookies to use in the request.
- **proxy** (string) - Optional - The proxy for this request. Format: `http://username:password@localhost:8030`.
- **proxy_auth** (tuple) - Optional - HTTP basic auth for proxy, tuple of (username, password).
- **proxies** (dict) - Optional - Dict of proxies to use. Format: `{"http": proxy_url, "https": proxy_url}`.
- **proxy_rotator** (ProxyRotator instance) - Optional - Automatic proxy rotation. Cannot be combined with `proxy` or `proxies`.
- **headers** (dict) - Optional - Headers to include in the request. Can override any header generated by `stealthy_headers`.
- **max_redirects** (integer) - Optional - Maximum number of redirects. Defaults to 30, use -1 for unlimited.
- **verify** (boolean) - Optional - Whether to verify HTTPS certificates. Defaults to True.
- **cert** (tuple) - Optional - Tuple of (cert, key) filenames for the client certificate.
- **selector_config** (dict) - Optional - Custom parsing arguments for `Selector`/`Response`.

#### Query Parameters
- **params** (dict) - Optional - Parameters to append to the URL for GET requests.

#### Request Body
Not applicable for GET requests.

### Request Example
```python
from scrapling.fetchers import Fetcher

# Basic GET
page = Fetcher.get('https://example.com')
page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
page = Fetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')

# With parameters
page = Fetcher.get('https://example.com/search', params={'q': 'query'})

# With headers
page = Fetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})

# Basic HTTP authentication
page = Fetcher.get("https://example.com", auth=("my_user", "password123"))

# Browser impersonation
page = Fetcher.get('https://example.com', impersonate='chrome')

# HTTP/3 support
page = Fetcher.get('https://example.com', http3=True)
```

### Response
#### Success Response (200)
- **page** (object) - The response object containing page content and metadata.
```

--------------------------------

### Set Up Pre-commit Hooks

Source: https://github.com/d4vinci/scrapling/blob/main/CONTRIBUTING.md

Install and set up pre-commit hooks to ensure code quality and consistency before committing changes.

```bash
pip install pre-commit
pre-commit install
```

--------------------------------

### Async Pre-Navigation Event Listeners

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/fetching/dynamic.md

The asynchronous version of `page_setup` for use with `async_fetch`. The provided function must be an async function.

```python
from playwright.async_api import Page

async def capture_websockets(page: Page):
    page.on("websocket", lambda ws: print(f"WebSocket opened: {ws.url}"))

page = await DynamicFetcher.async_fetch('https://example.com', page_setup=capture_websockets)
```

--------------------------------

### Get Ancestor Path

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/parsing/main_classes.md

Retrieves a list of all ancestor elements in the DOM tree, starting from the parent and going up to the root. Each ancestor is represented with its data and parent context.

```python
>>> article.path
[<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>,
 <data='<body> <div class="product-list"> <artic...' parent='<html><head><title>Some page</title></he...'>,
 <data='<html><head><title>Some page</title></he...'>]
```

--------------------------------

### Create and Use FetcherSession with Default Configuration

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/fetching/static.md

Demonstrates creating a FetcherSession with default settings and making multiple requests. Use this for managing consistent configurations and cookies across several HTTP calls.

```python
from scrapling.fetchers import FetcherSession

# Create a session with default configuration
with FetcherSession(
    impersonate='chrome',
    http3=True,
    stealthy_headers=True,
    timeout=30,
    retries=3
) as session:
    # Make multiple requests with the same settings and the same cookies
    page1 = session.get('https://scrapling.requestcatcher.com/get')
    page2 = session.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'})
    page3 = session.get('https://api.github.com/events')

    # All requests share the same session and connection pool
```

--------------------------------

### Find First Matching Element

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/parsing/main_classes.md

Locates the first HTML element that matches the given CSS selector. This is often used to get a starting point for further element-specific parsing.

```python
article = page.find('article')
```

--------------------------------

### Get Ancestor Path in DOM

Source: https://github.com/d4vinci/scrapling/blob/main/docs/parsing/main_classes.md

Retrieves a list of all ancestor elements in the DOM tree, starting from the parent and going up to the root. Each ancestor is represented with its tag and a snippet of its parent's context.

```python
>>> article.path
[<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>,
```

--------------------------------

### Define a Custom Spider in Scrapling

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/spiders/architecture.md

Subclass the Spider class to define custom crawling logic, including start URLs and parsing methods. This example shows how to extract links and scrape page titles.

```python
from scrapling.spiders import Spider, Response, Request

class MySpider(Spider):
    name = "my_spider"
    start_urls = ["https://example.com"]

    async def parse(self, response: Response):
        for link in response.css("a::attr(href)").getall():
            yield response.follow(link, callback=self.parse_page)

    async def parse_page(self, response: Response):
        yield {"title": response.css("h1::text").get("")}
```

--------------------------------

### Define and Start a Scrapy-like Spider

Source: https://github.com/d4vinci/scrapling/blob/main/docs/index.md

Create a custom spider by inheriting from the `Spider` class. Define the starting URLs and implement the `parse` method to process responses and yield extracted data. The spider can be started using the `start()` method.

```python
from scrapling.spiders import Spider, Response

class MySpider(Spider):
  name = "demo"
  start_urls = ["https://example.com/"]

  async def parse(self, response: Response):
      for item in response.css('.product'):
          yield {"title": item.css('h2::text').get()}

MySpider().start()
```

--------------------------------

### Custom Initial Requests with start_requests

Source: https://github.com/d4vinci/scrapling/blob/main/docs/spiders/advanced.md

Override the `start_requests` method to generate custom initial requests instead of relying on `start_urls`. This allows for complex scenarios like logging in before crawling.

```APIDOC
## POST /login and subsequent crawling

### Description
This example demonstrates how to override the `start_requests` method to perform an initial POST request for authentication, followed by crawling authenticated pages.

### Method
POST

### Endpoint
/login (for initial authentication)

### Request Body
- **user** (string) - Required - The username for login.
- **pass** (string) - Required - The password for login.

### Request Example
```json
{
  "user": "admin",
  "pass": "secret"
}
```

### Response
#### Success Response (200)
- **(response body)** - The response from the login endpoint, which is then passed to the `after_login` callback.

### Callback
`after_login` method is called with the login response.

## Crawling Authenticated Pages

### Description
After successful login, this section shows how to follow links to crawl authenticated pages.

### Method
GET (implicitly via `response.follow`)

### Endpoint
/dashboard (example path)

### Parameters
#### Query Parameters
None

### Request Example
(No direct request example, as it follows a successful login)

### Response
#### Success Response (200)
- **(response body)** - The content of the authenticated page.

### Callback
`parse` method is called with the response from the authenticated page.
```

--------------------------------

### E-commerce Data Collection Example

Source: https://github.com/d4vinci/scrapling/blob/main/docs/cli/interactive-shell.md

Scrape product links from a listing page, then fetch and extract details for a sample of products. Uses `FetcherSession` for efficient batch requests.

```python
# Start with product listing page
catalog = get('https://shop.example.com/products')

# Find product links
product_links = catalog.css('.product-link::attr(href)')
print(f"Found {len(product_links)} products")

# Sample a few products first
for link in product_links[:3]:
...     product = get(f"https://shop.example.com{link}")
...     name = product.css('.product-name::text').get('')
...     price = product.css('.price::text').get('')
...     print(f"{name}: {price}")

# Scale up with sessions for efficiency
from scrapling.fetchers import FetcherSession
with FetcherSession() as session:
...     products = []
...     for link in product_links:
...         product = session.get(f"https://shop.example.com{link}")
...         products.append({
...             'name': product.css('.product-name::text').get(''),
...             'price': product.css('.price::text').get(''),
...             'url': link
...         })
```

--------------------------------

### Start Scrapling MCP Server (Docker)

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/mcp-server.md

Launches the Scrapling MCP server using Docker. This involves pulling the image and running it with the MCP command.

```bash
docker pull pyd4vinci/scrapling
```

```bash
docker run -i --rm pyd4vinci/scrapling mcp
```

--------------------------------

### GET Request with HTTP/3 Support

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/fetching/static.md

Enable HTTP/3 for GET requests by setting `http3=True`.

```python
page = await AsyncFetcher.get('https://example.com', http3=True)
```

--------------------------------

### Scrapling MCP Server Initialization

Source: https://github.com/d4vinci/scrapling/blob/main/docs/api-reference/mcp-server.md

Methods to start the Scrapling MCP server either via CLI or Python integration.

```APIDOC
## Initialization

### CLI
`scrapling mcp`

### Python
```python
from scrapling.core.ai import ScraplingMCPServer
server = ScraplingMCPServer()
server.serve(http=False, host="0.0.0.0", port=8000)
```
```

--------------------------------

### GET Request with Proxy

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/fetching/static.md

Configure a proxy for GET requests using the `proxy` argument.

```python
page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')
```

--------------------------------

### GET Request with Parameters

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/fetching/static.md

Include query parameters in a GET request using the `params` dictionary.

```python
page = await AsyncFetcher.get('https://example.com/search', params={'q': 'query'})
```

--------------------------------

### Run Dynamic Session Example

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/examples/README.md

Execute the Python script for DynamicSession, which opens a visible browser for dynamic content scraping.

```bash
python examples/02_dynamic_session.py  # Opens a visible browser
```

--------------------------------

### Request Object Construction

Source: https://github.com/d4vinci/scrapling/blob/main/docs/spiders/requests-responses.md

Demonstrates how to create a Request object directly and how response.follow() can be used within callbacks.

```APIDOC
## Request Object

A `Request` represents a URL to be fetched. You create requests either directly or via `response.follow()`:

```python
from scrapling.spiders import Request

# Direct construction
request = Request(
    "https://example.com/page",
    callback=self.parse_page,
    priority=5,
)

# Via response.follow (preferred in callbacks)
request = response.follow("/page", callback=self.parse_page)
```

### Request Arguments

| Argument      | Type       | Default    | Description                                                                                           |
|---------------|------------|------------|-------------------------------------------------------------------------------------------------------|
| `url`         | `str`      | *required* | The URL to fetch                                                                                      |
| `sid`         | `str`      | `""`       | Session ID - routes the request to a specific session (see [Sessions](sessions.md))                   |
| `callback`    | `callable` | `None`     | Async generator method to process the response. Defaults to `parse()`                                 |
| `priority`    | `int`      | `0`        | Higher values are processed first                                                                     |
| `dont_filter` | `bool`     | `False`    | If `True`, skip deduplication (allow duplicate requests)                                              |
| `meta`        | `dict`     | `{}`       | Arbitrary metadata passed through to the response                                                     |
| `**kwargs`    |            |            | Additional keyword arguments passed to the session's fetch method (e.g., `headers`, `method`, `data`) |

### POST Request Example

Any extra keyword arguments are forwarded directly to the underlying session. For example, to make a POST request:

```python
yield Request(
    "https://example.com/api",
    method="POST",
    data={"key": "value"},
    callback=self.parse_result,
)
```
```

--------------------------------

### Scrapling CLI POST Request Examples

Source: https://github.com/d4vinci/scrapling/blob/main/docs/cli/extract-commands.md

Demonstrates how to make POST requests with form data or JSON payload. Specify the URL, output file, and data payload using --data or --json flags.

```bash
scrapling extract post "https://api.site.com/search" results.html --data "query=python&type=tutorial"
```

```bash
scrapling extract post "https://api.site.com" response.json --json '{"username": "test", "action": "search"}'
```

--------------------------------

### Initializing and Using the Selector Class

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/parsing/main_classes.md

Instantiate a Selector object with HTML content and an optional URL. Elements can then be selected using CSS selectors.

```python
page = Selector(
    '<html>...</html>',
    url='https://example.com'
)

# Then select elements as you like
elements = page.css('.product')
```

--------------------------------

### Synchronous GET Request and JSON Parsing

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/fetching/static.md

Make a synchronous GET request and parse the JSON response.

```python
page = Fetcher.get('https://api.github.com/events')
page.json()
```

--------------------------------

### Run Stealthy Session Example

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/examples/README.md

Execute the Python script for StealthySession, which opens a visible stealth browser for bypassing fingerprinting and Cloudflare.

```bash
python examples/03_stealthy_session.py # Opens a visible stealth browser
```

--------------------------------

### GET Request with Browser Impersonation

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/fetching/static.md

Impersonate a specific browser for GET requests using the `impersonate` argument.

```python
page = await AsyncFetcher.get('https://example.com', impersonate='chrome110')
```

--------------------------------

### GET Request with Custom Headers

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/fetching/static.md

Specify custom headers for a GET request using the `headers` dictionary.

```python
page = await AsyncFetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})
```

--------------------------------

### Spider Lifecycle Hook: on_start

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/spiders/advanced.md

Override the `on_start` method to perform setup tasks before crawling begins, such as loading data or initializing resources. It receives a `resuming` boolean indicating if the crawl is resuming.

```python
async def on_start(self, resuming: bool = False):
    self.logger.info("Spider starting up")
    # Load seed URLs from a database, initialize counters, etc.
```

--------------------------------

### GET Request with Stealthy Headers

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/fetching/static.md

Use `stealthy_headers=True` for GET requests to mimic real browser headers.

```python
page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
```

--------------------------------

### GET Request with Proxy

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/fetching/static.md

Make a GET request through a specified HTTP proxy. Ensure the proxy format is correct.

```python
page = Fetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')
```

--------------------------------

### Basic GET Request

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/fetching/static.md

Perform a basic GET request to a specified URL. Stealthy headers are enabled by default.

```python
page = Fetcher.get('https://example.com')
```

--------------------------------

### Load HTML Document with Selector

Source: https://github.com/d4vinci/scrapling/blob/main/docs/overview.md

Initialize the Selector with an HTML document string to begin parsing.

```python
from scrapling.parser import Selector
page = Selector(html_doc)
page  # <data='<html><head><title>Complex Web Page</tit...'>
```

--------------------------------

### Downloading Files with Fetcher

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/fetching/static.md

Shows how to download a file using Fetcher.get and save its content to a local file. This is useful for retrieving binary data like images or documents.

```python
from scrapling.fetchers import Fetcher

page = Fetcher.get('https://raw.githubusercontent.com/D4Vinci/Scrapling/main/docs/assets/main_cover.png')
with open(file='main_cover.png', mode='wb') as f:
   f.write(page.body)
```

--------------------------------

### Scrapling CLI PUT Request Examples

Source: https://github.com/d4vinci/scrapling/blob/main/docs/cli/extract-commands.md

Shows how to perform PUT requests with form data or JSON payload. Use the --data or --json flags to specify the payload and --impersonate to set the browser.

```bash
scrapling extract put "https://scrapling.requestcatcher.com/put" results.html --data "update=info" --impersonate "firefox"
```

```bash
scrapling extract put "https://scrapling.requestcatcher.com/put" response.json --json '{"username": "test", "action": "search"}'
```

--------------------------------

### Fetcher Basic GET Request

Source: https://github.com/d4vinci/scrapling/blob/main/docs/fetching/static.md

Demonstrates a basic GET request using the Fetcher class and extracting information from the response.

```APIDOC
## Basic HTTP Request

### Description
Performs a basic GET request and demonstrates how to access response data.

### Method
GET

### Endpoint
`https://example.com`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```python
from scrapling.fetchers import Fetcher

# Make a request
page = Fetcher.get('https://example.com')

# Check the status
if page.status == 200:
    # Extract title
    title = page.css('title::text').get()
    print(f"Page title: {title}")

    # Extract all links
    links = page.css('a::attr(href)').getall()
    print(f"Found {len(links)} links")
```
```

--------------------------------

### Scrapling Extract Get Command Help

Source: https://github.com/d4vinci/scrapling/blob/main/docs/cli/extract-commands.md

Displays the help message and available options for the `scrapling extract get` command.

```bash
Usage: scrapling extract get [OPTIONS] URL OUTPUT_FILE

  Perform a GET request and save the content to a file.

  The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.

Options:
  -H, --headers TEXT                             HTTP headers in format "Key: Value" (can be used multiple times)
  --cookies TEXT                                 Cookies string in format "name1=value1;name2=value2"
  --timeout INTEGER                              Request timeout in seconds (default: 30)
  --proxy TEXT                                   Proxy URL in format "http://username:password@host:port"
  -s, --css-selector TEXT                        CSS selector to extract specific content from the page. It returns all matches.
  -p, --params TEXT                              Query parameters in format "key=value" (can be used multiple times)
  --follow-redirects / --no-follow-redirects     Whether to follow redirects (default: True)
  --verify / --no-verify                         Whether to verify SSL certificates (default: True)
  --impersonate TEXT                             Browser to impersonate (e.g., chrome, firefox).
  --stealthy-headers / --no-stealthy-headers     Use stealthy browser headers (default: True)
  --ai-targeted                                  Extract only main content and sanitize hidden elements for AI consumption (default: False)
  --help                                         Show this message and exit.


```

--------------------------------

### GET Request with Basic HTTP Authentication

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/fetching/static.md

Perform GET requests with basic HTTP authentication using the `auth` tuple.

```python
page = await AsyncFetcher.get("https://example.com", auth=("my_user", "password123"))
```

--------------------------------

### Fetch Command Help

Source: https://github.com/d4vinci/scrapling/blob/main/docs/cli/extract-commands.md

Displays the help message for the scrapling extract fetch command, showing all available options.

```bash
Usage: scrapling extract fetch [OPTIONS] URL OUTPUT_FILE

  Use DynamicFetcher to fetch content with browser automation.

  The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.

Options:
  --headless / --no-headless                  Run browser in headless mode (default: True)
  --disable-resources / --enable-resources    Drop unnecessary resources for speed boost (default: False)
  --network-idle / --no-network-idle          Wait for network idle (default: False)
  --timeout INTEGER                           Timeout in milliseconds (default: 30000)
  --wait INTEGER                              Additional wait time in milliseconds after page load (default: 0)
  -s, --css-selector TEXT                     CSS selector to extract specific content from the page. It returns all matches.
  --wait-selector TEXT                        CSS selector to wait for before proceeding
  --locale TEXT                               Specify user locale. Defaults to the system default locale.
  --real-chrome/--no-real-chrome              If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)
  --proxy TEXT                                Proxy URL in format "http://username:password@host:port"
  -H, --extra-headers TEXT                    Extra headers in format "Key: Value" (can be used multiple times)
  --dns-over-https / --no-dns-over-https   Route DNS through Cloudflare's DoH to prevent DNS leaks when using proxies (default: False)
  --block-ads / --no-block-ads               Block requests to known ad and tracker domains (default: False)
  --ai-targeted                              Extract only main content and sanitize hidden elements for AI consumption (default: False)
  --help                                      Show this message and exit.
```

--------------------------------

### Basic GET Request with Scrapling

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/fetching/static.md

Perform a basic GET request to a URL. The `page` object returned is a Selector for parsing.

```python
page = await AsyncFetcher.get('https://example.com')
```

--------------------------------

### GET Request with Basic Authentication

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/fetching/static.md

Execute a GET request requiring basic HTTP authentication. Provide credentials as a tuple.

```python
page = Fetcher.get("https://example.com", auth=("my_user", "password123"))
```

--------------------------------

### GET Request with Custom Headers

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/fetching/static.md

Perform a GET request and override default headers, such as the User-Agent, using the `headers` argument.

```python
page = Fetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})
```

--------------------------------

### Scrapling Extract Available Commands Help

Source: https://github.com/d4vinci/scrapling/blob/main/docs/cli/extract-commands.md

Displays the available commands and options for the `scrapling extract` command.

```bash
Usage: scrapling extract [OPTIONS] COMMAND [ARGS]...

  Fetch web pages using various fetchers and extract full/selected HTML content as HTML, Markdown, or extract text content.

Options:
  --help  Show this message and exit.

Commands:
  get             Perform a GET request and save the content to a file.
  post            Perform a POST request and save the content to a file.
  put             Perform a PUT request and save the content to a file.
  delete          Perform a DELETE request and save the content to a file.
  fetch           Use DynamicFetcher to fetch content with browser...
  stealthy-fetch  Use StealthyFetcher to fetch content with advanced...

```

--------------------------------

### GET Request with HTTP/3 Support

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/fetching/static.md

Enable HTTP/3 protocol for a GET request. Note that this may cause issues when used with `impersonate`.

```python
page = Fetcher.get('https://example.com', http3=True)
```

--------------------------------

### GET Request with Stealthy Headers

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/fetching/static.md

Execute a GET request with explicitly enabled stealthy headers, which mimic real browser headers.

```python
page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
```

--------------------------------

### Install Scrapling Skill using Clawhub

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/README.md

This command installs the Scrapling agent skill directly using the Clawhub tool. Clawhub is a package manager for agent skills, compatible with OpenClaw and Claude Code. Ensure you have Clawhub installed and configured to use this command.

```bash
clawhub install scrapling-official
```

--------------------------------

### Set Up Proxy Rotation

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/fetching/dynamic.md

Configure a `ProxyRotator` to automatically switch proxies for subsequent requests within a `DynamicSession`. Each proxy gets its own browser context.

```python
from scrapling.fetchers import DynamicSession, ProxyRotator

# Set up proxy rotation
rotator = ProxyRotator([
    "http://proxy1:8080",
    "http://proxy2:8080",
    "http://proxy3:8080",
])

# Use with session - rotates proxy automatically with each request
with DynamicSession(proxy_rotator=rotator, headless=True) as session:
    page1 = session.fetch('https://example1.com')
    page2 = session.fetch('https://example2.com')

    # Override rotator for a specific request
    page3 = session.fetch('https://example3.com', proxy='http://specific-proxy:8080')
```

--------------------------------

### Start Scrapling MCP Server (HTTP Transport)

Source: https://github.com/d4vinci/scrapling/blob/main/agent-skill/Scrapling-Skill/references/mcp-server.md

Initiates the Scrapling MCP server using the Streamable HTTP transport. You can specify the host and port for custom configurations.

```bash
scrapling mcp --http
```

```bash
scrapling mcp --http --host 127.0.0.1 --port 8000
```