### GET Request Examples

Source: https://scrapling.readthedocs.io/en/latest/cli/extract-commands.html

Demonstrates various ways to use the 'get' command, including basic downloads, custom timeouts, CSS selector extraction, and adding headers/cookies.

```bash
# Basic download
scrapling extract get "https://news.site.com" news.md

```

```bash
# Download with custom timeout
scrapling extract get "https://example.com" content.txt --timeout 60

```

```bash
# Extract only specific content using CSS selectors
scrapling extract get "https://blog.example.com" articles.md --css-selector "article"

```

```bash
# Send a request with cookies
scrapling extract get "https://scrapling.requestcatcher.com" content.md --cookies "session=abc123; user=john"

```

```bash
# Add user agent
scrapling extract get "https://api.site.com" data.json -H "User-Agent: MyBot 1.0"

```

```bash
# Add multiple headers
scrapling extract get "https://site.com" page.html -H "Accept: text/html" -H "Accept-Language: en-US"

```

--------------------------------

### Setup Logic with on_start

Source: https://scrapling.readthedocs.io/en/latest/api-reference/spiders.html

Called before crawling starts. Override this method to implement custom setup logic, such as logging or initializing resources. It accepts a `resuming` boolean parameter.

```python
async def on_start(self, resuming: bool = False) -> None:
    """Called before crawling starts. Override for setup logic.

    :param resuming: It's enabled if the spider is resuming from a checkpoint, left for the user to use.
    """
    if resuming:
        self.logger.debug("Resuming spider from checkpoint")
    else:
        self.logger.debug("Starting spider")
```

--------------------------------

### on_start

Source: https://scrapling.readthedocs.io/en/latest/api-reference/spiders.html

Called before crawling starts. This method can be overridden to include custom setup logic before the crawling process begins.

```APIDOC
## on_start `async`

### Description
Called before crawling starts. Override for setup logic.

### Parameters
#### Path Parameters
- **resuming** (bool) - Optional - It's enabled if the spider is resuming from a checkpoint, left for the user to use. DEFAULT: False

### Method Signature
```python
async def on_start(self, resuming: bool = False) -> None:
```
```

--------------------------------

### Capture WebSockets with Page Setup (Sync)

Source: https://scrapling.readthedocs.io/en/latest/fetching/dynamic.html

Use `page_setup` to register event listeners, such as for WebSockets, before navigation. This example uses the synchronous API.

```python
from playwright.sync_api import Page

def capture_websockets(page: Page):
    page.on("websocket", lambda ws: print(f"WebSocket opened: {ws.url}"))

page = DynamicFetcher.fetch('https://example.com', page_setup=capture_websockets)

```

--------------------------------

### Start Browser Instance

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Initializes and starts a browser instance for the current context. It handles Playwright setup, including connecting over CDP or launching a new browser, and setting up the browser context with specified options. Raises a RuntimeError if the session has already been started.

```python
def start(self) -> None:
    """Create a browser for this instance and context."""
    if not self.playwright:
        self.playwright = sync_playwright().start()

        try:
            if self._config.cdp_url:  # pragma: no cover
                self.browser = self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
                if not self._config.proxy_rotator:
                    assert self.browser is not None
                    self.context = self.browser.new_context(**self._context_options)
            elif self._config.proxy_rotator:
                self.browser = self.playwright.chromium.launch(**self._browser_options)
            else:
                persistent_options = (
                    self._browser_options | self._context_options | {"user_data_dir": self._user_data_dir}
                )
                self.context = self.playwright.chromium.launch_persistent_context(**persistent_options)

            if self.context:
                self.context = self._initialize_context(self._config, self.context)

            self._is_alive = True
        except Exception:
            # Clean up playwright if browser setup fails
            self.playwright.stop()
            self.playwright = None
            raise
    else:
        raise RuntimeError("Session has been already started")
```

--------------------------------

### Scraping Product Links and Details

Source: https://scrapling.readthedocs.io/en/latest/cli/interactive-shell.html

Example demonstrating how to get a product listing page, extract product links, and then scrape details for a sample of products.

```python
# Start with product listing page
catalog = get('https://shop.example.com/products')

# Find product links
product_links = catalog.css('.product-link::attr(href)')
print(f"Found {len(product_links)} products")

# Sample a few products first
for link in product_links[:3]:
...     product = get(f"https://shop.example.com{link}")
...     name = product.css('.product-name::text').get('')
...     price = product.css('.price::text').get('')
...     print(f"{name}: {price}")

# Scale up with sessions for efficiency
from scrapling.fetchers import FetcherSession
with FetcherSession() as session:
...     products = []
...     for link in product_links:
...         product = session.get(f"https://shop.example.com{link}")
...         products.append({
...             'name': product.css('.product-name::text').get(''),
...             'price': product.css('.price::text').get(''),
...             'url': link
...         })
```

--------------------------------

### start

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Starts the browser instance and initializes the Playwright environment.

```APIDOC
## start

### Description
Create a browser for this instance and context. This method initializes the Playwright instance and sets up the browser and context based on the configuration.

### Method
`async def start(self) -> None`

### Parameters
None

### Response
None

### Errors
- `RuntimeError` - Raised if the session has already been started.
```

--------------------------------

### start

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Initializes and starts a browser instance for the current context. It can connect to an existing browser via CDP or launch a new one.

```APIDOC
## start

### Description
Create a browser for this instance and context. This method initializes the Playwright instance and launches a Chromium browser, either connecting over CDP or launching a new instance with specified options.

### Method
POST (assumed, as it creates a resource)

### Endpoint
`/browser/start` (assumed, based on function name and purpose)

### Parameters
None directly in the method signature, but relies on internal configuration (`self._config`).

### Request Example
```json
{
  "cdp_url": "http://localhost:9222/json",
  "proxy_rotator": false,
  "user_data_dir": "/path/to/user/data"
}
```

### Response
#### Success Response (200)
- **message** (string) - Indicates successful browser startup.

#### Response Example
```json
{
  "message": "Browser started successfully."
}
```

### Error Handling
- **RuntimeError**: If the session has already been started.
- **Exception**: Catches and re-raises exceptions during browser setup, cleaning up Playwright if necessary.
```

--------------------------------

### Install All Scrapling Features

Source: https://scrapling.readthedocs.io/en/latest/index.html

Installs Scrapling with all available features and dependencies. Ensure browser dependencies are installed using `scrapling install` after this.

```bash
pip install "scrapling[all]"

```

--------------------------------

### Asynchronous FetcherSession with Standard Methods

Source: https://scrapling.readthedocs.io/en/latest/fetching/static.html

Provides an example of using an asynchronous FetcherSession with common HTTP methods like GET, POST, PUT, and DELETE. Configuration options like impersonation and HTTP/3 are supported.

```python
async with FetcherSession(impersonate='firefox', http3=True) as session:
    # All standard HTTP methods available
    response = await session.get('https://example.com')
    response = await session.post('https://scrapling.requestcatcher.com/post', json={'data': 'value'})
    response = await session.put('https://scrapling.requestcatcher.com/put', data={'update': 'info'})
    response = await session.delete('https://scrapling.requestcatcher.com/delete')
```

--------------------------------

### Capture WebSockets with Page Setup (Async)

Source: https://scrapling.readthedocs.io/en/latest/fetching/dynamic.html

Use `page_setup` with the asynchronous API to capture WebSockets before navigation. The setup function must be async.

```python
from playwright.async_api import Page

async def capture_websockets(page: Page):
    page.on("websocket", lambda ws: print(f"WebSocket opened: {ws.url}"))

page = await DynamicFetcher.async_fetch('https://example.com', page_setup=capture_websockets)

```

--------------------------------

### __enter__

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Enters the runtime context manager, starting the browser engine and returning the instance.

```APIDOC
## __enter__

### Description
Enters the runtime context manager, starting the browser engine and returning the instance.

### Method
```python
def __enter__(self):
```

### Returns
* self - The browser engine instance.
```

--------------------------------

### start

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Initializes and starts a browser instance for the current session. This method creates a Playwright browser context and prepares it for use.

```APIDOC
## start

### Description
Create a browser for this instance and context.

### Method
POST (Implicit)

### Endpoint
`/start` (Implicit)

### Parameters
None

### Response
#### Success Response (200)
Indicates that the browser instance has been successfully started.

### Response Example
```json
{
  "status": "started"
}
```
```

--------------------------------

### Install Fetcher Dependencies

Source: https://scrapling.readthedocs.io/en/latest/cli/overview.html

Run this command to install all necessary browsers and their system dependencies, including fingerprint manipulation tools, required for Scrapling's fetchers.

```bash
scrapling install
```

--------------------------------

### Install Scrapling with AI Features

Source: https://scrapling.readthedocs.io/en/latest/index.html

Installs Scrapling with the MCP server feature. Remember to run `scrapling install` afterwards if browser dependencies are needed.

```bash
pip install "scrapling[ai]"

```

--------------------------------

### __aenter__

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Asynchronous context manager entry point. Starts the browser instance.

```APIDOC
## __aenter__

### Description
Asynchronous context manager entry point. This method is called when entering an 'async with' block. It initiates the browser instance by calling the `start` method.

### Method
`async def __aenter__(self)`

### Parameters
None

### Response
- `self` (object) - The current instance of the browser manager.
```

--------------------------------

### Start MCP Server via CLI

Source: https://scrapling.readthedocs.io/en/latest/api-reference/mcp-server.html

You can start the MCP server directly from the command line using the scrapling CLI tool.

```bash
scrapling mcp

```

--------------------------------

### Install Scrapling Fetchers via Python API

Source: https://scrapling.readthedocs.io/en/latest/index.html

Programmatically installs fetchers' dependencies and browser dependencies using the Scrapling Python API. Use `standalone_mode=False` for typical installations.

```python
from scrapling.cli import install

install([], standalone_mode=False)          # normal install
install(["--force"], standalone_mode=False) # force reinstall

```

--------------------------------

### Install Scrapling Parser Engine

Source: https://scrapling.readthedocs.io/en/latest/index.html

Installs the core parser engine and its dependencies. This is a minimal installation and does not include fetchers or command-line dependencies.

```bash
pip install scrapling

```

--------------------------------

### Install Camoufox and Dependencies

Source: https://scrapling.readthedocs.io/en/latest/fetching/stealthy.html

Install the Camoufox library and Playwright browser dependencies. This is a prerequisite for using Camoufox as a fetcher engine.

```bash
pip install camoufox
playwright install-deps firefox
camoufox fetch

```

--------------------------------

### Start Browser Instance

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Asynchronously starts a browser instance using Playwright. It handles launching a new browser or connecting to an existing one via CDP, and initializes the context. Raises a RuntimeError if the session has already been started.

```python
async def start(self) -> None:
    """Create a browser for this instance and context."""
    if not self.playwright:
        self.playwright = await async_playwright().start()
        try:
            if self._config.cdp_url:
                self.browser = await self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
                if not self._config.proxy_rotator:
                    assert self.browser is not None
                    self.context = await self.browser.new_context(**self._context_options)
            elif self._config.proxy_rotator:
                self.browser = await self.playwright.chromium.launch(**self._browser_options)
            else:
                persistent_options = (
                    self._browser_options | self._context_options | {"user_data_dir": self._user_data_dir}
                )
                self.context = await self.playwright.chromium.launch_persistent_context(**persistent_options)

            if self.context:
                self.context = await self._initialize_context(self._config, self.context)

            self._is_alive = True
        except Exception:
            # Clean up playwright if browser setup fails
            await self.playwright.stop()
            self.playwright = None
            raise
    else:
        raise RuntimeError("Session has been already started")
```

--------------------------------

### Scrapling CLI PUT Request Examples

Source: https://scrapling.readthedocs.io/en/latest/cli/extract-commands.html

Demonstrates how to send form data or JSON data in a PUT request using the Scrapling CLI. Includes examples for specifying the target URL, output file, and impersonating a browser.

```bash
# Send data
scrapling extract put "https://scrapling.requestcatcher.com/put" results.html --data "update=info" --impersonate "firefox"

```

```bash
# Send JSON data
scrapling extract put "https://scrapling.requestcatcher.com/put" response.json --json '{"username": "test", "action": "search"}'

```

--------------------------------

### Install Scrapling with Fetchers Dependencies

Source: https://scrapling.readthedocs.io/en/latest/index.html

Installs Scrapling along with dependencies required for fetchers and other extra features. This is recommended if you plan to use fetchers or spiders.

```bash
pip install "scrapling[fetchers]"

```

```bash
scrapling install

```

```bash
scrapling install  --force

```

--------------------------------

### API Integration and Testing with GET, POST, PUT

Source: https://scrapling.readthedocs.io/en/latest/cli/interactive-shell.html

Demonstrates interactive testing of API endpoints using GET, POST, and PUT requests, including JSON response handling.

```python
>>> # Test API endpoints interactively
>>> response = get('https://jsonplaceholder.typicode.com/posts/1')
>>> response.json()
{'userId': 1, 'id': 1, 'title': 'sunt aut...', 'body': 'quia et...'}

>>> # Test POST requests
>>> new_post = post('https://jsonplaceholder.typicode.com/posts', 
...                 json={'title': 'Test Post', 'body': 'Test content', 'userId': 1})
>>> new_post.json()['id']
101

>>> # Test with different data
>>> updated = put(f'https://jsonplaceholder.typicode.com/posts/{new_post.json()["id"]}',
...               json={'title': 'Updated Title'})
```

--------------------------------

### Scrapling CLI DELETE Request Examples

Source: https://scrapling.readthedocs.io/en/latest/cli/extract-commands.html

Provides examples of how to perform a DELETE request using the Scrapling CLI. Demonstrates sending data and impersonating a browser for the request.

```bash
# Send data
scrapling extract delete "https://scrapling.requestcatcher.com/delete" results.html

```

```bash
# Send JSON data
scrapling extract delete "https://scrapling.requestcatcher.com/" response.txt --impersonate "chrome"

```

--------------------------------

### Start Browser Instance

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Creates a browser instance for the current Scrapling session. This method initializes Playwright and can connect over CDP or launch a new browser instance with specified options. It raises a RuntimeError if the session has already been started.

```python
def start(self):
    """Create a browser for this instance and context."""
    if not self.playwright:
        self.playwright = sync_playwright().start()

        try:
            if self._config.cdp_url:  # pragma: no cover
                self.browser = self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
                if not self._config.proxy_rotator and self.browser:
                    self.context = self.browser.new_context(**self._context_options)
            elif self._config.proxy_rotator:
                self.browser = self.playwright.chromium.launch(**self._browser_options)
            else:
                persistent_options = (
                    self._browser_options | self._context_options | {"user_data_dir": self._user_data_dir}
                )
                self.context = self.playwright.chromium.launch_persistent_context(**persistent_options)

            if self.context:
                self.context = self._initialize_context(self._config, self.context)

            self._is_alive = True
        except Exception:
            # Clean up playwright if browser setup fails
            self.playwright.stop()
            self.playwright = None
            raise
    else:
        raise RuntimeError("Session has been already started")
```

--------------------------------

### Get Element Ancestors

Source: https://scrapling.readthedocs.io/en/latest/parsing/main_classes.html

Retrieve a list of all ancestor elements in the DOM tree, starting from the immediate parent.

```python
>>> article.path
[<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>,
 <data='<body> <div class="product-list"> <artic...' parent='<html><head><title>Some page</title></he...'>,
 <data='<html><head><title>Some page</title></he...'>]
```

--------------------------------

### start

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Initializes and creates a browser instance for the current session. This method must be called before any fetch operations.

```APIDOC
## start

### Description
Create a browser for this instance and context.

### Method
POST (Implicit)

### Endpoint
`/start` (Implicit)

### Parameters
This method does not accept any parameters.

### Response
#### Success Response (200)
This method does not return any specific data upon success, but indicates the browser is ready.

### Response Example
```json
{}
```
```

--------------------------------

### POST Request Examples

Source: https://scrapling.readthedocs.io/en/latest/cli/extract-commands.html

Demonstrates how to use the 'post' command to submit form data or send JSON payloads.

```bash
# Submit form data
scrapling extract post "https://api.site.com/search" results.html --data "query=python&type=tutorial"

```

```bash
# Send JSON data
scrapling extract post "https://api.site.com" response.json --json '{"username": "test", "action": "search"}'

```

--------------------------------

### Process Crawl Results

Source: https://scrapling.readthedocs.io/en/latest/spiders/advanced.html

Access the `CrawlResult` object returned by `start()` to get both the scraped items and detailed statistics. Items can be accessed and saved to a JSON file.

```python
result = MySpider().start()

# Items
print(f"Total items: {len(result.items)}")
result.items.to_json("output.json", indent=True)
```

--------------------------------

### Get Raw Page Body Content

Source: https://scrapling.readthedocs.io/en/latest/parsing/main_classes.html

Access the raw content of a page using the .body property. Starting from v0.4, this returns bytes for Response objects from fetchers.

```python
>>> page.body
'<html>\n  <head>\n    <title>Some page</title>\n  </head>\n  ...'
```

--------------------------------

### __enter__

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Enters the runtime context and starts the browser instance. This method is part of the context manager protocol, allowing the Fetcher instance to be used in a `with` statement.

```APIDOC
## __enter__

### Description
Enters the runtime context and starts the browser instance. This method is part of the context manager protocol, allowing the Fetcher instance to be used in a `with` statement.

### Method
```python
def __enter__(self):
```

### Parameters
None

### Request Example
None

### Response
#### Success Response (200)
Returns the Fetcher instance itself.

#### Response Example
None
```

--------------------------------

### Install Scrapling with Shell Features

Source: https://scrapling.readthedocs.io/en/latest/index.html

Installs Scrapling with shell features, including the interactive Web Scraping shell and the `extract` command. Browser dependencies must be installed separately with `scrapling install`.

```bash
pip install "scrapling[shell]"

```

--------------------------------

### Install Chrome for Playwright

Source: https://scrapling.readthedocs.io/en/latest/fetching/dynamic.html

If Google Chrome is not installed and you wish to use the 'real_chrome' option, you can install it via the Playwright CLI.

```bash
playwright install chrome
```

--------------------------------

### Install Scrapling with MCP Support

Source: https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html

Install Scrapling with AI/MCP server dependencies using pip. Ensure browser dependencies are also installed.

```bash
# Install Scrapling with MCP server dependencies
pip install "scrapling[ai]"

# Install browser dependencies
scrapling install

```

--------------------------------

### Start Browser Instance

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Asynchronously creates a browser instance and context. Handles initialization with Playwright, including options for connecting via CDP or launching directly, and supports proxy rotation.

```python
async def start(self) -> None:
    """Create a browser for this instance and context."""
    if not self.playwright:
        self.playwright = await async_playwright().start()
        try:
            if self._config.cdp_url:
                self.browser = await self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
                if not self._config.proxy_rotator and self.browser:
                    self.context = await self.browser.new_context(**self._context_options)
            elif self._config.proxy_rotator:
                self.browser = await self.playwright.chromium.launch(**self._browser_options)
            else:
                persistent_options = (
                    self._browser_options | self._context_options | {"user_data_dir": self._user_data_dir}
                )
                self.context = await self.playwright.chromium.launch_persistent_context(**persistent_options)

            if self.context:
                self.context = await self._initialize_context(self._config, self.context)

            self._is_alive = True
        except Exception:
            # Clean up playwright if browser setup fails
            await self.playwright.stop()
            self.playwright = None
            raise
    else:
        raise RuntimeError("Session has been already started")
```

--------------------------------

### Implement on_start Lifecycle Hook

Source: https://scrapling.readthedocs.io/en/latest/spiders/advanced.html

Override the `on_start` hook to perform setup tasks before crawling begins, such as loading data or initializing resources. It receives a `resuming` boolean indicating if the crawl is resuming from a paused state.

```python
async def on_start(self, resuming: bool = False):
    self.logger.info("Spider starting up")
    # Load seed URLs from a database, initialize counters, etc.
```

--------------------------------

### Configure Sessions with Different Instances

Source: https://scrapling.readthedocs.io/en/latest/spiders/sessions.html

Demonstrates using multiple instances of the same fetcher class with different configurations, such as impersonating different browsers.

```python
from scrapling.spiders import Spider, Response
from scrapling.fetchers import FetcherSession

class ProductSpider(Spider):
    name = "products"
    start_urls = ["https://shop.example.com/products"]

    def configure_sessions(self, manager):
        chrome_requests = FetcherSession(impersonate="chrome")
        firefox_requests = FetcherSession(impersonate="firefox")

        manager.add("chrome", chrome_requests)
        manager.add("firefox", firefox_requests)

    async def parse(self, response: Response):
        for link in response.css("a.product::attr(href)").getall():
            yield response.follow(link, callback=self.parse_product)

        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield response.follow(next_page, sid="firefox")

    async def parse_product(self, response: Response):
        yield {
            "name": response.css("h1::text").get(""),
            "price": response.css(".price::text").get(""),
        }

```

--------------------------------

### Asynchronously Start All Sessions

Source: https://scrapling.readthedocs.io/en/latest/api-reference/spiders.html

Starts all sessions that are not already alive and are not marked as lazy. This method ensures all necessary sessions are active before proceeding. It prevents re-starting if already started.

```python
async def start(self) -> None:
    """Start all sessions that aren't already alive."""
    if self._started:
        return

    for sid, session in self._sessions.items():
        if sid not in self._lazy_sessions and not session._is_alive:
            await session.__aenter__()

    self._started = True

```

--------------------------------

### Extract Product URLs and Prices

Source: https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html

This example shows how to extract product URLs from a category page and then fetch and parse details from the first few product pages.

```prompt
Go to the following category URL and extract all product URLs using the CSS selector "a". Then, fetch the first 3 product pages in parallel and extract each product’s price and details.

Keep the output in markdown format to reduce irrelevant content.

Category URL:
https://www.arnotts.ie/furniture/bedroom/bed-frames/
```

--------------------------------

### Custom Start Requests with POST

Source: https://scrapling.readthedocs.io/en/latest/spiders/advanced.html

Override `start_requests()` to generate custom initial requests, such as performing a login via a POST request before proceeding to crawl authenticated pages.

```python
async def start_requests(self):
    # POST request to log in first
    yield Request(
        "https://example.com/login",
        method="POST",
        data={"user": "admin", "pass": "secret"},
        callback=self.after_login,
    )

async def after_login(self, response: Response):
    # Now crawl the authenticated pages
    yield response.follow("/dashboard", callback=self.parse)
```

--------------------------------

### Get Previous Sibling: BeautifulSoup vs Scrapling

Source: https://scrapling.readthedocs.io/en/latest/tutorials/migrating_from_beautifulsoup.html

Shows how to get the immediately preceding sibling element.

```python
prev_element = element.previous_sibling
```

```python
prev_element = element.previous
```

--------------------------------

### Enter Context Manager

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Enters the context manager for the browser instance, starting the browser and returning the instance itself.

```python
def __enter__(self):
    self.start()
    return self

```

--------------------------------

### Get Next Sibling: BeautifulSoup vs Scrapling

Source: https://scrapling.readthedocs.io/en/latest/tutorials/migrating_from_beautifulsoup.html

Shows how to get the immediately following sibling element.

```python
next_element = element.next_sibling
```

```python
next_element = element.next
```

--------------------------------

### Instantiate ScraplingMCPServer

Source: https://scrapling.readthedocs.io/en/latest/api-reference/mcp-server.html

How to create an instance of the ScraplingMCPServer class.

```python
ScraplingMCPServer()
```

--------------------------------

### Asynchronous GET Requests with AsyncFetcher

Source: https://scrapling.readthedocs.io/en/latest/fetching/static.html

Illustrates how to perform asynchronous GET requests using AsyncFetcher. It mirrors the functionality of synchronous GET requests, including stealthy headers, proxies, parameters, custom headers, authentication, browser impersonation, and HTTP/3.

```python
from scrapling.fetchers import AsyncFetcher
# Basic GET
page = await AsyncFetcher.get('https://example.com')
page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')
# With parameters
 page = await AsyncFetcher.get('https://example.com/search', params={'q': 'query'})
# With headers
page = await AsyncFetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})
# Basic HTTP authentication
page = await AsyncFetcher.get("https://example.com", auth=("my_user", "password123"))
# Browser impersonation
page = await AsyncFetcher.get('https://example.com', impersonate='chrome110')
# HTTP/3 support
page = await AsyncFetcher.get('https://example.com', http3=True)
```

--------------------------------

### Extraction Methods

Source: https://scrapling.readthedocs.io/en/latest/api-reference/response.html

Methods for extracting data from selectors. `extract` gets all matches, while `extract_first` gets the first match.

```python
extract = getall
```

```python
extract_first = get
```

--------------------------------

### Downloading Files

Source: https://scrapling.readthedocs.io/en/latest/fetching/static.html

Illustrates how to download a file from a URL and save its content to a local file in binary mode.

```python
from scrapling.fetchers import Fetcher

page = Fetcher.get('https://raw.githubusercontent.com/D4Vinci/Scrapling/main/docs/assets/main_cover.png')
with open(file='main_cover.png', mode='wb') as f:
   f.write(page.body)


```

--------------------------------

### Initialize and Serve MCP Server Programmatically

Source: https://scrapling.readthedocs.io/en/latest/api-reference/mcp-server.html

Import and instantiate the ScraplingMCPServer class to run the server programmatically. Configure host and port, and choose whether to use HTTP.

```python
from scrapling.core.ai import ScraplingMCPServer

server = ScraplingMCPServer()
server.serve(http=False, host="0.0.0.0", port=8000)

```

--------------------------------

### Scrapling Extract Get Command Usage

Source: https://scrapling.readthedocs.io/en/latest/cli/extract-commands.html

Shows the basic syntax for performing a GET request to download website content.

```bash
scrapling extract get [URL] [OUTPUT_FILE] [OPTIONS]

```

--------------------------------

### start

Source: https://scrapling.readthedocs.io/en/latest/api-reference/spiders.html

Runs the spider and returns results. This is the main entry point for running a spider and handles asynchronous execution internally. It supports graceful shutdown via Ctrl+C and checkpoint saving if a crawl directory is specified.

```APIDOC
## start

### Description
Runs the spider and returns results. This is the main entry point for running a spider. Handles async execution internally via anyio.

Pressing Ctrl+C will initiate graceful shutdown (waits for active tasks to complete). Pressing Ctrl+C a second time will force immediate stop.
If crawldir is set, a checkpoint will also be saved on graceful shutdown, allowing you to resume the crawl later by running the spider again.

### Method Signature
`start(use_uvloop=False, **backend_options)`

### Parameters
* **use_uvloop** (bool) - Optional - Whether to use the faster uvloop/winloop event loop implementation, if available. Defaults to `False`.
* **backend_options** (Any) - Optional - Asyncio backend options to be used with `anyio.run`. Defaults to `{}`.
```

--------------------------------

### __aenter__()

Source: https://scrapling.readthedocs.io/en/latest/api-reference/spiders.html

Asynchronous context manager entry point. It starts the session manager and returns the manager instance.

```APIDOC
## __aenter__()

### Description
Asynchronous context manager entry point. It starts the session manager and returns the manager instance.

### Method
`async`

### Parameters
None

### Response
- **SessionManager** - The session manager instance.
```

--------------------------------

### Async Context Manager Entry

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Asynchronous entry point for context management. It starts the browser instance and returns the current instance.

```python
async def __aenter__(self):
    await self.start()
    return self
```

--------------------------------

### Scrape Product Details with Persistent Session

Source: https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html

Demonstrates using a persistent browser session to scrape main details from multiple product pages efficiently. Always remember to close the session when done.

```prompt
Open a stealthy browser session with 5 pages maximum pool, then use it to scrape the main details in bulk from the first 5 product pages on https://shop.example.com. Close the session when you're done.
```

--------------------------------

### Get Raw HTML Source: BeautifulSoup vs Scrapling

Source: https://scrapling.readthedocs.io/en/latest/tutorials/migrating_from_beautifulsoup.html

Illustrates how to get the raw, unformatted HTML content of a page or element.

```python
source = str(soup)
```

```python
source = page.html_content
```

--------------------------------

### get

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Sends an HTTP GET request to the specified URL asynchronously. This class method is part of the AsyncFetcher and returns an awaitable Response object.

```APIDOC
## GET /url

### Description
Sends an HTTP GET request to the specified URL asynchronously.

### Method
GET

### Endpoint
/url

### Parameters
#### Path Parameters
- **url** (string) - Required - The URL to send the GET request to.

#### Query Parameters
- **kwargs** (dict) - Optional - Additional keyword arguments to pass to the request.

### Request Example
```json
{
  "url": "http://example.com/data"
}
```

### Response
#### Success Response (200)
- **Response** (object) - An awaitable response object from the GET request.

#### Response Example
```json
{
  "status_code": 200,
  "body": "..."
}
```
```

--------------------------------

### Basic CrawlSpider with Rules

Source: https://scrapling.readthedocs.io/en/latest/spiders/generic-templates.html

Demonstrates a basic CrawlSpider setup with multiple rules for extracting author information and following pagination links.

```python
from scrapling.spiders import CrawlSpider, CrawlRule, LinkExtractor

class QuotesSpider(CrawlSpider):
    name = "blog"
    start_urls = ["https://quotes.toscrape.com/"]

    def rules(self):
        return [
            CrawlRule(LinkExtractor(allow=r"/author/"), callback=self.parse_author),
            CrawlRule(LinkExtractor(allow=r"/page/\d+/")),  # follow pagination, no callback
        ]

    async def parse_author(self, response):
        yield {
            '.author-title': response.css('.author-title::text').get(),
            "birthday": response.css('.author-born-date::text').get(),
            "url": response.url,
        }

result = QuotesSpider().start()

```

--------------------------------

### Perform HTTP GET Request

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

A class method to perform an HTTP GET request. It merges class-level configuration with provided keyword arguments.

```python
@classmethod
def get(cls, url: str, **kwargs: Unpack[GetRequestParams]) -> Response:
    return __FetcherClientInstance__.get(url, **_merge_selector_config(cls, kwargs))
```

--------------------------------

### configure

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Opens a browser and performs a request based on the specified options. This class method allows for extensive customization of the fetching process.

```APIDOC
## configure `classmethod`

### Description
Opens up a browser and do your request based on your chosen options below.

### Method
`classmethod`

### Parameters
#### Path Parameters
- **url** (str) - Required - Target url.

#### Query Parameters
- **headless** (bool) - Optional - Run the browser in headless/hidden (default), or headful/visible mode.
- **disable_resources** (list) - Optional - Drop requests for unnecessary resources for a speed boost.
- **blocked_domains** (set) - Optional - A set of domain names to block requests to. Subdomains are also matched (e.g., `"example.com"` blocks `"sub.example.com"` too).
- **block_ads** (bool) - Optional - Block requests to ~3,500 known ad/tracking domains. Can be combined with `blocked_domains`.
- **dns_over_https** (bool) - Optional - Route DNS queries through Cloudflare's DNS-over-HTTPS to prevent DNS leaks when using proxies.
- **useragent** (str) - Optional - Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
- **cookies** (dict) - Optional - Set cookies for the next request.
- **network_idle** (bool) - Optional - Wait for the page until there are no network connections for at least 500 ms.
- **load_dom** (bool) - Optional - Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- **timeout** (int) - Optional - The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000.
- **wait** (int) - Optional - The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
- **page_action** (function) - Optional - Added for automation. A function that takes the `page` object, runs after navigation, and does the automation you need.
- **page_setup** (function) - Optional - A function that takes the `page` object, runs before navigation. Use it to register event listeners or routes that must be set up before the page loads.
- **wait_selector** (str) - Optional - Wait for a specific CSS selector to be in a specific state.
- **init_script** (str) - Optional - An absolute path to a JavaScript file to be executed on page creation with this request.
- **locale** (str) - Optional - Set the locale for the browser if wanted. Defaults to the system default locale.
- **wait_selector_state** (str) - Optional - The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- **real_chrome** (bool) - Optional - If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
- **cdp_url** (str) - Optional - Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
- **google_search** (bool) - Optional - Enabled by default, Scrapling will set a Google referer header.
- **extra_headers** (dict) - Optional - A dictionary of extra headers to add to the request.
- **proxy** (str or dict) - Optional - The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- **extra_flags** (list) - Optional - A list of additional browser flags to pass to the browser on launch.
- **selector_config** (dict) - Optional - The arguments that will be passed in the end while creating the final Selector's class.
- **additional_args** (dict) - Optional - Additional arguments to be passed to Playwright's context as additional settings.

### Returns
#### Success Response
- **Response** (object) - A `Response` object.
```

--------------------------------

### E-commerce Data Collection with Bulk Fetch

Source: https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html

Collects product names, prices, and descriptions from multiple e-commerce URLs concurrently using bulk browser fetches.

```prompt
Extract product information from these e-commerce URLs using bulk browser fetches:
- https://shop1.com/product-a
- https://shop2.com/product-b  
- https://shop3.com/product-c

Get the product names, prices, and descriptions from each page.


```

--------------------------------

### Long Workflow: E-commerce Product Details

Source: https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html

Extracts all product URLs for a given furniture category, then retrieves prices and details for the first three products.

```prompt
Extract all product URLs for the following category, then return the prices and details for the first 3 products.

https://www.arnotts.ie/furniture/bedroom/bed-frames/


```

--------------------------------

### Install Scrapling Shell Dependency

Source: https://scrapling.readthedocs.io/en/latest/cli/overview.html

Install the 'shell' extra dependency group for Scrapling using pip. This is required to use the interactive shell functionality.

```bash
pip install "scrapling[shell]"
```

--------------------------------

### Get text of the first H1 element (alternative method)

Source: https://scrapling.readthedocs.io/en/latest/parsing/selection.html

An alternative way to get the text content is by selecting the element first and then accessing its `.text` attribute.

```python
title = page.css('h1')[0].text
title = page.xpath('//h1')[0].text
```

--------------------------------

### Synchronous FetcherSession with Configuration

Source: https://scrapling.readthedocs.io/en/latest/fetching/static.html

Illustrates creating a synchronous FetcherSession with custom configurations like impersonation, HTTP/3, stealthy headers, timeout, and retries. Requests made within the session share these settings and cookies.

```python
from scrapling.fetchers import FetcherSession

# Create a session with default configuration
with FetcherSession(
    impersonate='chrome',
    http3=True,
    stealthy_headers=True,
    timeout=30,
    retries=3
) as session:
    # Make multiple requests with the same settings and the same cookies
    page1 = session.get('https://scrapling.requestcatcher.com/get')
    page2 = session.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'})
    page3 = session.get('https://api.github.com/events')

    # All requests share the same session and connection pool
```

--------------------------------

### Simple HTTP GET Request

Source: https://scrapling.readthedocs.io/en/latest/overview.html

Use the Fetcher class for basic HTTP GET requests. The 'impersonate' argument can be used to mimic specific browser versions.

```python
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://scrapling.requestcatcher.com/get', impersonate="chrome")
```

--------------------------------

### Fetch a web page

Source: https://scrapling.readthedocs.io/en/latest/parsing/selection.html

Initializes the Fetcher and retrieves the content of a given URL.

```python
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://quotes.toscrape.com/')
```

--------------------------------

### Scrapling Extract Get Command Help

Source: https://scrapling.readthedocs.io/en/latest/cli/extract-commands.html

Displays the available options for the 'scrapling extract get' command, including details on output formats and extraction methods.

```bash
scrapling extract get --help

```

--------------------------------

### Get the serialized string of the first element

Source: https://scrapling.readthedocs.io/en/latest/api-reference/selector.html

Use `get` to retrieve the serialized string of the first element in the selector. If the selector is empty, it returns the specified default value.

```python
def get(self, default=None):
    """Returns the serialized string of the first element, or ``default`` if empty.
    :param default: the default value to return if the current list is empty
    """
    for x in self:
        return x.get()
    return default
```