Scrapling

https://scrapling.readthedocs.io/en/latest/
Admin
Scrapling is an adaptive web scraping library that learns from website changes, automatically...
Tokens:177,451
Snippets:1,350
Trust Score:8.3
Update:2 months ago
Show doc for...
Tokens:
Raw
### on_start

Source: https://scrapling.readthedocs.io/en/latest/api-reference/spiders.html

A lifecycle hook called before crawling starts. Override this for setup logic.

```APIDOC
## async on_start(resuming=False)

### Description
Called before crawling starts. Override for setup logic.

### Parameters
- **resuming** (bool) - Optional - It's enabled if the spider is resuming from a checkpoint. Default: False
```

--------------------------------

### GET Request Syntax and Examples

Source: https://scrapling.readthedocs.io/en/latest/cli/extract-commands.html

Perform GET requests with various options like timeouts, CSS selectors, cookies, and headers.

```bash
scrapling extract get [URL] [OUTPUT_FILE] [OPTIONS]
```

```bash
# Basic download
scrapling extract get "https://news.site.com" news.md

# Download with custom timeout
scrapling extract get "https://example.com" content.txt --timeout 60

# Extract only specific content using CSS selectors
scrapling extract get "https://blog.example.com" articles.md --css-selector "article"

# Send a request with cookies
scrapling extract get "https://scrapling.requestcatcher.com" content.md --cookies "session=abc123; user=john"

# Add user agent
scrapling extract get "https://api.site.com" data.json -H "User-Agent: MyBot 1.0"

# Add multiple headers
scrapling extract get "https://site.com" page.html -H "Accept: text/html" -H "Accept-Language: en-US"
```

--------------------------------

### start()

Source: https://scrapling.readthedocs.io/en/latest/api-reference/spiders.html

Starts all non-lazy sessions.

```APIDOC
## start()

### Description
Starts all sessions that are not marked as lazy and are not already alive.
```

--------------------------------

### Scraping Product Links and Details

Source: https://scrapling.readthedocs.io/en/latest/cli/interactive-shell.html

Example demonstrating how to get a product listing page, extract product links, and then scrape details for a sample of products.

```python
# Start with product listing page
catalog = get('https://shop.example.com/products')

# Find product links
product_links = catalog.css('.product-link::attr(href)')
print(f"Found {len(product_links)} products")

# Sample a few products first
for link in product_links[:3]:
...     product = get(f"https://shop.example.com{link}")
...     name = product.css('.product-name::text').get('')
...     price = product.css('.price::text').get('')
...     print(f"{name}: {price}")

# Scale up with sessions for efficiency
from scrapling.fetchers import FetcherSession
with FetcherSession() as session:
...     products = []
...     for link in product_links:
...         product = session.get(f"https://shop.example.com{link}")
...         products.append({
...             'name': product.css('.product-name::text').get(''),
...             'price': product.css('.price::text').get(''),
...             'url': link
...         })
```

--------------------------------

### Start Browser Instance

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Initializes and starts a browser instance for the current context. It handles Playwright setup, including connecting over CDP or launching a new browser, and setting up the browser context with specified options. Raises a RuntimeError if the session has already been started.

```python
def start(self) -> None:
    """Create a browser for this instance and context."""
    if not self.playwright:
        self.playwright = sync_playwright().start()

        try:
            if self._config.cdp_url:  # pragma: no cover
                self.browser = self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
                if not self._config.proxy_rotator:
                    assert self.browser is not None
                    self.context = self.browser.new_context(**self._context_options)
            elif self._config.proxy_rotator:
                self.browser = self.playwright.chromium.launch(**self._browser_options)
            else:
                persistent_options = (
                    self._browser_options | self._context_options | {"user_data_dir": self._user_data_dir}
                )
                self.context = self.playwright.chromium.launch_persistent_context(**persistent_options)

            if self.context:
                self.context = self._initialize_context(self._config, self.context)

            self._is_alive = True
        except Exception:
            # Clean up playwright if browser setup fails
            self.playwright.stop()
            self.playwright = None
            raise
    else:
        raise RuntimeError("Session has been already started")
```

--------------------------------

### start

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Starts the browser instance and initializes the Playwright environment.

```APIDOC
## start

### Description
Create a browser for this instance and context. This method initializes the Playwright instance and sets up the browser and context based on the configuration.

### Method
`async def start(self) -> None`

### Parameters
None

### Response
None

### Errors
- `RuntimeError` - Raised if the session has already been started.
```

--------------------------------

### start

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Initializes and starts a browser instance for the current context. It can connect to an existing browser via CDP or launch a new one.

```APIDOC
## start

### Description
Create a browser for this instance and context. This method initializes the Playwright instance and launches a Chromium browser, either connecting over CDP or launching a new instance with specified options.

### Method
POST (assumed, as it creates a resource)

### Endpoint
`/browser/start` (assumed, based on function name and purpose)

### Parameters
None directly in the method signature, but relies on internal configuration (`self._config`).

### Request Example
```json
{
  "cdp_url": "http://localhost:9222/json",
  "proxy_rotator": false,
  "user_data_dir": "/path/to/user/data"
}
```

### Response
#### Success Response (200)
- **message** (string) - Indicates successful browser startup.

#### Response Example
```json
{
  "message": "Browser started successfully."
}
```

### Error Handling
- **RuntimeError**: If the session has already been started.
- **Exception**: Catches and re-raises exceptions during browser setup, cleaning up Playwright if necessary.
```

--------------------------------

### GET Command Help

Source: https://scrapling.readthedocs.io/en/latest/cli/extract-commands.html

View detailed help and options for the GET command.

```bash
Usage: scrapling extract get [OPTIONS] URL OUTPUT_FILE

  Perform a GET request and save the content to a file.

  The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.

Options:
  -H, --headers TEXT                             HTTP headers in format "Key: Value" (can be used multiple times)
  --cookies TEXT                                 Cookies string in format "name1=value1;name2=value2"
  --timeout INTEGER                              Request timeout in seconds (default: 30)
  --proxy TEXT                                   Proxy URL in format "http://username:password@host:port"
  -s, --css-selector TEXT                        CSS selector to extract specific content from the page. It returns all matches.
  -p, --params TEXT                              Query parameters in format "key=value" (can be used multiple times)
  --follow-redirects / --no-follow-redirects     Whether to follow redirects (default: True)
  --verify / --no-verify                         Whether to verify SSL certificates (default: True)
  --impersonate TEXT                             Browser to impersonate (e.g., chrome, firefox).
  --stealthy-headers / --no-stealthy-headers     Use stealthy browser headers (default: True)
  --ai-targeted                                  Extract only main content and sanitize hidden elements for AI consumption (default: False)
  --help                                         Show this message and exit.
```

--------------------------------

### Install Scrapling and Dependencies

Source: https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html

Install the package with MCP support and the required browser binaries.

```bash
# Install Scrapling with MCP server dependencies
pip install "scrapling[ai]"

# Install browser dependencies
scrapling install
```

--------------------------------

### Asynchronous FetcherSession with Standard Methods

Source: https://scrapling.readthedocs.io/en/latest/fetching/static.html

Provides an example of using an asynchronous FetcherSession with common HTTP methods like GET, POST, PUT, and DELETE. Configuration options like impersonation and HTTP/3 are supported.

```python
async with FetcherSession(impersonate='firefox', http3=True) as session:
    # All standard HTTP methods available
    response = await session.get('https://example.com')
    response = await session.post('https://scrapling.requestcatcher.com/post', json={'data': 'value'})
    response = await session.put('https://scrapling.requestcatcher.com/put', data={'update': 'info'})
    response = await session.delete('https://scrapling.requestcatcher.com/delete')
```

--------------------------------

### __enter__

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Enters the runtime context manager, starting the browser engine and returning the instance.

```APIDOC
## __enter__

### Description
Enters the runtime context manager, starting the browser engine and returning the instance.

### Method
```python
def __enter__(self):
```

### Returns
* self - The browser engine instance.
```

--------------------------------

### Install Scrapling via pip

Source: https://scrapling.readthedocs.io/en/latest/index.html

Basic installation of the core parser engine.

```bash
pip install scrapling
```

--------------------------------

### Implement on_start Hook

Source: https://scrapling.readthedocs.io/en/latest/spiders/advanced.html

Override on_start to perform setup tasks before the crawl begins.

```python
async def on_start(self, resuming: bool = False):
    self.logger.info("Spider starting up")
    # Load seed URLs from a database, initialize counters, etc.
```

--------------------------------

### Install Fetchers and Browser Dependencies

Source: https://scrapling.readthedocs.io/en/latest/index.html

Install dependencies required for fetchers and browser automation.

```bash
pip install "scrapling[fetchers]"

scrapling install           # normal install
scrapling install  --force  # force reinstall
```

--------------------------------

### Start Sessions

Source: https://scrapling.readthedocs.io/en/latest/api-reference/spiders.html

Asynchronously start all non-lazy sessions that are not currently alive.

```python
start()
```

```python
async def start(self) -> None:
    """Start all sessions that aren't already alive."""
    if self._started:
        return

    for sid, session in self._sessions.items():
        if sid not in self._lazy_sessions and not session._is_alive:
            await session.__aenter__()

    self._started = True
```

--------------------------------

### Install Camoufox dependencies

Source: https://scrapling.readthedocs.io/en/latest/fetching/stealthy.html

Commands to install the necessary library and browser dependencies.

```bash
pip install camoufox
playwright install-deps firefox
camoufox fetch
```

--------------------------------

### start

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Initializes and starts a browser instance for the current session. This method creates a Playwright browser context and prepares it for use.

```APIDOC
## start

### Description
Create a browser for this instance and context.

### Method
POST (Implicit)

### Endpoint
`/start` (Implicit)

### Parameters
None

### Response
#### Success Response (200)
Indicates that the browser instance has been successfully started.

### Response Example
```json
{
  "status": "started"
}
```
```

--------------------------------

### Pre-Navigation Setup

Source: https://scrapling.readthedocs.io/en/latest/fetching/dynamic.html

Registers event listeners or scripts before navigation occurs.

```python
from playwright.sync_api import Page

def capture_websockets(page: Page):
    page.on("websocket", lambda ws: print(f"WebSocket opened: {ws.url}"))

page = DynamicFetcher.fetch('https://example.com', page_setup=capture_websockets)
```

```python
from playwright.async_api import Page

async def capture_websockets(page: Page):
    page.on("websocket", lambda ws: print(f"WebSocket opened: {ws.url}"))

page = await DynamicFetcher.async_fetch('https://example.com', page_setup=capture_websockets)
```

--------------------------------

### Install Fetcher Dependencies

Source: https://scrapling.readthedocs.io/en/latest/cli/overview.html

Run this command to install all necessary browsers and their system dependencies, including fingerprint manipulation tools, required for Scrapling's fetchers.

```bash
scrapling install
```

--------------------------------

### __aenter__

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Asynchronous context manager entry point. Starts the browser instance.

```APIDOC
## __aenter__

### Description
Asynchronous context manager entry point. This method is called when entering an 'async with' block. It initiates the browser instance by calling the `start` method.

### Method
`async def __aenter__(self)`

### Parameters
None

### Response
- `self` (object) - The current instance of the browser manager.
```

--------------------------------

### __aenter__

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Enters the asynchronous context manager, starting the browser and returning the instance.

```APIDOC
## __aenter__

### Description
Enters the asynchronous context manager, starting the browser and returning the instance.

### Method
```python
async def __aenter__(self):
```
```

--------------------------------

### Initialize and Serve Scrapling MCP Server

Source: https://scrapling.readthedocs.io/en/latest/api-reference/mcp-server.html

Import the server class and start the service programmatically.

```python
from scrapling.core.ai import ScraplingMCPServer

server = ScraplingMCPServer()
server.serve(http=False, host="0.0.0.0", port=8000)
```

--------------------------------

### Start Browser Instance

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Asynchronously starts a browser instance using Playwright. It handles launching a new browser or connecting to an existing one via CDP, and initializes the context. Raises a RuntimeError if the session has already been started.

```python
async def start(self) -> None:
    """Create a browser for this instance and context."""
    if not self.playwright:
        self.playwright = await async_playwright().start()
        try:
            if self._config.cdp_url:
                self.browser = await self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
                if not self._config.proxy_rotator:
                    assert self.browser is not None
                    self.context = await self.browser.new_context(**self._context_options)
            elif self._config.proxy_rotator:
                self.browser = await self.playwright.chromium.launch(**self._browser_options)
            else:
                persistent_options = (
                    self._browser_options | self._context_options | {"user_data_dir": self._user_data_dir}
                )
                self.context = await self.playwright.chromium.launch_persistent_context(**persistent_options)

            if self.context:
                self.context = await self._initialize_context(self._config, self.context)

            self._is_alive = True
        except Exception:
            # Clean up playwright if browser setup fails
            await self.playwright.stop()
            self.playwright = None
            raise
    else:
        raise RuntimeError("Session has been already started")
```

--------------------------------

### Enter Async Context

Source: https://scrapling.readthedocs.io/en/latest/api-reference/spiders.html

Starts the session manager within an asynchronous context.

```python
__aenter__()
```

```python
async def __aenter__(self) -> "SessionManager":
    await self.start()
    return self
```

--------------------------------

### Install Optional Scrapling Features

Source: https://scrapling.readthedocs.io/en/latest/index.html

Commands to install specific feature sets including AI, shell tools, or the full suite.

```bash
pip install "scrapling[ai]"
```

```bash
pip install "scrapling[shell]"
```

```bash
pip install "scrapling[all]"
```

--------------------------------

### Fetch dynamic content

Source: https://scrapling.readthedocs.io/en/latest/cli/extract-commands.html

Syntax and usage examples for fetching content from dynamic websites.

```bash
scrapling extract fetch [URL] [OUTPUT_FILE] [OPTIONS]
```

```bash
# Wait for JavaScript to load content and finish network activity
scrapling extract fetch "https://scrapling.requestcatcher.com/" content.md --network-idle

# Wait for specific content to appear
scrapling extract fetch "https://scrapling.requestcatcher.com/" data.txt --wait-selector ".content-loaded"

# Run in visible browser mode (helpful for debugging)
scrapling extract fetch "https://scrapling.requestcatcher.com/" page.html --no-headless --disable-resources
```

--------------------------------

### API Integration and Testing with GET, POST, PUT

Source: https://scrapling.readthedocs.io/en/latest/cli/interactive-shell.html

Demonstrates interactive testing of API endpoints using GET, POST, and PUT requests, including JSON response handling.

```python
>>> # Test API endpoints interactively
>>> response = get('https://jsonplaceholder.typicode.com/posts/1')
>>> response.json()
{'userId': 1, 'id': 1, 'title': 'sunt aut...', 'body': 'quia et...'}

>>> # Test POST requests
>>> new_post = post('https://jsonplaceholder.typicode.com/posts', 
...                 json={'title': 'Test Post', 'body': 'Test content', 'userId': 1})
>>> new_post.json()['id']
101

>>> # Test with different data
>>> updated = put(f'https://jsonplaceholder.typicode.com/posts/{new_post.json()["id"]}',
...               json={'title': 'Updated Title'})
```

--------------------------------

### Run a Spider

Source: https://scrapling.readthedocs.io/en/latest/spiders/getting-started.html

Instantiate and start a spider to begin the crawling process.

```python
result = QuotesSpider().start()
```

--------------------------------

### Start Browser Instance

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Creates a browser instance for the current Scrapling session. This method initializes Playwright and can connect over CDP or launch a new browser instance with specified options. It raises a RuntimeError if the session has already been started.

```python
def start(self):
    """Create a browser for this instance and context."""
    if not self.playwright:
        self.playwright = sync_playwright().start()

        try:
            if self._config.cdp_url:  # pragma: no cover
                self.browser = self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
                if not self._config.proxy_rotator and self.browser:
                    self.context = self.browser.new_context(**self._context_options)
            elif self._config.proxy_rotator:
                self.browser = self.playwright.chromium.launch(**self._browser_options)
            else:
                persistent_options = (
                    self._browser_options | self._context_options | {"user_data_dir": self._user_data_dir}
                )
                self.context = self.playwright.chromium.launch_persistent_context(**persistent_options)

            if self.context:
                self.context = self._initialize_context(self._config, self.context)

            self._is_alive = True
        except Exception:
            # Clean up playwright if browser setup fails
            self.playwright.stop()
            self.playwright = None
            raise
    else:
        raise RuntimeError("Session has been already started")
```

--------------------------------

### Synchronous GET Requests with Fetcher

Source: https://scrapling.readthedocs.io/en/latest/fetching/static.html

Demonstrates various ways to perform basic and advanced GET requests using the synchronous Fetcher. Supports custom headers, parameters, authentication, proxies, stealthy headers, browser impersonation, and HTTP/3.

```python
from scrapling.fetchers import Fetcher
# Basic GET
page = Fetcher.get('https://example.com')
page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
page = Fetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')
# With parameters
page = Fetcher.get('https://example.com/search', params={'q': 'query'})

# With headers
page = Fetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})
# Basic HTTP authentication
page = Fetcher.get("https://example.com", auth=("my_user", "password123"))
# Browser impersonation
page = Fetcher.get('https://example.com', impersonate='chrome')
# HTTP/3 support
page = Fetcher.get('https://example.com', http3=True)
```

--------------------------------

### Get Element Ancestors

Source: https://scrapling.readthedocs.io/en/latest/parsing/main_classes.html

Retrieve a list of all ancestor elements in the DOM tree, starting from the immediate parent.

```python
>>> article.path
[<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>,
 <data='<body> <div class="product-list"> <artic...' parent='<html><head><title>Some page</title></he...'>,
 <data='<html><head><title>Some page</title></he...'>]
```

--------------------------------

### start

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Initializes and creates a browser instance for the current session. This method must be called before any fetch operations.

```APIDOC
## start

### Description
Create a browser for this instance and context.

### Method
POST (Implicit)

### Endpoint
`/start` (Implicit)

### Parameters
This method does not accept any parameters.

### Response
#### Success Response (200)
This method does not return any specific data upon success, but indicates the browser is ready.

### Response Example
```json
{}
```
```

--------------------------------

### Enable uvloop for Spider

Source: https://scrapling.readthedocs.io/en/latest/spiders/advanced.html

Use the use_uvloop parameter in the start method to improve I/O throughput. Requires uvloop or winloop to be installed separately.

```python
result = MySpider().start(use_uvloop=True)
```

--------------------------------

### Get Raw Page Body Content

Source: https://scrapling.readthedocs.io/en/latest/parsing/main_classes.html

Access the raw content of a page using the .body property. Starting from v0.4, this returns bytes for Response objects from fetchers.

```python
>>> page.body
'<html>\n  <head>\n    <title>Some page</title>\n  </head>\n  ...'
```

--------------------------------

### __enter__

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Enters the runtime context and starts the browser instance. This method is part of the context manager protocol, allowing the Fetcher instance to be used in a `with` statement.

```APIDOC
## __enter__

### Description
Enters the runtime context and starts the browser instance. This method is part of the context manager protocol, allowing the Fetcher instance to be used in a `with` statement.

### Method
```python
def __enter__(self):
```

### Parameters
None

### Request Example
None

### Response
#### Success Response (200)
Returns the Fetcher instance itself.

#### Response Example
None
```

--------------------------------

### Perform a GET request with get()

Source: https://scrapling.readthedocs.io/en/latest/api-reference/mcp-server.html

Executes an asynchronous GET request to a single URL and returns a structured response. Suitable for low-to-mid protection levels.

```python
@staticmethod
async def get(
    url: str,
    impersonate: ImpersonateType = "chrome",
    extraction_type: extraction_types = "markdown",
    css_selector: Optional[str] = None,
    main_content_only: bool = True,
    params: Optional[Dict] = None,
    headers: Optional[Mapping[str, Optional[str]]] = None,
    cookies: Optional[Dict[str, str]] = None,
    timeout: Optional[int | float] = 30,
    follow_redirects: FollowRedirects = "safe",
    max_redirects: int = 30,
    retries: Optional[int] = 3,
    retry_delay: Optional[int] = 1,
    proxy: Optional[str] = None,
    proxy_auth: Optional[Dict[str, str]] = None,
    auth: Optional[Dict[str, str]] = None,
    verify: Optional[bool] = True,
    http3: Optional[bool] = False,
    stealthy_headers: Optional[bool] = True,
) -> ResponseModel:
    """Make GET HTTP request to a URL and return a structured output of the result.
    Note: This is only suitable for low-mid protection levels. For high-protection levels or websites that require JS loading, use the other tools directly.
    Note: If the `css_selector` resolves to more than one element, all the elements will be returned.

    :param url: The URL to request.
    :param impersonate: Browser version to impersonate its fingerprint. It's using the latest chrome version by default.
    :param extraction_type: The type of content to extract from the page. Defaults to "markdown". Options are:
        - Markdown will convert the page content to Markdown format.
        - HTML will return the raw HTML content of the page.
        - Text will return the text content of the page.
    :param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.
    :param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `<body>` tag.
    :param params: Query string parameters for the request.
    :param headers: Headers to include in the request.
    :param cookies: Cookies to use in the request.
    :param timeout: Number of seconds to wait before timing out.
    :param follow_redirects: Whether to follow redirects. Defaults to "safe", which follows redirects but rejects those targeting internal/private IPs (SSRF protection).
        Pass True to follow all redirects without restriction.
    :param max_redirects: Maximum number of redirects. Default 30, use -1 for unlimited.
    :param retries: Number of retry attempts. Defaults to 3.
    :param retry_delay: Number of seconds to wait between retry attempts. Defaults to 1 second.
    :param proxy: Proxy URL to use. Format: "http://username:password@localhost:8030".
                 Cannot be used together with the `proxies` parameter.
    :param proxy_auth: HTTP basic auth for proxy in dictionary format with `username` and `password` keys.
    :param auth: HTTP basic auth in dictionary format with `username` and `password` keys.
    :param verify: Whether to verify HTTPS certificates.
    :param http3: Whether to use HTTP3. Defaults to False. It might be problematic if used it with `impersonate`.
    :param stealthy_headers: If enabled (default), it creates and adds real browser headers. It also sets a Google referer header.
    """
    results = await ScraplingMCPServer.bulk_get(
        urls=[url],
        impersonate=impersonate,
        extraction_type=extraction_type,
        css_selector=css_selector,
        main_content_only=main_content_only,
        params=params,
        headers=headers,
        cookies=cookies,
        timeout=timeout,
        follow_redirects=follow_redirects,
        max_redirects=max_redirects,
        retries=retries,
        retry_delay=retry_delay,
        proxy=proxy,
        proxy_auth=proxy_auth,
        auth=auth,
        verify=verify,
        http3=http3,
        stealthy_headers=stealthy_headers,
    )
    return results[0]
```

--------------------------------

### start(use_uvloop=False, **backend_options)

Source: https://scrapling.readthedocs.io/en/latest/api-reference/spiders.html

The main entry point for running a spider. It handles async execution internally and supports graceful shutdown via Ctrl+C.

```APIDOC
## start(use_uvloop=False, **backend_options)

### Description
Run the spider and return results. This is the main entry point for running a spider, handling async execution internally via anyio.

### Parameters
- **use_uvloop** (bool) - Optional - Whether to use the faster uvloop/winloop event loop implementation, if available. Default: False.
- **backend_options** (Any) - Optional - Asyncio backend options to be used with anyio.run. Default: {}.
```

--------------------------------

### Start Browser Instance

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Asynchronously creates a browser instance and context. Handles initialization with Playwright, including options for connecting via CDP or launching directly, and supports proxy rotation.

```python
async def start(self) -> None:
    """Create a browser for this instance and context."""
    if not self.playwright:
        self.playwright = await async_playwright().start()
        try:
            if self._config.cdp_url:
                self.browser = await self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
                if not self._config.proxy_rotator and self.browser:
                    self.context = await self.browser.new_context(**self._context_options)
            elif self._config.proxy_rotator:
                self.browser = await self.playwright.chromium.launch(**self._browser_options)
            else:
                persistent_options = (
                    self._browser_options | self._context_options | {"user_data_dir": self._user_data_dir}
                )
                self.context = await self.playwright.chromium.launch_persistent_context(**persistent_options)

            if self.context:
                self.context = await self._initialize_context(self._config, self.context)

            self._is_alive = True
        except Exception:
            # Clean up playwright if browser setup fails
            await self.playwright.stop()
            self.playwright = None
            raise
    else:
        raise RuntimeError("Session has been already started")
```

--------------------------------

### Install Chrome for Playwright

Source: https://scrapling.readthedocs.io/en/latest/fetching/dynamic.html

Terminal command to install the Chrome browser for use with the library.

```bash
playwright install chrome
```

--------------------------------

### serve(http, host, port, allowed_hosts=())

Source: https://scrapling.readthedocs.io/en/latest/api-reference/mcp-server.html

Initializes and runs the MCP server. The server can be configured to run over HTTP or stdio, with optional host and port settings.

```APIDOC
## serve(http, host, port, allowed_hosts=())

### Description
Starts the MCP server. If running over HTTP, it is recommended to provide an authentication token to prevent unauthorized access to tools.

### Parameters
- **http** (bool) - Required - If True, runs the server over streamable-http; if False, runs over stdio.
- **host** (str) - Required - The host address to bind the server to.
- **port** (int) - Required - The port number to bind the server to.
- **allowed_hosts** (Sequence[str]) - Optional - A list of allowed hostnames for the server.
```

--------------------------------

### Configure Sessions with Different Instances

Source: https://scrapling.readthedocs.io/en/latest/spiders/sessions.html

Demonstrates using multiple instances of the same fetcher class with different configurations, such as impersonating different browsers.

```python
from scrapling.spiders import Spider, Response
from scrapling.fetchers import FetcherSession

class ProductSpider(Spider):
    name = "products"
    start_urls = ["https://shop.example.com/products"]

    def configure_sessions(self, manager):
        chrome_requests = FetcherSession(impersonate="chrome")
        firefox_requests = FetcherSession(impersonate="firefox")

        manager.add("chrome", chrome_requests)
        manager.add("firefox", firefox_requests)

    async def parse(self, response: Response):
        for link in response.css("a.product::attr(href)").getall():
            yield response.follow(link, callback=self.parse_product)

        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield response.follow(next_page, sid="firefox")

    async def parse_product(self, response: Response):
        yield {
            "name": response.css("h1::text").get(""),
            "price": response.css(".price::text").get(""),
        }

```

--------------------------------

### scrapling extract get

Source: https://scrapling.readthedocs.io/en/latest/cli/extract-commands.html

Performs an HTTP GET request to a specified URL and saves the content to a file.

```APIDOC
## scrapling extract get

### Description
Perform a GET request and save the content to a file. The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself based on the file extension.

### Usage
`scrapling extract get [URL] [OUTPUT_FILE] [OPTIONS]`

### Options
- **-H, --headers** (TEXT) - Optional - HTTP headers in format "Key: Value" (can be used multiple times)
- **--cookies** (TEXT) - Optional - Cookies string in format "name1=value1;name2=value2"
- **--timeout** (INTEGER) - Optional - Request timeout in seconds (default: 30)
- **--proxy** (TEXT) - Optional - Proxy URL in format "http://username:password@host:port"
- **-s, --css-selector** (TEXT) - Optional - CSS selector to extract specific content from the page
- **-p, --params** (TEXT) - Optional - Query parameters in format "key=value" (can be used multiple times)
- **--follow-redirects** (BOOLEAN) - Optional - Whether to follow redirects (default: True)
- **--verify** (BOOLEAN) - Optional - Whether to verify SSL certificates (default: True)
- **--impersonate** (TEXT) - Optional - Browser to impersonate (e.g., chrome, firefox)
- **--stealthy-headers** (BOOLEAN) - Optional - Use stealthy browser headers (default: True)
- **--ai-targeted** (BOOLEAN) - Optional - Extract only main content and sanitize hidden elements for AI consumption
```

--------------------------------

### Get Previous Sibling: BeautifulSoup vs Scrapling

Source: https://scrapling.readthedocs.io/en/latest/tutorials/migrating_from_beautifulsoup.html

Shows how to get the immediately preceding sibling element.

```python
prev_element = element.previous_sibling
```

```python
prev_element = element.previous
```

--------------------------------

### Enter Context Manager

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Enters the context manager for the browser instance, starting the browser and returning the instance itself.

```python
def __enter__(self):
    self.start()
    return self

```

--------------------------------

### Get Next Sibling: BeautifulSoup vs Scrapling

Source: https://scrapling.readthedocs.io/en/latest/tutorials/migrating_from_beautifulsoup.html

Shows how to get the immediately following sibling element.

```python
next_element = element.next_sibling
```

```python
next_element = element.next
```

--------------------------------

### Asynchronous GET Requests with AsyncFetcher

Source: https://scrapling.readthedocs.io/en/latest/fetching/static.html

Illustrates how to perform asynchronous GET requests using AsyncFetcher. It mirrors the functionality of synchronous GET requests, including stealthy headers, proxies, parameters, custom headers, authentication, browser impersonation, and HTTP/3.

```python
from scrapling.fetchers import AsyncFetcher
# Basic GET
page = await AsyncFetcher.get('https://example.com')
page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')
# With parameters
 page = await AsyncFetcher.get('https://example.com/search', params={'q': 'query'})
# With headers
page = await AsyncFetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})
# Basic HTTP authentication
page = await AsyncFetcher.get("https://example.com", auth=("my_user", "password123"))
# Browser impersonation
page = await AsyncFetcher.get('https://example.com', impersonate='chrome110')
# HTTP/3 support
page = await AsyncFetcher.get('https://example.com', http3=True)
```

--------------------------------

### Install Browser Dependencies via Python

Source: https://scrapling.readthedocs.io/en/latest/index.html

Programmatic installation of browser dependencies using the Scrapling CLI module.

```python
from scrapling.cli import install

install([], standalone_mode=False)          # normal install
install(["--force"], standalone_mode=False) # force reinstall
```

--------------------------------

### Start Scrapling MCP Server via CLI

Source: https://scrapling.readthedocs.io/en/latest/api-reference/mcp-server.html

Run the server directly from the command line interface.

```bash
scrapling mcp
```

--------------------------------

### Perform an asynchronous GET request

Source: https://scrapling.readthedocs.io/en/latest/api-reference/mcp-server.html

Executes a GET request with configurable parameters such as impersonation, extraction type, and proxy settings.

```python
get(
    url,
    impersonate="chrome",
    extraction_type="markdown",
    css_selector=None,
    main_content_only=True,
    params=None,
    headers=None,
    cookies=None,
    timeout=30,
    follow_redirects="safe",
    max_redirects=30,
    retries=3,
    retry_delay=1,
    proxy=None,
    proxy_auth=None,
    auth=None,
    verify=True,
    http3=False,
    stealthy_headers=True,
)
```

--------------------------------

### Downloading Files

Source: https://scrapling.readthedocs.io/en/latest/fetching/static.html

Illustrates how to download a file from a URL and save its content to a local file in binary mode.

```python
from scrapling.fetchers import Fetcher

page = Fetcher.get('https://raw.githubusercontent.com/D4Vinci/Scrapling/main/docs/assets/main_cover.png')
with open(file='main_cover.png', mode='wb') as f:
   f.write(page.body)


```

--------------------------------

### Perform bulk GET requests with bulk_get()

Source: https://scrapling.readthedocs.io/en/latest/api-reference/mcp-server.html

Executes multiple asynchronous GET requests simultaneously using the provided configuration parameters.

```python
bulk_get(
    urls,
    impersonate="chrome",
    extraction_type="markdown",
    css_selector=None,
    main_content_only=True,
    params=None,
    headers=None,
    cookies=None,
    timeout=30,
    follow_redirects="safe",
    max_redirects=30,
    retries=3,
    retry_delay=1,
    proxy=None,
    proxy_auth=None,
    auth=None,
    verify=True,
    http3=False,
    stealthy_headers=True,
)
```

--------------------------------

### Follow Method

Source: https://scrapling.readthedocs.io/en/latest/api-reference/response.html

Follows a link to a new URL with optional parameters.

```python
follow(
    url,
    sid="",
    callback=None,
    priority=None,
    dont_filter=False,
    meta=None,
    referer_flow=True,
    **kwargs
)
```

--------------------------------

### Get Raw HTML Source: BeautifulSoup vs Scrapling

Source: https://scrapling.readthedocs.io/en/latest/tutorials/migrating_from_beautifulsoup.html

Illustrates how to get the raw, unformatted HTML content of a page or element.

```python
source = str(soup)
```

```python
source = page.html_content
```

--------------------------------

### Initialize Browser Session

Source: https://scrapling.readthedocs.io/en/latest/api-reference/mcp-server.html

Configures and starts a new browser session, either stealthy or dynamic, based on the provided parameters.

```python
    common_kwargs: Dict[str, Any] = dict(
        wait=wait,
        proxy=proxy,
        locale=locale,
        timeout=timeout,
        cookies=cookies,
        cdp_url=cdp_url,
        headless=headless,
        block_ads=True,
        max_pages=max_pages,
        useragent=useragent,
        timezone_id=timezone_id,
        real_chrome=real_chrome,
        network_idle=network_idle,
        wait_selector=wait_selector,
        google_search=google_search,
        extra_headers=extra_headers,
        executable_path=self._resolve_executable_path(executable_path),
        disable_resources=disable_resources,
        wait_selector_state=wait_selector_state,
    )

    session: Union[AsyncDynamicSession, AsyncStealthySession]
    if session_type == "stealthy":
        session = AsyncStealthySession(
            **common_kwargs,
            hide_canvas=hide_canvas,
            block_webrtc=block_webrtc,
            allow_webgl=allow_webgl,
            solve_cloudflare=solve_cloudflare,
            additional_args=additional_args,
        )
    else:
        session = AsyncDynamicSession(**common_kwargs)

    await session.start()

    entry = _SessionEntry(session=session, session_type=session_type)
    self._sessions[session_id] = entry

    return SessionCreatedModel(
        session_id=session_id,
        session_type=session_type,
        created_at=entry.created_at,
        is_alive=True,
        message=f"Session '{session_id}' ({session_type}) created successfully.",
    )
```

--------------------------------

### get(url, ...)

Source: https://scrapling.readthedocs.io/en/latest/api-reference/mcp-server.html

Makes a GET HTTP request to a URL and returns a structured output of the result. Suitable for low-mid protection levels.

```APIDOC
## get(url, ...)

### Description
Make GET HTTP request to a URL and return a structured output of the result. Note: This is only suitable for low-mid protection levels.

### Parameters
- **url** (str) - Required - The URL to request.
- **impersonate** (ImpersonateType) - Optional - Browser version to impersonate. Defaults to "chrome".
- **extraction_type** (extraction_types) - Optional - Type of content to extract (markdown, html, text). Defaults to "markdown".
- **css_selector** (str) - Optional - CSS selector to extract content.
- **main_content_only** (bool) - Optional - Whether to extract only the main content. Defaults to True.
- **params** (Dict) - Optional - Query string parameters.
- **headers** (Mapping) - Optional - Headers to include.
- **cookies** (Dict) - Optional - Cookies to use.
- **timeout** (int | float) - Optional - Timeout in seconds. Defaults to 30.
- **follow_redirects** (FollowRedirects) - Optional - Redirect behavior. Defaults to "safe".
- **max_redirects** (int) - Optional - Max redirects. Defaults to 30.
- **retries** (int) - Optional - Number of retries. Defaults to 3.
- **retry_delay** (int) - Optional - Delay between retries. Defaults to 1.
- **proxy** (str) - Optional - Proxy URL.
- **proxy_auth** (Dict) - Optional - Proxy authentication.
- **auth** (Dict) - Optional - Basic authentication.
- **verify** (bool) - Optional - Verify HTTPS certificates. Defaults to True.
- **http3** (bool) - Optional - Use HTTP3. Defaults to False.
- **stealthy_headers** (bool) - Optional - Use stealthy headers. Defaults to True.
```

--------------------------------

### get

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Sends an HTTP GET request to the specified URL asynchronously. This class method is part of the AsyncFetcher and returns an awaitable Response object.

```APIDOC
## GET /url

### Description
Sends an HTTP GET request to the specified URL asynchronously.

### Method
GET

### Endpoint
/url

### Parameters
#### Path Parameters
- **url** (string) - Required - The URL to send the GET request to.

#### Query Parameters
- **kwargs** (dict) - Optional - Additional keyword arguments to pass to the request.

### Request Example
```json
{
  "url": "http://example.com/data"
}
```

### Response
#### Success Response (200)
- **Response** (object) - An awaitable response object from the GET request.

#### Response Example
```json
{
  "status_code": 200,
  "body": "..."
}
```
```

--------------------------------

### Perform HTTP GET Request

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

A class method to perform an HTTP GET request. It merges class-level configuration with provided keyword arguments.

```python
@classmethod
def get(cls, url: str, **kwargs: Unpack[GetRequestParams]) -> Response:
    return __FetcherClientInstance__.get(url, **_merge_selector_config(cls, kwargs))
```

--------------------------------

### Async Context Manager Entry

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Asynchronous entry point for context management. It starts the browser instance and returns the current instance.

```python
async def __aenter__(self):
    await self.start()
    return self
```

--------------------------------

### Serve MCP Server

Source: https://scrapling.readthedocs.io/en/latest/api-reference/mcp-server.html

Initializes and runs the MCP server with specified transport settings.

```python
serve(http, host, port, allowed_hosts=())
```

```python
def serve(self, http: bool, host: str, port: int, allowed_hosts: Sequence[str] = ());
    """Serve the MCP server."""
    if http and not self._auth_token:
        log.warning(
            f"The MCP server is running over HTTP without authentication, so anyone who can reach "
            f"{host}:{port} can use every tool, including fetching arbitrary URLs from this machine. "
            f"Pass `--auth-token` (or set the {MCP_AUTH_TOKEN_ENV} environment variable) to require a bearer token."
        )
    elif self._auth_token and not http:
        log.warning(
            "The authentication token only applies to the streamable-http transport, so it's ignored with stdio."
        )

    self._build_server(host, port, allowed_hosts).run(transport="stdio" if not http else "streamable-http")
```

--------------------------------

### Install Scrapling Shell Dependency

Source: https://scrapling.readthedocs.io/en/latest/cli/overview.html

Install the 'shell' extra dependency group for Scrapling using pip. This is required to use the interactive shell functionality.

```bash
pip install "scrapling[shell]"
```

--------------------------------

### Get text of the first H1 element (alternative method)

Source: https://scrapling.readthedocs.io/en/latest/parsing/selection.html

An alternative way to get the text content is by selecting the element first and then accessing its `.text` attribute.

```python
title = page.css('h1')[0].text
title = page.xpath('//h1')[0].text
```

--------------------------------

### Simple HTTP GET Request

Source: https://scrapling.readthedocs.io/en/latest/overview.html

Use the Fetcher class for basic HTTP GET requests. The 'impersonate' argument can be used to mimic specific browser versions.

```python
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://scrapling.requestcatcher.com/get', impersonate="chrome")
```

--------------------------------

### View FETCH command options

Source: https://scrapling.readthedocs.io/en/latest/cli/extract-commands.html

Displays the help menu for the extract fetch command.

```bash
Usage: scrapling extract fetch [OPTIONS] URL OUTPUT_FILE

  Use DynamicFetcher to fetch content with browser automation.

  The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.

Options:
  --headless / --no-headless                  Run browser in headless mode (default: True)
  --disable-resources / --enable-resources    Drop unnecessary resources for speed boost (default: False)
  --network-idle / --no-network-idle          Wait for network idle (default: False)
  --timeout INTEGER                           Timeout in milliseconds (default: 30000)
  --wait INTEGER                              Additional wait time in milliseconds after page load (default: 0)
  -s, --css-selector TEXT                     CSS selector to extract specific content from the page. It returns all matches.
  --wait-selector TEXT                        CSS selector to wait for before proceeding
  --locale TEXT                               Specify user locale. Defaults to the system default locale.
  --real-chrome/--no-real-chrome              If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)
  --proxy TEXT                                Proxy URL in format "http://username:password@host:port"
  -H, --extra-headers TEXT                    Extra headers in format "Key: Value" (can be used multiple times)
  --dns-over-https / --no-dns-over-https   Route DNS through Cloudflare's DoH to prevent DNS leaks when using proxies (default: False)
  --block-ads / --no-block-ads               Block requests to known ad and tracker domains (default: False)
  --executable-path TEXT                      Path to a custom Chromium-compatible browser executable. Falls back to the SCRAPLING_EXECUTABLE_PATH environment variable when not set.
  --ai-targeted                              Extract only main content and sanitize hidden elements for AI consumption (default: False)
  --help                                      Show this message and exit.
```

--------------------------------

### Fetch with Cloudflare and stealth options

Source: https://scrapling.readthedocs.io/en/latest/fetching/stealthy.html

Demonstrates enabling the automatic Cloudflare solver and combining it with various stealth settings and proxy configurations.

```python
# Automatic Cloudflare solver
page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)

# Works with other stealth options
page = StealthyFetcher.fetch(
    'https://protected-site.com',
    solve_cloudflare=True,
    block_webrtc=True,
    real_chrome=True,
    hide_canvas=True,
    google_search=True,
    proxy='http://username:password@host:port',  # It can also be a dictionary with only the keys 'server', 'username', and 'password'.
)
```

--------------------------------

### Asynchronous GET Request

Source: https://scrapling.readthedocs.io/en/latest/api-reference/fetchers.html

Use this class method to perform an asynchronous GET HTTP request. It merges selector configuration with provided keyword arguments and returns an awaitable response.

```python
@classmethod
def get(cls, url: str, **kwargs: Unpack[GetRequestParams]) -> Awaitable[Response]:
    return __AsyncFetcherClientInstance__.get(url, **_merge_selector_config(cls, kwargs))
```
Tokens:
Raw