web-poet

https://web-poet.readthedocs.io/en/stable
Admin
web-poet is a Python implementation of the page object pattern for web scraping that enables writing...
Tokens:101,752
Snippets:704
Trust Score:8.3
Update:8 months ago
Show doc for...
Tokens:
Raw
### Run Web-Poet Example Script (Python)

Source: https://web-poet.readthedocs.io/en/stable/_sources/intro/tutorial.rst

This Python script demonstrates how to use the `web_poet.example.get_item` function to retrieve data using a defined page object. It requires installing the `requests` library and calls `consume_modules` to register page objects before fetching an item.

```python
from web_poet.rules import consume_modules
from web_poet.example import get_item

# Assuming Book and BookPage are defined in a module named 'my_pages'
consume_modules("my_pages")

item = await get_item("http://books.toscrape.com/", Book)
print(item)

```

--------------------------------

### Example Framework

Source: https://web-poet.readthedocs.io/en/stable/api-reference

Functionality for the example framework.

```APIDOC
## Example Framework

### Description
Functionality related to the example framework.

### Functions
- `get_item()`
```

--------------------------------

### Install web-poet using pip

Source: https://web-poet.readthedocs.io/en/stable/_sources/intro/install.rst

This command installs the web-poet library from the Python Package Index (PyPI). Ensure you have pip installed and accessible in your environment.

```bash
pip install web-poet
```

--------------------------------

### Example Framework API

Source: https://web-poet.readthedocs.io/en/stable/_sources/api-reference.rst

Information about the example framework, intended for educational purposes and not for production use.

```APIDOC
## Example Framework API

### Description
The `web_poet.example` module provides a simplified, incomplete example of a web-poet framework. It is intended as support material for the tutorial and should not be used in production environments.

### Module
- `web_poet.example`: Contains example framework code.
```

--------------------------------

### Framework integration example (Python)

Source: https://web-poet.readthedocs.io/en/stable/_sources/intro/from-ground-up.rst

A conceptual example showing how a web-poet page object class (BookPage) would be used with an external framework to fetch and parse web content. The framework handles the downloading, and the page object handles the parsing.

```python
item = some_framework.get(url, BookPage)
```

--------------------------------

### Example Framework API

Source: https://web-poet.readthedocs.io/en/stable/api-reference

An example framework function for retrieving items from a URL using a page object.

```APIDOC
## POST /example/get_item

### Description
Returns an item built from the specified URL using a page object class from the default registry. This function is an example of a minimal, incomplete web-poet framework implementation, intended for use in the web-poet tutorial.

### Method
POST

### Endpoint
/example/get_item

### Parameters
#### Request Body
- **_url** (str) - Required - The URL to fetch the item from.
- **_item_cls** (type) - Required - The page object class to use for building the item.
- **_page_params** (dict[Any, Any] | None) - Optional - Additional parameters for the page object.

### Response
#### Success Response (200)
- **item** (Any) - The built item.

#### Response Example
```json
{
  "item": { "title": "Example Page", "content": "This is an example."} 
}
```
```

--------------------------------

### Example Framework Item Retrieval (Python)

Source: https://web-poet.readthedocs.io/en/stable/api-reference

A function from a simplified, incomplete example framework for web-poet, intended for tutorial support. It retrieves an item from a URL using a page object class from a default registry. This is not for production use.

```python
from typing import Any, Dict

def get_item(_url : str, _item_cls : type, *_ , _page_params : Dict[Any, Any] | None = None) -> Any:
    """Returns an item built from the specified URL using a page object class."""
    # Implementation details would go here
    pass
```

--------------------------------

### HTTP Response Handling in web_poet.example

Source: https://web-poet.readthedocs.io/en/stable/_modules/web_poet/example

This snippet demonstrates how to fetch an HTTP response and convert it into a web_poet HttpResponse object. It uses the requests library for making the HTTP GET request and extracts relevant information like URL, status code, content, and headers. This is a core part of fetching web page data within the example framework.

```python
from __future__ import annotations

from asyncio import run
from typing import Any
from warnings import warn

import andi
import requests

from . import default_registry
from .page_inputs import HttpClient, HttpResponse, PageParams
from .pages import ItemPage, is_injectable
from .utils import ensure_awaitable

warn(
    (
        "You should only be importing web_poet.example to follow the web-poet "
        "tutorial, never as part of production code."
    ),
    UserWarning,
    stacklevel=2,
)


class _HttpClient:
    async def get(self, url: str) -> HttpResponse:
        return _get_http_response(url)


def _get_http_response(url: str) -> HttpResponse:
    response = requests.get(url)
    return HttpResponse(
        response.url,
        status=response.status_code,
        body=response.content,
        headers=response.headers,
    )
```

--------------------------------

### Making HTTP Requests with HttpClient

Source: https://web-poet.readthedocs.io/en/stable/_sources/page-objects/additional-requests.rst

This snippet shows various ways to make HTTP requests using the HttpClient. It includes examples for GET, POST, and generic request methods, demonstrating how to pass URLs, bodies, and custom headers. It also mentions that methods can raise HttpError or return HttpResponse.

```python
http = HtpClient()
response = await http.get(url)
response = await http.post(url, body=b"...")
response = await http.request(url, method="...")
response = await http.execute(HttpRequest(url, method="..."))
```

```python
http.post(
    url,
    headers={"Content-Type": "application/json;charset=UTF-8"},
    body=json.dumps({"foo": "bar"}).encode("utf-8"),
)
```

--------------------------------

### HttpClient Initialization and Request Execution in Python

Source: https://web-poet.readthedocs.io/en/stable/_modules/web_poet/page_inputs/client

Demonstrates the initialization of the HttpClient class, which can optionally take a custom request downloader. It also shows how to execute a request using the `request` method, which creates an HttpRequest object and handles the response, including status code validation. The `get` method is a convenience shortcut for making GET requests.

```python
from web_poet.page_inputs.client import HttpClient

# Initialize HttpClient with a custom downloader (optional)
client = HttpClient(request_downloader=my_downloader_function)

# Or use the default downloader
client = HttpClient()

# Make a GET request
response = await client.get("https://example.com")

# Make a request with custom method, headers, and body
response = await client.request(
    url="https://example.com/submit",
    method="POST",
    headers={"Content-Type": "application/json"},
    body=b'{"key": "value"}'
)

# Allow specific status codes
response = await client.request(
    url="https://example.com/optional",
    allow_status="404"
)

# Allow any status code
response = await client.request(
    url="https://example.com/any_status",
    allow_status="*"
)
```

--------------------------------

### Simulate Button Click and Fetch Product Images with HttpClient

Source: https://web-poet.readthedocs.io/en/stable/_sources/page-objects/additional-requests.rst

This example demonstrates how to use HttpClient within a web-poet page object to simulate a button click, which triggers a background request to fetch product images. It includes error handling for HTTP requests and returns a list of Image items.

```python
import attrs
from web_poet import HttpClient, HttpError, field
from zyte_common_items import Image, ProductPage


@attrs.define
class MyProductPage(ProductPage):
    http: HttpClient

    @field
    def productId(self):
        return self.css("::attr(product-id)").get()

    @field
    async def images(self):
        url = f"https://api.example.com/v2/images?id={self.productId}"
        try:
            response = await self.http.get(url)
        except HttpError:
            return []
        else:
            urls = response.css(".product-images img::attr(src)").getall()
            return [Image(url=url) for url in urls]
```

--------------------------------

### Get Item from URL using web_poet.example

Source: https://web-poet.readthedocs.io/en/stable/_modules/web_poet/example

This function serves as the main entry point for retrieving an item from a given URL. It finds the appropriate page object class from the default registry, instantiates it using `_get_page`, and then calls the `to_item` method to extract and return the desired item. It handles potential errors if no page class is found for the URL.

```python
[docs]
def get_item(
    url: str,
    item_cls: type,
    *, 
    page_params: dict[Any, Any] | None = None,
) -> Any:
    """Returns an item built from the specified URL using a page object class
    from the default registry.

    This function is an example of a minimal, incomplete web-poet framework
    implementation, intended for use in the web-poet tutorial.
    """
    page_cls = default_registry.page_cls_for_item(url, item_cls)
    if page_cls is None:
        raise ValueError(f"No page object class found for URL: {url}")
    page = _get_page(url, page_cls, page_params=page_params)
    return run(ensure_awaitable(page.to_item()))
```

--------------------------------

### Get All Rules

Source: https://web-poet.readthedocs.io/en/stable/api-reference

Retrieves a list of all `ApplyRule` objects declared using the `@handle_urls` decorator. It's recommended to call `consume_modules()` beforehand to ensure all relevant rules are loaded.

```APIDOC
## GET /rules

### Description
Retrieves all `ApplyRule` objects registered in the system.

### Method
GET

### Endpoint
/rules

### Parameters
None

### Request Example
None

### Response
#### Success Response (200)
- **rules** (list[ApplyRule]) - A list of all registered ApplyRule objects.

#### Response Example
```json
[
  {
    "priority": 100,
    "use": "com.example.pages.HomePage",
    "include": [
      "https://example.com"
    ],
    "exclude": [],
    "override": null
  }
]
```
```

--------------------------------

### Get All Rules

Source: https://web-poet.readthedocs.io/en/stable/_modules/web_poet/rules

Retrieves a list of all declared ApplyRule objects managed by the registry. It's recommended to call `consume_modules` beforehand to ensure all submodules with `@handle_urls` decorators are imported.

```APIDOC
## GET /rules

### Description
Return all the :class:`~.ApplyRule` that were declared using the ``@handle_urls`` decorator.

### Method
GET

### Endpoint
/rules

### Parameters
#### Query Parameters
- **consume_modules** (boolean) - Optional - Whether to recursively import submodules containing ``@handle_urls`` decorators.

### Response
#### Success Response (200)
- **rules** (list[ApplyRule]) - A list of ApplyRule objects.

#### Response Example
```json
[
  {
    "use": "ProductPO",
    "instead_of": null,
    "meta": {},
    "priority": 0
  }
]
```
```

--------------------------------

### Define Rule Manually with ApplyRule - Python

Source: https://web-poet.readthedocs.io/en/stable/_sources/page-objects/rules.rst

This example shows how to manually create and register an ApplyRule object. It explicitly defines URL patterns, the page object class to use (MyPage), and the return type (MyItem). This provides more granular control over rule creation and registration with the default registry.

```python
from url_matcher import Patterns
from web_poet import ApplyRule, ItemPage, default_registry

from my_items import MyItem


class MyPage(ItemPage[MyItem]): ...


rule = ApplyRule(
    for_patterns=Patterns(include=["example.com"]),
    use=MyPage,
    to_return=MyItem,
)
default_registry.add_rule(rule)

```

--------------------------------

### Manually Build Page Object Instance

Source: https://web-poet.readthedocs.io/en/stable/page-objects/index

Provides an example of manually creating an instance of a page object class (`FooPage`). This involves directly instantiating the class and passing the required input parameters, such as an `HttpResponse` object, to its constructor. This method is less robust against changes in the page object's signature.

```python
from web_poet import HttpResponse

foo_page = FooPage(
    response=HttpResponse(
        "https://example.com",
        b"<!DOCTYPE html>\n<title>Foo</title>",
    ),
)
```

--------------------------------

### Get Fully Qualified Class Name

Source: https://web-poet.readthedocs.io/en/stable/_modules/web_poet/utils

Retrieves the fully qualified name of a given type (class). This includes the module path and the class name, useful for uniquely identifying types. Example usage with web_poet.Injectable and decimal.Decimal is provided.

```python
from __future__ import annotations

from web_poet import Injectable
from decimal import Decimal

def get_fq_class_name(cls: type) -> str:
    """Return the fully qualified name for a type.

    >>> from web_poet import Injectable
    >>> get_fq_class_name(Injectable)
    'web_poet.pages.Injectable'
    >>> from decimal import Decimal
    >>> get_fq_class_name(Decimal)
    'decimal.Decimal'
    """
    return f"{cls.__module__}.{cls.__qualname__}"

```

--------------------------------

### Perform GET Request using HttpClient

Source: https://web-poet.readthedocs.io/en/stable/_modules/web_poet/page_inputs/client

The `get` method is a convenience wrapper around the `request` method to perform an HTTP GET request. It takes a URL and optional headers, and returns an HttpResponse. It simplifies making GET requests by pre-setting the method to 'GET'.

```python
async def get(self,
        url: str | _Url,
        *, 
        headers: _Headers | None = None,
        allow_status: _StatusList | None = None,
    ) -> HttpResponse:
        """Similar to :meth:`~.HttpClient.request` but peforming a ``GET``
        request.
        """
        return await self.request(
            url=url,
            method="GET",
            headers=headers,
            allow_status=allow_status,
        )
```

--------------------------------

### Execute Web-Poet Tutorial Script (Bash)

Source: https://web-poet.readthedocs.io/en/stable/_sources/intro/tutorial.rst

This command executes the Python script `run.py` located in the `tutorial-project` directory. This script is part of the web-poet tutorial and demonstrates the usage of page objects and the `get_item` function.

```bash
python tutorial-project/run.py

```

--------------------------------

### Page Object Instantiation in web_poet.example

Source: https://web-poet.readthedocs.io/en/stable/_modules/web_poet/example

This function illustrates how to instantiate a page object using dependency injection with the 'andi' library. It plans the dependencies required by the page class (HttpClient, HttpResponse, PageParams) and provides them by calling helper functions or creating instances. This is crucial for setting up the page object with the necessary data before extracting items.

```python
def _get_page(
    url: str,
    page_cls: type[ItemPage],
    *, 
    page_params: dict[Any, Any] | None = None,
) -> ItemPage:
    plan = andi.plan(
        page_cls,
        is_injectable=is_injectable,
        externally_provided={
            HttpClient,
            HttpResponse,
            PageParams,
        },
    )
    instances: dict[Any, Any] = {}
    for fn_or_cls, kwargs_spec in plan:
        if fn_or_cls is HttpResponse:
            instances[fn_or_cls] = _get_http_response(url)
        elif fn_or_cls is HttpClient:
            instances[fn_or_cls] = _HttpClient()
        elif fn_or_cls is PageParams:
            instances[fn_or_cls] = PageParams(page_params or {})
        else:
            instances[fn_or_cls] = fn_or_cls(**kwargs_spec.kwargs(instances))
    return instances[page_cls]
```

--------------------------------

### HttpClient GET Method

Source: https://web-poet.readthedocs.io/en/stable/_modules/web_poet/page_inputs/client

A convenience method for making GET requests.

```APIDOC
## GET /get

### Description
A convenience method for making GET requests. It's a shortcut for calling `request` with the method set to 'GET'.

### Method
`get`

### Parameters
- **url** (str | _Url) - Required - The URL for the GET request.
- **headers** (_Headers | None) - Optional - Headers to include in the request.

### Response
#### Success Response (HttpResponse)
- **response** (HttpResponse) - The HttpResponse object for successful GET requests.

#### Error Response (HttpError or HttpResponseError)
- **exception** (HttpError | HttpResponseError) - Raised for connection errors, timeouts, or disallowed HTTP status codes.
```

--------------------------------

### HttpRequestHeaders Initialization

Source: https://web-poet.readthedocs.io/en/stable/api-reference

Shows how to instantiate HttpRequestHeaders using either an iterable of tuples or a dictionary of key-value pairs. It also highlights the case-insensitive lookup functionality inherited from multidict.CIMultiDict.

```python
from web_poet.page_inputs.http import HttpRequestHeaders

# Initialization with an iterable of tuples
pairs_tuple = [("Content-Encoding", "gzip"), ("content-length", "648")]
headers_from_tuple = HttpRequestHeaders(pairs_tuple)
print(headers_from_tuple)

# Initialization with a mapping of key-value pairs
pairs_dict = {"Content-Encoding": "gzip", "content-length": "648"}
headers_from_dict = HttpRequestHeaders(pairs_dict)
print(headers_from_dict)

# Case-insensitive lookup
print(headers_from_dict.get("content-encoding"))
print(headers_from_dict.get("Content-Length"))
```

--------------------------------

### GET Request

Source: https://web-poet.readthedocs.io/en/stable/_modules/web_poet/page_inputs/client

Performs an HTTP GET request to the specified URL. This is a convenience method that wraps the more general `request` method.

```APIDOC
## GET /url

### Description
Performs an HTTP GET request to the specified URL.

### Method
GET

### Endpoint
/url

### Parameters
#### Query Parameters
- **url** (str | _Url) - Required - The URL to send the GET request to.
- **headers** (_Headers | None) - Optional - A dictionary of headers to include in the request.
- **allow_status** (_StatusList | None) - Optional - A list of status codes to allow without raising an exception.

### Request Example
```json
{
  "url": "https://example.com",
  "headers": {
    "User-Agent": "MyClient/1.0"
  }
}
```

### Response
#### Success Response (200)
- **HttpResponse** - The HTTP response object.

#### Response Example
```json
{
  "status_code": 200,
  "body": "<html>...</html>"
}
```
```

--------------------------------

### Registering Rules with RulesRegistry in Python

Source: https://web-poet.readthedocs.io/en/stable/api-reference

Demonstrates how to use the RulesRegistry to store and retrieve ApplyRule instances. It shows how to access the default registry and use the `handle_urls` decorator to define rules for Page Objects.

```python
from web_poet import handle_urls, default_registry, WebPage
from my_items import Product


@handle_urls("example.com")
class ExampleComProductPage(WebPage[Product]): ...


rules = default_registry.get_rules()
```

--------------------------------

### Get Saved Responses

Source: https://web-poet.readthedocs.io/en/stable/api-reference

Retrieves previously saved requests and responses.

```APIDOC
## GET /get_saved_responses

### Description
Retrieves a collection of saved requests and responses.

### Method
GET

### Endpoint
/get_saved_responses

### Parameters
#### Path Parameters
None

#### Query Parameters
None

### Request Body
None

### Response
#### Success Response (200)
- **Iterable[_SavedResponseData]** - An iterable containing data for saved requests and responses.

#### Response Example
```json
[
  {
    "request": {
      "method": "GET",
      "url": "http://example.com",
      "headers": {},
      "body": null
    },
    "response": {
      "status_code": 200,
      "headers": {"Content-Type": "text/html"},
      "body": "<html><body>Hello</body></html>"
    }
  }
]
```
```

--------------------------------

### Separate download and parse functions

Source: https://web-poet.readthedocs.io/en/stable/_sources/intro/from-ground-up.rst

Refactors the scraping code into two distinct functions: 'download' for fetching the web page and 'parse' for extracting data. This separation improves modularity and allows for easier replacement of the download mechanism.

```python
import requests
from parsel import Selector


def parse(response: requests.Response) -> dict:
    selector = Selector(response.text)
    return {
        "url": response.url,
        "title": selector.css("h1").get(),
    }


def download(url: str) -> requests.Response:
    return requests.get(url)


url = "http://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
response = download(url)
item = parse(response)
```

--------------------------------

### Simplify Page Object Access with WebPage

Source: https://web-poet.readthedocs.io/en/stable/intro/from-ground-up

Further simplifies the page object by leveraging WebPage's shortcuts for accessing response attributes and methods directly.

```python
from web_poet import WebPage


class BookPage(WebPage):
    def to_item(self) -> dict:
        return {
            "url": self.url,
            "title": self.css("h1").get(),
        }


```

--------------------------------

### Get Saved Responses from Web-Poet

Source: https://web-poet.readthedocs.io/en/stable/_modules/web_poet/page_inputs/client

Retrieves all saved requests and responses from the internal storage. This method returns an iterable of _SavedResponseData objects.

```python
def get_saved_responses(self) -> Iterable[_SavedResponseData]:
    """Return saved requests and responses."""
    return self._saved_responses.values()
```

--------------------------------

### RequestDownloaderVarError Exception (Python)

Source: https://web-poet.readthedocs.io/en/stable/api-reference

The `RequestDownloaderVarError` is raised when the `web_poet.request_downloader_var` context variable is accessed before any value has been set. This typically occurs when request downloading is attempted without proper context setup.

```python
from web_poet.exceptions import RequestDownloaderVarError

# Example of catching the exception:
# try:
#     # code that uses request_downloader_var without it being set
# except RequestDownloaderVarError as e:
#     print(f"Context error: {e}")
```

--------------------------------

### Create a Web-Poet Page Object Class

Source: https://web-poet.readthedocs.io/en/stable/intro/from-ground-up

Demonstrates how to create a basic page object class using web-poet's Injectable base class. It takes an HttpResponse object and provides a to_item method to extract data.

```python
import requests
from web_poet import Injectable, HttpResponse


class BookPage(Injectable):
    def __init__(self, response: HttpResponse):
        self.response = response

    def to_item(self) -> dict:
        return {
            "url": self.response.url,
            "title": self.response.css("h1").get(),
        }


def download(url: str) -> Response:
    response = requests.get(url)
    return HttpResponse(
        url=response.url,
        body=response.content,
        headers=response.headers,
    )


url = "http://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
response = download(url)
book_page = BookPage(response=response)
item = book_page.to_item()

```

--------------------------------

### HttpResponseHeaders Initialization

Source: https://web-poet.readthedocs.io/en/stable/api-reference

Shows how to instantiate HttpResponseHeaders using an iterable of tuples, similar to HttpRequestHeaders. This class is designed to hold HTTP response headers.

```python
from web_poet.page_inputs.http import HttpResponseHeaders

pairs = [("Content-Encoding", "gzip"), ("content-length", "648")]
response_headers = HttpResponseHeaders(pairs)
print(response_headers)
```

--------------------------------

### Ensure Object is Awaitable (Python)

Source: https://web-poet.readthedocs.io/en/stable/api-reference

Returns the value of an object, awaiting it if it is an awaitable (e.g., a coroutine or a future). This function simplifies handling of asynchronous operations by providing a uniform way to get results.

```python
from typing import Any

async def ensure_awaitable(_obj_: Any) -> Any:
    """Return the value of obj, awaiting it if needed."""
    # Implementation details would go here
    pass
```

--------------------------------

### Define BookPage with fields using web-poet (Python)

Source: https://web-poet.readthedocs.io/en/stable/_sources/intro/from-ground-up.rst

Illustrates how to use web_poet's @field decorator to define individual parsing logic for fields like 'url' and 'title'. This approach eliminates the need for a manual to_item method and allows for lazy field evaluation.

```python
from web_poet import WebPage, field


class BookPage(WebPage):
    @field
    def url(self):
        return self.url

    @field
    def title(self):
        return self.css("h1").get()
```

--------------------------------

### Get Fully Qualified Class Name (Python)

Source: https://web-poet.readthedocs.io/en/stable/api-reference

Retrieves the fully qualified name of a given type. This is useful for introspection and serialization purposes. It handles built-in types and custom classes alike.

```python
from web_poet import Injectable

def get_fq_class_name(_cls : type_) -> str:
    """Return the fully qualified name for a type."""
    # Implementation details would go here
    pass

print(get_fq_class_name(Injectable))
from decimal import Decimal
print(get_fq_class_name(Decimal))
```

--------------------------------

### Get All Rules Declared with @handle_urls

Source: https://web-poet.readthedocs.io/en/stable/_modules/web_poet/rules

Retrieves a list of all ApplyRule objects that have been registered using the `@handle_urls` decorator. This function is useful for inspecting all available rules in the registry.

```python
def get_rules(self) -> list[ApplyRule]:
    """Return all the :class:`~.ApplyRule` that were declared using
    the ``@handle_urls`` decorator.

    .. note::

        Remember to consider calling :func:`~.web_poet.rules.consume_modules`
        beforehand to recursively import all submodules which contains the
        ``@handle_urls`` decorators from external Page Objects.
    """
    return list(self._rules.values())
```

--------------------------------

### Create HttpResponseHeaders from Bytes Dictionary (Python)

Source: https://web-poet.readthedocs.io/en/stable/api-reference

Shows how to instantiate HttpResponseHeaders from a dictionary where keys and values are in bytes format. It supports lists or tuples of bytes for multi-value headers and allows specifying the decoding encoding.

```python
>>> raw_values = {
...     b"Content-Encoding": [b"gzip", b"br"],
...     b"Content-Type": [b"text/html"],
...     b"content-length": b"648",
... }
>>> headers = _HttpHeaders.from_bytes_dict(raw_values)
>>> headers
<_HttpHeaders('Content-Encoding': 'gzip', 'Content-Encoding': 'br', 'Content-Type': 'text/html', 'content-length': '648')>
```

--------------------------------

### Get Generic Type Parameter (Python)

Source: https://web-poet.readthedocs.io/en/stable/api-reference

Recursively searches base classes breadth-first to find a generic class and return its parameter. It returns the parameter of the first found class that is a subclass of the expected type.

```python
from typing import TypeVar, Union, Tuple

TypeVarT = TypeVar('TypeVarT')

def get_generic_param(_cls : type, _expected : Union[type, Tuple[type, ...]]) -> Union[type, None]:
    """Search the base classes recursively breadth-first for a generic class and return its param."""
    # Implementation details would go here
    pass
```

--------------------------------

### Create HttpResponseHeaders from Key-Value Pairs (Python)

Source: https://web-poet.readthedocs.io/en/stable/api-reference

Demonstrates creating an HttpResponseHeaders object by passing a dictionary of key-value pairs. This constructor supports case-insensitive lookups for header keys.

```python
>>> pairs = {"Content-Encoding": "gzip", "content-length": "648"}
>>> headers = HttpResponseHeaders(pairs)
>>> headers
<HttpResponseHeaders('Content-Encoding': 'gzip', 'content-length': '648')>

>>> headers.get("content-encoding")
'gzip'
>>> headers.get("Content-Length")
'648'
```

--------------------------------

### Defining URL Patterns with ApplyRule in Python

Source: https://web-poet.readthedocs.io/en/stable/api-reference

Illustrates the use of the ApplyRule class to associate URL patterns with specific Page Objects. It explains the parameters like `for_patterns`, `use`, `instead_of`, and `to_return` for configuring rule behavior.

```python
from web_poet.rules import ApplyRule
from web_poet import ItemPage

# Example of creating an ApplyRule instance
rule = ApplyRule(
    for_patterns="example.com/products/*",
    use=ProductPage,
    instead_of=AnotherProductPage,
    to_return=Product,
    meta={"version": 1}
)
```

--------------------------------

### Refactor Page Object with WebPage Base Class

Source: https://web-poet.readthedocs.io/en/stable/intro/from-ground-up

Shows how to simplify a web-poet page object by inheriting from the WebPage base class, which provides a default __init__ method for the response object.

```python
from web_poet import WebPage


class BookPage(WebPage):
    def to_item(self) -> dict:
        return {
            "url": self.response.url,
            "title": self.response.css("h1").get(),
        }


```

--------------------------------

### Define RequestDownloaderVarError Exception for web-poet

Source: https://web-poet.readthedocs.io/en/stable/_modules/web_poet/exceptions/core

Defines the RequestDownloaderVarError exception, raised when the web_poet.request_downloader_var is accessed without a value set during request execution. Refer to the contextvars setup documentation for more details.

```python
class RequestDownloaderVarError(Exception):
    """The ``web_poet.request_downloader_var`` had its contents accessed but there
    wasn't any value set during the time requests are executed.

    See the documentation section about :ref:`setting up the contextvars <setup-contextvars>`
    to learn more about this.
    """
    pass
```

--------------------------------

### Get Fields Dictionary in Python

Source: https://web-poet.readthedocs.io/en/stable/_modules/web_poet/fields

This Python function retrieves a dictionary containing information about fields defined for a given class or instance. The keys are field names, and the values are FieldInfo instances.

```python
def get_fields_dict(cls_or_instance) -> dict[str, FieldInfo]:
    """Return a dictionary with information about the fields defined
    for the class: keys are field names, and values are
    :class:`web_poet.fields.FieldInfo` instances.
    """
    return getattr(cls_or_instance, _FIELDS_INFO_ATTRIBUTE_READ, {})
```

--------------------------------

### Define BookPage using WebPage shortcuts (Python)

Source: https://web-poet.readthedocs.io/en/stable/_sources/intro/from-ground-up.rst

Shows a refactored BookPage class inheriting from web_poet.WebPage, utilizing shortcuts provided by the base class for accessing response URL and CSS selectors directly. This simplifies the to_item method.

```python
from web_poet import WebPage


class BookPage(WebPage):
    def to_item(self) -> dict:
        return {
            "url": self.url,
            "title": self.css("h1").get(),
        }
```

--------------------------------

### Get Overriding Page Objects for URL

Source: https://web-poet.readthedocs.io/en/stable/_modules/web_poet/rules

Identifies page objects that override others for a specific URL. It returns a mapping where keys are the overridden page objects and values are the overriding ones.

```python
def overrides_for(self, url: _Url | str) -> Mapping[type[ItemPage], type[ItemPage]]:
    """Finds all of the page objects associated with the given URL and
    returns a Mapping where the 'key' represents the page object that is
    **overridden** by the page object in 'value'."""
    result: dict[type[ItemPage], type[ItemPage]] = {}
    for replaced_page, matcher in self._overrides_matchers.items():
        if replaced_page is None:
            continue
        page = self._match_url_for_page_object(url, matcher)
        if page:
            result[replaced_page] = page
    return result
```

--------------------------------

### Run pytest with Web-Poet test options

Source: https://web-poet.readthedocs.io/en/stable/_sources/page-objects/testing.rst

Demonstrates how to run pytest with specific options for Web-Poet to control test reporting granularity. The `--web-poet-test-per-item` flag changes the test structure from one test per field to one test per fixture.

```bash
python -m pytest --web-poet-test-per-item
```

--------------------------------

### Define BookPage with WebPage inheritance (Python)

Source: https://web-poet.readthedocs.io/en/stable/_sources/intro/from-ground-up.rst

Demonstrates a basic BookPage class inheriting from web_poet.WebPage, implementing the to_item method to extract URL and title from response data. This leverages the base class's __init__ for response handling.

```python
from web_poet import WebPage


class BookPage(WebPage):
    def to_item(self) -> dict:
        return {
            "url": self.response.url,
            "title": self.response.css("h1").get(),
        }
```

--------------------------------

### Web-Poet Page Object with Time Fields

Source: https://web-poet.readthedocs.io/en/stable/_sources/page-objects/testing.rst

An example of a Web-Poet page object that extracts and formats time-based data. It demonstrates capturing current time in UTC and local time, highlighting the importance of timezone handling in tests.

```python
import datetime
from web_poet import WebPage, validates_input

class DateItemPage(WebPage):
    @validates_input
    async def to_item(self) -> dict:
        # e.g. 2001-01-01 11:00:00 +00
        now = datetime.datetime.now(datetime.timezone.utc)
        return {
            # '2001-01-01T11:00:00Z'
            "time_utc": now.strftime("%Y-%M-%dT%H:%M:%SZ"),
            # if the current timezone is CET, then '2001-01-01T12:00:00+01:00'
            "time_local": now.astimezone().strftime("%Y-%M-%dT%H:%M:%S%z"),
        }
```

--------------------------------

### Ensuring Awaitable Execution with `ensure_awaitable`

Source: https://web-poet.readthedocs.io/en/stable/_sources/page-objects/index.rst

Provides an example of how to handle both synchronous and asynchronous `to_item` methods when retrieving a page object directly. The `ensure_awaitable` utility ensures consistent handling.

```python
from web_poet.utils import ensure_awaitable

# Assume 'foo_page' is an instance of a page object
# item = await ensure_awaitable(foo_page.to_item())
```

--------------------------------

### Search ApplyRules by Attributes

Source: https://web-poet.readthedocs.io/en/stable/api-reference

Searches the registry for ApplyRules that match all provided attributes. This is useful for finding specific rules based on criteria like 'use' or 'instead_of'.

```python
rules = registry.search(use=ProductPO, instead_of=GenericPO)
print(len(rules))  # 1
print(rules[0].use)  # ProductPO
print(rules[0].instead_of)  # GenericPO
```

--------------------------------

### Run Tests with Tox and Mypy

Source: https://web-poet.readthedocs.io/en/stable/contributing

This command executes the test suite for the web-poet project using tox, which also performs type checking with mypy across different Python versions. Ensure tox is installed in your environment before running.

```shell
tox
```

--------------------------------

### Resolve rule conflicts by subclassing and adjusting priority

Source: https://web-poet.readthedocs.io/en/stable/page-objects/rules

Provides an example of how to resolve rule conflicts when dealing with external packages. By subclassing a conflicting page object class and adjusting the priority, you can ensure your rule takes precedence.

```python
from package1 import A
from web_poet import handle_urls

# Assume package1.A and package2.B are conflicting page object classes
# with default priority (500).

@handle_urls(..., priority=510) # Adjust priority to ensure precedence
class NewA(A):
    pass

```

--------------------------------

### Parse with web-poet Page Object

Source: https://web-poet.readthedocs.io/en/stable/_sources/intro/from-ground-up.rst

Converts the parsing logic into a web-poet Page Object class. This approach encapsulates the parsing logic within a dedicated class, utilizing web-poet's HttpResponse for standardized response handling and providing helper methods like 'css'.

```python
import requests
from web_poet import Injectable, HttpResponse


class BookPage(Injectable):
    def __init__(self, response: HttpResponse):
        self.response = response

    def to_item(self) -> dict:
        return {
            "url": self.response.url,
            "title": self.response.css("h1").get(),
        }


def download(url: str) -> Response:
    response = requests.get(url)
    return HttpResponse(
        url=response.url,
        body=response.content,
        headers=response.headers,
    )


url = "http://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
response = download(url)
book_page = BookPage(response=response)
item = book_page.to_item()
```
Tokens:
Raw