python-zyte-api

https://python-zyte-api.readthedocs.io/en/stable
Admin
A command-line client and Python library for interacting with Zyte API to enable web scraping and...
Tokens:17,957
Snippets:197
Trust Score:8.3
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install zyte-api with x402 support

Source: https://python-zyte-api.readthedocs.io/en/stable/intro/install.html

Install the zyte-api package with x402 support. Python 3.10+ is required.

```bash
pip install zyte-api[x402]
```

--------------------------------

### Install zyte-api

Source: https://python-zyte-api.readthedocs.io/en/stable/intro/install.html

Use this command to install the base zyte-api package. Python 3.10+ is required.

```bash
pip install zyte-api
```

--------------------------------

### Sync API - Single Request

Source: https://python-zyte-api.readthedocs.io/en/stable/_sources/use/api.rst.txt

Demonstrates how to create a ZyteAPI client and perform a single request using the `get` method.

```APIDOC
## Sync API - Single Request

### Description
Use the `get` method of the `ZyteAPI` object to perform a single request.

### Method
`ZyteAPI.get(request_dict)`

### Parameters
#### Request Body
- **request_dict** (dict) - Required - A dictionary specifying the request parameters, e.g., `{"url": "https://toscrape.com", "httpResponseBody": True}`

### Request Example
```python
from zyte_api import ZyteAPI

client = ZyteAPI()
result = client.get({"url": "https://toscrape.com", "httpResponseBody": True})
```
```

--------------------------------

### Async API - Single Request

Source: https://python-zyte-api.readthedocs.io/en/stable/_sources/use/api.rst.txt

Illustrates how to create an `AsyncZyteAPI` client and perform a single asynchronous request using the `get` method.

```APIDOC
## Async API - Single Request

### Description
Use the `get` method of the `AsyncZyteAPI` object to perform a single asynchronous request within a coroutine.

### Method
`await client.get(request_dict)`

### Parameters
#### Request Body
- **request_dict** (dict) - Required - A dictionary specifying the request parameters, e.g., `{"url": "https://toscrape.com", "httpResponseBody": True}`

### Request Example
```python
import asyncio
from zyte_api import AsyncZyteAPI

async def main():
    client = AsyncZyteAPI()
    result = await client.get({"url": "https://toscrape.com", "httpResponseBody": True})

asyncio.run(main())
```
```

--------------------------------

### Custom Retry Policy Example

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_retry.html

Demonstrates how to create a custom retry policy by subclassing RetryFactory and modifying stop conditions for network and download errors.

```python
from zyte_api import \
    RetryFactory, \
    stop_after_uninterrupted_delay, \
    stop_on_download_error


class CustomRetryFactory(RetryFactory):
    network_error_stop = stop_after_uninterrupted_delay(30 * 60)
    download_error_stop = stop_on_download_error(max_total=8, max_permanent=4)


CUSTOM_RETRY_POLICY = CustomRetryFactory().build()
```

--------------------------------

### Custom Retry Policy Example

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_retry.html

Demonstrates subclassing `AggressiveRetryFactory` to define a custom retry policy. This example modifies the stop conditions for download, network, and undocumented errors.

```python
from zyte_api import (
    AggressiveRetryFactory,
    stop_after_uninterrupted_delay,
    stop_on_count,
    stop_on_download_error,
)


class CustomRetryFactory(AggressiveRetryFactory):
    download_error_stop = stop_on_download_error(max_total=16, max_permanent=8)
    network_error_stop = stop_after_uninterrupted_delay(30 * 60)
    undocumented_error_stop = stop_on_count(8)


CUSTOM_RETRY_POLICY = CustomRetryFactory().build()
```

--------------------------------

### Async API - Single Request

Source: https://python-zyte-api.readthedocs.io/en/stable/use/api.html

Demonstrates how to perform a single request using the asynchronous API by creating an `AsyncZyteAPI` object and awaiting its `get()` method.

```APIDOC
## Async API - Single Request

### Description
Performs a single request using the asynchronous API.

### Method
`AsyncZyteAPI.get(query)`

### Parameters
#### Request Body
- **query** (dict) - Required - A dictionary containing the request parameters, such as 'url' and 'httpResponseBody'.

### Request Example
```python
import asyncio
from zyte_api import AsyncZyteAPI

async def main():
    client = AsyncZyteAPI()
    result = await client.get({"url": "https://toscrape.com", "httpResponseBody": True})

asyncio.run(main())
```
```

--------------------------------

### Input File with URLs

Source: https://python-zyte-api.readthedocs.io/en/stable/_sources/use/cli.rst.txt

This is an example of a plain-text input file where each line is a URL to be processed by the Zyte API. The API will automatically set 'browserHtml' to true for each URL.

```none
https://books.toscrape.com
https://quotes.toscrape.com
```

--------------------------------

### Input File with JSON Lines Zyte API Parameters

Source: https://python-zyte-api.readthedocs.io/en/stable/_sources/use/cli.rst.txt

This example shows a JSON Lines input file where each line is a Zyte API request object. You can specify various parameters like 'url', 'browserHtml', 'geolocation', 'httpResponseBody', and 'productNavigation'.

```json
{"url": "https://a.example", "browserHtml": true, "geolocation": "GB"}
{"url": "https://b.example", "httpResponseBody": true}
{"url": "https://books.toscrape.com", "productNavigation": true}
```

--------------------------------

### Sync API - Single Request

Source: https://python-zyte-api.readthedocs.io/en/stable/use/api.html

Demonstrates how to perform a single request using the synchronous API by creating a `ZyteAPI` object and calling its `get()` method.

```APIDOC
## Sync API - Single Request

### Description
Performs a single request using the synchronous API.

### Method
`ZyteAPI.get(query)`

### Parameters
#### Request Body
- **query** (dict) - Required - A dictionary containing the request parameters, such as 'url' and 'httpResponseBody'.

### Request Example
```python
from zyte_api import ZyteAPI

client = ZyteAPI()
result = client.get({"url": "https://toscrape.com", "httpResponseBody": True})
```
```

--------------------------------

### Perform a Single Async Request

Source: https://python-zyte-api.readthedocs.io/en/stable/use/api.html

Use the async API from coroutines. Instantiate AsyncZyteAPI and await the get() method.

```python
import asyncio

from zyte_api import AsyncZyteAPI


async def main():
    client = AsyncZyteAPI()
    result = await client.get({"url": "https://toscrape.com", "httpResponseBody": True})


asyncio.run(main())
```

--------------------------------

### Perform a Single Sync Request

Source: https://python-zyte-api.readthedocs.io/en/stable/use/api.html

Use the sync API for simple scripts. Instantiate ZyteAPI and call the get() method with your request parameters.

```python
from zyte_api import ZyteAPI

client = ZyteAPI()
result = client.get({"url": "https://toscrape.com", "httpResponseBody": True})
```

--------------------------------

### get

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_sync.html

Sends a single query to the Zyte API and returns the result. It handles retries based on the specified policy or the default one.

```APIDOC
## get

### Description
Sends a single query to the Zyte API and returns the result. It handles retries based on the specified policy or the default one.

### Method
Synchronous method call.

### Parameters
- **query** (dict) - Required - The query to send to the Zyte API.
- **endpoint** (str) - Optional - The API endpoint to use. Defaults to "extract".
- **session** (aiohttp.ClientSession | None) - Optional - An existing aiohttp client session to use.
- **handle_retries** (bool) - Optional - Whether to handle retries. Defaults to True.
- **retrying** (AsyncRetrying | None) - Optional - A custom retry strategy.

### Response
- **dict** - The result of the Zyte API query.
```

--------------------------------

### Customizing Default Retry Policy

Source: https://python-zyte-api.readthedocs.io/en/stable/ref/api.html

Create a custom retry policy by subclassing RetryFactory and modifying retry parameters. This example doubles attempts for download errors and retry time for network errors.

```python
from zyte_api import (
    RetryFactory,
    stop_after_uninterrupted_delay,
    stop_on_download_error,
)


class CustomRetryFactory(RetryFactory):
    network_error_stop = stop_after_uninterrupted_delay(30 * 60)
    download_error_stop = stop_on_download_error(max_total=8, max_permanent=4)


CUSTOM_RETRY_POLICY = CustomRetryFactory().build()
```

--------------------------------

### Zyte API Python Sync API Usage

Source: https://python-zyte-api.readthedocs.io/en/stable/intro/basic.html

Use the synchronous Python API for basic scripts. Instantiate the ZyteAPI client with your API key and make a GET request.

```python
from zyte_api import ZyteAPI

client = ZyteAPI(api_key="YOUR_API_KEY")
response = client.get({"url": "https://toscrape.com", "httpResponseBody": True})

```

--------------------------------

### Send a Query with ZyteAPI.get()

Source: https://python-zyte-api.readthedocs.io/en/stable/ref/api.html

Use the get() method to send a single query to a specified Zyte API endpoint and retrieve the result. The endpoint defaults to 'extract'. Retry handling can be enabled or disabled.

```python
result = client.get(
    {
        "url": "http://example.com",
        "httpResponseBody": True
    }
)
```

--------------------------------

### Zyte API Python Async API Usage

Source: https://python-zyte-api.readthedocs.io/en/stable/intro/basic.html

Utilize the asynchronous Python API within an asyncio event loop. Initialize AsyncZyteAPI and perform an asynchronous GET request.

```python
import asyncio

from zyte_api import AsyncZyteAPI


async def main():
    client = AsyncZyteAPI(api_key="YOUR_API_KEY")
    response = await client.get(
        {"url": "https://toscrape.com", "httpResponseBody": True}
    )


asyncio.run(main())

```

--------------------------------

### Synchronous ZyteAPI Client Get Request

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_sync.html

Sends a single query to the specified Zyte API endpoint and returns the result. You can optionally provide a network session and control retry handling.

```python
from zyte_api import ZyteAPI

client = ZyteAPI()
result = client.get(
    query={"url": "http://example.com"},
    endpoint="extract",
    handle_retries=True,
)
```

--------------------------------

### Synchronous Get Request with Zyte API

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_sync.html

Sends a single query to the Zyte API and returns the result. Handles retries based on the provided policy or defaults. Use this for single, straightforward API calls.

```python
loop = _get_loop()
future = self._async_client.get(
    query=query,
    endpoint=endpoint,
    session=session,
    handle_retries=handle_retries,
    retrying=retrying,
)
return loop.run_until_complete(future)
```

--------------------------------

### Asynchronous GET Request with Zyte API

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_async.html

Use this method to make a single asynchronous request to a Zyte API endpoint. It handles authentication, retries, and session management automatically. Ensure you have aiohttp and the Zyte API library installed.

```python
async def get(
        self,
        query: dict[str, Any],
        *,
        endpoint: str = "extract",
        session: aiohttp.ClientSession | None = None,
        handle_retries: bool = True,
        retrying: AsyncRetrying | None = None,
    ) -> dict[str, Any]:
        """Asynchronous equivalent to :meth:`ZyteAPI.get`."""
        retrying = retrying or self.retrying
        owned_session: aiohttp.ClientSession | None = None
        if session is None:
            owned_session = create_session(self.n_conn, trust_env=self.trust_env)
            session = owned_session
        post = _post_func(session)

        url = self.api_url + endpoint
        query = _process_query(query)
        headers = {"User-Agent": self.user_agent, "Accept-Encoding": "br"}

        auth_kwargs = {}
        if isinstance(self._auth, str):
            auth_kwargs["auth"] = aiohttp.BasicAuth(self._auth)
        else:
            x402_headers = await self._auth.get_headers(url, query, headers, post)
            headers.update(x402_headers)

        post_kwargs = {
            "url": url,
            "json": query,
            "headers": headers,
            **auth_kwargs,
        }

        response_stats = []
        start_global = time.perf_counter()

        async def request() -> dict[str, Any]:
            stats = ResponseStats.create(start_global)
            self.agg_stats.n_attempts += 1

            try:
                async with self._semaphore, post(**post_kwargs) as resp:
                    stats.record_connected(resp.status, self.agg_stats)
                    if (
                        resp.status == 402
                        and isinstance(self._auth, _x402Handler)
                        and "X-Payment" in post_kwargs["headers"]
                    ):
                        self._auth.refresh_post_kwargs(post_kwargs, await resp.json())
                    if resp.status >= 400:
                        content = await resp.read()
                        resp.release()
                        stats.record_read()
                        stats.record_request_error(content, self.agg_stats)

                        raise RequestError(
                            request_info=resp.request_info,
                            history=resp.history,
                            status=resp.status,
                            message=resp.reason,
                            headers=resp.headers,
                            response_content=content,
                            query=query,
                        )

                    response = cast("dict[str, Any]", await resp.json())
                    stats.record_read(self.agg_stats)
                    return response
            except Exception as e:
                if not isinstance(e, RequestError):
                    self.agg_stats.n_errors += 1
                    stats.record_exception(e, agg_stats=self.agg_stats)
                raise
            finally:
                response_stats.append(stats)

        if handle_retries:
            request = retrying.wraps(request)

        try:
            try:
                # Try to make a request
                result = await request()
                self.agg_stats.n_success += 1
            except Exception:
                self.agg_stats.n_fatal_errors += 1
                raise

            return result
        finally:
            if owned_session is not None:
                await owned_session.close()

```

--------------------------------

### Initialize AsyncZyteAPI with Environment Variables

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_async.html

Initialize the asynchronous Zyte API client by automatically detecting the API key or Ethereum key from environment variables ('ZYTE_API_KEY' or 'ZYTE_API_ETH_KEY').

```python
from zyte_api import AsyncZyteAPI

# API key from ZYTE_API_KEY environment variable
client_api_env = AsyncZyteAPI()

# Ethereum key from ZYTE_API_ETH_KEY environment variable
# Requires zyte-api[x402] to be installed
client_eth_env = AsyncZyteAPI()
```

--------------------------------

### Client Initialization with Connection Count

Source: https://python-zyte-api.readthedocs.io/en/stable/_sources/use/api.rst.txt

Explains how to configure the number of concurrent connections when initializing `ZyteAPI` or `AsyncZyteAPI` clients using the `n_conn` parameter.

```APIDOC
## Client Initialization with Connection Count

### Description
Customize the number of concurrent connections used by the client by specifying the `n_conn` parameter during initialization. This setting applies globally to the client instance.

### Method
`ZyteAPI(n_conn=value)` or `AsyncZyteAPI(n_conn=value)`

### Parameters
#### Initialization Parameters
- **n_conn** (int) - Optional - The number of concurrent connections to use. Defaults to 15.

### Request Example
```python
# For synchronous client
client = ZyteAPI(n_conn=30)

# For asynchronous client
client = AsyncZyteAPI(n_conn=30)
```
```

--------------------------------

### Sync API - Multiple Requests

Source: https://python-zyte-api.readthedocs.io/en/stable/_sources/use/api.rst.txt

Shows how to use a `session` and the `iter` method for performing multiple requests efficiently in parallel.

```APIDOC
## Sync API - Multiple Requests

### Description
For multiple requests, use a `session` with the `ZyteAPI` object and the `iter` method to send requests in parallel. Results are yielded as they become available.

### Method
`session.iter(queries)`

### Parameters
#### Request Body
- **queries** (list of dict) - Required - A list of dictionaries, where each dictionary specifies a request.

### Request Example
```python
from zyte_api import ZyteAPI, RequestError

client = ZyteAPI()
with client.session() as session:
    queries = [
        {"url": "https://toscrape.com", "httpResponseBody": True},
        {"url": "https://books.toscrape.com", "httpResponseBody": True},
    ]
    for result_or_exception in session.iter(queries):
        if isinstance(result_or_exception, dict):
            # Process successful result
            ...
        elif isinstance(result_or_exception, RequestError):
            # Handle request error
            ...
        else:
            # Handle other exceptions
            assert isinstance(result_or_exception, Exception)
            ...
```
```

--------------------------------

### Set Ethereum Private Key (Windows CMD)

Source: https://python-zyte-api.readthedocs.io/en/stable/_sources/use/x402.rst.txt

Configure your Ethereum private key using the ZYTE_API_ETH_KEY environment variable on Windows CMD.

```shell
> set ZYTE_API_ETH_KEY=YOUR_ETH_PRIVATE_KEY
```

--------------------------------

### Sync API - Multiple Requests with Session

Source: https://python-zyte-api.readthedocs.io/en/stable/use/api.html

Shows how to perform multiple requests efficiently using a session and the `iter()` method for parallel processing.

```APIDOC
## Sync API - Multiple Requests with Session

### Description
Performs multiple requests in parallel using a session for better performance.

### Method
`ZyteAPI.session().iter(queries)`

### Parameters
#### Request Body
- **queries** (list of dict) - Required - A list of query dictionaries, where each dictionary represents a request.

### Request Example
```python
from zyte_api import ZyteAPI, RequestError

client = ZyteAPI()
with client.session() as session:
    queries = [
        {"url": "https://toscrape.com", "httpResponseBody": True},
        {"url": "https://books.toscrape.com", "httpResponseBody": True},
    ]
    for result_or_exception in session.iter(queries):
        if isinstance(result_or_exception, dict):
            # Process successful result
            pass
        elif isinstance(result_or_exception, RequestError):
            # Handle request error
            pass
        else:
            # Handle other exceptions
            assert isinstance(result_or_exception, Exception)
            pass
```

### Tip
`iter()` yields results as they come, not necessarily in their original order. Use echoData to track the source request.
```

--------------------------------

### List of URLs for Zyte API CLI

Source: https://python-zyte-api.readthedocs.io/en/stable/intro/basic.html

Create a text file containing a list of URLs to be processed by the zyte-api command-line client.

```text
https://books.toscrape.com
https://quotes.toscrape.com

```

--------------------------------

### AsyncZyteAPI Initialization with Custom Parameters

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_async.html

Initialize the asynchronous Zyte API client with custom connection count, retry strategy, and user agent. The 'trust_env' parameter controls whether to trust environment variables for proxy settings.

```python
from zyte_api import AsyncZyteAPI
from tenacity import AsyncRetrying

custom_retrying = AsyncRetrying(stop=None, wait=None, retry=None, reraise=True)

client = AsyncZyteAPI(
    api_key="YOUR_API_KEY",
    n_conn=10,
    retrying=custom_retrying,
    user_agent="MyCustomUserAgent/1.0",
    trust_env=True,
)
```

--------------------------------

### Synchronous ZyteAPI Client Initialization with Custom Settings

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_sync.html

Initializes the synchronous ZyteAPI client with custom settings for API URL, connection count, retry policy, user agent, and environment variable trust.

```python
from zyte_api import ZyteAPI
from tenacity import retry

@retry
def my_retry_policy():
    pass

client = ZyteAPI(
    api_url="https://some.custom.api.url/v1/",
    n_conn=10,
    retrying=my_retry_policy,
    user_agent="MyCustomUserAgent/1.0",
    trust_env=True,
)
```

--------------------------------

### Async Session Management

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_async.html

Demonstrates how to create and manage an asynchronous session with the Zyte API. Ensure to close the session when done.

```python
session = client.session()
try:
    ...
finally:
    await session.close()
```

--------------------------------

### Run Zyte API from Command Line

Source: https://python-zyte-api.readthedocs.io/en/stable/intro/basic.html

Execute the zyte-api command-line tool to process a list of URLs and save results to a JSON Lines file. Ensure you replace YOUR_API_KEY with your actual API key.

```bash
zyte-api url-list.txt --api-key YOUR_API_KEY --output results.jsonl

```

--------------------------------

### Synchronous ZyteAPI Client Initialization with Ethereum Key

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_sync.html

Initializes the synchronous ZyteAPI client using an Ethereum private key for payments. The API URL defaults to a specific endpoint for Ethereum-based payments.

```python
from zyte_api import ZyteAPI

client = ZyteAPI(eth_key="YOUR_ETH_KEY")
```

--------------------------------

### Basic Usage of zyte-api CLI

Source: https://python-zyte-api.readthedocs.io/en/stable/_sources/use/cli.rst.txt

Use this command to process a list of URLs from a file and save the results to a JSON Lines file. Ensure you have an input file (e.g., urls.txt) and specify the output file path.

```shell
zyte-api urls.txt --output result.jsonl
```

--------------------------------

### Async API - Multiple Requests with Session

Source: https://python-zyte-api.readthedocs.io/en/stable/use/api.html

Shows how to perform multiple asynchronous requests efficiently using a session and the `iter()` method for parallel processing.

```APIDOC
## Async API - Multiple Requests with Session

### Description
Performs multiple asynchronous requests in parallel using a session for better performance.

### Method
`AsyncZyteAPI.session().iter(queries)`

### Parameters
#### Request Body
- **queries** (list of dict) - Required - A list of query dictionaries, where each dictionary represents a request.

### Request Example
```python
import asyncio
from zyte_api import AsyncZyteAPI, RequestError

async def main():
    client = AsyncZyteAPI()
    async with client.session() as session:
        queries = [
            {"url": "https://toscrape.com", "httpResponseBody": True},
            {"url": "https://books.toscrape.com", "httpResponseBody": True},
        ]
        for future in session.iter(queries):
            try:
                result = await future
            except RequestError as e:
                # Handle request error
                pass
            except Exception as e:
                # Handle other exceptions
                pass

asyncio.run(main())
```

### Tip
`iter()` yields results as they come, not necessarily in their original order. Use echoData to track the source request.
```

--------------------------------

### Optimization - Concurrent Connections

Source: https://python-zyte-api.readthedocs.io/en/stable/use/api.html

Explains how to adjust the number of concurrent connections for `ZyteAPI` and `AsyncZyteAPI` to optimize performance.

```APIDOC
## Optimization - Concurrent Connections

### Description
Allows adjustment of the number of concurrent connections for performance optimization.

### Method
`ZyteAPI(n_conn=...)` or `AsyncZyteAPI(n_conn=...)

### Parameters
#### Initialization Parameters
- **n_conn** (int) - Optional - The number of concurrent connections to use. Defaults to 15.

### Request Example
```python
# For synchronous client
client = ZyteAPI(n_conn=30)

# For asynchronous client
# client = AsyncZyteAPI(n_conn=30)
```

### Note
The number of concurrent connections is enforced across all method calls, including different sessions of the same client.
```

--------------------------------

### Async API - Multiple Requests

Source: https://python-zyte-api.readthedocs.io/en/stable/_sources/use/api.rst.txt

Demonstrates using an `AsyncZyteAPI` session and the `iter` method for efficient parallel asynchronous requests.

```APIDOC
## Async API - Multiple Requests

### Description
For multiple asynchronous requests, use an `AsyncZyteAPI` session with the `iter` method to send requests in parallel. Each iteration yields a future that can be awaited.

### Method
`session.iter(queries)`

### Parameters
#### Request Body
- **queries** (list of dict) - Required - A list of dictionaries, where each dictionary specifies a request.

### Request Example
```python
import asyncio
from zyte_api import AsyncZyteAPI, RequestError

async def main():
    client = AsyncZyteAPI()
    async with client.session() as session:
        queries = [
            {"url": "https://toscrape.com", "httpResponseBody": True},
            {"url": "https://books.toscrape.com", "httpResponseBody": True},
        ]
        for future in session.iter(queries):
            try:
                result = await future
            except RequestError as e:
                # Handle request error
                ...
            except Exception as e:
                # Handle other exceptions
                ...

asyncio.run(main())
```
```

--------------------------------

### Instantiate Synchronous ZyteAPI with API Key

Source: https://python-zyte-api.readthedocs.io/en/stable/_sources/use/key.rst.txt

Pass your API key directly to the ZyteAPI constructor when initializing the synchronous Python client.

```python
from zyte_api import ZyteAPI

client = ZyteAPI(api_key="YOUR_API_KEY")
```

--------------------------------

### Zyte API CLI Usage

Source: https://python-zyte-api.readthedocs.io/en/stable/ref/cli.html

This snippet shows the general usage and available arguments for the zyte-api command-line tool. It outlines options for input/output types, connection limits, API keys, logging levels, and error handling.

```bash
usage: zyte-api [-h] [--intype {txt,jl}] [--limit LIMIT] [--output OUTPUT]
                [--n-conn N_CONN] [--api-key API_KEY | --eth-key ETH_KEY]
                [--api-url API_URL] [--loglevel {DEBUG,INFO,WARNING,ERROR}]
                [--shuffle] [--dont-retry-errors] [--store-errors]
                [--trust-env]
                INPUT
```

--------------------------------

### Initialize AsyncZyteAPI with Ethereum Private Key

Source: https://python-zyte-api.readthedocs.io/en/stable/_sources/use/x402.rst.txt

Pass your Ethereum private key directly to the Python client classes using the eth_key parameter when creating an AsyncZyteAPI object.

```python
from zyte_api import AsyncZyteAPI

client = AsyncZyteAPI(eth_key="YOUR_ETH_PRIVATE_KEY")
```

--------------------------------

### Export Ethereum Private Key (macOS/Linux)

Source: https://python-zyte-api.readthedocs.io/en/stable/_sources/use/x402.rst.txt

Configure your Ethereum private key using the ZYTE_API_ETH_KEY environment variable on macOS and Linux.

```shell
$ export ZYTE_API_ETH_KEY=YOUR_ETH_PRIVATE_KEY
```

--------------------------------

### Pass API Key to Command-Line Client

Source: https://python-zyte-api.readthedocs.io/en/stable/_sources/use/key.rst.txt

Use the --api-key switch to provide your API key directly when using the zyte-api command-line tool.

```shell
zyte-api --api-key YOUR_API_KEY …
```

--------------------------------

### Initialize ZyteAPI with Ethereum Private Key

Source: https://python-zyte-api.readthedocs.io/en/stable/_sources/use/x402.rst.txt

Pass your Ethereum private key directly to the Python client classes using the eth_key parameter when creating a ZyteAPI object.

```python
from zyte_api import ZyteAPI

client = ZyteAPI(eth_key="YOUR_ETH_PRIVATE_KEY")
```

--------------------------------

### Instantiate Asynchronous ZyteAPI with API Key

Source: https://python-zyte-api.readthedocs.io/en/stable/_sources/use/key.rst.txt

Pass your API key directly to the AsyncZyteAPI constructor when initializing the asynchronous Python client.

```python
from zyte_api import AsyncZyteAPI

client = AsyncZyteAPI(api_key="YOUR_API_KEY")
```

--------------------------------

### ZyteAPI.session()

Source: https://python-zyte-api.readthedocs.io/en/stable/ref/api.html

Provides a context manager to create a reusable network session for making Zyte API requests, improving performance by maintaining a pool of connections.

```APIDOC
## ZyteAPI.session()

### Description
Context manager to create a session with an `aiohttp.ClientSession` object for making Zyte API requests. Using a session improves performance by reusing connections.

### Method
`session(_**kwargs: Any_) -> _Session[source]`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
* **_kwargs** (Any) - Optional - Additional parameters to pass to `aiohttp.ClientSession`.

### Request Example
```python
# Example usage
client_session = client.session()
try:
    # Use client_session.get() or client_session.iter() here
    result = client_session.get({'url': 'http://example.com'})
finally:
    client_session.close()
```

### Response
#### Success Response (200)
* **_Session[source]** - A session object with the same API as the client, but without the `session()` method and with `get()`/`iter()` automatically using the session's `aiohttp.ClientSession`.
```

--------------------------------

### Set API Key Environment Variable (Windows)

Source: https://python-zyte-api.readthedocs.io/en/stable/_sources/use/key.rst.txt

Configure your API key using the 'set' command on Windows. This allows the Zyte API client to pick up the key automatically.

```shell
> set ZYTE_API_KEY=YOUR_API_API_KEY
```

--------------------------------

### session

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_sync.html

Creates a context manager for a session, which reuses an aiohttp.ClientSession for improved performance. It has a similar API to the client but without the session parameter in get/iter.

```APIDOC
## session

### Description
Creates a context manager for a session, which reuses an aiohttp.ClientSession for improved performance. It has a similar API to the client but without the session parameter in get/iter. You can pass kwargs to customize the aiohttp.ClientSession.

### Method
Synchronous method call.

### Parameters
- **kwargs** (Any) - Additional keyword arguments to pass to aiohttp.ClientSession.

### Response
- **_Session** - A session object with a similar API to the client, optimized for reusing connections.
```

--------------------------------

### Configure Concurrent Connections

Source: https://python-zyte-api.readthedocs.io/en/stable/use/api.html

Optimize performance by setting the number of concurrent connections using the n_conn parameter during client instantiation. This setting applies globally to the client.

```python
client = ZyteAPI(n_conn=30)
```

--------------------------------

### Manage ZyteAPI Session with Context Manager

Source: https://python-zyte-api.readthedocs.io/en/stable/ref/api.html

Use the session() method as a context manager to create a network session for making Zyte API requests. This reuses aiohttp.ClientSession for improved performance. Additional aiohttp.ClientSession parameters can be passed via kwargs.

```python
async with client.session() as session:
    result = await session.get(
        {
            "url": "http://example.com",
            "httpResponseBody": True
        }
    )
```

--------------------------------

### Perform Multiple Sync Requests with Session

Source: https://python-zyte-api.readthedocs.io/en/stable/use/api.html

For better performance with multiple requests, use a session and the iter() method. This allows sending requests in parallel. Handle potential RequestErrors.

```python
from zyte_api import ZyteAPI, RequestError

client = ZyteAPI()
with client.session() as session:
    queries = [
        {"url": "https://toscrape.com", "httpResponseBody": True},
        {"url": "https://books.toscrape.com", "httpResponseBody": True},
    ]
    for result_or_exception in session.iter(queries):
        if isinstance(result_or_exception, dict):
            ...
        elif isinstance(result_or_exception, RequestError):
            ...
        else:
            assert isinstance(result_or_exception, Exception)
            ...
```

--------------------------------

### Run Tests with Tox

Source: https://python-zyte-api.readthedocs.io/en/stable/contributing.html

Execute tests for the project across different Python versions using tox. This command also initiates type checks with mypy.

```bash
tox

```

--------------------------------

### Pass Ethereum Private Key to Zyte API CLI

Source: https://python-zyte-api.readthedocs.io/en/stable/_sources/use/x402.rst.txt

Use the --eth-key switch to pass your Ethereum private key directly to the command-line client.

```shell
zyte-api --eth-key YOUR_ETH_PRIVATE_KEY …
```

--------------------------------

### Run Tests with Tox

Source: https://python-zyte-api.readthedocs.io/en/stable/_sources/contributing.rst.txt

Execute tests and type checks using the tox command. This is the standard way to ensure code quality and compatibility across different Python versions.

```bash
tox
```

--------------------------------

### AsyncZyteAPI.session

Source: https://python-zyte-api.readthedocs.io/en/stable/ref/api.html

Asynchronous equivalent to `ZyteAPI.session()`. Manages asynchronous client sessions.

```APIDOC
## AsyncZyteAPI.session

```python
session(_** kwargs: Any_) → _AsyncSession[source]
```

Asynchronous equivalent to `ZyteAPI.session()`. You do not need to use `session()` as an async context manager as long as you await `close()` on the object it returns when you are done:
```
session = client.session()
try:
    ...
finally:
    await session.close()

```
```

--------------------------------

### Set API Key Environment Variable (macOS/Linux)

Source: https://python-zyte-api.readthedocs.io/en/stable/_sources/use/key.rst.txt

Configure your API key using the 'export' command on macOS and Linux. This allows the Zyte API client to pick up the key automatically.

```shell
$ export ZYTE_API_KEY=YOUR_API_KEY
```

--------------------------------

### Using Aggressive Retrying with ZyteAPI

Source: https://python-zyte-api.readthedocs.io/en/stable/use/api.html

Instantiate ZyteAPI with the aggressive_retrying policy to increase the number of retry attempts for various error scenarios.

```python
from zyte_api import ZyteAPI, aggressive_retrying

client = ZyteAPI(retrying=aggressive_retrying)
```

--------------------------------

### Synchronous ZyteAPI Client Session Context Manager

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_sync.html

Uses the synchronous ZyteAPI client's session as a context manager. This ensures that the network session is properly opened and closed, managing resources efficiently.

```python
from zyte_api import ZyteAPI

client = ZyteAPI()
with client.session() as session:
    result = session.get(query={"url": "http://example.com"})
    print(result)
```

--------------------------------

### AsyncZyteAPI.get

Source: https://python-zyte-api.readthedocs.io/en/stable/ref/api.html

Asynchronous equivalent to `ZyteAPI.get()`. Makes a single asynchronous request to a specified endpoint.

```APIDOC
## AsyncZyteAPI.get

```python
async get(_query : dict[str, Any]_, _*_ , _endpoint : str = 'extract'_, _session : ClientSession | None = None_, _handle_retries : bool = True_, _retrying : AsyncRetrying | None = None_) → dict[str, Any]
```

Asynchronous equivalent to `ZyteAPI.get()`. This method sends a query to the specified endpoint and returns the result.
```

--------------------------------

### Zyte API Session Management with Context Manager

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_sync.html

Creates a session for making multiple Zyte API requests using the same underlying aiohttp.ClientSession for improved performance. Can be used as a context manager or by manually calling close().

```python
session = client.session()
try:
    ...
finally:
    session.close()
```

--------------------------------

### Randomize Request Order

Source: https://python-zyte-api.readthedocs.io/en/stable/_sources/use/cli.rst.txt

Use the --shuffle option to randomize the order of requests, which can help distribute load more evenly when targeting multiple websites.

```shell
zyte-api urls.txt --shuffle …
```

--------------------------------

### Build AsyncRetrying Instance

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_retry.html

Constructs an `AsyncRetrying` instance with custom wait, retry, stop, and logging configurations. This is used internally by retry factories.

```python
def build(self) -> AsyncRetrying:
    return AsyncRetrying(
        wait=self.wait,
        retry=self.retry_condition,
        stop=self.stop,
        reraise=self.reraise(),
        before=before_log(logger, logging.DEBUG),
        after=after_log(logger, logging.DEBUG),
        before_sleep=before_sleep_log(logger, logging.DEBUG),
    )
```

--------------------------------

### Increase Concurrent Connections

Source: https://python-zyte-api.readthedocs.io/en/stable/_sources/use/cli.rst.txt

Adjust the number of concurrent connections for faster processing by using the --n-conn flag. The default is 20 connections.

```shell
zyte-api --n-conn 40 …
```

--------------------------------

### Synchronous ZyteAPI Client Session Iteration

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_sync.html

Iterates over multiple queries using a pre-established synchronous ZyteAPI client session. This allows for efficient batch processing of requests within a managed session.

```python
from zyte_api import ZyteAPI

client = ZyteAPI()
with client.session() as session:
    results = session.iter(
        queries=[
            {"url": "http://example.com/page1"},
            {"url": "http://example.com/page2"},
        ]
    )
    for result in results:
        print(result)
```

--------------------------------

### iter

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_sync.html

Sends multiple queries to the Zyte API in parallel and iterates over their results as they become available. Exceptions are yielded instead of raised.

```APIDOC
## iter

### Description
Sends multiple queries to the Zyte API in parallel and iterates over their results as they become available. Exceptions are yielded instead of raised. Results may not be in the original order.

### Method
Synchronous method call.

### Parameters
- **queries** (list[dict[str, Any]]) - Required - A list of queries to send to the Zyte API.
- **endpoint** (str) - Optional - The API endpoint to use. Defaults to "extract".
- **session** (aiohttp.ClientSession | None) - Optional - An existing aiohttp client session to use.
- **handle_retries** (bool) - Optional - Whether to handle retries. Defaults to True.
- **retrying** (AsyncRetrying | None) - Optional - A custom retry strategy.

### Response
- **Generator[dict[str, Any] | Exception, None, None]** - An iterator yielding results or exceptions as they are processed.
```

--------------------------------

### Iterate Over Multiple Queries with ZyteAPI.iter()

Source: https://python-zyte-api.readthedocs.io/en/stable/ref/api.html

The iter() method allows sending multiple queries in parallel and processing their results as they become available. Exceptions are yielded rather than raised. The number of concurrent requests is controlled by the client's n_conn setting.

```python
for result in client.iter(
    [
        {"url": "http://example.com/page1"},
        {"url": "http://example.com/page2"}
    ]
):
    # Process result
```

--------------------------------

### ZyteAPI.get

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_sync.html

Sends a query to the Zyte API and returns the result. Allows specifying the endpoint, network session, and retry behavior.

```APIDOC
## ZyteAPI.get

### Description
Sends a query to the Zyte API and returns the result. Allows specifying the endpoint, network session, and retry behavior.

### Method
GET

### Endpoint
/{endpoint}

### Parameters
#### Query Parameters
- **query** (dict) - Required - The query to send to the Zyte API.
- **endpoint** (str) - Optional - The Zyte API endpoint path relative to the client object api_url. Defaults to "extract".
- **session** (aiohttp.ClientSession) - Optional - The network session to use. Consider using :meth:`session` instead of this parameter.
- **handle_retries** (bool) - Optional - Determines whether or not a retry policy should be used. Defaults to True.
- **retrying** (tenacity.AsyncRetrying) - Optional - The retry policy to use, provided as a :class:`~tenacity.AsyncRetrying` instance.
```

--------------------------------

### AsyncZyteAPI.iter

Source: https://python-zyte-api.readthedocs.io/en/stable/ref/api.html

Asynchronous equivalent to `ZyteAPI.iter()`. Allows iterating over multiple asynchronous requests.

```APIDOC
## AsyncZyteAPI.iter

```python
iter(_queries : list[dict[str, Any]]_, _*_ , _endpoint : str = 'extract'_, _session : aiohttp.ClientSession | None = None_, _handle_retries : bool = True_, _retrying : AsyncRetrying | None = None_) → Iterator[_ResponseFuture]
```

Asynchronous equivalent to `ZyteAPI.iter()`. Yielded futures, when awaited, do raise their exceptions, instead of only returning them.
```

--------------------------------

### Perform Multiple Async Requests with Session

Source: https://python-zyte-api.readthedocs.io/en/stable/use/api.html

For parallel async requests, use a session with AsyncZyteAPI and its iter() method. Await each future and handle potential RequestErrors.

```python
import asyncio

from zyte_api import AsyncZyteAPI, RequestError


async def main():
    client = AsyncZyteAPI()
    async with client.session() as session:
        queries = [
            {"url": "https://toscrape.com", "httpResponseBody": True},
            {"url": "https://books.toscrape.com", "httpResponseBody": True},
        ]
        for future in session.iter(queries):
            try:
                result = await future
            except RequestError as e:
                ...
            except Exception as e:
                ...


asyncio.run(main())
```

--------------------------------

### ZyteAPI.get()

Source: https://python-zyte-api.readthedocs.io/en/stable/ref/api.html

Sends a single query to the Zyte API and returns the result. It allows specifying the endpoint, network session, retry handling, and retry policy.

```APIDOC
## ZyteAPI.get()

### Description
Sends a query to Zyte API and returns the result. Allows specifying the endpoint, network session, and retry behavior.

### Method
`get(_query: dict[str, Any], _*, _endpoint: str = 'extract'_, _session: ClientSession | None = None_, _handle_retries: bool = True_, _retrying: AsyncRetrying | None = None_) -> dict[str, Any]`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
* **_query** (dict[str, Any]) - Required - The query to send to Zyte API.
* **_endpoint** (str) - Optional - The Zyte API endpoint path relative to the client object `api_url`. Defaults to 'extract'.
* **_session** (ClientSession | None) - Optional - The network session to use. Consider using `session()` instead of this parameter.
* **_handle_retries** (bool) - Optional - Determines whether or not a retry policy should be used. Defaults to True.
* **_retrying** (AsyncRetrying | None) - Optional - The retry policy to use, provided `handle_retries` is `True`. If not specified, the default retry policy is used.

### Request Example
```python
# Example usage (assuming client is an instance of ZyteAPI)
result = client.get({'url': 'http://example.com'})
```

### Response
#### Success Response (200)
* **dict[str, Any]** - The result from the Zyte API.
```

--------------------------------

### Synchronous Iteration of Multiple Queries with Zyte API

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_sync.html

Sends multiple queries to the Zyte API in parallel and iterates over results as they become available. Exceptions are yielded rather than raised. Useful for processing a list of requests efficiently.

```python
loop = _get_loop()
for future in self._async_client.iter(
    queries=queries,
    endpoint=endpoint,
    session=session,
    handle_retries=handle_retries,
    retrying=retrying,
):
    try:
        yield loop.run_until_complete(future)
    except Exception as exception:
        yield exception
```

--------------------------------

### AsyncZyteAPI

Source: https://python-zyte-api.readthedocs.io/en/stable/ref/api.html

The asynchronous Zyte API client. It provides methods for making asynchronous requests to the Zyte API.

```APIDOC
## AsyncZyteAPI

```python
class AsyncZyteAPI(_*_ , _api_key : str | None = None_, _api_url : str | None = None_, _n_conn : int = 15_, _retrying : AsyncRetrying | None = None_, _user_agent : str | None = None_, _eth_key : str | None = None_, _trust_env : bool = False_)
```

Asynchronous Zyte API client. Parameters work the same as for `ZyteAPI`.
```

--------------------------------

### ZyteAPI.get

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_async.html

Asynchronously fetches data from a specified Zyte API endpoint. This method handles request creation, authentication, and response processing, including automatic retries if enabled.

```APIDOC
## async def get(query: dict[str, Any], *, endpoint: str = "extract", session: aiohttp.ClientSession | None = None, handle_retries: bool = True, retrying: AsyncRetrying | None = None) -> dict[str, Any]

### Description
Asynchronously makes a GET request to the Zyte API.

### Method
POST

### Endpoint
{api_url}/{endpoint}

### Parameters
#### Query Parameters
- **query** (dict[str, Any]) - Required - The parameters for the API request.
- **endpoint** (str) - Optional - The API endpoint to use. Defaults to "extract".
- **session** (aiohttp.ClientSession | None) - Optional - An existing aiohttp client session to use.
- **handle_retries** (bool) - Optional - Whether to automatically handle retries. Defaults to True.
- **retrying** (AsyncRetrying | None) - Optional - An instance of AsyncRetrying for custom retry logic.

### Request Example
```json
{
  "url": "https://api-x402.zyte.com/v1/extract",
  "json": {
    "url": "http://example.com"
  },
  "headers": {
    "User-Agent": "YourUserAgent",
    "Accept-Encoding": "br"
  }
}
```

### Response
#### Success Response (200)
- **dict[str, Any]** - The JSON response from the API.

#### Response Example
```json
{
  "status": "ok",
  "data": [
    {
      "url": "http://example.com",
      "content": "<html>...</html>"
    }
  ]
}
```
```

--------------------------------

### Custom Stop Condition: Stop on Download Errors

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_retry.html

A custom stop condition for tenacity that stops retrying based on the maximum number of total or permanent download errors encountered.

```python
class stop_on_download_error(stop_base):
    """Stop after the specified max numbers of total or permanent download
    errors."""

    def __init__(self, max_total: int, max_permanent: int) -> None:
        self._max_total = max_total
        self._max_permanent = max_permanent

    def __call__(self, retry_state: RetryCallState) -> bool:
        if not hasattr(retry_state, "counter"):
            retry_state.counter = Counter()  # type: ignore[attr-defined]

```

--------------------------------

### Asynchronous Iteration over Queries with Zyte API

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_async.html

Use this method to asynchronously process a list of queries. It returns an iterator of futures, allowing for efficient handling of multiple API requests. Exceptions raised by individual requests are propagated when the futures are awaited.

```python
def iter(
        self,
        queries: list[dict[str, Any]],
        *,
        endpoint: str = "extract",
        session: aiohttp.ClientSession | None = None,
        handle_retries: bool = True,
        retrying: AsyncRetrying | None = None,
    ) -> Iterator[_ResponseFuture]:
        """Asynchronous equivalent to :meth:`ZyteAPI.iter`.

        .. note:: Yielded futures, when awaited, do raise their exceptions,
                  instead of only returning them.
        """

        def _request(query: dict[str, Any]) -> _ResponseFuture:
            return self.get(
                query,
                endpoint=endpoint,
                session=session,
                handle_retries=handle_retries,
                retrying=retrying,
            )


```

--------------------------------

### ZyteAPI.iter

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_async.html

Asynchronously iterates over multiple API requests, yielding futures for each response. This is useful for processing a list of queries concurrently.

```APIDOC
## def iter(queries: list[dict[str, Any]], *, endpoint: str = "extract", session: aiohttp.ClientSession | None = None, handle_retries: bool = True, retrying: AsyncRetrying | None = None) -> Iterator[_ResponseFuture]

### Description
Asynchronously iterates over a list of queries, returning an iterator of futures.

### Method
POST (for each query)

### Endpoint
{api_url}/{endpoint}

### Parameters
#### Query Parameters
- **queries** (list[dict[str, Any]]) - Required - A list of query dictionaries to process.
- **endpoint** (str) - Optional - The API endpoint to use. Defaults to "extract".
- **session** (aiohttp.ClientSession | None) - Optional - An existing aiohttp client session to use.
- **handle_retries** (bool) - Optional - Whether to automatically handle retries for each request. Defaults to True.
- **retrying** (AsyncRetrying | None) - Optional - An instance of AsyncRetrying for custom retry logic.

### Response
#### Success Response (Iterator of Futures)
- **Iterator[_ResponseFuture]** - An iterator yielding futures, where each future resolves to the API response for a query.

### Note
Yielded futures, when awaited, will raise their exceptions directly, rather than returning them.
```

--------------------------------

### RetryFactory

Source: https://python-zyte-api.readthedocs.io/en/stable/ref/api.html

Factory class that builds the `tenacity.AsyncRetrying` object for the default retry policy.

```APIDOC
## RetryFactory

```python
class RetryFactory[source]
```

Factory class that builds the `tenacity.AsyncRetrying` object that defines the default retry policy. To create a custom retry policy, you can subclass this factory class, modify it as needed, and then call `build()` on your subclass to get the corresponding `tenacity.AsyncRetrying` object.
```

--------------------------------

### AsyncZyteAPI Authentication Info Property

Source: https://python-zyte-api.readthedocs.io/en/stable/_modules/zyte_api/_async.html

Access authentication details like the API key or Ethereum key using the 'auth' property. The 'type' attribute indicates whether it's a 'zyte' (API key) or 'eth' (Ethereum key) authentication.

```python
from zyte_api import AsyncZyteAPI

client = AsyncZyteAPI(api_key="YOUR_API_KEY")

print(f"Auth Type: {client.auth.type}")
print(f"Auth Key: {client.auth.key}")
```
Tokens:
Raw