"
""
""
""
"")
# Individual actions
await SetInputValue("#name", "World").run(page)
await ClickElement("#greet").run(page)
title = await GetText("#title").run(page)
print(f"Title: {title}") # "Hello, World!"
# Custom JS action
count = await AppendToOutput("Added via action").run(page)
print(f"Output children: {count}")
# Composed flow
flow = Flow([
ScrollTo(0, 0),
AppendToOutput("Added via flow"),
GetText("#output"),
])
result = await flow.run(page)
print(f"Output text: {result.last}")
await page.close()
asyncio.run(main())
```
--------------------------------
### Initialize Project Environment
Source: https://cascadinglabs.com/voidcrawl/contributing
Commands to clone the repository, build the project, and install dependencies.
```bash
git clone https://github.com/CascadingLabs/VoidCrawl
cd VoidCrawl
./build.sh
```
```bash
uv sync --all-groups
```
```bash
uvx prek install
```
--------------------------------
### Run Basic Navigation Example
Source: https://cascadinglabs.com/voidcrawl/guides/examples
Execute the basic navigation example script to see VoidCrawl in action. This script demonstrates launching a headless browser, navigating to a page, and reading its title and content.
```bash
python examples/basic_navigation.py
```
--------------------------------
### Verify VoidCrawl Installation
Source: https://cascadinglabs.com/voidcrawl/installation
Check the installation by importing the module and listing its contents.
```python
import voidcrawl
print(dir(voidcrawl))
# ['BrowserConfig', 'BrowserPool', 'BrowserSession', 'Page', 'PoolConfig', 'PooledTab']
```
--------------------------------
### Expected Output
Source: https://cascadinglabs.com/voidcrawl/guides/examples/basic-navigation
This is the expected output when running the basic navigation example.
```text
Title: Example Domain
URL: https://example.com/
HTML length: 1256 chars
```
--------------------------------
### Build VoidCrawl from Source
Source: https://cascadinglabs.com/voidcrawl/installation
Commands to clone, build, and install the project from the source repository.
```bash
git clone https://github.com/CascadingLabs/VoidCrawl
```
```bash
cd VoidCrawl
```
```bash
./build.sh
```
```bash
uv sync --all-groups
```
--------------------------------
### Start Headless Docker Compose
Source: https://cascadinglabs.com/voidcrawl/guides/docker
Use this command to start the headless Docker containers in detached mode. Ensure you are in the 'docker' directory.
```bash
cd docker
docker compose up -d
```
--------------------------------
### Install VoidCrawl via Package Managers
Source: https://cascadinglabs.com/voidcrawl/installation
Commands to install the package using common Python package managers.
```bash
uv add voidcrawl
```
```bash
pip install voidcrawl
```
```bash
poetry add voidcrawl
```
--------------------------------
### Conventional Commit Examples
Source: https://cascadinglabs.com/voidcrawl/contributing
Examples of commit messages following the conventional commits standard.
```bash
git commit -m "feat: add new pool eviction strategy"
git commit -m "fix: handle CDP timeout on slow networks"
git commit -m "docs: update stealth mode guide"
```
--------------------------------
### Run Headful Docker Container
Source: https://cascadinglabs.com/voidcrawl/guides/examples/docker-headful
Start the headful Docker container. Auto-detects GPU or specify with --gpu.
```bash
./docker/run-headful.sh # auto-detects your GPU
# or: ./docker/run-headful.sh --gpu amd
```
--------------------------------
### Running VoidCrawl Code with asyncio.run()
Source: https://cascadinglabs.com/voidcrawl/guides/async-native
Your entry point needs an event loop. Use `asyncio.run()` to start the event loop and execute your async main function.
```python
import asyncio
async def main():
# your VoidCrawl code here
pass
asyncio.run(main())
```
--------------------------------
### Scrape with QScrape
Source: https://cascadinglabs.com/voidcrawl/quickstart
Example of querying DOM elements from a target website using CSS selectors.
```python
import asyncio
from voidcrawl import BrowserPool, PoolConfig
async def main():
async with BrowserPool(PoolConfig()) as pool:
async with pool.acquire() as tab:
await tab.navigate(
"https://qscrape.dev/l1/eshop/catalog/"
"?cat=Forge%20%26%20Smithing"
)
title = await tab.title()
print(f"Page: {title}")
# Query product names from the DOM
products = await tab.query_selector_all(".product-name")
for p in products[:5]:
print(f" - {p}")
asyncio.run(main())
```
--------------------------------
### Run Headful Docker Script
Source: https://cascadinglabs.com/voidcrawl/guides/docker
Execute the run-headful.sh script to start a headful Docker container. This script auto-detects your GPU or allows manual specification.
```bash
# Auto-detects your GPU (AMD/Intel/NVIDIA) and starts everything
./docker/run-headful.sh
# Or specify GPU manually
./docker/run-headful.sh --gpu amd
./docker/run-headful.sh --gpu nvidia
./docker/run-headful.sh --gpu intel
./docker/run-headful.sh --gpu cpu # no GPU, software rendering
```
--------------------------------
### Navigate and Extract Page Info
Source: https://cascadinglabs.com/voidcrawl/guides/examples/basic-navigation
Use this snippet to perform basic web scraping tasks. Ensure asyncio is installed and VoidCrawl is imported. The BrowserPool manages browser instances and tabs for efficient reuse.
```python
import asyncio
from voidcrawl import BrowserPool, PoolConfig
async def main() -> None:
async with BrowserPool(PoolConfig()) as pool, pool.acquire() as tab:
await tab.navigate("https://example.com")
title = await tab.title()
url = await tab.url()
html = await tab.content()
print(f"Title: {title}")
print(f"URL: {url}")
print(f"HTML length: {len(html)} chars")
if __name__ == "__main__":
asyncio.run(main())
```
--------------------------------
### Define Custom CDP Action
Source: https://cascadinglabs.com/voidcrawl/guides/examples/actions-demo
Extends ActionNode to create a custom action that interacts with the browser via CDP. This example implements a double-click by dispatching mouse events.
```python
class CdpDoubleClick(ActionNode):
"""Custom action: double-click at coordinates via CDP."""
def __init__(self, x: float, y: float) -> None:
self.x = x
self.y = y
async def run(self, tab: Tab) -> None:
for _ in range(2):
await tab.dispatch_mouse_event(
"mousePressed", self.x, self.y, click_count=2
)
await tab.dispatch_mouse_event(
"mouseReleased", self.x, self.y, click_count=2
)
```
--------------------------------
### Instantiate and Run Actions
Source: https://cascadinglabs.com/voidcrawl/guides/builtin-actions
Instantiate actions with parameters and run them against a tab. Requires importing necessary actions.
```python
from voidcrawl.actions import ClickElement, GetText, SetInputValue
async with pool.acquire() as tab:
await tab.navigate("https://example.com")
# Each action is instantiated with parameters, then run against a tab
await SetInputValue("#search", "hello").run(tab)
await ClickElement("#submit").run(tab)
title = await GetText("h1").run(tab)
```
--------------------------------
### Initialize BrowserPool and Navigate
Source: https://cascadinglabs.com/voidcrawl
Demonstrates basic usage of the BrowserPool to acquire a tab, navigate to a URL, and retrieve page content.
```python
import asyncio
from voidcrawl import BrowserPool, PoolConfig
async def main():
async with BrowserPool(PoolConfig()) as pool:
async with pool.acquire() as tab:
await tab.navigate("https://example.com")
print(await tab.title())
print(len(await tab.content()))
asyncio.run(main())
```
--------------------------------
### Attribute and Text Retrieval
Source: https://cascadinglabs.com/voidcrawl/reference/api-reference
Functions for getting element attributes and text content.
```APIDOC
## `GetAttribute`
### Description
Read an HTML attribute from the first matching element. Returns `None` if the element is not found. The result is available as the return value of :meth:`run` (`str | None`).
### Method
GET
### Endpoint
/websites/cascadinglabs_voidcrawl/get_attribute
### Parameters
#### Query Parameters
- **selector** (str) - Required - CSS selector targeting the element.
- **attr** (str) - Required - Attribute name (e.g. `"href"`, `"data-id"`).
### Request Example
```json
{
"selector": "#my-link",
"attr": "href"
}
```
### Response
#### Success Response (200)
- **value** (str | None) - The attribute value or null if not found.
#### Response Example
```json
{
"value": "https://example.com"
}
```
## `GetText`
### Description
Read `textContent` from the first matching element. Returns `None` if the element is not found. The result is available as the return value of :meth:`run` (`str | None`).
### Method
GET
### Endpoint
/websites/cascadinglabs_voidcrawl/get_text
### Parameters
#### Query Parameters
- **selector** (str) - Required - CSS selector targeting the element.
### Request Example
```json
{
"selector": ".my-element"
}
```
### Response
#### Success Response (200)
- **value** (str | None) - The text content or null if not found.
#### Response Example
```json
{
"value": "Element text"
}
```
```
--------------------------------
### Manual Build Command
Source: https://cascadinglabs.com/voidcrawl/installation
Alternative command to build the Rust extension manually without using the build script.
```bash
maturin develop --release --manifest-path crates/pyo3_bindings/Cargo.toml
```
--------------------------------
### BrowserPool.acquire
Source: https://cascadinglabs.com/voidcrawl/reference/api-reference
Checks out a tab from the pool as an async context manager.
```APIDOC
## acquire()
### Description
Check out a tab from the pool as an async context manager. The tab is automatically returned to the pool when the context exits, even on exception.
### Returns
- **_AcquireContext** - An async context manager yielding a PooledTab.
```
--------------------------------
### Define Custom JavaScript Action
Source: https://cascadinglabs.com/voidcrawl/guides/examples/actions-demo
Extends JsActionNode to create a custom action that manipulates the DOM. This example appends text to a specified selector using inline JavaScript.
```python
class AppendToOutput(JsActionNode):
"""Custom action: append text to the #output div."""
js = inline_js("""
const el = document.querySelector(__params.selector);
el.innerHTML += '
' + __params.text + '
';
return el.children.length;
""")
def __init__(self, text: str, selector: str = "#output") -> None:
self.text = text
self.selector = selector
```
--------------------------------
### Run Integration Tests
Source: https://cascadinglabs.com/voidcrawl/contributing
Commands to execute integration tests for Rust and Python components.
```bash
cargo test -p voidcrawl_core -- --test-threads=1
```
```bash
uv run pytest tests/ -v
```
--------------------------------
### Rust Development Commands
Source: https://cascadinglabs.com/voidcrawl/contributing
Standard CLI commands for maintaining and testing the Rust core.
```bash
cargo check
```
```bash
cargo clippy --workspace --all-targets
```
```bash
cargo +nightly fmt --all
```
```bash
cargo test -p voidcrawl_core -- --test-threads=1
```
--------------------------------
### BrowserPool.warmup
Source: https://cascadinglabs.com/voidcrawl/reference/api-reference
Pre-opens tabs across all browser sessions to reduce latency.
```APIDOC
## warmup()
### Description
Pre-open tabs across all browser sessions. Call after entering the pool context to eliminate cold-start latency on the first acquire calls.
```
--------------------------------
### Python Development Commands
Source: https://cascadinglabs.com/voidcrawl/contributing
Standard CLI commands for linting, formatting, and testing the Python components.
```bash
uv run ruff check .
```
```bash
uv run ruff format
```
```bash
uv run mypy
```
```bash
uv run pytest tests/ -v
```
```bash
uvx prek run --all-files
```
--------------------------------
### Configure headful browser pool
Source: https://cascadinglabs.com/voidcrawl/guides/stealth
Initializes a browser pool with headful mode enabled to bypass WAF detection that targets headless browser characteristics.
```python
from voidcrawl import BrowserConfig, BrowserPool, PoolConfig
# For WAF-protected sites -- use headful
config = PoolConfig(
browser=BrowserConfig(headless=False),
)
async with BrowserPool(config) as pool:
async with pool.acquire() as tab:
await tab.navigate("https://waf-protected-site.com")
await tab.wait_for_stable_dom(timeout=15.0)
html = await tab.content()
```
--------------------------------
### Perform DOM Interactions with Voidcrawl
Source: https://cascadinglabs.com/voidcrawl/guides/examples/dom-interaction
Demonstrates querying elements, typing into inputs, and clicking buttons using CSS selectors. Requires a running BrowserSession.
```python
import asyncio
from voidcrawl import BrowserConfig, BrowserSession
FORM_PAGE = """data:text/html,
Hello
"""
async def main() -> None:
async with BrowserSession(BrowserConfig()) as browser:
page = await browser.new_page(FORM_PAGE)
# Query a single element (returns inner HTML or None)
heading = await page.query_selector("#greeting")
print(f"Heading HTML: {heading}")
# Query multiple elements
all_inputs = await page.query_selector_all("input")
print(f"Found {len(all_inputs)} input(s)")
# Type into the input and click the button
await page.type_into("#name", "World")
await page.click_element("#btn")
# Check the updated heading
updated = await page.query_selector("#greeting")
print(f"Updated heading: {updated}")
# Missing selectors return None
missing = await page.query_selector("#does-not-exist")
print(f"Missing element: {missing}")
await page.close()
if __name__ == "__main__":
asyncio.run(main())
```
--------------------------------
### Sequential Navigation and Data Fetching
Source: https://cascadinglabs.com/voidcrawl/guides/async-native
Navigate to a URL, then retrieve its title and HTML content sequentially within an async context.
```python
async with pool.acquire() as tab:
await tab.navigate("https://example.com")
title = await tab.title()
html = await tab.content()
```
--------------------------------
### Run Voidcrawl Actions Demo
Source: https://cascadinglabs.com/voidcrawl/guides/examples/actions-demo
Main asynchronous function to demonstrate Voidcrawl actions. It initializes a browser session, navigates to a demo page, and executes various actions including individual, custom JS, flow, and CDP actions.
```python
async def main() -> None:
async with BrowserSession(BrowserConfig()) as browser:
page = await browser.new_page(DEMO_PAGE)
# 1. Individual prebaked actions
print("--- Individual actions ---")
await SetInputValue("#name", "World").run(page)
await ClickElement("#greet").run(page)
title = await GetText("#title").run(page)
print(f"Title after greet: {title}")
# 2. Custom JS action
print("\n--- Custom JS action ---")
count = await AppendToOutput("First line").run(page)
print(f"Output children after append: {count}")
# 3. Composed flow
print("\n--- Flow ---")
flow = Flow([
ScrollTo(0, 0),
AppendToOutput("Added via flow"),
GetText("#output"),
])
result = await flow.run(page)
print(f"Flow results: {result.results}")
print(f"Last result (output text): {result.last}")
# 4. CDP-level action
print("\n--- CDP click ---")
await CdpClick(100.0, 50.0).run(page)
print("CDP click dispatched at (100, 50)")
await page.close()
if __name__ == "__main__":
asyncio.run(main())
```
--------------------------------
### Using Async Context Managers for BrowserPool and BrowserSession
Source: https://cascadinglabs.com/voidcrawl/guides/async-native
Employ `async with` for `BrowserPool` and `BrowserSession` to ensure proper cleanup of browser processes and tabs, even when exceptions occur.
```python
async with pool.acquire() as tab:
# Your code using the tab
pass
```
--------------------------------
### Configure BrowserPool via Environment Variables
Source: https://cascadinglabs.com/voidcrawl/guides/browser-pool
Load BrowserPool configuration from environment variables using PoolConfig.from_env(). This is useful for deployment scenarios where configuration is managed externally.
```python
from voidcrawl import BrowserPool, PoolConfig
# PoolConfig.from_env() reads all config from env vars
config = PoolConfig.from_env()
async with BrowserPool(config) as pool:
...
```
--------------------------------
### Action Framework
Source: https://cascadinglabs.com/voidcrawl/reference/api-reference
Classes and methods for defining and executing sequences of browser actions.
```APIDOC
## Class: ActionNode
### Description
Abstract base for all browser actions. Subclass and implement :meth:`run` to create a custom action.
### Method: run
`run(tab: Tab) -> object`
Execute this action against _tab_.
#### Parameters
- **tab** (Tab) - Required - Any object satisfying the :class:`~voidcrawl.actions.Tab` protocol (e.g. :class:`Page` or :class:`PooledTab`).
#### Returns
- **object** - The action result — type varies by action.
## Class: Flow
### Description
An ordered sequence of actions executed against a single tab.
### Method: add
`add(action: ActionNode) -> Flow`
Append an action and return _self_ for chaining.
#### Parameters
- **action** (ActionNode) - Required - The action to append.
#### Returns
- **Flow** - class:`Flow` instance (for builder-style chaining).
### Method: run
`run(tab: Tab) -> FlowResult`
Execute all actions sequentially against _tab_.
#### Parameters
- **tab** (Tab) - Required - Any object satisfying the :class:`~voidcrawl.actions.Tab` protocol.
#### Returns
- **FlowResult** - class:`FlowResult` containing one result per action.
## Class: FlowResult
### Description
Aggregated result of a :class:`Flow` execution.
### Attributes
- **results** (list[object]) - Ordered list of return values, one per action.
- **last** (object) - The return value of the final action, or None for empty flows (read-only property).
## Class: JsActionNode
### Description
Action executed by evaluating a JavaScript snippet in the page.
### Method: params
`params() -> dict[str, Any]`
Return the parameters injected as `__params` in the JS snippet.
#### Returns
- **dict[str, Any]** - A JSON-serialisable dict of parameter names to values.
### Method: run
`run(tab: JsTab) -> object`
Evaluate the JS snippet in _tab_ with the current :meth:`params`.
#### Parameters
- **tab** (JsTab) - Required - Any object satisfying :class:`~voidcrawl.actions.JsTab`.
#### Returns
- **object** - The JSON-deserialised return value from the snippet.
## Class: JsSource
### Description
Immutable wrapper around a JavaScript snippet string.
## Class: JsTab
### Description
Minimal protocol for JavaScript-only actions.
### Method: evaluate_js
`evaluate_js(expression: str) -> object`
Evaluate a JavaScript _expression_ in the page and return the result.
#### Parameters
- **expression** (str) - Required - JavaScript expression or IIFE string.
#### Returns
- **object** - The JSON-deserialised return value from the browser.
```
--------------------------------
### Browser Session Management
Source: https://cascadinglabs.com/voidcrawl/reference/api-reference
Methods for initializing new pages and retrieving browser version information.
```APIDOC
## new_page
### Description
Open a new tab and navigate to the specified URL.
### Parameters
#### Request Body
- **url** (str) - Required - The URL to load in the new tab.
### Response
- **Page** (object) - A handle for the new tab.
## version
### Description
Return the browser version string.
### Response
- **version** (str) - The Chrome/Chromium product version.
```
--------------------------------
### Run Chrome in Docker
Source: https://cascadinglabs.com/voidcrawl/quickstart
Deploy Chrome as a persistent daemon in Docker and connect via WebSocket URLs.
```bash
cd docker
docker compose up -d
```
```bash
export CHROME_WS_URLS="http://localhost:9222,http://localhost:9223"
python your_script.py
```
--------------------------------
### Create a CDP Action
Source: https://cascadinglabs.com/voidcrawl/guides/custom-js-actions
Implement low-level Chrome DevTools Protocol interactions by subclassing ActionNode and defining a custom run method.
```python
from voidcrawl.actions import ActionNode, Tab
class CdpDoubleClick(ActionNode):
"""Double-click at coordinates via CDP."""
def __init__(self, x: float, y: float) -> None:
self.x = x
self.y = y
async def run(self, tab: Tab) -> None:
for _ in range(2):
await tab.dispatch_mouse_event(
"mousePressed", self.x, self.y, click_count=2
)
await tab.dispatch_mouse_event(
"mouseReleased", self.x, self.y, click_count=2
)
```
--------------------------------
### Configure BrowserPool via Constructor
Source: https://cascadinglabs.com/voidcrawl/guides/browser-pool
Configure the BrowserPool using a PoolConfig object passed to the constructor. This allows setting parameters like the number of browsers, tabs per browser, and tab usage limits.
```python
from voidcrawl import BrowserConfig, BrowserPool, PoolConfig
config = PoolConfig(
browsers=2,
tabs_per_browser=4,
tab_max_uses=50,
tab_max_idle_secs=60,
browser=BrowserConfig(headless=True, stealth=True),
)
async with BrowserPool(config) as pool:
async with pool.acquire() as tab:
await tab.navigate("https://example.com")
```
--------------------------------
### VoidCrawl Headful Connection and Navigation
Source: https://cascadinglabs.com/voidcrawl/guides/examples/docker-headful
Connects VoidCrawl to existing Chrome instances in Docker and performs basic navigation and data extraction. Requires `headless=False` in BrowserConfig.
```python
import asyncio
from voidcrawl import BrowserConfig, BrowserPool, PoolConfig
async def main() -> None:
config = PoolConfig(
chrome_ws_urls=[
"http://localhost:19222",
"http://localhost:19223",
],
tabs_per_browser=2,
browser=BrowserConfig(headless=False),
)
async with BrowserPool(config) as pool:
# -- Basic navigation --
async with pool.acquire() as tab:
event = await tab.goto(
"https://en.wikipedia.org/wiki/Web_scraping",
timeout=30.0,
)
print(f"Wait event: {event}")
title = await tab.title()
html = await tab.content()
print(f"Title: {title}")
print(f"HTML: {len(html):,} chars")
# DOM queries
headings = await tab.query_selector_all("#toc li a")
print(f"Table of contents entries: {len(headings)}")
for h in headings[:5]:
print(f" - {h}")
# Screenshot
png_bytes = await tab.screenshot_png()
print(f"Screenshot: {len(png_bytes):,} bytes")
# JavaScript evaluation
link_count = await tab.evaluate_js(
'document.querySelectorAll("a").length'
)
print(f"Links on page: {link_count}")
# -- Parallel fetch (watch both tabs in VNC!) --
print("\nParallel fetch...")
async def fetch(url: str) -> tuple[str, int]:
async with pool.acquire() as tab:
await tab.goto(url)
t = await tab.title()
length = len(await tab.content())
return t or "(no title)", length
results = await asyncio.gather(
fetch("https://en.wikipedia.org/wiki/Web_scraping"),
fetch(
"https://en.wikipedia.org/"
"wiki/Rust_(programming_language)"
),
)
for t, length in results:
print(f" {t}: {length:,} chars")
print("\nDone! The Docker container is still running.")
print("Connect VNC to localhost:5900 to see the Chrome windows.")
if __name__ == "__main__":
asyncio.run(main())
```
--------------------------------
### Execute a Custom Action
Source: https://cascadinglabs.com/voidcrawl/guides/custom-js-actions
Run a custom action instance by calling the run method on a browser tab.
```python
async with pool.acquire() as tab:
await tab.navigate("https://example.com")
count = await AppendToOutput("Hello!").run(tab)
print(f"Children after append: {count}")
```
--------------------------------
### Set Custom Resolution for Headful Docker
Source: https://cascadinglabs.com/voidcrawl/guides/docker
Launch the headful Docker container with custom VNC resolution by setting VNC_WIDTH and VNC_HEIGHT environment variables.
```bash
VNC_WIDTH=2560 VNC_HEIGHT=1440 ./docker/run-headful.sh # 2K
VNC_WIDTH=1280 VNC_HEIGHT=720 ./docker/run-headful.sh # 720p (lower memory)
```
--------------------------------
### Tab Protocol Methods
Source: https://cascadinglabs.com/voidcrawl/reference/api-reference
Methods for dispatching low-level CDP input events to a browser tab.
```APIDOC
## dispatch_key_event
### Description
Send a low-level CDP Input.dispatchKeyEvent to the tab.
### Parameters
- **event_type** (str) - Required - "keyDown", "keyUp", "rawKeyDown", or "char".
- **key** (str) - Optional - DOM KeyboardEvent.key value.
- **code** (str) - Optional - Physical key code.
- **text** (str) - Optional - Character to insert.
- **modifiers** (int) - Optional - Bit field for modifier keys.
## dispatch_mouse_event
### Description
Send a low-level CDP Input.dispatchMouseEvent to the tab.
### Parameters
- **event_type** (str) - Required - "mousePressed", "mouseReleased", "mouseMoved", or "mouseWheel".
- **x** (float) - Required - Horizontal page coordinate.
- **y** (float) - Required - Vertical page coordinate.
- **button** (str) - Optional - "left", "right", or "middle".
- **click_count** (int) - Optional - Number of clicks.
- **delta_x** (float) - Optional - Horizontal scroll delta.
- **delta_y** (float) - Optional - Vertical scroll delta.
- **modifiers** (int) - Optional - Bit field for modifier keys.
```
--------------------------------
### PoolConfig.from_env
Source: https://cascadinglabs.com/voidcrawl/reference/api-reference
Builds a PoolConfig object by reading configuration from environment variables.
```APIDOC
## from_env()
### Description
Builds a PoolConfig from environment variables. Reads settings like CHROME_WS_URLS, BROWSER_COUNT, TABS_PER_BROWSER, TAB_MAX_USES, TAB_MAX_IDLE_SECS, CHROME_NO_SANDBOX, and CHROME_HEADLESS.
### Returns
- **PoolConfig** - A fully-populated PoolConfig object.
```
--------------------------------
### Connect via Native VNC Client
Source: https://cascadinglabs.com/voidcrawl/guides/docker
Commands to connect to the headful Docker container's VNC server using TigerVNC or Remmina. Note that VNC uses a binary protocol (RFB), not HTTP.
```bash
# TigerVNC
vncviewer localhost:5900
# Or Remmina: New connection -> Protocol: VNC -> Server: localhost:5900
```
--------------------------------
### Perform direct page operations with PooledTab
Source: https://cascadinglabs.com/voidcrawl/guides/builtin-actions
Use these methods for standard navigation, element selection, interaction, and JavaScript execution within an acquired tab.
```python
async with pool.acquire() as tab:
await tab.navigate(url)
# DOM queries
html = await tab.query_selector("#main")
items = await tab.query_selector_all(".item")
# Interaction
await tab.click_element("#btn")
await tab.type_into("#input", "text")
# JS evaluation
result = await tab.evaluate_js("document.title")
```
--------------------------------
### Connect VoidCrawl to Headful Docker Chrome
Source: https://cascadinglabs.com/voidcrawl/guides/docker
Python script to configure BrowserPool for headful Chrome instances, specifying CDP URLs and disabling headless mode. Requires asyncio.
```python
import asyncio
from voidcrawl import BrowserConfig, BrowserPool, PoolConfig
async def main():
config = PoolConfig(
chrome_ws_urls=["http://localhost:19222", "http://localhost:19223"],
tabs_per_browser=2,
browser=BrowserConfig(headless=False),
)
async with BrowserPool(config) as pool:
async with pool.acquire() as tab:
# Watch this in VNC!
await tab.navigate("https://en.wikipedia.org/wiki/Web_scraping")
print(f"Title: {await tab.title()}")
asyncio.run(main())
```
--------------------------------
### Input Actions
Source: https://cascadinglabs.com/voidcrawl/guides/builtin-actions
Manage input fields using SetInputValue, ClearInput, and SelectOption. These actions require a CSS selector and appropriate values.
```python
from voidcrawl.actions import SetInputValue, ClearInput, SelectOption
await SetInputValue("#name", "World").run(tab)
await ClearInput("#name").run(tab)
await SelectOption("#country", "US").run(tab)
```
--------------------------------
### Page Navigation and Content
Source: https://cascadinglabs.com/voidcrawl/reference/api-reference
Methods for navigating to URLs, retrieving page content, and managing page state.
```APIDOC
## goto
### Description
Navigate to a URL and wait for network idle.
### Parameters
#### Request Body
- **url** (str) - Required - The URL to load.
- **timeout** (float) - Optional - Maximum seconds to wait.
### Response
- **status** (str | None) - "networkIdle", "networkAlmostIdle", or None on timeout.
## content
### Description
Return the full page HTML.
### Response
- **html** (str) - The complete outer HTML of the document element.
```
--------------------------------
### PooledTab Methods
Source: https://cascadinglabs.com/voidcrawl/reference/api-reference
This section details the various methods available on a PooledTab object for interacting with a web page.
```APIDOC
## PooledTab Methods
### `wait_for_network_idle`
Waits for network activity to settle on the current page.
**Args:**
* `timeout` (float) - Maximum seconds to wait. Defaults to 30.0.
**Returns:**
* `str | None` - Returns "networkIdle" or "networkAlmostIdle" on success, or `None` if the timeout is reached.
```
```APIDOC
### `wait_for_stable_dom`
Waits until the DOM stabilizes, meaning it stops changing.
**Args:**
* `timeout` (float) - Maximum seconds to wait. Defaults to 10.0.
* `min_length` (int) - Minimum HTML length before stability checks begin. Defaults to 5000.
* `stable_checks` (int) - The number of consecutive unchanged polls required to consider the DOM stable. Defaults to 5.
**Returns:**
* `bool` - `True` if the DOM stabilized within the timeout, `False` otherwise.
```
```APIDOC
### `click_element`
Clicks the first HTML element that matches the provided CSS selector.
**Args:**
* `selector` (str) - The CSS selector string for the element to click.
```
```APIDOC
### `content`
Retrieves the complete HTML content of the current page.
**Returns:**
* `str` - The outer HTML of the `document.documentElement`.
```
```APIDOC
### `dispatch_key_event`
Sends a low-level CDP `Input.dispatchKeyEvent` to simulate keyboard input.
**Args:**
* `event_type` (str) - The type of key event (e.g., "keyDown", "keyUp", "rawKeyDown", "char").
* `key` (str | None) - The DOM `KeyboardEvent.key` value (e.g., "Enter").
* `code` (str | None) - The physical key code (e.g., "KeyA").
* `text` (str | None) - The character to insert (e.g., "a").
* `modifiers` (int | None) - A bit field representing modifier keys (e.g., Ctrl, Shift).
```
```APIDOC
### `dispatch_mouse_event`
Sends a low-level CDP `Input.dispatchMouseEvent` to simulate mouse input.
**Args:**
* `event_type` (str) - The type of mouse event (e.g., "mousePressed", "mouseReleased", "mouseMoved", "mouseWheel").
* `x` (float) - The horizontal page coordinate.
* `y` (float) - The vertical page coordinate.
* `button` (str) - The mouse button to use ("left", "right", or "middle"). Defaults to "left".
* `click_count` (int) - The number of clicks. Defaults to 1.
* `delta_x` (float | None) - Horizontal scroll delta (used for "mouseWheel" events).
* `delta_y` (float | None) - Vertical scroll delta (used for "mouseWheel" events).
* `modifiers` (int | None) - A bit field representing modifier keys.
```
```APIDOC
### `evaluate_js`
Evaluates a JavaScript expression within the page's context and returns the result.
**Args:**
* `expression` (str) - The JavaScript expression or IIFE string to evaluate.
**Returns:**
* `object` - The deserialized result of the JavaScript expression (e.g., dict, list, str, int, float, bool, None).
```
```APIDOC
### `goto`
Navigates to a specified URL and waits for the network to become idle.
**Args:**
* `url` (str) - The URL to navigate to.
* `timeout` (float) - Maximum seconds to wait for network idle. Defaults to 30.0.
**Returns:**
* `str | None` - Returns "networkIdle" or "networkAlmostIdle" on success, or `None` if the timeout is reached.
```
```APIDOC
### `navigate`
Navigates to a specified URL without waiting for any load events.
**Args:**
* `url` (str) - The URL to navigate to.
```
```APIDOC
### `query_selector`
Finds the first HTML element matching the given CSS selector and returns its outer HTML.
**Args:**
* `selector` (str) - The CSS selector string.
**Returns:**
* `str | None` - The outer HTML of the matched element, or `None` if no element is found.
```
```APIDOC
### `query_selector_all`
Finds all HTML elements matching the given CSS selector and returns their outer HTML.
**Args:**
* `selector` (str) - The CSS selector string.
**Returns:**
* `list[str]` - A list containing the outer HTML strings of all matched elements.
```
```APIDOC
### `screenshot_png`
Captures a full-page screenshot of the current page and returns it as PNG bytes.
**Returns:**
* `bytes` - The raw PNG image data.
```
```APIDOC
### `set_headers`
Sets custom HTTP headers that will be used for all subsequent requests made from this tab.
**Args:**
* `headers` (dict[str, str]) - A dictionary of header names and their corresponding values.
```
```APIDOC
### `title`
Retrieves the title of the current HTML document.
**Returns:**
* `str | None` - The document's title string, or `None` if it's unavailable.
```
--------------------------------
### Connect VoidCrawl to Headless Chrome
Source: https://cascadinglabs.com/voidcrawl/guides/docker
Python script to configure and use BrowserPool with headless Chrome instances exposed on specific WebSocket URLs. Requires asyncio.
```python
import asyncio
from voidcrawl import BrowserPool, PoolConfig
async def main():
config = PoolConfig(
chrome_ws_urls=["http://localhost:9222", "http://localhost:9223"],
tabs_per_browser=4,
)
async with BrowserPool(config) as pool:
async with pool.acquire() as tab:
await tab.navigate("https://example.com")
print(await tab.title())
asyncio.run(main())
```
--------------------------------
### DOM Query Actions
Source: https://cascadinglabs.com/voidcrawl/guides/builtin-actions
Retrieve data from the DOM using GetText and GetAttribute. SetAttribute can modify element attributes. Ensure the tab object is available.
```python
from voidcrawl.actions import GetText, GetAttribute
title = await GetText("h1").run(tab)
href = await GetAttribute("a.logo", "href").run(tab)
```
--------------------------------
### Open Multiple Pages with BrowserSession
Source: https://cascadinglabs.com/voidcrawl/guides/examples/multi-page
Use BrowserSession for low-level control over individual tabs. Ensure pages are closed after use to free browser resources.
```python
import asyncio
from voidcrawl import BrowserConfig, BrowserSession
URLS = [
"https://example.com",
"https://httpbin.org/html",
"https://www.iana.org/domains/reserved",
]
async def main() -> None:
async with BrowserSession(BrowserConfig()) as browser:
pages = [await browser.new_page(url) for url in URLS]
for page in pages:
title = await page.title()
url = await page.url()
print(f" {url} -> {title}")
for page in pages:
await page.close()
if __name__ == "__main__":
asyncio.run(main())
```
--------------------------------
### JavaScript Utilities
Source: https://cascadinglabs.com/voidcrawl/reference/api-reference
Functions for creating JsSource objects from inline code or external files.
```APIDOC
## inline_js
### Description
Create a JsSource from an inline string literal.
### Parameters
- **code** (str) - Required - Raw JavaScript source code.
## load_js
### Description
Load JavaScript from a .js file on disk.
### Parameters
- **path** (str | Path) - Required - Filesystem path to the .js file.
```
--------------------------------
### Voidcrawl Imports and Demo Page
Source: https://cascadinglabs.com/voidcrawl/guides/examples/actions-demo
Imports necessary modules from the voidcrawl library and defines the HTML content for the demo page as a data URL.
```python
import asyncio
from voidcrawl import BrowserConfig, BrowserSession
from voidcrawl.actions import (
ActionNode,
CdpClick,
ClickElement,
Flow,
GetText,
JsActionNode,
ScrollTo,
SetInputValue,
Tab,
inline_js,
)
DEMO_PAGE = """data:text/html,
Actions Demo
"""
```
--------------------------------
### Click Actions
Source: https://cascadinglabs.com/voidcrawl/guides/builtin-actions
Use ClickElement for CSS selector-based clicks and ClickAt for specific coordinates. Ensure the tab object is available.
```python
from voidcrawl.actions import ClickElement, ClickAt
await ClickElement("#submit-btn").run(tab)
await ClickAt(100, 200).run(tab)
```
--------------------------------
### Capture and Save PNG Screenshot
Source: https://cascadinglabs.com/voidcrawl/guides/examples/screenshots
Uses screenshot_png() to retrieve raw bytes and writes them to a file using Path.write_bytes(). Requires an active BrowserPool session.
```python
import asyncio
from pathlib import Path
from voidcrawl import BrowserPool, PoolConfig
OUTPUT_DIR = Path("output")
async def _capture() -> None:
async with BrowserPool(PoolConfig()) as pool, pool.acquire() as tab:
await tab.navigate("https://example.com")
# PNG screenshot
png_bytes = await tab.screenshot_png()
png_path = OUTPUT_DIR / "example.png"
png_path.write_bytes(png_bytes)
print(f"Screenshot saved: {png_path} ({len(png_bytes)} bytes)")
def main() -> None:
OUTPUT_DIR.mkdir(exist_ok=True)
asyncio.run(_capture())
if __name__ == "__main__":
main()
```
--------------------------------
### Parallel Fetching with asyncio.gather
Source: https://cascadinglabs.com/voidcrawl/guides/async-native
Fetch content from multiple URLs concurrently using `asyncio.gather`. Each fetch operation acquires a tab, navigates, and returns the content.
```python
async def fetch(pool, url):
async with pool.acquire() as tab:
await tab.navigate(url)
return await tab.content()
results = await asyncio.gather(
fetch(pool, "https://example.com"),
fetch(pool, "https://httpbin.org/html"),
)
```
--------------------------------
### Compare Stealth Mode Fingerprint Signals
Source: https://cascadinglabs.com/voidcrawl/guides/examples/stealth-mode
Uses BrowserConfig to toggle stealth mode and evaluates JavaScript to inspect navigator properties. Requires the voidcrawl library.
```python
import asyncio
import json
from voidcrawl import BrowserConfig, BrowserSession, Page
DETECTION_JS = """
JSON.stringify({
webdriver: navigator.webdriver,
plugins_count: navigator.plugins.length,
languages: navigator.languages,
has_chrome_runtime: typeof window.chrome !== 'undefined'
&& typeof window.chrome.runtime !== 'undefined',
})
"""
async def check_fingerprint(label: str, page: Page) -> None:
raw = await page.evaluate_js(DETECTION_JS)
fingerprint = json.loads(raw)
print(f"\n[{label}]")
for key, value in fingerprint.items():
print(f" {key}: {value}")
async def main() -> None:
# Stealth ON (default)
async with BrowserSession(BrowserConfig(stealth=True)) as browser:
page = await browser.new_page("https://example.com")
await check_fingerprint("stealth=True", page)
await page.close()
# Stealth OFF
async with BrowserSession(BrowserConfig(stealth=False)) as browser:
page = await browser.new_page("https://example.com")
await check_fingerprint("stealth=False", page)
await page.close()
if __name__ == "__main__":
asyncio.run(main())
```
--------------------------------
### Use BrowserSession for direct control
Source: https://cascadinglabs.com/voidcrawl/quickstart
Manage browser sessions manually without pooling. Requires explicit page closure.
```python
import asyncio
from voidcrawl import BrowserConfig, BrowserSession
async def main():
async with BrowserSession(BrowserConfig()) as session:
page = await session.new_page("https://example.com")
print(await page.title()) # "Example Domain"
print(len(await page.content()))
await page.close()
asyncio.run(main())
```
--------------------------------
### Create a Custom JS Action
Source: https://cascadinglabs.com/voidcrawl/guides/custom-js-actions
Subclass JsActionNode and use inline_js to define JavaScript that interacts with the page. Instance attributes are automatically serialized and accessible via the __params object in the JS context.
```python
from voidcrawl.actions import JsActionNode, inline_js
class AppendToOutput(JsActionNode):
"""Append text to an element."""
js = inline_js("""\
const el = document.querySelector(__params.selector);
el.innerHTML += '
' + __params.text + '
';
return el.children.length;
""")
def __init__(self, text: str, selector: str = "#output") -> None:
self.text = text
self.selector = selector
```
--------------------------------
### Running VoidCrawl within an Async Context
Source: https://cascadinglabs.com/voidcrawl/guides/async-native
Ensure VoidCrawl operations are initiated within an active event loop. Use `asyncio.run(main())` as the entry point if you don't have an existing loop.
```python
# Wrong -- no event loop
from voidcrawl import BrowserPool, PoolConfig
pool = BrowserPool(PoolConfig())
# Right -- use asyncio.run()
import asyncio
asyncio.run(main())
```
--------------------------------
### Use BrowserPool for efficient scraping
Source: https://cascadinglabs.com/voidcrawl/quickstart
Utilize a pool of pre-opened tabs for near-instant page loads. The context manager handles automatic release of tabs back to the pool.
```python
import asyncio
from voidcrawl import BrowserPool, PoolConfig
async def main():
async with BrowserPool(PoolConfig()) as pool:
async with pool.acquire() as tab:
await tab.navigate("https://example.com")
print(await tab.title()) # "Example Domain"
print(len(await tab.content()))
asyncio.run(main())
```
--------------------------------
### BrowserPool Configuration for Docker Integration
Source: https://cascadinglabs.com/voidcrawl/guides/browser-pool
Configure the BrowserPool to connect to existing Chrome instances via WebSocket URLs, typically used in Dockerized environments. This bypasses local Chrome launching.
```python
config = PoolConfig(
chrome_ws_urls=["http://localhost:9222", "http://localhost:9223"],
tabs_per_browser=4,
)
```
--------------------------------
### CDP-Tier Input Actions
Source: https://cascadinglabs.com/voidcrawl/guides/builtin-actions
Dispatch input events directly via Chrome DevTools Protocol. Use CdpClick for mouse clicks and CdpTypeText for typing. Ensure the tab object is available.
```python
from voidcrawl.actions import CdpClick, CdpTypeText
await CdpClick(100.0, 200.0).run(tab)
await CdpTypeText("hello world").run(tab)
```