### Quickstart Examples Set 1

Source: https://docs.crawl4ai.com/core/examples

Basic examples for getting started with Crawl4AI, covering fundamental crawling and content handling.

```javascript
// Basic crawling example
const { AsyncWebCrawler } = require('crawl4ai');

const crawler = new AsyncWebCrawler();

async function run() {
    await crawler.run('http://example.com');
}
```

--------------------------------

### Quickstart Examples Set 2

Source: https://docs.crawl4ai.com/core/examples

More advanced examples for working with Crawl4AI, including content cleaning, link analysis, and JavaScript execution.

```javascript
// Example with content cleaning and link analysis
const { AsyncWebCrawler } = require('crawl4ai');

const crawler = new AsyncWebCrawler({
    // Options for content cleaning and link analysis
});

async function run() {
    await crawler.run('http://example.com');
}
```

--------------------------------

### Run Example Script

Source: https://docs.crawl4ai.com/core/examples

Execute an example script from the Crawl4AI documentation. Ensure Crawl4AI is installed first.

```bash
python -m docs.examples.hello_world
Copy
```

--------------------------------

### Setup LLM Environment Variables

Source: https://docs.crawl4ai.com/core/self-hosting

Copy the example environment file and add your LLM API keys. This file should be in the project root directory.

```bash
# Make sure you are in the 'crawl4ai' root directory
cp deploy/docker/.llm.env.example .llm.env

# Now edit .llm.env and add your API keys
```

--------------------------------

### Complete Example with Function Hooks

Source: https://docs.crawl4ai.com/core/self-hosting

Demonstrates how to use Python functions as hooks for environment setup and content extraction when crawling with the Crawl4aiDockerClient. This example configures browser settings, blocks resources, adds headers, scrolls the page, and extracts metadata.

```python
from crawl4ai import Crawl4aiDockerClient, BrowserConfig, CrawlerRunConfig, CacheMode

# Define hooks as regular Python functions
async def setup_environment(page, context, **kwargs):
    """Setup crawling environment"""
    # Set viewport
    await page.set_viewport_size({"width": 1920, "height": 1080})

    # Block resources for speed
    await context.route("**/*.{png,jpg,gif}", lambda r: r.abort())

    # Add custom headers
    await page.set_extra_http_headers({
        "Accept-Language": "en-US",
        "X-Custom-Header": "Crawl4AI"
    })

    print("[HOOK] Environment configured")
    return page

async def extract_content(page, context, **kwargs):
    """Extract and prepare content"""
    # Scroll to load lazy content
    await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
    await page.wait_for_timeout(1000)

    # Extract metadata
    metadata = await page.evaluate('''() => ({
        title: document.title,
        links: document.links.length,
        images: document.images.length
    })
''')

    print(f"[HOOK] Page metadata: {metadata}")
    return page

async def main():
    async with Crawl4aiDockerClient(base_url="http://localhost:11235", verbose=True) as client:
        # Configure crawl
        browser_config = BrowserConfig(headless=True)
        crawler_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)

        # Crawl with hooks
        result = await client.crawl(
            ["https://httpbin.org/html"],
            browser_config=browser_config,
            crawler_config=crawler_config,
            hooks={
                "on_page_context_created": setup_environment,
                "before_retrieve_html": extract_content
            },
            hooks_timeout=30
        )

        if result.success:
            print(f"✅ Crawl successful!")
            print(f"   URL: {result.url}")
            print(f"   HTML: {len(result.html)} chars")
            print(f"   Markdown: {len(result.markdown)} chars")
        else:
            print(f"❌ Crawl failed: {result.error_message}")

if __name__ == "__main__":
    import asyncio
    asyncio.run(main())

```

--------------------------------

### Minimal Example of AsyncWebCrawler Usage

Source: https://docs.crawl4ai.com/core/browser-crawler-config

Shows a basic setup for using `AsyncWebCrawler` with a custom `BrowserConfig`. This example demonstrates how to instantiate `BrowserConfig` with specific browser settings and then use the crawler to fetch and print a portion of a webpage's content.

```python
from crawl4ai import AsyncWebCrawler, BrowserConfig

browser_conf = BrowserConfig(
    browser_type="firefox",
    headless=False,
    text_mode=True
)

async with AsyncWebCrawler(config=browser_conf) as crawler:
    result = await crawler.arun("https://example.com")
    print(result.markdown[:300])
```

--------------------------------

### Install and Run Tutorial Server

Source: https://docs.crawl4ai.com/core/c4a-script

Commands to clone the tutorial repository, install its Python dependencies, and launch the local server.

```bash
# Clone and navigate to the tutorial
cd docs/examples/c4a_script/tutorial/

# Install dependencies
pip install -r requirements.txt

# Launch the tutorial server
python server.py

# Open http://localhost:8000 in your browser
```

--------------------------------

### Quick Start with Undetected Browser Adapter

Source: https://docs.crawl4ai.com/advanced/undetected-browser

This example demonstrates how to set up and use the `UndetectedAdapter` with `AsyncPlaywrightCrawlerStrategy` for advanced bot detection evasion. Ensure `headless` is set to False for better stealth.

```python
import asyncio
from crawl4ai import (
    AsyncWebCrawler, 
    BrowserConfig, 
    CrawlerRunConfig,
    UndetectedAdapter
)
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy

async def main():
    # Create the undetected adapter
    undetected_adapter = UndetectedAdapter()

    # Create browser config
    browser_config = BrowserConfig(
        headless=False,  # Headless mode can be detected easier
        verbose=True,
    )

    # Create the crawler strategy with undetected adapter
    crawler_strategy = AsyncPlaywrightCrawlerStrategy(
        browser_config=browser_config,
        browser_adapter=undetected_adapter
    )

    # Create the crawler with our custom strategy
    async with AsyncWebCrawler(
        crawler_strategy=crawler_strategy,
        config=browser_config
    ) as crawler:
        # Your crawling code here
        result = await crawler.arun(
            url="https://example.com",
            config=CrawlerRunConfig()
        )
        print(result.markdown[:500])

asyncio.run(main())

```

--------------------------------

### Basic Crawl Setup

Source: https://docs.crawl4ai.com/core/simple-crawling

Set up a simple crawl using default BrowserConfig and CrawlerRunConfig. This is the starting point for most crawling tasks.

```python
import asyncio
from crawl4ai import AsyncWebCrawler
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig

async def main():
    browser_config = BrowserConfig()  # Default browser configuration
    run_config = CrawlerRunConfig()   # Default crawl run configuration

    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(
            url="https://example.com",
            config=run_config
        )
        print(result.markdown)  # Print clean markdown content

if __name__ == "__main__":
    asyncio.run(main())
```

--------------------------------

### Hello World Example

Source: https://docs.crawl4ai.com/core/examples

A simple introductory example demonstrating basic usage of AsyncWebCrawler with JavaScript execution and content filtering.

```javascript
const { AsyncWebCrawler } = require('crawl4ai');

const crawler = new AsyncWebCrawler({
    // Configuration options
});

async function run() {
    await crawler.run('http://example.com');
}

run();
```

--------------------------------

### Install Crawl4AI with All Features

Source: https://docs.crawl4ai.com/core/installation

Installs all available features of Crawl4AI. Requires running crawl4ai-setup afterwards.

```bash
pip install crawl4ai[all]
crawl4ai-setup
```

--------------------------------

### Install Crawl4AI

Source: https://docs.crawl4ai.com/core/examples

Install the Crawl4AI library using pip. This is a prerequisite for running most examples.

```bash
pip install crawl4ai
Copy
```

--------------------------------

### Complete AdaptiveCrawler Example

Source: https://docs.crawl4ai.com/api/adaptive-crawler

Demonstrates a full asynchronous crawling process using AdaptiveCrawler. It includes configuration, starting the crawl, retrieving results, and exporting knowledge.

```python
import asyncio
from crawl4ai import AsyncWebCrawler, AdaptiveCrawler, AdaptiveConfig

async def main():
    # Configure adaptive crawling
    config = AdaptiveConfig(
        confidence_threshold=0.75,
        max_pages=15,
        save_state=True,
        state_path="my_crawl.json"
    )

    async with AsyncWebCrawler() as crawler:
        adaptive = AdaptiveCrawler(crawler, config)

        # Start crawling
        state = await adaptive.digest(
            start_url="https://example.com/docs",
            query="authentication oauth2 jwt"
        )

        # Check results
        print(f"Confidence achieved: {adaptive.confidence:.0%}")
        adaptive.print_stats()

        # Get most relevant pages
        for page in adaptive.get_relevant_content(top_k=3):
            print(f"- {page['url']} (score: {page['score']:.2f})")

        # Export for later use
        adaptive.export_knowledge_base("auth_knowledge.jsonl")

if __name__ == "__main__":
    asyncio.run(main())
```

--------------------------------

### Complete Example: Safe Multi-Hook Crawling with httpbin.org

Source: https://docs.crawl4ai.com/core/self-hosting

Demonstrates a comprehensive setup for safe crawling using multiple hooks. It configures hooks for page context creation, before navigation, and before HTML retrieval, utilizing httpbin.org for testing and blocking specific resources.

```python
import requests
import json
import os

# Safe example using httpbin.org for testing
hooks_code = {
    "on_page_context_created": """
async def hook(page, context, **kwargs):
    # Set viewport and test cookies
    await page.set_viewport_size({"width": 1920, "height": 1080})
    await context.add_cookies([
        {"name": "test_cookie", "value": "test_value", "domain": ".httpbin.org", "path": "/"}
    ])

    # Block unnecessary resources for httpbin
    await context.route("**/*.{png,jpg,jpeg}", lambda route: route.abort())
    return page
""",

    "before_goto": """
async def hook(page, context, url, **kwargs):
    # Add custom headers for testing
    await page.set_extra_http_headers({
        "X-Test-Header": "crawl4ai-test",
        "Accept-Language": "en-US,en;q=0.9"
    })
    print(f"[HOOK] Navigating to: {url}")
    return page
""",

    "before_retrieve_html": """
async def hook(page, context, **kwargs):
    # Simple scroll for any lazy-loaded content
    await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
    await page.wait_for_timeout(1000)
    return page
"
}

# Make the request to safe testing endpoints
response = requests.post("http://localhost:11235/crawl", json={
    "urls": [
        "https://httpbin.org/html",
        "https://httpbin.org/json"
    ],
    "hooks": {
        "code": hooks_code,
        "timeout": 30
    },
    "crawler_config": {
        "cache_mode": "bypass"
    }
})

# Check results
if response.status_code == 200:
    data = response.json()

    # Check hook execution
    if data['hooks']['status']['status'] == 'success':
        print(f"✅ All {len(data['hooks']['status']['attached_hooks'])} hooks executed successfully")
        print(f"Execution stats: {data['hooks']['summary']}")

    # Process crawl results
    for result in data['results']:
        print(f"Crawled: {result['url']} - Success: {result['success']}")
else:
    print(f"Error: {response.status_code}")
```

--------------------------------

### Run Virtual Scroll Examples

Source: https://docs.crawl4ai.com/advanced/virtual-scroll

Navigate to the examples directory and execute the Python script to run the virtual scrolling demonstrations.

```bash
cd docs/examples
python virtual_scroll_example.py
```

--------------------------------

### Basic AsyncWebCrawler Setup

Source: https://docs.crawl4ai.com/assets/llm.txt/txt/simple_crawling.txt

Demonstrates the minimal setup required to initialize AsyncWebCrawler and perform a basic crawl. Imports necessary modules and uses default configurations for browser and crawl settings.

```python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

async def main():
    browser_config = BrowserConfig()  # Default browser settings
    run_config = CrawlerRunConfig()   # Default crawl settings

    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(
            url="https://example.com",
            config=run_config
        )
        print(result.markdown)

if __name__ == "__main__":
    asyncio.run(main())
```

--------------------------------

### Quick SeedingConfig Examples

Source: https://docs.crawl4ai.com/assets/llm.txt/txt/url_seeder.txt

Provides examples of creating quick SeedingConfig instances for common use cases like blog posts, API documentation, and product pages.

```python
# Quick configurations for common use cases
blog_config = SeedingConfig(
    source="sitemap",
    pattern="*/blog/*", 
    extract_head=True
)

api_docs_config = SeedingConfig(
    source="sitemap+cc",
    pattern="*/docs/*",
    query="API reference documentation",
    scoring_method="bm25",
    score_threshold=0.5
)

product_pages_config = SeedingConfig(
    source="cc",
    pattern="*/product/*",
    live_check=True,
    max_urls=500
)
```

--------------------------------

### Programmatic Profile Creation Example

Source: https://docs.crawl4ai.com/assets/llm.txt/txt/cli.txt

Example of creating browser profiles programmatically using Python. This allows for automated setup and management of browser sessions.

```python
from crawl4ai import BrowserProfile

profile = BrowserProfile(name="my-programmatic-profile")
profile.create()

# You can then log in to websites within this profile
# For example, by launching the browser associated with the profile:
# profile.launch_browser()
# Or by using it in a crawl:
# crwl("https://example.com", profile="my-programmatic-profile")
```

--------------------------------

### Basic Adaptive Crawling Usage

Source: https://docs.crawl4ai.com/core/adaptive-crawling

Demonstrates the basic setup and execution of an adaptive crawler. It initializes the crawler, starts crawling with a specific query, prints statistics, and retrieves the most relevant content.

```python
from crawl4ai import AsyncWebCrawler, AdaptiveCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        # Create an adaptive crawler (config is optional)
        adaptive = AdaptiveCrawler(crawler)

        # Start crawling with a query
        result = await adaptive.digest(
            start_url="https://docs.python.org/3/",
            query="async context managers"
        )

        # View statistics
        adaptive.print_stats()

        # Get the most relevant content
        relevant_pages = adaptive.get_relevant_content(top_k=5)
        for page in relevant_pages:
            print(f"- {page['url']} (score: {page['score']:.2f})")
```

--------------------------------

### Example: Export Knowledge Base

Source: https://docs.crawl4ai.com/api/adaptive-crawler

An example demonstrating how to export the knowledge base to a file named 'my_knowledge.jsonl'.

```python
adaptive.export_knowledge_base("my_knowledge.jsonl")
Copy

```

--------------------------------

### Full Browser Crawler Configuration Example

Source: https://docs.crawl4ai.com/core/browser-crawler-config

This snippet demonstrates a complete setup for an asynchronous web crawler. It includes browser configuration, a JSON CSS extraction strategy, LLM content filtering with Gemini, and a crawler run configuration with cache bypass. Use this as a template for complex crawling tasks.

```python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig, LLMContentFilter, DefaultMarkdownGenerator
from crawl4ai import JsonCssExtractionStrategy

async def main():
    # 1) Browser config: headless, bigger viewport, no proxy
    browser_conf = BrowserConfig(
        headless=True,
        viewport_width=1280,
        viewport_height=720
    )

    # 2) Example extraction strategy
    schema = {
        "name": "Articles",
        "baseSelector": "div.article",
        "fields": [
            {"name": "title", "selector": "h2", "type": "text"},
            {"name": "link", "selector": "a", "type": "attribute", "attribute": "href"}
        ]
    }
    extraction = JsonCssExtractionStrategy(schema)

    # 3) Example LLM content filtering

    gemini_config = LLMConfig(
        provider="gemini/gemini-1.5-pro", 
        api_token = "env:GEMINI_API_TOKEN"
    )

    # Initialize LLM filter with specific instruction
    filter = LLMContentFilter(
        llm_config=gemini_config,  # or your preferred provider
        instruction="""
        Focus on extracting the core educational content.
        Include:
        - Key concepts and explanations
        - Important code examples
        - Essential technical details
        Exclude:
        - Navigation elements
        - Sidebars
        - Footer content
        Format the output as clean markdown with proper code blocks and headers.
        """,
        chunk_token_threshold=500,  # Adjust based on your needs
        verbose=True
    )

    md_generator = DefaultMarkdownGenerator(
        content_filter=filter,
        options={"ignore_links": True}
    )

    # 4) Crawler run config: skip cache, use extraction
    run_conf = CrawlerRunConfig(
        markdown_generator=md_generator,
        extraction_strategy=extraction,
        cache_mode=CacheMode.BYPASS,
    )

    async with AsyncWebCrawler(config=browser_conf) as crawler:
        # 4) Execute the crawl
        result = await crawler.arun(url="https://example.com/news", config=run_conf)

        if result.success:
            print("Extracted content:", result.extracted_content)
        else:
            print("Error:", result.error_message)

if __name__ == "__main__":
    asyncio.run(main())

```

--------------------------------

### Host Discovery Example

Source: https://docs.crawl4ai.com/core/domain-mapping

Illustrates the host discovery process, showing subdomains and the sources used for discovery.

```text
superdesign.dev
├── crt.sh           → docs, app, cloud, insights, staging-api, ui2web, ...
├── Wayback CDX      → api, app, docs, www, ...
├── Common Crawl     → app, www, ...
└── DNS guessing     → www, app, api, docs, blog, admin, cloud, ...

Result: 13 validated hosts
```

--------------------------------

### Build Docker Image with Specific Features

Source: https://docs.crawl4ai.com/core/self-hosting

Use this command to build the Docker image with specific features enabled via build arguments. This example builds for multiple platforms and installs all features.

```bash
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --build-arg INSTALL_TYPE=all \
  -t yourname/crawl4ai-all:latest \
  --load \
  . # Build from root context
```

--------------------------------

### Docker Configuration Example

Source: https://docs.crawl4ai.com/core/examples

Demonstrates the creation and utilization of Docker configuration objects within Crawl4AI.

```python
# Example code for Docker Config would go here, likely involving Docker SDK or configuration files.
```

--------------------------------

### Initialize CrawlerMonitor

Source: https://docs.crawl4ai.com/advanced/multi-url-crawling

Example of initializing a CrawlerMonitor with specified display settings for real-time crawling visibility.

```python
from crawl4ai import CrawlerMonitor, DisplayMode
monitor = CrawlerMonitor(
    # Maximum rows in live display
    max_visible_rows=15,          

    # DETAILED or AGGREGATED view
    display_mode=DisplayMode.DETAILED  
)
```

--------------------------------

### Install Undetected Browser Dependencies

Source: https://docs.crawl4ai.com/advanced/undetected-browser

Run this command to install all necessary browser dependencies for both regular and undetected modes.

```bash
crawl4ai-setup
```

--------------------------------

### Display Crawl4AI CLI Usage Examples

Source: https://docs.crawl4ai.com/core/cli

View available usage examples for the Crawl4AI CLI to understand its capabilities.

```bash
crwl --example
```

--------------------------------

### Initialize CrawlerRunConfig

Source: https://docs.crawl4ai.com/api/parameters

Example of initializing CrawlerRunConfig with various content processing and streaming options.

```python
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

run_cfg = CrawlerRunConfig(
    wait_for="css:.main-content",
    word_count_threshold=15,
    excluded_tags=["nav", "footer"],
    exclude_external_links=True,
    stream=True,  # Enable streaming for arun_many()
)

```

--------------------------------

### Comprehensive Crawl Example

Source: https://docs.crawl4ai.com/assets/llm.txt/txt/simple_crawling.txt

A complete example demonstrating advanced configuration options including verbose logging, content filtering, iframe processing, cache control, and detailed result processing for media and links.

```python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

async def comprehensive_crawl():
    browser_config = BrowserConfig(verbose=True)
    
    run_config = CrawlerRunConfig(
        # Content filtering
        word_count_threshold=10,
        excluded_tags=['form', 'header', 'nav'],
        exclude_external_links=True,
        
        # Content processing
        process_iframes=True,
        remove_overlay_elements=True,
        
        # Cache control
        cache_mode=CacheMode.ENABLED
    )

    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(
            url="https://example.com",
            config=run_config
        )
        
        if result.success:
            # Display content summary
            print(f"Title: {result.metadata.get('title', 'No title')}")
            print(f"Content: {result.markdown[:500]}...")
            
            # Process media
            images = result.media.get("images", [])
            print(f"Found {len(images)} images")
            for img in images[:3]:  # First 3 images
                print(f"  - {img.get('src', 'No src')}")
            
            # Process links
            internal_links = result.links.get("internal", [])
            print(f"Found {len(internal_links)} internal links")
            for link in internal_links[:3]:  # First 3 links
                print(f"  - {link.get('href', 'No href')}")
                
        else:
            print(f"❌ Crawl failed: {result.error_message}")
            print(f"Status: {result.status_code}")

if __name__ == "__main__":
    asyncio.run(comprehensive_crawl())
```

--------------------------------

### Complete Example: Cloud Platform Job Cancellation

Source: https://docs.crawl4ai.com/core/deep-crawling

This example demonstrates a full integration of cancellable deep crawls with an external job management system using Redis for status and state tracking. It includes checking for cancellation, saving progress, and reporting final status.

```python
import asyncio
import json
import redis.asyncio as redis
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

async def run_cancellable_crawl(job_id: str, start_url: str):
    redis_client = redis.Redis(host='localhost', port=6379, db=0)

    # Check external cancellation source
    async def check_cancelled():
        status = await redis_client.get(f"job:{job_id}:status")
        return status == b"cancelled"

    # Save progress for monitoring and recovery
    async def save_progress(state: dict):
        await redis_client.set(
            f"job:{job_id}:state",
            json.dumps(state)
        )
        # Update job progress
        await redis_client.set(
            f"job:{job_id}:pages_crawled",
            state["pages_crawled"]
        )

    strategy = BFSDeepCrawlStrategy(
        max_depth=3,
        max_pages=500,
        should_cancel=check_cancelled,
        on_state_change=save_progress,
    )

    config = CrawlerRunConfig(
        deep_crawl_strategy=strategy,
        stream=True,
    )

    results = []
    try:
        async with AsyncWebCrawler() as crawler:
            async for result in await crawler.arun(start_url, config=config):
                results.append(result)
                print(f"Crawled: {result.url}")
    finally:
        # Report final status
        if strategy.cancelled:
            await redis_client.set(f"job:{job_id}:status", "cancelled")
            print(f"Job cancelled after {len(results)} pages")
        else:
            await redis_client.set(f"job:{job_id}:status", "completed")
            print(f"Job completed with {len(results)} pages")

        await redis_client.close()

    return results

# Usage
# asyncio.run(run_cancellable_crawl("job-123", "https://example.com"))
#
# To cancel from another process:
# redis_client.set("job:job-123:status", "cancelled")
```

--------------------------------

### Complete Example: Redis-Based Crash Recovery

Source: https://docs.crawl4ai.com/core/deep-crawling

A full implementation demonstrating how to use Redis for storing and resuming crawl states. This example includes checking for existing state, persisting progress, and handling potential crawl interruptions.

```python
import asyncio
import json
import redis.asyncio as redis
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

REDIS_KEY = "crawl4ai:crawl_state"

async def main():
    redis_client = redis.Redis(host='localhost', port=6379, db=0)

    # Check for existing state
    saved_state = None
    existing = await redis_client.get(REDIS_KEY)
    if existing:
        saved_state = json.loads(existing)
        print(f"Resuming from checkpoint: {saved_state['pages_crawled']} pages already crawled")

    # State persistence callback
    async def persist_state(state: dict):
        await redis_client.set(REDIS_KEY, json.dumps(state))

    # Create strategy with recovery support
    strategy = BFSDeepCrawlStrategy(
        max_depth=3,
        max_pages=100,
        resume_state=saved_state,
        on_state_change=persist_state,
    )

    config = CrawlerRunConfig(deep_crawl_strategy=strategy, stream=True)

    try:
        async with AsyncWebCrawler() as crawler:
            async for result in await crawler.arun("https://example.com", config=config):
                print(f"Crawled: {result.url}")
    except Exception as e:
        print(f"Crawl interrupted: {e}")
        print("State saved - restart to resume")
    finally:
        await redis_client.close()

if __name__ == "__main__":
    asyncio.run(main())
```

--------------------------------

### Install and Test Crawl4AI in Google Colab

Source: https://docs.crawl4ai.com/assets/llm.txt/txt/installation.txt

Instructions for installing Crawl4AI within a Google Colab environment, including basic installation, advanced features, and troubleshooting Playwright installations. Includes a quick test to verify successful setup.

```python
# Install in Colab
!pip install crawl4ai
!crawl4ai-setup

# If setup fails, manually install Playwright browsers
!playwright install chromium

# Install with all features (may take 5-10 minutes)
!pip install crawl4ai[all]
!crawl4ai-setup
!crawl4ai-download-models

# If still having issues, force Playwright install
!playwright install chromium --force

# Quick test
import asyncio
from crawl4ai import AsyncWebCrawler

async def test_crawl():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://example.com")
        print("✅ Installation successful!")
        print(f"Content length: {len(result.markdown)}")

# Run test in Colab
await test_crawl()
```

--------------------------------

### Verify Installation with a Simple Crawl

Source: https://docs.crawl4ai.com/core/installation

A minimal Python script to demonstrate a basic crawl using AsyncWebCrawler, BrowserConfig, and CrawlerRunConfig. It loads example.com and prints the first 300 characters of markdown.

```python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://www.example.com",
        )
        print(result.markdown[:300])  # Show the first 300 characters of extracted text

if __name__ == "__main__":
    asyncio.run(main())
```

--------------------------------

### Integrate with Chainlit

Source: https://docs.crawl4ai.com/core/examples

Provides a guide on integrating Crawl4AI with the Chainlit framework for building conversational AI applications. Requires Chainlit installation.

```python
# Example code for Chainlit Integration would go here, showing how to connect Crawl4AI outputs to Chainlit UI.
```

--------------------------------

### AdaptiveCrawler.digest()

Source: https://docs.crawl4ai.com/api/adaptive-crawler

The primary method to perform adaptive crawling starting from a URL with a specific query, guiding the process based on coverage, consistency, and saturation.

```APIDOC
## AdaptiveCrawler.digest()

### Description
The main method that performs adaptive crawling starting from a URL with a specific query.

### Method
`async def digest(
    start_url: str,
    query: str,
    resume_from: Optional[Union[str, Path]] = None
) -> CrawlState`

### Parameters
- **start_url** (`str`): The starting URL for crawling.
- **query** (`str`): The search query that guides the crawling process.
- **resume_from** (`Optional[Union[str, Path]]`): Path to a saved state file to resume from.

### Returns
- **CrawlState**: The final crawl state containing all crawled URLs, knowledge base, and metrics.

### Example
```python
async with AsyncWebCrawler() as crawler:
    adaptive = AdaptiveCrawler(crawler)
    state = await adaptive.digest(
        start_url="https://docs.python.org",
        query="async context managers"
    )
```
```

--------------------------------

### Run Crawl4AI Container (Basic)

Source: https://docs.crawl4ai.com/core/self-hosting

Starts the Crawl4AI Docker container in detached mode, mapping port 11235. This is a basic setup without LLM support.

```bash
docker run -d \
  -p 11235:11235 \
  --name crawl4ai \
  --shm-size=1g \
  unclecode/crawl4ai:latest
```

--------------------------------

### Download Multiple Files Example

Source: https://docs.crawl4ai.com/advanced/file-downloading

Demonstrates downloading multiple files by iterating through download links and clicking them with JavaScript, with delays between clicks. Includes configuration for download path and waiting time.

```python
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig
import os
from pathlib import Path

async def download_multiple_files(url: str, download_path: str):
    config = BrowserConfig(accept_downloads=True, downloads_path=download_path)
    async with AsyncWebCrawler(config=config) as crawler:
        run_config = CrawlerRunConfig(
            js_code="""
                const downloadLinks = document.querySelectorAll('a[download]');
                for (const link of downloadLinks) {
                    link.click();
                    // Delay between clicks
                    await new Promise(r => setTimeout(r, 2000));  
                }
            """,
            wait_for=10  # Wait for all downloads to start
        )
        result = await crawler.arun(url=url, config=run_config)

        if result.downloaded_files:
            print("Downloaded files:")
            for file in result.downloaded_files:
                print(f"- {file}")
        else:
            print("No files downloaded.")

# Usage
download_path = os.path.join(Path.home(), ".crawl4ai", "downloads")
os.makedirs(download_path, exist_ok=True)

asyncio.run(download_multiple_files("https://www.python.org/downloads/windows/", download_path))
```

--------------------------------

### digest()

Source: https://docs.crawl4ai.com/api/digest

The primary interface for adaptive web crawling. It intelligently crawls websites starting from a given URL, guided by a query, and automatically determines when sufficient information has been gathered.

```APIDOC
## digest()

### Description
The `digest()` method is the primary interface for adaptive web crawling. It intelligently crawls websites starting from a given URL, guided by a query, and automatically determines when sufficient information has been gathered.

### Method Signature
```python
async def digest(
    start_url: str,
    query: str,
    resume_from: Optional[Union[str, Path]] = None
) -> CrawlState
```

### Parameters
#### start_url
* **Type** : `str`
* **Required** : Yes
* **Description** : The starting URL for the crawl. This should be a valid HTTP/HTTPS URL that serves as the entry point for information gathering.

#### query
* **Type** : `str`
* **Required** : Yes
* **Description** : The search query that guides the crawling process. This should contain key terms related to the information you're seeking. The crawler uses this to evaluate relevance and determine which links to follow.

#### resume_from
* **Type** : `Optional[Union[str, Path]]`
* **Default** : `None`
* **Description** : Path to a previously saved crawl state file. When provided, the crawler resumes from the saved state instead of starting fresh.

### Return Value
Returns a `CrawlState` object containing:
* **crawled_urls** (`Set[str]`): All URLs that have been crawled
* **knowledge_base** (`List[CrawlResult]`): Collection of crawled pages with content
* **pending_links** (`List[Link]`): Links discovered but not yet crawled
* **metrics** (`Dict[str, float]`): Performance and quality metrics
* **query** (`str`): The original query
* Additional statistical information for scoring
```

--------------------------------

### Authentication Workflow Examples

Source: https://docs.crawl4ai.com/assets/llm.txt/txt/cli.txt

Demonstrates end-to-end workflows for authenticated crawling, including profile creation, login, and data extraction for sites like LinkedIn, GitHub, and Twitter/X.

```bash
# Complete workflow for LinkedIn scraping
# 1. Create authenticated profile
crwl profiles
# Select "Create new profile" → login to LinkedIn in browser → press 'q' to save

# 2. Use profile for crawling
crwl https://linkedin.com/in/someone -p linkedin-profile -o markdown

# 3. Extract structured data with authentication
crwl https://linkedin.com/search/results/people/ \
    -p linkedin-profile \
    -j "Extract people profiles with names, titles, and companies" \
    -b "headless=false,viewport_width=1920"
```

```bash
# GitHub authenticated crawling
crwl profiles  # Create github-profile
crwl https://github.com/settings/profile -p github-profile
```

```bash
# Twitter/X authenticated access
crwl profiles  # Create twitter-profile  
crwl https://twitter.com/home -p twitter-profile -o markdown
```

--------------------------------

### Full Async Web Crawler Example

Source: https://docs.crawl4ai.com/api/async-webcrawler

Demonstrates setting up BrowserConfig and CrawlerRunConfig for a single URL crawl with CSS extraction. It shows how to initialize the crawler, execute a crawl, and process the results, including extracted content.

```python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai import JsonCssExtractionStrategy
import json

async def main():
    # 1.  Browser config
    browser_cfg = BrowserConfig(
        browser_type="firefox",
        headless=False,
        verbose=True
    )

    # 2.  Run config
    schema = {
        "name": "Articles",
        "baseSelector": "article.post",
        "fields": [
            {
                "name": "title", 
                "selector": "h2", 
                "type": "text"
            },
            {
                "name": "url", 
                "selector": "a", 
                "type": "attribute", 
                "attribute": "href"
            }
        ]
    }

    run_cfg = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        extraction_strategy=JsonCssExtractionStrategy(schema),
        word_count_threshold=15,
        remove_overlay_elements=True,
        wait_for="css:.post"  # Wait for posts to appear
    )

    async with AsyncWebCrawler(config=browser_cfg) as crawler:
        result = await crawler.arun(
            url="https://example.com/blog",
            config=run_cfg
        )

        if result.success:
            print("Cleaned HTML length:", len(result.cleaned_html))
            if result.extracted_content:
                articles = json.loads(result.extracted_content)
                print("Extracted articles:", articles[:2])
        else:
            print("Error:", result.error_message)

asyncio.run(main())
Copy
```

--------------------------------

### Implement Advanced Crawler Hooks

Source: https://docs.crawl4ai.com/assets/llm.txt/txt/config_objects.txt

Utilize hooks to customize crawler behavior at various stages, such as browser creation, page context setup, navigation, and HTML retrieval. This example demonstrates blocking images and setting viewport size.

```python
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig
from playwright.async_api import Page, BrowserContext

async def advanced_crawler_with_hooks():
    browser_config = BrowserConfig(headless=True, verbose=True)
    crawler = AsyncWebCrawler(config=browser_config)

    # Hook functions for different stages
    async def on_browser_created(browser, **kwargs):
        print("[HOOK] Browser created successfully")
        return browser

    async def on_page_context_created(page: Page, context: BrowserContext, **kwargs):
        print("[HOOK] Setting up page & context")
        
        # Block images for faster crawling
        async def route_filter(route):
            if route.request.resource_type == "image":
                await route.abort()
            else:
                await route.continue_()
        
        await context.route("**", route_filter)
        
        # Simulate login if needed
        # await page.goto("https://example.com/login")
        # await page.fill("input[name='username']", "testuser")
        # await page.fill("input[name='password']", "password123")
        # await page.click("button[type='submit']")
        
        await page.set_viewport_size({"width": 1080, "height": 600})
        return page

    async def before_goto(page: Page, context: BrowserContext, url: str, **kwargs):
        print(f"[HOOK] About to navigate to: {url}")
        await page.set_extra_http_headers({"Custom-Header": "my-value"})
        return page

    async def after_goto(page: Page, context: BrowserContext, url: str, response, **kwargs):
        print(f"[HOOK] Successfully loaded: {url}")
        try:
            await page.wait_for_selector('.content', timeout=1000)
            print("[HOOK] Content found!")
        except:
            print("[HOOK] Content not found, continuing")
        return page

    async def before_retrieve_html(page: Page, context: BrowserContext, **kwargs):
        print("[HOOK] Final actions before HTML retrieval")
        await page.evaluate("window.scrollTo(0, document.body.scrollHeight);")
        return page

    # Attach hooks
    crawler.crawler_strategy.set_hook("on_browser_created", on_browser_created)
    crawler.crawler_strategy.set_hook("on_page_context_created", on_page_context_created)
    crawler.crawler_strategy.set_hook("before_goto", before_goto)
    crawler.crawler_strategy.set_hook("after_goto", after_goto)
    crawler.crawler_strategy.set_hook("before_retrieve_html", before_retrieve_html)

    await crawler.start()
    
    config = CrawlerRunConfig()
    result = await crawler.arun("https://example.com", config=config)
    
    if result.success:
        print(f"Crawled successfully: {len(result.html)} chars")
    
    await crawler.close()
```

--------------------------------

### AsyncWebCrawler Configuration and Run

Source: https://docs.crawl4ai.com/api/parameters

Demonstrates how to configure and run the AsyncWebCrawler with detailed browser and run settings. Includes options for proxy, text mode, cache bypass, CSS selectors, excluded tags, waiting for elements, screenshots, and streaming.

```python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

async def main():
    # Configure the browser
    browser_cfg = BrowserConfig(
        headless=False,
        viewport_width=1280,
        viewport_height=720,
        proxy_config="http://user:pass@myproxy:8080",
        text_mode=True
    )

    # Configure the run
    run_cfg = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        session_id="my_session",
        css_selector="main.article",
        excluded_tags=["script", "style"],
        exclude_external_links=True,
        wait_for="css:.article-loaded",
        screenshot=True,
        stream=True
    )

    async with AsyncWebCrawler(config=browser_cfg) as crawler:
        result = await crawler.arun(
            url="https://example.com/news",
            config=run_cfg
        )
        if result.success:
            print("Final cleaned_html length:", len(result.cleaned_html))
            if result.screenshot:
                print("Screenshot captured (base64, length):", len(result.screenshot))
        else:
            print("Crawl failed:", result.error_message)

if __name__ == "__main__":
    asyncio.run(main())
```

--------------------------------

### Proxy Configuration Example

Source: https://docs.crawl4ai.com/core/browser-crawler-config

Define proxy server details including address, username, and password. Leave as None if no proxy is required.

```json
{
    "server": "http://proxy.example.com:8080", 
    "username": "...", 
    "password": "..."
}
```

--------------------------------

### Install Crawl4AI Core Library

Source: https://docs.crawl4ai.com/assets/llm.txt/txt/installation.txt

Installs the main crawl4ai library using pip. Run crawl4ai-setup to install Playwright browsers and crawl4ai-doctor to verify the installation.

```bash
# Install core library
pip install crawl4ai

# Initial setup (installs Playwright browsers)
crawl4ai-setup

# Verify installation
crawl4ai-doctor
```

--------------------------------

### Basic HTTP Crawler Setup

Source: https://docs.crawl4ai.com/assets/llm.txt/txt/http_based_crawler_strategy.txt

Initializes and runs a basic HTTP crawler using AsyncWebCrawler and AsyncHTTPCrawlerStrategy. Requires asyncio and crawl4ai libraries.

```python
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, HTTPCrawlerConfig, CacheMode
from crawl4ai.async_crawler_strategy import AsyncHTTPCrawlerStrategy
from crawl4ai.async_logger import AsyncLogger

async def main():
    # Initialize HTTP strategy
    http_strategy = AsyncHTTPCrawlerStrategy(
        browser_config=HTTPCrawlerConfig(
            method="GET",
            verify_ssl=True,
            follow_redirects=True
        ),
        logger=AsyncLogger(verbose=True)
    )

    # Use with AsyncWebCrawler
    async with AsyncWebCrawler(crawler_strategy=http_strategy) as crawler:
        result = await crawler.arun("https://example.com")
        print(f"Status: {result.status_code}")
        print(f"Content: {len(result.html)} chars")

if __name__ == "__main__":
    asyncio.run(main())
```

--------------------------------

### Install Crawl4AI with Advanced Features

Source: https://docs.crawl4ai.com/assets/llm.txt/txt/installation.txt

Installs optional features for Crawl4AI, including PyTorch-based functionalities (text clustering, semantic chunking) or Transformers (Hugging Face models). Run crawl4ai-setup after installation. crawl4ai-download-models can be used to pre-download models.

```bash
# PyTorch-based features (text clustering, semantic chunking)
pip install crawl4ai[torch]
crawl4ai-setup

# Transformers (Hugging Face models)
pip install crawl4ai[transformer]
crawl4ai-setup

# All features (large download)
pip install crawl4ai[all]
crawl4ai-setup

# Pre-download models (optional)
crawl4ai-download-models
```

--------------------------------

### Practical Crawl4AI Examples

Source: https://docs.crawl4ai.com/blog/articles/llm-context-revolution

This snippet demonstrates practical usage of Crawl4AI, showcasing JavaScript execution, CSS selector-based data extraction, and session-based crawling with custom headers. These examples represent the 'Examples' pillar of the LLM Context Protocol.

```python
# Crawling with JavaScript execution
result = await crawler.arun(
    url="https://example.com",
    js_code="window.scrollTo(0, document.body.scrollHeight);",
    wait_for="css:.lazy-loaded-content"
)

# Extracting structured data with CSS selectors
result = await crawler.arun(
    url="https://shop.example.com",
    extraction_strategy=CSSExtractionStrategy({
        "prices": "span.price::text",
        "titles": "h2.product-title::text"
    })
)

# Session-based crawling with custom headers
async with crawler:
    result1 = await crawler.arun(url1, session_id="product_scan")
    result2 = await crawler.arun(url2, session_id="product_scan")
...
```

--------------------------------

### Basic Browser Configuration

Source: https://docs.crawl4ai.com/assets/llm.txt/txt/config_objects.txt

Sets up a basic browser environment using Chromium in headless mode with specified viewport dimensions and verbose logging.

```python
from crawl4ai import BrowserConfig, AsyncWebCrawler

# Basic browser configuration
browser_config = BrowserConfig(
    browser_type="chromium",  # "chromium", "firefox", "webkit"
    headless=True,           # False for visible browser (debugging)
    viewport_width=1280,
    viewport_height=720,
    verbose=True
)
```

--------------------------------

### Initialize and Use RateLimiter

Source: https://docs.crawl4ai.com/advanced/multi-url-crawling

Example of initializing a RateLimiter with custom settings for managing request pacing and retries.

```python
from crawl4ai import RateLimiter

# Create a RateLimiter with custom settings
rate_limiter = RateLimiter(
    base_delay=(2.0, 4.0),  # Random delay between 2-4 seconds
    max_delay=30.0,         # Cap delay at 30 seconds
    max_retries=5,          # Retry up to 5 times on rate-limiting errors
    rate_limit_codes=[429, 503]  # Handle these HTTP status codes
)

# RateLimiter will handle delays and retries internally
# No additional setup is required for its operation
```

--------------------------------

### Get Hook Information

Source: https://docs.crawl4ai.com/core/self-hosting

Retrieve information about available hook points and their signatures using the GET /hooks/info endpoint.

```HTTP
GET /hooks/info
```

```Shell
curl http://localhost:11235/hooks/info
```

--------------------------------

### Q&A with Authenticated Content Examples

Source: https://docs.crawl4ai.com/assets/llm.txt/txt/cli.txt

Ask questions about authenticated content directly via the CLI. Supports single questions or multi-step Q&A workflows involving viewing content first.

```bash
# Ask questions about authenticated content
crwl https://private-dashboard.com -p dashboard-profile \
    -q "What are the key metrics shown in my dashboard?"
```

```bash
# Multiple questions workflow
crwl https://company-intranet.com -p work-profile -o markdown  # View content
crwl https://company-intranet.com -p work-profile \
    -q "Summarize this week's announcements"
crwl https://company-intranet.com -p work-profile \
    -q "What are the upcoming deadlines?"
```

--------------------------------

### Authenticate with Basic Auth (httpbin.org)

Source: https://docs.crawl4ai.com/core/self-hosting

This example demonstrates basic authentication using httpbin.org for testing purposes. It encodes credentials in base64 for the Authorization header.

```python
# Safe testing with httpbin.org (a service designed for HTTP testing)
hooks_code = {
    "before_goto": """
async def hook(page, context, url, **kwargs):
    import base64
    # httpbin.org/basic-auth expects username="user" and password="passwd"
    credentials = base64.b64encode(b"user:passwd").decode('ascii')

    await page.set_extra_http_headers({
        'Authorization': f'Basic {credentials}'
    })
    return page
"""
}

response = requests.post("http://localhost:11235/crawl", json={
    "urls": ["https://httpbin.org/basic-auth/user/passwd"],
    "hooks": {"code": hooks_code, "timeout": 15}
})
Copy

```

--------------------------------

### Built-in Browser Guide

Source: https://docs.crawl4ai.com/core/examples

A guide for using the built-in browser capabilities within Crawl4AI. This feature allows for browser automation.

```python
# Example code for Built-in Browser Guide would go here, demonstrating browser interactions.
```