### Docker Compose Setup (Bash)

Source: https://github.com/zleap-ai/sag/blob/main/README_en.md

Instructions for setting up and running the SAG project using Docker Compose. This includes cloning the repository, configuring environment variables, downloading necessary data, and starting the services.

```bash
# 1. Clone the project
git clone https://github.com/Zleap-AI/SAG.git
cd SAG

# 2. Configure environment
cp .env.example .env
# Edit .env:
#   LLM_API_KEY=sk-xxx
#   MYSQL_PASSWORD=your_password

# 3. download nltk_data (first only)
python scripts/download_nltk_data.py

# 4. Start services
docker compose up -d

# 5. Access
# Frontend: http://localhost
# API: http://localhost/api/docs
```

--------------------------------

### Install SAG Python SDK

Source: https://context7.com/zleap-ai/sag/llms.txt

Install the SAG Python package for programmatic access to its features. Supports installation from PyPI (when available) or directly from source, including development dependencies.

```bash
# Install from PyPI (when available)
pip install sag

# Or install from source
git clone https://github.com/Zleap-AI/SAG.git
cd SAG
pip install -e .

# Install with development dependencies
pip install -e ".[dev]"

# Verify installation
python -c "import sag; print(f'SAG version: {sag.__version__}')"
```

--------------------------------

### Explainability Example (JSON)

Source: https://github.com/zleap-ai/sag/blob/main/README_en.md

Provides an example of the JSON output used for explaining search results. It details the event, its score, and the clues leading to it, categorized by search stage (recall, expand). This aids in understanding how results are generated.

```json
{
  "event": "Sparse Expert Layer Design of MoE Architecture",
  "score": 0.89,
  "clues": [
    {
      "stage": "recall",
      "from": "Query: Large Model Optimization",
      "to": "Entity: MoE Architecture",
      "confidence": 0.92
    },
    {
      "stage": "expand", 
      "from": "Event: MoE Architecture Advantages",
      "to": "Event: Sparse Expert Layer Design",
      "shared_entities": ["MoE Architecture", "Expert Layer"],
      "confidence": 0.85
    }
  ]
}
```

--------------------------------

### Deploy SAG with Docker Compose

Source: https://context7.com/zleap-ai/sag/llms.txt

Deploy the SAG application and its dependencies (API, web UI, database, search, cache) using Docker Compose. This simplifies the setup process for development and production environments.

```bash
# Start all services
docker compose up -d

# Check service status
docker compose ps

# View logs
docker compose logs -f sag-api

# Stop services
docker compose down

# Rebuild after code changes
docker compose up -d --build
```

--------------------------------

### Manage Tasks using cURL

Source: https://context7.com/zleap-ai/sag/llms.txt

Interact with the SAG API to list, retrieve, and get statistics for asynchronous tasks. This requires a running SAG API instance and uses cURL for HTTP requests.

```bash
# List all tasks
curl "http://localhost:8000/api/v1/tasks?status=running&page=1&page_size=20"

# Get task details
curl "http://localhost:8000/api/v1/tasks/task-550e8400-e29b-41d4-a716-446655440000"

# Get task statistics
curl "http://localhost:8000/api/v1/tasks/stats"
```

--------------------------------

### Source Configuration Management

Source: https://context7.com/zleap-ai/sag/llms.txt

Enables the creation and management of information sources, including defining custom entity types and metadata. Supports creating, listing, retrieving, updating, and deleting source configurations.

```bash
# Create source configuration
curl -X POST "http://localhost:8000/api/v1/sources" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "project-docs-001",
    "name": "Project Documentation",
    "description": "Technical documentation for the main project",
    "status": "active",
    "entity_types": [
      {
        "type": "component",
        "name": "Component",
        "description": "Software components",
        "weight": 1.4
      },
      {
        "type": "api_endpoint",
        "name": "API Endpoint",
        "description": "REST API endpoints",
        "weight": 1.3
      }
    ],
    "metadata": {
      "project": "main-app",
      "team": "backend-team"
    }
  }'
```

```bash
# List all sources
curl "http://localhost:8000/api/v1/sources?page=1&page_size=20"
```

```bash
# Get source details
curl "http://localhost:8000/api/v1/sources/project-docs-001"
```

```bash
# Update source
curl -X PATCH "http://localhost:8000/api/v1/sources/project-docs-001" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Updated Project Documentation",
    "status": "active"
  }'
```

```bash
# Delete source (cascades to all related data)
curl -X DELETE "http://localhost:8000/api/v1/sources/project-docs-001"
```

--------------------------------

### GET /api/v1/tasks/{task_id}

Source: https://context7.com/zleap-ai/sag/llms.txt

Retrieves the status and results of a previously submitted pipeline task.

```APIDOC
## GET /api/v1/tasks/{task_id}

### Description
Retrieves the status and results of a previously submitted pipeline task using its unique identifier.

### Method
GET

### Endpoint
/api/v1/tasks/{task_id}

### Parameters
#### Path Parameters
- **task_id** (string) - Required - The unique identifier of the task to check.

### Request Example
```bash
curl "http://localhost:8000/api/v1/tasks/task-550e8400-e29b-41d4-a716-446655440000"
```

### Response
#### Success Response (200)
- **success** (boolean) - Indicates if the operation was successful.
- **data** (object) - Contains the task details and results.
  - **task_id** (string) - The unique identifier of the task.
  - **status** (string) - The current status of the task (e.g., 'completed', 'running', 'failed').
  - **progress** (integer) - The completion progress of the task (0-100).
  - **result** (object) - The results of the pipeline task if completed.
    - **article_id** (string) - Identifier for the processed article.
    - **events** (array) - List of extracted events.
    - **search_results** (array) - Results from the search stage.

#### Response Example
```json
{
  "success": true,
  "data": {
    "task_id": "task-550e8400-e29b-41d4-a716-446655440000",
    "status": "completed",
    "progress": 100,
    "result": {
      "article_id": "article_123",
      "events": ["event_1", "event_2", "event_3"],
      "search_results": [...] 
    }
  }
}
```
```

--------------------------------

### POST /api/v1/pipeline/load

Source: https://context7.com/zleap-ai/sag/llms.txt

Initiates only the document loading stage of the pipeline.

```APIDOC
## POST /api/v1/pipeline/load

### Description
Initiates only the document loading stage of the pipeline. This allows for processing and uploading documents without proceeding to extraction or search.

### Method
POST

### Endpoint
/api/v1/pipeline/load

### Parameters
#### Query Parameters
- **source_config_id** (string) - Required - Identifier for the data source configuration.
- **path** (string) - Required - Path to the documents to load.
- **background** (string) - Optional - Background information for the loaded documents.

### Request Example
```bash
curl -X POST "http://localhost:8000/api/v1/pipeline/load?source_config_id=docs-001&path=./articles/&background=Tech%20articles" \
  -H "Content-Type: application/json"
```

### Response
#### Success Response (200)
- **success** (boolean) - Indicates if the operation was successful.
- **data** (object) - Contains information about the loaded documents.
  - **article_id** (string) - Identifier for the loaded article or document batch.
  - **sections** (array) - List of processed sections.
  - **stats** (object) - Statistics about the loading process.
    - **section_count** (integer) - Number of sections processed.
    - **chunk_count** (integer) - Number of chunks created.
    - **total_tokens** (integer) - Total tokens processed.
    - **avg_tokens_per_chunk** (integer) - Average tokens per chunk.
- **message** (string) - A message indicating the load stage completion.

#### Response Example
```json
{
  "success": true,
  "data": {
    "article_id": "article_789",
    "sections": ["section_1", "section_2", "section_3"],
    "stats": {
      "section_count": 15,
      "chunk_count": 8,
      "total_tokens": 12500,
      "avg_tokens_per_chunk": 1562
    }
  },
  "message": "Load stage completed"
}
```
```

--------------------------------

### Initialize SAG Engine

Source: https://context7.com/zleap-ai/sag/llms.txt

Initializes the core orchestration engine for SAG pipeline execution. Requires a source configuration ID to define pipeline stages, entity types, model preferences, and metadata.

```python
import asyncio
from sag import SAGEngine

async def main():
    # Initialize engine with source configuration ID
    engine = SAGEngine(source_config_id="my-project-001")

    # The engine will use this source_config_id for all pipeline stages
    # Source configs define entity types, model preferences, and metadata

    print(f"Engine initialized for source: {engine.source_config_id}")

asyncio.run(main())
```

--------------------------------

### Configure Environment Variables (.env)

Source: https://context7.com/zleap-ai/sag/llms.txt

Set up environment variables for database connections, AI models, and API server settings by creating and populating a .env file. This is crucial for configuring the SAG application.

```bash
# Copy example configuration
cp .env.example .env

# Edit .env file
cat > .env << 'EOF'
# === Database Configuration ===
MYSQL_HOST=localhost
MYSQL_PORT=3306
MYSQL_USER=sag_user
MYSQL_PASSWORD=your_secure_password
MYSQL_DATABASE=sag_db

# === Elasticsearch Configuration ===
ES_HOST=localhost
ES_PORT=9200
ES_USERNAME=elastic
ES_PASSWORD=elastic_password
ES_INDEX_PREFIX=sag

# === Redis Configuration ===
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_PASSWORD=redis_password
REDIS_DB=0

# === LLM Configuration ===
LLM_API_KEY=sk-your-openai-api-key
LLM_MODEL=gpt-4-turbo
LLM_BASE_URL=https://api.openai.com/v1
LLM_TEMPERATURE=0.3
LLM_MAX_TOKENS=4000
LLM_TIMEOUT=60
LLM_MAX_RETRIES=3

# === Embedding Configuration ===
EMBEDDING_API_KEY=sk-your-openai-api-key
EMBEDDING_MODEL_NAME=text-embedding-3-large
EMBEDDING_DIMENSIONS=3072
EMBEDDING_BASE_URL=https://api.openai.com/v1

# === API Server Configuration ===
API_HOST=0.0.0.0
API_PORT=8000
API_WORKERS=4
DEBUG=false
EOF

# Verify configuration
python -c "from sag.core.config.settings import get_settings; s = get_settings(); print(f'Config loaded: {s.mysql_host}:{s.mysql_port}')"
```

--------------------------------

### Python SDK for Document Processing and Retrieval

Source: https://github.com/zleap-ai/sag/blob/main/README_en.md

Demonstrates how to use the SAG Python SDK to load, extract information from, and search documents. It initializes the SAGEngine, processes documents from a specified path, extracts events in parallel, and performs intelligent retrieval based on natural language queries. Results are then printed with their scores and summaries.

```python
import asyncio
from sag import SAGEngine
from sag.modules.load.config import LoadBaseConfig
from sag.modules.extract.config import ExtractBaseConfig
from sag.modules.search.config import SearchBaseConfig

async def main():
    # Initialize
    engine = SAGEngine(source_config_id="my-project")
    
    # Load documents
    await engine.load(LoadBaseConfig(
        type="path",
        origin=["./docs/article.md"],
        background="Technical Documentation"
    ))
    
    # Extract events
    await engine.extract(ExtractBaseConfig(
        parallel=True,
        background="AI Large Model Documentation"
    ))
    
    # Intelligent retrieval
    result = await engine.search(SearchBaseConfig(
        query="How to optimize large model inference speed?",
        depth=2,
        top_k=10
    ))
    
    # View results
    for event in result.events:
        print(f"[{event.score:.2f}] {event.title}")
        print(f"  {event.summary}\n")

asyncio.run(main())
```

--------------------------------

### Configure Three-Stage Search Parameters in Python

Source: https://context7.com/zleap-ai/sag/llms.txt

This Python code configures and executes a three-stage search (recall, expand, rerank) using the SAGEngine. It demonstrates setting parameters for entity recall, multi-hop expansion, and result reranking using PageRank. The code takes a query, rewrites it, and then processes the results, printing statistics and knowledge graph connections.

```python
import asyncio
from sag import SAGEngine, SearchBaseConfig
from sag.modules.search.config import RecallConfig, ExpandConfig, RerankConfig, RerankStrategy

async def main():
    engine = SAGEngine(source_config_id="knowledge-base")

    # === Stage 1: Recall Configuration ===
    # Purpose: Find initial entities and events from query
    recall_config = RecallConfig(
        use_fast_mode=True,              # Skip LLM entity extraction, use query vector directly
        vector_top_k=15,                 # Return top 15 similar entities
        vector_candidates=20,            # Search pool size for vector search
        entity_similarity_threshold=0.4, # Minimum entity similarity score
        event_similarity_threshold=0.4,  # Minimum event similarity score
        max_entities=25,                 # Maximum entities to recall
        max_events=60,                   # Maximum events to recall
        entity_weight_threshold=0.05,    # Minimum entity weight to keep
        final_entity_count=15            # Final entity count after filtering
    )

    # === Stage 2: Expand Configuration ===
    # Purpose: Multi-hop exploration through shared entities
    expand_config = ExpandConfig(
        enabled=True,                    # Enable expansion stage
        max_hops=3,                      # Maximum hop depth (BFS)
        entities_per_hop=10,             # New entities per hop
        weight_change_threshold=0.1,     # Convergence threshold
        event_similarity_threshold=0.3,  # Filter events by similarity
        min_events_per_hop=5,            # Minimum events to continue
        max_events_per_hop=100           # Maximum events per hop
    )

    # === Stage 3: Rerank Configuration ===
    # Purpose: Rank results using PageRank or RRF
    rerank_config = RerankConfig(
        strategy=RerankStrategy.PAGERANK,    # Use PageRank algorithm
        score_threshold=0.5,                 # Minimum final score
        max_results=10,                      # Return top 10 results
        max_key_recall_results=30,           # Step1 key recall limit
        max_query_recall_results=30,         # Step2 query recall limit
        pagerank_damping_factor=0.85,        # PageRank damping factor (0-1)
        pagerank_max_iterations=100,         # PageRank max iterations
        rrf_k=60                             # RRF fusion parameter
    )

    # Execute search with custom configuration
    await engine.search_async(SearchBaseConfig(
        query="How do transformer models handle long sequences?",
        enable_query_rewrite=True,
        recall=recall_config,
        expand=expand_config,
        rerank=rerank_config
    ))

    result = engine.get_result()

    if result.search_result and result.search_result.status == "success":
        events = result.search_result.data_full
        stats = result.search_result.stats

        print(f"✅ Found {len(events)} results")
        print(f"📊 Algorithm stats:")
        print(f"  - Recall: {stats.get('recall_entity_count', 0)} entities")
        print(f"  - Expand: {stats.get('expand_hops', 0)} hops, {stats.get('expand_entity_count', 0)} entities")
        print(f"  - Rerank: {stats.get('rerank_strategy', 'N/A')} strategy")

        # Display knowledge graph connections
        clues = stats.get('clues', [])
        print(f"\n🔗 Knowledge graph: {len(clues)} connections")

        recall_clues = [c for c in clues if c['stage'] == 'recall']
        expand_clues = [c for c in clues if c['stage'] == 'expand']

        print(f"  - Recall connections: {len(recall_clues)}")
        for clue in recall_clues[:3]:
            print(f"    • {clue['from']} → {clue['to']} (confidence: {clue.get('confidence', 0):.2f})")

        print(f"  - Expand connections: {len(expand_clues)}")
        for clue in expand_clues[:3]:
            shared = clue.get('shared_entities', [])
            print(f"    • {clue['from']} → {clue['to']} (via: {', '.join(shared[:2])})")

asyncio.run(main())

```

--------------------------------

### Document Upload and Management

Source: https://context7.com/zleap-ai/sag/llms.txt

Facilitates the uploading of documents for processing and subsequent management. Supports uploading files, listing documents associated with a source, retrieving document details, and fetching associated events.

```bash
# Upload document
curl -X POST "http://localhost:8000/api/v1/documents/upload" \
  -F "file=@./technical_report.pdf" \
  -F "source_config_id=docs-001" \
  -F "background=Technical report on system architecture"
```

```bash
# List documents
curl "http://localhost:8000/api/v1/documents?source_config_id=docs-001&page=1&page_size=20"
```

```bash
# Get document details
curl "http://localhost:8000/api/v1/documents/doc_456"
```

```bash
# Get document events
curl "http://localhost:8000/api/v1/documents/doc_456/events?page=1&page_size=50"
```

--------------------------------

### POST /api/v1/pipeline/run

Source: https://context7.com/zleap-ai/sag/llms.txt

Submits a pipeline task for asynchronous processing. This endpoint initiates a multi-stage process including loading, extracting, and searching data.

```APIDOC
## POST /api/v1/pipeline/run

### Description
Submits a pipeline task for asynchronous processing. This endpoint initiates a multi-stage process including loading, extracting, and searching data.

### Method
POST

### Endpoint
/api/v1/pipeline/run

### Parameters
#### Query Parameters
None

#### Request Body
- **source_config_id** (string) - Required - Identifier for the data source configuration.
- **task_name** (string) - Required - Name of the task.
- **task_description** (string) - Optional - Description of the task.
- **background** (string) - Optional - Context or background information for the task.
- **load** (object) - Optional - Configuration for the data loading stage.
  - **path** (string) - Required - Path to the data source.
  - **max_tokens** (integer) - Optional - Maximum tokens to process.
  - **auto_vector** (boolean) - Optional - Whether to automatically create vectors.
  - **min_content_length** (integer) - Optional - Minimum content length for processing.
  - **merge_short_sections** (boolean) - Optional - Whether to merge short sections.
- **extract** (object) - Optional - Configuration for the data extraction stage.
  - **max_concurrency** (integer) - Optional - Maximum concurrency for extraction.
  - **auto_vector** (boolean) - Optional - Whether to automatically create vectors.
  - **custom_entity_types** (array) - Optional - Array of custom entity types to extract.
    - **name** (string) - Required - Name of the entity type.
    - **description** (string) - Optional - Description of the entity type.
- **search** (object) - Optional - Configuration for the search stage.
  - **query** (string) - Required - The search query.
  - **enable_query_rewrite** (boolean) - Optional - Whether to enable query rewriting.
  - **return_type** (string) - Optional - The type of return for search results (e.g., 'event').
  - **recall** (object) - Optional - Recall parameters for search.
    - **use_fast_mode** (boolean) - Optional - Use fast mode for recall.
    - **vector_top_k** (integer) - Optional - Top K for vector search.
    - **max_entities** (integer) - Optional - Maximum entities to return.
  - **expand** (object) - Optional - Expansion parameters for search.
    - **enabled** (boolean) - Optional - Whether to enable expansion.
    - **max_hops** (integer) - Optional - Maximum hops for expansion.
    - **entities_per_hop** (integer) - Optional - Entities per hop.
  - **rerank** (object) - Optional - Reranking parameters for search.
    - **strategy** (string) - Optional - Reranking strategy (e.g., 'pagerank').
    - **max_results** (integer) - Optional - Maximum reranked results.

### Request Example
```json
{
  "source_config_id": "source-001",
  "task_name": "Process AI documentation",
  "task_description": "Analyze technical AI papers",
  "background": "AI and machine learning research papers",
  "load": {
    "path": "./docs/transformers.md",
    "max_tokens": 8000,
    "auto_vector": true,
    "min_content_length": 100,
    "merge_short_sections": true
  },
  "extract": {
    "max_concurrency": 10,
    "auto_vector": true,
    "custom_entity_types": [
      {
        "name": "Algorithm",
        "description": "ML algorithm names"
      }
    ]
  },
  "search": {
    "query": "transformer attention mechanisms",
    "enable_query_rewrite": true,
    "return_type": "event",
    "recall": {
      "use_fast_mode": true,
      "vector_top_k": 15,
      "max_entities": 25
    },
    "expand": {
      "enabled": true,
      "max_hops": 3,
      "entities_per_hop": 10
    },
    "rerank": {
      "strategy": "pagerank",
      "max_results": 10
    }
  }
}
```

### Response
#### Success Response (200)
- **success** (boolean) - Indicates if the operation was successful.
- **data** (object) - Contains task-related information.
  - **task_id** (string) - The unique identifier for the submitted task.
  - **status** (string) - The initial status of the task (e.g., 'pending').
  - **message** (string) - A message indicating the task creation status.
- **message** (string) - A general message about the pipeline start.

#### Response Example
```json
{
  "success": true,
  "data": {
    "task_id": "task-550e8400-e29b-41d4-a716-446655440000",
    "status": "pending",
    "message": "Task created and running..."
  },
  "message": "Pipeline started"
}
```
```

--------------------------------

### Model Configuration Management API

Source: https://context7.com/zleap-ai/sag/llms.txt

Endpoints for managing AI model configurations, including creating, listing, updating, and deleting configurations.

```APIDOC
## POST /api/v1/model-configs

### Description
Creates a new model configuration.

### Method
POST

### Endpoint
/api/v1/model-configs

### Request Body
- **type** (string) - Required - The type of the model (e.g., 'llm').
- **scenario** (string) - Required - The scenario for which the model is configured (e.g., 'extract').
- **model_name** (string) - Required - The name of the AI model.
- **api_key** (string) - Required - The API key for accessing the model service.
- **base_url** (string) - Required - The base URL for the model API.
- **temperature** (number) - Optional - The sampling temperature for generation.
- **max_tokens** (integer) - Optional - The maximum number of tokens to generate.
- **timeout** (integer) - Optional - The request timeout in seconds.
- **max_retries** (integer) - Optional - The maximum number of retries for failed requests.

### Request Example
```json
{
  "type": "llm",
  "scenario": "extract",
  "model_name": "gpt-4-turbo",
  "api_key": "sk-xxx",
  "base_url": "https://api.openai.com/v1",
  "temperature": 0.3,
  "max_tokens": 4000,
  "timeout": 60,
  "max_retries": 3
}
```

### Response
#### Success Response (201 Created)
- **message** (string) - Confirmation message.
- **config_id** (string) - The ID of the newly created model configuration.

### Response Example
```json
{
  "message": "Model configuration created successfully.",
  "config_id": "model_config_abc123"
}
```

## GET /api/v1/model-configs

### Description
Lists available model configurations with optional filtering.

### Method
GET

### Endpoint
/api/v1/model-configs

### Query Parameters
- **type** (string) - Optional - Filter configurations by model type.
- **scenario** (string) - Optional - Filter configurations by scenario.

### Response
#### Success Response (200)
- **success** (boolean) - Indicates if the request was successful.
- **data** (array) - An array of model configuration objects.
  - Each object contains details like `config_id`, `type`, `scenario`, `model_name`, etc.

### Response Example
```json
{
  "success": true,
  "data": [
    {
      "config_id": "model_config_123",
      "type": "llm",
      "scenario": "extract",
      "model_name": "gpt-4-turbo"
    }
  ]
}
```

## PATCH /api/v1/model-configs/{config_id}

### Description
Updates an existing model configuration.

### Method
PATCH

### Endpoint
/api/v1/model-configs/{config_id}

### Path Parameters
- **config_id** (string) - Required - The ID of the model configuration to update.

### Request Body
- **temperature** (number) - Optional - The sampling temperature for generation.
- **max_tokens** (integer) - Optional - The maximum number of tokens to generate.
- **timeout** (integer) - Optional - The request timeout in seconds.
- **max_retries** (integer) - Optional - The maximum number of retries for failed requests.

### Request Example
```json
{
  "temperature": 0.2,
  "max_tokens": 8000
}
```

### Response
#### Success Response (200)
- **message** (string) - Confirmation message.

### Response Example
```json
{
  "message": "Model configuration updated successfully."
}
```

## DELETE /api/v1/model-configs/{config_id}

### Description
Deletes a specific model configuration.

### Method
DELETE

### Endpoint
/api/v1/model-configs/{config_id}

### Path Parameters
- **config_id** (string) - Required - The ID of the model configuration to delete.

### Response
#### Success Response (200)
- **message** (string) - Confirmation message.

### Response Example
```json
{
  "message": "Model configuration deleted successfully."
}
```
```

--------------------------------

### Execute Complete SAG Pipeline (Load, Extract, Search)

Source: https://context7.com/zleap-ai/sag/llms.txt

Executes the full SAG pipeline: loading documents, extracting events and entities, and performing a three-stage search. It takes various configuration options for each stage and returns detailed results upon success or error.

```python
import asyncio
from sag import SAGEngine, DocumentLoadConfig, ExtractBaseConfig, SearchBaseConfig

async def main():
    engine = SAGEngine(source_config_id="tech-docs")

    # Stage 1: Load document and generate chunks
    await engine.load_async(DocumentLoadConfig(
        source_config_id="tech-docs",
        path="./documents/ai_overview.md",
        recursive=True,
        pattern="*.md",
        max_tokens=8000,
        auto_vector=True,
        min_content_length=100,
        merge_short_sections=True
    ))

    # Stage 2: Extract events and entities
    await engine.extract_async(ExtractBaseConfig(
        parallel=True,
        max_concurrency=10,
        auto_vector=True,
        background="Technical documentation about AI and machine learning"
    ))

    # Stage 3: Search with three-stage algorithm
    await engine.search_async(SearchBaseConfig(
        query="How to optimize large language model inference?",
        enable_query_rewrite=True
    ))

    # Get complete results
    result = engine.get_result()

    if result.is_success():
        print(f"✅ Pipeline completed successfully")
        print(f"📄 Article ID: {result.article_id}")
        print(f"📦 Events extracted: {len(result.extract_result.data_ids)}")
        print(f"🔍 Search results: {len(result.search_result.data_full)}")

        # Display search results
        for event in result.search_result.data_full:
            print(f"\n[Score: {event.rank:.2f}] {event.title}")
            print(f"Summary: {event.summary}")
            print(f"Time: {event.start_time} → {event.end_time}")
    else:
        print(f"❌ Error: {result.error}")

asyncio.run(main())
```

--------------------------------

### Source Configuration Management

Source: https://context7.com/zleap-ai/sag/llms.txt

Create and manage information sources with custom entity types. This API allows for defining and organizing data sources for information extraction and analysis.

```APIDOC
## Source Configuration Management

### Create Source Configuration

**Method:** POST
**Endpoint:** `/api/v1/sources`

**Description:** Creates a new source configuration.

**Request Body:**
```json
{
  "id": "project-docs-001",
  "name": "Project Documentation",
  "description": "Technical documentation for the main project",
  "status": "active",
  "entity_types": [
    {
      "type": "component",
      "name": "Component",
      "description": "Software components",
      "weight": 1.4
    },
    {
      "type": "api_endpoint",
      "name": "API Endpoint",
      "description": "REST API endpoints",
      "weight": 1.3
    }
  ],
  "metadata": {
    "project": "main-app",
    "team": "backend-team"
  }
}
```

**Response (Success):**
```json
{
  "success": true,
  "data": {
    "id": "project-docs-001",
    "name": "Project Documentation",
    "created_at": "2024-01-15T10:30:00Z"
  }
}
```

### List All Sources

**Method:** GET
**Endpoint:** `/api/v1/sources`

**Query Parameters:**
- **page** (integer) - Optional - Page number for pagination.
- **page_size** (integer) - Optional - Number of sources per page.

### Get Source Details

**Method:** GET
**Endpoint:** `/api/v1/sources/{source_id}`

**Path Parameters:**
- **source_id** (string) - Required - The ID of the source to retrieve.

### Update Source

**Method:** PATCH
**Endpoint:** `/api/v1/sources/{source_id}`

**Path Parameters:**
- **source_id** (string) - Required - The ID of the source to update.

**Request Body:**
```json
{
  "name": "Updated Project Documentation",
  "status": "active"
}
```

### Delete Source

**Method:** DELETE
**Endpoint:** `/api/v1/sources/{source_id}`

**Description:** Deletes a source and cascades to all related data.

**Path Parameters:**
- **source_id** (string) - Required - The ID of the source to delete.
```

--------------------------------

### Load Documents Only

Source: https://context7.com/zleap-ai/sag/llms.txt

Initiates the document loading stage of the pipeline without proceeding to extraction or search. This API call is useful for pre-processing and indexing documents. It requires a source configuration ID and a path to the documents.

```bash
curl -X POST "http://localhost:8000/api/v1/pipeline/load?source_config_id=docs-001&path=./articles/&background=Tech%20articles" \
  -H "Content-Type: application/json"
```

--------------------------------

### POST /api/v1/pipeline/run-sync

Source: https://context7.com/zleap-ai/sag/llms.txt

Executes the pipeline synchronously, waiting for completion. Suitable for smaller datasets or testing.

```APIDOC
## POST /api/v1/pipeline/run-sync

### Description
Executes the pipeline synchronously, meaning the API call will block until the entire pipeline process is complete. This is convenient for smaller datasets or for testing purposes where immediate results are needed.

### Method
POST

### Endpoint
/api/v1/pipeline/run-sync

### Parameters
#### Query Parameters
None

#### Request Body
- **source_config_id** (string) - Required - Identifier for the data source configuration.
- **load** (object) - Optional - Configuration for the data loading stage.
  - **path** (string) - Required - Path to the data source.
- **extract** (object) - Optional - Configuration for the data extraction stage.
  - **max_concurrency** (integer) - Optional - Maximum concurrency for extraction.
- **search** (object) - Optional - Configuration for the search stage.
  - **query** (string) - Required - The search query.

### Request Example
```json
{
  "source_config_id": "quick-test",
  "load": {
    "path": "./docs/sample.md"
  },
  "extract": {
    "max_concurrency": 5
  },
  "search": {
    "query": "key findings"
  }
}
```

### Response
#### Success Response (200)
- **success** (boolean) - Indicates if the operation was successful.
- **data** (object) - Contains the final results of the pipeline.
  - **article_id** (string) - Identifier for the processed article.
  - **load_stats** (object) - Statistics from the load stage.
  - **extract_stats** (object) - Statistics from the extract stage.
  - **search_results** (array) - The search results obtained.
- **message** (string) - A message indicating the pipeline completion.

#### Response Example
```json
{
  "success": true,
  "data": {
    "article_id": "article_456",
    "load_stats": {...},
    "extract_stats": {...},
    "search_results": [
      {
        "id": "event_001",
        "title": "Key Finding: Performance Improvement",
        "summary": "The system showed 40% improvement...",
        "rank": 0.92,
        "entities": [...] 
      }
    ]
  },
  "message": "Pipeline completed"
}
```
```

--------------------------------

### Submit Pipeline Task (Asynchronous)

Source: https://context7.com/zleap-ai/sag/llms.txt

Submits a pipeline task for asynchronous processing. It defines configurations for loading, extracting, and searching data, returning a task ID for status tracking. Dependencies include a running Zleap AI API service.

```bash
curl -X POST "http://localhost:8000/api/v1/pipeline/run" \
  -H "Content-Type: application/json" \
  -d '{
    "source_config_id": "source-001",
    "task_name": "Process AI documentation",
    "task_description": "Analyze technical AI papers",
    "background": "AI and machine learning research papers",
    "load": {
      "path": "./docs/transformers.md",
      "max_tokens": 8000,
      "auto_vector": true,
      "min_content_length": 100,
      "merge_short_sections": true
    },
    "extract": {
      "max_concurrency": 10,
      "auto_vector": true,
      "custom_entity_types": [
        {
          "name": "Algorithm",
          "description": "ML algorithm names"
        }
      ]
    },
    "search": {
      "query": "transformer attention mechanisms",
      "enable_query_rewrite": true,
      "return_type": "event",
      "recall": {
        "use_fast_mode": true,
        "vector_top_k": 15,
        "max_entities": 25
      },
      "expand": {
        "enabled": true,
        "max_hops": 3,
        "entities_per_hop": 10
      },
      "rerank": {
        "strategy": "pagerank",
        "max_results": 10
      }
    }
  }'
```

--------------------------------

### POST /api/v1/pipeline/search

Source: https://context7.com/zleap-ai/sag/llms.txt

Executes the search stage across multiple specified sources.

```APIDOC
## POST /api/v1/pipeline/search

### Description
Executes the search stage of the pipeline, allowing queries across multiple specified information sources simultaneously. It supports query rewriting, fast mode search, and advanced ranking options.

### Method
POST

### Endpoint
/api/v1/pipeline/search

### Parameters
#### Query Parameters
None

#### Request Body
- **source_config_ids** (array) - Required - Array of source configuration identifiers to search across.
- **query** (string) - Required - The search query.
- **enable_query_rewrite** (boolean) - Optional - Whether to enable query rewriting for better results.
- **use_fast_mode** (boolean) - Optional - Enables a faster search mode, potentially at the cost of some accuracy.
- **vector_top_k** (integer) - Optional - The number of top results to consider from vector search.
- **max_entities** (integer) - Optional - Maximum number of entities to return in the results.
- **max_events** (integer) - Optional - Maximum number of events to return in the results.
- **expand_enabled** (boolean) - Optional - Whether to enable query expansion.
- **max_hops** (integer) - Optional - Maximum hops for query expansion.
- **entities_per_hop** (integer) - Optional - Number of entities to retrieve per hop during expansion.
- **strategy** (string) - Optional - The strategy for re-ranking results (e.g., 'pagerank').
- **max_results** (integer) - Optional - The maximum number of final results to return.
- **pagerank_damping_factor** (number) - Optional - Damping factor for the PageRank algorithm if used for re-ranking.

### Request Example
```json
{
  "source_config_ids": ["source_001", "source_002", "source_003"],
  "query": "neural network optimization techniques",
  "enable_query_rewrite": true,
  "use_fast_mode": true,
  "vector_top_k": 20,
  "max_entities": 30,
  "max_events": 100,
  "expand_enabled": true,
  "max_hops": 3,
  "entities_per_hop": 12,
  "strategy": "pagerank",
  "max_results": 15,
  "pagerank_damping_factor": 0.85
}
```

### Response
#### Success Response (200)
- **success** (boolean) - Indicates if the operation was successful.
- **data** (object) - Contains the search results.
  - **events** (array) - A list of events matching the search criteria.
    - **id** (string) - Unique identifier for the event.
    - **title** (string) - Title of the event.
    - **summary** (string) - A brief summary of the event.
    - **content** (string) - Full content of the event.
    - **rank** (number) - Relevance rank of the event.
    - **start_time** (string) - Start time associated with the event.
    - **end_time** (string) - End time associated with the event.
    - **category** (string) - Category of the event.
    - **entities** (array) - Entities found within the event.
- **message** (string) - A message indicating the search stage completion.

#### Response Example
```json
{
  "success": true,
  "data": {
    "events": [
      {
        "id": "event_123",
        "title": "Gradient Descent Optimization",
        "summary": "Techniques for accelerating gradient descent...",
        "content": "Full content here...",
        "rank": 0.95,
        "start_time": "2024-01-15",
        "end_time": "2024-01-15",
        "category": "TECHNICAL",
        "entities": [
          {
            "name": "Adam Optimizer",
            "type": "Algorithm"
          }
        ]
      }
    ]
  },
  "message": "Search stage completed"
}
```
```

--------------------------------

### Search with Multi-Source Support

Source: https://context7.com/zleap-ai/sag/llms.txt

Performs a search query across multiple specified data sources. This endpoint allows for advanced search configurations including query rewriting, recall settings, entity expansion, and reranking strategies. It is designed for comprehensive information retrieval.

```bash
curl -X POST "http://localhost:8000/api/v1/pipeline/search" \
  -H "Content-Type: application/json" \
  -d '{
    "source_config_ids": ["source_001", "source_002", "source_003"],
    "query": "neural network optimization techniques",
    "enable_query_rewrite": true,
    "use_fast_mode": true,
    "vector_top_k": 20,
    "max_entities": 30,
    "max_events": 100,
    "expand_enabled": true,
    "max_hops": 3,
    "entities_per_hop": 12,
    "strategy": "pagerank",
    "max_results": 15,
    "pagerank_damping_factor": 0.85
  }'
```