Deep Searcher

https://github.com/zilliztech/deep-searcher
Admin
Open Source Deep Research Alternative to Reason and Search on Private Data. Written in Python.
Tokens:17,974
Snippets:173
Trust Score:9.8
License:Apache-2.0
Update:1 year ago
Show doc for...
Tokens:
Raw
### Start DeepSearcher FastAPI Web Service

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/usage/deployment.md

This command starts the DeepSearcher web service, which is built using FastAPI. It runs the `main.py` script, making the service accessible at the default address `localhost:8000`. This is the primary way to run the application directly.

```shell
python main.py
```

--------------------------------

### Install DeepSearcher using pip

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Instructions to create a Python virtual environment and install the DeepSearcher package using pip, including an example for optional dependencies like Ollama.

```bash
python -m venv .venv
source .venv/bin/activate
pip install deepsearcher
pip install "deepsearcher[ollama]"
```

--------------------------------

### DeepSearcher quick start demo with Python

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

A Python example demonstrating how to initialize DeepSearcher configuration, set up LLM and embedding providers, load data from local files or websites, and perform a query.

```python
from deepsearcher.configuration import Configuration, init_config
from deepsearcher.online_query import query

config = Configuration()

# Customize your config here,
# more configuration see the Configuration Details section below.
config.set_provider_config("llm", "OpenAI", {"model": "o1-mini"})
config.set_provider_config("embedding", "OpenAIEmbedding", {"model": "text-embedding-ada-002"})
init_config(config = config)

# Load your local data
from deepsearcher.offline_loading import load_from_local_files
load_from_local_files(paths_or_directory=your_local_path)

# (Optional) Load from web crawling (`FIRECRAWL_API_KEY` env variable required)
from deepsearcher.offline_loading import load_from_website
load_from_website(urls=website_url)

# Query
result = query("Write a report about xxx.") # Your question here
```

--------------------------------

### Verify DeepSearcher Installation

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/installation/index.md

This Python snippet demonstrates how to verify a successful DeepSearcher installation. It initializes the default configuration using the `Configuration` class from `deepsearcher.configuration` and prints a confirmation message.

```python
from deepsearcher.configuration import Configuration
from deepsearcher.online_query import query

# Initialize with default configuration
config = Configuration()
print("DeepSearcher installed successfully!")
```

--------------------------------

### Configure Google Gemini LLM Provider

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Example of configuring Google Gemini. Ensure your `GEMINI_API_KEY` is set as an environment variable and `pip install google-genai` is executed. This configuration uses the 'gemini-2.0-flash' model.

```Python
config.set_provider_config('llm', 'Gemini', { 'model': 'gemini-2.0-flash' })
```

--------------------------------

### Install MkDocs and required plugins

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/README.md

Installs MkDocs, MkDocs Material theme, MkDocs Jupyter plugin, and PyMdown Extensions using pip, which are necessary for building and serving the documentation.

```bash
pip install mkdocs mkdocs-material mkdocs-jupyter pymdown-extensions
```

--------------------------------

### Clone Repository and Install Dependencies with uv (Shell)

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/contributing/index.md

Steps to set up the DeepSearcher development environment. This involves cloning the repository, installing project dependencies using `uv sync`, and activating the created virtual environment.

```shell
git clone https://github.com/zilliztech/deep-searcher.git && cd deep-searcher
```

```shell
uv sync
```

```shell
source .venv/bin/activate
```

--------------------------------

### Install DeepSearcher with All Optional Dependencies

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/installation/pip.md

This command installs DeepSearcher including all available optional dependencies, providing full functionality and integrations.

```bash
pip install "deepsearcher[all]"
```

--------------------------------

### Install uv package manager via PowerShell (Windows)

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/installation/development.md

Downloads and executes the `uv` installation script using PowerShell on Windows systems.

```powershell
irm https://astral.sh/uv/install.ps1 | iex
```

--------------------------------

### Configure VoyageAI Embedding Model

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

This example sets up the VoyageAI embedding model. It requires the `VOYAGE_API_KEY` environment variable to be prepared. Additionally, the `voyageai` Python package must be installed to use this provider.

```Python
config.set_provider_config("embedding", "VoyageEmbedding", {"model": "voyage-3"})
```

--------------------------------

### Install DeepSearcher in development mode with uv

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Steps to clone the DeepSearcher repository and install its development dependencies using uv, a fast package installer.

```shell
git clone https://github.com/zilliztech/deep-searcher.git && cd deep-searcher
uv sync
source .venv/bin/activate
```

--------------------------------

### Configure OpenRouter LLM Provider (via OpenAI interface)

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Example of configuring OpenRouter using the OpenAI provider interface. This setup specifies a Qwen3 model, a custom base URL for OpenRouter, and requires an `OPENROUTER_API_KEY`.

```Python
config.set_provider_config("llm", "OpenAI", {"model": "qwen/qwen3-235b-a22b:free", "base_url": "https://openrouter.ai/api/v1", "api_key": "OPENROUTER_API_KEY"})
```

--------------------------------

### Install nest_asyncio for Jupyter Notebook Asyncio Compatibility

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/faq/index.md

Install the `nest_asyncio` package to resolve common `asyncio` issues when running DeepSearcher or similar applications within a Jupyter notebook environment.

```bash
pip install nest_asyncio
```

--------------------------------

### Configure IBM watsonx.ai LLM Provider

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Example Python configuration for integrating IBM watsonx.ai as an LLM provider. Requires `WATSONX_APIKEY`, `WATSONX_URL`, and `WATSONX_PROJECT_ID` environment variables, and the `ibm-watsonx-ai` Python package to be installed.

```Shell
pip install ibm-watsonx-ai
```

```Python
config.set_provider_config("llm", "watsonx", {"model": "us.deepseek.r1-v1:0"})
```

--------------------------------

### Set up DeepSearcher development environment with pip

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/installation/development.md

Creates a Python virtual environment, activates it, and installs DeepSearcher in editable mode with development and all optional dependencies using `pip`.

```bash
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
pip install -e ".[dev,all]"
```

--------------------------------

### Install DeepSearcher with Ollama Integration

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/installation/pip.md

This command installs DeepSearcher along with its optional dependencies required for integration with Ollama, enabling local LLM deployment capabilities.

```bash
pip install "deepsearcher[ollama]"
```

--------------------------------

### DeepSearcher Python Example: Querying PDF with Token Tracking

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/examples/oracle.md

This Python script demonstrates how to initialize DeepSearcher, load a PDF document into a vector database, and perform a complex query. It includes path manipulation to import modules from a parent directory and showcases how to retrieve the answer, retrieved context, and consumed tokens from the query operation. It also shows how to disable the httpx logger.

```python
import sys, os
from pathlib import Path
script_directory = Path(__file__).resolve().parent.parent
sys.path.append(os.path.abspath(script_directory))

import logging

httpx_logger = logging.getLogger("httpx")  # disable openai's logger output
httpx_logger.setLevel(logging.WARNING)

current_dir = os.path.dirname(os.path.abspath(__file__))

# Customize your config here
from deepsearcher.configuration import Configuration, init_config

config = Configuration()
init_config(config=config)

# Load your local data
# Hint: You can load from a directory or a single file, please execute it in the root directory of the deep searcher project

from deepsearcher.offline_loading import load_from_local_files

load_from_local_files(
    paths_or_directory=os.path.join(current_dir, "data/WhatisMilvus.pdf"),
    collection_name="milvus_docs",
    collection_description="All Milvus Documents",
    # force_new_collection=True, # If you want to drop origin collection and create a new collection every time, set force_new_collection to True
)

# Query
from deepsearcher.online_query import query

question = 'Write a report comparing Milvus with other vector databases.'
answer, retrieved_results, consumed_token = query(question)
print(answer)

# get consumed tokens, about: 2.5~3w tokens when using openai gpt-4o model
# print(f"Consumed tokens: {consumed_token}")
```

--------------------------------

### Serve DeepSearcher documentation locally

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/README.md

Starts a local development server for the MkDocs documentation, allowing you to preview changes in real-time. The documentation will be accessible in your web browser, typically at http://127.0.0.1:8000/.

```bash
mkdocs serve
```

--------------------------------

### Install DeepSearcher Package

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/installation/pip.md

This command uses pip, Python's package installer, to download and install the DeepSearcher library and its core dependencies into the active virtual environment.

```bash
pip install deepsearcher
```

--------------------------------

### Install uv package manager via pip

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/installation/development.md

Installs the `uv` Python package manager using the standard `pip` command. `uv` is recommended for faster dependency management.

```bash
pip install uv
```

--------------------------------

### Basic Usage of DeepSearcher with Python

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/usage/quick_start.md

This snippet demonstrates the fundamental steps to use DeepSearcher. It covers initializing the configuration, setting up LLM and embedding providers, loading data from local files or websites, and performing a query. Ensure 'OPENAI_API_KEY' and 'FIRECRAWL_API_KEY' (for website loading) are set in your environment variables.

```python
# Import configuration modules
from deepsearcher.configuration import Configuration, init_config
from deepsearcher.online_query import query

# Initialize configuration
config = Configuration()

# Customize your config here
# (See the Configuration Details section below for more options)
config.set_provider_config("llm", "OpenAI", {"model": "o1-mini"})
config.set_provider_config("embedding", "OpenAIEmbedding", {"model": "text-embedding-ada-002"})
init_config(config=config)

# Load data from local files
from deepsearcher.offline_loading import load_from_local_files
load_from_local_files(paths_or_directory=your_local_path)

# (Optional) Load data from websites
# Requires FIRECRAWL_API_KEY environment variable
from deepsearcher.offline_loading import load_from_website
load_from_website(urls=website_url)

# Query your data
result = query("Write a report about xxx.")  # Replace with your question
print(result)
```

--------------------------------

### Install Unstructured Ingest Core

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/file_loader.md

Installs the essential `unstructured-ingest` package, a prerequisite for local processing with Unstructured.

```bash
pip install unstructured-ingest
```

--------------------------------

### Configure DeepSeek via TogetherAI LLM Provider

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Example of configuring DeepSeek through TogetherAI. Ensure your `TOGETHER_API_KEY` is set as an environment variable and `pip install together` is executed. This configuration uses the 'deepseek-ai/DeepSeek-R1' model.

```Python
config.set_provider_config("llm", "TogetherAI", {"model": "deepseek-ai/DeepSeek-R1"})
```

--------------------------------

### Configure GLM LLM Provider

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Example Python configuration for integrating GLM as an LLM provider. Requires `GLM_API_KEY` environment variable and the `zhipuai` Python package to be installed.

```Shell
pip install zhipuai
```

```Python
config.set_provider_config("llm", "GLM", {"model": "glm-4-plus"})
```

--------------------------------

### Install uv package manager via curl (Unix/macOS)

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/installation/development.md

Downloads and installs the `uv` package manager on Unix-like systems (macOS, Linux) by executing a script via `curl`.

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

--------------------------------

### Configure Crawl4AI Web Crawler and Install Package

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/web_crawler.md

Configures DeepSearcher to use Crawl4AI, a Python package for web crawling with browser automation. Provides an example of browser configuration and installation steps.

```python
config.set_provider_config("web_crawler", "Crawl4AICrawler", {"browser_config": {"headless": True, "verbose": True}})
```

```bash
pip install crawl4ai
```

```bash
crawl4ai-setup
```

--------------------------------

### Install Optional and Development Dependencies with uv (Shell)

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/contributing/index.md

Commands for installing additional dependencies. This includes installing all optional and development dependencies, or specific optional dependencies like `ollama`, to enable extended project features.

```shell
uv sync --all-extras --dev
```

```shell
uv sync --extra ollama
```

--------------------------------

### Configure Deep Searcher with Crawl4AICrawler

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Configures Deep Searcher to use Crawl4AICrawler. This setup requires the 'crawl4ai' Python package to be installed and 'crawl4ai-setup' to be executed. An example demonstrates how to pass browser configuration arguments.

```Python
config.set_provider_config("web_crawler", "Crawl4AICrawler", {"browser_config": {"headless": True, "verbose": True}})
```

--------------------------------

### Configure Volcengine LLM

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/llm.md

This example demonstrates how to configure the Volcengine LLM. To use this provider, you must set the `VOLCENGINE_API_KEY` environment variable.

```python
config.set_provider_config("llm", "Volcengine", {"model": "deepseek-r1-250120"})
```

--------------------------------

### Configure Llama 4 via TogetherAI LLM Provider

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Example of configuring Llama 4 through TogetherAI. Ensure your `TOGETHER_API_KEY` is set as an environment variable and `pip install together` is executed. This configuration uses the 'meta-llama/Llama-4-Scout-17B-16E-Instruct' model.

```Python
config.set_provider_config("llm", "TogetherAI", {"model": "meta-llama/Llama-4-Scout-17B-16E-Instruct"})
```

--------------------------------

### Display help for DeepSearcher 'load' command

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/usage/cli.md

Shows detailed help documentation specifically for the `load` subcommand of DeepSearcher. This includes all available options, arguments, and examples pertinent to data ingestion.

```shell
deepsearcher load --help
```

--------------------------------

### Set up DeepSearcher development environment with uv

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/installation/development.md

Synchronizes project dependencies using `uv` and activates the created Python virtual environment for DeepSearcher development.

```bash
uv sync
source .venv/bin/activate
```

--------------------------------

### Configure Amazon Bedrock LLM

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/llm.md

This example shows how to configure the Amazon Bedrock LLM. This requires installing the `boto3` Python package and setting the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables.

```python
config.set_provider_config("llm", "Bedrock", {"model": "us.deepseek.r1-v1:0"})
```

--------------------------------

### Perform Semantic Search with DeepSearcher in Python

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/examples/basic_example.md

This Python script demonstrates how to initialize DeepSearcher, load a local PDF file into a collection, and then query the loaded data semantically. It also shows how to track consumed tokens from the language model.

```python
import logging
import os

from deepsearcher.offline_loading import load_from_local_files
from deepsearcher.online_query import query
from deepsearcher.configuration import Configuration, init_config

httpx_logger = logging.getLogger("httpx")  # disable openai's logger output
httpx_logger.setLevel(logging.WARNING)

current_dir = os.path.dirname(os.path.abspath(__file__))

config = Configuration()  # Customize your config here
init_config(config=config)


# You should clone the milvus docs repo to your local machine first, execute:
# git clone https://github.com/milvus-io/milvus-docs.git
# Then replace the path below with the path to the milvus-docs repo on your local machine
# import glob
# all_md_files = glob.glob('xxx/milvus-docs/site/en/**/*.md', recursive=True)
# load_from_local_files(paths_or_directory=all_md_files, collection_name="milvus_docs", collection_description="All Milvus Documents")

# Hint: You can also load a single file, please execute it in the root directory of the deep searcher project
load_from_local_files(
    paths_or_directory=os.path.join(current_dir, "data/WhatisMilvus.pdf"),
    collection_name="milvus_docs",
    collection_description="All Milvus Documents",
    # force_new_collection=True, # If you want to drop origin collection and create a new collection every time, set force_new_collection to True
)

question = "Write a report comparing Milvus with other vector databases."

_, _, consumed_token = query(question, max_iter=1)
print(f"Consumed tokens: {consumed_token}")
```

--------------------------------

### Configure Amazon Bedrock LLM Provider

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Example Python configuration for integrating Amazon Bedrock as an LLM provider. Requires `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables, and the `boto3` Python package to be installed.

```Shell
pip install boto3
```

```Python
config.set_provider_config("llm", "Bedrock", {"model": "us.deepseek.r1-v1:0"})
```

--------------------------------

### Install Unstructured for All Document Types

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/file_loader.md

Installs the `unstructured` library with support for all document formats, providing comprehensive local document processing capabilities.

```bash
pip install "unstructured[all-docs]"
```

--------------------------------

### Configure IBM watsonx.ai LLM

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/llm.md

This example demonstrates configuring the IBM watsonx.ai LLM, including options for custom parameters and using a `space_id`. Before use, install the `ibm-watsonx-ai` Python package and set the `WATSONX_APIKEY`, `WATSONX_URL`, and `WATSONX_PROJECT_ID` environment variables.

```python
config.set_provider_config("llm", "WatsonX", {"model": "ibm/granite-3-3-8b-instruct"})
```

```python
config.set_provider_config("llm", "WatsonX", {
    "model": "ibm/granite-3-3-8b-instruct",
    "max_new_tokens": 1000,
    "temperature": 0.7,
    "top_p": 0.9,
    "top_k": 50
})
```

```python
config.set_provider_config("llm", "WatsonX", {
        "model": "ibm/granite-3-3-8b-instruct""
    })
```

--------------------------------

### Verify DeepSearcher Installation

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/installation/pip.md

This Python snippet imports the __version__ attribute from the DeepSearcher library and prints its value, confirming that the library has been successfully installed and is accessible.

```python
from deepsearcher import __version__
print(f"DeepSearcher version: {__version__}")
```

--------------------------------

### Clone Repository and Setup Development Environment with uv (Shell)

Source: https://github.com/zilliztech/deep-searcher/blob/master/CONTRIBUTING.md

Instructions to clone the DeepSearcher repository and set up the development environment using uv. This includes synchronizing dependencies and activating the virtual environment for project development.

```shell
git clone https://github.com/zilliztech/deep-searcher.git && cd deep-searcher
```

```shell
uv sync
source .venv/bin/activate
```

--------------------------------

### Configure DeepSearcher with Docling for Local File and Web Data Ingestion (Python)

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/examples/docling.md

This Python script demonstrates how to set up DeepSearcher to utilize Docling as both a file loader and a web crawler. It initializes the DeepSearcher configuration, specifies Docling for file and web data processing, then proceeds to load data from a local file path and crawl a list of URLs, including Markdown and PDF documents. Finally, it shows how to query the aggregated data, incorporating basic error handling for file loading operations.

```python
import logging
import os
from deepsearcher.offline_loading import load_from_local_files, load_from_website
from deepsearcher.online_query import query
from deepsearcher.configuration import Configuration, init_config

# Suppress unnecessary logging from third-party libraries
logging.getLogger("httpx").setLevel(logging.WARNING)

def main():
    # Step 1: Initialize configuration
    config = Configuration()

    # Configure Vector Database and Docling providers
    config.set_provider_config("vector_db", "Milvus", {})
    config.set_provider_config("file_loader", "DoclingLoader", {})
    config.set_provider_config("web_crawler", "DoclingCrawler", {})

    # Apply the configuration
    init_config(config)

    # Step 2a: Load data from a local file using DoclingLoader
    local_file = "your_local_file_or_directory"
    local_collection_name = "DoclingLocalFiles"
    local_collection_description = "Milvus Documents loaded using DoclingLoader"

    print("\n=== Loading local files using DoclingLoader ===")
    
    try:
        load_from_local_files(
            paths_or_directory=local_file, 
            collection_name=local_collection_name, 
            collection_description=local_collection_description,
            force_new_collection=True
        )
        print(f"Successfully loaded: {local_file}")
    except ValueError as e:
        print(f"Validation error: {str(e)}")
    except Exception as e:
        print(f"Error: {str(e)}")

    print("Successfully loaded all local files")

    # Step 2b: Crawl URLs using DoclingCrawler
    urls = [
        # Markdown documentation files
        "https://milvus.io/docs/quickstart.md",
        "https://milvus.io/docs/overview.md",
        # PDF example - can handle various URL formats
        "https://arxiv.org/pdf/2408.09869"
    ]
    web_collection_name = "DoclingWebCrawl"
    web_collection_description = "Milvus Documentation crawled using DoclingCrawler"

    print("\n=== Crawling web pages using DoclingCrawler ===")
    
    load_from_website(
        urls=urls,
        collection_name=web_collection_name,
        collection_description=web_collection_description,
        force_new_collection=True
    )
    print("Successfully crawled all URLs")

    # Step 3: Query the loaded data
    question = "What is Milvus?"
    result = query(question)


if __name__ == "__main__":
    main()
```

--------------------------------

### Install Optional and Development Dependencies with uv (Shell)

Source: https://github.com/zilliztech/deep-searcher/blob/master/CONTRIBUTING.md

Commands to install all optional and development dependencies, or specific optional dependencies, using uv. This is useful for enabling additional features or development tools like testing frameworks.

```shell
uv sync --all-extras --dev
```

```shell
uv sync --extra ollama
```

--------------------------------

### Display general help for DeepSearcher CLI

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/usage/cli.md

Provides an overview of the DeepSearcher command-line interface, listing all available top-level commands and general usage instructions. This is the starting point for understanding the CLI's capabilities.

```shell
deepsearcher --help
```

--------------------------------

### Configure DeepSeek and Llama from TogetherAI

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/llm.md

This example shows how to configure various LLM models from TogetherAI, including DeepSeek R1 and Llama 4. Before running, you must install the `together` Python package via pip and set the `TOGETHER_API_KEY` environment variable.

```python
config.set_provider_config("llm", "TogetherAI", {"model": "deepseek-ai/DeepSeek-R1"})
```

```python
config.set_provider_config("llm", "TogetherAI", {"model": "meta-llama/Llama-4-Scout-17B-16E-Instruct"})
```

--------------------------------

### Install Unstructured for Specific Document Types (e.g., PDF)

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/file_loader.md

Installs the `unstructured` library with support for a specific document format, such as PDF, for targeted local processing.

```bash
pip install "unstructured[pdf]"
```

--------------------------------

### Configure DeepSeek from PPIO

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/llm.md

This example shows how to configure the DeepSeek LLM using the PPIO provider. Ensure the `PPIO_API_KEY` environment variable is set for this configuration to function correctly.

```python
config.set_provider_config("llm", "PPIO", {"model": "deepseek/deepseek-r1-turbo"})
```

--------------------------------

### Install DeepSearcher Test Dependencies with uv (Shell)

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/contributing/index.md

Command to ensure all necessary development and optional dependencies, including testing tools like `pytest`, are installed. This is a prerequisite for running the project's test suite.

```shell
uv sync --all-extras --dev
```

--------------------------------

### Run DeepSearcher tests with pytest

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/installation/development.md

Executes the test suite for DeepSearcher using the `pytest` framework, targeting the `tests/` directory.

```bash
pytest tests/
```

--------------------------------

### Install Docling Library

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/file_loader.md

Installs the `docling` Python library, which is required for using the DoclingLoader in DeepSearcher.

```bash
pip install docling
```

--------------------------------

### Start DeepSearcher FastAPI Service

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Command to run the main Python script, which starts the DeepSearcher service as a FastAPI application on localhost:8000.

```shell
$ python main.py
```

--------------------------------

### Configure Volcengine LLM Provider

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Example Python configuration for integrating Volcengine as an LLM provider. Requires `VOLCENGINE_API_KEY` environment variable to be set.

```Python
config.set_provider_config("llm", "Volcengine", {"model": "deepseek-r1-250120"})
```

--------------------------------

### Run DeepSearcher Docker Container with Configuration

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/usage/deployment.md

This command runs the previously built DeepSearcher Docker image as a container. It maps port 8000, sets the `OPENAI_API_KEY` environment variable, and mounts local `data`, `logs`, and `config.yaml` directories/files into the container. This provides a portable and isolated environment for running DeepSearcher.

```shell
docker run -p 8000:8000 \
  -e OPENAI_API_KEY=your_openai_api_key \
  -v $(pwd)/data:/app/data \
  -v $(pwd)/logs:/app/logs \
  -v $(pwd)/deepsearcher/config.yaml:/app/deepsearcher/config.yaml \
  deepsearcher:latest
```

--------------------------------

### Install nest_asyncio for Jupyter Notebook

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Installs the `nest_asyncio` library, required to run DeepSearcher's asynchronous operations within a Jupyter Notebook environment.

```shell
pip install nest_asyncio
```

--------------------------------

### Display help for DeepSearcher 'query' command

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/usage/cli.md

Provides specific help documentation for the `query` subcommand of DeepSearcher. It outlines the parameters, usage, and examples for performing searches on indexed data.

```shell
deepsearcher query --help
```

--------------------------------

### Configure DeepSeek via PPIO LLM Provider

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Example of configuring DeepSeek through PPIO. Ensure your `PPIO_API_KEY` is set as an environment variable. This configuration uses the 'deepseek/deepseek-r1-turbo' model.

```Python
config.set_provider_config("llm", "PPIO", {"model": "deepseek/deepseek-r1-turbo"})
```

--------------------------------

### Configure DoclingCrawler and Install Package

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/web_crawler.md

Configures DeepSearcher to use DoclingCrawler, which integrates web crawling with document processing. Provides instructions for installing the Docling package.

```python
config.set_provider_config("web_crawler", "DoclingCrawler", {})
```

```bash
pip install docling
```

--------------------------------

### Activate Python Virtual Environment

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/installation/pip.md

These commands activate the previously created virtual environment, allowing subsequent Python commands to use the environment's specific dependencies. The command differs based on the operating system.

```bash
source .venv/bin/activate
```

```batch
.venv\Scripts\activate
```

--------------------------------

### Web Crawler Document Loaders

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/integrations/index.md

Details on web crawlers supported for document loading, including their description and setup requirements.

```APIDOC
Crawler: FireCrawl
  Description: Crawler designed for AI applications
  Required Environment Variables/Setup: FIRECRAWL_API_KEY

Crawler: Jina Reader
  Description: High-accuracy web content extraction
  Required Environment Variables/Setup: JINA_API_TOKEN

Crawler: Crawl4AI
  Description: Browser automation crawler
  Required Environment Variables/Setup: Run `crawl4ai-setup` for first-time use
```

--------------------------------

### Configure OpenAI LLM Provider

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Example of configuring OpenAI as the LLM provider. Ensure your `OPENAI_API_KEY` is set as an environment variable. This configuration uses the 'o1-mini' model.

```Python
config.set_provider_config("llm", "OpenAI", {"model": "o1-mini"})
```

--------------------------------

### Create Python Virtual Environment

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/installation/pip.md

This command creates a new isolated Python virtual environment named '.venv' in the current directory. It helps manage project dependencies without interfering with the system's Python installation.

```bash
python -m venv .venv
```

--------------------------------

### Configure XAI Grok LLM Provider

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Example of configuring XAI Grok. Ensure your `XAI_API_KEY` is set as an environment variable. This configuration uses the 'grok-2-latest' model.

```Python
config.set_provider_config("llm", "XAI", {"model": "grok-2-latest"})
```

--------------------------------

### Configure Aliyun Bailian LLM Provider

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Example of configuring Aliyun Bailian as the LLM provider. Ensure your `DASHSCOPE_API_KEY` is set as an environment variable. This configuration uses the 'qwen-plus-latest' model.

```Python
config.set_provider_config("llm", "Aliyun", {"model": "qwen-plus-latest"})
```

--------------------------------

### Access DeepSearcher Command-Line Help

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Shows how to get general help for the DeepSearcher CLI and specific help for subcommands like 'load' and 'query'.

```shell
deepsearcher --help
```

```shell
deepsearcher load --help
```

```shell
deepsearcher query --help
```

--------------------------------

### Configure Volcengine Embedding Model

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

This example sets up the Volcengine embedding model. It requires the `VOLCENGINE_API_KEY` environment variable to be configured. The model `doubao-embedding-text-240515` is specified for this provider.

```Python
config.set_provider_config("embedding", "VolcengineEmbedding", {"model": "doubao-embedding-text-240515"})
```

--------------------------------

### Configure FastEmbed Embedding Provider

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Sets up the FastEmbed embedding provider with a specified model. The `fastembed` library must be installed to use this provider.

```Python
config.set_provider_config("embedding", "FastEmbedEmbedding", {"model": "intfloat/multilingual-e5-large"})
```

--------------------------------

### Clone DeepSearcher Git repository

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/installation/development.md

Clones the DeepSearcher source code from GitHub and changes the current directory into the newly created repository folder.

```bash
git clone https://github.com/zilliztech/deep-searcher.git
cd deep-searcher
```

--------------------------------

### Configure DeepSeek Official LLM Provider

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Example of configuring DeepSeek directly. Ensure your `DEEPSEEK_API_KEY` is set as an environment variable. This configuration uses the 'deepseek-reasoner' model.

```Python
config.set_provider_config("llm", "DeepSeek", {"model": "deepseek-reasoner"})
```

--------------------------------

### Configure Anthropic Claude LLM

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/llm.md

This example demonstrates configuring the Anthropic Claude LLM. To enable this provider, ensure your `ANTHROPIC_API_KEY` environment variable is set.

```python
config.set_provider_config("llm", "Anthropic", {"model": "claude-sonnet-4-0"})
```

--------------------------------

### Configure Gemini Embedding Provider

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Sets up the Gemini embedding provider using a specified model. This configuration requires the `google-genai` library to be installed.

```Python
config.set_provider_config("embedding", "GeminiEmbedding", {"model": "text-embedding-004"})
```

--------------------------------

### Apply nest_asyncio Patch in Jupyter Notebook

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/faq/index.md

After installing `nest_asyncio`, add these lines at the beginning of your Jupyter notebook to apply the patch, enabling proper `asyncio` loop management within the notebook.

```python
import nest_asyncio
nest_asyncio.apply()
```

--------------------------------

### Configure DeepSeek via SiliconFlow LLM Provider

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Example of configuring DeepSeek through SiliconFlow. Ensure your `SILICONFLOW_API_KEY` is set as an environment variable. This configuration uses the 'deepseek-ai/DeepSeek-R1' model.

```Python
config.set_provider_config("llm", "SiliconFlow", {"model": "deepseek-ai/DeepSeek-R1"})
```

--------------------------------

### Configure PPIO Cloud Embedding Model

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/embedding.md

Example of configuring the PPIO cloud embedding model with `baai/bge-m3`. This requires the `PPIO_API_KEY` environment variable for authentication.

```python
config.set_provider_config("embedding", "PPIOEmbedding", {"model": "baai/bge-m3"})
```

--------------------------------

### Integrate FireCrawl with DeepSearcher for Web Crawling and Querying in Python

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/examples/firecrawl.md

This Python script demonstrates how to integrate FireCrawl with DeepSearcher to crawl a website, load its content into a Milvus vector database, and then query the loaded data. It covers setting up API keys for OpenAI and FireCrawl, configuring DeepSearcher's vector database and web crawler providers, and executing the data loading and querying process. It also includes logging suppression and commented-out examples for advanced crawling options like max_depth.

```python
import logging
import os
from deepsearcher.offline_loading import load_from_website
from deepsearcher.online_query import query
from deepsearcher.configuration import Configuration, init_config

# Suppress unnecessary logging from third-party libraries
logging.getLogger("httpx").setLevel(logging.WARNING)

# Set API keys (ensure these are set securely in real applications)
os.environ['OPENAI_API_KEY'] = 'sk-***************'
os.environ['FIRECRAWL_API_KEY'] = 'fc-***************'


def main():
    # Step 1: Initialize configuration
    config = Configuration()

    # Set up Vector Database (Milvus) and Web Crawler (FireCrawlCrawler)
    config.set_provider_config("vector_db", "Milvus", {})
    config.set_provider_config("web_crawler", "FireCrawlCrawler", {})

    # Apply the configuration
    init_config(config)

    # Step 2: Load data from a website into Milvus
    website_url = "https://example.com"  # Replace with your target website
    collection_name = "FireCrawl"
    collection_description = "All Milvus Documents"

    # crawl a single webpage
    load_from_website(urls=website_url, collection_name=collection_name, collection_description=collection_description)
    # only applicable if using Firecrawl: deepsearcher can crawl multiple webpages, by setting max_depth, limit, allow_backward_links
    # load_from_website(urls=website_url, max_depth=2, limit=20, allow_backward_links=True, collection_name=collection_name, collection_description=collection_description)

    # Step 3: Query the loaded data
    question = "What is Milvus?"  # Replace with your actual question
    result = query(question)


if __name__ == "__main__":
    main()
```

--------------------------------

### Build DeepSearcher Docker Image

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/usage/deployment.md

This command builds a Docker image for DeepSearcher. It uses the `Dockerfile` located in the current directory and tags the resulting image as `deepsearcher:latest`. This step is necessary before running the application in a Docker container.

```shell
docker build -t deepsearcher:latest .
```

--------------------------------

### Configure Siliconflow Embedding Model

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

This snippet configures the Siliconflow embedding model. It requires the `SILICONFLOW_API_KEY` environment variable to be prepared. The example uses the `BAAI/bge-m3` model with this provider.

```Python
config.set_provider_config("embedding", "SiliconflowEmbedding", {"model": "BAAI/bge-m3"})
```

--------------------------------

### Configure Volcengine Embedding Model

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/embedding.md

Example of configuring the Volcengine embedding model with `doubao-embedding-text-240515`. This requires the `VOLCENGINE_API_KEY` environment variable.

```python
config.set_provider_config("embedding", "VolcengineEmbedding", {"model": "doubao-embedding-text-240515"})
```

--------------------------------

### Configure Ollama Embedding Provider

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Sets up the Ollama embedding provider with a specified model. Ensure the `ollama` Python package is installed before use.

```Python
config.set_provider_config("embedding", "OllamaEmbedding", {"model": "bge-m3"})
```

--------------------------------

### Access DeepSearcher API Endpoints via Swagger UI

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/usage/deployment.md

This section describes how to access and interact with the DeepSearcher API endpoints using the Swagger UI. Users can navigate to the `/docs` endpoint in their browser, view available APIs, and test them interactively by filling in parameters and executing requests. This provides a user-friendly way to explore and validate API functionality.

```APIDOC
API Access via Browser:
  Endpoint: http://localhost:8000/docs
  Functionality:
    - Displays Swagger UI with all available API endpoints.
    - Allows interactive testing of endpoints.
    - Steps:
      1. Open browser and navigate to the endpoint.
      2. The Swagger UI will display all available API endpoints.
      3. Click the "Try it out" button on any endpoint to interact with it.
      4. Fill in the required parameters and execute the request.
```

--------------------------------

### Configure Novita AI Embedding Model

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/embedding.md

Example of configuring the Novita AI embedding model with `baai/bge-m3`. This requires the `NOVITA_API_KEY` environment variable for authentication.

```python
config.set_provider_config("embedding", "NovitaEmbedding", {"model": "baai/bge-m3"})
```

--------------------------------

### Configure IBM WatsonX Embedding Model

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/embedding.md

Example of configuring the IBM WatsonX embedding model with `ibm/slate-125m-english-rtrvr-v2`. This requires the `ibm-watsonx-ai` Python package.

```python
config.set_provider_config("embedding", "WatsonXEmbedding", {"model": "ibm/slate-125m-english-rtrvr-v2"})
```

--------------------------------

### Configure Anthropic Claude LLM Provider

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Example of configuring Anthropic Claude. Ensure your `ANTHROPIC_API_KEY` is set as an environment variable. This configuration uses the 'claude-sonnet-4-0' model.

```Python
config.set_provider_config("llm", "Anthropic", {"model": "claude-sonnet-4-0"})
```

--------------------------------

### Configure DeepSearcher Components in Python

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/index.md

This snippet demonstrates the basic method for initializing and setting provider-specific configurations for DeepSearcher components using the `Configuration` class and `init_config` function. It shows how to create a configuration object, set a provider's options for a specific component, and then apply the configuration.

```python
from deepsearcher.configuration import Configuration, init_config

# Create configuration
config = Configuration()

# Set provider configurations
config.set_provider_config("[component]", "[provider]", {"option": "value"})

# Initialize with configuration
init_config(config=config)
```

--------------------------------

### Configure Novita AI Embedding Model

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

This example demonstrates configuring the Novita AI embedding model. It requires the `NOVITA_API_KEY` environment variable to be set. The model `baai/bge-m3` is specified for use with this provider.

```Python
config.set_provider_config("embedding", "NovitaEmbedding", {"model": "baai/bge-m3"})
```

--------------------------------

### Configure Google Gemini LLM

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/llm.md

This snippet provides the configuration for integrating the Google Gemini LLM. Prior to use, install the `google-genai` Python package and set the `GEMINI_API_KEY` environment variable.

```python
config.set_provider_config('llm', 'Gemini', { 'model': 'gemini-2.0-flash' })
```

--------------------------------

### Integrate Unstructured with DeepSearcher for Document Parsing

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/examples/unstructured.md

This Python script demonstrates how to integrate Unstructured with DeepSearcher to load and query documents. It initializes DeepSearcher's configuration, sets up Unstructured as the file loader, loads data from local files into a Milvus vector database, and then queries the loaded data. It also shows optional API key configuration for Unstructured's cloud service.

```python
import logging
import os
from deepsearcher.offline_loading import load_from_local_files
from deepsearcher.online_query import query
from deepsearcher.configuration import Configuration, init_config

# Suppress unnecessary logging from third-party libraries
logging.getLogger("httpx").setLevel(logging.WARNING)

# (Optional) Set API keys (ensure these are set securely in real applications)
os.environ['UNSTRUCTURED_API_KEY'] = '***************'
os.environ['UNSTRUCTURED_API_URL'] = '***************'


def main():
    # Step 1: Initialize configuration
    config = Configuration()

    # Configure Vector Database (Milvus) and File Loader (UnstructuredLoader)
    config.set_provider_config("vector_db", "Milvus", {})
    config.set_provider_config("file_loader", "UnstructuredLoader", {})

    # Apply the configuration
    init_config(config)

    # Step 2: Load data from a local file or directory into Milvus
    input_file = "your_local_file_or_directory"  # Replace with your actual file path
    collection_name = "Unstructured"
    collection_description = "All Milvus Documents"

    load_from_local_files(paths_or_directory=input_file, collection_name=collection_name, collection_description=collection_description)

    # Step 3: Query the loaded data
    question = "What is Milvus?"  # Replace with your actual question
    result = query(question)


if __name__ == "__main__":
    main()
```

--------------------------------

### Configure DeepSeek LLM Provider

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/llm.md

Example of configuring DeepSeek as the LLM provider in DeepSearcher, specifying the 'deepseek-reasoner' model. This configuration requires the `DEEPSEEK_API_KEY` environment variable to be set.

```python
config.set_provider_config("llm", "DeepSeek", {"model": "deepseek-reasoner"})
```

--------------------------------

### Add New Project Dependencies with uv (Shell)

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/contributing/index.md

Commands to add new dependencies to the `pyproject.toml` file. This covers adding standard dependencies and adding dependencies to specific optional extras, keeping the default installation lightweight.

```shell
uv add <package_name>
```

```shell
uv add <package_name> --optional <extra>
```

--------------------------------

### Configure GLM LLM

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/llm.md

This snippet provides the configuration for the GLM LLM. Before using, install the `zhipuai` Python package and ensure the `GLM_API_KEY` environment variable is set.

```python
config.set_provider_config("llm", "GLM", {"model": "glm-4-plus"})
```

--------------------------------

### Configure Google Gemini Embedding Model

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/embedding.md

Example of configuring the Google Gemini embedding model with `text-embedding-004`. This requires the `GEMINI_API_KEY` environment variable and the `google-genai` Python package.

```python
config.set_provider_config("embedding", "GeminiEmbedding", {"model": "text-embedding-004"})
```

--------------------------------

### Configure OpenAI LLM Provider

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/llm.md

Example of configuring OpenAI as the LLM provider in DeepSearcher, specifying the 'o1-mini' model. This configuration requires the `OPENAI_API_KEY` environment variable to be set.

```python
config.set_provider_config("llm", "OpenAI", {"model": "o1-mini"})
```

--------------------------------

### Configure Ollama LLM

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/llm.md

This snippet configures the Ollama LLM. This setup assumes you have a local Ollama instance running, with models pulled and accessible via its default REST API at `http://localhost:11434`.

```python
config.set_provider_config("llm", "Ollama", {"model": "qwq"})
```

--------------------------------

### Query DeepSearcher from Command Line

Source: https://github.com/zilliztech/deep-searcher/blob/master/README.md

Demonstrates how to run a query using the DeepSearcher command-line interface.

```shell
deepsearcher query "Write a report about xxx."
```

--------------------------------

### Configure Milvus Standalone Server Connection

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/configuration/vector_db.md

Connects to a standalone Milvus server by providing its network URI. This setup is suitable for larger datasets requiring a dedicated, self-hosted Milvus instance.

```python
config.set_provider_config("vector_db", "Milvus", {"uri": "http://localhost:19530", "token": ""})
```

--------------------------------

### Configure DeepSearcher LLM API Key in YAML

Source: https://github.com/zilliztech/deep-searcher/blob/master/docs/usage/deployment.md

This YAML snippet shows how to configure the Large Language Model (LLM) provider and API key for DeepSearcher. It specifies OpenAI as the provider and includes a placeholder for the API key. This configuration is crucial for the application to interact with the LLM service.

```yaml
llm:
  provider: "OpenAI"
  api_key: "your_openai_api_key_here"
```
Tokens:
Raw