### TreeSearch Python Quick Start Source: https://github.com/shibing624/treesearch/blob/main/README.md A basic example of how to initialize and use the TreeSearch Python library. It demonstrates searching for a query across specified directories and iterating through results. ```python from treesearch import TreeSearch # Just pass directories — auto-discovers all supported files ts = TreeSearch("project_root/", "docs/") results = ts.search("How does auth work?") for doc in results["documents"]: for node in doc["nodes"]: print(f"[{node['score']:.2f}] {node['title']}") print(f" {node['text'][:200]}") ``` -------------------------------- ### Clone and Install TreeSearch Source: https://github.com/shibing624/treesearch/blob/main/CONTRIBUTING.md Clone the repository and install the project in development mode with its dependencies. ```bash git clone https://github.com/shibing624/TreeSearch.git cd TreeSearch pip install -e ".[dev]" ``` -------------------------------- ### Install Voice Call Plugin from npm Source: https://github.com/shibing624/treesearch/blob/main/examples/data/markdowns/voice-call.md Use this command to install the voice-call plugin from npm. A Gateway restart is required after installation. ```bash openclaw plugins install @openclaw/voice-call ``` -------------------------------- ### Benchmark Execution Commands Source: https://github.com/shibing624/treesearch/blob/main/docs/tree_search_experiments.md Provides commands to install dependencies and run benchmarks for QASPER, FinanceBench, and CodeSearchNet using different strategies. ```bash # 安装依赖 pip install -e ".[dev,pdf]" # 运行 QASPER benchmark (含 Tree 模式) python examples/benchmark/qasper_benchmark.py \ --strategies fts5 tree \ --max-samples 50 --max-papers 20 # 运行 FinanceBench benchmark python examples/benchmark/financebench_benchmark.py \ --max-samples 150 # 运行 CodeSearchNet benchmark python examples/benchmark/codesearchnet_benchmark.py \ --max-samples 50 # 运行所有测试 pytest tests/ -v ``` -------------------------------- ### Install Voice Call Plugin from Local Folder Source: https://github.com/shibing624/treesearch/blob/main/examples/data/markdowns/voice-call.md Install the voice-call plugin from a local directory for development purposes. This method avoids copying files. A Gateway restart is required afterwards. ```bash openclaw plugins install ./extensions/voice-call cd ./extensions/voice-call && pnpm install ``` -------------------------------- ### Quick Start Search Source: https://github.com/shibing624/treesearch/blob/main/rust/README.md Perform a quick search in the current directory using the default auto mode. The query is 'How does auth work?'. ```bash ts "How does auth work?" . ``` -------------------------------- ### Install TreeSearch with Homebrew Source: https://github.com/shibing624/treesearch/blob/main/rust/README.md Install the treesearch CLI using Homebrew on macOS or Linux. After installation, you can run `ts --help` to see available commands. ```bash brew tap shibing624/tap brew install treesearch ts --help ``` -------------------------------- ### Search Mode Examples Source: https://github.com/shibing624/treesearch/blob/main/rust/README.md Illustrates how to use different search modes: auto (default), flat, and tree. The auto mode uses a three-layer decision logic to select the best retrieval strategy. ```bash ts "query" . # auto (default) ``` ```bash ts "query" . --mode flat # force flat ``` ```bash ts "query" . --mode tree # force tree ``` -------------------------------- ### Install TreeSearch with Cargo Source: https://github.com/shibing624/treesearch/blob/main/rust/README.md Install the treesearch CLI using Cargo, Rust's package manager. This method is suitable if you have Rust and Cargo installed. ```bash cargo install treesearch ts --help ``` -------------------------------- ### Install and Use TreeSearch CLI Source: https://github.com/shibing624/treesearch/blob/main/README.md Install the TreeSearch command-line interface using pip. This allows you to use TreeSearch directly from your terminal. Use '--help' for a list of commands. ```bash pip install -U pytreesearch treesearch --help ``` -------------------------------- ### Install TreeSearch Python Library Source: https://github.com/shibing624/treesearch/blob/main/README.md Use this command to install the TreeSearch Python library. This is required for using TreeSearch from Python scripts, backend services, or data pipelines. ```bash pip install -U pytreesearch ``` -------------------------------- ### TreeSearch CLI Common Commands Source: https://github.com/shibing624/treesearch/blob/main/README.md Examples of common commands for the TreeSearch CLI. These demonstrate searching for terms in specified directories and managing the search index. ```bash treesearch "How does auth work?" src/ docs/ ``` ```bash treesearch index --paths src/ docs/ ``` ```bash treesearch search --db ./index.db --query "auth" ``` -------------------------------- ### TreeSearch CLI Commands Source: https://github.com/shibing624/treesearch/blob/main/README.md Examples of using the TreeSearch command-line interface for indexing and searching. Supports default lazy indexing and searching, as well as advanced separate index building and searching. ```bash # Default mode: one command does everything (lazy index + search) treesearch "How does auth work?" src/ docs/ treesearch "configure Redis" project/ # With options treesearch "auth" src/ --max-nodes 10 --db ./my_index.db # Advanced: build index separately (for large codebases) treesearch index --paths src/ docs/ --add-description treesearch index --paths "docs/*.md" "src/**/*.py" --add-description # Advanced: search a pre-built index treesearch search --index_dir ./indexes/ --query "How does auth work?" ``` -------------------------------- ### Wildcard Query Examples Source: https://github.com/shibing624/treesearch/blob/main/rust/README.md Demonstrates various ways to use wildcard characters in search queries for prefix, contains-style, and raw FTS5 expressions. Invalid regex patterns will raise an error. ```bash ts "auth*" . ``` ```bash ts "*auth*" . ``` ```bash ts --regex "o?auth" . ``` ```bash ts --fts-expression "auth*" . ``` -------------------------------- ### Add New Dataset Loader Source: https://github.com/shibing624/treesearch/blob/main/examples/benchmark/README.md Example of how to implement a custom dataset loader function within `benchmark_utils.py`. This function should parse the dataset and return a list of BenchmarkSample objects. ```python # 1. 在 benchmark_utils.py 实现加载函数 def load_my_dataset(max_samples: int = 200) -> list[BenchmarkSample]: samples = [] # 解析数据集... return samples ``` -------------------------------- ### Document Tree Structure Example Source: https://github.com/shibing624/treesearch/blob/main/docs/architecture.md Illustrates the hierarchical structure of a document represented as a tree, with nodes for sections and parent-child relationships. ```text Document ├── Chapter 1: Introduction │ ├── 1.1 Background │ └── 1.2 Motivation ├── Chapter 2: Methods │ ├── 2.1 Data Collection │ └── 2.2 Model Design └── Chapter 3: Results ``` -------------------------------- ### TreeSearch Rust CLI Wildcard and Regex Queries Source: https://github.com/shibing624/treesearch/blob/main/README.md Examples of advanced query controls in the Rust CLI version of TreeSearch. These include using raw regex patterns or FTS5 expressions for more specific searches. ```bash ts --regex "o?auth" . ``` ```bash ts search --regex "o?auth" ``` ```bash ts --fts-expression "auth*" . ``` ```bash ts search --fts-expression "auth*" ``` -------------------------------- ### Build Index and Search Technical Docs Source: https://github.com/shibing624/treesearch/blob/main/README.md Use this to index directories containing technical documentation and then search for specific queries. Ensure the output directory exists. ```python from treesearch import build_index, search # 1. Build index — just pass directories (run once) docs = await build_index( paths=["docs/", "specs/"], output_dir="./indexes" ) # 2. Search — millisecond response result = await search( query="How to configure Redis cluster?", documents=docs, ) # 3. Results — complete sections, not fragments for doc in result["documents"]: print(f"Doc: {doc['doc_name']}") for node in doc["nodes"]: print(f" Section: {node['title']}") print(f" Content: {node['text'][:200]}...") ``` -------------------------------- ### Retain Section Example in Daily Log Source: https://github.com/shibing624/treesearch/blob/main/examples/data/markdowns/memory.md An example of how to structure the 'Retain' section at the end of a daily log file (YYYY-MM-DD.md) to capture narrative, self-contained facts. ```markdown ## Retain - The agent decided to prioritize the "warelay" task due to its potential impact on the project timeline. (decision; warelay) - User feedback on the "The Castle" prototype was generally positive, with specific suggestions for UI improvements. (feedback; The-Castle) - A new entity, "Peter", was identified as a key stakeholder in the upcoming "warelay" discussion. (entity; Peter) ``` -------------------------------- ### TreeSearch Python Initialization with Mixed Inputs Source: https://github.com/shibing624/treesearch/blob/main/README.md Demonstrates initializing TreeSearch with a mix of directories, specific files, and glob patterns. This allows for flexible input specification for the search index. ```python # All three input types work together ts = TreeSearch("src/", "docs/*.md", "README.md") results = ts.search("authentication") ``` -------------------------------- ### Run Benchmark with Samples Source: https://github.com/shibing624/treesearch/blob/main/examples/benchmark/README.md Call this function to execute benchmarks with specified samples, documents, and search strategies. It returns the benchmark reports. ```python reports = await run_benchmark_with_samples( samples=samples, documents=documents, strategies=["fts5", "tree", "auto"], ) ``` -------------------------------- ### Import TreeSearch in Python Source: https://github.com/shibing624/treesearch/blob/main/README.md Import the TreeSearch class in your Python code after installation. This is the entry point for using the library programmatically. ```python from treesearch import TreeSearch ``` -------------------------------- ### Initialize FTS5 Index Source: https://github.com/shibing624/treesearch/blob/main/docs/api.md Create an FTS5 index instance. Specify the database path for persistent storage. ```python from treesearch.fts import FTS5Index ts = FTS5Index(db_path="./index.db") ``` -------------------------------- ### Run QASPER Benchmark Source: https://github.com/shibing624/treesearch/blob/main/README.md Execute the benchmark for academic paper retrieval on the QASPER dataset. Includes an option to use embeddings. ```bash python examples/benchmark/qasper_benchmark.py --max-samples 50 --max-papers 20 --with-embedding ``` -------------------------------- ### Build Indexes from Glob Pattern (CLI) Source: https://github.com/shibing624/treesearch/blob/main/docs/api.md Use the CLI to build indexes from multiple Markdown files matching a glob pattern. Includes an option to add descriptions. ```bash treesearch index --paths "docs/*.md" --add-description ``` -------------------------------- ### Run HotpotQA Benchmark Source: https://github.com/shibing624/treesearch/blob/main/examples/benchmark/README.md Executes the HotpotQA benchmark for multi-hop reasoning questions. Specify the maximum number of samples for the benchmark. ```bash python examples/benchmark/hotpotqa_benchmark.py --max-samples 50 ``` -------------------------------- ### TreeSearch Get Indexed Files Method Source: https://github.com/shibing624/treesearch/blob/main/docs/api.md Retrieve a list of files that have been indexed by the `TreeSearch` instance. Each item in the list is a dictionary containing file information. ```python def get_indexed_files(self) -> list[dict] ``` -------------------------------- ### Run FinanceBench Benchmark Source: https://github.com/shibing624/treesearch/blob/main/README.md Execute the benchmark for financial document retrieval on the FinanceBench dataset. Includes an option to use embeddings. ```bash python examples/benchmark/financebench_benchmark.py --max-samples 50 --with-embedding ``` -------------------------------- ### Build Index from Mixed File Types (CLI) Source: https://github.com/shibing624/treesearch/blob/main/docs/api.md Use the CLI to build an index from various file types and specify an output directory for the indexes. ```bash treesearch index --paths docs/*.md paper.txt -o ./indexes ``` -------------------------------- ### Run FinanceBench Benchmark Source: https://github.com/shibing624/treesearch/blob/main/examples/benchmark/README.md Executes the FinanceBench benchmark for SEC financial reports. Specify the maximum number of samples for the benchmark. ```bash python examples/benchmark/financebench_benchmark.py --max-samples 50 ``` -------------------------------- ### Override OpenAI Model for Voice Calls Source: https://github.com/shibing624/treesearch/blob/main/examples/data/markdowns/voice-call.md This snippet shows a deep-merge example where only the OpenAI model and voice are overridden for voice calls, while keeping other ElevenLabs settings (if any) from the core TTS configuration. ```json5 { plugins: { entries: { "voice-call": { config: { tts: { openai: { model: "gpt-4o-mini-tts", voice: "marin" } } } } } } } ``` -------------------------------- ### FTS5Index: Full-Text Search Engine Demo Source: https://context7.com/shibing624/treesearch/llms.txt Demonstrates building, indexing, searching, and maintaining an FTS5 index. Includes keyword search, advanced FTS5 syntax, aggregations, scoring, and index management. ```python import asyncio from treesearch import FTS5Index, Document, md_to_tree, load_index, save_index async def fts_demo(): # Build a tree from Markdown and save to DB result = await md_to_tree("docs/voice-call.md", if_add_node_summary=True) save_index(result, "indexes/voice-call.db") # Load saved document doc = load_index("indexes/voice-call.db") # returns a Document # Create FTS5 index (persistent or in-memory) fts = FTS5Index(db_path="indexes/fts.db") # persistent # fts = FTS5Index() # in-memory (db_path=None) # Index documents fts.index_documents([doc]) # Simple keyword search with BM25 scoring results = fts.search("authentication config", top_k=5) for r in results: print(f"[{r['fts_score']:.4f}] {r['title']} (depth={r['depth']})") # Advanced FTS5 syntax: AND, OR, NOT, NEAR, phrase quotes, column filters results = fts.search("", fts_expression='title:auth AND body:config', top_k=5) results = fts.search("", fts_expression='"voice call" webhook', top_k=3) results = fts.search("", fts_expression='NEAR(auth config, 3)', top_k=5) # Per-document aggregation: total hits, best score, avg score agg = fts.search_with_aggregation("authentication", group_by_doc=True, top_k=10) for doc_agg in agg: print(f"{doc_agg['doc_name']}: {doc_agg['hit_count']} hits, " f"best={doc_agg['best_score']:.4f}, avg={doc_agg['avg_score']:.4f}") # score_nodes: PreFilter protocol — returns {node_id: score} dict scores = fts.score_nodes("auth", doc.doc_id) # {node_id: normalized_score} # Batch scoring across all docs in one SQL query (fast path for tree search) batch_scores = fts.score_nodes_batch( "configuration", doc_ids=[doc.doc_id], ancestor_decay=0.6, # propagate child scores to parent nodes ) # {doc_id: {node_id: score}} # Build FTS5 expression programmatically expr = FTS5Index.build_fts_expression( ["auth", "login"], operator="AND", column="title", # restrict to title column only prefix=True, # "auth*" matches auth, authentication, ... ) print(expr) # "title : (auth* AND login*)" expr_near = FTS5Index.build_fts_expression( ["machine", "learning"], near_distance=5, # NEAR(machine learning, 5) ) # Delete a document atomically (clears fts_nodes, nodes, documents, index_meta) deleted = fts.delete_document(doc.doc_id) # True if existed, False if not found # Index health check and repair report = fts.verify_index() print(f"Healthy: {report['healthy']}") if not report["healthy"]: removed = fts.repair_index(drop_missing_files=False) print(f"Repaired: {removed}") # Maintenance fts.optimize() # FTS5 merge optimization fts.rebuild() # rebuild FTS5 index from scratch fts.wal_checkpoint("TRUNCATE") # fold WAL back into DB stats = fts.get_stats() print(f"{stats['document_count']} documents, {stats['node_count']} nodes") fts.close() asyncio.run(fts_demo()) ``` -------------------------------- ### Run QASPER Benchmark Source: https://github.com/shibing624/treesearch/blob/main/examples/benchmark/README.md Executes the QASPER benchmark for academic paper QA, including comparison with embedding strategies. Adjust max samples and include embedding if needed. ```bash python examples/benchmark/qasper_benchmark.py --strategies fts5 tree auto --max-samples 50 --with-embedding ``` -------------------------------- ### Agent Tool Voice Call Actions Source: https://github.com/shibing624/treesearch/blob/main/examples/data/markdowns/voice-call.md These are the available actions for the `voice_call` agent tool. They correspond to different voice call operations like initiating, continuing, speaking, ending, and getting status. ```text Tool name: `voice_call` Actions: - `initiate_call` (message, to?, mode?) - `continue_call` (callId, message) - `speak_to_user` (callId, message) - `end_call` (callId) - `get_status` (callId) ``` -------------------------------- ### Build Index from Markdown (CLI) Source: https://github.com/shibing624/treesearch/blob/main/docs/api.md Use the CLI to build an index from a single Markdown file. ```bash treesearch index --paths document.md ``` -------------------------------- ### Override TTS for Voice Calls (ElevenLabs with API Key) Source: https://github.com/shibing624/treesearch/blob/main/examples/data/markdowns/voice-call.md This example demonstrates how to override the TTS provider to ElevenLabs exclusively for voice calls, including providing an API key and specific voice/model details. It ensures core TTS settings remain unchanged for other parts of the application. ```json5 { plugins: { entries: { "voice-call": { config: { tts: { provider: "elevenlabs", elevenlabs: { apiKey: "elevenlabs_key", voiceId: "pMsXgVXv3BLzUgSXRplE", modelId: "eleven_multilingual_v2" } } } } } } } ``` -------------------------------- ### TreeSearchConfig: Inspecting and Setting Global Configuration Source: https://context7.com/shibing624/treesearch/llms.txt Shows how to inspect default configuration values and override them globally using `set_config`. Supports custom FTS5 weights and CJK tokenization settings. ```python from treesearch import TreeSearchConfig, set_config, reset_config, get_config # --- Inspect defaults --- cfg = get_config() print(cfg.search_mode) # "auto" print(cfg.max_nodes_per_doc) # 5 print(cfg.fts_body_weight) # 10.0 # --- Override globally for aggressive tree expansion --- set_config(TreeSearchConfig( search_mode="tree", anchor_top_k=8, # max anchor nodes per document max_expansions=60, # max tree walk expansions max_hops=5, # max depth offset from anchor max_siblings=4, # max sibling nodes per step path_top_k=5, # top paths to return min_frontier_score=0.05, )) # --- Custom FTS5 column weights --- set_config(TreeSearchConfig( fts_title_weight=10.0, # boost title matches more fts_body_weight=5.0, fts_code_weight=3.0, fts_summary_weight=2.0, fts_front_matter_weight=1.0, )) # --- Chinese tokenization with custom jieba dictionary --- set_config(TreeSearchConfig( cjk_tokenizer="jieba", jieba_user_dict_paths=["domain_terms.txt"], # jieba format: "word [freq] [tag]" jieba_user_words=["石墨烯", "量子计算 1000"], # in-memory terms jieba_del_words=["的"], # words to remove )) ``` -------------------------------- ### Optimization Phases Overview Source: https://github.com/shibing624/treesearch/blob/main/docs/tree_search_experiments.md Outlines the two phases of optimization, detailing the techniques implemented in each phase and their orthogonal relationship. ```text Phase 1 (已完成): Generic Demotion (×0.60) → 压低 QASPER 虚高 generic sections Title-Prefix Propagation (30%) → 补偿 ::: 子节点 Walk Boost (15%) → FTS5+Walk 双确认加分 Term Density Boost (10%) → 多术语查询加分 Phase 2 (本轮): Walk-Only Injection → ★ 新增:发现 FTS5 漏检节点 Source-Type Degrade → ★ 新增:code 自动退化 Phase 2 不修改 Phase 1 的任何参数,只在 Stage 3 的 Walk Boost 分支中增加了 `fts_s == 0` 的处理路径。两轮优化完全正交。 ``` -------------------------------- ### Wildcard and Regex Queries Source: https://context7.com/shibing624/treesearch/llms.txt Supports prefix, contains-style, and raw regex queries for flexible searching. ```bash treesearch "auth*" src/ # prefix matching ``` ```bash treesearch "*auth*" src/ # contains-style ``` ```bash treesearch --regex "o?auth" src/ # raw Python regex ``` ```bash treesearch --fts-expression "auth* AND config*" src/ # raw FTS5 syntax ``` -------------------------------- ### TreeSearch Initialization and Basic Search Source: https://context7.com/shibing624/treesearch/llms.txt Initialize TreeSearch with source paths and perform a basic keyword search. The index is auto-built on the first search and persisted to an SQLite file. ```APIDOC ## TreeSearch Initialization and Basic Search ### Description Initialize TreeSearch with source paths (directories, files, or glob patterns) and perform a basic keyword search. The index is auto-built on the first search and persisted to an SQLite file. ### Method `TreeSearch(source_paths..., db_path=None)` `search(query, search_mode='auto', ...)` ### Parameters #### TreeSearch Initialization - **source_paths** (str or list[str]) - Paths to directories, files, or glob patterns to index. - **db_path** (str, optional) - Path to the SQLite database file. If None, operates in memory. #### Search Method - **query** (str) - The search query string. - **search_mode** (str, optional) - The retrieval mode ('flat', 'tree', 'auto'). Defaults to 'auto'. ### Request Example ```python from treesearch import TreeSearch ts = TreeSearch("src/", "docs/", "README.md") results = ts.search("How does authentication work?") ``` ### Response #### Success Response - **documents** (list[dict]) - A list of documents containing search results. - **doc_name** (str) - The name of the document. - **nodes** (list[dict]) - A list of nodes within the document. - **score** (float) - The relevance score of the node. - **title** (str) - The title of the node. - **line_start** (int) - The starting line number of the node. - **line_end** (int) - The ending line number of the node. - **text** (str) - The text content of the node (truncated). ### Response Example ```json { "documents": [ { "doc_name": "example.md", "nodes": [ { "score": 0.85, "title": "Authentication", "line_start": 10, "line_end": 25, "text": "This section explains how authentication works..." } ] } ] } ``` ``` -------------------------------- ### Async and Sync Search with search and search_sync Source: https://context7.com/shibing624/treesearch/llms.txt Perform searches using the `search` (async) or `search_sync` (sync) functions. Supports various options like custom filters, regex queries, and ancestor inclusion. The `search` function requires `asyncio`. ```python import asyncio from treesearch import build_index, search, search_sync, GrepFilter async def demo(): docs = await build_index(["src/", "docs/"]) # Async search result = await search( query="user login authentication", documents=docs, top_k_docs=5, max_nodes_per_doc=5, value_threshold=0.3, text_mode="full", # include full node text include_ancestors=True, # include parent section titles merge_strategy="interleave", ) # Result structure: # { # "documents": [ # { # "doc_id": "auth_service", # "doc_name": "auth_service", # "nodes": [ # {"node_id": "0003", "title": "UserAuthenticator.login", # "text": "def login(username, password):\n ...", "score": 0.87, "line_start": 42, "line_end": 68} # ] # } # ], # "query": "user login authentication", # "mode": "flat", # } # Custom GrepFilter for precise symbol matching grep_filter = GrepFilter(docs) result_grep = await search( query="def authenticate", documents=docs, pre_filter=grep_filter, ) # Regex query via CLI/search flags result_regex = await search( query="o?auth", documents=docs, regex=True, # treat query as raw regex ) asyncio.run(demo()) # Synchronous wrapper — no asyncio needed result = search_sync( query="cache invalidation strategy", documents=docs, max_nodes_per_doc=3, ) for doc in result["documents"]: for node in doc["nodes"]: print(f"[{node['score']:.2f}] {node['title']}: {node['text'][:150]}") ``` -------------------------------- ### TreeSearch CLI Wildcard and Regex Queries Source: https://github.com/shibing624/treesearch/blob/main/README.md Demonstrates advanced query options in the TreeSearch CLI, including wildcard shortcuts for prefix and contains matching, and explicit regex or FTS5 expression usage. ```bash treesearch --regex "o?auth" src/ ``` ```bash treesearch search --query "o?auth" --regex ``` ```bash treesearch --fts-expression "auth*" src/ ``` ```bash treesearch search --fts-expression "auth*" ``` -------------------------------- ### Search with Persistent FTS5 Database (CLI) Source: https://github.com/shibing624/treesearch/blob/main/docs/api.md Use the CLI to search an index directory with FTS5, utilizing a persistent FTS5 database file. ```bash treesearch search --index_dir ./indexes/ --query "auth" --fts --fts-db ./indexes/fts.db ``` -------------------------------- ### Search with FTS5 (CLI) Source: https://github.com/shibing624/treesearch/blob/main/docs/api.md Use the CLI to search an index directory using FTS5 with a given query. ```bash treesearch search --index_dir ./indexes/ --query "How does authentication work?" ``` -------------------------------- ### Basic TreeSearch Usage and Search Source: https://context7.com/shibing624/treesearch/llms.txt Initialize TreeSearch with directories, files, or glob patterns. The first search automatically builds and persists the index to './index.db'. Results include document names, node scores, titles, line numbers, and text snippets. ```python from treesearch import TreeSearch # --- Basic usage: pass directories, files, or glob patterns --- ts = TreeSearch("src/", "docs/", "README.md") # First search auto-builds the index and persists it to ./index.db results = ts.search("How does authentication work?") for doc in results["documents"]: print(f"\n[{doc['doc_name']}]") for node in doc["nodes"]: print(f" [{node['score']:.2f}] {node['title']}") print(f" lines {node['line_start']}-{node['line_end']}") print(f" {node['text'][:200]}") ``` -------------------------------- ### Persist and Load Tree Indexes with Tree Utilities Source: https://context7.com/shibing624/treesearch/llms.txt Use `save_index` and `load_index` to persist and load document trees, and `load_documents` to load multiple indexes from a directory. Includes utilities like `print_toc`, `flatten_tree`, and node navigation. ```python import asyncio from treesearch import ( md_to_tree, save_index, load_index, load_documents, flatten_tree, print_toc, Document, ) async def utilities_demo(): # Build and persist a tree result = await md_to_tree("docs/architecture.md", if_add_node_text=True) save_index(result, "indexes/architecture.db") # Load back as a Document object doc = load_index("indexes/architecture.db") print(f"Loaded: {doc.doc_name}, {len(doc.structure)} root nodes") print(f"source_type: {doc.source_type}") print(f"source_path: {doc.source_path}") # Load multiple documents from a directory of .db files all_docs = load_documents("indexes/") # returns list[Document] # Print table of contents (tree-style indented outline) print_toc(doc.structure) # Output: # Installation # Python Library # Rust CLI # Quick Start # In-Memory Mode # Tree Mode # Architecture # Flat Mode # Tree Mode # Flatten tree into a list of all nodes (depth-first) all_nodes = flatten_tree(doc.structure) print(f"Total nodes: {len(all_nodes)}") for node in all_nodes[:3]: print(f" [{node['node_id']}] {node['title']} ({len(node.get('text',''))} chars)") # Get tree without text fields (for inspection/serialization) skeleton = doc.get_tree_without_text() # Find a node by ID node = doc.get_node_by_id("0003") if node: print(f"Node: {node['title']}") asyncio.run(utilities_demo()) ``` -------------------------------- ### Tree Search Benchmark Configuration Source: https://github.com/shibing624/treesearch/blob/main/docs/tree_search_experiments.md Configuration for Tree Search optimized for benchmarking, utilizing larger parameters for more comprehensive evaluation. ```python TreeSearchConfig( path_top_k=max(k_values), anchor_top_k=max(k_values), max_expansions=60, ) ``` -------------------------------- ### Rust CLI: Basic Search Source: https://context7.com/shibing624/treesearch/llms.txt Use the standalone Rust CLI binary 'ts' for quick searches without a Python environment. ```bash ts "How does auth work?" src/ docs/ ``` -------------------------------- ### Run TreeSearch Tests Source: https://github.com/shibing624/treesearch/blob/main/CONTRIBUTING.md Execute the test suite for the TreeSearch project with verbose output and short traceback. ```bash pytest tests/ -v ``` ```bash pytest tests/ -v --tb=short ``` -------------------------------- ### Search a Pre-built Index Source: https://context7.com/shibing624/treesearch/llms.txt Perform searches against an existing index file. Supports various query types and result limiting. ```bash treesearch search --db ./indexes/index.db --query "auth" ``` ```bash treesearch search --db ./indexes/index.db --query "auth" --regex ``` ```bash treesearch search --db ./indexes/index.db --fts-expression "title:auth*" ``` ```bash treesearch search --db ./indexes/index.db --query "user login" --top-k 10 ``` -------------------------------- ### Async Batch Indexing with build_index Source: https://context7.com/shibing624/treesearch/llms.txt Use `build_index` to concurrently index multiple files, directories, or glob patterns into a SQLite database. Supports incremental updates and provides detailed statistics. Ensure `asyncio` is used for execution. ```python import asyncio from treesearch import build_index, search async def main(): # Index directories and glob patterns concurrently docs = await build_index( paths=["docs/", "src/**/*.py", "specs/*.md"], output_dir="./indexes", # default: ./indexes/index.db if_add_node_summary=True, if_add_doc_description=True, if_add_node_text=True, max_concurrency=8, # default: os.cpu_count() force=False, # skip unchanged files (incremental) ) # Access index statistics stats = docs.stats print(stats.summary()) # Output: # Index Statistics # Total files discovered: 42 # Indexed (new/changed): 38 # Skipped (unchanged): 4 # Total nodes generated: 1247 # Total time: 0.831s # Database: ./indexes/index.db (512.0 KB) # # Per file type: # markdown 12 file(s) 384 nodes 0.210s # code 26 file(s) 863 nodes 0.621s # Search the indexed documents result = await search( query="How to configure Redis cluster?", documents=docs, top_k_docs=3, max_nodes_per_doc=5, value_threshold=0.3, # minimum relevance score merge_strategy="interleave", # "interleave" | "per_doc" | "global_score" ) for doc in result["documents"]: print(f"\nDoc: {doc['doc_name']}") for node in doc["nodes"]: print(f" Section: {node['title']}") print(f" Score: {node['score']:.4f}") print(f" Text: {node['text'][:300]}...") asyncio.run(main()) ``` -------------------------------- ### Adaptive Degradation for Code Sources Source: https://github.com/shibing624/treesearch/blob/main/docs/tree_search_experiments.md Explains how adaptive degradation bypasses Tree Searcher instantiation and Walk loops for code, relying solely on FTS5 for scoring and ranking. ```text 而自适应降级让 code 直接走 flat path: - 省去 TreeSearcher 实例化 + Walk 循环的开销 - 分数完全由 FTS5 决定,排序与 flat mode 完全一致 - CodeSearchNet 上 Tree 模式 == FTS5 模式(MRR 一致) ``` -------------------------------- ### Configure Optional Tool Allowlist Source: https://github.com/shibing624/treesearch/blob/main/examples/data/markdowns/agent-tools.md Enable optional tools for an agent by adding their names or plugin identifiers to the agent's tool allowlist in the configuration. ```json { agents: { list: [ { id: "main", tools: { allow: [ "workflow_tool", // specific tool name "workflow", // plugin id (enables all tools from that plugin) "group:plugins" // all plugin tools ] } } ] } } ``` -------------------------------- ### Register a Basic Agent Tool Source: https://github.com/shibing624/treesearch/blob/main/examples/data/markdowns/agent-tools.md Register a tool that is always available to the LLM. Ensure tool names do not conflict with core tools. ```typescript import { Type } from "@sinclair/typebox"; export default function (api) { api.registerTool({ name: "my_tool", description: "Do a thing", parameters: Type.Object({ input: Type.String(), }), async execute(_id, params) { return { content: [{ type: "text", text: params.input }] }; }, }); } ``` -------------------------------- ### FTS5 Standalone Usage with TreeSearch Source: https://github.com/shibing624/treesearch/blob/main/README.md Demonstrates building an index from Markdown, loading saved documents, and performing FTS5 searches. Requires `treesearch` and optionally `PyMuPDF`, `python-docx`, `beautifulsoup4` for full functionality. ```python from treesearch import FTS5Index, Document, load_index, save_index, md_to_tree import asyncio # Option 1: Build from Markdown and save to DB result = asyncio.run(md_to_tree(md_path="docs/voice-call.md", if_add_node_summary=True)) save_index(result, "indexes/voice-call.db") # Option 2: Load a previously saved document from DB doc = load_index("indexes/voice-call.db") # returns a Document object # Create FTS5 index and search ts = FTS5Index(db_path="indexes/fts.db") # persistent, or omit for in-memory ts.index_documents([doc]) # Simple keyword search results = fts.search("authentication config", top_k=5) for r in results: print(f"[{r['fts_score']:.4f}] {r['title']}") # Advanced FTS5 query syntax results = fts.search("auth", fts_expression='title:auth AND body:config', top_k=5) # Per-document aggregation agg = fts.search_with_aggregation("authentication", group_by_doc=True) for doc_agg in agg: print(f"{doc_agg['doc_name']}: {doc_agg['hit_count']} hits, best={doc_agg['best_score']:.4f}") ``` -------------------------------- ### Build Index Separately Source: https://context7.com/shibing624/treesearch/llms.txt For large codebases, build the index in a separate step. Supports glob patterns and forcing re-indexing. ```bash treesearch index --paths src/ docs/ --add-description ``` ```bash treesearch index --paths "docs/*.md" "src/**/*.py" ``` ```bash treesearch index --paths . --force # force re-index all files ``` -------------------------------- ### Query-Aware Demotion Implementation Source: https://github.com/shibing624/treesearch/blob/main/docs/tree_search_experiments.md Implements a query-aware demotion strategy in Stage 1b to prevent demoting sections when the query explicitly targets them. ```python # Stage 1b — query-aware demotion if plan and plan.terms: base_title = title.split(" ::: ")[0].strip().lower() if any(t in base_title for t in plan.terms): continue # Skip demotion — query targets this section node_scores[key] *= 0.70 ``` -------------------------------- ### Preview Files to be Indexed Source: https://context7.com/shibing624/treesearch/llms.txt Use `resolve_glob_files()` to preview which files would be indexed by TreeSearch based on the provided paths and glob patterns, before initiating the actual indexing process. ```python # --- Preview which files would be indexed (before indexing) --- files = ts.resolve_glob_files() print(f"{len(files)} files matched") ``` -------------------------------- ### In-Memory Mode TreeSearch Initialization Source: https://github.com/shibing624/treesearch/blob/main/README.md Use in-memory mode by setting `db_path=None` to avoid writing an index file to disk. This is suitable for quick searches or ephemeral use cases. Performance is excellent, but indexes are lost when the process exits. ```python # In-memory mode — no index.db file, all indexes kept in memory ts = TreeSearch("docs/", db_path=None) results = ts.search("voice calls") ``` -------------------------------- ### build_index Source: https://github.com/shibing624/treesearch/blob/main/docs/api.md Build tree indexes for multiple files concurrently. Supports glob patterns for file selection and auto-dispatches parsers based on file extension. ```APIDOC ## build_index ### Description Build tree indexes for multiple files concurrently. Accepts glob patterns (e.g. `["docs/*.md", "src/**/*.py"]`). Auto-dispatches via `ParserRegistry` based on file extension. ### Parameters #### Path Parameters - **paths** (list[str]) - Required - A list of file paths or glob patterns. #### Keyword Arguments - **output_dir** (str) - Optional - The directory to save the indexes. Defaults to "./indexes". - **if_add_node_summary** (bool) - Optional - Whether to add a summary to each node. Defaults to True. - **if_add_doc_description** (bool) - Optional - Whether to add a description to the document. Defaults to True. - **if_add_node_text** (bool) - Optional - Whether to add the full text to each node. Defaults to True. - **if_add_node_id** (bool) - Optional - Whether to add an ID to each node. Defaults to True. - **max_concurrency** (int) - Optional - Maximum number of concurrent builds. Defaults to `min(os.cpu_count(), 64)`. - **force** (bool) - Optional - Whether to force re-indexing. Defaults to False. ### Returns - **list[Document]**: A list of `Document` objects representing the built indexes. ``` -------------------------------- ### File Indexing and Management Source: https://context7.com/shibing624/treesearch/llms.txt Preview files to be indexed, retrieve a list of currently indexed files, and force re-indexing of specific files. ```APIDOC ## File Indexing and Management ### Description Manage the files indexed by TreeSearch. Preview which files match glob patterns, retrieve a list of files already in the index, and force re-indexing of updated files. ### Methods - `resolve_glob_files()` - `get_indexed_files()` - `index(file_path)` ### Parameters #### `index(file_path)` Method - **file_path** (str or list[str]) - Path(s) to the file(s) to be re-indexed. ### Request Example ```python # Preview files that would be indexed files = ts.resolve_glob_files() print(f"{len(files)} files matched") # Get list of currently indexed files indexed = ts.get_indexed_files() for info in indexed: print(f"[{info['source_type']:>8}] {info['source_path']}") # Force re-index a specific file ts.index("docs/updated_file.md") ``` ### Response #### `resolve_glob_files()` Response - Returns a list of file paths matching the glob patterns provided during initialization. #### `get_indexed_files()` Response - **indexed** (list[dict]) - A list of dictionaries, each containing information about an indexed file. - **source_type** (str) - The type of the source file (e.g., 'markdown', 'python'). - **source_path** (str) - The path to the indexed file. - **doc_id** (str) - The unique identifier for the document in the index. #### `index()` Response - Returns the number of files successfully re-indexed. ``` -------------------------------- ### Show Traversal Paths in Tree Mode Source: https://context7.com/shibing624/treesearch/llms.txt Use the --show-path flag in tree mode to visualize the traversal path of search results. ```bash treesearch "webhook URL" docs/ --show-path ``` -------------------------------- ### build_index — Async Batch Indexing Source: https://context7.com/shibing624/treesearch/llms.txt Concurrently indexes multiple files, directories, or glob patterns into a persistent SQLite database. Supports incremental updates, orphan pruning, and move/rename detection. ```APIDOC ## `build_index` — Async Batch Indexing Concurrently indexes multiple files, directories, or glob patterns into a persistent SQLite database. Returns a list of `Document` objects ready for `search()`. Supports incremental updates (only re-indexes changed files), orphan pruning, and move/rename detection. ```python import asyncio from treesearch import build_index, search async def main(): # Index directories and glob patterns concurrently docs = await build_index( paths=["docs/", "src/**/*.py", "specs/*.md"], output_dir="./indexes", # default: ./indexes/index.db if_add_node_summary=True, if_add_doc_description=True, if_add_node_text=True, max_concurrency=8, # default: os.cpu_count() force=False, # skip unchanged files (incremental) ) # Access index statistics stats = docs.stats print(stats.summary()) # Output: # Index Statistics # Total files discovered: 42 # Indexed (new/changed): 38 # Skipped (unchanged): 4 # Total nodes generated: 1247 # Total time: 0.831s # Database: ./indexes/index.db (512.0 KB) # # Per file type: # markdown 12 file(s) 384 nodes 0.210s # code 26 file(s) 863 nodes 0.621s # Search the indexed documents result = await search( query="How to configure Redis cluster?", documents=docs, top_k_docs=3, max_nodes_per_doc=5, value_threshold=0.3, # minimum relevance score merge_strategy="interleave", # "interleave" | "per_doc" | "global_score" ) for doc in result["documents"]: print(f"\nDoc: {doc['doc_name']}") for node in doc["nodes"]: print(f" Section: {node['title']}") print(f" Score: {node['score']:.4f}") print(f" Text: {node['text'][:300]}...") asyncio.run(main()) ``` ``` -------------------------------- ### High-level TreeSearch API Initialization Source: https://github.com/shibing624/treesearch/blob/main/docs/api.md Initialize the `TreeSearch` class for a high-level API that combines indexing and search. It supports glob patterns for file selection and an optional persistent SQLite database for incremental updates. ```python class TreeSearch: def __init__( self, *patterns: str, db_path: str = "", ) ```