structchunk

https://github.com/yzp0111/structchunk
Admin
structchunk is a structure-aware markdown text chunker for RAG pipelines that preserves document...
Tokens:21,850
Snippets:197
Trust Score:3.6
License:MIT
Update:3 weeks ago
Show doc for...
Tokens:
Raw
### Install structchunk from source

Source: https://github.com/yzp0111/structchunk/blob/master/README.md

Clones the repository and installs the package in editable mode with test dependencies.

```bash
git clone https://github.com/yzp0111/structchunk
cd structchunk
pip install -e ".[test]"
```

--------------------------------

### Install structchunk with uv from PyPI

Source: https://github.com/yzp0111/structchunk/blob/master/docs/installation.md

Installs structchunk using the uv package manager. Requires uv to be installed.

```bash
uv pip install structchunk
```

--------------------------------

### Install package with test dependencies

Source: https://github.com/yzp0111/structchunk/blob/master/CONTRIBUTING.md

Install the package in editable mode with test dependencies.

```bash
pip install -e ".[test]"
```

--------------------------------

### Install built structchunk wheel

Source: https://github.com/yzp0111/structchunk/blob/master/docs/installation.md

Installs the wheel built in the previous step from the dist/ directory.

```bash
pip install dist/structchunk-0.1.0-py3-none-any.whl
```

--------------------------------

### Install structchunk from source with uv

Source: https://github.com/yzp0111/structchunk/blob/master/docs/installation.md

Clones the repository, creates a virtual environment with uv, activates it, and installs in editable mode with the test extra.

```bash
git clone https://github.com/yzp0111/structchunk.git
cd structchunk
uv venv .venv
source .venv/bin/activate
uv pip install -e ".[test]"
```

--------------------------------

### Install structchunk from source with pip

Source: https://github.com/yzp0111/structchunk/blob/master/docs/installation.md

Clones the repository and installs in editable mode with the test extra, which installs pytest>=7.0 for running the test suite.

```bash
git clone https://github.com/yzp0111/structchunk.git
cd structchunk
pip install -e ".[test]"
```

--------------------------------

### Output of structchunk.chunk() quick start

Source: https://github.com/yzp0111/structchunk/blob/master/README.md

Example output from the quick start example using the default `hierarchical` algorithm. The breadcrumb includes the `#` prefix and the H1 title is present in every chunk.

```text
[0] ['# Title']
# Title

Some content with a long paragraph that needs splitting.
```

--------------------------------

### Build structchunk wheel and sdist from source

Source: https://github.com/yzp0111/structchunk/blob/master/docs/installation.md

Clones the repository, installs the build package, and builds a wheel and sdist. The output is placed in the dist/ directory.

```bash
git clone https://github.com/yzp0111/structchunk.git
cd structchunk
python -m pip install build
python -m build
```

--------------------------------

### Example output filenames

Source: https://github.com/yzp0111/structchunk/blob/master/docs/cli.md

Example output filenames when chunking report.md with the hierarchical algorithm at 14:32:05 on 2026-06-12.

```text
report-切块结果-hierarchical-20260612_143205.json
report-切块结果-hierarchical-20260612_143205.md
```

--------------------------------

### Install structchunk from PyPI with pip

Source: https://github.com/yzp0111/structchunk/blob/master/docs/installation.md

Installs the latest release of structchunk from PyPI. This is the simplest method for most users.

```bash
pip install structchunk
```

--------------------------------

### structchunk CLI usage examples

Source: https://github.com/yzp0111/structchunk/blob/master/README.md

Shows the available command-line options for the structchunk CLI. The default algorithm is hierarchical with a 500-character cap, and the default output directory is ./test_result/. Use --quiet to suppress the summary table and --output-dir to specify a custom directory.

```bash
structchunk document.md                                       # hierarchical, 500c cap
structchunk document.md --algorithm linear                    # greedy block-by-block
structchunk document.md --max-chars 300 --format json          # 300c cap, JSON output
structchunk document.md --quiet                                # suppress summary
structchunk document.md --output-dir /tmp/chunks               # custom output directory
```

--------------------------------

### Verify structchunk CLI availability

Source: https://github.com/yzp0111/structchunk/blob/master/docs/installation.md

Displays the CLI help to confirm the command-line interface is installed.

```bash
structchunk --help
```

--------------------------------

### Chunk chain linking example for a long code block

Source: https://github.com/yzp0111/structchunk/blob/master/docs/metadata.md

Shows how three chunks A, B, C are linked via continuation, has_more, prev_chunk_id, and next_chunk_id. Use this to understand the linked-list pointers for walking a chain forward or backward.

```python
A.continuation = False
A.has_more = True
A.next_chunk_id = B.chunk_id

B.continuation = True
B.has_more = True
B.prev_chunk_id = A.chunk_id
B.next_chunk_id = C.chunk_id

C.continuation = True
C.has_more = False
C.prev_chunk_id = B.chunk_id
```

--------------------------------

### Breadcrumb header list example

Source: https://github.com/yzp0111/structchunk/blob/master/docs/metadata.md

Provides an example of the 'header_breadcrumb' field, which is a list of strings representing the markdown-level prefix, such as the heading hierarchy from H1 to H2.

```python
["# Title", "## Section"]
```

--------------------------------

### Create feature branch

Source: https://github.com/yzp0111/structchunk/blob/master/CONTRIBUTING.md

Creates a new feature branch from master. Use this command at the start of development work.

```bash
git checkout -b feat/your-feature-name
```

--------------------------------

### chunk_linear(content: str, **kwargs) -> list[MarkdownChunk]

Source: https://github.com/yzp0111/structchunk/blob/master/docs/api.md

Greedy block-by-block chunking. Each block (paragraph, table, list, code fence) is added to the current chunk until the size cap is reached, then a new chunk starts. Oversized blocks are delegated to type-specific sub-splitters.

```APIDOC
## chunk_linear(content: str, **kwargs) -> list[MarkdownChunk]

### Description
Greedy block-by-block chunking. Each block (paragraph, table, list, code fence) is added to the current chunk until the size cap is reached, then a new chunk starts. Oversized blocks are delegated to type-specific sub-splitters: paragraphs split at sentence boundaries, tables at row boundaries, lists at item boundaries, code fences at line boundaries.

### Method
Function

### Signature
chunk_linear(content: str, **kwargs) -> list[MarkdownChunk]

### Parameters
- **content** (str) - Required - The content to chunk.
- **kwargs** - Optional - Additional keyword arguments (e.g., size cap).

### Returns
- **list[MarkdownChunk]** - A list of MarkdownChunk objects.
```

--------------------------------

### Clone the repository

Source: https://github.com/yzp0111/structchunk/blob/master/CONTRIBUTING.md

Clone your fork of the repository and navigate into the project directory.

```bash
git clone https://github.com/your-username/structchunk.git
cd structchunk
```

--------------------------------

### Check documentation heading parity with scripts/check_doc_sync.py

Source: https://github.com/yzp0111/structchunk/blob/master/CONTRIBUTING.md

Run this script from the project root to verify that every heading in the English documentation has a matching heading in the Chinese translation. If it reports mismatches, fix them before committing; out-of-sync docs will be held up in review.

```bash
python scripts/check_doc_sync.py
```

--------------------------------

### structchunk CLI synopsis

Source: https://github.com/yzp0111/structchunk/blob/master/docs/cli.md

Shows the general command-line syntax for the structchunk CLI. The input argument is required and must be a UTF-8 encoded markdown file.

```bash
structchunk <input> [flags]
```

--------------------------------

### Hello World: chunk a markdown string and print chunks

Source: https://github.com/yzp0111/structchunk/blob/master/docs/quickstart.md

Chunks a markdown string with the default hierarchical algorithm and prints each chunk's index, header breadcrumb, and content. The output shows the expected result for the given input.

```python
import structchunk

chunks = structchunk.chunk(
    "# Title\n\nSome content with a long paragraph that needs splitting.",
    max_chars=500,
)

for c in chunks:
    print(f"[{c.metadata.chunk_index}] {c.metadata.header_breadcrumb}")
    print(c.content)
    print()
```

```text
[0] ['# Title']
# Title

Some content with a long paragraph that needs splitting.
```

--------------------------------

### MongoDB: Insert chunks document

Source: https://github.com/yzp0111/structchunk/blob/master/docs/database-schema.md

JavaScript example for inserting a chunks document in MongoDB. Uses NumberLong (BSON int64) for chunk_id, native BSON array for header_breadcrumb, and array of floats for embedding. For vector search, consider MongoDB Atlas Vector Search or pgvector-mongo integrations.

```javascript
db.chunks.insertOne({
    chunk_id: NumberLong("188704004562132992"),
    doc_id: "abc123def4567890",
    chunk_index: 0,
    total_chunks: 5,
    content: "...",
    content_length: 487,
    total_length: 487,
    header_breadcrumb: ["# Title", "## Section"],
    embedding: [...]  // array of floats
})
```

--------------------------------

### chunk_linear() function signature

Source: https://github.com/yzp0111/structchunk/blob/master/docs/api.md

Greedy block-by-block chunking. Each block (paragraph, table, list, code fence) is added to the current chunk until the size cap is reached, then a new chunk starts. Oversized blocks are delegated to type-specific sub-splitters: paragraphs split at sentence boundaries, tables at row boundaries, lists at item boundaries, code fences at line boundaries.

```python
chunk_linear(content: str, **kwargs) -> list[MarkdownChunk]
```

--------------------------------

### Choose algorithm: hierarchical vs linear

Source: https://github.com/yzp0111/structchunk/blob/master/docs/quickstart.md

Shows how to switch between the two built-in algorithms using the `algorithm` keyword. `hierarchical` is the default and is structure-aware; `linear` is greedy and delegates oversized blocks to type-specific sub-splitters.

```python
# Hierarchical (default — section-aware)
chunks = structchunk.chunk(content, algorithm="hierarchical", max_chars=500)

# Linear (greedy)
chunks = structchunk.chunk(content, algorithm="linear", max_chars=500)
```

--------------------------------

### Project layout

Source: https://github.com/yzp0111/structchunk/blob/master/CONTRIBUTING.md

Directory tree showing the project layout. Core chunking logic is in linear.py and hierarchical.py; private modules handle parsing, models, and splitting.

```text
src/structchunk/
├── __init__.py         # public API: chunk, chunk_file, chunk_to_dicts
├── _version.py         # __version__
├── _config.py          # ChunkerConfig
├── _models.py          # AtomicBlock, BlockType, MarkdownChunk, ChunkMetadata
├── _parser.py          # _parse_atomic_blocks, regex constants
├── _splitters.py       # sentence / table / list / code sub-splitters
├── _enrich.py          # breadcrumb, RAG metadata
├── linear.py           # linear (greedy) chunker
├── hierarchical.py     # hierarchical (tree) chunker
└── _cli.py             # CLI entry point
examples/
└── basic.py            # basic usage demo (both algorithms)
tests/                  # pytest test suite
```

--------------------------------

### Run tests

Source: https://github.com/yzp0111/structchunk/blob/master/CONTRIBUTING.md

Run the test suite. All 192 tests should pass; requires Python >= 3.9 and pytest >= 7.0.

```bash
pytest
```

--------------------------------

### Output file naming pattern

Source: https://github.com/yzp0111/structchunk/blob/master/docs/cli.md

Shows the naming pattern for output files. The stem is the source file's name without extension, followed by the algorithm and a timestamp.

```text
<stem>-切块结果-<algorithm>-<YYYYMMDD_HHMMSS>.json
<stem>-切块结果-<algorithm>-<YYYYMMDD_HHMMSS>.md
```

--------------------------------

### Chunk markdown text with structchunk.chunk()

Source: https://github.com/yzp0111/structchunk/blob/master/README.md

Demonstrates the main entry point `chunk()` which splits markdown text into `MarkdownChunk` objects. The `max_chars` parameter caps chunk size. The output shows the chunk index and header breadcrumb.

```python
import structchunk

chunks = structchunk.chunk(
    "# Title\n\nSome content with a long paragraph that needs splitting.",
    max_chars=500,
)

for c in chunks:
    print(f"[{c.metadata.chunk_index}] {c.metadata.header_breadcrumb}")
    print(c.content)
    print()
```

--------------------------------

### Import structchunk and access public names

Source: https://github.com/yzp0111/structchunk/blob/master/docs/api.md

Shows how to import the structchunk package and access its public names, such as chunk(), MarkdownChunk, and BlockType.

```python
import structchunk

structchunk.chunk(...)
structchunk.MarkdownChunk
structchunk.BlockType
# etc.
```

--------------------------------

### chunk()

Source: https://github.com/yzp0111/structchunk/blob/master/docs/api.md

Chunk markdown content for RAG. This is the recommended entry point for most users.

```APIDOC
## chunk()

### Description
Chunk markdown content for RAG. This is the recommended entry point for most users.

### Function Signature
```python
chunk(
    content: str,
    *,
    algorithm: str = "hierarchical",
    max_chars: int | None = None,
    forward_intro_text: bool = True,
    **kwargs,
) -> list[MarkdownChunk]
```

### Parameters
- **content** (str) - Required - The raw markdown text to chunk.
- **algorithm** (str) - Optional, default `"hierarchical"` - `"linear"` for greedy block-by-block chunking; `"hierarchical"` for document-tree based, semantically coherent chunks.
- **max_chars** (int | None) - Optional, default `None` - Convenience cap (e.g. 500 for Dify). If set, every chunk is guaranteed to be at most `max_chars` characters. Sets both `max_chunk_size` (80%) and `hard_max_size` (100%).
- **forward_intro_text** (bool) - Optional, default `True` - Whether to forward introductory text for tables and lists (see `ChunkerConfig`).
- **kwargs** - Optional - Additional keyword arguments forwarded to the selected algorithm's chunk function. If `doc_id` is not provided, it is auto-derived from a SHA-256 hash of the content (first 16 hex characters).

### Returns
- `list[MarkdownChunk]` - The list of chunks.

### Raises
- `ValueError` - If `algorithm` is not `"linear"` or `"hierarchical"`, or if `max_chars` is set to a non-positive value.
```

--------------------------------

### Run specific test file

Source: https://github.com/yzp0111/structchunk/blob/master/CONTRIBUTING.md

Runs a specific test file with verbose output. Useful for debugging individual test failures.

```bash
pytest tests/test_structchunk.py -v
```

--------------------------------

### Verify structchunk import and version

Source: https://github.com/yzp0111/structchunk/blob/master/docs/installation.md

Checks that the package imports and prints its version. Expected output is 0.1.0.

```bash
python -c "import structchunk; print(structchunk.__version__)"
```

--------------------------------

### structchunk CLI Synopsis

Source: https://github.com/yzp0111/structchunk/blob/master/docs/cli.md

The primary command invocation for the structchunk command-line tool.

```APIDOC
## structchunk <input> [flags]

### Description
Run the structchunk chunking algorithm on a markdown file. Reads the input file, chunks it, and saves results in JSON and/or Markdown formats.

### Method
CLI Command

### Arguments
- **input** (string) - Required - Path to the input markdown file (must be UTF-8).

### Options
- **--algorithm** (string: "linear" or "hierarchical") - Optional - Default: "hierarchical" - Chunking algorithm to use.
- **--max-chars** (integer) - Optional - Default: 500 - Hard cap on chunk size in characters.
- **--format** (string: "json", "md", or "both") - Optional - Default: "both" - Output format(s).
- **--quiet** (boolean) - Optional - Default: false - Suppress the chunk summary printout.
- **--output-dir** (string) - Optional - Default: "./test_result/" - Directory for output files (created if not exists).

### Examples

#### Default usage
```
structchunk document.md
```

#### Use the linear algorithm
```
structchunk document.md --algorithm linear
```

#### Tighter chunk size
```
structchunk document.md --max-chars 300
```

#### JSON output only
```
structchunk document.md --format json
```

#### Custom output directory
```
structchunk document.md --output-dir /tmp/chunks
```

### Output Files
Output filenames include the source file's stem, the algorithm, and a timestamp:

- `<stem>-切块结果-<algorithm>-<YYYYMMDD_HHMMSS>.json`
- `<stem>-切块结果-<algorithm>-<YYYYMMDD_HHMMSS>.md`

The `json` file contains an array of chunk dicts (the same shape as `chunk_to_dicts()`). The `md` file is a human-readable summary with one section per chunk.

### Exit Codes
| Code | Meaning |
|------|---------|
| `0` | Success. Output files written. |
| `1` | Failure. Input file not found or not UTF-8 decodeable. Error printed to stderr. |

```

--------------------------------

### Use linear algorithm: structchunk --algorithm linear

Source: https://github.com/yzp0111/structchunk/blob/master/docs/cli.md

Uses the linear (greedy) chunking algorithm instead of the default hierarchical one.

```bash
structchunk document.md --algorithm linear
```

--------------------------------

### Customize chunking with keyword arguments

Source: https://github.com/yzp0111/structchunk/blob/master/docs/quickstart.md

Customizes chunking with common keyword arguments: `max_chars` sets the hard cap, `forward_intro_text` forwards leading paragraphs to table/list continuations, and `doc_id` overrides the auto-derived document ID.

```python
chunks = structchunk.chunk(
    content,
    max_chars=500,           # hard cap (Dify-friendly)
    forward_intro_text=True, # forward leading paragraph to table/list continuations
    doc_id="my-doc-001",     # override the auto-derived doc_id
)
```

--------------------------------

### Package version constant

Source: https://github.com/yzp0111/structchunk/blob/master/docs/api.md

Defines the package version string following Semantic Versioning. The current version is 0.1.0.

```python
__version__: str = "0.1.0"
```

--------------------------------

### Chunk content with hierarchical and linear algorithms

Source: https://github.com/yzp0111/structchunk/blob/master/README.md

Shows the two chunking algorithms available in structchunk. The hierarchical algorithm is the default and builds a section tree from the document's header hierarchy, walking it bottom-up to emit one chunk per section that fits within the size cap. The linear algorithm uses greedy block-by-block assembly, adding each block until the size cap is exceeded, and delegates oversized blocks to type-specific sub-splitters.

```python
# Hierarchical (default, section-tree based)
chunks = structchunk.chunk(content, algorithm="hierarchical", max_chars=500)

# Linear (greedy block-by-block)
chunks = structchunk.chunk(content, algorithm="linear", max_chars=500)
```

--------------------------------

### Default usage: structchunk document.md

Source: https://github.com/yzp0111/structchunk/blob/master/docs/cli.md

Runs the default chunking: hierarchical algorithm, max-chars=500, and saves both JSON and Markdown outputs to ./test_result/.

```bash
structchunk document.md
```

--------------------------------

### Chunk ID int, string, and hex conversion

Source: https://github.com/yzp0111/structchunk/blob/master/docs/metadata.md

Shows how to access the raw chunk ID as an integer and convert it to decimal string or 16-character hex via chunk_id_str and chunk_id_hex. chunk_id is a 64-bit Snowflake int; 0 is a sentinel meaning no predecessor/successor.

```python
c = chunks[0]
print(c.metadata.chunk_id)        # int, e.g. 188704004562132992
print(c.chunk_id_str)             # str, "188704004562132992"
print(c.chunk_id_hex)             # str, "029f0c9d210b8000" (16 hex chars)
```

--------------------------------

### Custom output directory: structchunk --output-dir

Source: https://github.com/yzp0111/structchunk/blob/master/docs/cli.md

Writes output files to a custom directory (/tmp/chunks) instead of the default ./test_result/. The directory is created if it does not exist.

```bash
structchunk document.md --output-dir /tmp/chunks
```

--------------------------------

### Forward Linked-List Traversal

Source: https://github.com/yzp0111/structchunk/blob/master/docs/metadata.md

Walks the chunk chain forward from the first chunk, printing each chunk's ID and first 50 characters. Uses metadata.next_chunk_id to find the next chunk, stopping when no next ID exists.

```python
chunks = structchunk.chunk(content)
current = chunks[0]
while current is not None:
    print(current.chunk_id_str, current.content[:50])
    next_id = current.metadata.next_chunk_id
    current = next(
        (c for c in chunks if c.metadata.chunk_id == next_id),
        None,
    ) if next_id else None
```

--------------------------------

### JSON output only: structchunk --format json

Source: https://github.com/yzp0111/structchunk/blob/master/docs/cli.md

Emits only a single JSON file containing all chunks, instead of both JSON and Markdown.

```bash
structchunk document.md --format json
```

--------------------------------

### chunk_file()

Source: https://github.com/yzp0111/structchunk/blob/master/docs/api.md

Read a markdown file and chunk it. The file's absolute path is used as `doc_id` unless the caller provides one explicitly.

```APIDOC
## chunk_file()

### Description
Read a markdown file and chunk it. The file's absolute path is used as `doc_id` unless the caller provides one explicitly.

### Function Signature
```python
chunk_file(
    path: str,
    *,
    algorithm: str = "hierarchical",
    forward_intro_text: bool = True,
    **kwargs,
) -> list[MarkdownChunk]
```

### Parameters
- **path** (str) - Required - The path to the markdown file to read and chunk.
- **algorithm** (str) - Optional, default `"hierarchical"` - The chunking algorithm to use.
- **forward_intro_text** (bool) - Optional, default `True` - Whether to forward introductory text for tables and lists.
- **kwargs** - Optional - Additional keyword arguments forwarded to the selected algorithm's chunk function.

### Returns
- `list[MarkdownChunk]` - The list of chunks.
```

--------------------------------

### Backward Linked-List Traversal

Source: https://github.com/yzp0111/structchunk/blob/master/docs/metadata.md

Walks the chunk chain backward from the last chunk, printing each chunk's ID and first 50 characters. Uses metadata.prev_chunk_id to find the previous chunk, stopping when no previous ID exists.

```python
chunks = structchunk.chunk(content)
current = chunks[-1]
while current is not None:
    print(current.chunk_id_str, current.content[:50])
    prev_id = current.metadata.prev_chunk_id
    current = next(
        (c for c in chunks if c.metadata.chunk_id == prev_id),
        None,
    ) if prev_id else None
```

--------------------------------

### chunk() function signature

Source: https://github.com/yzp0111/structchunk/blob/master/docs/api.md

Signature for the chunk() function, the primary entry point for chunking markdown content for RAG. It accepts content, algorithm, max_chars, forward_intro_text, and additional kwargs.

```python
chunk(
    content: str,
    *,
    algorithm: str = "hierarchical",
    max_chars: int | None = None,
    forward_intro_text: bool = True,
    **kwargs,
) -> list[MarkdownChunk]
```
Tokens:
Raw