### Install structchunk from source
Source: https://github.com/yzp0111/structchunk/blob/master/README.md
Clones the repository and installs the package in editable mode with test dependencies.
```bash
git clone https://github.com/yzp0111/structchunk
cd structchunk
pip install -e ".[test]"
```
--------------------------------
### Install structchunk with uv from PyPI
Source: https://github.com/yzp0111/structchunk/blob/master/docs/installation.md
Installs structchunk using the uv package manager. Requires uv to be installed.
```bash
uv pip install structchunk
```
--------------------------------
### Install package with test dependencies
Source: https://github.com/yzp0111/structchunk/blob/master/CONTRIBUTING.md
Install the package in editable mode with test dependencies.
```bash
pip install -e ".[test]"
```
--------------------------------
### Install built structchunk wheel
Source: https://github.com/yzp0111/structchunk/blob/master/docs/installation.md
Installs the wheel built in the previous step from the dist/ directory.
```bash
pip install dist/structchunk-0.1.0-py3-none-any.whl
```
--------------------------------
### Install structchunk from source with uv
Source: https://github.com/yzp0111/structchunk/blob/master/docs/installation.md
Clones the repository, creates a virtual environment with uv, activates it, and installs in editable mode with the test extra.
```bash
git clone https://github.com/yzp0111/structchunk.git
cd structchunk
uv venv .venv
source .venv/bin/activate
uv pip install -e ".[test]"
```
--------------------------------
### Install structchunk from source with pip
Source: https://github.com/yzp0111/structchunk/blob/master/docs/installation.md
Clones the repository and installs in editable mode with the test extra, which installs pytest>=7.0 for running the test suite.
```bash
git clone https://github.com/yzp0111/structchunk.git
cd structchunk
pip install -e ".[test]"
```
--------------------------------
### Output of structchunk.chunk() quick start
Source: https://github.com/yzp0111/structchunk/blob/master/README.md
Example output from the quick start example using the default `hierarchical` algorithm. The breadcrumb includes the `#` prefix and the H1 title is present in every chunk.
```text
[0] ['# Title']
# Title
Some content with a long paragraph that needs splitting.
```
--------------------------------
### Build structchunk wheel and sdist from source
Source: https://github.com/yzp0111/structchunk/blob/master/docs/installation.md
Clones the repository, installs the build package, and builds a wheel and sdist. The output is placed in the dist/ directory.
```bash
git clone https://github.com/yzp0111/structchunk.git
cd structchunk
python -m pip install build
python -m build
```
--------------------------------
### Example output filenames
Source: https://github.com/yzp0111/structchunk/blob/master/docs/cli.md
Example output filenames when chunking report.md with the hierarchical algorithm at 14:32:05 on 2026-06-12.
```text
report-切块结果-hierarchical-20260612_143205.json
report-切块结果-hierarchical-20260612_143205.md
```
--------------------------------
### Install structchunk from PyPI with pip
Source: https://github.com/yzp0111/structchunk/blob/master/docs/installation.md
Installs the latest release of structchunk from PyPI. This is the simplest method for most users.
```bash
pip install structchunk
```
--------------------------------
### structchunk CLI usage examples
Source: https://github.com/yzp0111/structchunk/blob/master/README.md
Shows the available command-line options for the structchunk CLI. The default algorithm is hierarchical with a 500-character cap, and the default output directory is ./test_result/. Use --quiet to suppress the summary table and --output-dir to specify a custom directory.
```bash
structchunk document.md # hierarchical, 500c cap
structchunk document.md --algorithm linear # greedy block-by-block
structchunk document.md --max-chars 300 --format json # 300c cap, JSON output
structchunk document.md --quiet # suppress summary
structchunk document.md --output-dir /tmp/chunks # custom output directory
```
--------------------------------
### Verify structchunk CLI availability
Source: https://github.com/yzp0111/structchunk/blob/master/docs/installation.md
Displays the CLI help to confirm the command-line interface is installed.
```bash
structchunk --help
```
--------------------------------
### Chunk chain linking example for a long code block
Source: https://github.com/yzp0111/structchunk/blob/master/docs/metadata.md
Shows how three chunks A, B, C are linked via continuation, has_more, prev_chunk_id, and next_chunk_id. Use this to understand the linked-list pointers for walking a chain forward or backward.
```python
A.continuation = False
A.has_more = True
A.next_chunk_id = B.chunk_id
B.continuation = True
B.has_more = True
B.prev_chunk_id = A.chunk_id
B.next_chunk_id = C.chunk_id
C.continuation = True
C.has_more = False
C.prev_chunk_id = B.chunk_id
```
--------------------------------
### Breadcrumb header list example
Source: https://github.com/yzp0111/structchunk/blob/master/docs/metadata.md
Provides an example of the 'header_breadcrumb' field, which is a list of strings representing the markdown-level prefix, such as the heading hierarchy from H1 to H2.
```python
["# Title", "## Section"]
```
--------------------------------
### Create feature branch
Source: https://github.com/yzp0111/structchunk/blob/master/CONTRIBUTING.md
Creates a new feature branch from master. Use this command at the start of development work.
```bash
git checkout -b feat/your-feature-name
```
--------------------------------
### chunk_linear(content: str, **kwargs) -> list[MarkdownChunk]
Source: https://github.com/yzp0111/structchunk/blob/master/docs/api.md
Greedy block-by-block chunking. Each block (paragraph, table, list, code fence) is added to the current chunk until the size cap is reached, then a new chunk starts. Oversized blocks are delegated to type-specific sub-splitters.
```APIDOC
## chunk_linear(content: str, **kwargs) -> list[MarkdownChunk]
### Description
Greedy block-by-block chunking. Each block (paragraph, table, list, code fence) is added to the current chunk until the size cap is reached, then a new chunk starts. Oversized blocks are delegated to type-specific sub-splitters: paragraphs split at sentence boundaries, tables at row boundaries, lists at item boundaries, code fences at line boundaries.
### Method
Function
### Signature
chunk_linear(content: str, **kwargs) -> list[MarkdownChunk]
### Parameters
- **content** (str) - Required - The content to chunk.
- **kwargs** - Optional - Additional keyword arguments (e.g., size cap).
### Returns
- **list[MarkdownChunk]** - A list of MarkdownChunk objects.
```
--------------------------------
### chunk_linear() function signature
Source: https://github.com/yzp0111/structchunk/blob/master/docs/api.md
Greedy block-by-block chunking. Each block (paragraph, table, list, code fence) is added to the current chunk until the size cap is reached, then a new chunk starts. Oversized blocks are delegated to type-specific sub-splitters: paragraphs split at sentence boundaries, tables at row boundaries, lists at item boundaries, code fences at line boundaries.
```python
chunk_linear(content: str, **kwargs) -> list[MarkdownChunk]
```
--------------------------------
### MongoDB: Insert chunks document
Source: https://github.com/yzp0111/structchunk/blob/master/docs/database-schema.md
JavaScript example for inserting a chunks document in MongoDB. Uses NumberLong (BSON int64) for chunk_id, native BSON array for header_breadcrumb, and array of floats for embedding. For vector search, consider MongoDB Atlas Vector Search or pgvector-mongo integrations.
```javascript
db.chunks.insertOne({
chunk_id: NumberLong("188704004562132992"),
doc_id: "abc123def4567890",
chunk_index: 0,
total_chunks: 5,
content: "...",
content_length: 487,
total_length: 487,
header_breadcrumb: ["# Title", "## Section"],
embedding: [...] // array of floats
})
```
--------------------------------
### Clone the repository
Source: https://github.com/yzp0111/structchunk/blob/master/CONTRIBUTING.md
Clone your fork of the repository and navigate into the project directory.
```bash
git clone https://github.com/your-username/structchunk.git
cd structchunk
```
--------------------------------
### Check documentation heading parity with scripts/check_doc_sync.py
Source: https://github.com/yzp0111/structchunk/blob/master/CONTRIBUTING.md
Run this script from the project root to verify that every heading in the English documentation has a matching heading in the Chinese translation. If it reports mismatches, fix them before committing; out-of-sync docs will be held up in review.
```bash
python scripts/check_doc_sync.py
```
--------------------------------
### structchunk CLI synopsis
Source: https://github.com/yzp0111/structchunk/blob/master/docs/cli.md
Shows the general command-line syntax for the structchunk CLI. The input argument is required and must be a UTF-8 encoded markdown file.
```bash
structchunk [flags]
```
--------------------------------
### Hello World: chunk a markdown string and print chunks
Source: https://github.com/yzp0111/structchunk/blob/master/docs/quickstart.md
Chunks a markdown string with the default hierarchical algorithm and prints each chunk's index, header breadcrumb, and content. The output shows the expected result for the given input.
```python
import structchunk
chunks = structchunk.chunk(
"# Title\n\nSome content with a long paragraph that needs splitting.",
max_chars=500,
)
for c in chunks:
print(f"[{c.metadata.chunk_index}] {c.metadata.header_breadcrumb}")
print(c.content)
print()
```
```text
[0] ['# Title']
# Title
Some content with a long paragraph that needs splitting.
```
--------------------------------
### Choose algorithm: hierarchical vs linear
Source: https://github.com/yzp0111/structchunk/blob/master/docs/quickstart.md
Shows how to switch between the two built-in algorithms using the `algorithm` keyword. `hierarchical` is the default and is structure-aware; `linear` is greedy and delegates oversized blocks to type-specific sub-splitters.
```python
# Hierarchical (default — section-aware)
chunks = structchunk.chunk(content, algorithm="hierarchical", max_chars=500)
# Linear (greedy)
chunks = structchunk.chunk(content, algorithm="linear", max_chars=500)
```
--------------------------------
### Project layout
Source: https://github.com/yzp0111/structchunk/blob/master/CONTRIBUTING.md
Directory tree showing the project layout. Core chunking logic is in linear.py and hierarchical.py; private modules handle parsing, models, and splitting.
```text
src/structchunk/
├── __init__.py # public API: chunk, chunk_file, chunk_to_dicts
├── _version.py # __version__
├── _config.py # ChunkerConfig
├── _models.py # AtomicBlock, BlockType, MarkdownChunk, ChunkMetadata
├── _parser.py # _parse_atomic_blocks, regex constants
├── _splitters.py # sentence / table / list / code sub-splitters
├── _enrich.py # breadcrumb, RAG metadata
├── linear.py # linear (greedy) chunker
├── hierarchical.py # hierarchical (tree) chunker
└── _cli.py # CLI entry point
examples/
└── basic.py # basic usage demo (both algorithms)
tests/ # pytest test suite
```
--------------------------------
### Run tests
Source: https://github.com/yzp0111/structchunk/blob/master/CONTRIBUTING.md
Run the test suite. All 192 tests should pass; requires Python >= 3.9 and pytest >= 7.0.
```bash
pytest
```
--------------------------------
### Output file naming pattern
Source: https://github.com/yzp0111/structchunk/blob/master/docs/cli.md
Shows the naming pattern for output files. The stem is the source file's name without extension, followed by the algorithm and a timestamp.
```text
-切块结果--.json
-切块结果--.md
```
--------------------------------
### Chunk markdown text with structchunk.chunk()
Source: https://github.com/yzp0111/structchunk/blob/master/README.md
Demonstrates the main entry point `chunk()` which splits markdown text into `MarkdownChunk` objects. The `max_chars` parameter caps chunk size. The output shows the chunk index and header breadcrumb.
```python
import structchunk
chunks = structchunk.chunk(
"# Title\n\nSome content with a long paragraph that needs splitting.",
max_chars=500,
)
for c in chunks:
print(f"[{c.metadata.chunk_index}] {c.metadata.header_breadcrumb}")
print(c.content)
print()
```
--------------------------------
### Import structchunk and access public names
Source: https://github.com/yzp0111/structchunk/blob/master/docs/api.md
Shows how to import the structchunk package and access its public names, such as chunk(), MarkdownChunk, and BlockType.
```python
import structchunk
structchunk.chunk(...)
structchunk.MarkdownChunk
structchunk.BlockType
# etc.
```
--------------------------------
### chunk()
Source: https://github.com/yzp0111/structchunk/blob/master/docs/api.md
Chunk markdown content for RAG. This is the recommended entry point for most users.
```APIDOC
## chunk()
### Description
Chunk markdown content for RAG. This is the recommended entry point for most users.
### Function Signature
```python
chunk(
content: str,
*,
algorithm: str = "hierarchical",
max_chars: int | None = None,
forward_intro_text: bool = True,
**kwargs,
) -> list[MarkdownChunk]
```
### Parameters
- **content** (str) - Required - The raw markdown text to chunk.
- **algorithm** (str) - Optional, default `"hierarchical"` - `"linear"` for greedy block-by-block chunking; `"hierarchical"` for document-tree based, semantically coherent chunks.
- **max_chars** (int | None) - Optional, default `None` - Convenience cap (e.g. 500 for Dify). If set, every chunk is guaranteed to be at most `max_chars` characters. Sets both `max_chunk_size` (80%) and `hard_max_size` (100%).
- **forward_intro_text** (bool) - Optional, default `True` - Whether to forward introductory text for tables and lists (see `ChunkerConfig`).
- **kwargs** - Optional - Additional keyword arguments forwarded to the selected algorithm's chunk function. If `doc_id` is not provided, it is auto-derived from a SHA-256 hash of the content (first 16 hex characters).
### Returns
- `list[MarkdownChunk]` - The list of chunks.
### Raises
- `ValueError` - If `algorithm` is not `"linear"` or `"hierarchical"`, or if `max_chars` is set to a non-positive value.
```
--------------------------------
### Run specific test file
Source: https://github.com/yzp0111/structchunk/blob/master/CONTRIBUTING.md
Runs a specific test file with verbose output. Useful for debugging individual test failures.
```bash
pytest tests/test_structchunk.py -v
```
--------------------------------
### Verify structchunk import and version
Source: https://github.com/yzp0111/structchunk/blob/master/docs/installation.md
Checks that the package imports and prints its version. Expected output is 0.1.0.
```bash
python -c "import structchunk; print(structchunk.__version__)"
```
--------------------------------
### structchunk CLI Synopsis
Source: https://github.com/yzp0111/structchunk/blob/master/docs/cli.md
The primary command invocation for the structchunk command-line tool.
```APIDOC
## structchunk [flags]
### Description
Run the structchunk chunking algorithm on a markdown file. Reads the input file, chunks it, and saves results in JSON and/or Markdown formats.
### Method
CLI Command
### Arguments
- **input** (string) - Required - Path to the input markdown file (must be UTF-8).
### Options
- **--algorithm** (string: "linear" or "hierarchical") - Optional - Default: "hierarchical" - Chunking algorithm to use.
- **--max-chars** (integer) - Optional - Default: 500 - Hard cap on chunk size in characters.
- **--format** (string: "json", "md", or "both") - Optional - Default: "both" - Output format(s).
- **--quiet** (boolean) - Optional - Default: false - Suppress the chunk summary printout.
- **--output-dir** (string) - Optional - Default: "./test_result/" - Directory for output files (created if not exists).
### Examples
#### Default usage
```
structchunk document.md
```
#### Use the linear algorithm
```
structchunk document.md --algorithm linear
```
#### Tighter chunk size
```
structchunk document.md --max-chars 300
```
#### JSON output only
```
structchunk document.md --format json
```
#### Custom output directory
```
structchunk document.md --output-dir /tmp/chunks
```
### Output Files
Output filenames include the source file's stem, the algorithm, and a timestamp:
- `-切块结果--.json`
- `-切块结果--.md`
The `json` file contains an array of chunk dicts (the same shape as `chunk_to_dicts()`). The `md` file is a human-readable summary with one section per chunk.
### Exit Codes
| Code | Meaning |
|------|---------|
| `0` | Success. Output files written. |
| `1` | Failure. Input file not found or not UTF-8 decodeable. Error printed to stderr. |
```
--------------------------------
### Use linear algorithm: structchunk --algorithm linear
Source: https://github.com/yzp0111/structchunk/blob/master/docs/cli.md
Uses the linear (greedy) chunking algorithm instead of the default hierarchical one.
```bash
structchunk document.md --algorithm linear
```
--------------------------------
### Customize chunking with keyword arguments
Source: https://github.com/yzp0111/structchunk/blob/master/docs/quickstart.md
Customizes chunking with common keyword arguments: `max_chars` sets the hard cap, `forward_intro_text` forwards leading paragraphs to table/list continuations, and `doc_id` overrides the auto-derived document ID.
```python
chunks = structchunk.chunk(
content,
max_chars=500, # hard cap (Dify-friendly)
forward_intro_text=True, # forward leading paragraph to table/list continuations
doc_id="my-doc-001", # override the auto-derived doc_id
)
```
--------------------------------
### Package version constant
Source: https://github.com/yzp0111/structchunk/blob/master/docs/api.md
Defines the package version string following Semantic Versioning. The current version is 0.1.0.
```python
__version__: str = "0.1.0"
```
--------------------------------
### Chunk content with hierarchical and linear algorithms
Source: https://github.com/yzp0111/structchunk/blob/master/README.md
Shows the two chunking algorithms available in structchunk. The hierarchical algorithm is the default and builds a section tree from the document's header hierarchy, walking it bottom-up to emit one chunk per section that fits within the size cap. The linear algorithm uses greedy block-by-block assembly, adding each block until the size cap is exceeded, and delegates oversized blocks to type-specific sub-splitters.
```python
# Hierarchical (default, section-tree based)
chunks = structchunk.chunk(content, algorithm="hierarchical", max_chars=500)
# Linear (greedy block-by-block)
chunks = structchunk.chunk(content, algorithm="linear", max_chars=500)
```
--------------------------------
### Default usage: structchunk document.md
Source: https://github.com/yzp0111/structchunk/blob/master/docs/cli.md
Runs the default chunking: hierarchical algorithm, max-chars=500, and saves both JSON and Markdown outputs to ./test_result/.
```bash
structchunk document.md
```
--------------------------------
### Chunk ID int, string, and hex conversion
Source: https://github.com/yzp0111/structchunk/blob/master/docs/metadata.md
Shows how to access the raw chunk ID as an integer and convert it to decimal string or 16-character hex via chunk_id_str and chunk_id_hex. chunk_id is a 64-bit Snowflake int; 0 is a sentinel meaning no predecessor/successor.
```python
c = chunks[0]
print(c.metadata.chunk_id) # int, e.g. 188704004562132992
print(c.chunk_id_str) # str, "188704004562132992"
print(c.chunk_id_hex) # str, "029f0c9d210b8000" (16 hex chars)
```
--------------------------------
### Custom output directory: structchunk --output-dir
Source: https://github.com/yzp0111/structchunk/blob/master/docs/cli.md
Writes output files to a custom directory (/tmp/chunks) instead of the default ./test_result/. The directory is created if it does not exist.
```bash
structchunk document.md --output-dir /tmp/chunks
```
--------------------------------
### Forward Linked-List Traversal
Source: https://github.com/yzp0111/structchunk/blob/master/docs/metadata.md
Walks the chunk chain forward from the first chunk, printing each chunk's ID and first 50 characters. Uses metadata.next_chunk_id to find the next chunk, stopping when no next ID exists.
```python
chunks = structchunk.chunk(content)
current = chunks[0]
while current is not None:
print(current.chunk_id_str, current.content[:50])
next_id = current.metadata.next_chunk_id
current = next(
(c for c in chunks if c.metadata.chunk_id == next_id),
None,
) if next_id else None
```
--------------------------------
### JSON output only: structchunk --format json
Source: https://github.com/yzp0111/structchunk/blob/master/docs/cli.md
Emits only a single JSON file containing all chunks, instead of both JSON and Markdown.
```bash
structchunk document.md --format json
```
--------------------------------
### chunk_file()
Source: https://github.com/yzp0111/structchunk/blob/master/docs/api.md
Read a markdown file and chunk it. The file's absolute path is used as `doc_id` unless the caller provides one explicitly.
```APIDOC
## chunk_file()
### Description
Read a markdown file and chunk it. The file's absolute path is used as `doc_id` unless the caller provides one explicitly.
### Function Signature
```python
chunk_file(
path: str,
*,
algorithm: str = "hierarchical",
forward_intro_text: bool = True,
**kwargs,
) -> list[MarkdownChunk]
```
### Parameters
- **path** (str) - Required - The path to the markdown file to read and chunk.
- **algorithm** (str) - Optional, default `"hierarchical"` - The chunking algorithm to use.
- **forward_intro_text** (bool) - Optional, default `True` - Whether to forward introductory text for tables and lists.
- **kwargs** - Optional - Additional keyword arguments forwarded to the selected algorithm's chunk function.
### Returns
- `list[MarkdownChunk]` - The list of chunks.
```
--------------------------------
### Backward Linked-List Traversal
Source: https://github.com/yzp0111/structchunk/blob/master/docs/metadata.md
Walks the chunk chain backward from the last chunk, printing each chunk's ID and first 50 characters. Uses metadata.prev_chunk_id to find the previous chunk, stopping when no previous ID exists.
```python
chunks = structchunk.chunk(content)
current = chunks[-1]
while current is not None:
print(current.chunk_id_str, current.content[:50])
prev_id = current.metadata.prev_chunk_id
current = next(
(c for c in chunks if c.metadata.chunk_id == prev_id),
None,
) if prev_id else None
```
--------------------------------
### chunk() function signature
Source: https://github.com/yzp0111/structchunk/blob/master/docs/api.md
Signature for the chunk() function, the primary entry point for chunking markdown content for RAG. It accepts content, algorithm, max_chars, forward_intro_text, and additional kwargs.
```python
chunk(
content: str,
*,
algorithm: str = "hierarchical",
max_chars: int | None = None,
forward_intro_text: bool = True,
**kwargs,
) -> list[MarkdownChunk]
```
--------------------------------
### chunk_hierarchical(content: str, **kwargs) -> list[MarkdownChunk]
Source: https://github.com/yzp0111/structchunk/blob/master/docs/api.md
Document-tree based chunking. Builds a section tree from the document's header hierarchy, then walks it bottom-up, emitting one chunk per section that fits within the size cap. Oversized sections are sub-split at natural boundaries. Adjacent same-level sibling sections are greedily merged when they fit together.
```APIDOC
## chunk_hierarchical(content: str, **kwargs) -> list[MarkdownChunk]
### Description
Document-tree based chunking. Builds a section tree from the document's header hierarchy, then walks it bottom-up, emitting one chunk per section that fits within the size cap. Oversized sections are sub-split at natural boundaries. Adjacent same-level sibling sections are greedily merged when they fit together.
### Method
Function
### Signature
chunk_hierarchical(content: str, **kwargs) -> list[MarkdownChunk]
### Parameters
- **content** (str) - Required - The content to chunk.
- **kwargs** - Optional - Additional keyword arguments (e.g., size cap).
### Returns
- **list[MarkdownChunk]** - A list of MarkdownChunk objects.
```
--------------------------------
### Tighter chunk size: structchunk --max-chars 300
Source: https://github.com/yzp0111/structchunk/blob/master/docs/cli.md
Sets a hard cap of 300 characters per chunk, overriding the default of 500.
```bash
structchunk document.md --max-chars 300
```
--------------------------------
### Serialize chunks with structchunk.chunk_to_dicts()
Source: https://github.com/yzp0111/structchunk/blob/master/README.md
Converts a list of `MarkdownChunk` objects to dictionaries for JSON serialization.
```python
dicts = structchunk.chunk_to_dicts(chunks)
```
--------------------------------
### Structured header breadcrumb with HeaderInfo
Source: https://github.com/yzp0111/structchunk/blob/master/docs/metadata.md
Shows the 'header_path' field, a list of HeaderInfo objects with level and text, parallel to header_breadcrumb. This provides structured header information for each chunk.
```python
[
HeaderInfo(level=1, text="Title"),
HeaderInfo(level=2, text="Section"),
]
```
--------------------------------
### chunk_file() function signature
Source: https://github.com/yzp0111/structchunk/blob/master/docs/api.md
Signature for the chunk_file() function, which reads a markdown file and chunks it. The file's absolute path is used as doc_id unless provided explicitly.
```python
chunk_file(
path: str,
*,
algorithm: str = "hierarchical",
forward_intro_text: bool = True,
**kwargs,
) -> list[MarkdownChunk]
```
--------------------------------
### chunk_to_dicts()
Source: https://github.com/yzp0111/structchunk/blob/master/docs/api.md
Serialize a chunk list to JSON-friendly dicts. Each chunk is converted via `MarkdownChunk.to_dict()`. Identifier fields are emitted as decimal strings.
```APIDOC
## chunk_to_dicts()
### Description
Serialize a chunk list to JSON-friendly dicts. Each chunk is converted via `MarkdownChunk.to_dict()`. Identifier fields are emitted as decimal strings.
### Function Signature
```python
chunk_to_dicts(chunks: list[MarkdownChunk]) -> list[dict]
```
### Parameters
- **chunks** (list[MarkdownChunk]) - Required - The list of chunks to serialize.
### Returns
- `list[dict]` - The JSON-friendly dict representation of the chunks.
```
--------------------------------
### ChunkerConfig Dataclass
Source: https://github.com/yzp0111/structchunk/blob/master/docs/api.md
Configuration shared by the linear and hierarchical chunkers. Pass individual fields as keyword arguments to chunk(), chunk_linear(), or chunk_hierarchical().
```APIDOC
## ChunkerConfig Dataclass
### Description
Configuration shared by the linear and hierarchical chunkers. Pass individual fields as keyword arguments to `chunk()`, `chunk_linear()`, or `chunk_hierarchical()`.
### Fields
- **max_chunk_size** (int) - Default: `1500` - Soft upper limit on chunk size in characters.
- **hard_max_size** (int) - Default: `3000` - Absolute maximum. A chunk is never larger than this.
- **min_chunk_size** (int) - Default: `50` - Don't produce chunks smaller than this (merge with neighbours).
- **sub_split_table** (bool) - Default: `True` - Allow splitting oversized tables at row boundaries.
- **sub_split_code** (bool) - Default: `True` - Allow splitting oversized code blocks at line boundaries.
- **sub_split_list** (bool) - Default: `True` - Allow splitting oversized lists at item boundaries.
- **sub_split_paragraph** (bool) - Default: `True` - Allow splitting oversized paragraphs at sentence boundaries.
- **preserve_table_header** (bool) - Default: `True` - When a table is split across chunks, prepend the header row(s) to each continuation chunk.
- **preserve_code_fence** (bool) - Default: `True` - When a code block is split, re-open/close fences in each chunk.
- **doc_id** (str) - Default: `""` - Optional document identifier propagated to every chunk's metadata.
- **forward_intro_text** (bool) - Default: `True` - When a section's first table or list is immediately preceded by a plain-text paragraph (not a markdown header), the paragraph's last sentence is forwarded to split continuations as an implicit title. Set to `False` to disable.
```
--------------------------------
### chunk_hierarchical() function signature
Source: https://github.com/yzp0111/structchunk/blob/master/docs/api.md
Document-tree based chunking. Builds a section tree from the document's header hierarchy, then walks it bottom-up, emitting one chunk per section that fits within the size cap. Oversized sections are sub-split at natural boundaries. Adjacent same-level sibling sections are greedily merged when they fit together.
```python
chunk_hierarchical(content: str, **kwargs) -> list[MarkdownChunk]
```
--------------------------------
### chunk_to_dicts() function signature
Source: https://github.com/yzp0111/structchunk/blob/master/docs/api.md
Signature for the chunk_to_dicts() function, which serializes a chunk list to JSON-friendly dicts. Each chunk is converted via MarkdownChunk.to_dict(), and identifier fields are emitted as decimal strings.
```python
chunk_to_dicts(chunks: list[MarkdownChunk]) -> list[dict]
```
--------------------------------
### Chunk a file with chunk_file
Source: https://github.com/yzp0111/structchunk/blob/master/docs/quickstart.md
Chunks a file by path. The file's absolute path is used as the `doc_id` automatically, so the same path yields the same `doc_id`.
```python
chunks = structchunk.chunk_file("path/to/document.md", max_chars=500)
```
--------------------------------
### Serialize chunks to dicts with chunk_to_dicts
Source: https://github.com/yzp0111/structchunk/blob/master/docs/quickstart.md
Converts chunk objects to JSON-friendly dicts for storage or network transfer. Comments show how to access the content, Snowflake chunk ID, and header breadcrumb from the resulting dicts.
```python
dicts = structchunk.chunk_to_dicts(chunks)
# dicts[0]["content"] -> the text
# dicts[0]["metadata"]["chunk_id"] -> Snowflake id as decimal string
# dicts[0]["metadata"]["header_breadcrumb"] -> ['# Title', '## Section']
```
--------------------------------
### MySQL: Create chunks table
Source: https://github.com/yzp0111/structchunk/blob/master/docs/database-schema.md
SQL DDL for the chunks table in MySQL. Uses BIGINT for chunk_id, JSON for header_breadcrumb (MySQL 5.7+), and LONGBLOB for embedding. Note that BIGINT is the same as PostgreSQL.
```sql
CREATE TABLE chunks (
chunk_id BIGINT PRIMARY KEY,
doc_id VARCHAR(64) NOT NULL,
chunk_index INT NOT NULL,
total_chunks INT NOT NULL,
content TEXT NOT NULL,
content_length INT NOT NULL,
total_length INT NOT NULL,
header_breadcrumb JSON, -- store as JSON array
embedding LONGBLOB -- or use a vector extension
) ENGINE=InnoDB;
```
--------------------------------
### PostgreSQL schema with pgvector for structchunk
Source: https://github.com/yzp0111/structchunk/blob/master/docs/database-schema.md
Complete PostgreSQL schema for structchunk integration with pgvector. Defines the chunks table with a Snowflake BIGINT primary key, content metadata, a TEXT[] breadcrumb, and a VECTOR(1024) embedding column, plus indexes on doc_id and chunk_id. Adjust the embedding dimension to match your model (e.g., 1536 for OpenAI text-embedding-3-small).
```sql
CREATE TABLE chunks (
chunk_id BIGINT PRIMARY KEY, -- Snowflake int, 8 bytes
doc_id TEXT NOT NULL, -- content hash or file path
chunk_index INT NOT NULL,
total_chunks INT NOT NULL,
content TEXT NOT NULL,
content_length INT NOT NULL, -- pre-computed, filterable
total_length INT NOT NULL, -- same as content_length since v0.4.0
header_breadcrumb TEXT[],
embedding VECTOR(1024), -- pgvector column
created_at_ms BIGINT GENERATED ALWAYS AS
(structchunk.chunk_id_timestamp_ms(chunk_id)) STORED
);
CREATE INDEX idx_chunks_doc_id ON chunks (doc_id);
CREATE INDEX idx_chunks_chunk_id ON chunks (chunk_id);
```
--------------------------------
### SQLite: Create chunks table
Source: https://github.com/yzp0111/structchunk/blob/master/docs/database-schema.md
SQL DDL for the chunks table in SQLite. Uses INTEGER (8-byte) for chunk_id to hold full Snowflake id, and TEXT for header_breadcrumb as a JSON-encoded string.
```sql
CREATE TABLE chunks (
chunk_id INTEGER PRIMARY KEY, -- 8-byte integer
doc_id TEXT NOT NULL,
chunk_index INTEGER NOT NULL,
total_chunks INTEGER NOT NULL,
content TEXT NOT NULL,
content_length INTEGER NOT NULL,
total_length INTEGER NOT NULL,
header_breadcrumb TEXT -- JSON-encoded array
);
```
--------------------------------
### Sentence Boundary Split Regex (Chinese and English)
Source: https://github.com/yzp0111/structchunk/blob/master/docs/algorithms.md
Regex used to split text into sentences, matching either a run of non-terminator characters followed by terminators (., !, ?, 。，！，？) plus optional whitespace, or a trailing fragment without terminators.
```regex
[^.!?。！？]+[.!?。！？]+\s*|[^.!?。！？]+\s*$
```
--------------------------------
### PostgreSQL generated column for timestamp recovery
Source: https://github.com/yzp0111/structchunk/blob/master/docs/database-schema.md
Defines a generated column that uses the structchunk.chunk_id_timestamp_ms() function to recover the millisecond timestamp from a chunk_id. Requires the SQL function to be defined in the database.
```sql
created_at_ms BIGINT GENERATED ALWAYS AS
(structchunk.chunk_id_timestamp_ms(chunk_id)) STORED
```
--------------------------------
### BlockType Enum
Source: https://github.com/yzp0111/structchunk/blob/master/docs/api.md
Enumeration of atomic markdown block types. Each value corresponds to a distinct structural element in a markdown document.
```APIDOC
## BlockType Enum
### Description
Enumeration of atomic markdown block types. Each value corresponds to a distinct structural element in a markdown document.
### Members
- **FRONTMATTER** - YAML/TOML frontmatter block at the start of a document.
- **HEADER** - A markdown header (`#` through `######`).
- **CODE_FENCE** - A fenced code block.
- **TABLE** - A complete markdown table.
- **TABLE_ROW** - An individual row within a table.
- **LIST** - A list (ordered or unordered).
- **BLOCKQUOTE** - A blockquote section.
- **MATH_BLOCK** - A math block (e.g. `$$...$$`).
- **HR** - A horizontal rule (`---`, `***`, `___`).
- **PARAGRAPH** - A plain text paragraph.
- **BLANK** - An empty/blank line.
```
--------------------------------
### plpgsql function structchunk_chunk_id_timestamp_ms
Source: https://github.com/yzp0111/structchunk/blob/master/docs/database-schema.md
Defines a plpgsql function that mirrors the lib's timestamp extraction from a Snowflake ID. It shifts the chunk_id right by 22 bits and adds the epoch offset 1735689600000. Use this for SQL-side recovery; the in-Python function produces identical results.
```sql
-- Python reference (lib's _models.py:201-203)
-- timestamp_bits = 64 - 10 - 12 = 42
-- _EPOCH_MS = 1735689600000 (2026-01-01 00:00:00 UTC)
-- extracted = (value >> 12 >> 10) + 1735689600000
-- Equivalent in plpgsql:
CREATE OR REPLACE FUNCTION structchunk_chunk_id_timestamp_ms(chunk_id BIGINT)
RETURNS BIGINT AS $$
BEGIN
RETURN (chunk_id >> 22) + 1735689600000;
END;
$$ LANGUAGE plpgsql IMMUTABLE;
```
--------------------------------
### chunk_id_timestamp_ms()
Source: https://github.com/yzp0111/structchunk/blob/master/docs/api.md
Recovers the millisecond timestamp embedded in a Snowflake chunk ID. Accepts an integer or a string in decimal or hex format.
```APIDOC
## chunk_id_timestamp_ms()
### Description
Recovers the millisecond timestamp embedded in a Snowflake chunk ID. Accepts the raw `int` (canonical) or a string in either decimal (preferred, what `str(int_chunk_id)` produces) or hex (for legacy callers). Returns 0 for non-Snowflake inputs.
### Signature
```python
chunk_id_timestamp_ms(chunk_id: int | str) -> int
```
### Parameters
- **chunk_id** (int | str) - Required - The chunk ID, either as an integer or a string in decimal or hex format.
### Returns
- **int** - Unix timestamp in milliseconds, or 0 if the input is not a valid Snowflake ID.
```
=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.