ChunkNorris

https://github.com/wikit-ai/chunknorris
Admin
ChunkNorris is a Python package that optimizes document chunking from various sources (HTML, PDFs,...
Tokens:17,672
Snippets:208
Trust Score:8
License:MIT
Update:5 months ago
Show doc for...
Tokens:
Raw
### Development Setup for chunknorris

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/pages/installation.md

Install the library with development dependencies for contributing to the project.

```bash
git clone https://github.com/wikit-ai/chunknorris.git
cd chunknorris
pip install " .[dev]"
```

--------------------------------

### Install chunknorris from Source

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/pages/installation.md

Clone the repository and install locally to use the latest development versions.

```bash
git clone https://github.com/wikit-ai/chunknorris.git
cd chunknorris
pip install .
```

--------------------------------

### Install chunknorris

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/pdf_parsing.ipynb

Install the chunknorris library if it's not already present. A kernel restart might be required after installation.

```python
# If needed, install chunknorris
%pip install chunknorris -q
```

--------------------------------

### ChunkNorris CLI Usage Examples

Source: https://context7.com/wikit-ai/chunknorris/llms.txt

Examples of how to use the ChunkNorris command-line interface to chunk different file types with various configurations.

```bash
# Chunk a PDF file with default settings
chunknorris --filepath "report.pdf"
```

```bash
# Chunk a Markdown file with custom word-count limits
chunknorris --filepath "docs.md" \
  --max_headers_to_use h3 \
  --max_chunk_word_count 150 \
  --hard_max_chunk_word_count 300 \
  --min_chunk_word_count 20
```

```bash
# Chunk a PDF with OCR enabled for scanned pages
chunknorris --filepath "scanned.pdf" \
  --use_ocr auto \
  --ocr_language "eng+fra" \
  --extract_tables true
```

```bash
# All available options
chunknorris -h
```

--------------------------------

### Python String Joining Example

Source: https://github.com/wikit-ai/chunknorris/blob/main/tests/test_files/file.ipynb

Demonstrates joining a list of strings into a single string using Python's join method.

```python
" ".join(["content", "of", "output", "cell"])
```

--------------------------------

### Pipeline Setup for PDF Chunking

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/pdf_chunking.ipynb

Imports necessary classes from chunknorris and sets the logging level. This code is required before initializing the pipeline.

```python
from chunknorris.parsers import PdfParser
from chunknorris.chunkers import MarkdownChunker
from chunknorris.pipelines import BasePipeline
from chunknorris import set_log_level
from IPython.display import Markdown

set_log_level("info")
```

--------------------------------

### Install chunknorris with Pip

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/pages/installation.md

Use this command to install the library from the Python Package Index.

```bash
pip install chunknorris
```

--------------------------------

### Python Code Cell Example

Source: https://github.com/wikit-ai/chunknorris/blob/main/tests/test_files/file.ipynb

A basic example of a Python code cell.

```python
# content of python cell
```

--------------------------------

### Chunk Markdown File with BasePipeline

Source: https://github.com/wikit-ai/chunknorris/blob/main/README.md

Example of chunking a Markdown file using MarkdownParser, MarkdownChunker, and BasePipeline. This pipeline processes the file and outputs chunks that can be printed or saved.

```python
from chunknorris.parsers import MarkdownParser
from chunknorris.chunkers import MarkdownChunker
from chunknorris.pipelines import BasePipeline

# Instanciate components
parser = MarkdownParser()
chunker = MarkdownChunker()
pipeline = BasePipeline(parser, chunker)

# Get some chunks !
chunks = pipeline.chunk_file(filepath="myfile.md")

# Print or save :
for chunk in chunks:
    print(chunk.get_text())
pipeline.save_chunks(chunks)
```

--------------------------------

### Define Markdown String

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/advanced_chunking.ipynb

Define a sample Markdown string to be used for chunking examples. This string includes various header levels.

```python
md_string = """
# This is header 1

This is some intruction text after header 1

## This is SUBheader 1

This is some intruction text after header 1

### This is an h3 header

This is the content of the h3 subsection

### This is ANOTHER h3 header

This is the other content of the h3 subsection
"""
Markdown(md_string)
```

--------------------------------

### JavaScript Code Cell Example

Source: https://github.com/wikit-ai/chunknorris/blob/main/tests/test_files/file.ipynb

A basic example of a JavaScript code cell.

```javascript
# content of javascript cell
```

--------------------------------

### Implementing a Custom Parser in Python

Source: https://context7.com/wikit-ai/chunknorris/llms.txt

Provides an example of extending `AbstractParser` to support custom file formats. Requires implementing `parse_file()` and `parse_string()` methods to return a `MarkdownDoc`. Useful for integrating proprietary or unsupported file types into the ChunkNorris pipeline.

```python
from chunknorris.parsers.abstract_parser import AbstractParser
from chunknorris.core.components import MarkdownDoc
from chunknorris.chunkers import MarkdownChunker
from chunknorris.pipelines import BasePipeline


class MyCustomParser(AbstractParser):
    """Example: parse plain-text files as Markdown."""

    def parse_file(self, filepath: str) -> MarkdownDoc:
        with open(filepath, "r", encoding="utf-8") as f:
            content = f.read()
        return self.parse_string(content)

    def parse_string(self, string: str) -> MarkdownDoc:
        # Apply any custom cleaning / conversion logic here
        cleaned = string.strip()
        return MarkdownDoc.from_string(cleaned)


pipeline = BasePipeline(parser=MyCustomParser(), chunker=MarkdownChunker())
chunks = pipeline.chunk_file("notes.txt")
for chunk in chunks:
    print(chunk.get_text())
```

--------------------------------

### View Chunk Details

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/pdf_chunking.ipynb

Iterates through specific chunks of a PDF file and prints their start page, end page, and text content. This helps in understanding how PDF content is segmented.

```python
for chunk_idx in [10, 11]: # choose any
    chunk = chunks[chunk_idx]
    print(f"\n===== Start page: {chunk.start_page} --- End page: {chunk.end_page} ======\n")
    print(chunk.get_text())
```

--------------------------------

### Get document as markdown string

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/pdf_parsing.ipynb

Convert the parsed PDF document into a markdown string. This is useful for viewing the extracted text, headers, and tables in a formatted way. Only a sample of the string is displayed.

```python
# Let's view a sample of the document
md_string = parsed_doc.to_string()
Markdown(
    "___________
[...]" + md_string[26400:34000] + "[...]
_______________")
```

--------------------------------

### Initialize BasePipeline with NotebookParser

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/custom_parser.ipynb

Shows how to set up the `BasePipeline` by providing a `NotebookParser` instance and a `MarkdownChunker`. This pipeline is then used to chunk the content of a notebook file.

```python
from chunknorris.chunkers import MarkdownChunker
from chunknorris.pipelines import BasePipeline
```

```python
pipe = BasePipeline(
    parser=NotebookParser(), 
    chunker=MarkdownChunker(max_chunk_word_count=100)
    )

chunks = pipe.chunk_file(path_to_notebook)
print(f"Got {len(chunks)} chunks !")
for i, chunk in enumerate(chunks[:3]):
    print(f"============ chunk {i} ============")
    print(chunk)
```

--------------------------------

### Instantiate and Use NotebookParser

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/custom_parser.ipynb

Demonstrates how to create an instance of `NotebookParser` and use its `parse_file` method to process a notebook. The output is then displayed as markdown.

```python
path_to_notebook = "./custom_parser.ipynb" # as an example we will use... this notebook !
notebook_parser = NotebookParser(include_code_cells_outputs=False)
md_doc = notebook_parser.parse_file(path_to_notebook)
```

```python
# Before feeding the parsed result to the chunker, **let's have a look** at the markdown it outputs.
Markdown(md_doc.to_string()[:1400] + " [...]" ) # only print out the first 1400 caracters
```

--------------------------------

### Instantiate DOCX Chunking Pipeline

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/docx_chunking.ipynb

Create an instance of the BasePipeline, configuring it with DocxParser for handling .docx files and MarkdownChunker for processing the parsed content.

```python
# instanciate
pipeline = BasePipeline(DocxParser(), MarkdownChunker())
```

--------------------------------

### Initialize and Run PDF Chunking Pipeline

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/pdf_chunking.ipynb

Sets up the PdfParser and MarkdownChunker within a BasePipeline and then chunks a specified PDF file. The number of generated chunks is printed.

```python
# Setup the pipe. Feel free to play with the parser and chunker's arguments.
pipeline = BasePipeline(
    PdfParser(),
    MarkdownChunker(),
)

chunks = pipeline.chunk_file("./example_data/sample.pdf")
print(f"Got {len(chunks)} chunks !")
```

--------------------------------

### BasePipeline

Source: https://context7.com/wikit-ai/chunknorris/llms.txt

`BasePipeline` is the recommended entry point for most use cases, combining a parser and `MarkdownChunker` to process files or strings and save the resulting chunks.

```APIDOC
## `BasePipeline` — Parse and Chunk in One Step

`BasePipeline` combines any parser with `MarkdownChunker` and exposes `chunk_file()`, `chunk_string()`, and `save_chunks()`. It is the recommended entry point for most use cases.

```python
from chunknorris.parsers import MarkdownParser
from chunknorris.chunkers import MarkdownChunker
from chunknorris.pipelines import BasePipeline

pipeline = BasePipeline(
    parser=MarkdownParser(),
    chunker=MarkdownChunker()
)

# Chunk a file
chunks = pipeline.chunk_file(filepath="README.md")

# Chunk a raw string
raw_md = "# Section 1\n\nSome content here.\n\n## Subsection\n\nMore content."
chunks = pipeline.chunk_string(raw_md)

# Inspect results
for chunk in chunks:
    print(f"[words: {chunk.word_count}] pages {chunk.start_page}–{chunk.end_page}")
    print(chunk.get_text())
    print("---")

# Persist to JSON  →  [{"text": "...", "start_page": null, "end_page": null, "start_line": 0}, ...]
pipeline.save_chunks(chunks, output_filename="output-chunks.json", remove_links=False)
```
```

--------------------------------

### Instantiate Chunknorris Pipeline Components

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/markdown_chunking.ipynb

Create instances of the MarkdownParser, MarkdownChunker, and BasePipeline to set up the markdown chunking process.

```python
# instanciate
parser = MarkdownParser()
chunker = MarkdownChunker()
pipeline = BasePipeline(parser, chunker)
```

--------------------------------

### Import necessary modules

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/pdf_parsing.ipynb

Import the PdfParser, set_log_level function, and Markdown for displaying output. Set the log level to 'info' for detailed logs during parsing.

```python
from chunknorris.parsers import PdfParser
from chunknorris import set_log_level
from IPython.display import Markdown

set_log_level("info")
```

--------------------------------

### Parse PDF file from local path

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/pdf_parsing.ipynb

Instantiate PdfParser and parse a local PDF file. This operation includes table extraction and may take a few seconds depending on the file size and hardware.

```python
path_to_pdf = "./example_data/sample.pdf" # Mitel phones user manual, 265 pages.

# Instanciate parser and parse (should take around 2s)
parser = PdfParser()
parsed_doc = parser.parse_file(path_to_pdf)
```

--------------------------------

### Chunk Component

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/reference/components/component_chunk.md

The Chunk is the entity returned by chunknorris's chunkers. It contains various elements related to the chunks: its text content, headers, the pages it comes from (if from paginated documents), etc. You might essentially need to use `Chunk.get_text()` to get the cleaned chunk's content as text preceded by its headers.

```APIDOC
## Chunk

### Description
The `Chunk` object represents a piece of text extracted from a document by a chunker. It stores the text content, associated headers, and information about the original page if the document was paginated.

### Usage

To retrieve the cleaned text content of a chunk, including any preceding headers, use the `get_text()` method.

```python
chunk.get_text()
```

### Methods

- **get_text()**: Returns the cleaned text content of the chunk, prefixed with its headers.
```

--------------------------------

### Instantiate Chunknorris Pipeline

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/html_chunking.ipynb

Create an instance of the BasePipeline, combining the HTMLParser and MarkdownChunker. This pipeline will handle the conversion and chunking process.

```python
# instanciate
pipeline = BasePipeline(HTMLParser(), MarkdownChunker())
```

--------------------------------

### Chunking with Default Parameters

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/advanced_chunking.ipynb

Process the Markdown string using the BasePipeline with default MarkdownParser and MarkdownChunker settings. Observe the initial chunking result.

```python
# Pipeline with default parameters:
pipeline = BasePipeline(MarkdownParser(), MarkdownChunker())
chunks = pipeline.chunk_string(md_string)
print_chunking_result(chunks)
```

--------------------------------

### Import Necessary Libraries

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/advanced_chunking.ipynb

Import the required classes for parsing and chunking Markdown text, as well as the base pipeline.

```python
from chunknorris.parsers import MarkdownParser # <- you can use any parser you want as long as the are compatible with MarkdownChunker
from chunknorris.chunkers import MarkdownChunker # <- tutorial is essentially about this guy
from chunknorris.pipelines import BasePipeline
from IPython.display import Markdown
```

--------------------------------

### Configure MarkdownChunker with Advanced Options

Source: https://github.com/wikit-ai/chunknorris/blob/main/README.md

Instantiate MarkdownChunker with custom parameters to control chunking behavior. Specify maximum header levels, word counts (soft and hard limits), minimum chunk size, and token limits with a custom tokenizer.

```python
import tiktoken
from chunknorris.chunkers import MarkdownChunker

chunker = MarkdownChunker(
    max_headers_to_use="h4",
    max_chunk_word_count=250,
    hard_max_chunk_word_count=400,
    min_chunk_word_count=15,
    hard_max_chunk_token_count=8000,
    tokenizer=tiktoken.encoding_for_model("text-embedding-3-large"),
)
```

--------------------------------

### Display First Three Chunks

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/markdown_chunking.ipynb

Iterate through the first three generated chunks and print their text content to inspect the chunking results.

```python
# Let's look at the chunks
for i, chunk in enumerate(chunks[:3]): # we only look at the 3 first chunks
    print(f"\n------------- chunk {i} ----------------\n")
    print(chunk.get_text())
```

--------------------------------

### Inspecting Chunk Objects in Python

Source: https://context7.com/wikit-ai/chunknorris/llms.txt

Demonstrates how to inspect Chunk objects returned by the pipeline, including retrieving full text with or without headers, and accessing metadata like word count and page numbers. Useful for understanding chunk content and properties.

```python
from chunknorris.parsers import PdfParser
from chunknorris.chunkers import MarkdownChunker
from chunknorris.pipelines import BasePipeline

pipeline = BasePipeline(parser=PdfParser(use_ocr="never"), chunker=MarkdownChunker())
chunks = pipeline.chunk_file("paper.pdf")

chunk = chunks[0]

# Full text including ancestor headers prepended for context
print(chunk.get_text())
# # Introduction
# ## Background
# This paper presents ...

# Text without Markdown link syntax (link text preserved, URL stripped)
print(chunk.get_text(remove_links=True))

# Text without prepended headers (content only)
print(chunk.get_text(prepend_headers=False))

# Metadata
print(chunk.word_count)   # int: words in content (headers excluded)
print(chunk.start_page)   # int | None: first page of content (0-based)
print(chunk.end_page)     # int | None: last page of content (0-based)
print(chunk.start_line)   # int: line index in the source MarkdownDoc

# Serialize to dict for downstream use (e.g. vector store ingestion)
chunk_dict = chunk.model_dump()
# {'headers': [...], 'content': [...], 'start_line': 4,
#  'word_count': 87, 'start_page': 1, 'end_page': 2}
```

--------------------------------

### Display First Two Chunks

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/html_chunking.ipynb

Iterate through the first two chunks generated from the HTML file and print their text content. This allows inspection of the chunking results.

```python
# Let's look at the chunks
for i, chunk in enumerate(chunks[:2]): # we only look at the 2 first chunks
    print(f"\n------------- chunk {i} \n")
    print(chunk.get_text())
```

--------------------------------

### Define a Custom NotebookParser Class

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/custom_parser.ipynb

Implement a custom parser by inheriting from AbstractParser and defining the parse_file and parse_string methods. These methods must return a MarkdownDoc object.

```python
# Base of our class
class NotebookParser(AbstractParser): # inherit from abstract parser
def parse_file(self, filepath: str) -> MarkdownDoc:
pass

def parse_string(self, string: str) -> MarkdownDoc:
pass # We have to fill this
```

--------------------------------

### Working with MarkdownDoc in Python

Source: https://context7.com/wikit-ai/chunknorris/llms.txt

Shows how to create, manipulate, and serialize MarkdownDoc objects, which represent the intermediate parsed document format. Useful for building documents from strings, converting back to strings, saving to files, and accessing individual lines.

```python
from chunknorris.core.components import MarkdownDoc

# Build from a markdown string
md_doc = MarkdownDoc.from_string("# Title\n\nParagraph text.\n\n## Section\n\nMore text.")

# Convert back to string
print(md_doc.to_string())

# Per-page string (relevant when built from a paginated parser like PdfParser)
page_strings = md_doc.to_string(keep_track_of_page=True)
# {None: "# Title\n\nParagraph text.\n\n## Section\n\nMore text."}

# Save the markdown to a file
md_doc.save("output.md")

# Access individual lines
for line in md_doc.content:
    if line.is_header:
        print(f"H{line.get_header_level()}: {line.text}")
    print(f"  page={line.page}, in_code_block={line.isin_code_block}")
```

--------------------------------

### Chunk a DOCX File

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/docx_chunking.ipynb

Use the pipeline to chunk a specified .docx file. The number of chunks generated is printed to the console.

```python
# Get those chunks !
path_to_file = "../../tests/test_files/file.docx"
chunks = pipeline.chunk_file(path_to_file)
print(f"Got {len(chunks)} chunks !")
```

--------------------------------

### Import Chunknorris Components

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/markdown_chunking.ipynb

Import the necessary components from the chunknorris library for markdown file processing: MarkdownParser, MarkdownChunker, and BasePipeline.

```python
# imported the required chunknorris components
from chunknorris.parsers import MarkdownParser
from chunknorris.chunkers import MarkdownChunker
from chunknorris.pipelines import BasePipeline
```

--------------------------------

### Chunk HTML Files with Python

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/pages/getting_started.md

Utilize HTMLParser and MarkdownChunker for chunking HTML documents. The output chunks are in Markdown format.

```python
from chunknorris.parsers import HTMLParser
from chunknorris.chunkers import MarkdownChunker
from chunknorris.pipelines import BasePipeline

# Instanciate components
pipeline = BasePipeline(
    parser=HTMLParser(),
    chunker=MarkdownChunker()
    )

# Get some chunks !
chunks = pipeline.chunk_file(filepath="myfile.html")

# Save chunks
pipeline.save_chunks(chunks)

# Print chunks:
for chunk in chunks:
    print(chunk.get_text())
```

--------------------------------

### Import Necessary Components for Custom Parser

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/custom_parser.ipynb

Import the base AbstractParser class and other required components for creating a custom parser. Ensure you inherit from AbstractParser to integrate with chunknorris.

```python
from typing import Any
import json
from IPython.display import Markdown
from chunknorris.parsers import AbstractParser # <-- our custom parser must inherit from this
from chunknorris.parsers.markdown.components import MarkdownDoc # <-- object ot be fed in chunker
```

--------------------------------

### Import Necessary Components

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/custom_parser.ipynb

Import the required components from the chunknorris library and typing for custom parser implementation.

```python
# Import components
from typing import Any
import json
from IPython.display import Markdown
from chunknorris.parsers import AbstractParser # <-- our custom parser must inherit from this
from chunknorris.core.components import MarkdownDoc # <-- object ot be fed in chunker
```

--------------------------------

### Chunk Excel Files with Python

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/pages/getting_started.md

Use ExcelParser to chunk .xlsx, .xls, .odt, or other Excel-like files. The output chunks are in Markdown format.

```python
from chunknorris.parsers import ExcelParser
from chunknorris.chunkers import MarkdownChunker
from chunknorris.pipelines import BasePipeline

# Instanciate components
pipeline = BasePipeline(
    parser=ExcelParser(),
    chunker=MarkdownChunker()
    )

# Get some chunks !
chunks = pipeline.chunk_file(filepath="myfile.xslx") #  .xls, .odt or any excel-like file is compatible.

# Save chunks
pipeline.save_chunks(chunks)

# Print chunks:
for chunk in chunks:
    print(chunk.get_text())
```

--------------------------------

### Import Chunknorris Components for DOCX Chunking

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/docx_chunking.ipynb

Import the necessary classes from chunknorris for parsing DOCX files and chunking the content. This includes DocxParser, MarkdownChunker, and BasePipeline.

```python
# imported the required chunknorris components
from chunknorris.parsers import DocxParser
from chunknorris.chunkers import MarkdownChunker
from chunknorris.pipelines import BasePipeline
```

--------------------------------

### Optimize PdfParser for speed

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/pdf_parsing.ipynb

Create a PdfParser instance with options to disable table extraction and OCR for faster parsing when these features are not needed.

```python
parser = PdfParser(
    extract_tables=False,
    use_ocr="never"
)
```

--------------------------------

### Display detected tables

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/pdf_parsing.ipynb

Access and display a specific detected table in markdown format. Note that tables without visible lines might not be detected.

```python
# You may also want to look at the tables detected
print(f"Amount of tables detected : {len(parser.tables)}")
table_idx = 2 # Choose the idx you want
Markdown(parser.tables[table_idx].to_markdown())
```

--------------------------------

### Import Chunknorris Components

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/html_chunking.ipynb

Import the necessary classes from the chunknorris library for HTML parsing and markdown chunking. These components form the basis of the processing pipeline.

```python
# imported the required chunknorris components
from chunknorris.parsers import HTMLParser
from chunknorris.chunkers import MarkdownChunker
from chunknorris.pipelines import BasePipeline
```

--------------------------------

### Chunk DOCX Files with Python

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/pages/getting_started.md

Use DocxParser and MarkdownChunker to chunk Microsoft Word (.docx) documents. The output is in Markdown format.

```python
from chunknorris.parsers import DocxParser
from chunknorris.chunkers import MarkdownChunker
from chunknorris.pipelines import BasePipeline

# Instanciate components
pipeline = BasePipeline(
    parser=DocxParser(),
    chunker=MarkdownChunker()
    )

# Get some chunks !
chunks = pipeline.chunk_file(filepath="myfile.docx")

# Save chunks
pipeline.save_chunks(chunks)

# Print chunks:
for chunk in chunks:
    print(chunk.get_text())
```

--------------------------------

### Chunk HTML File

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/html_chunking.ipynb

Process an HTML file using the configured pipeline to generate chunks. The number of chunks obtained is printed to the console.

```python
# Get those chunks !
path_to_html_file = "../../tests/test_files/file.html"
chunks = pipeline.chunk_file(path_to_html_file)
print(f"Got {len(chunks)} chunks !")
```

--------------------------------

### Chunk Markdown Files with Python

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/pages/getting_started.md

Use MarkdownParser and MarkdownChunker to process and save chunks from a Markdown file.

```python
from chunknorris.parsers import MarkdownParser
from chunknorris.chunkers import MarkdownChunker
from chunknorris.pipelines import BasePipeline

# Instanciate components
pipeline = BasePipeline(
    parser=MarkdownParser(),
    chunker=MarkdownChunker()
    )

# Get some chunks !
chunks = pipeline.chunk_file(filepath="myfile.md")

# Save chunks
pipeline.save_chunks(chunks)

# Print chunks:
for chunk in chunks:
    print(chunk.get_text())
```

--------------------------------

### Chunk Documents using CLI

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/pages/getting_started.md

Run Chunknorris from the terminal to chunk files. Specify the filepath and Chunknorris will save the chunks to a JSON file.

```bash
chunknorris --filepath "path/to/myfile.pdf" # .pdf, .md or .html
```

--------------------------------

### Chunk a Markdown File

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/markdown_chunking.ipynb

Process a specified markdown file using the configured pipeline to generate text chunks. The number of resulting chunks is printed.

```python
# Get those chunks !
path_to_md_file = "../../tests/test_files/file.md"
chunks = pipeline.chunk_file(path_to_md_file)
print(f"Got {len(chunks)} chunks !")
```

--------------------------------

### Chunk PDF Files with Python

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/pages/getting_started.md

Process PDF documents using PdfParser and MarkdownChunker. The resulting chunks are in Markdown format.

```python
from chunknorris.parsers import PdfParser
from chunknorris.chunkers import MarkdownChunker
from chunknorris.pipelines import BasePipeline

# Instanciate components
pipeline = BasePipeline(
    parser=PdfParser(),
    chunker=MarkdownChunker()
    )

# Get some chunks !
chunks = pipeline.chunk_file(filepath="myfile.pdf")

# Save chunks
pipeline.save_chunks(chunks)

# Print chunks:
for chunk in chunks:
    print(chunk.get_text())
```

--------------------------------

### Define NotebookParser Class

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/custom_parser.ipynb

Implements methods to read and parse .ipynb files, extracting markdown and code cells. It handles different cell types and optionally includes code cell outputs.

```python
class NotebookParser(AbstractParser):
    def __init__(self, include_code_cells_outputs: bool = False) -> None:
        self.include_code_cells_outputs = include_code_cells_outputs

    def parse_file(self, filepath: str) -> MarkdownDoc:
        """chunks a notebook .ipynb file"""
        file_content = self.read_file(filepath)
        md_string = self.parse_notebook_content(file_content)
        
        return MarkdownDoc.from_string(md_string) # we don't return directly the markdown string, but build a MarkdownDoc with

    def parse_string(self, string: str) -> MarkdownDoc:
        raise NotImplementedError # We won't implement this as it is unlikely that the notebook content fill be passed as a string.

    @staticmethod
    def read_file(filepath: str) -> dict[str, Any]:
        """Reads a .ipynb file and returns its 
        content as a json dict.

        Args:
            filepath (str): path to the file

        Returns:
            dict[str, Any]: the json content of the ipynb file
        """
        if not filepath.endswith(".ipynb"):
            raise ValueError("Only .ipynb files can be passed to NotebookParser.")
        with open(filepath, "r", encoding="utf8") as file:
            content = json.load(file)

        return content
    
    def parse_notebook_content(self, notebook_content: dict[str, Any]) -> str:
        """Parses

        Args:
            notebook_content (dict[str, Any]): the content of the notebook, as a json file.
                It should be a dict of structure:
                {'cells': [{
                    'cell_type': 'markdown',
                    'metadata': {},
                    'source': <list of lines>
                    }...] 

        Returns:
            str: the markdown string parsed from the notebook content
        """
        kernel_language = notebook_content["metadata"]["kernelspec"]["language"]
            if notebook_content["metadata"] else ""
        md_string = ""
        for cell in notebook_content["cells"]:
            match cell["cell_type"]:
                case "markdown" | "raw":
                    md_string += "".join(cell["source"]) + "\n\n"
                case "code":
                    language = cell["metadata"]["kernelspec"]["language"] if cell["metadata"] else kernel_language
                    md_string += "```" + language + "\n" + "".join(cell['source']) + "\n```\n\n"
                    if self.include_code_cells_outputs:
                        md_string += "".join(cell["outputs"].get("data", {}).get('text/plain', "")) + "\n\n"
                case _:
                    pass

        return md_string
```

--------------------------------

### BasePipeline: Parse and Chunk Files/Strings

Source: https://context7.com/wikit-ai/chunknorris/llms.txt

Use BasePipeline to combine a parser and MarkdownChunker for a unified interface to chunk files or strings. Persist results to JSON.

```python
from chunknorris.parsers import MarkdownParser
from chunknorris.chunkers import MarkdownChunker
from chunknorris.pipelines import BasePipeline

pipeline = BasePipeline(
    parser=MarkdownParser(),
    chunker=MarkdownChunker()
)

# Chunk a file
chunks = pipeline.chunk_file(filepath="README.md")

# Chunk a raw string
raw_md = "# Section 1\n\nSome content here.\n\n## Subsection\n\nMore content."
chunks = pipeline.chunk_string(raw_md)

# Inspect results
for chunk in chunks:
    print(f"[words: {chunk.word_count}] pages {chunk.start_page}–{chunk.end_page}")
    print(chunk.get_text())
    print("---")

# Persist to JSON  →  [{"text": "...", "start_page": null, "end_page": null, "start_line": 0}, ...]
pipeline.save_chunks(chunks, output_filename="output-chunks.json", remove_links=False)
```

--------------------------------

### BasePipeline Class

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/reference/pipelines/pipeline_base.md

The BasePipeline class orchestrates the process of parsing text and then chunking the parsed output. It's a straightforward implementation for sequential processing.

```APIDOC
## Class: BasePipeline

### Description

This pipeline takes the output from a provided parser and uses it as the input for a provided chunker. It represents a basic sequential processing flow.

### Usage

```python
from chunknorris.pipelines.base_pipeline import BasePipeline

# Assuming you have initialized parser and chunker objects
# parser = YourParser()
# chunker = YourChunker()

# pipeline = BasePipeline(parser=parser, chunker=chunker)
# result = pipeline.process(text)
```

### Parameters

- **parser** (object) - Required - An initialized parser object.
- **chunker** (object) - Required - An initialized chunker object.

### Methods

#### process(text: str) -> list

Processes the input text through the parser and then the chunker.

- **Parameters**:
  - **text** (str) - The input text to process.

- **Returns**:
  - list - The result of the chunking process.
```

--------------------------------

### Parse PDF from URL (commented out)

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/pdf_parsing.ipynb

This commented-out code block shows how to parse a PDF file directly from a URL using the requests library and PdfParser. It requires fetching the content first.

```python
# Use the following block to parse a pdf file from an url

# import requests
# r = requests.get("myurl.pdf")
# data = r.content
# parser = PdfParser()
# parsed_doc = parser.parse_string(data)
```

--------------------------------

### Define Custom NotebookParser

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/custom_parser.ipynb

Define a custom NotebookParser class that inherits from AbstractParser and implements the parse_file and parse_string methods. This forms the base for parsing notebook content.

```python
# Base of our class
class NotebookParser(AbstractParser): # inherit from abstract parser
    def parse_file(self, filepath: str) -> MarkdownDoc:
        pass

    def parse_string(self, string: str) -> MarkdownDoc:
        pass # We have to fill this
```

--------------------------------

### Chunk CSV Files with Python

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/pages/getting_started.md

Process CSV files using CSVParser and MarkdownChunker. The resulting chunks are in Markdown format.

```python
from chunknorris.parsers import CSVParser
from chunknorris.chunkers import MarkdownChunker
from chunknorris.pipelines import BasePipeline

# Instanciate components
pipeline = BasePipeline(
    parser=CSVParser(),
    chunker=MarkdownChunker()
    )

# Get some chunks !
chunks = pipeline.chunk_file(filepath="myfile.csv")

# Save chunks
pipeline.save_chunks(chunks)

# Print chunks:
for chunk in chunks:
    print(chunk.get_text())
```

--------------------------------

### Chunk Jupyter Notebooks with Python

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/pages/getting_started.md

Process Jupyter Notebook files (.ipynb) using JupyterNotebookParser and MarkdownChunker. The output chunks are in Markdown format.

```python
from chunknorris.parsers import JupyterNotebookParser
from chunknorris.chunkers import MarkdownChunker
from chunknorris.pipelines import BasePipeline

# Instanciate components
pipeline = BasePipeline(
    parser=JupyterNotebookParser(),
    chunker=MarkdownChunker()
    )

# Get some chunks !
chunks = pipeline.chunk_file(filepath="myfile.ipynb")

# Save chunks
pipeline.save_chunks(chunks)

# Print chunks:
for chunk in chunks:
    print(chunk.get_text())
```

--------------------------------

### Parse HTML Files and Strings with HTMLParser

Source: https://context7.com/wikit-ai/chunknorris/llms.txt

Converts HTML to Markdown, stripping images and normalizing whitespace. The conversion loop repeats for stable results, handling nested structures. Use for parsing HTML from files or raw strings.

```python
from chunknorris.parsers import HTMLParser
from chunknorris.chunkers import MarkdownChunker
from chunknorris.pipelines import BasePipeline

# Parse from file
pipeline = BasePipeline(parser=HTMLParser(), chunker=MarkdownChunker())
chunks = pipeline.chunk_file("page.html")

# Parse from raw HTML string
parser = HTMLParser()
html = """
<html><body>
  <h1>Product Overview</h1>
  <p>This product does X and Y.</p>
  <h2>Installation</h2>
  <p>Run <code>pip install mypackage</code>.</p>
  <table>
    <tr><th>Feature</th><th>Status</th></tr>
    <tr><td>Auth</td><td>Stable</td></tr>
  </table>
</body></html>
"""
md_doc = parser.parse_string(html)
print(md_doc.to_string())
# # Product Overview
# This product does X and Y.
# ## Installation
# Run `pip install mypackage`.
# | Feature | Status |
# | --- | --- |
# | Auth | Stable |
```

--------------------------------

### Chunking with max_chunk_word_count=50

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/advanced_chunking.ipynb

Chunk the Markdown string with a reduced max_chunk_word_count of 50. This demonstrates how limiting the word count per chunk influences splitting behavior, separating the introduction from the rest of the content.

```python
chunker = MarkdownChunker(
    max_chunk_word_count=50, 
    min_chunk_word_count=0 # we set this to 0 because the chunker automatically discard chunks below 15 words by default
    )
pipeline = BasePipeline(MarkdownParser(), chunker)
chunks = pipeline.chunk_string(md_string)
print_chunking_result(chunks)
```

--------------------------------

### Display Table of Contents (TOC)

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/pdf_parsing.ipynb

Access and print the detected table of contents items. The parser attempts to find TOC from metadata, then using regex, and finally infers it from font size.

```python
# Let's diplay the first elements of the table of content : 
print(f"Amount of TOC items detected : {len(parser.toc)}.\nSample : ")
parser.toc[:4]
```

--------------------------------

### Parse Raw String with YAML Front-matter

Source: https://context7.com/wikit-ai/chunknorris/llms.txt

Parses a raw string containing YAML front-matter and Markdown content. The metadata is extracted into a dictionary, and the content is processed into MarkdownLine objects.

```python
raw = """
---
title: My Doc
version: 1.0
---

# Introduction

Welcome to the guide.

## Details

More content here.
"""
md_doc = parser.parse_string(raw)
print(md_doc.metadata)           # {'title': 'My Doc', 'version': 1.0}
print(len(md_doc.content))       # Number of MarkdownLine objects
```

--------------------------------

### Utility Function for Printing Chunks

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/advanced_chunking.ipynb

A helper function to display the results of chunking, showing the number of chunks and their content.

```python
# utility functions
def print_chunking_result(chunks):
    print(f"\n======= Got {len(chunks)} chunks ! ========\n")
    for i, chunk in enumerate(chunks):
        print(f"--------------------- chunk {i} ---------------------")
        print(chunk.get_text())
```

--------------------------------

### Iterate Through Chunk Headers and Content

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/markdown_chunking.ipynb

Access and print the headers and content of a specific chunk. Assumes 'chunks' is a pre-defined list of chunk objects.

```python
for line in chunks[10].headers:
    print(line)
print("=======================")
for line in chunks[10].content:
    print(line)
```

--------------------------------

### ExcelParser Usage

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/reference/parsers/parser_excel.md

This snippet demonstrates how to use the ExcelParser to parse spreadsheet files. It automatically processes all sheets within the provided file.

```APIDOC
## ExcelParser

### Description

The ``ExcelParser`` enables parsing spreadsheets, such as .xslx files. All sheets in the notebook will be parsed.

### Usage

```python
from chunknorris.parsers.sheets.excel_parser import ExcelParser

# Assuming 'file_path' is the path to your .xlsx file
parser = ExcelParser(file_path)
parsed_data = parser.parse()

# 'parsed_data' will contain the content of all sheets in the spreadsheet
```

### Parameters

- **file_path** (str): The path to the Excel file to be parsed.

### Returns

- **parsed_data** (dict): A dictionary where keys are sheet names and values are the parsed content of each sheet.
```

--------------------------------

### Invoke ChunkNorris from Terminal

Source: https://github.com/wikit-ai/chunknorris/blob/main/README.md

Command-line usage for chunking a file directly from the terminal. Use '-h' for available options.

```bash
chunknorris --filepath "path/to/myfile.pdf"
```

--------------------------------

### MarkdownChunker: Configurable Chunking Engine

Source: https://context7.com/wikit-ai/chunknorris/llms.txt

Configure MarkdownChunker with various limits for headers, word counts, and token counts. It processes a MarkdownDoc and returns a list of Chunk objects.

```python
import tiktoken
from chunknorris.chunkers import MarkdownChunker
from chunknorris.parsers import MarkdownParser

# Fully configured chunker
chunker = MarkdownChunker(
    max_headers_to_use="h3",          # Only use h1/h2/h3 as split points; h4–h6 ignored
    max_chunk_word_count=150,          # Soft limit: try to split further if exceeded
    hard_max_chunk_word_count=300,     # Hard limit: force-split by newlines if exceeded
    min_chunk_word_count=20,           # Discard chunks smaller than this
    hard_max_chunk_token_count=512,    # Hard token limit (requires tokenizer)
    tokenizer=tiktoken.encoding_for_model("text-embedding-3-large"),
)

parser = MarkdownParser()
md_doc = parser.parse_file("my_document.md")

chunks = chunker.chunk(md_doc)
# chunker is also directly callable:
# chunks = chunker(md_doc)

for chunk in chunks:
    print(f"Word count: {chunk.word_count}")
    print(chunk.get_text())
```

--------------------------------

### Save Chunks to File

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/docx_chunking.ipynb

Saves the obtained text chunks to a specified file. Ensure the 'chunks' variable is populated before calling this method.

```python
pipeline.save_chunks(chunks, "mychunk.json")
```

--------------------------------

### Display Extracted Chunks

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/docx_chunking.ipynb

Iterate through a subset of the generated chunks and print their text content. This helps in inspecting the quality and content of the chunking.

```python
# Let's look at the chunks
for i, chunk in enumerate(chunks[1:4]): # we only look at the 2 first chunks
    print(f"\n------------- chunk {i} \n")
    print(chunk.get_text())
```

--------------------------------

### Chunking with max_chunk_word_count=0

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/advanced_chunking.ipynb

Set max_chunk_word_count to 0 to force the chunker to use all available headers for splitting. This results in the maximum number of chunks possible based on the Markdown structure.

```python
chunker = MarkdownChunker(
    max_chunk_word_count=0, 
    min_chunk_word_count=0 # we set this to 0 because the chunker automatically discard chunks below 15 words by default
    )
pipeline = BasePipeline(MarkdownParser(), chunker)
chunks = pipeline.chunk_string(md_string)
print_chunking_result(chunks)
```

--------------------------------

### Chunk Component API

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/reference/components/component_chunk.md

This details the Chunk component's API. Use `Chunk.get_text()` to retrieve the cleaned text content, including any preceding headers.

```python
class Chunk:
    def __init__(self, text: str, headers: list[str] | None = None, page: int | None = None):
        self.text = text
        self.headers = headers if headers is not None else []
        self.page = page

    def get_text(self) -> str:
        """Returns the chunk's text content, preceded by its headers."""
        header_text = "\n".join(self.headers) if self.headers else ""
        return f"{header_text}\n{self.text}".strip()

    def __str__(self) -> str:
        return self.get_text()

    def __repr__(self) -> str:
        return f"Chunk(text='{self.text[:20]}...', headers={self.headers}, page={self.page})"
```

--------------------------------

### Python String Joining Result

Source: https://github.com/wikit-ai/chunknorris/blob/main/tests/test_files/file.ipynb

The output of the Python string joining operation.

```text
Result:
'content of output cell'
```

--------------------------------

### CLI Usage

Source: https://context7.com/wikit-ai/chunknorris/llms.txt

ChunkNorris can be invoked directly from the terminal to process files. The output is saved as a JSON file containing the extracted chunks.

```APIDOC
## CLI Usage

`chunknorris` can be invoked on any supported file directly from a terminal. The output is saved as `<filename>-chunks.json`.

```bash
# Chunk a PDF file with default settings
chunknorris --filepath "report.pdf"

# Chunk a Markdown file with custom word-count limits
chunknorris --filepath "docs.md" \
  --max_headers_to_use h3 \
  --max_chunk_word_count 150 \
  --hard_max_chunk_word_count 300 \
  --min_chunk_word_count 20

# Chunk a PDF with OCR enabled for scanned pages
chunknorris --filepath "scanned.pdf" \
  --use_ocr auto \
  --ocr_language "eng+fra" \
  --extract_tables true

# All available options
chunknorris -h
```
```

--------------------------------

### Parse Jupyter Notebooks with JupyterNotebookParser

Source: https://context7.com/wikit-ai/chunknorris/llms.txt

Extracts code and markdown cells from .ipynb files, converting them into a unified MarkdownDoc. Use this parser for processing Jupyter Notebooks.

```python
from chunknorris.parsers import JupyterNotebookParser
from chunknorris.chunkers import MarkdownChunker
from chunknorris.pipelines import BasePipeline

pipeline = BasePipeline(
    parser=JupyterNotebookParser(),
    chunker=MarkdownChunker()
)
chunks = pipeline.chunk_file("analysis.ipynb")

for chunk in chunks:
    print(chunk.get_text())
```

--------------------------------

### MarkdownParser: Parse Markdown Files and Strings

Source: https://context7.com/wikit-ai/chunknorris/llms.txt

Use MarkdownParser to clean Markdown strings, parse YAML front-matter, and convert headers. It can parse files or raw strings.

```python
from chunknorris.parsers import MarkdownParser

parser = MarkdownParser()

# Parse a file
md_doc = parser.parse_file("docs/guide.md")
print(md_doc.metadata)           # {"title": "Guide", "author": "..."} if front-matter present
print(md_doc.to_string())        # Full cleaned markdown string
```

--------------------------------

### Parse Tabular Files with CSVParser and ExcelParser

Source: https://context7.com/wikit-ai/chunknorris/llms.txt

Converts tabular data from CSV or Excel files into Markdown-formatted tables. These parsers are typically used within a pipeline with MarkdownChunker.

```python
from chunknorris.parsers import CSVParser, ExcelParser
from chunknorris.chunkers import MarkdownChunker
from chunknorris.pipelines import BasePipeline

# CSV
csv_pipeline = BasePipeline(parser=CSVParser(), chunker=MarkdownChunker())
csv_chunks = csv_pipeline.chunk_file("data.csv")

# Excel (.xlsx, .xls, .odt and other Excel-compatible formats)
excel_pipeline = BasePipeline(parser=ExcelParser(), chunker=MarkdownChunker())
excel_chunks = excel_pipeline.chunk_file("spreadsheet.xlsx")

for chunk in excel_chunks:
    print(chunk.get_text())
```

--------------------------------

### Token-Based Chunking with Custom Tokenizer

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/advanced_chunking.ipynb

Set a hard limit on chunk size using tokens with `hard_max_chunk_token_count`. This requires providing a `tokenizer` object with an `encode(str) -> list[int]` method, such as one from the `tiktoken` library, to accurately count tokens.

```python
import tiktoken

chunker = MarkdownChunker(
    min_chunk_word_count=0, # we set this to 0 because the chunker automatically discard chunks below 15 words by default
    hard_max_chunk_token_count=20,
    tokenizer=tiktoken.encoding_for_model("text-embedding-3-small"),
    )
pipeline = BasePipeline(MarkdownParser(), chunker)
chunks = pipeline.chunk_string(md_string)
print_chunking_result(chunks)
```

--------------------------------

### Parse PDF Files with PdfParser

Source: https://context7.com/wikit-ai/chunknorris/llms.txt

Extracts text from PDFs, detecting and excluding headers/footers, and reconstructing text blocks. It can extract tables, identify titles, and optionally use OCR for scanned pages. Use for basic parsing, specific page ranges, or byte inputs.

```python
from chunknorris.parsers import PdfParser
from chunknorris.chunkers import MarkdownChunker
from chunknorris.pipelines import BasePipeline

# Basic usage — OCR in auto mode (falls back to OCR for image-only pages)
pipeline = BasePipeline(
    parser=PdfParser(
        extract_tables=True,
        add_headers=True,
        use_ocr="auto",          # "always" | "auto" | "never"
        ocr_language="eng+fra",
    ),
    chunker=MarkdownChunker()
)
chunks = pipeline.chunk_file("annual_report.pdf")

# Parse only specific pages (0-based, exclusive end)
parser = PdfParser(use_ocr="never")
md_doc = parser.parse_file("report.pdf", page_start=2, page_end=10)
print(md_doc.to_string())

# Parse from bytes (e.g. downloaded from an API)
import requests
pdf_bytes = requests.get("https://example.com/doc.pdf").content
md_doc = parser.parse_string(pdf_bytes)

# Per-page string dict (useful for page-level citation tracking)
page_dict = md_doc.to_string(keep_track_of_page=True)
for page_num, text in page_dict.items():
    print(f"--- Page {page_num} ---\n{text[:200]}")

# Release memory after parsing
parser.cleanup_memory()
```

--------------------------------

### MarkdownChunker Class

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/reference/chunkers/chunker_markdown.md

Reference for the MarkdownChunker class, detailing its usage for processing Markdown documents.

```APIDOC
## MarkdownChunker

### Description
The `MarkdownChunker` is used to process documents that have been parsed into a `MarkdownDoc` object.

### Usage
This class is intended for internal use within the chunknorris library for handling markdown-specific document processing.
```

--------------------------------

### Save Chunks to JSON

Source: https://github.com/wikit-ai/chunknorris/blob/main/docs/examples/html_chunking.ipynb

Save the processed chunks to a JSON file. This method is part of the pipeline and allows for persistent storage of the chunked data.

```python
# Let's save the chunks. We can just pass the chunks we obtain and the filename we want
pipeline.save_chunks(chunks, "mychunk.json")
```
Tokens:
Raw