### Install Python 3.13.5 with Pyenv

Source: https://github.com/lindera/lindera-python/blob/main/README.md

Use pyenv to install a specific Python version. Ensure pyenv is installed first.

```shell
# Install Python
% pyenv install 3.13.5
```

--------------------------------

### Install Lindera-Python with Training Support

Source: https://github.com/lindera/lindera-python/blob/main/README.md

Command to install the lindera-python package with the 'train' feature enabled, necessary for dictionary training.

```shell
# Install with training support
(.venv) % maturin develop --features train

```

--------------------------------

### Install Lindera-Python as a Library

Source: https://github.com/lindera/lindera-python/blob/main/README.md

Install the lindera-python library within the activated virtual environment. This process can be time-consuming as it builds the library with all dictionaries.

```shell
(.venv) % make develop
```

--------------------------------

### Setup Repository and Activate Virtual Environment

Source: https://github.com/lindera/lindera-python/blob/main/README.md

Clone the lindera-python repository, set the local Python version, create a virtual environment, and activate it. Finally, initialize the project using make.

```shell
# Clone lindera-python project repository
% git clone git@github.com:lindera/lindera-python.git
% cd lindera-python

# Set Python version for this project
% pyenv local 3.13.5

# Make Python virtual environment
% python -m venv .venv

# Activate Python virtual environment
% source .venv/bin/activate

# Initialize lindera-python project
(.venv) % make init
```

--------------------------------

### Advanced Filter Configuration Examples

Source: https://github.com/lindera/lindera-python/blob/main/README.md

Illustrates advanced configuration for character and token filters, including parameters like 'kind', 'mapping', 'min', 'max', and 'tags'.

```python
from lindera import TokenizerBuilder

builder = TokenizerBuilder()
builder.set_dictionary("embedded://ipadic")

# Character filters with dict configuration
builder.append_character_filter("unicode_normalize", {"kind": "nfkc"})
builder.append_character_filter("japanese_iteration_mark", {
    "normalize_kanji": "true",
    "normalize_kana": "true"
})
builder.append_character_filter("mapping", {
    "mapping": {"リンデラ": "lindera", "トウキョウ": "東京"}
})

# Token filters with dict configuration  
builder.append_token_filter("japanese_katakana_stem", {"min": 3})
builder.append_token_filter("length", {"min": 2, "max": 10})
builder.append_token_filter("japanese_stop_tags", {
    "tags": ["助詞", "助動詞", "記号"]
})

# Filters without configuration can omit the dict
builder.append_token_filter("lowercase")
builder.append_token_filter("japanese_base_form")

tokenizer = builder.build()

```

--------------------------------

### Load Tokenizer Configuration from File

Source: https://context7.com/lindera/lindera-python/llms.txt

Use `from_file` on TokenizerBuilder to load a complete tokenizer configuration from a YAML or JSON file, ensuring reproducible setups.

```python
from lindera import TokenizerBuilder

# resources/lindera.yml defines dictionary, mode, and filter pipeline
builder = TokenizerBuilder()
tokenizer = builder.from_file("resources/lindera.yml").build()

tokens = tokenizer.tokenize("関西国際空港")
for token in tokens:
    print(token["surface"], token.get("detail"))
```

--------------------------------

### TokenizerBuilder.from_file - Load tokenizer config from file

Source: https://context7.com/lindera/lindera-python/llms.txt

Load a complete tokenizer configuration, including dictionary path, mode, and filter pipeline, from a YAML or JSON configuration file. This is useful for reproducible and externally managed tokenizer setups.

```APIDOC
## TokenizerBuilder.from_file

### Description
Loads a complete tokenizer configuration from a specified file (YAML/JSON), allowing for reproducible and externally managed tokenizer setups.

### Usage
```python
from lindera import TokenizerBuilder

# resources/lindera.yml defines dictionary, mode, and filter pipeline
builder = TokenizerBuilder()
tokenizer = builder.from_file("resources/lindera.yml").build()

tokens = tokenizer.tokenize("関西国際空港")
for token in tokens:
    print(token["surface"], token.get("detail"))
```
```

--------------------------------

### Build User Dictionary from CSV

Source: https://context7.com/lindera/lindera-python/llms.txt

Compiles a CSV user dictionary into binary form. The CSV format mirrors the dictionary schema. Includes an example of loading and using the compiled user dictionary.

```python
import lindera

# user.csv format (IPADIC): surface,reading,pronunciation
# 東京スカイツリー,トウキョウスカイツリー,トウキョウスカイツリー
# 関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクウコウ

lindera.build_user_dictionary(
    _kind="ipadic",            # reserved, currently unused
    input_file="user.csv",
    output_dir="/path/to/user_dict_output",
    # metadata=None uses default Metadata
)

# Load and use
from lindera import load_dictionary, load_user_dictionary, Tokenizer
dic  = load_dictionary("embedded://ipadic")
udic = load_user_dictionary("/path/to/user_dict_output", dic.metadata())
tok  = Tokenizer(dic, mode="normal", user_dictionary=udic)
print([t["surface"] for t in tok.tokenize("東京スカイツリーへ行く")])
# ['東京スカイツリー', 'へ', '行く']
```

--------------------------------

### Retrieve Lindera-Python Package Version

Source: https://context7.com/lindera/lindera-python/llms.txt

Shows how to get the current version of the lindera-python package. This is useful for checking compatibility or reporting issues.

```python
import lindera

v = lindera.version()
print(v)  # e.g. "2.0.0"
```

--------------------------------

### Tokenization with Character Filters

Source: https://github.com/lindera/lindera-python/blob/main/README.md

Shows how to configure Lindera tokenizers with character filters for text preprocessing. This example adds a mapping filter for character replacement and a Unicode normalization filter (NFKC).

```python
from lindera import TokenizerBuilder

# Create tokenizer builder
builder = TokenizerBuilder()
builder.set_mode("normal")
builder.set_dictionary("embedded://ipadic")

# Add character filters
builder.append_character_filter("mapping", {"mapping": {"ー": "-"}})
builder.append_character_filter("unicode_normalize", {"kind": "nfkc"})

# Build tokenizer with filters
tokenizer = builder.build()
text = "テストー１２３"
tokens = tokenizer.tokenize(text)  # Will apply filters automatically
```

--------------------------------

### Tokenization with Token Filters

Source: https://github.com/lindera/lindera-python/blob/main/README.md

Illustrates the use of token filters for post-processing tokenized text in Lindera. This example applies lowercase transformation, length filtering, and Japanese stop word filtering based on specific tags.

```python
from lindera import TokenizerBuilder

# Create tokenizer builder
builder = TokenizerBuilder()
builder.set_mode("normal")
builder.set_dictionary("embedded://ipadic")

# Add token filters
builder.append_token_filter("lowercase")
builder.append_token_filter("length", {"min": 2, "max": 10})
builder.append_token_filter("japanese_stop_tags", {"tags": ["助詞", "助動詞"]})

# Build tokenizer with filters
tokenizer = builder.build()
tokens = tokenizer.tokenize("テキストの解析")
```

--------------------------------

### Tokenize Text and Get Detailed Tokens

Source: https://context7.com/lindera/lindera-python/llms.txt

The `tokenize` method processes text and returns a list of dictionaries, each containing the surface form and detailed morphological information (like POS tags and readings) from the dictionary.

```python
from lindera import Tokenizer, load_dictionary

dictionary = load_dictionary("embedded://ipadic")
tokenizer = Tokenizer(dictionary, mode="normal")

tokens = tokenizer.tokenize("すもももももももものうち")
for token in tokens:
    surface = token["surface"]
    detail  = token.get("detail", [])
    print(f"{surface:8} -> {detail}")

# Expected output:
# すもも   -> ['名詞', '一般', '*', '*', '*', '*', 'すもも', 'スモモ', 'スモモ']
# も       -> ['助詞', '係助詞', ...]
# もも     -> ['名詞', '一般', ...]
# も       -> ['助詞', '係助詞', ...]
# もも     -> ['名詞', '一般', ...]
# の       -> ['助詞', '連体化', ...]
# うち     -> ['名詞', '非自立', ...]
```

--------------------------------

### Configure and Use Tokenizer with Filters

Source: https://github.com/lindera/lindera-python/blob/main/README.md

Demonstrates how to configure character and token filters using TokenizerBuilder and then build and use the tokenizer.

```python
from lindera import TokenizerBuilder

builder = TokenizerBuilder()
builder.set_dictionary("embedded://ipadic")

# Add character filters
builder.append_character_filter("mapping", {"mapping": {"ー": "-"}})
builder.append_character_filter("unicode_normalize", {"kind": "nfkc"})

# Add token filters  
builder.append_token_filter("lowercase")
builder.append_token_filter("japanese_base_form")

# Build and use
tokenizer = builder.build()
tokens = tokenizer.tokenize("コーヒーショップ")

```

--------------------------------

### Define and Use Custom Schema

Source: https://context7.com/lindera/lindera-python/llms.txt

Demonstrates how to create a custom schema for dictionary fields and validate records against it. Ensure the schema fields match the expected data.

```python
import lindera

# Custom schema for a simplified dictionary
custom_schema = lindera.Schema(["surface", "reading", "pronunciation"])
print(custom_schema.field_count())   # 3
print(custom_schema.get_field_index("reading"))  # 1

field_def = custom_schema.get_field_by_name("surface")
print(field_def.index, field_def.name, str(field_def.field_type))
# 0, surface, surface

# Validate a CSV record against the schema
try:
    custom_schema.validate_record(["東京", "トウキョウ", "トウキョウ"])
    print("Record is valid")
except ValueError as e:
    print(f"Invalid record: {e}")
```

--------------------------------

### Export Trained Model and Build Dictionary

Source: https://context7.com/lindera/lindera-python/llms.txt

Explains how to export a trained model into dictionary source files and then compile them into a loadable dictionary. This process is necessary to use a custom-trained model for tokenization.

```python
import lindera

lindera.export(
    model="model/model.dat",       # trained model from lindera.train()
    output="exported_dict/",       # output directory
    metadata="data/metadata.json", # optional: base metadata to update and copy
)
# Files created in exported_dict/:
#   lex.csv, matrix.def, unk.def, char.def, metadata.json (if metadata provided)

# After export, compile into a loadable dictionary
meta = lindera.Metadata.from_json_file("exported_dict/metadata.json")
lindera.build_dictionary(
    input_dir="exported_dict",
    output_dir="compiled_dict",
    metadata=meta,
)

# Use the trained dictionary for tokenization
from lindera import TokenizerBuilder
tokenizer = TokenizerBuilder().set_dictionary("compiled_dict").build()
tokens = tokenizer.tokenize("外国人参政権")
print([t["surface"] for t in tokens])
```

--------------------------------

### Load User Dictionary for Tokenizer

Source: https://context7.com/lindera/lindera-python/llms.txt

Loads a pre-built binary user dictionary and integrates it with a main dictionary for tokenization. The metadata must match the main dictionary.

```python
from lindera import Tokenizer, load_dictionary, load_user_dictionary

# Load main dictionary first to obtain its metadata
dictionary = load_dictionary("embedded://ipadic")
metadata   = dictionary.metadata()

# Load binary user dictionary (built from a CSV via build_user_dictionary)
user_dict = load_user_dictionary("resources/ipadic_simple_userdic.csv", metadata)

# Create tokenizer that recognises custom words
tokenizer = Tokenizer(dictionary, mode="normal", user_dictionary=user_dict)

tokens = tokenizer.tokenize("東京スカイツリーへようこそ")
for token in tokens:
    print(token["surface"])
# Custom entries such as 東京スカイツリー are returned as a single token
```

--------------------------------

### Build Lindera Dictionary from Source

Source: https://context7.com/lindera/lindera-python/llms.txt

Compiles raw dictionary source files (MeCab format) into Lindera's binary representation. Requires a Metadata object for configuration.

```python
import lindera

metadata = lindera.Metadata(
    name="my-ipadic",
    encoding="EUC-JP",
    compress_algorithm=lindera.CompressionAlgorithm.Deflate,
    flexible_csv=True,
    skip_invalid_cost_or_id=True,
)

# input_dir must contain: lex.csv (or *.csv), matrix.def, char.def, unk.def
lindera.build_dictionary(
    input_dir="/path/to/ipadic-source",
    output_dir="/path/to/output-dict",
    metadata=metadata,
)
print("Dictionary built successfully")
```

--------------------------------

### build_dictionary

Source: https://context7.com/lindera/lindera-python/llms.txt

Compiles raw dictionary source files (in MeCab format) into the binary representation used by Lindera. Requires a Metadata object describing encoding, compression, and schema.

```APIDOC
## build_dictionary — Compile a dictionary from source files

Compiles raw dictionary source files (in MeCab format) into the binary representation used by Lindera. Requires a `Metadata` object describing encoding, compression, and schema.

```python
import lindera

metadata = lindera.Metadata(
    name="my-ipadic",
    encoding="EUC-JP",
    compress_algorithm=lindera.CompressionAlgorithm.Deflate,
    flexible_csv=True,
    skip_invalid_cost_or_id=True,
)

# input_dir must contain: lex.csv (or *.csv), matrix.def, char.def, unk.def
lindera.build_dictionary(
    input_dir="/path/to/ipadic-source",
    output_dir="/path/to/output-dict",
    metadata=metadata,
)
print("Dictionary built successfully")
```
```

--------------------------------

### Configure Tokenization Mode and Penalty

Source: https://context7.com/lindera/lindera-python/llms.txt

Shows how to set tokenization modes (normal, decompose) and customize penalties for word decomposition in decompose mode. The penalty settings affect how compound words are split.

```python
import lindera

# Inspect mode
mode_normal    = lindera.Mode("normal")
mode_decompose = lindera.Mode("decompose")
print(mode_normal.is_normal())     # True
print(mode_decompose.is_decompose()) # True

# Custom penalty for decompose mode
penalty = lindera.Penalty(
    kanji_penalty_length_threshold=2,   # penalise kanji segments longer than 2 chars
    kanji_penalty_length_penalty=3000,
    other_penalty_length_threshold=7,   # penalise other segments longer than 7 chars
    other_penalty_length_penalty=1700,
)
print(penalty)
# Penalty(kanji_threshold=2, kanji_penalty=3000, other_threshold=7, other_penalty=1700)

# Pass mode string directly to the builder
tokenizer = (
    lindera.TokenizerBuilder()
    .set_mode("decompose")
    .set_dictionary("embedded://ipadic")
    .build()
)
tokens = tokenizer.tokenize("関西国際空港")
print([t["surface"] for t in tokens])  # ['関西', '国際', '空港']
```

--------------------------------

### load_user_dictionary

Source: https://context7.com/lindera/lindera-python/llms.txt

Loads a pre-built binary user dictionary (.bin) produced by `build_user_dictionary`. The metadata must match the main dictionary's metadata for correct ID interpretation.

```APIDOC
## load_user_dictionary — Load a compiled user dictionary

Loads a pre-built binary user dictionary (`.bin`) produced by `build_user_dictionary`. The `metadata` argument must match the main dictionary's metadata so that cost/context IDs are interpreted correctly.

```python
from lindera import Tokenizer, load_dictionary, load_user_dictionary

# Load main dictionary first to obtain its metadata
dictionary = load_dictionary("embedded://ipadic")
metadata   = dictionary.metadata()

# Load binary user dictionary (built from a CSV via build_user_dictionary)
user_dict = load_user_dictionary("resources/ipadic_simple_userdic.csv", metadata)

# Create tokenizer that recognises custom words
tokenizer = Tokenizer(dictionary, mode="normal", user_dictionary=user_dict)

tokens = tokenizer.tokenize("東京スカイツリーへようこそ")
for token in tokens:
    print(token["surface"])
# Custom entries such as 東京スカイツリー are returned as a single token
```
```

--------------------------------

### Basic Japanese Text Tokenization

Source: https://github.com/lindera/lindera-python/blob/main/README.md

Demonstrates basic Japanese text tokenization using the default settings of Lindera. Imports TokenizerBuilder and tokenizes a sample Japanese sentence, printing the text and position of each token.

```python
from lindera import TokenizerBuilder

# Create a tokenizer with default settings
builder = TokenizerBuilder()
builder.set_mode("normal")
builder.set_dictionary("embedded://ipadic")
tokenizer = builder.build()

# Tokenize Japanese text
text = "すもももももももものうち"
tokens = tokenizer.tokenize(text)

for token in tokens:
    print(f"Text: {token.text}, Position: {token.position}")
```

--------------------------------

### Build Tokenizer with Integrated Filters

Source: https://github.com/lindera/lindera-python/blob/main/README.md

Demonstrates building a Lindera tokenizer with an integrated pipeline of character and token filters. This approach allows for a streamlined tokenization process.

```python
from lindera import TokenizerBuilder

# Build tokenizer with integrated filters
builder = TokenizerBuilder()
builder.set_mode("normal")
builder.set_dictionary("embedded://ipadic")


```

--------------------------------

### Load and Inspect Dictionary Metadata

Source: https://github.com/lindera/lindera-python/blob/main/README.md

Shows how to load metadata for a specific dictionary and access its details like name, version, and schema.

```python
from lindera import Metadata

# Get metadata for a specific dictionary
metadata = Metadata.load("embedded://ipadic")
print(f"Dictionary: {metadata.dictionary_name}")
print(f"Version: {metadata.dictionary_version}")

# Access schema information
schema = metadata.dictionary_schema
print(f"Schema has {len(schema.fields)} fields")
print(f"Fields: {schema.fields[:5]}" ) # First 5 fields

```

--------------------------------

### Tokenizer - Direct tokenizer construction

Source: https://context7.com/lindera/lindera-python/llms.txt

Instantiate a Tokenizer directly using a pre-loaded Dictionary and an optional UserDictionary. This method is suitable when dictionaries are shared across multiple tokenizers or loaded once for reuse.

```APIDOC
## Tokenizer

### Description
Allows direct instantiation of a `Tokenizer` object using a pre-loaded `Dictionary` and an optional `UserDictionary`. This bypasses the `TokenizerBuilder` and is useful for scenarios where dictionaries are loaded once and reused.

### Usage
```python
from lindera import Tokenizer, load_dictionary

# Load embedded IPADIC dictionary
dictionary = load_dictionary("embedded://ipadic")

# Normal mode tokenizer (no user dictionary)
tokenizer_normal = Tokenizer(dictionary, mode="normal")

# Decompose mode tokenizer (splits compound words)
tokenizer_decompose = Tokenizer(dictionary, mode="decompose")

text = "関西国際空港限定トートバッグを東京スカイツリーの最寄り駅で買う"

print("--- Normal mode ---")
for token in tokenizer_normal.tokenize(text):
    print(token["surface"])

print("--- Decompose mode ---")
for token in tokenizer_decompose.tokenize(text):
    print(token["surface"])
# Decompose mode yields finer-grained segments, e.g. 関西 / 国際 / 空港
```
```

--------------------------------

### Train Custom Morphological Model

Source: https://context7.com/lindera/lindera-python/llms.txt

Illustrates the process of training a custom CRF-based morphological model using various input files. Ensure all specified input files exist before execution.

```python
import lindera

# All input files must exist before calling train()
lindera.train(
    seed="data/seed.csv",          # surface,left_id,right_id,cost,features...
    corpus="data/corpus.txt",      # surface\tfeatures lines, sentences separated by EOS
    char_def="data/char.def",      # character type definitions
    unk_def="data/unk.def",        # unknown word definitions
    feature_def="data/feature.def",# CRF feature templates (UNIGRAM/BIGRAM)
    rewrite_def="data/rewrite.def",# feature normalisation rules
    output="model/model.dat",      # output trained model
    lambda_=0.01,                  # L1 regularisation strength
    max_iter=100,                  # maximum training iterations
    max_threads=None,              # None = auto-detect CPU cores
)
print("Training complete: model/model.dat")
```

--------------------------------

### TokenizerBuilder - Fluent tokenizer configuration

Source: https://context7.com/lindera/lindera-python/llms.txt

The TokenizerBuilder allows for fluent configuration of tokenizers using a builder pattern. You can chain calls to set the dictionary, tokenization mode, whitespace handling, and filter pipelines before building the tokenizer.

```APIDOC
## TokenizerBuilder

### Description
The `TokenizerBuilder` class provides a fluent interface for configuring and building `Tokenizer` instances. It allows users to specify dictionaries, tokenization modes, and apply various character and token filters.

### Usage
```python
from lindera import TokenizerBuilder

# Build a tokenizer with IPADIC, Unicode normalization, and stop-tag filtering
builder = TokenizerBuilder()
builder.set_mode("normal")
builder.set_dictionary("embedded://ipadic")

# Character filters (applied before tokenization)
builder.append_character_filter("unicode_normalize", {"kind": "nfkc"})
builder.append_character_filter("japanese_iteration_mark", {
    "normalize_kanji": "true",
    "normalize_kana": "true"
})
builder.append_character_filter("mapping", {"mapping": {"リンデラ": "lindera"}})

# Token filters (applied after tokenization)
builder.append_token_filter("japanese_stop_tags", {
    "tags": ["助詞", "助動詞", "記号", "フィラー"]
})
builder.append_token_filter("japanese_katakana_stem", {"min": 3})
builder.append_token_filter("lowercase")
builder.append_token_filter("japanese_base_form")

tokenizer = builder.build()

tokens = tokenizer.tokenize("Ｌｉｎｄｅｒａは形態素解析ｴﾝｼﾞﾝです。")
for token in tokens:
    print(token["surface"])
# Output (after filters): lindera, 形態素, 解析, エンジン
```
```

--------------------------------

### Direct Tokenizer Construction

Source: https://context7.com/lindera/lindera-python/llms.txt

Instantiate Tokenizer directly with a pre-loaded Dictionary and optional UserDictionary, bypassing the builder pattern. This is efficient when sharing dictionaries across multiple tokenizers.

```python
from lindera import Tokenizer, load_dictionary

# Load embedded IPADIC dictionary
dictionary = load_dictionary("embedded://ipadic")

# Normal mode tokenizer (no user dictionary)
tokenizer_normal = Tokenizer(dictionary, mode="normal")

# Decompose mode tokenizer (splits compound words)
tokenizer_decompose = Tokenizer(dictionary, mode="decompose")

text = "関西国際空港限定トートバッグを東京スカイツリーの最寄り駅で買う"

print("--- Normal mode ---")
for token in tokenizer_normal.tokenize(text):
    print(token["surface"])

print("--- Decompose mode ---")
for token in tokenizer_decompose.tokenize(text):
    print(token["surface"])
# Decompose mode yields finer-grained segments, e.g. 関西 / 国際 / 空港
```

--------------------------------

### build_user_dictionary

Source: https://context7.com/lindera/lindera-python/llms.txt

Compiles a user dictionary from a CSV file into binary form. The CSV format mirrors the dictionary schema (e.g., surface,reading,pronunciation for IPADIC).

```APIDOC
## build_user_dictionary — Compile a user dictionary from CSV

Compiles a CSV user dictionary into binary form. The CSV format mirrors the dictionary schema: `surface,reading,pronunciation` by default for IPADIC-style dictionaries.

```python
import lindera

# user.csv format (IPADIC): surface,reading,pronunciation
# 東京スカイツリー,トウキョウスカイツリー,トウキョウスカイツリー
# 関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクウコウ

lindera.build_user_dictionary(
    _kind="ipadic",            # reserved, currently unused
    input_file="user.csv",
    output_dir="/path/to/user_dict_output",
    # metadata=None uses default Metadata
)

# Load and use
from lindera import load_dictionary, load_user_dictionary, Tokenizer
dic  = load_dictionary("embedded://ipadic")
udic = load_user_dictionary("/path/to/user_dict_output", dic.metadata())
tok  = Tokenizer(dic, mode="normal", user_dictionary=udic)
print([t["surface"] for t in tok.tokenize("東京スカイツリーへ行く")])
# ['東京スカイツリー', 'へ', '行く']
```
```

--------------------------------

### Configure Tokenizer with Builder Pattern

Source: https://context7.com/lindera/lindera-python/llms.txt

Use TokenizerBuilder to configure dictionaries, tokenization modes, and filter pipelines before building a tokenizer. Character filters are applied before tokenization, and token filters are applied after.

```python
from lindera import TokenizerBuilder

# Build a tokenizer with IPADIC, Unicode normalization, and stop-tag filtering
builder = TokenizerBuilder()
builder.set_mode("normal")
builder.set_dictionary("embedded://ipadic")

# Character filters (applied before tokenization)
builder.append_character_filter("unicode_normalize", {"kind": "nfkc"})
builder.append_character_filter("japanese_iteration_mark", {
    "normalize_kanji": "true",
    "normalize_kana": "true"
})
builder.append_character_filter("mapping", {"mapping": {"リンデラ": "lindera"}})

# Token filters (applied after tokenization)
builder.append_token_filter("japanese_stop_tags", {
    "tags": ["助詞", "助動詞", "記号", "フィラー"]
})
builder.append_token_filter("japanese_katakana_stem", {"min": 3})
builder.append_token_filter("lowercase")
builder.append_token_filter("japanese_base_form")

tokenizer = builder.build()

tokens = tokenizer.tokenize("Ｌｉｎｄｅｒａは形態素解析ｴﾝｼﾞﾝです。")
for token in tokens:
    print(token["surface"])
# Output (after filters): lindera, 形態素, 解析, エンジン
```

--------------------------------

### Export Trained Model to Dictionary Files

Source: https://github.com/lindera/lindera-python/blob/main/README.md

Python function call to export a trained model into dictionary files, including optional metadata.

```python
# Export trained model to dictionary files
lindera.export(
    model="model.dat",              # Trained model
    output="exported_dict/",        # Output directory
    metadata="metadata.json"        # Optional metadata file
)

```

--------------------------------

### Configure Lindera Metadata

Source: https://context7.com/lindera/lindera-python/llms.txt

Manages configuration parameters for building or loading dictionaries, including name, encoding, compression, and CSV parsing flags. Supports creation with defaults, explicit values, and loading from JSON.

```python
import lindera

# Create with defaults
meta = lindera.Metadata()

# Create with explicit values
meta = lindera.Metadata(
    name="custom-dict",
    encoding="UTF-8",
    compress_algorithm=lindera.CompressionAlgorithm.Deflate,
    default_word_cost=-10000,
    default_left_context_id=1288,
    default_right_context_id=1288,
    flexible_csv=False,
    skip_invalid_cost_or_id=False,
    normalize_details=False,
)

# Load from a JSON file (standard Lindera metadata.json)
meta = lindera.Metadata.from_json_file("resources/ipadic_metadata.json")

print(meta.name)              # "ipadic"
print(meta.encoding)          # "EUC-JP"
print(str(meta.compress_algorithm))  # "deflate"
print(meta.to_dict())         # Full dict representation
```

--------------------------------

### Tokenizer.tokenize - Tokenize text and return token dicts

Source: https://context7.com/lindera/lindera-python/llms.txt

The `tokenize` method processes an input string and returns a list of dictionaries. Each dictionary contains the 'surface' form and morphological details (like POS tags, reading, base form) from the underlying dictionary.

```APIDOC
## Tokenizer.tokenize

### Description
Processes a given text string and returns a list of token dictionaries. Each dictionary includes the surface form of the token and its associated morphological details, such as part-of-speech tags, readings, and base forms.

### Parameters
*   **text** (string) - The input text to tokenize.

### Returns
*   List of dictionaries, where each dictionary represents a token and contains at least a `"surface"` key and optionally a `"detail"` key with morphological information.

### Usage
```python
from lindera import Tokenizer, load_dictionary

dictionary = load_dictionary("embedded://ipadic")
tokenizer = Tokenizer(dictionary, mode="normal")

tokens = tokenizer.tokenize("すもももももももものうち")
for token in tokens:
    surface = token["surface"]
    detail  = token.get("detail", [])
    print(f"{surface:8} -> {detail}")

# Expected output:
# すもも   -> ['名詞', '一般', '*', '*', '*', '*', 'すもも', 'スモモ', 'スモモ']
# も       -> ['助詞', '係助詞', ...]
# もも     -> ['名詞', '一般', ...]
# も       -> ['助詞', '係助詞', ...]
# もも     -> ['名詞', '一般', ...]
# の       -> ['助詞', '連体化', ...]
# うち     -> ['名詞', '非自立', ...]
```
```

--------------------------------

### Train a Morphological Analysis Model

Source: https://github.com/lindera/lindera-python/blob/main/README.md

Python function call to train a morphological analysis model using various input files and parameters like regularization and iteration count.

```python
import lindera

# Train a model from corpus
lindera.train(
    seed="path/to/seed.csv",           # Seed lexicon
    corpus="path/to/corpus.txt",       # Training corpus
    char_def="path/to/char.def",       # Character definitions
    unk_def="path/to/unk.def",         # Unknown word definitions
    feature_def="path/to/feature.def", # Feature templates
    rewrite_def="path/to/rewrite.def", # Rewrite rules
    output="model.dat",                # Output model file
    lambda_=0.01,                      # L1 regularization
    max_iter=100,                      # Max iterations
    max_threads=None                   # Auto-detect CPU cores
)

```

--------------------------------

### Metadata

Source: https://context7.com/lindera/lindera-python/llms.txt

Represents dictionary metadata configuration, including name, encoding, compression algorithm, and CSV parsing flags. Can be created explicitly or loaded from a JSON file.

```APIDOC
## Metadata — Dictionary metadata configuration

`Metadata` holds all configuration parameters used when building or loading a dictionary, including name, encoding, compression algorithm, default word costs, CSV parsing flags, and schema definitions.

```python
import lindera

# Create with defaults
meta = lindera.Metadata()

# Create with explicit values
meta = lindera.Metadata(
    name="custom-dict",
    encoding="UTF-8",
    compress_algorithm=lindera.CompressionAlgorithm.Deflate,
    default_word_cost=-10000,
    default_left_context_id=1288,
    default_right_context_id=1288,
    flexible_csv=False,
    skip_invalid_cost_or_id=False,
    normalize_details=False,
)

# Load from a JSON file (standard Lindera metadata.json)
meta = lindera.Metadata.from_json_file("resources/ipadic_metadata.json")

print(meta.name)              # "ipadic"
print(meta.encoding)          # "EUC-JP"
print(str(meta.compress_algorithm))  # "deflate"
print(meta.to_dict())         # Full dict representation
```
```

--------------------------------

### load_dictionary

Source: https://context7.com/lindera/lindera-python/llms.txt

Loads a Dictionary object from an embedded dictionary name or a file-system path. Embedded dictionaries are pre-compiled into the package.

```APIDOC
## load_dictionary — Load a built-in or custom dictionary

`load_dictionary` loads a `Dictionary` object from an embedded dictionary name or a file-system path. Embedded dictionaries (prefixed with `embedded://`) are compiled into the binary when the package is built with the corresponding feature flags.

```python
import lindera

# Load an embedded dictionary (compiled into the wheel)
ipadic    = lindera.load_dictionary("embedded://ipadic")
unidic    = lindera.load_dictionary("embedded://unidic")
ko_dic    = lindera.load_dictionary("embedded://ko-dic")
cc_cedict = lindera.load_dictionary("embedded://cc-cedict")

# Load from a local directory (custom-built dictionary)
custom = lindera.load_dictionary("/path/to/my/custom_dict")

# Inspect dictionary metadata
print(ipadic.metadata_name())      # e.g. "ipadic"
print(ipadic.metadata_encoding())  # e.g. "EUC-JP"

meta = ipadic.metadata()
print(meta.name, meta.encoding, meta.compress_algorithm)
```
```

--------------------------------

### Schema

Source: https://context7.com/lindera/lindera-python/llms.txt

Describes the ordered list of field names in dictionary CSV entries, providing index lookups and validation helpers used during dictionary building or inspection.

```APIDOC
## Schema — Dictionary field schema

`Schema` describes the ordered list of field names in dictionary CSV entries. It provides index lookups and validation helpers used when building or inspecting dictionaries.

```python
import lindera

# Default IPADIC-style schema (13 fields)
schema = lindera.Schema.create_default()
print(schema.fields)
# ['surface', 'left_context_id', 'right_context_id', 'cost',

```

--------------------------------

### Load Lindera Dictionaries

Source: https://context7.com/lindera/lindera-python/llms.txt

Loads a Dictionary object from embedded names or file-system paths. Embedded dictionaries are compiled into the package.

```python
import lindera

# Load an embedded dictionary (compiled into the wheel)
ipadic    = lindera.load_dictionary("embedded://ipadic")
unidic    = lindera.load_dictionary("embedded://unidic")
ko_dic    = lindera.load_dictionary("embedded://ko-dic")
cc_cedict = lindera.load_dictionary("embedded://cc-cedict")

# Load from a local directory (custom-built dictionary)
custom = lindera.load_dictionary("/path/to/my/custom_dict")

# Inspect dictionary metadata
print(ipadic.metadata_name())      # e.g. "ipadic"
print(ipadic.metadata_encoding())  # e.g. "EUC-JP"

meta = ipadic.metadata()
print(meta.name, meta.encoding, meta.compress_algorithm)
```