### Build sudachi.rs and Install CLI

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/README.md

Clones the sudachi.rs repository, builds the release version, installs the command-line interface, and fetches the necessary dictionary. This is a quick setup for command-line usage.

```bash
git clone https://github.com/WorksApplications/sudachi.rs.git
cd ./sudachi.rs

cargo build --release
cargo install --path sudachi-cli/
./fetch_dictionary.sh
```

--------------------------------

### SudachiPy Installation Guide

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/README.md

Instructions for installing the SudachiPy library and its core dictionary using pip. This is the first step to using SudachiPy for Japanese morphological analysis.

```bash
pip install sudachipy
pip install sudachidict_core
```

--------------------------------

### Install SudachiPy from Source - Bash Script

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/README.md

Instructions for building and installing SudachiPy from its source distribution. This involves installing Python development tools, running a build script, and then installing the generated source distribution using pip.

```bash
./build-sdist.sh
```

--------------------------------

### Install and Use SudachiPy with Full Dictionary (CLI)

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/README.md

Shows how to install the full dictionary package and use it with the `sudachipy` command-line tool for tokenizing text. The full dictionary offers the most comprehensive vocabulary.

```bash
pip install sudachidict_full
echo "外国人参政権" | sudachipy -s full
```

--------------------------------

### Install and Use SudachiPy with Small Dictionary (CLI)

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/README.md

Demonstrates how to install the small dictionary package and use it with the `sudachipy` command-line tool to tokenize text. This is useful for quick analysis or scripting.

```bash
pip install sudachidict_small
echo "外国人参政権" | sudachipy -s small
```

--------------------------------

### Check sudachi CLI Installation

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/README.md

Verifies the installation of the sudachi command-line tool and displays its help information. This confirms that the installation was successful.

```bash
which sudachi
sudachi -h
```

--------------------------------

### Install sudachi-cli Tool

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/README.ja.md

Command to install the sudachi command-line interface tool after building the sudachi.rs project. It also shows how to verify the installation and check the tool's version.

```bash
$ cd sudachi.rs/
$ cargo install --path sudachi-cli/

$ which sudachi
/Users/<USER>/.cargo/bin/sudachi

$ sudachi -h
sudachi 0.6.0
A Japanese tokenizer
...
```

--------------------------------

### Install sudachi CLI

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/README.md

Installs the sudachi command-line interface globally using Cargo. This makes the 'sudachi' command available in your system's PATH.

```bash
cd sudachi.rs/
cargo install --path sudachi-cli/
```

--------------------------------

### Install SudachiPy (Python) and sudachi.rs (Rust)

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/README.ja.md

Provides commands to install the Python version of SudachiPy using pip and to clone the sudachi.rs repository. It also includes steps for building the Rust project and installing the command-line tool.

```bash
pip install --upgrade 'sudachipy>=0.6.10'

$ git clone https://github.com/WorksApplications/sudachi.rs.git
$ cd ./sudachi.rs

$ cargo build --release
$ cargo install --path sudachi-cli/
$ ./fetch_dictionary.sh

$ echo "高輪ゲートウェイ駅" | sudachi
高輪ゲートウェイ駅  名詞,固有名詞,一般,*,*,*    高輪ゲートウェイ駅
EOS
```

--------------------------------

### Install Python Sudachi

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/README.md

Installs or upgrades the Python version of Sudachi using pip. Ensure you have Python and pip installed.

```bash
pip install --upgrade 'sudachipy>=0.6.10'
```

--------------------------------

### Python Tokenization and Morpheme Information (SudachiPy)

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/README.md

Provides Python code examples for initializing the SudachiPy tokenizer and performing morphological analysis. It demonstrates accessing surface form, reading form, and part-of-speech tags for individual morphemes, as well as tokenizing with different split modes.

```python
from sudachipy import Dictionary, SplitMode

tokenizer = Dictionary().create()

morphemes = tokenizer.tokenize("国会議事堂前駅")
print(morphemes[0].surface())  # '国会議事堂前駅'
print(morphemes[0].reading_form())  # 'コッカイギジドウマエエキ'
print(morphemes[0].part_of_speech())  # ['名詞', '固有名詞', '一般', '*', '*', '*']

morphemes = tokenizer.tokenize("国会議事堂前駅", SplitMode.A)
print([m.surface() for m in morphemes])  # ['国会', '議事', '堂', '前', '駅']
```

--------------------------------

### Install SudachiPy for Development - Python Pip

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/README.md

This command installs SudachiPy in an editable mode using pip. This is useful for developers who want to make changes to the SudachiPy code and test them immediately without reinstalling the package.

```python
python3 -m pip install -e .
```

--------------------------------

### Initialize SudachiPy Tokenizer with Default Dictionary (Python)

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/README.md

Initializes the SudachiPy tokenizer using the default configuration, which typically points to the 'core' dictionary. This is the simplest way to get a tokenizer object in Python.

```python
from sudachipy import Dictionary

# default: sudachidict_core
tokenizer_obj = Dictionary().create()
```

--------------------------------

### Normalized Form Output with sudachi CLI

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/README.md

Demonstrates how to get the normalized form of tokens using the sudachi command-line tool. This is useful for standardizing text variations.

```bash
echo 打込む かつ丼 附属 vintage | sudachi
```

--------------------------------

### Sudachi.rs CLI: Tokenization Modes and Outputs

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/README.ja.md

Demonstrates the usage of the sudachi command-line tool for Japanese morphological analysis. It shows examples for different splitting modes (default, 'A') and for displaying normalized forms with whitespace separation.

```sh
$ echo 選挙管理委員会 | sudachi
選挙管理委員会  名詞,固有名詞,一般,*,*,*        選挙管理委員会
EOS

$ echo 選挙管理委員会 | sudachi --mode A
選挙    名詞,普通名詞,サ変可能,*,*,*    選挙
管理    名詞,普通名詞,サ変可能,*,*,*    管理
委員    名詞,普通名詞,一般,*,*,*        委員
会      名詞,普通名詞,一般,*,*,*        会
EOS

$ echo 打込む かつ丼 附属 vintage | sudachi
打込む  動詞,一般,*,*,五段-マ行,終止形-一般     打ち込む
        空白,*,*,*,*,*
かつ丼  名詞,普通名詞,一般,*,*,*        カツ丼
        空白,*,*,*,*,*
附属    名詞,普通名詞,サ変可能,*,*,*    付属
        空白,*,*,*,*,*
vintage 名詞,普通名詞,一般,*,*,*        ビンテージ
EOS
```

--------------------------------

### Fetch Sudachi Dictionary using Script

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/README.ja.md

Provides shell commands to automatically download Sudachi dictionaries using the provided fetch script. It shows how to get the latest core dictionary or a specific version and type.

```sh
# fetch latest core dictionary
./fetch_dictionary.sh

# fetch dictionary of specified version and type
./fetch_dictionary.sh 20241021 small
```

--------------------------------

### Debug Sudachi.rs Fuzzing Failures

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/sudachi-fuzz/README.md

This command enables debugging of fuzzing failures by running the fuzzer with a debugger attached. It requires the lldb debugger to be installed. The command executes specific fuzzing targets within the hfuzz_workspace directory, allowing for step-by-step analysis of crashes.

```shell
cargo hfuzz run-debug sudachi-fuzz hfuzz_workspace/*/*.fuzz
```

--------------------------------

### Sudachi CLI Help and Options

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/README.md

Displays the help message and available options for the Sudachi command-line interface, including arguments for input files, commands like 'build' and 'ubuild', and various output formatting options.

```bash
$ sudachi -h
A Japanese tokenizer

Usage: sudachi [OPTIONS] [FILE] [COMMAND]

Commands:
  build
          Builds system dictionary
  ubuild
          Builds user dictionary
  dump

  help
          Print this message or the help of the given subcommand(s)

Arguments:
  [FILE]
          Input text file: If not present, read from STDIN

Options:
  -r, --config-file <CONFIG_FILE>
          Path to the setting file in JSON format
  -p, --resource_dir <RESOURCE_DIR>
          Path to the root directory of resources
  -m, --mode <MODE>
          Split unit: "A" (short), "B" (middle), or "C" (Named Entity) [default: C]
  -o, --output <OUTPUT_FILE>
          Output text file: If not present, use stdout
  -a, --all
          Prints all fields
  -w, --wakati
          Outputs only surface form
  -d, --debug
          Debug mode: Print the debug information
  -l, --dict <DICTIONARY_PATH>
          Path to sudachi dictionary. If None, it refer config and then baked dictionary
      --split-sentences <SPLIT_SENTENCES>
          How to split sentences [default: yes]
  -h, --help
          Print help (see more with '--help')
  -V, --version
          Print version

```

--------------------------------

### Sudachi CLI Help and Options

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/README.ja.md

Displays the help message for the sudachi command-line tool, listing available commands, arguments, and options for configuration, output formatting, and dictionary path specification.

```bash
$ sudachi -h
A Japanese tokenizer

Usage: sudachi [OPTIONS] [FILE] [COMMAND]

Commands:
  build
          Builds system dictionary
  ubuild
          Builds user dictionary
  dump

  help
          Print this message or the help of the given subcommand(s)

Arguments:
  [FILE]
          Input text file: If not present, read from STDIN

Options:
  -r, --config-file <CONFIG_FILE>
          Path to the setting file in JSON format
  -p, --resource_dir <RESOURCE_DIR>
          Path to the root directory of resources
  -m, --mode <MODE>
          Split unit: "A" (short), "B" (middle), or "C" (Named Entity) [default: C]
  -o, --output <OUTPUT_FILE>
          Output text file: If not present, use stdout
  -a, --all
          Prints all fields
  -w, --wakati
          Outputs only surface form
  -d, --debug
          Debug mode: Print the debug information
  -l, --dict <DICTIONARY_PATH>
          Path to sudachi dictionary. If None, it refer config and then baked dictionary
      --split-sentences <SPLIT_SENTENCES>
          How to split sentences [default: yes]
  -h, --help
          Print help (see more with '--help')
  -V, --version
          Print version
```

--------------------------------

### SudachiPy Tokenize CLI Help

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/README.md

Displays the help message for the `sudachipy tokenize` command, outlining available arguments and their descriptions. This is useful for understanding the various options for command-line tokenization.

```bash
$ sudachipy tokenize -h
usage: sudachipy tokenize [-h] [-r file] [-m {A,B,C}] [-o file] [-s string]
                          [-a] [-d] [-v]
                          [file [file ...]]

Tokenize Text

positional arguments:
  file           text written in utf-8

optional arguments:
  -h, --help     show this help message and exit
  -r file        the setting file in JSON format
  -m {A,B,C}     the mode of splitting
  -o file        the output file
  -s string      sudachidict type
  -a             print all of the fields
  -d             print the debug information
  -v, --version  print sudachipy version
```

--------------------------------

### Basic Tokenization with sudachi CLI

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/README.md

Demonstrates basic Japanese text tokenization using the sudachi command-line tool. It takes input via stdin and outputs tokens with their part-of-speech information.

```bash
echo "高輪ゲートウェイ駅" | sudachi
```

--------------------------------

### Initialize SudachiPy Tokenizer with Configuration File (Python)

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/README.md

Initializes the SudachiPy tokenizer by specifying a path to a configuration file (`sudachi.json`). This allows for custom dictionary paths or other settings defined within the JSON.

```python
from sudachipy import Dictionary

# The dictionary given by the `systemDict` key in the config file (/path/to/sudachi.json) will be used
tokenizer_obj = Dictionary(config="/path/to/sudachi.json").create()
```

--------------------------------

### Build User Dictionary with SudachiPy CLI

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/README.md

Provides the command-line usage for `sudachipy ubuild`, which is used to compile user dictionary source files (CSV format) into a binary dictionary file. Options include specifying output file and adding description comments.

```bash
sudachipy ubuild -h
sudachipy ubuild [-h] [-o file] [-d string] -s file file [file ...]
```

--------------------------------

### Build Sudachi Dictionary - Bash Command

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/README.md

This bash command displays the help message for the `sudachipy build` command, outlining its usage for creating a Sudachi dictionary from source files. It specifies required arguments like source files and connection matrix, and optional arguments for output file and description.

```bash
sudachipy build -h
```

--------------------------------

### Clone sudachi.rs Repository

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/README.ja.md

Command to clone the sudachi.rs project repository from GitHub. This is the initial step for setting up the project locally.

```bash
git clone https://github.com/WorksApplications/sudachi.rs.git
```

--------------------------------

### Build sudachi.rs with Baked Dictionary

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/README.md

Builds the sudachi.rs release version with the 'bake_dictionary' feature enabled, embedding the dictionary into the binary. Requires setting the SUDACHI_DICT_PATH environment variable.

```bash
# Download dictionary to resources/system.dic
$ ./fetch_dictionary.sh

# Build with bake_dictionary feature (relative path)
$ env SUDACHI_DICT_PATH=resources/system.dic cargo build --release --features bake_dictionary

# or

# Build with bake_dictionary feature (absolute path)
$ env SUDACHI_DICT_PATH=/path/to/my-sudachi.dic cargo build --release --features bake_dictionary
```

--------------------------------

### Command-Line Tokenization with Full Output (SudachiPy)

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/README.md

Demonstrates using the `sudachipy` command with the `-a` flag to display all available fields for tokenized output. This includes surface form, part-of-speech, normalized form, dictionary form, reading form, dictionary ID, and synonym group IDs.

```bash
$ echo "外国人参政権" | sudachipy -a
外国人参政権	名詞,普通名詞,一般,*,*,*	外国人参政権	外国人参政権	ガイコクジンサンセイケン	0	[]
EOS

echo "阿quei" | sudachipy -a
阿	名詞,普通名詞,一般,*,*,*	阿	阿		-1	[]	(OOV)
quei	名詞,普通名詞,一般,*,*,*	quei	quei		-1	[]	(OOV)
EOS
```

--------------------------------

### Multi-granular Tokenization with sudachi CLI

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/README.md

Shows how to perform multi-granular tokenization using the sudachi command-line tool by specifying the '--mode A' option. This breaks down text into smaller units.

```bash
echo 選挙管理委員会 | sudachi --mode A
```

--------------------------------

### Build sudachi.rs Project (Release Mode)

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/README.ja.md

Command to compile the sudachi.rs project in release mode. The `--all` flag ensures that all included plugins are also built.

```bash
cargo build --release
```

--------------------------------

### Specify Sudachi Configuration - Bash Command

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/README.md

This bash command shows how to run SudachiPy using a specific configuration file. The `-r` option allows you to point to a `sudachi.json` file, which can include settings for a custom system dictionary.

```bash
sudachipy -r path/to/sudachi.json
```

--------------------------------

### Specify System Dictionary in SudachiPy Settings File (JSON)

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/README.md

Illustrates the structure of a `sudachi.json` configuration file, specifically showing how to define the `systemDict` key to point to a custom system dictionary file. This allows for flexible dictionary management.

```json
{
    "systemDict" : "relative/path/from/resourceDir/to/system.dic",
    ...
}
```

--------------------------------

### Basic Command-Line Tokenization (SudachiPy)

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/README.md

Demonstrates basic Japanese text tokenization using the SudachiPy command-line tool. It shows default output and different splitting modes (A, B, C) for varying granularity. The output includes surface form, part-of-speech tags, and normalized form.

```bash
$ pip install sudachipy sudachidict_core

$ echo "高輪ゲートウェイ駅" | sudachipy
高輪ゲートウェイ駅	名詞,固有名詞,一般,*,*,*	高輪ゲートウェイ駅
EOS

$ echo "高輪ゲートウェイ駅" | sudachipy -m A
高輪	名詞,固有名詞,地名,一般,*,*	高輪
ゲートウェイ	名詞,普通名詞,一般,*,*,*	ゲートウェー
駅	名詞,普通名詞,一般,*,*,*	駅
EOS

$ echo "空缶空罐空きカン" | sudachipy -a
空缶	名詞,普通名詞,一般,*,*,*	空き缶	空缶	アキカン	0
空罐	名詞,普通名詞,一般,*,*,*	空き缶	空罐	アキカン	0
空きカン	名詞,普通名詞,一般,*,*,*	空き缶	空きカン	アキカン	0
EOS
```

--------------------------------

### Initialize SudachiPy Tokenizer with Specific Dictionary Edition (Python)

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/README.md

Initializes the SudachiPy tokenizer and explicitly selects a dictionary edition ('small', 'core', or 'full') using the `dict` argument. This provides direct control over the dictionary used.

```python
from sudachipy import Dictionary

# The dictionary specified by `dict` will be used.
tokenizer_obj = Dictionary(dict="core").create()  # sudachidict_core (same as default)
tokenizer_obj = Dictionary(dict="small").create()  # sudachidict_small
tokenizer_obj = Dictionary(dict="full").create()  # sudachidict_full
```

--------------------------------

### Configure SudachiPy with Custom Dictionary - JSON

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/README.md

A JSON configuration snippet demonstrating how to specify a custom system dictionary for SudachiPy. The `systemDict` key should be set to the relative path of the `system.dic` file from the `sudachi.json` configuration file.

```json
{
    "systemDict" : "relative/path/to/system.dic",
    ...
}
```

--------------------------------

### Build sudachi.rs with Embedded Dictionary

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/README.ja.md

Demonstrates how to build the sudachi.rs project with the 'bake_dictionary' feature enabled, which embeds the dictionary into the executable. It shows how to specify the dictionary path using the SUDACHI_DICT_PATH environment variable.

```bash
# resources/system.dic への辞書ダウンロード
$ ./fetch_dictionary.sh

# bake_dictionary フィーチャーフラグ付きでビルド (辞書を相対パスで指定)
$ env SUDACHI_DICT_PATH=resources/system.dic cargo build --release --features bake_dictionary

# もしくは

# bake_dictionary フィーチャーフラグ付きでビルド (辞書を絶対パスで指定)
$ env SUDACHI_DICT_PATH=/path/to/my-sudachi.dic cargo build --release --features bake_dictionary
```

--------------------------------

### Run Sudachi.rs Fuzzing with Cargo

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/sudachi-fuzz/README.md

This command initiates the fuzzing process for Sudachi.rs using the cargo hfuzz runner. It leverages coverage-based fuzzing to discover potential issues. Ensure you are in the project's root directory before execution.

```shell
nice -n 19 cargo hfuzz run sudachi-fuzz
```

--------------------------------

### Wakati (Space-Delimited Surface Form) Output with sudachi CLI

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/README.md

Illustrates how to output text with tokens separated by spaces (wakati format) using the sudachi command-line tool. Reads input from a file.

```bash
cat lemon.txt
sudachi --wakati lemon.txt
```

--------------------------------

### Sudachi Tokenization with '-a' Flag

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/README.md

Demonstrates how to use the Sudachi tokenizer with the '-a' flag to output all fields, including surface form, part-of-speech tags, normalized form, dictionary form, reading form, dictionary ID, and synonym group IDs. It also shows how Out-of-Vocabulary (OOV) words are indicated.

```bash
$ echo "外国人参政権" | sudachi -a
外国人参政権	名詞,普通名詞,一般,*,*,*	外国人参政権	外国人参政権	ガイコクジンサンセイケン	0	[]
```

```bash
echo "阿quei" | sudachipy -a
阿	名詞,普通名詞,一般,*,*,*	阿	阿		-1	[]	(OOV)
quei	名詞,普通名詞,一般,*,*,*	quei	quei		-1	[]	(OOV)
```

--------------------------------

### Specify User Dictionary in SudachiPy Settings File (JSON)

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/README.md

Shows how to configure a user dictionary (`user.dic`) within the `sudachi.json` settings file. The `userDict` key accepts a list of relative paths to user dictionary files.

```json
{
    "userDict" : ["relative/path/to/user.dic"],
    ...
}
```

--------------------------------

### Sudachi.rs CLI: Wakati (Word Segmentation) Output

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/README.ja.md

Illustrates how to use the sudachi command-line tool with the '--wakati' option to perform word segmentation and output only the surface forms of the tokens, separated by spaces.

```sh
$ cat lemon.txt
えたいの知れない不吉な塊が私の心を始終圧えつけていた。
焦躁と言おうか、嫌悪と言おうか――酒を飲んだあとに宿酔があるように、酒を毎日飲んでいると宿酔に相当した時期がやって来る。
それが来たのだ。これはちょっといけなかった。

$ sudachi --wakati lemon.txt
えたい の 知れ ない 不吉 な 塊 が 私 の 心 を 始終 圧え つけ て い た 。
焦躁 と 言おう か 、 嫌悪 と 言おう か ― ― 酒 を 飲ん だ あと に 宿酔 が ある よう に 、 酒 を 毎日 飲ん で いる と 宿酔 に 相当 し た 時期 が やっ て 来る 。
それ が 来 た の だ 。 これ は ちょっと いけ なかっ た 。
```

--------------------------------

### Override Config Dictionary with 'dict' Argument (Python)

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/README.md

Demonstrates how to use both `config` and `dict` arguments when initializing the SudachiPy tokenizer. The `dict` argument takes precedence, overriding any dictionary specified in the configuration file.

```python
from sudachipy import Dictionary

# The dictionary specified by `dict` overrides those defined in the config.
# In the following code, `sudachidict_full` will be used regardless of a dictionary defined in the config file.
tokenizer_obj = Dictionary(config="/path/to/sudachi.json", dict="full").create()
```

--------------------------------

### Sudachi Wakati Tokenization with '-w' Flag

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/README.md

Shows the use of the '-w' or '--wakati' flag with Sudachi to output only the surface forms of tokens, separated by spaces.

```bash
$ echo "外国人参政権" | sudachi -m A -w
外国 人 参政 権
```

--------------------------------

### SudachiPy: Subset WordInfo Fields

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/docs/source/topics/subsetting.rst

Demonstrates how to create a SudachiPy dictionary with a subset of WordInfo fields. The 'fields' parameter accepts a set of strings, specifying which data to include. Passing an empty set is allowed for tokenization-only use cases.

```python
from sudachipy import Dictionary, SplitMode

dic = Dictionary()
# Example: Load only the 'surface' and 'pos' fields
word_infos = dic.create(fields={'surface', 'pos'})

# Example: Load only tokenization (empty set)
word_infos_tokenization_only = dic.create(fields={})
```

```python
from sudachipy import Dictionary, SplitMode

dic = Dictionary()
# Implicitly loads 'splits_b' when SplitMode.B is used
word_infos_with_splits = dic.create(SplitMode.B, fields={})
```

--------------------------------

### Multi-granular Tokenization in Python (SudachiPy)

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/README.md

Illustrates how to perform tokenization with different granularities using SudachiPy in Python. The `SplitMode` enum (C, B, A) controls the level of detail in the tokenized output.

```python
# Multi-granular Tokenization

# SplitMode.C is the default mode
[m.surface() for m in tokenizer_obj.tokenize("国家公務員", SplitMode.C)]
# => ['国家公務員']

[m.surface() for m in tokenizer_obj.tokenize("国家公務員", SplitMode.B)]
# => ['国家', '公務員']

[m.surface() for m in tokenizer_obj.tokenize("国家公務員", SplitMode.A)]
# => ['国家', '公務', '員']
```

--------------------------------

### Text Normalization (SudachiPy Python)

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/README.md

Demonstrates the text normalization capabilities of SudachiPy in Python. It shows how SudachiPy can convert between different forms of words, such as kanji variations or katakana to hiragana.

```python
# Normalization

tokenizer_obj.tokenize("附属", mode)[0].normalized_form()
# => '付属'
tokenizer_obj.tokenize("SUMMER", mode)[0].normalized_form()
# => 'サマー'
tokenizer_obj.tokenize("シュミレーション", mode)[0].normalized_form()

```

--------------------------------

### SudachiPy: Reuse MorphemeList with Tokenizer

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/docs/source/topics/out_param.rst

Demonstrates the basic usage of reusing a MorphemeList object with SudachiPy's tokenize method. This pattern helps reduce memory allocation overhead by passing an existing MorphemeList as an output parameter.

```python
tok = dic.create(Mode.A)
morphemes = tok.tokenize("")
for line in data:
    tok.tokenize(line, out=morphemes)
    process(morphemes)
```

--------------------------------

### Minimize Sudachi.rs Fuzzing Crashes

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/sudachi-fuzz/README.md

When fuzzing encounters crashes or panics, this command helps minimize the problematic inputs. By setting the HFUZZ_RUN_ARGS environment variable to '-M', it instructs the fuzzer to focus on reducing the size of crashing test cases. This is useful for easier debugging.

```shell
HFUZZ_RUN_ARGS='-M' cargo -vv hfuzz run sudachi-fuzz
```

--------------------------------

### Extracting Morpheme Information (SudachiPy Python)

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/README.md

Shows how to extract detailed information about individual morphemes in Python using SudachiPy. This includes accessing the surface form, dictionary form, reading form, and part-of-speech tags.

```python
# Morpheme information

m = tokenizer_obj.tokenize("食べ")[0]

m.surface() # => '食べ'
m.dictionary_form() # => '食べる'
m.reading_form() # => 'タベ'
m.part_of_speech() # => ['動詞', '一般', '*', '*', '下一段-バ行', '連用形-一般']
```

--------------------------------

### SudachiPy: Memory Reuse with Morpheme.split (Potential Issue)

Source: https://github.com/worksapplications/sudachi.rs/blob/develop/python/docs/source/topics/out_param.rst

Illustrates memory reuse with SudachiPy's Morpheme.split method. It warns about potential issues where the resulting MorphemeList might refer to invalidated data if the parent list is reused as an output parameter in subsequent tokenization.

```python
ml1 = tok.tokenize("外国人参政権")
subl1 = ml1[0].split(SplitMode.A)
tok.tokenize("something", out=ml1)
subl1[0].surface() # can raise an exception!
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.