### Running Core kiwi-rs Examples

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/docs/medium_kiwi_rs_performance_guide_en.md

This section provides commands to run various core examples of the kiwi-rs library, demonstrating different functionalities like basic tokenization, options analysis, blocklists, pre-tokenized data, native batch processing, and UTF-16 API usage.

```bash
# Core examples
car go run --example basic
car go run --example analyze_options
car go run --example blocklist_and_pretokenized
car go run --example native_batch
car go run --example utf16_api
```

--------------------------------

### Install Kiwi via Windows PowerShell Script

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/README.md

Shows how to install the Kiwi library and its models on Windows using a PowerShell script. This method is suitable for Windows environments and handles the necessary file operations for setup.

```powershell
cd kiwi-rs
powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\install_kiwi.ps1

```

--------------------------------

### Install Kiwi via Linux/macOS Shell Script

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/README.md

Provides instructions for installing the Kiwi library and its models on Linux and macOS systems using a `make` command. This script typically handles downloading and setting up the necessary files in a default location.

```bash
cd kiwi-rs
make install-kiwi

```

--------------------------------

### Initialize Kiwi with Automatic Bootstrap (Rust)

Source: https://context7.com/jaichangpark/kiwi-rs/llms.txt

Initializes the Kiwi library, automatically downloading necessary assets (library and model files) if they are not found locally. This is the simplest way to get started. It returns a `Kiwi` instance ready for use.

```rust
use kiwi_rs::Kiwi;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Downloads assets to cache if not found locally
    let kiwi = Kiwi::init()?;

    // Ready to use
    let tokens = kiwi.tokenize("한국어 형태소 분석")?;
    println!("Token count: {}", tokens.len());
    Ok(())
}
```

--------------------------------

### Rust: Initialize Kiwi with Automatic Bootstrap

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/skills/kiwi-rs-assistant/SKILL.md

Demonstrates the simplest way to initialize the kiwi-rs library using `Kiwi::init()`. This method handles automatic bootstrap, making it convenient for quick setups. It's suitable when environment variables for model paths are not explicitly managed.

```rust
use kiwi_rs::Kiwi;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Use Kiwi::init() for automatic bootstrap
    let kiwi = Kiwi::init()?;

    let text = "안녕하세요";
    let tokens = kiwi.tokenize(text)?;

    println!("Tokens: {:?}", tokens);
    Ok(())
}

```

--------------------------------

### Initialize and Tokenize Korean Text with kiwi-rs

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/docs/medium_kiwi_rs_performance_guide_en.md

Demonstrates the basic usage of kiwi-rs for Korean morphological analysis. It initializes the Kiwi analyzer, tokenizes a given Korean text, and prints detailed information about each token. This example assumes local assets are available or will be bootstrapped from cache/download.

```rust
use kiwi_rs::Kiwi;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // If local assets are missing, init() bootstraps from cache/download.
    let kiwi = Kiwi::init()?;

    let text = "아버지가방에들어가신다.";
    let tokens = kiwi.tokenize(text)?;

    for token in tokens {
        println!(
            "{} / {} (pos={}, len={}, word={}, sent={}, score={}, typo_cost={})",
            token.form,
            token.tag,
            token.position,
            token.length,
            token.word_position,
            token.sent_position,
            token.score,
            token.typo_cost
        );
    }

    Ok(())
}
```

--------------------------------

### Local Quality Checks for kiwi-rs

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/README.md

This snippet outlines the standard commands used to ensure code quality and correctness within the kiwi-rs project. It includes formatting, compilation checks, testing, linting, example compilation, and package building.

```bash
cd kiwi-rs
cargo fmt
cargo check
cargo test
cargo clippy --all-targets -- -D warnings
cargo check --examples
cargo package --allow-dirty
```

--------------------------------

### Rust Build and Check Commands for kiwi-rs

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/docs/medium_kiwi_rs_performance_guide_en.md

This snippet outlines essential Cargo commands for maintaining the kiwi-rs project. It includes formatting, checking, testing, linting, example compilation, and packaging.

```bash
cd kiwi-rs
car go fmt
car go check
car go test
car go clippy --all-targets -- -D warnings
car go check --examples
car go package --allow-dirty
```

--------------------------------

### Bash: Benchmark Kiwi-rs with Varied Input

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/docs/medium_kiwi_rs_performance_guide_en.md

Executes a Python script to benchmark Kiwi-rs under varied input conditions, simulating a near no-cache scenario. This command requires a Python environment with the necessary benchmarking scripts and dependencies installed, outputting results to markdown and JSON files.

```bash
cd kiwi-rs
mkdir -p tmp

# varied
.venv-bench/bin/python scripts/compare_feature_bench.py \
  --text "아버지가방에들어가신다." \
  --warmup 100 --iters 5000 \
  --batch-size 256 --batch-iters 500 \
  --input-mode varied --variant-pool 8192 \
  --repeats 1 \
  --md-out tmp/feature_bench_varied.md \
  --json-out tmp/feature_bench_varied.json

```

--------------------------------

### Bash: Benchmark Kiwi-rs with Repeated Input

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/docs/medium_kiwi_rs_performance_guide_en.md

Executes a Python script to benchmark Kiwi-rs under repeated input conditions, simulating a warm cache scenario. This command requires a Python environment with the necessary benchmarking scripts and dependencies installed, outputting results to markdown and JSON files.

```bash
cd kiwi-rs
mkdir -p tmp

# repeated
.venv-bench/bin/python scripts/compare_feature_bench.py \
  --text "아버지가방에들어가신다." \
  --warmup 100 --iters 5000 \
  --batch-size 256 --batch-iters 500 \
  --input-mode repeated --variant-pool 4096 \
  --repeats 1 \
  --md-out tmp/feature_bench_repeated.md \
  --json-out tmp/feature_bench_repeated.json

```

--------------------------------

### Rust: Tokenize Korean Text with Kiwi Rs

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/skills/kiwi-rs-assistant/SKILL.md

Provides a basic example of tokenizing Korean text using the `tokenize` API of the kiwi-rs library. This snippet assumes a `Kiwi` instance has already been initialized.

```rust
use kiwi_rs::Kiwi;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let kiwi = Kiwi::init()?;
    let text = "이것은 예시 문장입니다.";
    let tokens = kiwi.tokenize(text)?;
    println!("Tokens: {:?}", tokens);
    Ok(())
}

```

--------------------------------

### Configure Global Settings and Options in Rust

Source: https://context7.com/jaichangpark/kiwi-rs/llms.txt

Demonstrates how to initialize Kiwi-RS, read, modify, and apply global configuration settings such as cutoff thresholds and space penalties. It also shows how to retrieve the library version and check API support.

```rust
use kiwi_rs::Kiwi;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut kiwi = Kiwi::init()?;

    // Read current global config
    let config = kiwi.global_config()?;
    println!("Current cutoff: {}", config.cut_off_threshold);
    println!("Space penalty: {}", config.space_penalty);

    // Modify individual settings
    kiwi.set_cutoff_threshold(3.0)?;
    kiwi.set_space_penalty(5.0)?;
    kiwi.set_integrate_allomorph(true)?;

    // Check library version
    println!("Kiwi version: {}", kiwi.library_version()?);

    // Check API support
    println!("UTF-16 API supported: {}", kiwi.supports_utf16_api());
    println!("Batch UTF-16 supported: {}", kiwi.supports_analyze_mw());

    Ok(())
}
```

--------------------------------

### Initialize Kiwi with Configuration (Rust)

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/docs/medium_kiwi_rs_performance_guide_en.md

Demonstrates how to initialize the Kiwi library using a configuration object, specifying paths for dynamic libraries and models, and adding user-defined words. This is typically used in production environments where exact asset paths are pinned.

```rust
use kiwi_rs::{Kiwi, KiwiConfig};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = KiwiConfig::default()
        .with_library_path("/path/to/libkiwi.dylib")
        .with_model_path("/path/to/models/cong/base")
        .add_user_word("러스트", "NNP", 0.0);

    let kiwi = Kiwi::from_config(config)?;
    let analyses = kiwi.analyze_top_n("형태소 분석 예시", 2)?;
    println!("{} candidates", analyses.len());

    Ok(())
}
```

--------------------------------

### Install kiwi-rs Dependency in Rust

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/README.md

This snippet shows how to add the kiwi-rs crate as a dependency to your Rust project using Cargo. It's a standard TOML format for specifying dependencies.

```toml
[dependencies]
kiwi-rs = "0.1"

```

--------------------------------

### Rust: Initialize Kiwi with Builder Customization

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/skills/kiwi-rs-assistant/SKILL.md

Demonstrates initializing kiwi-rs using `KiwiLibrary` and its `builder` pattern. This approach is ideal for scenarios requiring pre-configuration of user words, regex rules, dictionaries, or typo sets before building the `Kiwi` instance.

```rust
use kiwi_rs::{Kiwi, KiwiLibrary};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Use KiwiLibrary + builder(...) for customization before building Kiwi
    let mut builder = KiwiLibrary::builder();
    // Add custom dictionaries, user words, regex rules, or typo sets here
    // builder.add_dictionary(...);
    // builder.add_user_word(...);
    let kiwi = builder.build()?;

    let text = "안녕하세요";
    let tokens = kiwi.tokenize(text)?;

    println!("Tokens: {:?}", tokens);
    Ok(())
}

```

--------------------------------

### Rust: Initialize Kiwi with Explicit Configuration

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/skills/kiwi-rs-assistant/SKILL.md

Illustrates initializing the kiwi-rs library with `Kiwi::from_config()`, allowing for explicit and reproducible paths to the library and model. This is useful when precise control over resource locations is required.

```rust
use kiwi_rs::{Kiwi, KiwiConfig};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Use Kiwi::from_config() for explicit, reproducible paths
    let config = KiwiConfig::default(); // Customize paths as needed
    let kiwi = Kiwi::from_config(config)?;

    let text = "안녕하세요";
    let tokens = kiwi.tokenize(text)?;

    println!("Tokens: {:?}", tokens);
    Ok(())
}

```

--------------------------------

### Running kiwi-rs Benchmarks

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/docs/medium_kiwi_rs_performance_guide_en.md

Commands for executing performance benchmarks within the kiwi-rs project. This includes tokenization benchmarks with specified iterations and warmup periods, and feature benchmarks with batch processing configurations.

```bash
# Rust-side benchmarks
car go run --release --example bench_tokenize -- --iters 1000 --warmup 100
car go run --release --example bench_features -- --iters 5000 --warmup 100 --batch-size 256 --batch-iters 500
```

--------------------------------

### Initialize Kiwi with Explicit Configuration (Rust)

Source: https://context7.com/jaichangpark/kiwi-rs/llms.txt

Initializes the Kiwi library using explicit paths for the library and model files, along with custom configuration options. This method allows for fine-grained control over the initialization process, including adding custom user dictionary words and setting default analysis options.

```rust
use kiwi_rs::{Kiwi, KiwiConfig, AnalyzeOptions};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = KiwiConfig::default()
        .with_library_path("/path/to/libkiwi.dylib")
        .with_model_path("/path/to/models/cong/base")
        .add_user_word("러스트", "NNP", 0.0)
        .add_user_word("형태소분석기", "NNG", 0.0)
        .with_default_analyze_options(AnalyzeOptions::default().with_top_n(3));

    let kiwi = Kiwi::from_config(config)?;
    let tokens = kiwi.tokenize("러스트 형태소분석기 테스트")?;

    for token in tokens {
        println!("{}/{} @{}", token.form, token.tag, token.position);
    }
    Ok(())
}
```

--------------------------------

### Initialize Kiwi with Manual Configuration in Rust

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/README.md

Illustrates initializing the Kiwi library in Rust using `Kiwi::from_config()`. This method allows for explicit configuration of library and model paths, as well as adding user-defined words with their parts of speech and weights. It requires the `kiwi_rs` crate.

```rust
use kiwi_rs::{Kiwi, KiwiConfig};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = KiwiConfig::default()
        .with_library_path("/path/to/libkiwi.dylib")
        .with_model_path("/path/to/models/cong/base")
        .add_user_word("러스트", "NNP", 0.0);

    let kiwi = Kiwi::from_config(config)?;
    let analyses = kiwi.analyze_top_n("형태소 분석 예시", 2)?;
    println!("{} candidates", analyses.len());
    Ok(())
}

```

--------------------------------

### Rust: Initialize Kiwi with Environment Variables

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/skills/kiwi-rs-assistant/SKILL.md

Shows how to initialize the kiwi-rs library using `Kiwi::new()`. This method assumes that the `KIWI_LIBRARY_PATH` and `KIWI_MODEL_PATH` environment variables are already set and managed. It's a common approach when the environment is pre-configured.

```rust
use kiwi_rs::Kiwi;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Use Kiwi::new() when environment variables are managed
    let kiwi = Kiwi::new()?;

    let text = "안녕하세요";
    let tokens = kiwi.tokenize(text)?;

    println!("Tokens: {:?}", tokens);
    Ok(())
}

```

--------------------------------

### Comparing Rust and Python Performance with kiwi-rs

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/docs/medium_kiwi_rs_performance_guide_en.md

This command demonstrates how to run a side-by-side performance comparison between the Rust implementation (kiwi-rs) and its Python counterpart (kiwipiepy), using a specific Korean text and benchmark parameters.

```bash
# Rust vs Python side-by-side
python3 scripts/bench_kiwipiepy.py --text "아버지가방에들어가신다." --warmup 100 --iters 5000
```

--------------------------------

### Run Expanded Feature Benchmark Snapshot

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/README.md

Generates a benchmark snapshot for dataset v2, including multi-run analysis. This command uses compare_feature_bench.py with 'repeated' input mode to capture warm-cache performance. It outputs detailed Markdown and JSON files for review and defense purposes.

```bash
cd kiwi-rs
mkdir -p tmp

# warm-cache overall (repeated input)
.venv-bench/bin/python scripts/compare_feature_bench.py \
  --dataset-tsv benchmarks/datasets/swe_textset_v2.tsv \
  --input-mode repeated \
  --warmup 20 --iters 300 \
  --batch-size 128 --batch-iters 60 \
  --repeats 5 \
  --engine-order alternate \
  --sleep-between-engines-ms 100 \
  --sleep-between-runs-ms 200 \
  --bootstrap-samples 2000 \
  --equivalence-band 0.05 \
  --strict-sink-check \
  --md-out tmp/feature_dataset_matrix_v2_repeated_r5_i300/overall.md \
  --json-out tmp/feature_dataset_matrix_v2_repeated_r5_i300/overall.json
```

--------------------------------

### Run Dataset-Driven Benchmark

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/README.md

Executes dataset-driven benchmarks using compare_feature_bench.py and compare_feature_dataset.py. These commands utilize a specified TSV dataset to evaluate performance. They are configured with specific warmup, iteration, batch size, and repeat counts, generating Markdown and JSON outputs or a directory of results.

```bash
cd kiwi-rs
mkdir -p tmp
.venv-bench/bin/python scripts/compare_feature_bench.py \
  --dataset-tsv benchmarks/datasets/swe_textset_v2.tsv \
  --input-mode varied \
  --warmup 20 --iters 300 \
  --batch-size 128 --batch-iters 60 \
  --repeats 5 \
  --engine-order alternate \
  --sleep-between-engines-ms 100 \
  --sleep-between-runs-ms 200 \
  --sink-warning-threshold 0.05 \
  --bootstrap-samples 2000 \
  --equivalence-band 0.05 \
  --strict-sink-check \
  --md-out tmp/feature_bench_dataset_v2.md \
  --json-out tmp/feature_bench_dataset_v2.json

.venv-bench/bin/python scripts/compare_feature_dataset.py \
  --dataset-tsv benchmarks/datasets/swe_textset_v2.tsv \
  --input-mode varied \
  --warmup 20 --iters 300 \
  --batch-size 128 --batch-iters 60 \
  --repeats 5 \
  --engine-order alternate \
  --sleep-between-engines-ms 100 \
  --sleep-between-runs-ms 200 \
  --out-dir tmp/feature_dataset_matrix_v2
```

--------------------------------

### Run Publication-Grade Benchmark (Recommended)

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/README.md

Executes publication-grade benchmarks using the compare_feature_bench.py script. It supports both 'repeated' and 'varied' input modes to assess performance under different caching conditions. The commands generate Markdown and JSON output files for detailed analysis.

```bash
cd kiwi-rs
mkdir -p tmp
.venv-bench/bin/python scripts/compare_feature_bench.py \
  --text "아버지가방에들어가신다." \
  --warmup 100 --iters 5000 \
  --batch-size 256 --batch-iters 500 \
  --input-mode repeated --variant-pool 4096 \
  --repeats 7 \
  --engine-order alternate \
  --sleep-between-engines-ms 200 \
  --sleep-between-runs-ms 500 \
  --sink-warning-threshold 0.05 \
  --bootstrap-samples 2000 \
  --equivalence-band 0.05 \
  --strict-sink-check \
  --md-out tmp/feature_bench_repeated_r7.md \
  --json-out tmp/feature_bench_repeated_r7.json

.venv-bench/bin/python scripts/compare_feature_bench.py \
  --text "아버지가방에들어가신다." \
  --warmup 100 --iters 5000 \
  --batch-size 256 --batch-iters 500 \
  --input-mode varied --variant-pool 8192 \
  --repeats 7 \
  --engine-order alternate \
  --sleep-between-engines-ms 200 \
  --sleep-between-runs-ms 500 \
  --sink-warning-threshold 0.05 \
  --bootstrap-samples 2000 \
  --equivalence-band 0.05 \
  --strict-sink-check \
  --md-out tmp/feature_bench_varied_r7.md \
  --json-out tmp/feature_bench_varied_r7.json
```

--------------------------------

### Run Benchmark Script

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/README.md

This command executes a Python script to compare feature performance across different dataset input modes. It configures warmup iterations, batch sizes, and output directories for detailed analysis.

```bash
.venv-bench/bin/python scripts/compare_feature_dataset.py \
  --dataset-tsv benchmarks/datasets/swe_textset_v2.tsv \
  --input-mode varied \
  --warmup 20 --iters 300 \
  --batch-size 128 --batch-iters 60 \
  --repeats 5 \
  --bootstrap-samples 2000 \
  --equivalence-band 0.05 \
  --engine-order alternate \
  --sleep-between-engines-ms 100 \
  --sleep-between-runs-ms 200 \
  --out-dir tmp/feature_dataset_matrix_v2_varied_r5_i300
```

--------------------------------

### Rust: Check and Use UTF-16 API with Kiwi-rs

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/docs/medium_kiwi_rs_performance_guide_en.md

Demonstrates how to initialize Kiwi-rs, check for UTF-16 API support, and then use UTF-16 encoded text for tokenization, analysis, and sentence splitting. It requires the `kiwi_rs` crate and standard Rust error handling.

```rust
use kiwi_rs::{AnalyzeOptions, Kiwi};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let kiwi = Kiwi::init()?;

    if !kiwi.supports_utf16_api() {
        println!("Loaded Kiwi library does not support UTF-16 API on this runtime.");
        return Ok(());
    }

    let text = "UTF16 경로도 동일하게 분석할 수 있습니다.";
    let utf16: Vec<u16> = text.encode_utf16().collect();

    let tokens = kiwi.tokenize_utf16_with_options(&utf16, AnalyzeOptions::default())?;
    println!("token count: {}", tokens.len());

    let candidates =
        kiwi.analyze_utf16_with_options(&utf16, AnalyzeOptions::default().with_top_n(2))?;
    println!("candidate count: {}", candidates.len());

    let sentences =
        kiwi.split_into_sents_utf16_with_options(&utf16, AnalyzeOptions::default(), true, true)?;
    println!("sentence count: {}", sentences.len());

    Ok(())
}

```

--------------------------------

### Subword Tokenization with SwTokenizer in Rust

Source: https://context7.com/jaichangpark/kiwi-rs/llms.txt

Illustrates how to use a Kiwi-compatible subword tokenizer model for neural network applications in Rust. It initializes Kiwi, opens a subword tokenizer model file, and then demonstrates encoding text to token IDs with character offsets, decoding token IDs back to text, and simple encoding without offsets.

```rust
use kiwi_rs::Kiwi;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let kiwi = Kiwi::init()?;

    // Open subword tokenizer model file
    let tokenizer = kiwi.open_sw_tokenizer("/path/to/tokenizer.json")?;

    let text = "형태소 분석기";

    // Encode text to token IDs with character offsets
    let (token_ids, offsets) = tokenizer.encode_with_offsets(text)?;
    println!("Token IDs: {:?}", token_ids);
    println!("Offsets: {:?}", offsets);

    // Decode token IDs back to text
    let decoded = tokenizer.decode(&token_ids)?;
    println!("Decoded: {}", decoded);

    // Simple encode without offsets
    let ids_only = tokenizer.encode(text)?;
    println!("IDs only: {:?}", ids_only);

    Ok(())
}
```

--------------------------------

### Benchmark Visualization: Repeated Input Ratio (Split + Glue)

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/README.md

Mermaid chart illustrating the performance ratio of kiwi-rs to kiwipiepy for 'split_into_sents' and 'glue' features under the 'repeated' input mode. This focuses on features with potentially higher ratios.

```mermaid
xychart-beta
    title "Repeated Input Ratio (Split + Glue)"
    x-axis ["split_into_sents","glue"]
    y-axis "kiwi-rs / kiwipiepy (x)" 0 --> 10000
    bar [9445.91,542.54]
```

--------------------------------

### Rust: Customize Kiwi Builder with User Dictionary

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/skills/kiwi-rs-assistant/SKILL.md

Explains how to customize the kiwi-rs behavior by adding a user-defined dictionary using the builder pattern. This allows the library to recognize specific terms or phrases. The `KiwiLibrary` must be used to access the builder.

```rust
use kiwi_rs::{Kiwi, KiwiLibrary, DictionaryBuilder};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut dict_builder = DictionaryBuilder::new();
    dict_builder.add_word("커스텀단어", "NNG"); // Add a custom word with its part-of-speech tag

    let mut builder = KiwiLibrary::builder();
    builder.add_dictionary(dict_builder.build()?);
    let kiwi = builder.build()?;

    let text = "이것은 커스텀단어 예시입니다.";
    let tokens = kiwi.tokenize(text)?;
    println!("Tokens with custom word: {:?}", tokens);
    Ok(())
}

```

--------------------------------

### Benchmark Visualization: Repeated Input Ratio

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/README.md

Mermaid chart visualizing the performance ratio of kiwi-rs to kiwipiepy for selected features under the 'repeated' input mode. It shows the relative throughput for features like 'tokenize' and 'analyze_top1'.

```mermaid
xychart-beta
    title "Repeated Input Ratio (Selected)"
    x-axis ["tokenize","analyze_top1","split_with_tokens","join","analyze_many_native","tokenize_many_batch","space_many_batch"]
    y-axis "kiwi-rs / kiwipiepy (x)" 0 --> 170
    bar [156.03,148.44,86.64,4.30,24.10,24.62,14.23]
```

--------------------------------

### Benchmark Visualization: Varied Input Ratio (Moderate Range)

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/README.md

Mermaid chart displaying the performance ratio of kiwi-rs to kiwipiepy for features under the 'varied' input mode, focusing on a moderate range of ratios. It includes features like 'tokenize', 'space', and 'glue'.

```mermaid
xychart-beta
    title "Varied Input Ratio (Moderate Range)"
    x-axis ["tokenize","analyze_top1","split","space","glue","join","analyze_many_native","space_many_batch"]
    y-axis "kiwi-rs / kiwipiepy (x)" 0 --> 5
    bar [1.49,1.00,1.06,1.12,1.56,3.85,0.92,0.98]
```

--------------------------------

### Handle N-best Analysis Candidates (Rust)

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/docs/medium_kiwi_rs_performance_guide_en.md

Shows how to perform N-best analysis using `analyze_top_n` and process the results, including iterating through candidates and their tokens. This pattern is crucial for handling ambiguous strings and improving downstream model performance by providing N-best features.

```rust
use kiwi_rs::{AnalyzeOptions, Kiwi, KIWI_MATCH_ALL_WITH_NORMALIZING};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let kiwi = Kiwi::init()?;

    let options = AnalyzeOptions::default()
        .with_top_n(3)
        .with_match_options(KIWI_MATCH_ALL_WITH_NORMALIZING)
        .with_open_ending(false);

    let text = "형태소 분석 결과 후보를 여러 개 보고 싶습니다.";
    let candidates = kiwi.analyze_with_options(text, options)?;

    for (index, candidate) in candidates.iter().enumerate() {
        println!("candidate #{index} prob={}", candidate.probability);
        for token in &candidate.tokens {
            println!("  {}/{} @{}+", token.form, token.tag, token.position, token.length);
        }
    }

    Ok(())
}
```

--------------------------------

### Domain Constraints with Blocklist and Pretokenization (Rust)

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/docs/medium_kiwi_rs_performance_guide_en.md

Illustrates how to enforce domain-specific tokenization using a blocklist (`MorphemeSet`) and pretokenized spans (`Pretokenized`). This is useful for ensuring deterministic token behavior for specific entities or phrases in enterprise search and data processing.

```rust
use kiwi_rs::{AnalyzeOptions, Kiwi};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let kiwi = Kiwi::init()?;

    let mut blocklist = kiwi.new_morphset()?;
    // Block a specific morpheme candidate.
    let _ = blocklist.add("하", Some("VV"))?;

    let mut pretokenized = kiwi.new_pretokenized()?;
    let text = "AI엔지니어링팀에서테스트중";

    // Force [0, 2) to be a single token: AI/NNP.
    let span_id = pretokenized.add_span(0, 2)?;
    pretokenized.add_token_to_span(span_id, "AI", "NNP", 0, 2)?;

    let tokens = kiwi.tokenize_with_blocklist_and_pretokenized(
        text,
        AnalyzeOptions::default(),
        Some(&blocklist),
        Some(&pretokenized),
    )?;

    for token in tokens {
        println!("{}/{} @{}+", token.form, token.tag, token.position, token.length);
    }

    Ok(())
}
```

--------------------------------

### Rust: Batch Process Text with Kiwi Rs

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/skills/kiwi-rs-assistant/SKILL.md

Shows how to efficiently process multiple text inputs using batch APIs in kiwi-rs, like `analyze_many`. This is beneficial for performance when dealing with large datasets. Ensure the `Kiwi` instance is initialized.

```rust
use kiwi_rs::Kiwi;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let kiwi = Kiwi::init()?;
    let texts = vec!["첫 번째 문장", "두 번째 문장"];
    let results = kiwi.analyze_many(&texts)?;
    println!("Batch Analysis Results: {:?}", results);
    Ok(())
}

```

--------------------------------

### Benchmark Visualization: Varied Input Ratio (High-Range Features)

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/README.md

Mermaid chart showing the performance ratio of kiwi-rs to kiwipiepy for 'split_with_tokens' and 'tokenize_many_batch' features under the 'varied' input mode. This chart highlights features with higher performance differences.

```mermaid
xychart-beta
    title "Varied Input Ratio (High-Range Features)"
    x-axis ["split_with_tokens","tokenize_many_batch"]
    y-axis "kiwi-rs / kiwipiepy (x)" 0 --> 70
    bar [67.75,23.18]
```

--------------------------------

### Utilize UTF-16 API for Tokenization and Analysis in Rust

Source: https://context7.com/jaichangpark/kiwi-rs/llms.txt

Demonstrates how to use Kiwi-RS's UTF-16 API for processing Korean text, which is useful for interoperability with systems using UTF-16 encoding. It covers tokenizing and analyzing UTF-16 encoded input and checks for API availability.

```rust
use kiwi_rs::{Kiwi, AnalyzeOptions};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let kiwi = Kiwi::init()?;

    // Check if UTF-16 API is available
    if !kiwi.supports_utf16_api() {
        println!("UTF-16 API not available in this Kiwi version");
        return Ok(())
    }

    // Convert text to UTF-16
    let text = "한국어 텍스트";
    let utf16: Vec<u16> = text.encode_utf16().collect();

    // Tokenize with UTF-16 input
    let tokens = kiwi.tokenize_utf16(&utf16)?;
    for token in tokens {
        println!("{}/{}", token.form, token.tag);
    }

    // Analyze with UTF-16
    let candidates = kiwi.analyze_utf16_with_options(
        &utf16,
        AnalyzeOptions::default().with_top_n(2),
    )?;

    for candidate in candidates {
        println!("Probability: {:.4}", candidate.probability);
    }

    Ok(())
}
```

--------------------------------

### Rust: Join Tokens with Spaces using Kiwi Rs

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/skills/kiwi-rs-assistant/SKILL.md

Illustrates using the `join` API in kiwi-rs to reconstruct text from tokens, inserting spaces appropriately. This is the inverse operation of tokenization. The `Kiwi` instance needs to be initialized.

```rust
use kiwi_rs::Kiwi;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let kiwi = Kiwi::init()?;
    let text = "자연어 처리";
    let tokens = kiwi.tokenize(text)?;
    // Assuming 'join' is the correct API for reconstructing with spaces
    // Note: The actual API might be 'join_with_space' or similar depending on kiwi-rs version
    // For demonstration, let's simulate joining tokens
    let reconstructed_text = tokens.iter().map(|t| t.text).collect::<Vec<&str>>().join(" ");
    println!("Reconstructed: {}", reconstructed_text);
    Ok(())
}

```

--------------------------------

### Rust: Split Korean Text into Sentences with Kiwi Rs

Source: https://github.com/jaichangpark/kiwi-rs/blob/main/skills/kiwi-rs-assistant/SKILL.md

Shows how to split a block of Korean text into individual sentences using the `split` API in kiwi-rs. This is useful for processing documents sentence by sentence. The `Kiwi` instance must be initialized beforehand.

```rust
use kiwi_rs::Kiwi;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let kiwi = Kiwi::init()?;
    let text = "첫 번째 문장입니다. 두 번째 문장입니다.";
    let sentences = kiwi.split(text)?;
    println!("Sentences: {:?}", sentences);
    Ok(())
}

```