SudachiDict

https://github.com/worksapplications/sudachidict
Admin
SudachiDict is a lexicon for the Japanese tokenizer Sudachi, providing various dictionary types...
Tokens:24,831
Snippets:395
Trust Score:8.4
Update:2 months ago
Show doc for...
Tokens:
Raw
### Setup Virtual Environment

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/06-release-and-distribution.md

Commands to create and activate a Python virtual environment and install necessary build dependencies.

```bash
# Create isolated environment
python3 -m venv .venv

# Activate
source .venv/bin/activate  # Linux/macOS
# or
.venv\Scripts\activate     # Windows

# Install build dependencies
pip install -r scripts/requirements.txt
```

--------------------------------

### Verify Installation

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/06-release-and-distribution.md

Command to test the installation of the updated dictionary package.

```bash
pip install --upgrade sudachidict_core
```

--------------------------------

### Setup CoreTest Class

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/05-dictionary-compilation.md

Example of extending the base test class to configure the tokenizer for core dictionary tests.

```java
public class CoreTest extends SudachiTest {
    @Before
    public void setUp() throws IOException {
        getTokenizer("system_core.dic");
    }
    // ... test methods
}
```

--------------------------------

### Troubleshoot installation issues

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/04-python-package-api.md

Commands for verifying installations, resolving download timeouts, and checking version compatibility.

```bash
pip show sudachidict_core
```

```bash
pip install --force-reinstall sudachidict_core
```

```bash
pip install --default-timeout=1000 sudachidict_core
```

```bash
pip install ./sudachi-dictionary-20260723-core.zip
```

```bash
pip show SudachiPy sudachidict_core
```

--------------------------------

### Install SudachiDict-full

Source: https://github.com/worksapplications/sudachidict/blob/develop/python/README.md

Installs the full Sudachi Dictionary package. This provides the most comprehensive linguistic data.

```bash
pip install sudachidict_full
```

--------------------------------

### Install SudachiDict-core

Source: https://github.com/worksapplications/sudachidict/blob/develop/python/README.md

Installs the core Sudachi Dictionary package. Use this for a balanced set of features.

```bash
pip install sudachidict_core
```

--------------------------------

### Install Dictionary via S3 or Private Mirror

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/09-deployment-and-operations.md

Alternative installation methods for when PyPI is unavailable or for private index usage.

```bash
pip install https://d2ej7fkh96fzlu.cloudfront.net/sudachidict/sudachi-dictionary-20260723-core.zip
```

```bash
pip install -i https://pypi.example.com/simple sudachidict_core
```

--------------------------------

### Python Installation Dependency Chain

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/07-architecture-and-data-flow.md

Overview of the installation flow for sudachidict_core, including JNI bindings and resource placement.

```text
pip install sudachidict_core
   │
   ├─ SudachiPy >= 0.5.0, < 0.7.0
   │  └─ Sudachi (JNI binding to Java)
   │     └─ Java Runtime Environment
   │
   └─ Dictionary Binary (downloaded at install time)
      └─ Placed in site-packages/sudachidict_core/resources/
```

--------------------------------

### Dictionary Configuration Examples

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/README.md

Common configuration parameters for dictionary builds and package requirements.

```text
dict.version: 20260723
dict.release: false
DictionaryBuilder -o: system_core.dic
install_requires: ["SudachiPy>=0.5,<0.7"]
```

--------------------------------

### Install Build Dependencies

Source: https://github.com/worksapplications/sudachidict/blob/develop/RELEASE.md

Installs necessary dependencies for the build environment from a requirements file.

```bash
pip install -r scripts/requirements.txt
```

--------------------------------

### Install SudachiDict-small

Source: https://github.com/worksapplications/sudachidict/blob/develop/python/README.md

Installs the small Sudachi Dictionary package. This is suitable for applications with limited resource constraints.

```bash
pip install sudachidict_small
```

--------------------------------

### Install and Use SudachiDict in Python

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/07-architecture-and-data-flow.md

Commands and code for installing the dictionary package via pip and initializing the tokenizer in Python.

```bash
$ pip install sudachidict_core
    ├─ PyPI download
    ├─ setup.py executes
    ├─ Downloads sudachi-dictionary-20260723-core.zip from CloudFront
    ├─ Extracts system.dic
    ├─ Places in sudachidict_core/resources/
    └─ SudachiPy auto-discovers at runtime

$ python
>>> from sudachipy.tokenizer import Tokenizer
>>> tokenizer = Tokenizer.create()  # Auto-discovers sudachidict_core
>>> tokens = tokenizer.tokenize("こんにちは")
```

--------------------------------

### Verify PyPI Installation

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/09-deployment-and-operations.md

Commands to validate the package installation and verify the tokenizer import in a Python environment.

```bash
pip install --upgrade sudachidict_core
from sudachipy.tokenizer import Tokenizer
```

--------------------------------

### Setup Release Virtual Environment

Source: https://github.com/worksapplications/sudachidict/blob/develop/RELEASE.md

Sets up a Python virtual environment for the release process. Activate the environment using 'source .venv/bin/activate'.

```bash
python3 -m venv .venv
source .venv/bin/activate
```

--------------------------------

### Integrate Sudachidict in Java

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/10-quick-reference.md

Example of initializing a dictionary and tokenizing text in a Java application.

```java
import com.worksap.nlp.sudachi.*;
import java.nio.file.Paths;

public class MyApp {
    public static void main(String[] args) throws IOException {
        Config cfg = Config.defaultConfig()
            .systemDictionary(Paths.get("system_core.dic"));
        Dictionary dict = new DictionaryFactory().create(cfg);
        JapaneseTokenizer tokenizer = (JapaneseTokenizer) dict.create();
        
        List<Morpheme> tokens = tokenizer.tokenize("こんにちは");
        for (Morpheme m : tokens) {
            System.out.println(m.surface() + " : " + m.partOfSpeech());
        }
    }
}
```

--------------------------------

### Integrate Sudachidict in Python

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/10-quick-reference.md

Example of initializing a tokenizer and processing text in a Python application.

```python
from sudachipy.tokenizer import Tokenizer

# Auto-discovers sudachidict_core
tokenizer = Tokenizer.create()

tokens = tokenizer.tokenize("こんにちは", Tokenizer.SplitMode.C)
for token in tokens:
    print(f"{token.surface()} : {token.part_of_speech()}")
```

--------------------------------

### Install Full Edition via Source Distribution

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/06-release-and-distribution.md

Forces pip to install from the source distribution to bypass wheel size limitations.

```bash
pip install --no-binary :all: SudachiDict-full
```

--------------------------------

### Version Output Format

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/11-api-surface-summary.md

Example output formats for the versioning script.

```text
Release: 20260723
Snapshot: v1.0-5-g1a2b3c4-SNAPSHOT
Dirty: v1.0-5-g1a2b3c4-SNAPSHOT+dirty
```

--------------------------------

### Install Python Package Locally

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/10-quick-reference.md

Installs the sudachidict package in development mode. Note that dictionaries are fetched from S3 rather than local build outputs.

```bash
pip install -e python/
```

--------------------------------

### Synonym Source Format Example (Full)

Source: https://github.com/worksapplications/sudachidict/blob/develop/docs/synonyms.md

This example demonstrates the full CSV format for the synonym source, including all fields.

```csv
000127,1,0,1,0,0,0,(医療),インフルエンザ,,
000127,1,0,1,0,0,1,(医療),influenza,,
000127,1,0,1,0,2,0,(医療),インフル,,
000127,1,0,2,0,0,0,(医療),流行性感冒,,
000127,1,0,2,0,2,0,(医療),流感,,

000008,1,0,1,0,0,0,(),支払い,,
000008,1,0,1,0,0,2,(),支払,,
000008,2,0,2,0,0,0,(),支払う,,
000008,1,1,3,0,0,0,(),勘定,,
000008,1,1,4,0,0,0,(),精算,,
000008,1,1,5,0,0,0,(),会計,,
000008,1,1,6,0,0,0,(),御愛想,,
000008,1,1,6,0,0,2,(),お愛想,,
```

--------------------------------

### Compile Dictionary Variants

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/05-dictionary-compilation.md

Examples of compiling different dictionary sizes by specifying source names.

```bash
DictionaryBuilder -o system_small.dic -m matrix.def -d 20260723 small
```

```bash
DictionaryBuilder -o system_core.dic -m matrix.def -d 20260723 small core
```

```bash
DictionaryBuilder -o system_full.dic -m matrix.def -d 20260723 small core notcore
```

--------------------------------

### Package directory structure

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/04-python-package-api.md

The expected layout of the installed package directory containing the dictionary binary.

```text
sudachidict_core/                 # Package directory (PKG_DIR)
├── __init__.py                   # Package marker
├── resources/                    # RESOURCE_DIR
│   └── system.dic                # Compiled dictionary binary
└── ... (other package files)
```

--------------------------------

### Package Versioning Mapping

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/09-deployment-and-operations.md

Example mapping of PyPI package versions to specific dictionary binary releases.

```text
SudachiDict-core 1.0.0 ──→ sudachi-dictionary-20260723-core.zip
SudachiDict-core 1.0.1 ──→ sudachi-dictionary-20260723-core.zip (pkg fix)
SudachiDict-core 2.0.0 ──→ sudachi-dictionary-20260801-core.zip (new dict)
```

--------------------------------

### Define Dictionary ZIP URL in Python

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/07-architecture-and-data-flow.md

URL construction logic used in setup.py to locate dictionary packages for installation.

```python
ZIP_URL = "https://d2ej7fkh96fzlu.cloudfront.net/sudachidict/" \
          "sudachi-dictionary-{}-{}.zip".format(DICT_VERSION, DICT_EDITION)
```

--------------------------------

### Install SudachiDict Packages

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/04-python-package-api.md

Use pip to install the desired dictionary edition. Requires SudachiPy version 0.5.0 or higher, but less than 0.7.0.

```bash
pip install sudachidict_core
```

```bash
pip install sudachidict_small
pip install sudachidict_full
```

--------------------------------

### Synonym Dictionary CSV Example

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/11-api-surface-summary.md

Sample entries demonstrating the synonym dictionary CSV format.

```csv
000127,1,0,1,0,0,0,(医療),インフルエンザ,,
000127,1,0,1,0,0,1,(医療),influenza,,
000127,1,0,1,0,2,0,(医療),インフル,,
```

--------------------------------

### Synonym Source Format Example (Abbreviated)

Source: https://github.com/worksapplications/sudachidict/blob/develop/docs/synonyms.md

This example shows the synonym source format with omitted fields, demonstrating a more concise representation.

```csv
000127,,,,,,,,インフルエンザ,,
000127,,,,,,,,influenza,,
000127,,,,,,,,インフル,,,
000127,,,,,,,,流行性感冒,,
000127,,,,,,,,流感,,
```

--------------------------------

### Memory Scaling Example

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/09-deployment-and-operations.md

Calculates memory usage based on dictionary size and concurrent tokenizer threads.

```text
1 server, core dict: 100 MB + 50 MB (tokenizer overhead) = 150 MB
10 servers, 4 tokenizer threads each: 10 × 4 × 10 MB = 400 MB
```

--------------------------------

### Define dictionary download paths

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/04-python-package-api.md

Variables used during the installation process to locate and extract the dictionary binary.

```python
ZIP_URL = "https://d2ej7fkh96fzlu.cloudfront.net/sudachidict/" \
          "sudachi-dictionary-{}-{}.zip".format(DICT_VERSION, DICT_EDITION)
ZIP_NAME = urlparse(ZIP_URL).path.split("/")[-1]
UNZIP_NAME = "sudachi-dictionary-{}".format(DICT_VERSION)
PKG_DIR = "sudachidict_{}".format(DICT_EDITION)
RESOURCE_DIR = os.path.join(PKG_DIR, "resources")
BINARY_NAME = "system_{}.dic".format(DICT_EDITION)
```

--------------------------------

### Storage Estimates

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/09-deployment-and-operations.md

Provides approximate disk space requirements for Python package installation, build caches, and S3 storage.

```text
Python package (installed): 150-600 MB per edition
  + site-packages overhead: 5-10 MB
  + total in single venv: ~600 MB (if all three installed)

Build cache (gradle): 200-300 MB per dictionary version
  (shared across team, reusable)

PyPI bandwidth: Scales with popularity
  (externally managed, transparent to SudachiDict)

S3 storage: ~1 GB per 10 recent versions
```

--------------------------------

### CI/CD Gradle Cache Configuration

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/08-configuration-and-customization.md

Example GitHub Actions workflow configuration to cache Gradle dependencies and build artifacts.

```yaml
# .github/workflows/build.yml example
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/cache@v3
        with:
          path: build/cache
          key: ${{ runner.os }}-gradle-cache
          restore-keys: |
            ${{ runner.os }}-gradle-
      - run: ./gradlew build
```

--------------------------------

### View Build Output Directory Structure

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/06-release-and-distribution.md

Displays the expected directory layout after running the release script.

```text
build/python/
├── dist/           # Source distributions (.tar.gz)
├── wheels/         # Binary wheels (.whl)
└── ... (metadata, build artifacts)
```

--------------------------------

### Initialize NotCoreTest class

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/05-dictionary-compilation.md

Sets up the tokenizer using the system_full.dic dictionary for testing extended vocabulary.

```java
public class NotCoreTest extends SudachiTest {
    @Before
    public void setUp() throws IOException {
        getTokenizer("system_full.dic");
    }
    // ... test methods
}
```

--------------------------------

### Run Release Process

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/10-quick-reference.md

Automates the release process, including uploading binary dictionaries to S3 and preparing Python packages for PyPI.

```bash
bash do_release.sh 20260723 /path/to/csv aws-profile arn:aws:iam::ACCOUNT:mfa/user
```

--------------------------------

### Configure setuptools for SudachiDict

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/04-python-package-api.md

Defines the package metadata and dependencies required for the SudachiDict Python package.

```python
setuptools.setup(
    name="SudachiDict-{}".format(DICT_EDITION),
    version=PKG_VERSION,
    description="Sudachi Dictionary for SudachiPy - {} Edition".format(DICT_EDITION.title()),
    url="https://github.com/WorksApplications/SudachiDict",
    license="Apache-2.0",
    author="Works Applications",
    author_email="sudachi@worksap.co.jp",
    packages=setuptools.find_packages(),
    package_data={"": ["resources/*"]},
    install_requires=[
        "SudachiPy>=0.5,<0.7"
    ],
)
```

--------------------------------

### Retrieve Package Metadata

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/07-architecture-and-data-flow.md

Uses importlib.metadata to access the version information of the installed dictionary package.

```python
import importlib.metadata
version_info = importlib.metadata.distribution('sudachidict-core')
# Returns package version (e.g., "1.0.0")
# Dictionary version encoded in package metadata
```

--------------------------------

### Execute Build and Tests

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/08-configuration-and-customization.md

Runs a clean build and test suite via Gradle.

```bash
./gradlew clean build
```

--------------------------------

### Build All Dictionaries

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/10-quick-reference.md

Executes a full build of all dictionary variants, including system_small, system_core, and system_full.

```bash
./gradlew build
```

--------------------------------

### CI/CD Build Time Estimates

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/09-deployment-and-operations.md

Breakdown of build times for clean versus cached environments.

```text
Clean build (all editions): 30-60 minutes
  - Download: 5-10 minutes
  - Compile small: 5 minutes
  - Compile core: 10 minutes
  - Compile full: 10-15 minutes
  - Test: 5 minutes

Cached build (with downloads cached): 15-20 minutes
```

--------------------------------

### GET PyPI Package JSON API

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/11-api-surface-summary.md

Retrieves metadata about the SudachiDict-core package from PyPI in JSON format.

```APIDOC
## GET https://pypi.org/pypi/SudachiDict-core/json

### Description
Fetches package metadata for SudachiDict-core from the PyPI JSON API.

### Method
GET

### Endpoint
https://pypi.org/pypi/SudachiDict-core/json
```

--------------------------------

### Show Current Version

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/10-quick-reference.md

Displays the current dictionary version string.

```bash
./gradlew showVersion
```

--------------------------------

### GET Download Compiled Dictionary

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/11-api-surface-summary.md

Retrieves a compiled dictionary ZIP file from the distribution S3 bucket.

```APIDOC
## GET https://d2ej7fkh96fzlu.cloudfront.net/sudachidict/sudachi-dictionary-{VERSION}-{EDITION}.zip

### Description
Downloads the compiled dictionary archive for a specific version and edition.

### Method
GET

### Endpoint
https://d2ej7fkh96fzlu.cloudfront.net/sudachidict/sudachi-dictionary-{VERSION}-{EDITION}.zip
```

--------------------------------

### Define S3 Compiled Binary Structure

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/06-release-and-distribution.md

Represents the directory hierarchy for compiled dictionary binaries used during installation.

```text
s3://sudachi/sudachidict/
├── sudachi-dictionary-{version}-small.zip
├── sudachi-dictionary-{version}-core.zip
├── sudachi-dictionary-{version}-full.zip
├── sudachi-dictionary-latest-small.zip
├── sudachi-dictionary-latest-core.zip
└── sudachi-dictionary-latest-full.zip
```

--------------------------------

### Influenza Synonym CSV Data

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/03-synonym-dictionary-format.md

Example CSV entries demonstrating the hierarchical fields for the influenza synonym group.

```csv
000127,1,0,1,0,0,0,(医療),インフルエンザ,,
000127,1,0,1,0,0,1,(医療),influenza,,
000127,1,0,1,0,2,0,(医療),インフル,,
000127,1,0,2,0,0,0,(医療),流行性感冒,,
000127,1,0,2,0,2,0,(医療),流感,,
```

--------------------------------

### Common Gradle Build Commands

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/02-build-system.md

Standard commands for building, testing, and cleaning the project artifacts.

```bash
# Full build (all dictionaries and tests)
./gradlew build

# Build without tests
./gradlew build -x test

# Compile only core dictionary
./gradlew compileCoreDict

# Show current version
./gradlew showVersion

# Clean build artifacts
./gradlew clean
```

--------------------------------

### Development Build Optimization

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/08-configuration-and-customization.md

Speed up development cycles by skipping full dictionary compilation or using pre-built dictionary files.

```bash
# Build without full edition (faster)
./gradlew build -x compileFullDict

# Or use pre-built dictionaries
cp /production/system_core.dic build/dict/bin/20260723/
./gradlew test
```

--------------------------------

### GET Download Raw CSV Sources

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/11-api-surface-summary.md

Retrieves the raw CSV source files for a specific dictionary version and edition.

```APIDOC
## GET https://d2ej7fkh96fzlu.cloudfront.net/sudachidict-raw/{VERSION}/{EDITION}_lex.zip

### Description
Downloads the raw CSV source files for the specified version and edition.

### Method
GET

### Endpoint
https://d2ej7fkh96fzlu.cloudfront.net/sudachidict-raw/{VERSION}/{EDITION}_lex.zip
```

--------------------------------

### Execute Basic Release Script

Source: https://github.com/worksapplications/sudachidict/blob/develop/RELEASE.md

Runs the main release script with required arguments for dictionary version, CSV paths, AWS profile, and MFA ARN. An optional Python package version can also be provided.

```bash
bash do_release.sh DIC_VERSION /path/to/csv/dics aws-profile arn:aws:iam::0123456789:mfa/iam_user
```

--------------------------------

### Remove deprecated dictionary link command

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/04-python-package-api.md

The 'sudachipy link' command is removed in v0.5.2+. Use pip installation for automatic discovery.

```bash
sudachipy link sudachidict_core
```

--------------------------------

### MANIFEST.in Configuration

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/04-python-package-api.md

Ensures that compiled dictionary binary files are included in the Python wheel distribution.

```text
recursive-include sudachidict_* resources
```

--------------------------------

### Display current version

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/08-configuration-and-customization.md

Execute the Gradle task to print the current version string to standard output.

```bash
./gradlew showVersion
# Prints current version to stdout
```

--------------------------------

### Initialize Dictionary in Java Tests

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/02-build-system.md

Retrieves the build directory from system properties to configure and instantiate a dictionary object.

```java
Path basicPath = Paths.get(System.getProperty("buildDirectory"));
Config cfg = Config.defaultConfig().systemDictionary(basicPath.resolve("system_core.dic"));
Dictionary dict = new DictionaryFactory().create(cfg);
```

--------------------------------

### DictionaryBuilder Command Pattern

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/02-build-system.md

Shows the command-line usage for the DictionaryBuilder tool.

```bash
DictionaryBuilder -o <output.dic> -m matrix.def -d {version} {source_names}
```

--------------------------------

### Build Dependency Graph

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/02-build-system.md

Visual representation of the task execution order for the build process.

```text
[build] (main target)
  ├── zipSmallDict
  │   └── compileSmallDict
  │       ├── unzipSmallDict
  │       │   └── downloadSmallZip
  │       └── unzipMatrixDef
  │           └── downloadMatrixDef
  ├── zipCoreDict
  │   └── compileCoreDict
  │       ├── unzipSmallDict / unzipCoreDict
  │       └── unzipMatrixDef
  └── zipFullDict
      └── compileFullDict
          ├── unzipSmallDict / unzipCoreDict / unzipNotcoreDict
          └── unzipMatrixDef
```

--------------------------------

### Configure Raw Dictionary Upload Options

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/06-release-and-distribution.md

Configuration class for the raw dictionary upload script.

```python
@classopt(default_long=True)
class Opts:
    input: Path                      # Input directory with small_lex.zip, etc.
    version: str                     # Dictionary version (e.g., 20260723)
    aws_profile: str                 # AWS credentials profile
    aws_mfa: str                     # MFA device ARN
    aws_region: str = "ap-northeast-1"
    s3_bucket: str = "sudachi"
    s3_prefix: str = "sudachidict-raw"
```

--------------------------------

### Define Build Properties

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/02-build-system.md

Sets the dictionary version and release status in the gradle.properties file.

```properties
dict.version=20260723
dict.release=false
```

--------------------------------

### Integrate SudachiPy with Dictionary Packages

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/04-python-package-api.md

Demonstrates automatic dictionary discovery and explicit configuration for SudachiPy tokenizers.

```python
from sudachipy.tokenizer import Tokenizer

# Automatically finds sudachidict_core if installed
tokenizer = Tokenizer.create()

# Or explicitly specify dictionary edition
config = {
    "systemDict": None,  # Let SudachiPy auto-discover from installed packages
}
tokenizer = Tokenizer.create(config)

tokens = tokenizer.tokenize("京都", SplitMode.C)
```

--------------------------------

### Optimize Gradle Build Performance

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/10-quick-reference.md

Commands to speed up builds by skipping tasks, using cached directories, or enabling parallel execution.

```bash
# Skip slow full dictionary
./gradlew build -x compileFullDict

# Use cached downloads
export GRADLE_USER_HOME=/shared/gradle-cache

# Parallel build (if available)
./gradlew build -j 8
```

--------------------------------

### Execute dictionary tests via CLI

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/05-dictionary-compilation.md

Commands to run the full test suite or target specific test classes and methods.

```bash
# All tests
./gradlew test

# Specific test class
./gradlew test --tests CoreTest

# Specific test method
./gradlew test --tests CoreTest.partOfSpeech
```

--------------------------------

### Measure Tokenization Performance

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/09-deployment-and-operations.md

Includes a JIT warmup loop before measuring the execution time of the tokenizer.

```java
// Warm up JIT
for (int i = 0; i < 1000; i++) {
    tokenizer.tokenize("暖機運転");
}

// Measure
long start = System.nanoTime();
List<Morpheme> morphemes = tokenizer.tokenize("こんにちは");
long timeUs = (System.nanoTime() - start) / 1000;
System.out.println("Tokenization: " + timeUs + "µs");
```

--------------------------------

### Initialize Sudachi Dictionary in Java

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/01-project-overview.md

Configures the dictionary path and initializes the tokenizer for morphological analysis.

```java
Config cfg = Config.defaultConfig()
    .systemDictionary(dictionaryPath.resolve("system_core.dic"));
Dictionary dict = new DictionaryFactory().create(cfg);
JapaneseTokenizer tokenizer = (JapaneseTokenizer) dict.create();
List<Morpheme> morphemes = tokenizer.tokenize("京都");
```

--------------------------------

### Run Tests

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/10-quick-reference.md

Executes the test suite, covering morphology, POS tagging, and extended vocabulary handling.

```bash
./gradlew test
```

--------------------------------

### Initialize Sudachi Dictionary in Test Suite

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/05-dictionary-compilation.md

Base class for initializing the Sudachi tokenizer using a dictionary file from the build directory.

```java
abstract class SudachiTest {
    Dictionary dict;
    JapaneseTokenizer tokenizer;

    void getTokenizer(String dictName) throws IOException {
        Path basicPath = Paths.get(System.getProperty("buildDirectory"));
        Config cfg = Config.defaultConfig()
            .systemDictionary(basicPath.resolve(dictName));
        dict = new DictionaryFactory().create(cfg);
        tokenizer = (JapaneseTokenizer)dict.create();
    }
}
```

--------------------------------

### Upload packages to PyPI

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/04-python-package-api.md

Use twine to upload distribution files and wheels to PyPI, requiring specific environment variables for authentication.

```bash
twine upload dist/* wheels/*.whl
```

--------------------------------

### Dictionary Download Tasks

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/02-build-system.md

Lists the Gradle tasks responsible for fetching raw dictionary and matrix definition files.

```gradle
downloadSmallZip        // Downloads small_lex.zip
downloadCoreZip         // Downloads core_lex.zip
downloadNotcoreZip      // Downloads notcore_lex.zip
downloadMatrixDef       // Downloads matrix.def.zip (MeCab format definition)
```

--------------------------------

### Dictionary Packaging Tasks

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/02-build-system.md

Lists the Gradle tasks responsible for packaging compiled dictionaries into distribution ZIP files.

```gradle
zipSmallDict            // Creates: sudachi-dictionary-{version}-small.zip
zipCoreDict             // Creates: sudachi-dictionary-{version}-core.zip
zipFullDict             // Creates: sudachi-dictionary-{version}-full.zip
```

--------------------------------

### Configure Dictionary Caching

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/08-configuration-and-customization.md

Uses a shared Gradle user home directory to persist downloaded dictionary files across builds.

```bash
# Share gradle cache directory
export GRADLE_USER_HOME=/shared/gradle-cache
./gradlew build

# Second build reuses downloaded ZIPs
./gradlew clean build  # Downloads skipped if cache still exists
```

--------------------------------

### Execute Release Script

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/09-deployment-and-operations.md

Run the automated release process using the specified dictionary version, source path, and AWS profile.

```bash
bash do_release.sh 20260723 /path/to/csv aws-profile arn:...
```

--------------------------------

### Configure Gradle Plugins

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/02-build-system.md

Defines the required plugins for the Java library and external file downloads.

```gradle
plugins {
    id 'java-library'
    id "de.undercouch.download" version "5.4.0"
}
```

--------------------------------

### Configure Compiled Dictionary Upload Options

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/06-release-and-distribution.md

Configuration class for the compiled dictionary upload script.

```python
@classopt(default_long=True)
class Opts:
    input: Path                      # Directory with built .zip files
    version: str = None              # Filter by version (optional)
    aws_profile: str = None
    aws_mfa: str = None
    aws_region: str = "ap-northeast-1"
    s3_bucket: str = "sudachi"
    s3_prefix: str = "sudachidict"
    no_latest: bool = False          # Skip uploading "latest" symlinks
```

--------------------------------

### Project Directory Structure

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/01-project-overview.md

Visual representation of the project source tree.

```text
sudachidict/
├── src/
│   ├── main/text/
│   │   └── synonyms.txt          # Synonym dictionary source (97,637 lines)
│   └── test/java/                # Integration tests for dictionary compilation
├── python/                         # Python package wrapper
│   ├── setup.py                  # PyPI distribution configuration
│   ├── INFO.json                 # Version and edition metadata
│   └── README.md                 # Python package documentation
├── scripts/                        # Deployment and release automation
├── build.gradle                   # Primary build configuration
└── gradle.properties              # Build properties and versions
```

--------------------------------

### Configure Twine for PyPI Upload

Source: https://github.com/worksapplications/sudachidict/blob/develop/RELEASE.md

Sets environment variables for Twine to authenticate with PyPI using token-based authentication. Ensure these commands are run in a way that does not save them to bash history.

```bash
 export TWINE_USERNAME=__token__
 export TWINE_PASSWORD=pypi-starting-login-string
 export TWINE_REPOSITORY_URL=https://upload.pypi.org/legacy/
```

--------------------------------

### Configure Maven Repositories

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/08-configuration-and-customization.md

Configures the project to resolve dependencies from Maven Central or custom mirrors. Use mirrors to improve download speeds in restricted network environments.

```gradle
repositories {
    mavenCentral()
}
```

```gradle
repositories {
    maven {
        url "https://mirrors.aliyun.com/maven/repository/mavenCentral"  // Aliyun mirror
    }
    mavenCentral()  // Fallback to official repo
}
```

--------------------------------

### Load Metadata from INFO.json

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/08-configuration-and-customization.md

Reads package and dictionary metadata from a local JSON file.

```python
with open("INFO.json") as fh:
    dict_info = json.load(fh)
PKG_VERSION = dict_info["version"]
DICT_VERSION = dict_info["dict_version"]
DICT_EDITION = dict_info["edition"]
```

--------------------------------

### Build Core Dictionary

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/10-quick-reference.md

Compiles only the core dictionary to reduce build time.

```bash
./gradlew compileCoreDict
```

--------------------------------

### Perform Clean Build

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/10-quick-reference.md

Removes existing build artifacts before running a fresh build.

```bash
rm -rf build/
./gradlew clean build
```

--------------------------------

### Tokenize Text with SudachiPy

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/11-api-surface-summary.md

Initialize the tokenizer and perform morphological analysis using the default dictionary.

```python
from sudachipy.tokenizer import Tokenizer

tokenizer = Tokenizer.create()  # Auto-discovers sudachidict_core
tokens = tokenizer.tokenize("text", Tokenizer.SplitMode.C)
```

--------------------------------

### Gradle Configuration for Custom Dictionaries

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/08-configuration-and-customization.md

Update build.gradle to include custom source files and register a task to unzip the dictionary source.

```gradle
def dictionarySrc = [
    core: ['small', 'core', 'custom']  // Add custom
]

// Add task to prepare custom source
tasks.register('unzipCustomDict', Copy) {
    from file('src/main/resources/custom_lex.zip')
    into new File(project.buildDir, "dict/raw/${version}/custom")
}

// Update compile task dependency
compileCoreDict.dependsOn(unzipCustomDict)
```

--------------------------------

### Configure JVM for Compilation

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/08-configuration-and-customization.md

Sets heap size and system properties for dictionary compilation tasks. Adjust these values based on available system memory and encoding requirements.

```gradle
t.maxHeapSize = "4g"
t.systemProperty('file.encoding', 'UTF-8')
t.systemProperty('buildDirectory', builtDictDir.toString())
```

```gradle
// For 8GB available memory
t.maxHeapSize = "6g"

// For 2GB system (small dictionaries only)
t.maxHeapSize = "2g"
```

--------------------------------

### Measure Dictionary Loading Time

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/09-deployment-and-operations.md

Tracks the time taken to initialize the dictionary factory in Java.

```java
long start = System.nanoTime();
Dictionary dict = new DictionaryFactory().create(cfg);
long loadTimeMs = (System.nanoTime() - start) / 1_000_000;
System.out.println("Dictionary loaded in " + loadTimeMs + "ms");
```

--------------------------------

### Build Cache Directory Structure

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/07-architecture-and-data-flow.md

Layout of the cached dictionary files within the build directory.

```text
build/cache/
├── matrix.def.zip
└── {version}/
    ├── small/
    │   └── small_lex.zip
    ├── core/
    │   └── core_lex.zip
    └── notcore/
        └── notcore_lex.zip
```

--------------------------------

### Configure Gradle Test System Properties

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/02-build-system.md

Sets the build directory and encoding for test tasks within the Gradle build script.

```gradle
systemProperty('buildDirectory', builtDictDir.toString())
systemProperty('file.encoding', 'UTF-8')
```

--------------------------------

### System Architecture Diagram

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/09-deployment-and-operations.md

Visual representation of the data flow from S3 buckets through CloudFront to end-user distribution channels.

```text
┌─────────────────────────────────────┐
│        Amazon S3 Buckets            │
├─────────────────────────────────────┤
│ Bucket: sudachi                     │
│ ├─ sudachidict-raw/                │
│ │  ├─ {version}/                   │
│ │  │  ├─ small_lex.zip             │
│ │  │  ├─ core_lex.zip              │
│ │  │  └─ notcore_lex.zip           │
│ │  └─ matrix.def.zip               │
│ └─ sudachidict/                    │
│    ├─ sudachi-dictionary-*.zip     │
│    └─ index.html (listing)         │
└─────────────────────┬───────────────┘
                      │ (cached & distributed)
                      ▼
        ┌─────────────────────────┐
        │   CloudFront CDN        │
        │   d2ej7fkh96fzlu.     │
        │   cloudfront.net       │
        └─────────────────────────┘
                      │
        ┌─────────────┼─────────────┐
        ▼             ▼             ▼
    PyPI         Direct HTTP    GitHub
    (setup.py)   Download       Releases
```

--------------------------------

### Invoke Release Automation Script

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/11-api-surface-summary.md

Executes the full dictionary release pipeline including compilation, S3 uploads, and package building.

```bash
bash do_release.sh DIC_VERSION CSV_PATH AWS_PROFILE MFA_ARN [PKG_VERSION]
```

--------------------------------

### Invoke Raw Dictionary Upload Script

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/06-release-and-distribution.md

Command line execution for uploading raw dictionary files to S3.

```bash
python scripts/01_upload_raw_dictionaries.py \
    --input /path/to/dicts \
    --version 20260723 \
    --aws-profile sudachi \
    --aws-mfa arn:aws:iam::ACCOUNT:mfa/user
```

--------------------------------

### Configure Dictionary CDN URL

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/08-configuration-and-customization.md

Sets the remote URL for downloading dictionary files.

```python
ZIP_URL = "https://d2ej7fkh96fzlu.cloudfront.net/sudachidict/" \
          "sudachi-dictionary-{}-{}.zip".format(DICT_VERSION, DICT_EDITION)
```

```python
# Use alternate CDN
ZIP_URL = "https://my-cdn.example.com/sudachidict/" \
          "sudachi-dictionary-{}-{}.zip".format(DICT_VERSION, DICT_EDITION)

# Or local file path
ZIP_URL = f"file:///data/dictionaries/sudachi-dictionary-{DICT_VERSION}-{DICT_EDITION}.zip"
```

--------------------------------

### Dictionary Compilation Workflow

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/10-quick-reference.md

Visual representation of the dictionary compilation task dependencies.

```text
downloadSmallZip → unzipSmallDict ┐
downloadCoreZip  → unzipCoreDict  ├→ compileSmallDict → zipSmallDict
downloadNotcoreZip → unzipNotcoreDict
downloadMatrixDef → unzipMatrixDef
```

--------------------------------

### Register Custom Matrix Definition

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/08-configuration-and-customization.md

Copies a custom matrix definition file into the build directory for domain-specific tokenization.

```gradle
def matrixDefDownload = tasks.register("downloadMatrixDef", Copy) {
    from './src/main/resources/custom-matrix.def'  // Use local file
    into new File(project.buildDir, "dict/raw/matrix")
}
```

--------------------------------

### DictionaryBuilder Command Signature

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/05-dictionary-compilation.md

The basic command-line syntax for invoking the DictionaryBuilder tool.

```bash
DictionaryBuilder -o <output_file> -m <matrix_file> -d <version> [sources...]
```

--------------------------------

### Download Dictionary ZIP via Gradle

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/07-architecture-and-data-flow.md

Gradle task configuration for fetching raw dictionary ZIP files from S3 during the build process.

```gradle
def downloadTask = tasks.register("download${capitalName}Zip", Download) {
    src "https://d2ej7fkh96fzlu.cloudfront.net/sudachidict-raw/${version}/${name}_lex.zip"
    dest new File(downloadCacheDir, "${version}/$name/$filename")
    overwrite false
}
```

--------------------------------

### Execute Gradle Test Commands

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/08-configuration-and-customization.md

Various command-line operations for running, filtering, or skipping tests during the build process.

```bash
./gradlew test
```

```bash
./gradlew test --tests CoreTest
```

```bash
./gradlew build -x test
```

```bash
./gradlew test -x compileCoreDict -x compileFullDict
# Caution: Tests will fail if dictionaries aren't pre-built
```

--------------------------------

### Optimize Memory Usage

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/08-configuration-and-customization.md

Limits heap size and restricts dictionary builds to the small subset to accommodate limited RAM environments.

```gradle
t.maxHeapSize = "2g"

// And limit to small dictionary only
def dictionarySrc = [
    small: ['small']
]
```

--------------------------------

### INFO.json Metadata Structure

Source: https://github.com/worksapplications/sudachidict/blob/develop/_autodocs/04-python-package-api.md

Defines the package versioning and dictionary edition metadata used during the build process.

```json
{
    "version": "%%VERSION%%",
    "dict_version": "%%DICT_VERSION%%",
    "edition": "%%DICT_TYPE%%"
}
```
Tokens:
Raw