Sudachi

https://github.com/worksapplications/sudachi
Admin
Sudachi is a Japanese morphological analyzer that performs text segmentation, part-of-speech...
Tokens:14,029
Snippets:95
Trust Score:8.4
Update:7 months ago
Show doc for...
Tokens:
Raw
### Sudachi Analysis with Extended Output (Command Line)

Source: https://github.com/worksapplications/sudachi/blob/develop/README.md

This command-line example shows how to use the '-a' option with Sudachi to get extended output, including the dictionary form, reading form, dictionary ID, synonym group ID list, and OOV flag for each token in the sentence '東京都へ行く'.

```bash
$ echo 東京都へ行く | java -jar target/sudachi.jar -a
東京都	名詞,固有名詞,地名,一般,*,*	東京都	東京都	トウキョウト	0	[]
へ	助詞,格助詞,*,*,*,*	へ	へ	ヘ	0	[]
行く	動詞,非自立可能,*,*,五段-カ行,終止形-一般	行く	行く	イク	0	[]
EOS
```

--------------------------------

### Basic Sudachi Analysis Example (Command Line)

Source: https://github.com/worksapplications/sudachi/blob/develop/README.md

This example demonstrates a basic morphological analysis of the Japanese sentence '東京都へ行く' using the Sudachi command-line tool. The output shows the segmented words, their part-of-speech tags, and their dictionary forms.

```bash
$ echo 東京都へ行く | java -jar target/sudachi.jar
東京都	名詞,固有名詞,地名,一般,*,*	東京都
へ	助詞,格助詞,*,*,*,*	へ
行く	動詞,非自立可能,*,*,五段-カ行,終止形-一般	行く
EOS
```

--------------------------------

### Sudachi Analysis with Mode A Splitting (Command Line)

Source: https://github.com/worksapplications/sudachi/blob/develop/README.md

This example demonstrates Sudachi's mode A splitting for the sentence '東京都へ行く'. Mode A divides text into the shortest units equivalent to UniDic short units, resulting in a more granular segmentation.

```bash
$ echo 東京都へ行く | java -jar target/sudachi.jar -m A
東京	名詞,固有名詞,地名,一般,*,*	東京
都	名詞,普通名詞,一般,*,*,*	都
へ	助詞,格助詞,*,*,*,*	へ
行く	動詞,非自立可能,*,*,五段-カ行,終止形-一般	行く
EOS
```

--------------------------------

### Match Twitter Handles with RegexOovProvider in Sudachi

Source: https://github.com/worksapplications/sudachi/blob/develop/docs/oov_handlers.md

This example configures Sudachi's RegexOovProvider to match Twitter handles, requiring a minimum length of 4 characters. It includes settings for class, IDs, cost, the regex pattern, POS tags, and a 'relaxed' boundary mode.

```json5
{
    "class": "com.worksap.nlp.sudachi.RegexOovProvider",
    "leftId": 5968,
    "rightId": 5968,
    "cost": 500,
    "regex": "^@[0-9a-z_\\.\\-‐]{4,}",
    "pos": [ "補助記号", "一般", "twitter", "*", "*", "*" ],
    "userPOS": "allow",
    "boundaries": "relaxed"
}
```

--------------------------------

### Match URLs with RegexOovProvider in Sudachi

Source: https://github.com/worksapplications/sudachi/blob/develop/docs/oov_handlers.md

This configuration demonstrates how to use Sudachi's RegexOovProvider to identify URLs. It employs a regex pattern that matches common URL structures starting with http, https, or www, along with associated metadata like POS tags and user POS settings.

```json5
{
  "class": "com.worksap.nlp.sudachi.RegexOovProvider",
  "leftId": 5968,
  "rightId": 5968,
  "cost": 500,
  "regex": "^(?:https?://|www)[\\-_.!~*'a-zA-Z0-9;/?:@&=+$,%#¯−―]+",
  "pos": [ "補助記号", "一般", "URL", "*", "*", "*" ],
  "userPOS": "allow"
}
```

--------------------------------

### Sudachi Analysis with Space Separation Output (Command Line)

Source: https://github.com/worksapplications/sudachi/blob/develop/README.md

This command-line example shows how to use the '-t' option with Sudachi to output segmented words separated by spaces for the sentence '東京都へ行く'. This provides a simple, space-delimited list of the analyzed tokens.

```bash
$ echo 東京都へ行く | java -jar target/sudachi.jar -t
東京都 へ 行く
```

--------------------------------

### Tokenize Japanese Text with Sudachi (Java)

Source: https://context7.com/worksapplications/sudachi/llms.txt

Demonstrates basic Japanese text tokenization using Sudachi's Tokenizer interface in Java. It shows how to tokenize text with default Mode C (longest units), and also illustrates tokenization with Mode A (shortest units) and Mode B (middle units). Includes examples of tokenizing multiple sentences and lazy tokenization for large inputs.

```java
import com.worksap.nlp.sudachi.*;
import java.io.IOException;
import java.io.StringReader;
import java.util.Iterator;
import java.util.List;

public class TokenizerExample {
    public static void main(String[] args) throws IOException {
        try (Dictionary dict = new DictionaryFactory().create(Config.defaultConfig())) {
            Tokenizer tokenizer = dict.create();

            // Basic tokenization with default Mode C (longest units)
            MorphemeList morphemes = tokenizer.tokenize("東京都へ行く");
            for (Morpheme m : morphemes) {
                System.out.printf("%s\t%s\t%s%n",
                    m.surface(),                           // Surface form: 東京都
                    String.join(",", m.partOfSpeech()),    // POS: 名詞,固有名詞,地名,一般,*,*
                    m.normalizedForm());                   // Normalized: 東京都
            }

            // Tokenization with Mode A (shortest units)
            MorphemeList shortUnits = tokenizer.tokenize(Tokenizer.SplitMode.A, "選挙管理委員会");
            // Result: 選挙 / 管理 / 委員 / 会

            // Tokenization with Mode B (middle units)
            MorphemeList middleUnits = tokenizer.tokenize(Tokenizer.SplitMode.B, "選挙管理委員会");
            // Result: 選挙 / 管理 / 委員会

            // Tokenization with Mode C (longest units - named entities)
            MorphemeList longUnits = tokenizer.tokenize(Tokenizer.SplitMode.C, "選挙管理委員会");
            // Result: 選挙管理委員会

            // Tokenize multiple sentences from text
            String text = "今日は天気がいい。明日も晴れるだろう。";
            for (MorphemeList sentence : tokenizer.tokenizeSentences(text)) {
                System.out.println("---" Sentence ---");
                for (Morpheme m : sentence) {
                    System.out.println(m.surface());
                }
            }

            // Lazy tokenization for large inputs (memory efficient)
            StringReader reader = new StringReader("大量のテキストデータを処理する。");
            Iterator<List<Morpheme>> lazyIterator = tokenizer.lazyTokenizeSentences(
                Tokenizer.SplitMode.C, reader);
            while (lazyIterator.hasNext()) {
                List<Morpheme> sentenceMorphemes = lazyIterator.next();
                // Process each sentence...
            }
        }
    }
}

```

--------------------------------

### Sudachi CLI: Basic Tokenization and Analysis

Source: https://context7.com/worksapplications/sudachi/llms.txt

Demonstrates basic tokenization using Sudachi's command-line interface. It shows default mode (C), detailed output (-a), shortest unit mode (A), and space-separated output (-t). It also illustrates how to specify dictionaries and process files.

```bash
# Basic tokenization (Mode C - longest units, default)
echo "東京都へ行く" | java -jar sudachi.jar
# Output:
# 東京都  名詞,固有名詞,地名,一般,*,*     東京都
# へ      助詞,格助詞,*,*,*,*     へ
# 行く    動詞,非自立可能,*,*,五段-カ行,終止形-一般       行く
# EOS

# Detailed output with dictionary form, reading, dictionary ID, and synonym groups
echo "東京都へ行く" | java -jar sudachi.jar -a
# Output:
# 東京都  名詞,固有名詞,地名,一般,*,*     東京都  東京都  トウキョウト    0       []
# へ      助詞,格助詞,*,*,*,*     へ      へ      ヘ      0       []
# 行く    動詞,非自立可能,*,*,五段-カ行,終止形-一般       行く    行く    イク    0       []
# EOS

# Mode A - shortest units (UniDic equivalent)
echo "東京都へ行く" | java -jar sudachi.jar -m A
# Output:
# 東京    名詞,固有名詞,地名,一般,*,*     東京
# 都      名詞,普通名詞,一般,*,*,*        都
# へ      助詞,格助詞,*,*,*,*     へ
# 行く    動詞,非自立可能,*,*,五段-カ行,終止形-一般       行く
# EOS

# Wakachi-gaki (space-separated) output
echo "東京都へ行く" | java -jar sudachi.jar -t
# Output: 東京都 へ 行く

# Use small dictionary instead of core
java -jar sudachi.jar -s '{"systemDict":"system_small.dic"}'

# Use user dictionary
java -jar sudachi.jar -s '{"userDict":["user1.dic","user2.dic"]}'

# Process input file and write to output file
java -jar sudachi.jar -o output.txt input.txt
```

--------------------------------

### Programmatic Sudachi Configuration (Java)

Source: https://context7.com/worksapplications/sudachi/llms.txt

Demonstrates building Sudachi configurations programmatically in Java. This includes setting system and user dictionaries, character definitions, and adding plugins with custom settings. It shows how to load configurations from files, JSON strings, and build them from scratch.

```java
import com.worksap.nlp.sudachi.*;
import java.io.IOException;
import java.nio.file.Paths;

public class ConfigExample {
    public static void main(String[] args) throws IOException {
        // Build config from scratch
        Config config = Config.empty()
            .systemDictionary(Paths.get("/path/to/system_core.dic"))
            .addUserDictionary(Paths.get("/path/to/user1.dic"))
            .addUserDictionary(Paths.get("/path/to/user2.dic"))
            .characterDefinition(Paths.get("/path/to/char.def"))
            .allowEmptyMorpheme(false);

        // Load from filesystem and merge with defaults
        Config fileConfig = Config.fromFile(Paths.get("sudachi.json"))
            .withFallback(Config.defaultConfig());

        // Load from JSON string
        Config jsonConfig = Config.fromJsonString(
            "{\"systemDict\":\"system_full.dic\",\"userDict\":[\"custom.dic\"]}",
            PathAnchor.filesystem(Paths.get("/dict/directory"))
        );

        // Add plugins programmatically
        Config pluginConfig = Config.empty()
            .systemDictionary(Paths.get("system_core.dic"));

        // Add input text plugin with settings
        pluginConfig.addInputTextPlugin(DefaultInputTextPlugin.class);

        pluginConfig.addInputTextPlugin(ProlongedSoundMarkInputTextPlugin.class)
            .addList("prolongedSoundMarks", "ー", "-", "〜")
            .add("replacementSymbol", "ー");

        // Add OOV provider plugin
        pluginConfig.addOovProviderPlugin(SimpleOovProviderPlugin.class)
            .addList("oovPOS", "補助記号", "一般", "*", "*", "*", "*")
            .add("leftId", 5968)
            .add("rightId", 5968)
            .add("cost", 3857);

        // Use configuration
        try (Dictionary dict = new DictionaryFactory().create(config)) {
            Tokenizer tokenizer = dict.create();
            MorphemeList result = tokenizer.tokenize("テスト文章");
        }
    }
}

```

--------------------------------

### Using Sudachi with a Specific System Dictionary (Command Line)

Source: https://github.com/worksapplications/sudachi/blob/develop/README.md

This command demonstrates how to run the Sudachi command-line tool and specify a particular system dictionary, such as 'system_small.dic', using a JSON configuration string. This allows for customized analysis based on the chosen dictionary.

```bash
java -jar sudachi-XX.jar -s '{"systemDict":"system_small.dic"}'
```

--------------------------------

### Java: Creating Sudachi Dictionary Instances with DictionaryFactory

Source: https://context7.com/worksapplications/sudachi/llms.txt

Shows how to create Sudachi Dictionary instances using the DictionaryFactory class in Java. It covers creating dictionaries with default configurations, from filesystem JSON files, with custom system and user dictionaries, and merging settings from JSON strings.

```java
import com.worksap.nlp.sudachi.*;
import java.io.IOException;
import java.nio.file.Paths;

public class DictionaryExample {
    public static void main(String[] args) throws IOException {
        // Create dictionary with default configuration (from classpath sudachi.json)
        try (Dictionary dict = new DictionaryFactory().create(Config.defaultConfig())) {
            Tokenizer tokenizer = dict.create();
            // Use tokenizer...
        }

        // Create dictionary from filesystem configuration file
        Config config = Config.fromFile(Paths.get("/path/to/sudachi.json"));
        try (Dictionary dict = new DictionaryFactory().create(config)) {
            Tokenizer tokenizer = dict.create();
            // Use tokenizer...
        }

        // Create dictionary with custom system dictionary path
        Config customConfig = Config.empty()
            .systemDictionary(Paths.get("/path/to/system_core.dic"))
            .addUserDictionary(Paths.get("/path/to/user.dic"));
        try (Dictionary dict = new DictionaryFactory().create(customConfig)) {
            Tokenizer tokenizer = dict.create();
            // Use tokenizer...
        }

        // Create dictionary from JSON string with merged settings
        Config mergedConfig = Config.fromJsonString(
            "{\"systemDict\":\"system_full.dic\"}",
            PathAnchor.filesystem(Paths.get("/dict/path"))
        ).withFallback(Config.defaultConfig());
        try (Dictionary dict = new DictionaryFactory().create(mergedConfig)) {
            Tokenizer tokenizer = dict.create();
            // Use tokenizer...
        }
    }
}
```

--------------------------------

### Sudachi Command-Line Tool Usage and Options

Source: https://github.com/worksapplications/sudachi/blob/develop/README.md

This command shows the general usage of the Sudachi command-line tool, including various options for configuration, mode selection, output formatting, and error handling. It allows users to customize the morphological analysis process.

```bash
java -jar sudachi-XX.jar [-r conf] [-s json] [-m mode] [-a] [-d] [-f] [-o output] [file...]
```

--------------------------------

### Configure Sudachi OOV Plugins (JSON5)

Source: https://github.com/worksapplications/sudachi/blob/develop/docs/oov_handlers.md

This configuration snippet shows how to define an array of OOV plugins for Sudachi. It allows reordering and overriding default configurations by specifying the class name. The order matters, as the first matching plugin is used.

```json5
{
  "oovPlugins": [
    {
      "class": "com.worksap.nlp.sudachi.SimpleOovProviderPlugin"
    },
    {
      "class" : "com.worksap.nlp.sudachi.MeCabOovProviderPlugin"
    }
  ]
}
```

--------------------------------

### Build Sudachi User Dictionary from CSV (Bash)

Source: https://context7.com/worksapplications/sudachi/llms.txt

Provides a bash command to build a Sudachi user dictionary from a CSV source file. This allows for adding custom vocabulary or overriding system dictionary entries. The command specifies output, system dictionary, description, and the input CSV file.

```bash
# Build user dictionary from CSV source
java -Dfile.encoding=UTF-8 -cp sudachi.jar \
  com.worksap.nlp.sudachi.dictionary.UserDictionaryBuilder \
  -o user.dic \
  -s system_core.dic \
  -d "Custom dictionary v1.0" \
  user_entries.csv

```

--------------------------------

### Use Sudachi User Dictionary (Java)

Source: https://context7.com/worksapplications/sudachi/llms.txt

Illustrates how to use a custom user dictionary with Sudachi in Java. It shows two methods: configuring via a JSON string and programmatically adding user dictionaries. User dictionary entries are prioritized over the system dictionary during tokenization.

```java
// Use user dictionary in configuration
import com.worksap.nlp.sudachi.*;
import java.io.IOException;
import java.nio.file.Paths;

public class UserDictExample {
    public static void main(String[] args) throws IOException {
        // Via JSON configuration
        Config config = Config.fromJsonString(
            "{\"userDict\":[\"user1.dic\",\"user2.dic\"]}",
            PathAnchor.filesystem(Paths.get("."))
        ).withFallback(Config.defaultConfig());

        // Or programmatically
        Config programmaticConfig = Config.empty()
            .systemDictionary(Paths.get("system_core.dic"))
            .addUserDictionary(Paths.get("domain_terms.dic"))
            .addUserDictionary(Paths.get("company_names.dic"));

        try (Dictionary dict = new DictionaryFactory().create(config)) {
            Tokenizer tokenizer = dict.create();
            // User dictionary entries take priority over system dictionary
            MorphemeList result = tokenizer.tokenize("舞台芸術について");
        }
    }
}

```

--------------------------------

### Filter Morphemes by Part-of-Speech using PosMatcher in Java

Source: https://context7.com/worksapplications/sudachi/llms.txt

Demonstrates how to create and use PosMatcher to filter morphemes based on their part-of-speech tags. It covers creating matchers for specific POS, partial matching, combining matchers, and inverting them. Requires Sudachi library.

```java
import com.worksap.nlp.sudachi.*;
import com.worksap.nlp.sudachi.dictionary.POS;
import java.io.IOException;
import java.util.stream.Collectors;
import java.util.stream.StreamSupport;

public class PosMatcherExample {
    public static void main(String[] args) throws IOException {
        try (Dictionary dict = new DictionaryFactory().create(Config.defaultConfig())) {
            Tokenizer tokenizer = dict.create();

            // Create matcher for nouns (名詞)
            PosMatcher nounMatcher = dict.posMatcher(pos ->
                pos.get(0).equals("名詞"));

            // Create matcher for verbs (動詞)
            PosMatcher verbMatcher = dict.posMatcher(pos ->
                pos.get(0).equals("動詞"));

            // Create matcher for proper nouns (固有名詞)
            PosMatcher properNounMatcher = dict.posMatcher(pos ->
                pos.get(0).equals("名詞") && pos.get(1).equals("固有名詞"));

            // Use partial POS matching
            PosMatcher particleMatcher = dict.posMatcher(
                new PartialPOS("助詞"),           // Matches all particles
                new PartialPOS("助詞", "格助詞")  // Matches case particles specifically
            );

            MorphemeList morphemes = tokenizer.tokenize("東京都で映画を見る");

            // Filter nouns
            System.out.println("Nouns:");
            for (Morpheme m : morphemes) {
                if (nounMatcher.test(m)) {
                    System.out.println("  " + m.surface());  // 東京都, 映画
                }
            }

            // Filter verbs
            System.out.println("Verbs:");
            for (Morpheme m : morphemes) {
                if (verbMatcher.test(m)) {
                    System.out.println("  " + m.surface());  // 見る
                }
            }

            // Combine matchers with union
            PosMatcher nounOrVerb = nounMatcher.union(verbMatcher);
            System.out.println("Nouns or Verbs:");
            for (Morpheme m : morphemes) {
                if (nounOrVerb.test(m)) {
                    System.out.println("  " + m.surface());
                }
            }

            // Invert matcher (everything except nouns)
            PosMatcher notNoun = nounMatcher.invert();

            // Iterate all POS tags matched by this matcher
            System.out.println("All noun POS tags:");
            for (POS pos : nounMatcher) {
                System.out.println("  " + pos);
            }
        }
    }
}

```

--------------------------------

### Perform Standalone Text Normalization using TextNormalizer in Java

Source: https://context7.com/worksapplications/sudachi/llms.txt

Illustrates how to use TextNormalizer for text normalization tasks without requiring a full morphological analysis. It covers default normalization, normalization using dictionary rules, and various character transformations like case folding and expansion. Requires Sudachi library.

```java
import com.worksap.nlp.sudachi.*;
import java.io.IOException;

public class TextNormalizerExample {
    public static void main(String[] args) throws IOException {
        // Create default text normalizer
        TextNormalizer normalizer = TextNormalizer.defaultTextNormalizer();

        // Normalize various text patterns
        System.out.println(normalizer.normalize("ＡＢＣＤ"));      // abcd (full-width to half-width, lowercase)
        System.out.println(normalizer.normalize("㍻"));            // 平成 (composed character expansion)
        System.out.println(normalizer.normalize("ｶﾀｶﾅ"));         // カタカナ (half-width to full-width katakana)
        System.out.println(normalizer.normalize("①②③"));         // 1 2 3 (circled numbers)

        // Create normalizer from existing dictionary (uses same plugins)
        try (Dictionary dict = new DictionaryFactory().create(Config.defaultConfig())) {
            JapaneseDictionary jdict = (JapaneseDictionary) dict;
            TextNormalizer dictNormalizer = TextNormalizer.fromDictionary(jdict);

            // Normalizes using the same rules as morphological analysis
            String normalized = dictNormalizer.normalize("ラーメン〜〜〜");
            System.out.println(normalized);  // ラーメンーーー (prolonged sound marks normalized)
        }
    }
}

```

--------------------------------

### Work with Morpheme Lists using Sudachi Java

Source: https://context7.com/worksapplications/sudachi/llms.txt

Illustrates how to use the MorphemeList interface in Java to manage collections of morphemes resulting from tokenization. It covers obtaining lists in different split modes, converting between modes, accessing individual morphemes, and iterating through the list. It also shows how to retrieve the internal path cost for advanced analysis.

```java
import com.worksap.nlp.sudachi.*;
import java.io.IOException;

public class MorphemeListExample {
    public static void main(String[] args) throws IOException {
        try (Dictionary dict = new DictionaryFactory().create(Config.defaultConfig())) {
            Tokenizer tokenizer = dict.create();

            // Get morpheme list in Mode C
            MorphemeList listC = tokenizer.tokenize(Tokenizer.SplitMode.C, "消費者安全調査委員会");
            System.out.println("Mode C count: " + listC.size());  // 1 morpheme

            // Convert entire list to finer split mode
            MorphemeList listA = listC.split(Tokenizer.SplitMode.A);
            System.out.println("Mode A count: " + listA.size());  // 6 morphemes: 消費/者/安全/調査/委員/会

            MorphemeList listB = listC.split(Tokenizer.SplitMode.B);
            System.out.println("Mode B count: " + listB.size());  // 4 morphemes: 消費者/安全/調査/委員会

            // Access individual morphemes by index
            Morpheme first = listC.get(0);
            System.out.println("First morpheme: " + first.surface());

            // Iterate using enhanced for loop
            for (Morpheme m : listA) {
                System.out.print(m.surface() + " ");
            }
            // Output: 消費 者 安全 調査 委員 会

            // Get internal path cost (for advanced analysis)
            int cost = listC.getInternalCost();
            System.out.println("Path cost: " + cost);
        }
    }
}

```

--------------------------------

### Access Morpheme Properties with Sudachi Java

Source: https://context7.com/worksapplications/sudachi/llms.txt

Demonstrates how to use the Morpheme interface in Java to access detailed linguistic properties of a tokenized word. This includes surface form, part-of-speech, dictionary form, reading, and normalization. It also shows how to split a morpheme into finer sub-tokens.

```java
import com.worksap.nlp.sudachi.*;
import java.io.IOException;
import java.util.List;

public class MorphemeExample {
    public static void main(String[] args) throws IOException {
        try (Dictionary dict = new DictionaryFactory().create(Config.defaultConfig())) {
            Tokenizer tokenizer = dict.create();
            MorphemeList morphemes = tokenizer.tokenize("シュミレーションを打込む");

            for (Morpheme m : morphemes) {
                // Basic properties
                System.out.println("Surface: " + m.surface());           // 打込む
                System.out.println("Begin: " + m.begin());               // Start character index
                System.out.println("End: " + m.end());                   // End character index

                // Part-of-speech information
                List<String> pos = m.partOfSpeech();                     // [動詞, 一般, *, *, 五段-マ行, 連体形-一般]
                System.out.println("POS: " + String.join(",", pos));
                System.out.println("POS ID: " + m.partOfSpeechId());     // Numeric POS identifier

                // Dictionary and normalized forms
                System.out.println("Dictionary form: " + m.dictionaryForm());  // 打ち込む (lemma)
                System.out.println("Normalized form: " + m.normalizedForm());  // 打ち込む (spelling normalization)
                System.out.println("Reading: " + m.readingForm());             // ウチコム (katakana reading)

                // Dictionary information
                System.out.println("Dictionary ID: " + m.getDictionaryId());   // 0=system, >0=user, <0=OOV
                System.out.println("Word ID: " + m.getWordId());
                System.out.println("Is OOV: " + m.isOOV());                    // Out-of-vocabulary flag

                // Synonym groups
                int[] synonyms = m.getSynonymGroupIds();
                System.out.println("Synonym groups: " + java.util.Arrays.toString(synonyms));

                // Split morpheme into finer units
                List<Morpheme> splitA = m.split(Tokenizer.SplitMode.A);
                for (Morpheme sub : splitA) {
                    System.out.println("  Sub-token: " + sub.surface());
                }
                System.out.println("---");
            }
        }
    }
}

```

--------------------------------

### MeCab-compatible OOV Handler Configuration (JSON5)

Source: https://github.com/worksapplications/sudachi/blob/develop/docs/oov_handlers.md

Configuration for the MeCab-compatible OOV handler in Sudachi. This handler supports MeCab's unknown word handling. The `charDef` and `unkDef` parameters are optional and default to 'char.def' and 'unk.def' respectively.

```json5
{
 "class"   : "com.worksap.nlp.sudachi.MeCabOovProviderPlugin",
  "charDef" : "char.def",
  "unkDef"  : "unk.def"
}
```

--------------------------------

### Simple OOV Handler Configuration (JSON5)

Source: https://github.com/worksapplications/sudachi/blob/develop/docs/oov_handlers.md

Configuration for the Simple OOV handler in Sudachi. This handler creates nodes based on character class boundaries. All fields, including class, oovPOS, leftId, rightId, and cost, are required for proper functioning.

```json5
{
  "class" : "com.worksap.nlp.sudachi.SimpleOovProviderPlugin",
  "oovPOS" : [ "補助記号", "一般", "*", "*", "*", "*" ],
  "leftId" : 5968,
  "rightId" : 5968,
  "cost" : 3857
}
```

--------------------------------

### Match Emails with RegexOovProvider in Sudachi

Source: https://github.com/worksapplications/sudachi/blob/develop/docs/oov_handlers.md

This configuration uses Sudachi's RegexOovProvider to identify and tokenize email addresses based on a provided regular expression. It sets specific left and right IDs, cost, POS tags, and enables user POS tagging.

```json5
{
    "class": "com.worksap.nlp.sudachi.RegexOovProvider",    
    "leftId": 5968,
    "rightId": 5968,
    "cost": 500,
    "regex": "^[\\w!#$%&’*+/=?`{|}~^-]+(?:\\.[\\w!#$%&’*+/=?`{|}~^-]+)*@(?:[0-9a-z\\-]+\\.)+[a-z]{2,6}",
    "pos": [ "補助記号", "一般", "e-mail", "*", "*", "*" ],
    "userPOS": "allow"
}
```

--------------------------------

### Tokenize Text and Extract Content Words in Java

Source: https://context7.com/worksapplications/sudachi/llms.txt

This Java code demonstrates how to use the Sudachi library to tokenize Japanese text and extract specific content words (nouns, verbs, adjectives). It initializes Sudachi with default settings, creates a tokenizer, defines a part-of-speech matcher, and processes a sample sentence.

```java
// Complete example: tokenize text and extract content words
import com.worksap.nlp.sudachi.*;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

public class ContentWordExtractor {
    public static void main(String[] args) throws IOException {
        try (Dictionary dict = new DictionaryFactory().create(Config.defaultConfig())) {
            Tokenizer tokenizer = dict.create();

            // Create matcher for content words (nouns and verbs)
            PosMatcher contentMatcher = dict.posMatcher(pos -> {
                String category = pos.get(0);
                return category.equals("名詞") || category.equals("動詞") || category.equals("形容詞");
            });

            String text = "人工知能の研究が急速に発展している";
            MorphemeList morphemes = tokenizer.tokenize(text);

            List<String> contentWords = new ArrayList<>();
            List<String> normalizedForms = new ArrayList<>();

            for (Morpheme m : morphemes) {
                if (contentMatcher.test(m)) {
                    contentWords.add(m.surface());
                    normalizedForms.add(m.normalizedForm());
                }
            }

            System.out.println("Content words: " + contentWords);
            // Output: [人工知能, 研究, 急速, 発展]

            System.out.println("Normalized: " + normalizedForms);
            // Output: [人工知能, 研究, 急速, 発展]
        }
    }
}
```

--------------------------------

### Sudachi Maven Dependency Configuration

Source: https://github.com/worksapplications/sudachi/blob/develop/README.md

This XML snippet shows how to declare Sudachi as a dependency in a Maven project. By including this in the pom.xml file, developers can easily integrate the Sudachi API into their Java applications.

```xml
<dependency>
  <groupId>com.worksap.nlp</groupId>
  <artifactId>sudachi</artifactId>
  <version>0.5.3</version>
</dependency>
```

--------------------------------

### Regex OOV Handler Configuration (JSON5)

Source: https://github.com/worksapplications/sudachi/blob/develop/docs/oov_handlers.md

Configuration for the Regex OOV handler in Sudachi. This handler generates OOV nodes based on a provided regular expression. Required fields include regex, leftId, rightId, cost, and oovPOS. Optional fields like userPOS, maxLength, and boundaries can modify its behavior.

```json5
{
  "class": "com.worksap.nlp.sudachi.RegexOovProvider",
  "regex": "[0-9a-z-]+",
  "leftId": 500,
  "rightId": 500,
  "cost": 5000,
  "oovPOS": [ "補助記号", "一般", "*", "*", "*", "*" ],
  "userPOS": "allow",
  "maxLength": 32,
  "boundaries": "relaxed"
 }
```
Tokens:
Raw