### Installing Fugashi and Dictionaries Source: https://context7.com/polm/fugashi/llms.txt Instructions for installing fugashi with different Unidic dictionary variants via pip and verifying the installation. ```bash pip install 'fugashi[unidic-lite]' pip install 'fugashi[unidic]' python -m unidic download python -c "from fugashi import Tagger; t = Tagger(); print(t.parse('テスト'))" ``` -------------------------------- ### Installing Fugashi with UniDic Dictionaries Source: https://github.com/polm/fugashi/blob/main/README.md Shows how to install the fugashi library along with UniDic dictionaries using pip. It covers installing a lightweight version ('unidic-lite') and the full version ('unidic'), including the necessary download step for the full version. ```sh pip install 'fugashi[unidic-lite]' # The full version of UniDic requires a separate download step pip install 'fugashi[unidic]' python -m unidic download ``` -------------------------------- ### Fugashi Command Line Interface Usage Source: https://context7.com/polm/fugashi/llms.txt Provides examples for using the fugashi CLI for basic tokenization, wakati output, dictionary information retrieval, and user dictionary building. ```bash echo "日本語のテスト" | fugashi echo "日本語のテスト" | fugashi -Owakati fugashi-info fugashi-build-dict -d /path/to/system/dic -u user.dic user.csv ``` -------------------------------- ### Basic Tokenization with Fugashi Tagger Source: https://github.com/polm/fugashi/blob/main/README.md Demonstrates the basic usage of the Tagger class from the fugashi library for Japanese text tokenization. It shows how to initialize the tagger with a specific output format ('-Owakati') and how to parse text to get space-separated words or iterate through words with their features. ```python from fugashi import Tagger tagger = Tagger('-Owakati') text = "麩菓子は、麩を主材料とした日本の菓子。" tagger.parse(text) # => '麩 菓子 は 、 麩 を 主材 料 と し た 日本 の 菓子 。' for word in tagger(text): print(word, word.feature.lemma, word.pos, sep='\t') # "feature" is the Unidic feature data as a named tuple ``` -------------------------------- ### Get Dictionary Metadata (Python) Source: https://context7.com/polm/fugashi/llms.txt The dictionary_info property provides metadata about the loaded dictionaries, including filename, character encoding, size, and version. This information is useful for understanding the dictionary configuration and debugging. ```python from fugashi import Tagger, GenericTagger tagger = Tagger() # Get dictionary information for info in tagger.dictionary_info: print(f"Filename: {info['filename']}") print(f"Charset: {info['charset']}") print(f"Size: {info['size']} entries") print(f"Version: {info['version']}") print("---") # Output example: # Filename: /path/to/unidic/dicdir # Charset: UTF-8 # Size: 756463 entries # Version: 102 # --- ``` -------------------------------- ### Tagger - Japanese Tokenization Source: https://context7.com/polm/fugashi/llms.txt The Tagger class is the primary interface for tokenizing Japanese text. It automatically detects installed Unidic dictionaries and provides access to morphological features. ```APIDOC ## Tagger(options) ### Description Initializes the MeCab tagger. Automatically detects Unidic versions or accepts custom dictionary paths via options. ### Method Constructor ### Parameters #### Query Parameters - **options** (string) - Optional - MeCab command-line options (e.g., '-Owakati') ### Request Example ```python tagger = Tagger('-Owakati') ``` ### Response #### Success Response (200) - **tagger** (object) - An instance of the Tagger class ready for parsing. ``` -------------------------------- ### Initialize Tagger and Access Token Properties Source: https://context7.com/polm/fugashi/llms.txt Demonstrates how to initialize the Fugashi Tagger and iterate through tokens to access surface forms, POS tags, lemmas, and readings. ```python from fugashi import Tagger tagger = Tagger() text = "麩菓子は、麩を主材料とした日本の菓子。" tokens = tagger(text) for token in tokens: print(f"Surface: {token.surface}") print(f"POS: {token.pos}") print(f"Lemma: {token.feature.lemma}") print(f"Reading: {token.feature.pron}") print(f"Is Unknown: {token.is_unk}") print("---") ``` -------------------------------- ### N-Best Tokenization as Node Lists (Python) Source: https://context7.com/polm/fugashi/llms.txt The nbestToNodeList() method returns multiple tokenization hypotheses, each as a list of Node objects. This provides full morphological information for each alternative segmentation, allowing detailed analysis of each hypothesis. ```python from fugashi import Tagger tagger = Tagger() text = "外国人参政権" # Get top 3 tokenizations as node lists hypotheses = tagger.nbestToNodeList(text, 3) for i, nodes in enumerate(hypotheses): print(f"Hypothesis {i + 1}:") surfaces = [node.surface for node in nodes] print(f" Tokens: {' | '.join(surfaces)}") # Access full features for each hypothesis for node in nodes: print(f" {node.surface}: {node.pos}") print() # Output: Hypothesis 1: Tokens: 外国 | 人参 | 政権 外国: 名詞,普通名詞,一般,* 人参: 名詞,普通名詞,一般,* 政権: 名詞,普通名詞,一般,* Hypothesis 2: Tokens: 外国 | 人 | 参政 | 権 外国: 名詞,普通名詞,一般,* 人: 名詞,普通名詞,一般,* 参政: 名詞,普通名詞,一般,* 権: 名詞,普通名詞,一般,* ``` -------------------------------- ### Parse Text to String Output Source: https://context7.com/polm/fugashi/llms.txt Shows how to use the parse() method to retrieve tokenized text as a string, including space-separated output using the -Owakati option. ```python from fugashi import Tagger tagger = Tagger('-Owakati') text = "深海魚は、深海に生息する魚類の総称。" result = tagger.parse(text) print(result) tagger_full = Tagger() full_result = tagger_full.parse("日本語です") print(full_result) ``` -------------------------------- ### N-Best Tokenization as String (Python) Source: https://context7.com/polm/fugashi/llms.txt The nbest() method returns multiple possible tokenizations of the input text as a newline-separated string. This is useful for handling ambiguous segmentations and retrieving the top N hypotheses. ```python from fugashi import Tagger # Use wakati mode for readable output tagger = Tagger('-Owakati') # Ambiguous text: "foreign + ginseng + political rights" vs "foreigner + suffrage" text = "外国人参政権" # Get top 3 tokenizations result = tagger.nbest(text, 3) print(result) # Output: # 外国 人参 政権 # 外国 人 参政 権 # 外国 人 参 政権 # Another ambiguous example text2 = "東京都の大人気ない主材料" result2 = tagger.nbest(text2, 2) print(result2) # Output: # 東京 都 の 大人気 ない 主材 料 # 東京 都 の 大人気 ない 主 材料 ``` -------------------------------- ### Using GenericTagger with Custom Dictionaries Source: https://github.com/polm/fugashi/blob/main/README.md Illustrates how to use the GenericTagger for dictionaries other than UniDic. It shows basic parsing and accessing feature data by field numbers. It also demonstrates creating a custom feature wrapper for named tuple access to features. ```python from fugashi import GenericTagger, create_feature_wrapper tagger = GenericTagger() # parse can be used as normal tagger.parse('something') # features from the dictionary can be accessed by field numbers for word in tagger(text): print(word.surface, word.feature[0]) CustomFeatures = create_feature_wrapper('CustomFeatures', 'alpha beta gamma') tagger = GenericTagger(wrapper=CustomFeatures) for word in tagger.parseToNodeList(text): print(word.surface, word.feature.alpha) ``` -------------------------------- ### parse() - Tokenized String Output Source: https://context7.com/polm/fugashi/llms.txt The parse() method returns the tokenized text as a string, useful for simple tokenization or formatted output based on MeCab modes. ```APIDOC ## Tagger.parse(text) ### Description Parses the input string and returns the result as a formatted string based on the tagger's configuration. ### Method GET/POST (Internal) ### Parameters #### Request Body - **text** (string) - Required - The Japanese text to tokenize. ### Request Example ```python result = tagger.parse("日本語です") ``` ### Response #### Success Response (200) - **result** (string) - The tokenized output string. ``` -------------------------------- ### Access Detailed Node Properties Source: https://context7.com/polm/fugashi/llms.txt Explains how to access advanced node metadata such as byte length, whitespace preservation, character type, and status codes. ```python from fugashi import Tagger tagger = Tagger() text = "これは テストです" nodes = tagger(text) for node in nodes: print(f"Surface: '{node.surface}'") print(f" Length: {node.length}") print(f" White Space: '{node.white_space}'") print(f" Char Type: {node.char_type}") print(f" Stat: {node.stat}") ``` -------------------------------- ### Retrieving Dictionary Metadata Source: https://context7.com/polm/fugashi/llms.txt Shows how to programmatically access the dictionary information currently in use by the Fugashi Tagger instance. ```python from fugashi import Tagger tagger = Tagger() info = tagger.dictionary_info[0] print(f"Using dictionary: {info['filename']}") print(f"Dictionary size: {info['size']} entries") ``` -------------------------------- ### Accessing Morphological Features in Fugashi Source: https://context7.com/polm/fugashi/llms.txt Demonstrates how to access common morphological features from a token object. It also shows how to safely check for version-specific fields like accent types using Python's hasattr. ```python print(f"POS1: {feat.pos1}") print(f"POS2: {feat.pos2}") print(f"Lemma: {feat.lemma}") print(f"Orth: {feat.orth}") print(f"Pron: {feat.pron}") if hasattr(feat, 'aType'): print(f"Accent Type: {feat.aType}") ``` -------------------------------- ### Parse Text to Node List Source: https://context7.com/polm/fugashi/llms.txt Demonstrates using parseToNodeList() to obtain a list of Node objects, allowing for granular access to morphological features. ```python from fugashi import Tagger tagger = Tagger() text = "あなたは新米の魔女。" nodes = tagger.parseToNodeList(text) for node in nodes: surface = node.surface feat = node.feature print(f"{surface}\t{feat.pos1}-{feat.pos2}\t{feat.lemma}") ``` -------------------------------- ### Generic Tagger with Custom Dictionaries (Python) Source: https://context7.com/polm/fugashi/llms.txt The GenericTagger class supports non-Unidic dictionaries like IPAdic or custom dictionaries. Features can be accessed by index or through a custom wrapper, providing flexibility for different dictionary schemas. ```python from fugashi import GenericTagger # Using with ipadic (if installed) import ipadic tagger = GenericTagger(ipadic.MECAB_ARGS) text = "日本語ですよ" result = tagger.parse(text) print(result) # Access features by index for node in tagger(text): surface = node.surface # IPAdic features: pos1, pos2, pos3, pos4, conjugation_type, conjugation_form, base_form, reading, pronunciation pos1 = node.feature[0] # Major POS category pos2 = node.feature[1] # Sub-category base = node.feature[6] # Base/dictionary form reading = node.feature[7] if len(node.feature) > 7 else None print(f"{surface}\t{pos1}/{pos2}\t{base}\t{reading}") # Using wakati mode with custom dictionary tagger_wakati = GenericTagger(ipadic.MECAB_ARGS + ' -Owakati') print(tagger_wakati.parse("すももももももももの内")) # Output: すもも も もも も もも の 内 ``` -------------------------------- ### Custom Feature Wrapper with Named Tuple (Python) Source: https://context7.com/polm/fugashi/llms.txt The create_feature_wrapper() function generates a named tuple wrapper for dictionary features, enabling attribute-based access with custom dictionaries. This improves code readability and maintainability when working with non-standard dictionary schemas. ```python from fugashi import GenericTagger, create_feature_wrapper # Define custom feature wrapper matching your dictionary schema # Example for IPAdic-style dictionary with 9 fields IpadicFeatures = create_feature_wrapper( 'IpadicFeatures', 'pos1 pos2 pos3 pos4 cType cForm baseForm reading pronunciation' ) # Use with GenericTagger import ipadic tagger = GenericTagger(ipadic.MECAB_ARGS, wrapper=IpadicFeatures) text = "東京に行きます" for node in tagger(text): print(f"Surface: {node.surface}") print(f" POS: {node.feature.pos1}/{node.feature.pos2}") print(f" Base Form: {node.feature.baseForm}") print(f" Reading: {node.feature.reading}") # Output: # Surface: 東京 # POS: 名詞/固有名詞 # Base Form: 東京 # Reading: トウキョウ # Surface: に # POS: 助詞/格助詞 # Base Form: に # Reading: ニ # ... ``` -------------------------------- ### Unidic Feature Schemas (Python) Source: https://context7.com/polm/fugashi/llms.txt Fugashi provides pre-defined named tuples for different Unidic versions (e.g., UnidicFeatures17, UnidicFeatures26, UnidicFeatures29). The feature schema is automatically detected and applied when using the standard Tagger, simplifying access to morphological information. ```python from fugashi import Tagger, UnidicFeatures17, UnidicFeatures26, UnidicFeatures29 tagger = Tagger() tokens = tagger("日本語") # The feature type depends on your installed Unidic version token = tokens[0] feat = token.feature # UnidicFeatures17 (unidic 2.1.2 src): 17 fields ``` -------------------------------- ### parseToNodeList() - Morphological Analysis Source: https://context7.com/polm/fugashi/llms.txt Returns a list of Node objects, providing granular access to morphological features like POS, lemma, and reading. ```APIDOC ## Tagger.parseToNodeList(text) ### Description Tokenizes text and returns a list of Node objects containing full morphological data. ### Method GET/POST (Internal) ### Parameters #### Request Body - **text** (string) - Required - The Japanese text to analyze. ### Request Example ```python nodes = tagger.parseToNodeList("あなたは新米の魔女。") ``` ### Response #### Success Response (200) - **nodes** (list) - A list of UnidicNode objects containing surface, feature, and metadata. ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.