### Install PyKoSpacing via PyPI Source: https://github.com/haven-jeon/pykospacing/blob/master/README.md Standard installation using pip for Python 3. Ensure TensorFlow and Keras are installed first. ```bash pip install tensorflow pip install keras ``` -------------------------------- ### Install TensorFlow on M1 Mac with Miniforge3 Source: https://github.com/haven-jeon/pykospacing/blob/master/README.md Instructions for installing TensorFlow on M1 Macs using Miniforge3. This involves installing Miniforge, activating its environment, and then installing specific TensorFlow packages for macOS. ```bash # Install Miniforge3 for mac curl -O https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh chmod +x Miniforge3-MacOSX-arm64.sh sh Miniforge3-MacOSX-arm64.sh # Activate Miniforge3 virtualenv # You should use Python version 3.10 or less. source ~/miniforge3/bin/activate # Install the Tensorflow dependencies conda install -c apple tensorflow-deps # Install base tensorflow python -m pip install tensorflow-macos # Install metal plugin python -m pip install tensorflow-metal ``` -------------------------------- ### Install PyKoSpacing from GitHub Source: https://github.com/haven-jeon/pykospacing/blob/master/README.md Install the package directly from the GitHub repository using pip. This is useful for obtaining the latest development version. ```bash pip install git+https://github.com/haven-jeon/PyKoSpacing.git ``` -------------------------------- ### Initialize and Use Spacing Model Source: https://context7.com/haven-jeon/pykospacing/llms.txt Initialize the Spacing model and use it for basic Korean word spacing. Handles ambiguous text by predicting correct spaces. ```python from pykospacing import Spacing # Initialize the spacing model spacing = Spacing() # Basic usage - add spaces to unspaced Korean text text = "김형호영화시장분석가는'1987'의네이버영화정보네티즌10점평에서언급된단어들을지난해12월27일부터올해1월10일까지통계프로그램R과KoNLP패키지로텍스트마이닝하여분석했다." result = spacing(text) print(result) # Output: "김형호 영화시장 분석가는 '1987'의 네이버 영화 정보 네티즌 10점 평에서 언급된 단어들을 지난해 12월 27일부터 올해 1월 10일까지 통계 프로그램 R과 KoNLP 패키지로 텍스트마이닝하여 분석했다." # Simple example demonstrating disambiguation ambiguous = "아버지가방에들어가신다." print(spacing(ambiguous)) # Output: "아버지가 방에 들어가신다." (My father enters the room) ``` -------------------------------- ### Use Command Line Interface for Batch Processing Source: https://context7.com/haven-jeon/pykospacing/llms.txt Process text files or standard input directly from the terminal using the pykospacing module. ```bash # Basic usage - read from file, output to stdout python -m pykospacing.pykos input.txt # Output to a specific file python -m pykospacing.pykos input.txt output.txt # Overwrite the input file with spaced result python -m pykospacing.pykos -o input.txt # Read from stdin echo "아버지가방에들어가신다." | python -m pykospacing.pykos /dev/stdin ``` -------------------------------- ### Setting Spacing Rules from CSV Source: https://github.com/haven-jeon/pykospacing/blob/master/README.md Configure custom spacing rules by loading them from a CSV file using the `set_rules_by_csv` method. Specify the column containing the words to be used as rules. ```bash $ cat test.csv 인덱스,단어 1,네이버영화 2,언급된단어 ``` ```python >>> from pykospacing import Spacing >>> spacing = Spacing(rules=['']) >>> spacing.set_rules_by_csv('./test.csv', '단어') >>> spacing("김형호영화시장분석가는'1987'의네이버영화정보네티즌10점평에서언급된단어들을지난해12월27일부터올해1월10일까지통계프로그램R과KoNLP패키지로텍스트마이닝하여분석했다.") "김형호 영화시장 분석가는 '1987'의 네이버영화 정보 네티즌 10점 평에서 언급된단어들을 지난해 12월 27일부터 올해 1월 10일까지 통계 프로그램 R과 KoNLP 패키지로 텍스트마이닝하여 분석했다." ``` -------------------------------- ### Troubleshoot libstdc++ error on Ubuntu Source: https://github.com/haven-jeon/pykospacing/blob/master/README.md If you encounter a GLIBCXX_3.4.22 not found error on Ubuntu, these commands can help resolve the issue by upgrading the C++ standard library. ```bash sudo apt-get install libstdc++6 sudo apt-get install ppa:ubuntu-toolchain-r/test sudo apt-get update sudo apt-get upgrade sudo apt-get dist-upgrade (This takes long time.) ``` -------------------------------- ### Handling English Characters with `ignore='none'` Source: https://github.com/haven-jeon/pykospacing/blob/master/README.md Demonstrates the default behavior of the `ignore` parameter, where no special pre- or post-processing is applied for English characters. Spacing may occur within English words. ```python >>> from pykospacing import Spacing >>> spacing = Spacing() >>> spacing("친구와함께bmw썬바이저를썼다.", ignore='none') "친구와 함께 bm w 썬바이저를 썼다." >>> spacing("chicken박스를열고닭다리를꺼내입에문다.crispy한튀김옷덕에내입주변은glossy해진다.", ignore='none') "chicken박스를 열고 닭다리를 꺼내 입에 문다. crispy 한튀김 옷 덕에 내 입 주변은 glossy해진다." >>> spacing("김형호영화시장분석가는'1987'의네이버영화정보네티즌10점평에서언급된단어들을지난해12월27일부터올해1월10일까지통계프로그램R과KoNLP패키지로텍스트마이닝하여분석했다.", ignore='none') "김형호 영화시장 분석가는 '1987'의 네이버 영화 정보 네티즌 10점 평에서 언급된 단어들을 지난해 12월 27일부터 올해 1월 10일까지 통계 프로그램 R과 KoNLP 패키지로 텍스트마이닝하여 분석했다." ``` -------------------------------- ### Command Line Spacing Source: https://github.com/haven-jeon/pykospacing/blob/master/README.md Utilize PyKoSpacing directly from the command line to process text files. This is convenient for batch processing. ```bash $ cat test_in.txt 김형호영화시장분석가는'1987'의네이버영화정보네티즌10점평에서언급된단어들을지난해12월27일부터올해1월10일까지통계프로그램R과KoNLP패키지로텍스트마이닝하여분석했다. 아버지가방에들어가신다. $ python -m pykospacing.pykos test_in.txt 김형호 영화시장 분석가는 '1987'의 네이버 영화 정보 네티즌 10점 평에서 언급된 단어들을 지난해 12월 27일부터 올해 1월 10일까지 통계 프로그램 R과 KoNLP 패키지로 텍스트마이닝하여 분석했다. 아버지가 방에 들어가신다. ``` -------------------------------- ### Load Rules from CSV File Source: https://context7.com/haven-jeon/pykospacing/llms.txt Load custom compound word rules from a CSV file using `set_rules_by_csv()`. Specify the column containing the words to preserve. ```python from pykospacing import Spacing # CSV file format (test.csv): # 인덱스,단어 # 1,네이버영화 # 2,언급된단어 # Initialize and load rules from CSV spacing = Spacing(rules=['']) # Initialize with empty rules spacing.set_rules_by_csv('./test.csv', '단어') # Load rules from '단어' column text = "김형호영화시장분석가는'1987'의네이버영화정보네티즌10점평에서언급된단어들을지난해12월27일부터올해1월10일까지통계프로그램R과KoNLP패키지로텍스트마이닝하여분석했다." result = spacing(text) print(result) # Output: "김형호 영화시장 분석가는 '1987'의 네이버영화 정보 네티즌 10점 평에서 언급된단어들을 지난해 12월 27일부터 올해 1월 10일까지 통계 프로그램 R과 KoNLP 패키지로 텍스트마이닝하여 분석했다." # Without specifying key - reads all columns spacing2 = Spacing(rules=['']) spacing2.set_rules_by_csv('./words.csv') # Uses all words in CSV ``` -------------------------------- ### Basic Korean Text Spacing Source: https://github.com/haven-jeon/pykospacing/blob/master/README.md Instantiate the Spacing class and use it to correct spacing in Korean text. This is the default behavior without custom rules. ```python >>> from pykospacing import Spacing >>> spacing = Spacing() >>> spacing("김형호영화시장분석가는'1987'의네이버영화정보네티즌10점평에서언급된단어들을지난해12월27일부터올해1월10일까지통계프로그램R과KoNLP패키지로텍스트마이닝하여분석했다.") "김형호 영화시장 분석가는 '1987'의 네이버 영화 정보 네티즌 10점 평에서 언급된 단어들을 지난해 12월 27일부터 올해 1월 10일까지 통계 프로그램 R과 KoNLP 패키지로 텍스트마이닝하여 분석했다." ``` -------------------------------- ### Handling English Characters with `ignore='pre2'` Source: https://github.com/haven-jeon/pykospacing/blob/master/README.md Uses pre-processing to remove characters matching `ignore_pattern` and then predicts spacing on both the pre-processed and original text. This allows for more accurate spacing around ignored characters but doubles computation time. ```python >>> spacing("친구와함께bmw썬바이저를썼다.", ignore='pre2') "친구와 함께 bmw 썬바이저를 썼다." >>> spacing("chicken박스를열고닭다리를꺼내입에문다.crispy한튀김옷덕에내입주변은glossy해진다.", ignore='pre2') "chicken박스를 열고 닭다리를 꺼내 입에 문다. crispy 한 튀김옷 덕에 내 입 주변은 glossy해진다." >>> spacing("김형호영화시장분석가는'1987'의네이버영화정보네티즌10점평에서언급된단어들을지난해12월27일부터올해1월10일까지통계프로그램R과KoNLP패키지로텍스트마이닝하여분석했다.", ignore='pre2') ``` -------------------------------- ### Process Multi-line Text in Python Source: https://context7.com/haven-jeon/pykospacing/llms.txt Iterate through lines of text to apply spacing individually, which is effective for handling longer documents. ```python from pykospacing import Spacing spacing = Spacing() # Multi-line text processing korean_text = """아즈사쨩이살인자가되는건싫어요... 그런어둡고우울한이야기,전싫어요 그게진실이라고,이세계의본질이라고해도,전싫어요! 저는좋아하는게있어요! 평범하고,별다른개성도없는저이지만....자신이좋아하는것에한해선절대로양보못해요!""" # Process each line separately result_lines = [] for line in korean_text.split('\n'): spaced_line = spacing(line) result_lines.append(spaced_line) print(spaced_line) # Join results final_text = '\n'.join(result_lines) ``` -------------------------------- ### Applying Non-Spacing Rules Source: https://github.com/haven-jeon/pykospacing/blob/master/README.md Use the Spacing class with a list of words that should not be spaced. This is useful for specific terms or names. ```python >>> # Apply a list of words that must be non-spacing >>> spacing('귀밑에서턱까지잇따라난수염을구레나룻이라고한다.') '귀 밑에서 턱까지 잇따라 난 수염을 구레나 룻이라고 한다.' >>> spacing = Spacing(rules=['구레나룻']) >>> spacing('귀밑에서턱까지잇따라난수염을구레나룻이라고한다.') '귀 밑에서 턱까지 잇따라 난 수염을 구레나룻이라고 한다.' ``` -------------------------------- ### Handling English Characters with `ignore='pre'` Source: https://github.com/haven-jeon/pykospacing/blob/master/README.md Applies pre-processing to remove characters matching `ignore_pattern` before spacing. Deleted characters are merged back after prediction, potentially causing incorrect spacing after them. ```python >>> spacing("친구와함께bmw썬바이저를썼다.", ignore='pre') "친구와 함께bmw 썬바이저를 썼다." >>> spacing("chicken박스를열고닭다리를꺼내입에문다.crispy한튀김옷덕에내입주변은glossy해진다.", ignore='pre') "chicken박스를 열고 닭다리를 꺼내 입에 문다.crispy 한 튀김옷 덕에 내 입 주변은glossy 해진다." >>> spacing("김형호영화시장분석가는'1987'의네이버영화정보네티즌10점평에서언급된단어들을지난해12월27일부터올해1월10일까지통계프로그램R과KoNLP패키지로텍스트마이닝하여분석했다.", ignore='pre') "김형호 영화시장 분석가는 '1987'의 네이버 영화 정보 네티즌 10점 평에서 언급된 단어들을 지난해 12월 27일부터 올해 1월 10일까지 통계 프로그램R과KoNLP 패키지로 텍스트마이닝하여 분석했다." ``` -------------------------------- ### set_rules_by_csv Source: https://github.com/haven-jeon/pykospacing/blob/master/README.md Configures non-spacing rules from a CSV file. ```APIDOC ## POST /spacing/rules/csv ### Description Loads a list of words that should not be spaced from a CSV file. ### Method POST ### Endpoint /spacing/rules/csv ### Parameters #### Request Body - **file_path** (string) - Required - Path to the CSV file. - **column_name** (string) - Required - The column name containing the words to protect from spacing. ``` -------------------------------- ### Apply Custom Ignore Patterns in Python Source: https://context7.com/haven-jeon/pykospacing/llms.txt Use custom regex patterns to prevent specific characters or sequences from being affected by the spacing algorithm. ```python # Custom pattern - only ignore uppercase English letters text = "BMW자동차를타고SEOUL에서BUSAN까지달렸다." custom_pattern = r'[A-Z]+' print(spacing(text, ignore='pre2', ignore_pattern=custom_pattern)) # Output: "BMW 자동차를 타고 SEOUL에서 BUSAN까지 달렸다." # Pattern to ignore numbers only number_text = "2024년1월1일에시작한다." number_pattern = r'[0-9]+' print(spacing(number_text, ignore='pre2', ignore_pattern=number_pattern)) ``` -------------------------------- ### Handling English Characters with `ignore='post'` Source: https://github.com/haven-jeon/pykospacing/blob/master/README.md Applies post-processing to ignore model outputs on characters matching `ignore_pattern`. This can lead to English characters affecting spacing of nearby non-English characters. ```python >>> spacing("친구와함께bmw썬바이저를썼다.", ignore='post') "친구와 함께 bm w 썬바이저를 썼다." >>> spacing("chicken박스를열고닭다리를꺼내입에문다.crispy한튀김옷덕에내입주변은glossy해진다.", ignore='post') "chicken박스를 열고 닭다리를 꺼내 입에 문다. crispy 한튀김 옷 덕에 내 입 주변은 glossy해진다." >>> spacing("김형호영화시장분석가는'1987'의네이버영화정보네티즌10점평에서언급된단어들을지난해12월27일부터올해1월10일까지통계프로그램R과KoNLP패키지로텍스트마이닝하여분석했다.", ignore='post') "김형호 영화시장 분석가는 '1987'의 네이버 영화 정보 네티즌 10점 평에서 언급된 단어들을 지난해 12월 27일부터 올해 1월 10일까지 통계 프로그램 R과 KoNLP 패키지로 텍스트마이닝하여 분석했다." ``` -------------------------------- ### Handling English Characters with Ignore Parameter Source: https://context7.com/haven-jeon/pykospacing/llms.txt Control how non-Korean characters, especially English, are handled during spacing using the `ignore` parameter. Options include 'none', 'pre', 'post', and 'pre2'. ```python from pykospacing import Spacing spacing = Spacing() text = "친구와함께bmw썬바이저를썼다." # ignore='none' - No preprocessing, model output as-is (may split English words) print(spacing(text, ignore='none')) # Output: "친구와 함께 bm w 썬바이저를 썼다." # ignore='pre' - Remove non-Korean before prediction, merge after (space after English) print(spacing(text, ignore='pre')) # Output: "친구와 함께bmw 썬바이저를 썼다." # ignore='post' - Ignore model output on non-Korean characters print(spacing(text, ignore='post')) # Output: "친구와 함께 bm w 썬바이저를 썼다." # ignore='pre2' - Double prediction for accurate spacing around English (slower but best) print(spacing(text, ignore='pre2')) # Output: "친구와 함께 bmw 썬바이저를 썼다." # Complex example with multiple English words mixed_text = "chicken박스를열고닭다리를꺼내입에문다.crispy한튀김옷덕에내입주변은glossy해진다." print(spacing(mixed_text, ignore='pre2')) # Output: "chicken박스를 열고 닭다리를 꺼내 입에 문다. crispy 한 튀김옷 덕에 내 입 주변은 glossy해진다." ``` -------------------------------- ### Spacing Correction API Source: https://github.com/haven-jeon/pykospacing/blob/master/README.md The primary interface for correcting spacing in Korean text strings. ```APIDOC ## POST /spacing ### Description Corrects the spacing of a provided Korean text string using the Spacing model. ### Method POST ### Endpoint /spacing ### Parameters #### Request Body - **text** (string) - Required - The Korean text to be corrected. - **rules** (list) - Optional - A list of words that must not be separated by spaces. - **ignore** (string) - Optional - Strategy for handling non-Korean characters ('none', 'pre', 'post', 'pre2'). Default is 'none'. - **ignore_pattern** (string) - Optional - Regex pattern for characters to ignore. ### Response #### Success Response (200) - **result** (string) - The spacing-corrected text. ``` -------------------------------- ### Custom Ignore Pattern for Character Handling Source: https://context7.com/haven-jeon/pykospacing/llms.txt Use the `ignore_pattern` parameter with a regex to define custom character sets to be handled by the `ignore` modes. This offers fine-grained control over character processing. ```python from pykospacing import Spacing spacing = Spacing() # Default pattern matches non-Korean, non-ASCII symbol sequences ``` -------------------------------- ### Custom Rules for Compound Words Source: https://context7.com/haven-jeon/pykospacing/llms.txt Specify custom rules to preserve compound words that should not be split by the spacing model. Useful for proper nouns or technical terms. ```python from pykospacing import Spacing # Without rules - model incorrectly splits "구레나룻" (sideburns) spacing = Spacing() text = "귀밑에서턱까지잇따라난수염을구레나룻이라고한다." print(spacing(text)) # Output: "귀 밑에서 턱까지 잇따라 난 수염을 구레나 룻이라고 한다." # With rules - preserve "구레나룻" as one word spacing_with_rules = Spacing(rules=['구레나룻']) print(spacing_with_rules(text)) # Output: "귀 밑에서 턱까지 잇따라 난 수염을 구레나룻이라고 한다." # Multiple rules can be specified spacing_multi = Spacing(rules=['네이버영화', '텍스트마이닝', '구레나룻']) text2 = "네이버영화에서텍스트마이닝을한다." print(spacing_multi(text2)) # Output: "네이버영화에서 텍스트마이닝을 한다." ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.