### Install soynlp

Source: https://context7.com/lovit/soynlp/llms.txt

Install the soynlp library using pip.

```bash
pip install soynlp
```

--------------------------------

### Initialize Tokenizers with SoyNLP

Source: https://github.com/lovit/soynlp/blob/master/tutorials/tokenizer_usage.ipynb

Imports necessary tokenizer classes from the soynlp library. Ensure the library is installed and accessible in your Python path.

```python
from pprint import pprint
import sys
sys.path.append('../')

from soynlp.tokenizer import RegexTokenizer, LTokenizer, MaxScoreTokenizer
```

--------------------------------

### Generate Initial Candidates Example

Source: https://github.com/lovit/soynlp/blob/master/tutorials/tagger_lecture.ipynb

Demonstrates the usage of the _initialize_L function to generate word candidates from a sample Korean sentence with potential spacing errors.

```python
sent = '아이오아이는공연에서너무너무너무를불렀다'

candidates = _initialize_L(sent)
pprint(candidates)
```

--------------------------------

### Example Lemmatization Output

Source: https://github.com/lovit/soynlp/blob/master/test/lemmatizer_test.ipynb

Sample output from the `lemma_candidate` function, showing potential lemma candidates for various input root-ending pairs. Note the variety of possible derivations.

```text
(깨달, 아) -> {('깨달', '아'), ('깨닫', '아'), ('깨다', 'ㄹ아'), ('깨닿', 'ㄹ아'), ('깨드', '알아')}
(굴, 러) -> {('구', 'ㄹ러'), ('굴', '러'), ('궇', 'ㄹ러'), ('구르', '어')}
(더러, 워서) -> {('더러', '워서'), ('더렇', '워서'), ('더르', '어워서'), ('더럿', '워서'), ('더럽', '어서')}
(도, 왔다) -> {('돟', '왔다'), ('도', '왔다'), ('돗', '왔다'), ('돕', '았다')}
(부, 었다) -> {('붛', '었다'), ('붓', '었다'), ('부', '었다')}
(똥퍼, ) -> {('똥펗', ''), ('똥퍼', ''), ('똥푸', '어'), ('똥프', '어')}
(퍼, ) -> {('프', '어'), ('푸', '어'), ('펗', ''), ('퍼', '')}
(줬, 어) -> {('주', '었어'), ('줬', '어'), ('줘', 'ㅆ어'), ('줳', 'ㅆ어')}
(꺼, ) -> {('꺼', ''), ('끄', '어'), ('껗', '')}
(텄, 어) -> {('트', '었어'), ('텄', '어'), ('텋', 'ㅆ어'), ('터', 'ㅆ어')}
(가, 거라) -> {('갛', '거라'), ('가', '거라'), ('그', '아거라')}
(돌아오, 거라) -> {('돌아오', '거라'), ('돌아옿', '거라')}
(돌아왔, 다) -> {('돌아와', 'ㅆ다'), ('돌아오', '았다'), ('돌아왔', '다'), ('돌아왛', 'ㅆ다')}
(이르, 러) -> {('이르', '러'), ('이릏', '러')}
(파라, 면) -> {('파르', '아면'), ('파라', '면'), ('파랗', '면')}
(시퍼렜, 다) -> {('시퍼렇', '었다'), ('시퍼렣', 'ㅆ다'), ('시퍼레', 'ㅆ다'), ('시퍼렜', '다')}
(파랬, 다) -> {('파랳', 'ㅆ다'), ('파랗', '았다'), ('파래', 'ㅆ다'), ('파랬', '다')}
(파래, ) -> {('파랗', '아'), ('파래', ''), ('파랳', '')}
(간, ) -> {('갛', 'ㄴ'), ('가', 'ㄴ'), ('간', ''), ('그', '안')}
(푸른, ) -> {('푸르', 'ㄴ'), ('푸릏', 'ㄴ'), ('푸른', '')}
(한, ) -> {('한', ''), ('흐', '안'), ('핳', 'ㄴ'), ('하', 'ㄴ')}
(이른, ) -> {('이르', 'ㄴ'), ('이릏', 'ㄴ'), ('이른', '')}
(불, 어) -> {('붛', 'ㄹ어'), ('불', '어'), ('붇', '어'), ('부', 'ㄹ어')}
(부, 어) -> {('붓', '어'), ('부', '어'), ('붛', '어')}
(일, 러) -> {('일', '러'), ('잏', 'ㄹ러'), ('이', 'ㄹ러'), ('이르', '어')}
(이르, 니) -> {('이릏', '니'), ('이르', '니')}
(이른, ) -> {('이르', 'ㄴ'), ('이릏', 'ㄴ'), ('이른', '')}
(뵈, 고) -> {('뵣', '고'), ('뵈', '고')}
(뵙, 고) -> {('뵙', '고'), ('뵈', 'ㅂ고'), ('뵣', 'ㅂ고')}
(뵙, 는) -> {('뵣', 'ㅂ는'), ('뵈', 'ㅂ는'), ('뵙', '는')}
(그래, ) -> {('그렇', '아'), ('그래', ''), ('그랳', '')}
```

--------------------------------

### Get SoyNLP Version

Source: https://github.com/lovit/soynlp/blob/master/tutorials/tagger_usage.ipynb

Retrieves the currently installed version of the SoyNLP library. Ensure SoyNLP is installed in your environment.

```python
import sys
sys.path.append('../')
import soynlp

print(soynlp.__version__)
```

--------------------------------

### Iterate Through Eojeols and Their Counts

Source: https://context7.com/lovit/soynlp/llms.txt

Provides an example of iterating through all unique eojeols and their corresponding frequencies stored in the EojeolCounter.

```python
from soynlp.utils import EojeolCounter

sentences = [
    '아이오아이가 노래를 불렀다',
    '아이오아이의 무대가 멋있다',
    '트와이스가 신곡을 발표했다'
]

counter = EojeolCounter(sents=sentences)

for eojeol, count in counter.items():
    print(f'{eojeol}: {count}')
```

--------------------------------

### Initialize KoNLPy Taggers and Check Version

Source: https://github.com/lovit/soynlp/blob/master/tutorials/nounextractor-v1_usage.ipynb

Initializes Kkma, Okt, and Hannanum taggers from KoNLPy and prints the installed KoNLPy version. Ensure you have KoNLPy version 0.5.1 or compatible.

```python
# -*- encoding:utf8 -*-
import sys
if sys.version_info <= (2,7):
    reload(sys)
    sys.setdefaultencoding('utf-8')
import konlpy
from konlpy.tag import Kkma, Okt, Hannanum

kkma = Kkma()
okt = Okt()
hannanum = Hannanum()
print('konlpy version = %s' % konlpy.__version__)
```

--------------------------------

### POS Tagging Output for Specific Sentence

Source: https://github.com/lovit/soynlp/blob/master/tutorials/tagger_lecture.ipynb

Example of POS tagging for a sentence with different words.

```python
sent = '아이오아이는 공연에서 너무너무너무를 불렀다'
tag(sent)
```

--------------------------------

### Expand Right-Hand Candidates Example

Source: https://github.com/lovit/soynlp/blob/master/tutorials/tagger_lecture.ipynb

Shows how to use the _expand_R function to extend the initial word candidates with possible right-hand parts, demonstrating the creation of LR structures.

```python
candidates = _expand_R(sent, candidates)
pprint(candidates)
```

--------------------------------

### Example Korean Sentence and Tokenized Output

Source: https://github.com/lovit/soynlp/blob/master/tutorials/tagger_lecture.ipynb

Illustrates a sample Korean sentence and its desired tokenized output with part-of-speech tags. This serves as a target for POS tagging.

```text
문장: '너무너무너무는 아이오아이의 노래입니다'
단어열: [('너무너무너무', '명사'), ('는', '조사'), ('아이오아이', '명사'), ('의', '조사'), ('노래', '명사'), ('입니다', '조사')]
```

--------------------------------

### Import SoyNLP Lemmatizer

Source: https://github.com/lovit/soynlp/blob/master/test/lemmatizer_test.ipynb

Imports the necessary `lemma_candidate` function from the SoyNLP library. Ensure SoyNLP is installed and accessible in your Python path.

```python
import sys
sys.path.insert(0, '../')

import soynlp
from soynlp.lemmatizer import lemma_candidate
```

--------------------------------

### POS Tagging Candidate Scoring Example (Full Sentence)

Source: https://github.com/lovit/soynlp/blob/master/tutorials/tagger_lecture.ipynb

Demonstrates two potential tokenization candidates for a Korean sentence, highlighting the need for a scoring mechanism to select the most appropriate one.

```text
문장 : 너무너무너무는 아이오아이의 노래입니다
후보 1: [너무너무너무/명사, 는/조사, 아이오아이/명사, 의/조사, 노래/명사, 입니다/조사]
후보 2: [너무너무너무/명사, 는/조사, 아이/명사, 오/명사, 아이/명사, 의/조사, 노래/명사, 입니다/조사]
```

--------------------------------

### Initialize Word Candidates (_initialize_L)

Source: https://github.com/lovit/soynlp/blob/master/tutorials/tagger_lecture.ipynb

Generates initial word candidates from a given text 't' by checking substrings against a dictionary. It identifies words belonging to specific POS tags (noun, adjective, verb, adverb, interjection) and stores them with their tags and positions. Candidates are sorted by their starting position.

```python
ltags = {'명사', '형용사', '동사', '부사', '감탄사'}

def _pos_L(word):
    poses = dictionary.get_pos(word)
    poses = {pos for pos in poses if pos in ltags}
    return poses

def _initialize_L(t):
    n = len(t)
    candidates = []

    for b in range(n):
        for e in range(b+1, min(n, b + dictionary.max_length) + 1):
            l = t[b:e]
            l_tags = _pos_L(l)

            if not l_tags:
                continue

            for l_tag in l_tags:
                candidates.append([l, l_tag, b, e])

    return sorted(candidates, key=lambda x:x[2])
```

--------------------------------

### Example Korean Word Dictionary for POS Tagging

Source: https://github.com/lovit/soynlp/blob/master/tutorials/tagger_lecture.ipynb

A sample dictionary mapping Korean words to their parts of speech (POS). This dictionary can be used in the candidate generation phase of a POS tagger.

```python
단어 사전 = {
               '부사': {'너무', '매우'},
               '명사': {'너무너무너무', '아이오아이', '아이', '노래', '오'},
               '조사': {'는', '의', '이다', '입니다'}
              }
```

--------------------------------

### POS Tagging Candidate Scoring Example (Eojeol Unit)

Source: https://github.com/lovit/soynlp/blob/master/tutorials/tagger_lecture.ipynb

Shows two tokenization candidates for a specific Korean 'eojeol' (word segment), illustrating how scoring can be applied at a smaller unit for POS tagging.

```text
어절: 아이오아이의 
후보 1: [아이오아이/명사, 의/조사]
후보 2: [아이/명사, 오/명사, 아이/명사, 의/조사]
```

--------------------------------

### Get Multiple POS Tags for a Word

Source: https://github.com/lovit/soynlp/blob/master/tutorials/tagger_lecture.ipynb

Demonstrates retrieving multiple POS tags for a word that is present in the dictionary under different categories. For example, '이' can be both a noun and a particle.

```python
dictionary.get_pos('이')
```

--------------------------------

### Instantiate and Use SimpleTagger (Flattened)

Source: https://github.com/lovit/soynlp/blob/master/tutorials/tagger_lecture.ipynb

Demonstrates how to create an instance of SimpleTagger and use it to tag a sentence, returning a flattened list of (word, tag) tuples.

```python
tagger = SimpleTagger(generator, evaluator)
tagger.tag(sent)
```

--------------------------------

### Count Sentences in Corpus

Source: https://github.com/lovit/soynlp/blob/master/tutorials/doublespace_line_corpus_(with_noun_extraction).ipynb

Get the number of sentences in the corpus when iter_sent is set to True.

```python
print('number of sents = {}'.format(len(corpus)))
```

--------------------------------

### Check SoyNLP Version

Source: https://github.com/lovit/soynlp/blob/master/tutorials/doublespace_line_corpus_(with_noun_extraction).ipynb

Import the SoyNLP library and print its version. Ensure you have version 0.0.46 or later for this tutorial.

```python
import sys
sys.path.insert(0, '../')

import soynlp
print(soynlp.__version__)
```

--------------------------------

### Get Branching Entropy with SoyNLP

Source: https://github.com/lovit/soynlp/blob/master/tutorials/wordextractor_lecture.ipynb

Use `all_branching_entropy()` to compute branching entropy for all words. Individual scores can be accessed by word.

```python
branching_entropy = word_extractor.all_branching_entropy()
branching_entropy['아이오아이'] # (left_branching_entropy, right_branching_entropy)
```

```text
all branching entropies was computed # words = 355061
```

```text
Result:
(3.0548011243339506, 2.766022241109869)
```

--------------------------------

### Initialize BaseVectorizer with Custom Parameters

Source: https://github.com/lovit/soynlp/blob/master/README.md

Configure BaseVectorizer for creating sparse matrices from text. Parameters like minimum/maximum term frequency and document frequency can be adjusted. Verbose mode provides feedback during vectorization.

```python
vectorizer = BaseVectorizer(
    tokenizer=tokenizer,
    min_tf=0,
    max_tf=10000,
    min_df=0,
    max_df=1.0,
    stopwords=None,
    lowercase=True,
    verbose=True
)

corpus.iter_sent = False
x = vectorizer.fit_transform(corpus)
```

--------------------------------

### Select Best Candidate

Source: https://github.com/lovit/soynlp/blob/master/tutorials/tagger_lecture.ipynb

Shows how to use the `select_best` method of the `SimpleEojeolEvaluator` to determine the highest-scoring word sequence candidate from a list of possibilities.

```python
evaluator.select_best(candidates)
```

--------------------------------

### Print SoyNLP Version

Source: https://github.com/lovit/soynlp/blob/master/tutorials/normalizer_usage.ipynb

Imports the SoyNLP library and prints its current version. Ensure the library is installed and accessible in your Python path.

```python
import sys
sys.path.insert(0, '../')
import soynlp
from soynlp.normalizer import *

print(soynlp.__version__)
```

--------------------------------

### Instantiate and Use SimpleTagger (Non-Flattened)

Source: https://github.com/lovit/soynlp/blob/master/tutorials/tagger_lecture.ipynb

Shows how to use the SimpleTagger instance to tag a sentence and receive the output as a list of lists, where each inner list corresponds to the segmented words and tags for an eojeol.

```python
tagger.tag(sent, flatten=False)
```

--------------------------------

### Save and Load EojeolCounter

Source: https://context7.com/lovit/soynlp/llms.txt

Demonstrates how to save the current state of an EojeolCounter to a file and load it back later.

```python
from soynlp.utils import EojeolCounter

sentences = [
    '아이오아이가 노래를 불렀다',
    '아이오아이의 무대가 멋있다',
    '트와이스가 신곡을 발표했다'
]

counter = EojeolCounter(sents=sentences)
counter.save('eojeol_counts.txt')
counter.load('eojeol_counts.txt')
```

--------------------------------

### Getting the Number of Extracted Words

Source: https://github.com/lovit/soynlp/blob/master/tutorials/wordextractor_lecture.ipynb

Check the total number of word candidates extracted by the WordExtractor. This provides an overview of the vocabulary identified by the model.

```python
len(words)
```

```text
Result:
8163
```

--------------------------------

### Display POS Tagset

Source: https://github.com/lovit/soynlp/blob/master/tutorials/tagger_usage.ipynb

Prints the available Part-of-Speech (POS) tags and their Korean descriptions used by the SoyNLP tagger. No additional setup is required.

```python
from soynlp.pos.tagset import tagset
from pprint import pprint
pprint(tagset)
```

--------------------------------

### Get Accessor Variety with SoyNLP

Source: https://github.com/lovit/soynlp/blob/master/tutorials/wordextractor_lecture.ipynb

Use `all_accessor_variety()` to calculate accessor variety for all words. Scores for specific words are available via dictionary access.

```python
accessor_variety = word_extractor.all_accessor_variety()
accessor_variety['아이오아이'] # (left_accessor_variety, right_accessor_variety)
```

```text
all accessor variety was computed # words = 355061
```

```text
Result:
(32, 22)
```

--------------------------------

### Instantiate LRTemplateMatcher

Source: https://github.com/lovit/soynlp/blob/master/tutorials/tagger_lecture.ipynb

Creates an instance of LRTemplateMatcher, passing a dictionary object to it. This prepares the matcher for generating token candidates.

```python
lr_generator = LRTemplateMatcher(dictionary)
```

--------------------------------

### Get Cohesion Scores with SoyNLP

Source: https://github.com/lovit/soynlp/blob/master/tutorials/wordextractor_lecture.ipynb

Use `all_cohesion_scores()` to retrieve cohesion scores for all words. Access individual word scores using dictionary-like lookup.

```python
cohesion_scores = word_extractor.all_cohesion_scores()
cohesion_scores['아이오아이'] # (cohesion_forward, cohesion_backward)
```

```text
all cohesion probabilities was computed. # words = 16942
```

```text
Result:
(0.30063636035733476, 0)
```

--------------------------------

### Create and Populate a Custom POS Dictionary

Source: https://github.com/lovit/soynlp/blob/master/tutorials/tagger_usage.ipynb

Demonstrates the creation of a custom POS dictionary using a Python dictionary and initializing the Dictionary class with it. This dictionary is used by the SimpleTagger.

```python
from soynlp.pos import Dictionary
from soynlp.pos import LRTemplateMatcher
from soynlp.pos import LREvaluator
from soynlp.pos import SimpleTagger
from soynlp.pos import UnknowLRPostprocessor

pos_dict = {
    'Adverb': {'너무', '매우'},
    'Noun': {'너무너무너무', '아이오아이', '아이', '노래', '오', '이', '고양'},
    'Josa': {'는', '의', '이다', '입니다', '이', '이는', '를', '라', '라는'},
    'Verb': {'하는', '하다', '하고'},
    'Adjective': {'예쁜', '예쁘다'},
    'Exclamation': {'우와'}

}

dictionary = Dictionary(pos_dict)
```

--------------------------------

### Tokenize Sentence with LTokenizer (Flattened Output)

Source: https://github.com/lovit/soynlp/blob/master/README.md

Tokenize a Korean sentence using LTokenizer, with flatten=True (default) to get a flat list of recognized words.

```python
print(tokenizer.tokenize(sent))
# ['데이터마이닝', '을', '공부', '중이다']
```

--------------------------------

### Initialize LRTemplateMatcher

Source: https://github.com/lovit/soynlp/blob/master/tutorials/tagger_lecture.ipynb

Initializes the LRTemplateMatcher with custom or default tag sets and templates. The dictionary is a required argument.

```python
class LRTemplateMatcher(BaseTemplateMatcher):
    def __init__(self, dictionary, ltags=None, templates=None):
        if not ltags:
            ltags = {'명사', '형용사', '동사', '부사', '감탄사'}
        if not templates:
            templates = {'명사': ('조사', '동사', '형용사')}
        
        self.dictionary = dictionary
        self.ltags = ltags
        self.rtags = {tag for tags in templates.values() for tag in tags}
        self.templates = templates
```

--------------------------------

### Inspect LR Templates

Source: https://github.com/lovit/soynlp/blob/master/tutorials/tagger_usage.ipynb

The `generator.templates` attribute shows the predefined patterns used for matching. For example, it indicates that 'Noun' can be followed by 'Josa', 'Verb', or 'Adjective'.

```python
generator.templates
```