Morfessor 2.0

https://morfessor.readthedocs.io/en/latest/
Admin
Morfessor is an unsupervised morphological segmentation tool that discovers and segments morphemes...
Tokens:7,084
Snippets:156
Trust Score:9.6
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install from tarball/zip file using setup.py

Source: https://morfessor.readthedocs.io/en/latest/installation.html

Unpack the tarball or zip file and run the setup.py script to install Morfessor.

```bash
python setup.py install

```

--------------------------------

### Install from PyPI using pip

Source: https://morfessor.readthedocs.io/en/latest/installation.html

Use pip to automatically download and install the latest version of Morfessor from PyPI.

```bash
pip install morfessor

```

--------------------------------

### Install from PyPI using easy_install

Source: https://morfessor.readthedocs.io/en/latest/installation.html

Use easy_install to automatically download and install the latest version of Morfessor from PyPI.

```bash
easy_install morfessor

```

--------------------------------

### Install from tarball/zip file using pip

Source: https://morfessor.readthedocs.io/en/latest/installation.html

Use pip to install Morfessor directly from the tarball or zip file.

```bash
pip install morfessor-VERSION.tar.gz

```

--------------------------------

### Training and Segmentation Example

Source: https://morfessor.readthedocs.io/en/latest/cmdtools.html

Example of training a model, saving it in Morfessor 1.0 text format, and segmenting a test set.

```bash
morfessor -t inputdata.txt -S model.segm -T test.txt

```

--------------------------------

### morfessor-train example

Source: https://morfessor.readthedocs.io/en/latest/cmdtools.html

Example of training a Morfessor model from a wordcount list with specific encoding, logging, and saving options.

```bash
morfessor-train --encoding=ISO_8859-15 --traindata-list --logfile=log.log -s model.bin -d ones traindata.txt

```

--------------------------------

### Annotation file example

Source: https://morfessor.readthedocs.io/en/latest/filetypes.html

An example of an annotation file.

```text
kahvikakku kahvi kakku, kahvi kak ku
kahvikilon kahvi kilon
kahvikoneemme kahvi konee mme, kah vi ko nee mme

```

--------------------------------

### morfessor-segment example

Source: https://morfessor.readthedocs.io/en/latest/cmdtools.html

Example of loading a binary model and segmenting words in a test data file.

```bash
morfessor-segment -l model.bin testdata.txt

```

--------------------------------

### morfessor-evaluate example

Source: https://morfessor.readthedocs.io/en/latest/cmdtools.html

Example of evaluating three different models against a golden standard, outputting results in latex table format.

```bash
morfessor-evaluate --format-template=latex goldstd.txt model1.bin model2.segm model3.bin

```

--------------------------------

### Text corpus file example

Source: https://morfessor.readthedocs.io/en/latest/filetypes.html

An example of a text corpus file.

```text
kavhikakku kahvikilon kahvikilon
kahvikoneemme kahvikakku

```

--------------------------------

### Word list file example 2

Source: https://morfessor.readthedocs.io/en/latest/filetypes.html

An example of a word list corpus file without counts.

```text
kahvikakku
kahvikilon
kahvikoneemme

```

--------------------------------

### Word list file example 1

Source: https://morfessor.readthedocs.io/en/latest/filetypes.html

An example of a word list corpus file with counts.

```text
10 kahvikakku
5 kahvikilon
24 kahvikoneemme

```

--------------------------------

### Morfessor 1.0 style text model example

Source: https://morfessor.readthedocs.io/en/latest/filetypes.html

An example of a Morfessor 1.0 style text model.

```text
10 kahvi + kakku
5 kahvi + kilo + n
24 kahvi + kone + emme

```

--------------------------------

### Command-line equivalent for testing type vs token models

Source: https://morfessor.readthedocs.io/en/latest/libinterface.html

Provides the command-line instructions to replicate the Python script for testing different Mofessor models.

```bash
morfessor-train -s model_types -d ones training_data
morfessor-train -s model_logtokens -d log training_data
morfessor-train -s model_tokens training_data

morfessor-evaluate gold_std morfessor-train morfessor-train morfessor-train
```

--------------------------------

### morfessor-train basic command structure

Source: https://morfessor.readthedocs.io/en/latest/cmdtools.html

The basic command structure for training a Morfessor model.

```bash
morfessor-train [arguments] traindata-file [traindata-file ...]

```

--------------------------------

### Segmenting new data using an existing model

Source: https://morfessor.readthedocs.io/en/latest/libinterface.html

Demonstrates how to load a pre-trained Mofessor model and use it to segment new words.

```python
import morfessor

io = morfessor.MorfessorIO()

model = io.read_binary_model_file('model.bin')

words = ['words', 'segmenting', 'morfessor', 'unsupervised']

for word in words:
    print(model.viterbi_segment(word))
```

--------------------------------

### morfessor-evaluate basic command structure

Source: https://morfessor.readthedocs.io/en/latest/cmdtools.html

The basic command structure for evaluating a Morfessor model against a gold-standard.

```bash
morfessor-evaluate [arguments] <goldstandard> <model> [<model> ...]

```

--------------------------------

### morfessor-segment basic command structure

Source: https://morfessor.readthedocs.io/en/latest/cmdtools.html

The basic command structure for segmenting test data with a Morfessor model.

```bash
morfessor-segment [arguments] testcorpus-file [testcorpus-file ...]

```

--------------------------------

### Testing type vs token models

Source: https://morfessor.readthedocs.io/en/latest/libinterface.html

Compares different Mofessor models (type, log-token, token) by training them on provided data and evaluating their performance against gold standard annotations.

```python
import morfessor
import math

io = morfessor.MorfessorIO()

train_data = list(io.read_corpus_file('training_data'))

model_types = morfessor.BaselineModel()
model_logtokens = morfessor.BaselineModel()
model_tokens = morfessor.BaselineModel()

model_types.load_data(train_data, count_modifier=lambda x: 1)
def log_func(x):
    return int(round(math.log(x + 1, 2)))
model_logtokens.load_data(train_data, count_modifier=log_func)
model_tokens.load_data(train_data)

models = [model_types, model_logtokens, model_tokens]

for model in models:
    model.train_batch()

goldstd_data = io.read_annotations_file('gold_std')
ev = morfessor.MorfessorEvaluation(goldstd_data)
results = [ev.evaluate_model(m) for m in models]

wsr = morfessor.WilcoxonSignedRank()
r = wsr.significance_test(results)
WilcoxonSignedRank.print_table(r)
```

--------------------------------

### Annotation file specification

Source: https://morfessor.readthedocs.io/en/latest/filetypes.html

The specification for an annotation file.

```text
<compound> <analysis1construction1>[ <analysis1constructionN>][, <analysis2construction1> [<analysis2constructionN>]*]*

```

--------------------------------

### Word list file specification

Source: https://morfessor.readthedocs.io/en/latest/filetypes.html

The specification for a word list corpus file.

```text
[<int><space>]<COMPOUND>

```

--------------------------------

### Morfessor 1.0 style text model specification

Source: https://morfessor.readthedocs.io/en/latest/filetypes.html

The specification for the Morfessor 1.0 style text model format.

```text
<int><space><CONSTRUCTION>[<space>+<space><CONSTRUCTION>]*

```
Tokens:
Raw