### Install from tarball/zip file using setup.py Source: https://morfessor.readthedocs.io/en/latest/installation.html Unpack the tarball or zip file and run the setup.py script to install Morfessor. ```bash python setup.py install ``` -------------------------------- ### Install from PyPI using pip Source: https://morfessor.readthedocs.io/en/latest/installation.html Use pip to automatically download and install the latest version of Morfessor from PyPI. ```bash pip install morfessor ``` -------------------------------- ### Install from PyPI using easy_install Source: https://morfessor.readthedocs.io/en/latest/installation.html Use easy_install to automatically download and install the latest version of Morfessor from PyPI. ```bash easy_install morfessor ``` -------------------------------- ### Install from tarball/zip file using pip Source: https://morfessor.readthedocs.io/en/latest/installation.html Use pip to install Morfessor directly from the tarball or zip file. ```bash pip install morfessor-VERSION.tar.gz ``` -------------------------------- ### Training and Segmentation Example Source: https://morfessor.readthedocs.io/en/latest/cmdtools.html Example of training a model, saving it in Morfessor 1.0 text format, and segmenting a test set. ```bash morfessor -t inputdata.txt -S model.segm -T test.txt ``` -------------------------------- ### morfessor-train example Source: https://morfessor.readthedocs.io/en/latest/cmdtools.html Example of training a Morfessor model from a wordcount list with specific encoding, logging, and saving options. ```bash morfessor-train --encoding=ISO_8859-15 --traindata-list --logfile=log.log -s model.bin -d ones traindata.txt ``` -------------------------------- ### Annotation file example Source: https://morfessor.readthedocs.io/en/latest/filetypes.html An example of an annotation file. ```text kahvikakku kahvi kakku, kahvi kak ku kahvikilon kahvi kilon kahvikoneemme kahvi konee mme, kah vi ko nee mme ``` -------------------------------- ### morfessor-segment example Source: https://morfessor.readthedocs.io/en/latest/cmdtools.html Example of loading a binary model and segmenting words in a test data file. ```bash morfessor-segment -l model.bin testdata.txt ``` -------------------------------- ### morfessor-evaluate example Source: https://morfessor.readthedocs.io/en/latest/cmdtools.html Example of evaluating three different models against a golden standard, outputting results in latex table format. ```bash morfessor-evaluate --format-template=latex goldstd.txt model1.bin model2.segm model3.bin ``` -------------------------------- ### Text corpus file example Source: https://morfessor.readthedocs.io/en/latest/filetypes.html An example of a text corpus file. ```text kavhikakku kahvikilon kahvikilon kahvikoneemme kahvikakku ``` -------------------------------- ### Word list file example 2 Source: https://morfessor.readthedocs.io/en/latest/filetypes.html An example of a word list corpus file without counts. ```text kahvikakku kahvikilon kahvikoneemme ``` -------------------------------- ### Word list file example 1 Source: https://morfessor.readthedocs.io/en/latest/filetypes.html An example of a word list corpus file with counts. ```text 10 kahvikakku 5 kahvikilon 24 kahvikoneemme ``` -------------------------------- ### Morfessor 1.0 style text model example Source: https://morfessor.readthedocs.io/en/latest/filetypes.html An example of a Morfessor 1.0 style text model. ```text 10 kahvi + kakku 5 kahvi + kilo + n 24 kahvi + kone + emme ``` -------------------------------- ### Command-line equivalent for testing type vs token models Source: https://morfessor.readthedocs.io/en/latest/libinterface.html Provides the command-line instructions to replicate the Python script for testing different Mofessor models. ```bash morfessor-train -s model_types -d ones training_data morfessor-train -s model_logtokens -d log training_data morfessor-train -s model_tokens training_data morfessor-evaluate gold_std morfessor-train morfessor-train morfessor-train ``` -------------------------------- ### morfessor-train basic command structure Source: https://morfessor.readthedocs.io/en/latest/cmdtools.html The basic command structure for training a Morfessor model. ```bash morfessor-train [arguments] traindata-file [traindata-file ...] ``` -------------------------------- ### Segmenting new data using an existing model Source: https://morfessor.readthedocs.io/en/latest/libinterface.html Demonstrates how to load a pre-trained Mofessor model and use it to segment new words. ```python import morfessor io = morfessor.MorfessorIO() model = io.read_binary_model_file('model.bin') words = ['words', 'segmenting', 'morfessor', 'unsupervised'] for word in words: print(model.viterbi_segment(word)) ``` -------------------------------- ### morfessor-evaluate basic command structure Source: https://morfessor.readthedocs.io/en/latest/cmdtools.html The basic command structure for evaluating a Morfessor model against a gold-standard. ```bash morfessor-evaluate [arguments] [ ...] ``` -------------------------------- ### morfessor-segment basic command structure Source: https://morfessor.readthedocs.io/en/latest/cmdtools.html The basic command structure for segmenting test data with a Morfessor model. ```bash morfessor-segment [arguments] testcorpus-file [testcorpus-file ...] ``` -------------------------------- ### Testing type vs token models Source: https://morfessor.readthedocs.io/en/latest/libinterface.html Compares different Mofessor models (type, log-token, token) by training them on provided data and evaluating their performance against gold standard annotations. ```python import morfessor import math io = morfessor.MorfessorIO() train_data = list(io.read_corpus_file('training_data')) model_types = morfessor.BaselineModel() model_logtokens = morfessor.BaselineModel() model_tokens = morfessor.BaselineModel() model_types.load_data(train_data, count_modifier=lambda x: 1) def log_func(x): return int(round(math.log(x + 1, 2))) model_logtokens.load_data(train_data, count_modifier=log_func) model_tokens.load_data(train_data) models = [model_types, model_logtokens, model_tokens] for model in models: model.train_batch() goldstd_data = io.read_annotations_file('gold_std') ev = morfessor.MorfessorEvaluation(goldstd_data) results = [ev.evaluate_model(m) for m in models] wsr = morfessor.WilcoxonSignedRank() r = wsr.significance_test(results) WilcoxonSignedRank.print_table(r) ``` -------------------------------- ### Annotation file specification Source: https://morfessor.readthedocs.io/en/latest/filetypes.html The specification for an annotation file. ```text [ ][, []*]* ``` -------------------------------- ### Word list file specification Source: https://morfessor.readthedocs.io/en/latest/filetypes.html The specification for a word list corpus file. ```text [] ``` -------------------------------- ### Morfessor 1.0 style text model specification Source: https://morfessor.readthedocs.io/en/latest/filetypes.html The specification for the Morfessor 1.0 style text model format. ```text [+]* ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.