### Install from tarball/zip using setup.py Source: https://github.com/aalto-speech/morfessor/blob/master/docs/source/installation.rst Unpack the tarball or zip file and run the setup.py script to install Morfessor. ```bash python setup.py install ``` -------------------------------- ### Install from PyPI using easy_install Source: https://github.com/aalto-speech/morfessor/blob/master/docs/source/installation.rst Install the latest version of Morfessor from the Python Package Index (PyPI) using easy_install. ```bash easy_install morfessor ``` -------------------------------- ### Install from tarball/zip using pip Source: https://github.com/aalto-speech/morfessor/blob/master/docs/source/installation.rst Use pip to install Morfessor directly from the tarball or zip file. ```bash pip install morfessor-VERSION.tar.gz ``` -------------------------------- ### Install from PyPI using pip Source: https://github.com/aalto-speech/morfessor/blob/master/docs/source/installation.rst Install the latest version of Morfessor from the Python Package Index (PyPI) using pip. ```bash pip install morfessor ``` -------------------------------- ### morfessor-segment example Source: https://github.com/aalto-speech/morfessor/blob/master/docs/source/cmdtools.rst Example of loading a binary model and segmenting words from a test data file. ```bash morfessor-segment -l model.bin testdata.txt ``` -------------------------------- ### Training and Segmenting Example Source: https://github.com/aalto-speech/morfessor/blob/master/docs/source/cmdtools.rst Example of training a model from input data, saving it in morfessor1 format, and segmenting a test set. ```bash morfessor -t inputdata.txt -S model.segm -T test.txt ``` -------------------------------- ### morfessor-train example Source: https://github.com/aalto-speech/morfessor/blob/master/docs/source/cmdtools.rst Example of training a morfessor model from a wordcount list with specific encoding, logging, and saving options. ```bash morfessor-train --encoding=ISO_8859-15 --traindata-list --logfile=log.log -s model.bin -d ones traindata.txt ``` -------------------------------- ### Annotation file example Source: https://github.com/aalto-speech/morfessor/blob/master/docs/source/filetypes.rst Example of an annotation file. ```text kahvikakku kahvi kakku, kahvi kak ku kahvikilon kahvi kilon kahvikoneemme kahvi konee mme, kah vi ko nee mme ``` -------------------------------- ### morfessor-evaluate example Source: https://github.com/aalto-speech/morfessor/blob/master/docs/source/cmdtools.rst Example of evaluating three different models against a golden standard, outputting results in latex table format. ```bash morfessor-evaluate --format-template=latex goldstd.txt model1.bin model2.segm model3.bin ``` -------------------------------- ### Text corpus file example Source: https://github.com/aalto-speech/morfessor/blob/master/docs/source/filetypes.rst Example of a text corpus file. ```text kavhikakku kahvikilon kahvikilon kahvikoneemme kahvikakku ``` -------------------------------- ### Word list file example 1 Source: https://github.com/aalto-speech/morfessor/blob/master/docs/source/filetypes.rst Example of a word list corpus file with counts. ```text 10 kahvikakku 5 kahvikilon 24 kahvikoneemme ``` -------------------------------- ### Word list file example 2 Source: https://github.com/aalto-speech/morfessor/blob/master/docs/source/filetypes.rst Example of a word list corpus file without counts. ```text kahvikakku kahvikilon kahvikoneemme ``` -------------------------------- ### Testing type vs token models Source: https://github.com/aalto-speech/morfessor/blob/master/docs/source/libinterface.rst This example shows how to train and evaluate Morfessor models with different data loading strategies (type, log-token, token) and perform a significance test. ```python import morfessor import math io = morfessor.MorfessorIO() train_data = list(io.read_corpus_file('training_data')) model_types = morfessor.BaselineModel() model_logtokens = morfessor.BaselineModel() model_tokens = morfessor.BaselineModel() model_types.load_data(train_data, count_modifier=lambda x: 1) def log_func(x): return int(round(math.log(x + 1, 2))) model_logtokens.load_data(train_data, count_modifier=log_func) model_tokens.load_data(train_data) models = [model_types, model_logtokens, model_tokens] for model in models: model.train_batch() goldstd_data = io.read_annotations_file('gold_std') ev = morfessor.MorfessorEvaluation(goldstd_data) results = [ev.evaluate_model(m) for m in models] wsr = morfessor.WilcoxonSignedRank() r = wsr.significance_test(results) WilcoxonSignedRank.print_table(r) ``` -------------------------------- ### Morfessor 1.0 style text model example Source: https://github.com/aalto-speech/morfessor/blob/master/docs/source/filetypes.rst Example of a Morfessor 1.0 style text model file. ```text 10 kahvi + kakku 5 kahvi + kilo + n 24 kahvi + kone + emme ``` -------------------------------- ### morfessor-train basic command structure Source: https://github.com/aalto-speech/morfessor/blob/master/docs/source/cmdtools.rst The basic command structure for training a morfessor model. ```bash morfessor-train [arguments] traindata-file [traindata-file ...] ``` -------------------------------- ### Command line equivalent for testing type vs token models Source: https://github.com/aalto-speech/morfessor/blob/master/docs/source/libinterface.rst This shows the command-line equivalent for training and evaluating Morfessor models with different data loading strategies. ```bash morfessor-train -s model_types -d ones training_data morfessor-train -s model_logtokens -d log training_data morfessor-train -s model_tokens training_data morfessor-evaluate gold_std morfessor-train morfessor-train morfessor-train ``` -------------------------------- ### morfessor-evaluate basic command structure Source: https://github.com/aalto-speech/morfessor/blob/master/docs/source/cmdtools.rst The basic command structure for evaluating a morfessor model against a gold-standard. ```bash morfessor-evaluate [arguments] [ ...] ``` -------------------------------- ### morfessor-segment basic command structure Source: https://github.com/aalto-speech/morfessor/blob/master/docs/source/cmdtools.rst The basic command structure for segmenting test data with a morfessor model. ```bash morfessor-segment [arguments] testcorpus-file [testcorpus-file ...] ``` -------------------------------- ### Segmenting new data using an existing model Source: https://github.com/aalto-speech/morfessor/blob/master/docs/source/libinterface.rst This code snippet demonstrates how to load a pre-trained Morfessor model from a binary file and use it to segment new words. ```python import morfessor io = morfessor.MorfessorIO() model = io.read_binary_model_file('model.bin') words = ['words', 'segmenting', 'morfessor', 'unsupervised'] for word in words: print(model.viterbi_segment(word)) ``` -------------------------------- ### Annotation file specification Source: https://github.com/aalto-speech/morfessor/blob/master/docs/source/filetypes.rst Specification for the format of an annotation file. ```text [ ][, []*]* ``` -------------------------------- ### Word list file specification Source: https://github.com/aalto-speech/morfessor/blob/master/docs/source/filetypes.rst Specification for the format of a word list corpus file. ```text [] ``` -------------------------------- ### Morfessor 1.0 style text model specification Source: https://github.com/aalto-speech/morfessor/blob/master/docs/source/filetypes.rst Specification for the format of Morfessor 1.0 style text model files. ```text [+]* ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.