OpenGrm Libraries

https://www.openfst.org/twiki/bin/view/GRM/
Admin
OpenGrm is a collection of open-source libraries for constructing, combining, applying and searching...
Tokens:23,355
Snippets:153
Trust Score:8.6
Update:1 year ago
Show doc for...
Tokens:
Raw
### Install Pynini with Conda

Source: https://www.openfst.org/twiki/bin/view/GRM/Pynini

This command installs Pynini and its dependencies using the conda-forge channel. It's the recommended method for users who have Conda installed.

```Shell
conda install -c conda-forge pynini
```

--------------------------------

### SFst Installation Instructions

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Presents installation instructions for OpenGrm SFst, including copyright information.

```English
OpenGrm SFst INSTALL html version Installation Instructions Copyright (C) 1994, 1995, 1996, 1999, 2000, 2001, 2002, 2004, 2005, 2006,...
```

--------------------------------

### SFst Quick Tour

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

A tutorial on the OpenGrm SFST library using command-line utilities, based on a running example.

```text
Below is a brief tutorial on the OpenGrm SFST library based on a running example. We use the command line SFST utilities for this; we could have...
```

--------------------------------

### Import Pynini Module

Source: https://www.openfst.org/twiki/bin/view/GRM/PyniniDocs

Demonstrates how to import the Pynini library after installation. This is the initial step for using Pynini's functionalities.

```Python
import pynini
```

--------------------------------

### OpenFst MPDT Push Definition Example

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

Illustrates the 'push' definition in OpenFst's MPDT extension, showing how parentheses are pushed onto a stack after each byte value. This example uses a specific format for defining push operations.

```OpenFst
push = Optimize[
    "[1][1(1]"
  | "[2][2(1]"
  | "[3][3(1]"
  | "[4][4(1]"
  | "[5][5(1]"
  | "[6][6(1]"

```

--------------------------------

### Pynini Documentation Example

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

A snippet of Pynini documentation showing a character sequence, possibly related to grammar compilation or string manipulation.

```text
dP```88``````Y8, Yb, 88 `8b `` 88 ,8P gg gg...
```

--------------------------------

### OpenFst Replace Transducer Example

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

This example demonstrates how to use the 'Replace' class in OpenFst. It defines a root structure with placeholders for 'ROOT', 'NUMBER', and 'MEASURE', and then specifies the FSTs that correspond to these placeholders, including the definition of numbers and measures.

```OpenFst
Replace["[ROOT][NUMBER][MEASURE]",
     "[NUMBER]  [MEASURE]",
     n.NUMBERS,
     measures]
```

--------------------------------

### Stemmer Example

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

An example of a stemmer transducer using the CDRewrite operation. This transducer defines how to rewrite strings, typically for processing stems.

```C++
CDRewrite["+" sigma_star : "", "", "[EOS]", sigma_star];
```

--------------------------------

### Install Thrax using Conda

Source: https://www.openfst.org/twiki/bin/view/GRM/Thrax

This command installs the Thrax library and its dependencies using the conda-forge channel. It is intended for Linux (x86) and Mac OS X users who have conda installed.

```bash
conda install -c conda-forge thrax
```

--------------------------------

### Test Thrax Grammar Compilation

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

Demonstrates how to test a compiled Thrax grammar using `thraxrewrite-tester`. It specifies the compiled FAR file and the rules to apply, then shows example input and output strings.

```text
$ thraxrewrite-tester --far=example.far --rules=TOKENIZER
Input string: Well, I can't eat muffins in an agitated manner. 
Output string: Well , I ca n't eat muffins in an agitated manner .
Input string: Mr. Ernest Worthing, B. 4, The Albany.
Output string: Mr. Ernest Worthing , B. four , The Albany .
Input string: Lieutenant 1840,
Output string: Lieutenant eighteen forty ,
Input string: Uncle Jack!
Output string: Uncle Ernest !
```

--------------------------------

### Analyzer Mapping Example

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

Illustrates the mapping performed by an analyzer. It shows how inflected forms are converted into their analyzed forms, including stems and grammatical features.

```C++
maximus -> maxim+us[cas=nom][num=sg]
boni -> bon+i[cas=nom][num=pl]
```

--------------------------------

### Inflector Mapping Example

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

Shows the inversion of a lemmatizer, acting as an inflector. This transducer maps dictionary forms with features back to their inflected forms.

```C++
maximus[cas=gen][num=pl] -> maximorum
```

--------------------------------

### Lemmatizer Mapping Example

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

Demonstrates the mapping performed by a lemmatizer, which converts inflected forms to their dictionary (lemma) forms. This is achieved by composing an analyzer with a boundary deleter.

```C++
maximorum -> maximus[cas=gen][num=pl]
```

--------------------------------

### T9 Example Usage

Source: https://www.openfst.org/twiki/bin/view/GRM/PyniniDocs

Demonstrates the usage of T9 encoding and decoding functions. It encodes a sample English sentence into its T9 ciphertext and then deciphers it using the k-best function to retrieve the most probable plaintext.

```Python
pt = "THE SINGLE MOST POPULAR CHEESE IN THE WORLD"
ct = encode_string(pt)
print("CIPHERTEXT:", ct)
print("DECIPHERED PLAINTEXT:", k_best(ct, 1).string())
```

--------------------------------

### OpenFst MPDT Pop Definition Example

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

Shows the 'pop' definition for the second stack in OpenFst's MPDT extension. This definition matches closing parentheses on the second stack with corresponding opening parentheses, completing the stack balancing process.

```OpenFst
pop = Optimize[
    ("" : "[1]") "[1)2]"
  | ("" : "[2]") "[2)2]"
  | ("" : "[3]") "[3)2]"
  | ("" : "[4]") "[4)2]"
  | ("" : "[5]") "[5)2]"
  | ("" : "[6]") "[6)2]"
 

```

--------------------------------

### Get N-gram Model Information with ngraminfo

Source: https://www.openfst.org/twiki/bin/view/GRM/NGramQuickTour

ngraminfo displays detailed statistics about an n-gram language model in FST format, including the number of states, arcs, n-grams of different orders, and normalization status.

```bash
$ ngraminfo earnest.mod
```

--------------------------------

### Read N-gram Models with ngramread

Source: https://www.openfst.org/twiki/bin/view/GRM/NGramQuickTour

ngramread converts textual representations of n-gram models into FSTs. The example shows generating an FST model from an ARPA format file.

```bash
$ ngramread --ARPA earnest.ARPA earnest.mod
```

--------------------------------

### Apply Paradigm Replacement

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

Applies the ParadigmReplace function to create a modified paradigm. This example replaces the default nominative singular form with a new one for declension 2.

```C++
declension2r = ParadigmReplace[declension2, decl2nomsg, decl2nomsgr];
```

--------------------------------

### OpenFst MPDT Pop-Push Definition Example

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

Demonstrates the 'poppush' definition in OpenFst's MPDT extension. This definition handles popping a right parenthesis from one stack and pushing a new left parenthesis onto a second stack, facilitating operations like copying.

```OpenFst
poppush = Optimize[
    "[1)1][1(2]"
  | "[2)1][2(2]"
  | "[3)1][3(2]"
  | "[4)1][4(2]"
  | "[5)1][5(2]"
  | "[6)1][6(2]"

```

--------------------------------

### Grm FST Generation Statements

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

Shows examples of FST generation statements within a grm file body. It includes assigning a string to a variable and exporting a union of a variable and a string.

```grm
foo = "abc";
export bar = foo | "xyz";
```

--------------------------------

### Attaching Weights to FSTs

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

This example shows how to attach a weight to an FST using angle brackets. The content within the brackets is an arbitrary string that is interpreted as the appropriate weight type for the arc.

```grm
foo = "aaa" <1>;
goo = "aaa" : "bbb" <-1>;
```

--------------------------------

### Print N-gram Models with ngramprint

Source: https://www.openfst.org/twiki/bin/view/GRM/NGramQuickTour

ngramprint reads n-gram models and outputs them to text files, supporting various formats like ARPA. It can process both raw counts and normalized models encoded in FSTs. The example shows printing a portion of a model in ARPA format.

```bash
$ ngramprint --ARPA earnest.mod earnest.ARPA
$ head -15 earnest.ARPA
```

--------------------------------

### Download Pynini Library

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Provides download information for the Pynini library, a finite state grammar development tool in Python. Includes current version and SHA 256 hash.

```text
Current version: pynini 2.1.7.tar.gz
SHA 256: a2df40e21a6dd66c4bc1f7ecdfd72d85758ec394bf0d042fe0f3dd0d5378b820
```

--------------------------------

### Compile and Build Grammar with Thrax

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

This snippet shows the basic commands to compile a Thrax grammar and build an FST archive (far). It uses `thraxmakedep` to generate dependencies and `make` to build the archive.

```bash
$ thraxmakedep example.grm
$ make
```

--------------------------------

### Download Thrax Grammar Development Tools

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Provides download information for the OpenGrm Thrax Grammar Development Tools, including the current version and SHA 256 hash. Requires C 17.

```text
Current version: thrax 1.3.10.tar.gz (requires C 17)
SHA 256: 78dedada58a0a8543b4ea90c77a36783ac82495cf5456bec5d83baafac74b...
```

--------------------------------

### Download pynini Source Code Archives

Source: https://www.openfst.org/twiki/bin/view/GRM/PyniniDownload

This section lists various versions of the pynini library available for download as compressed tar.gz archives. These archives contain the source code for the pynini library, which is likely a component or related project to OpenFST GRM.

```bash
wget https://example.com/path/to/pynini-0.1.tar.gz
wget https://example.com/path/to/pynini-0.2.tar.gz
wget https://example.com/path/to/pynini-0.3.tar.gz
wget https://example.com/path/to/pynini-0.4.tar.gz
wget https://example.com/path/to/pynini-0.5.tar.gz
wget https://example.com/path/to/pynini-0.6.tar.gz
wget https://example.com/path/to/pynini-0.7.tar.gz
wget https://example.com/path/to/pynini-0.8.tar.gz
wget https://example.com/path/to/pynini-0.9.tar.gz
wget https://example.com/path/to/pynini-1.0.tar.gz
wget https://example.com/path/to/pynini-1.1.tar.gz
wget https://example.com/path/to/pynini-1.2.tar.gz
wget https://example.com/path/to/pynini-1.3.tar.gz
wget https://example.com/path/to/pynini-1.4.tar.gz
wget https://example.com/path/to/pynini-1.5.tar.gz
wget https://example.com/path/to/pynini-1.6.tar.gz
wget https://example.com/path/to/pynini-1.7.tar.gz
wget https://example.com/path/to/pynini-1.8.tar.gz
wget https://example.com/path/to/pynini-1.9.1.tar.gz
wget https://example.com/path/to/pynini-1.9.2.tar.gz
wget https://example.com/path/to/pynini-1.9.3.tar.gz
wget https://example.com/path/to/pynini-2.0.0.tar.gz
wget https://example.com/path/to/pynini-2.0.1.tar.gz
wget https://example.com/path/to/pynini-2.0.2.tar.gz
wget https://example.com/path/to/pynini-2.0.3.tar.gz
wget https://example.com/path/to/pynini-2.0.4.tar.gz
wget https://example.com/path/to/pynini-2.0.5.tar.gz
wget https://example.com/path/to/pynini-2.0.6.tar.gz
wget https://example.com/path/to/pynini-2.0.7.tar.gz
wget https://example.com/path/to/pynini-2.0.8.tar.gz
wget https://example.com/path/to/pynini-2.0.9.post1.tar.gz
wget https://example.com/path/to/pynini-2.0.9.post2.tar.gz
wget https://example.com/path/to/pynini-2.0.9.tar.gz
wget https://example.com/path/to/pynini-2.1.0.tar.gz
wget https://example.com/path/to/pynini-2.1.1.tar.gz
wget https://example.com/path/to/pynini-2.1.2.tar.gz
wget https://example.com/path/to/pynini-2.1.3.tar.gz
wget https://example.com/path/to/pynini-2.1.4.tar.gz
wget https://example.com/path/to/pynini-2.1.5.post1.tar.gz
wget https://example.com/path/to/pynini-2.1.5.post2.tar.gz
wget https://example.com/path/to/pynini-2.1.5.tar.gz
wget https://example.com/path/to/pynini-2.1.6.post1.tar.gz
wget https://example.com/path/to/pynini-2.1.6.tar.gz
wget https://example.com/path/to/pynini-2.1.7.tar.gz
```

--------------------------------

### Define Replacement Acceptor

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

Defines a new acceptor for a specific form that will be used in a paradigm replacement. This example redefines the nominative singular ending for declension 2.

```C++
decl2nomsgr =  sigma* "+" "[cas=nom][num=sg]";
```

--------------------------------

### Download SFst Library

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Provides download information for the OpenGrm SFst Library, including the current version and SHA 256 hash. Requires C 17.

```text
Current version: sfst 1.2.1.tar.gz (requires C 17)
SHA 256: 3da1473a45cb0cd4eda06a528808dc5fb1f5cfca659189a349aeb1f98018031c...
```

--------------------------------

### NGram Library Quick Tour

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

A quick tour of the OpenGrm NGram Library, covering n-gram model creation, modification, and usage, including corpus I/O.

```text
This tour is organized around the stages of n gram model creation, modification and use: corpus I/O ( ngramsymbols , farcompilestrings...
```

--------------------------------

### Pynini String (De)compilation

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Documentation on functions for mapping strings into FSAs (compilation) and mapping string FSTs onto strings (printing) in Pynini.

```text
This directory contains functions useful for mapping strings into FSAs ( compilation ) and for mapping string FSTs onto strings ( printing...
```

--------------------------------

### Generate Sentence from Language Model using ngramrandgen

Source: https://www.openfst.org/twiki/bin/view/GRM/NGramQuickTour

Generates a sample sentence from a trained n-gram language model. The output is piped to farprintstrings to display the generated sentence, including epsilon transitions representing backoffs.

```bash
$ ngramrandgen earnest.mod | farprintstrings
```

--------------------------------

### Export FST Symbol

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

The 'export' keyword is used to specify that a particular FST symbol or rule should be written out to the FST archive. This example demonstrates exporting an FST named 'foo'.

```grm
export foo = "abc";
```

--------------------------------

### Test PDT Rewrites with thraxrewrite-tester

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

Shows how to use the `thraxrewrite-tester` command-line tool to test Pushdown Transducer (PDT) grammars. It requires specifying the FAR file and the rule associated with the exported parenthesis transducer.

```Shell
$ thraxrewrite-tester --far=pdt.far --rules=PDT\$PARENS
Input string: aabb
Output string: aabb
Input string: ab
Output string: ab
Input string: aaaabbbb
Output string: aaaabbbb
Input string: abb
Rewrite failed.
```

--------------------------------

### SFst README: Release Information

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Provides the README for OpenGrm SFst Release 1.0, outlining its capabilities in normalizing, sampling, combining, and approximating stochastic finite state transducers.

```English
OpenGrm SFst Release 1.0 OpenGrm SFst is a library for normalizing, sampling, combining, and approximating stochastic (or probabilistic) finite...
```

--------------------------------

### Thrax README: Release Information

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Provides the README for Thrax Release 1.2, a toolkit for compiling grammars based on regular expressions and context-dependent rewrite rules into weighted finite state transducers.

```English
Thrax README Thrax Release 1.2 Thrax is a toolkit for compiling grammars based on regular expressions and context dependent rewrite rules into weighted finite state...
```

--------------------------------

### Sample from N-gram Model using ngramrandgen

Source: https://www.openfst.org/twiki/bin/view/GRM/NGramQuickTour

The `ngramrandgen` utility samples sentences from a specified n-gram model. The output can be piped to `farprintstrings` to display the generated sentences.

```bash
$ ngramrandgen --max_sents=1 earnest.mod | farprintstrings
IT IS SIMPLY A VERY INEXCUSABLE MANNER
```

--------------------------------

### Pynini Extensions

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Information on additional free libraries built using Pynini, such as the pynini rewrite Python module for applying rewrite rules.

```text
The following are additional free libraries built using Pynini: pynini rewrite: Python module for ap plying rewrite rules edit transducer...
```

--------------------------------

### Prune N-gram Models with ngramshrink

Source: https://www.openfst.org/twiki/bin/view/GRM/NGramQuickTour

ngramshrink prunes n-gram models using specified methods like relative entropy. The example demonstrates shrinking a 5-gram model to approximately 1/10th of its original size using a theta value of 0.00015.

```bash
$ ngramshrink --method=relative_entropy --theta=.00015 earnest.mod  earnest.pru
```

--------------------------------

### Download NGram Library

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Provides download information for the OpenGrm NGram Library, including the current version and SHA 256 hash. Requires C 17.

```text
Current version: ngram 1.3.16.tar.gz (requires C 17)
SHA 256: a5cc0fd15556f07fb463616c9a1e16687e3084f3c9944dd6c812385497ec112e...
```

--------------------------------

### Pynini Operators Documentation

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Documentation for specialty FST functions in Pynini, including the `fst::Cross` operation for generating transducers from acceptors.

```text
fst::Cross The cross product operation generates a transducer from two acceptors...
```

--------------------------------

### Download Baum Welch Library

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Provides download information for the Baum Welch library, including the current version and SHA 256 hash. Requires C 17.

```text
Current version: baumwelch 0.3.10.tar.gz (requires C 17)
SHA 256: 80cb35a7d625cf471bd874db0a01ff649261b87a05b409aa958164601bb99a...
```

--------------------------------

### Pynini FST Optimization

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Documentation on FST optimization in Pynini, mentioning the C template function `fst::Optimize` as the underlying mechanism.

```text
There are several ways to `optimize` a weighted finite state transducer (WFST). The C template function fst::Optimize underlies Pynini`s optimize...
```

--------------------------------

### SFst Background Material

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Background reading material concerning stochastic finite state transducers and related topics for the OpenGrm SFST library.

```text
The following is provided as background reading about stochastic finite state transducers and related material. For material...
```

--------------------------------

### Pynini String Map and String File

Source: https://www.openfst.org/twiki/bin/view/GRM/PyniniDocs

Describes 'string_map' and 'string_file' for compiling unions of string pair cross-products using a compact prefix tree representation. Mentions options for arc types.

```Python
# Example usage of string_map (conceptual)
# string_pairs = [("a", "b"), ("c", "d")]
# fst_map = pynini.string_map(string_pairs, arc_type='standard')
```

```Python
# Example usage of string_file (conceptual)
# fst_file = pynini.string_file("path/to/file.tsv", arc_type='standard')
```

--------------------------------

### ngramtdisttrain.sh: OpenGrm NGram pipeline script

Source: https://www.openfst.org/twiki/bin/view/GRM/NGramQuickTour

The `ngramtdisttrain.sh` script is a convenience shell script for running common OpenGrm NGram pipelines. It supports distributed computation and various options for input/output types, smoothing, and pruning methods.

```bash
$ ngramtdisttrain.sh --itype=text_sents --otype=pruned_lm --ifile=in.txt --ofile=lm.fst --symbols=in.syms --order=5 --smooth_method=katz --shrink_method=relative_entropy --theta=.00015
```

--------------------------------

### Load Symbol Table and Parse String

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

Demonstrates loading a symbol table from a file and then parsing a string using that table. This is useful for mapping string elements to specific FST labels.

```python
      arctic_symbol_table = SymbolTable['/path/to/bears.symtab'];
      pb = "polar bear".arctic_symbol_table;
```

--------------------------------

### Pynini Transducer Construction with Cross

Source: https://www.openfst.org/twiki/bin/view/GRM/PyniniDocs

Explains the 'cross' function for creating transducers from FSTs or strings. It details how string arguments are compiled, the cross-product interpretation, and options for weights, arc types, and token types.

```Python
# Example usage of cross function (conceptual, actual FST objects needed)
# transducer = pynini.cross(fst1, fst2, weight=1.0, arc_type='standard', input_token_type='byte', output_token_type='byte')
```

--------------------------------

### Thrax Rewrite Tester Command

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

Shows how to use the 'thraxrewrite-tester' command-line tool with an MPDT. It requires specifying the FAR file, the rules, and the associated PARENS and ASSIGNMENTS transducers.

```bash
$ thraxrewrite-tester --far=mpdt.far --rules=INFIXING_REDUPLICATION\$PARENS\$ASSIGNMENTS
```

--------------------------------

### Cite OpenGrm NGram

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxDownload

This section provides the recommended citation for users of OpenGrm NGram software in their publications. It includes the full bibliographic details of a relevant paper.

```bibtex
Brian Roark, Richard Sproat, Cyril Allauzen, Michael Riley, Jeffrey Sorensen and Terry Tai.  
2012.  The OpenGrm open-source finite-state grammar software libraries.  In _Proceedings 
of the ACL 2012 System Demonstrations_, pp. 61-66.  http://www.opengrm.org.
```

--------------------------------

### NGramInfo: Print information about an n-gram model

Source: https://www.openfst.org/twiki/bin/view/GRM/NGramQuickTour

The NGramInfo operation prints various information about an n-gram model. It takes the n-gram model file as input.

```bash
ngraminfo [in.mod]
```

--------------------------------

### Baum Welch Documentation - Training

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Documentation for the Baum Welch library's training function, which takes a FAR or WFST representing plaintext and a channel model.

```text
Training is performed using the Train function. It takes as arguments a FAR or WFST representing the plaintext, and channel model...
```

--------------------------------

### Load Symbol Table

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

Loads and returns a symbol table from a specified path. An optional third column in the symbol file can assign weights to mappings.

```OpenFst
SymbolTable['path/to/symtab']
```

--------------------------------

### Compose PDT with Other Transducers

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

Demonstrates how to compose a Pushdown Transducer (PDT) with other ordinary transducers using the `PdtCompose` function. This is necessary when integrating PDT logic with standard finite-state transducers, ensuring correct handling of parenthesis symbols.

```OpenFst
PdtCompose[sigma*, PDT, PARENS, 'right_pdt'];
```

--------------------------------

### Download Thrax Source Code Archives

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxDownload

This section lists various versions of the Thrax software available for download as compressed tar.gz archives. Each entry includes the version number, file size, and upload date, along with the uploader's name. The latest version requires C++17.

```shell
thrax-1.3.10.tar.gz (requires C++17)
SHA-256: 78dedada58a0a8543b4ea90c77a36783ac82495cf5456bec5d83baafac74b764

openfst-1.8.4.tar.gz
thrax-0.1.0.tar.gz
thrax-1.0.0.tar.gz
thrax-1.0.1.tar.gz
thrax-1.0.2.tar.gz
thrax-1.1.0.tar.gz
thrax-1.2.0.tar.gz
thrax-1.2.1.tar.gz
thrax-1.2.2.tar.gz
thrax-1.2.3.tar.gz
thrax-1.2.4.tar.gz
thrax-1.2.5.tar.gz
thrax-1.2.6.tar.gz
thrax-1.2.7.tar.gz
thrax-1.2.8.tar.gz
thrax-1.2.9.tar.gz
thrax-1.3.0.tar.gz
thrax-1.3.1.tar.gz
thrax-1.3.10.tar.gz
thrax-1.3.2.tar.gz
thrax-1.3.3.tar.gz
thrax-1.3.4.tar.gz
thrax-1.3.5.tar.gz
thrax-1.3.6.tar.gz
thrax-1.3.7.tar.gz
thrax-1.3.8.tar.gz
thrax-1.3.9.tar.gz
```

--------------------------------

### Pynini Paths: PathIterator

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Describes the fst::PathIterator class for iterating over paths in an FST, providing basic iterator functionality.

```English
fst::PathIterator This template class provides a basic iterator over paths. It is constructed...
```

--------------------------------

### Thrax Known Bugs

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Lists known bugs for Thrax, specifically mentioning a temporary issue with AssertNull and AssertEmpty registration in version 1.2.2.

```English
Known Bugs Temporary bug in thrax 1.2.2 where AssertNull and AssertEmpty are not being properly registered. This will get fixed soon, but in the meantime as a...
```

--------------------------------

### NGramSymbols Description

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Description of the NGramSymbols command-line utility, which produces a symbol table from an input text corpus.

```text
Command line utility to produce a symbol table from an input text corpus. Creates a symbol entry for every type in the corpus, as well as...
```

--------------------------------

### Importing GRM Files in OpenFst

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

Demonstrates how to import GRM files in OpenFst, specifying a path and an alias for referencing imported functions and symbols. It also shows how to access nested symbols from imported files.

```OpenFst GRM
import 'path/to/file.grm' as alias;

// Accessing a function from the imported file:
alias.foo[...];

// Accessing nested symbols from imported files:
alias.utils.common.kUpperCaseLetters.
```

--------------------------------

### NGram README: Release Information

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Provides the README for OpenGrm NGram Release 1.3, focusing on the library for creating and modifying n-gram language models encoded as weighted finite state transducers.

```English
OpenGrm NGram README OpenGrm NGram Release 1.3 The OpenGrm NGram library is used for making and modifying n gram language models encoded as weighted finite state...
```

--------------------------------

### Compile String to Acceptor (Byte Arcs)

Source: https://www.openfst.org/twiki/bin/view/GRM/PyniniDocs

Compiles a UTF-8 bytestring into a deterministic acceptor using Pynini's `accep` function. By default, each arc corresponds to a byte in the input string. The output shows the state transitions and byte values for each character.

```Python
# UTF-8 bytestring, byte arcs.
print(pynini.accep("Pont l'Evêque"))
# Output below...

0   1   80   80
1   2   111   111
2   3   110   110
3   4   116   116
4   5   32   32
5   6   108   108
6   7   39   39
7   8   69   69
8   9   118   118
9   10   195   195
10   11   170   170
11   12   113   113
12   13   117   117
13   14   101   101
14
```

--------------------------------

### Pynini Acceptor with Byte/UTF8 Modes

Source: https://www.openfst.org/twiki/bin/view/GRM/PyniniDocs

Demonstrates Pynini's handling of bracketed sequences in byte or utf8 modes. It shows how sequences are parsed as integers or treated as generated symbols, and how to escape literal brackets.

```Python
assert pynini.accep("b[0x61][97]") == pynini.accep("baa")  # OK.
```

```Python
x = pynini.accep("[It's not much of a cheese shop really]")
y = pynini.accep("[It's][not][much][of][a][cheese][shop][really]")
assert x == y   # OK.
```

```Python
bracket = pynini.accep("\\[")  # OK.

unused = pynini.accep("[")  # Not OK: Raises FstStringCompilationError.
```

--------------------------------

### NGramMake Operation

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Explains the NGramMake operation for producing smoothed, normalized language models from input n-gram count FSTs, detailing smoothing methods.

```English
NGramMake Description This operation produces a smoothed, normalized language model from input n gram count FST. It smooths the model in one of six ways: witten...
```

--------------------------------

### Load FST from File or FAR

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

Loads FSTs either directly from a file path or by extracting them from a FAR (Fst Archive).

```OpenFst
LoadFst['path/to/fst']
```

```OpenFst
LoadFstFromFar['path/to/far', 'fst_name']
```

--------------------------------

### Thrax Release Notes

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Details release information for the Thrax Grammar Compiler, including version updates and dependency changes.

```English
Thrax Release 0.1 (Alpha version.) Thrax Release 1.0 Removed dependency on ICU for UTF8 string parsing: with icu configuration flag no longer needed and...
```

--------------------------------

### Pynini Optimize Instance Method

Source: https://www.openfst.org/twiki/bin/view/GRM/PyniniOptimizeDoc

The `optimize` instance method in Pynini is used to optimize a WFST. This method is underpinned by the C++ template function `fst::Optimize` and performs operations like epsilon-removal, determinization, and minimization.

```Python
import pynini

# Assuming 'fst' is a Pynini WFST object
optimized_fst = fst.optimize()
```

--------------------------------

### NGramSymbols: Create symbol table from corpus

Source: https://www.openfst.org/twiki/bin/view/GRM/NGramQuickTour

The NGramSymbols operation creates a symbol table from a corpus. It allows specifying symbols for epsilon and out-of-vocabulary (OOV) tokens. Input is a text file (corpus), and output is a text file (symbol table).

```bash
ngramsmybols [--epsilon_symbol] [--OOV_symbol] [in.txt [out.txt]]
```

--------------------------------

### NGram Library Suggestions

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Presents suggestions for improving the OpenGrm NGram library, including memory usage, counting from cyclic FSTs, and support for specific features.

```English
OpenGrm NGram Library Suggestions 1. Improve memory usage of ngramread uses 14 gb for 1.5gb input (Brian) 2. Counting from cyclic FSTs (Cyril) 3. support...
```

--------------------------------

### Create Analyzer

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

Constructs an analyzer that maps inflected forms to their analyses. It takes a paradigm acceptor, a set of stems, a stemmer transducer, and a deleter transducer as input.

```C++
Analyzer[paradigm, stems, stemmer, deleter]
```

--------------------------------

### Thrax Copying License

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Details the licensing information for Thrax under the Apache License, Version 2.0.

```English
Thrax COPYING Licensed under the Apache License, Version 2.0 (the `License`); you may not use these files except in compliance with the License. You may obtain a copy...
```

--------------------------------

### Function Definition and Return

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

This snippet demonstrates how to declare and define a function in GRM. It uses the 'func' keyword to define the function and 'return' to specify the value returned to the caller. Statements after 'return' are ignored.

```grm
func Cleanup[fst] {
  return RmEpsilon[Determinize[fst]];
}
```

--------------------------------

### Compile String File to FST

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

Loads a file containing strings or string pairs and compiles it into an FST representing their union. Can be more efficient than manual union for large lists. Supports different parsing modes and symbol tables.

```OpenFst
StringFile['strings_file']
```

```OpenFst
StringFile['strings_file', 'byte', my_symbols]
```

--------------------------------

### Apply N-gram Model using ngramapply

Source: https://www.openfst.org/twiki/bin/view/GRM/NGramQuickTour

The `ngramapply` utility applies an n-gram model to a concatenated archive of automata. The result is a FAR (Finite Automata Representation) weighted by the n-gram model.

```bash
$ ngramapply earnest.mod earnest.far | farprintstrings --print_weight
```

--------------------------------

### NGram Copying License

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Details the licensing information for OpenGrm NGram under the Apache License, Version 2.0.

```English
OpenGrm NGram COPYING Licensed under the Apache License, Version 2.0 (the `License`); you may not use these files except in compliance with the License. You may obtain...
```

--------------------------------

### SFst Copying License

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Details the licensing information for OpenGrm SFst under the Apache License, Version 2.0.

```English
OpenGrm SFst COPYING Licensed under the Apache License, Version 2.0 (the `License`); you may not use these files except in compliance with the License. You may obtain...
```

--------------------------------

### Convert Text Corpus to FAR Archive using farcompilestrings

Source: https://www.openfst.org/twiki/bin/view/GRM/NGramQuickTour

Converts a text corpus into a binary FAR archive using a specified symbol table. This archive format is efficient for processing by NGram Library utilities. The --keep_symbols flag ensures symbol information is retained.

```bash
$ farcompilestrings --fst_type=compact --symbols=earnest.syms --keep_symbols earnest.txt earnest.far
```

--------------------------------

### NGramRandgen: Sample sentences from an n-gram model

Source: https://www.openfst.org/twiki/bin/view/GRM/NGramQuickTour

The NGramRandgen operation randomly samples sentences from an n-gram model. It allows setting limits on the maximum number of sentences and their length, and a seed for reproducibility. Input is a model FST, and output can be an FST archive.

```bash
ngramrandgen [--max_sents] [--max_length] [--seed] [in.mod [out.far]]
```

--------------------------------

### FST CDRewrite

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

Generates a new FST for context-dependent rewriting. It takes a transducer, context acceptors, an alphabet machine, and options for rewrite direction and obligation.

```OpenFST Syntax
CDRewrite[tau, lambda, rho, sigma_star, 'ltr'|'rtl'|'sim', 'obl'|'opt']
CDRewrite["s" : "z", "", "d[EOS]", sigma_star]   ## ltr obligatory rule changing "s" into "z" before a "d" at the end of a string
```

--------------------------------

### Generate Random Sentence with ngramrandgen

Source: https://www.openfst.org/twiki/bin/view/GRM/NGramQuickTour

ngramrandgen generates a random sentence from a pruned n-gram model. The output can be piped to other utilities like farprintstrings for display.

```bash
$ ngramrandgen earnest.pru | farprintstrings
```

--------------------------------

### Evaluate N-gram Model Perplexity using ngramperplexity

Source: https://www.openfst.org/twiki/bin/view/GRM/NGramQuickTour

The `ngramperplexity` command evaluates an n-gram model by calculating the perplexity of given text. It requires input text piped to it, often after processing with `farcompilestrings`.

```bash
echo -e "A HAND BAG\nBAG HAND A" \
    | farcompilestrings --generate_keys=1 -symbols=earnest.syms --keep_symbols \
    | ngramperplexity --v=1 earnest.mod -
```

--------------------------------

### OpenFst MPdtCompose Usage

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

Demonstrates the usage of the MPdtCompose operation in OpenFst for combining an MPDT with another transducer. It specifies the arguments required for correct composition, including PARENS and ASSIGNMENTS transducers.

```OpenFst
MPdtCompose[MPDT, fst, PARENS, ASSIGNMENTS, 'left_mpdt'];
```

--------------------------------

### Print FAR Archive Content using farprintstrings

Source: https://www.openfst.org/twiki/bin/view/GRM/NGramQuickTour

Prints the content of a FAR archive to standard output, typically redirected to a text file. This is useful for inspecting the contents of the archive or for converting it back to a human-readable format.

```bash
$ farprintstrings earnest.far > earnest.txt
```

--------------------------------

### Define Linguistic Features in Thrax

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

Demonstrates how to define linguistic features such as case, number, and gender using Thrax's Feature constructor. These definitions are then translated into simple 2-state FSTs.

```Thrax
case = Feature['case', 'nom', 'acc', 'gen', 'dat'];
number = Feature['number', 'sg', 'pl'];
gender = Feature['gender', 'mas', 'fem', 'neu'];
```

--------------------------------

### NGramMake: Smooth and normalize n-gram models

Source: https://www.openfst.org/twiki/bin/view/GRM/NGramQuickTour

The NGramMake operation performs n-gram model smoothing and normalization. It supports various smoothing methods like Katz, Kneser-Ney, Witten-Bell, and Absolute Discounting. It can also handle unsmoothed models. Input and output are FST files.

```bash
ngrammake [--method] [--backoff] [--bins] [--witten_bell_k] [--discount_D] [in.fst [out.fst]]
```

```c++
NGramAbsolute(&CountFst);
```

```c++
NGramKatz(&CountFst);
```

```c++
NGramKneserNey(&CountFst);
```

```c++
NGramUnsmoothed(&CountFst);
```

```c++
NGramWittenBell(&CountFst);
```

--------------------------------

### Re-estimate N-gram Models with ngrammarginalize

Source: https://www.openfst.org/twiki/bin/view/GRM/NGramQuickTour

ngrammarginalize re-estimates smoothed n-gram models using marginalization constraints, similar to Kneser-Ney smoothing. This utility is available from version 1.1.0 onwards and may require multiple iterations using the --iterations switch.

```bash
$ ngrammarginalize earnest.mod earnest.marg.mod
```

--------------------------------

### FST Parsing Modes

Source: https://www.openfst.org/twiki/bin/view/GRM/ThraxQuickTour

This snippet illustrates different parsing modes for strings when creating FST arcs. It shows byte-by-byte parsing, UTF8 character parsing, and symbol table parsing.

```grm
a = "avarice";                # Parse mode = byte.
b = "brackish".byte;          # Parse mode = byte.
c = "calumny".utf8;           # Parse mode = UTF8.
d = "decrepitude".my_symtab;  # Parse mode = symbol table.
```

--------------------------------

### Extract Symbols from Corpus using ngramsymbols

Source: https://www.openfst.org/twiki/bin/view/GRM/NGramQuickTour

Extracts symbols from a text corpus to create an OpenFst-style symbol table. This is the first step in processing text corpora for the NGram Library. The output symbol table can be used with other utilities like farcompilestrings.

```bash
$ ngramsymbols earnest.txt earnest.syms
```

--------------------------------

### NGramRead Description

Source: https://www.openfst.org/twiki/bin/view/GRM/WebChanges

Description of the NGramRead utility, detailing flags for specifying text input format, with a default of reading as sorted text.

```text
It has flags for specifying the format of the text input, currently one of two options: By default, the text file is read as a sorted...
```
Tokens:
Raw