Malware Report Documentation

https://pypi.org/project/sacremoses/
Admin
This project provides comprehensive documentation for submitting malware reports, offering guidance...
Tokens:4,888
Snippets:56
Trust Score:10
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install Sacremoses

Source: https://pypi.org/project/sacremoses/0.1.1

Install the latest version of Sacremoses using pip.

```bash
pip install -U sacremoses
```

--------------------------------

### Chaining Sacremoses Commands in a Pipeline

Source: https://pypi.org/project/sacremoses

This example demonstrates how to chain multiple Sacremoses commands (normalize, tokenize, truecase) in a single pipeline for batch processing text files. It shows how to use options like removing control characters, aggressive dash splitting, and loading/training a truecase model.

```bash
cat big.txt | sacremoses -l en -j 4 \
    normalize -c tokenize -a truecase -a -m big.truemodel \
    > big.txt.norm.tok.true
```

--------------------------------

### Train and Use Truecaser

Source: https://pypi.org/project/sacremoses/0.1.1

Shows how to train a truecase model from a text file, save it, and then use it to truecase strings. Also demonstrates loading a pre-trained model and using specific options.

```python
from sacremoses import MosesTruecaser, MosesTokenizer

# Train a new truecaser from a 'big.txt' file.
>>> mtr = MosesTruecaser()
>>> mtok = MosesTokenizer(lang='en')

# Save the truecase model to 'big.truecasemodel' using `save_to`
>> tokenized_docs = [mtok.tokenize(line) for line in open('big.txt')] 
>>> mtr.train(tokenized_docs, save_to='big.truecasemodel')
```

```python
# Save the truecase model to 'big.truecasemodel' after training
# (just in case you forgot to use `save_to`)
>>> mtr = MosesTruecaser()
>>> mtr.train('big.txt')
>>> mtr.save_model('big.truecasemodel')
```

```python
# Truecase a string after training a model.
>>> mtr = MosesTruecaser()
>>> mtr.train('big.txt')
>>> mtr.truecase("THE ADVENTURES OF SHERLOCK HOLMES")
['the', 'adventures', 'of', 'Sherlock', 'Holmes']
```

```python
# Loads a model and truecase a string using trained model.
>>> mtr = MosesTruecaser('big.truecasemodel')
>>> mtr.truecase("THE ADVENTURES OF SHERLOCK HOLMES")
['the', 'adventures', 'of', 'Sherlock', 'Holmes']
>>> mtr.truecase("THE ADVENTURES OF SHERLOCK HOLMES", use_known=True)
['the', 'ADVENTURES', 'OF', 'SHERLOCK', 'HOLMES']
>>> mtr.truecase("THE ADVENTURES OF SHERLOCK HOLMES", return_str=True)
'the adventures of Sherlock Holmes'
```

--------------------------------

### Train and Use Truecaser

Source: https://pypi.org/project/sacremoses

Shows how to train a truecase model from a text file and then use it to truecase strings. Supports loading pre-trained models.

```python
from sacremoses import MosesTruecaser, MosesTokenizer

# Train a new truecaser from a 'big.txt' file.
>>> mtr = MosesTruecaser()
>>> mtok = MosesTokenizer(lang='en')

# Save the truecase model to 'big.truecasemodel' using `save_to`
>> tokenized_docs = [mtok.tokenize(line) for line in open('big.txt')] # doctest: +SKIP
>>> mtr.train(tokenized_docs, save_to='big.truecasemodel')

# Save the truecase model to 'big.truecasemodel' after training
# (just in case you forgot to use `save_to`)
>>> mtr = MosesTruecaser()
>>> mtr.train('big.txt') # doctest: +SKIP
>>> mtr.save_model('big.truecasemodel')

# Truecase a string after training a model.
>>> mtr = MosesTruecaser()
>>> mtr.train('big.txt') # doctest: +SKIP
>>> mtr.truecase("THE ADVENTURES OF SHERLOCK HOLMES")
['the', 'adventures', 'of', 'Sherlock', 'Holmes']

# Loads a model and truecase a string using trained model.
>>> mtr = MosesTruecaser('big.truecasemodel')
>>> mtr.truecase("THE ADVENTURES OF SHERLOCK HOLMES")
['the', 'adventures', 'of', 'Sherlock', 'Holmes']
>>> mtr.truecase("THE ADVENTURES OF SHERLOCK HOLMES", use_known=True)
['the', 'ADVENTURES', 'OF', 'SHERLOCK', 'HOLMES']
>>> mtr.truecase("THE ADVENTURES OF SHERLOCK HOLMES", return_str=True)
'the adventures of Sherlock Holmes'
```

--------------------------------

### Sacremoses Normalize Help

Source: https://pypi.org/project/sacremoses/0.1.1

Shows the help information for the sacremoses normalize command, detailing its normalization options.

```bash
$ sacremoses normalize --help
```

--------------------------------

### Sacremoses CLI Help

Source: https://pypi.org/project/sacremoses/0.1.1

Displays the help message for the Sacremoses command-line interface, showing available global options and commands.

```bash
$ sacremoses --help
Usage: sacremoses [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

Options:
  -l, --language TEXT      Use language specific rules when tokenizing
  -j, --processes INTEGER  No. of processes.
  -e, --encoding TEXT      Specify encoding of file.
  -q, --quiet              Disable progress bar.
  --version                Show the version and exit.
  -h, --help               Show this message and exit.

Commands:
  detokenize
  detruecase
  normalize
  tokenize
  train-truecase
  truecase

```

--------------------------------

### Sacremoses Tokenizer Help

Source: https://pypi.org/project/sacremoses

Displays the help message for the sacremoses tokenize command, outlining available options for tokenization.

```bash
$ sacremoses tokenize --help
Usage: sacremoses tokenize [OPTIONS]

Options:
  -a, --aggressive-dash-splits   Triggers dash split rules.
  -x, --xml-escape               Escape special characters for XML.
  -p, --protected-patterns TEXT  Specify file with patters to be protected in
                                 tokenisation.
  -c, --custom-nb-prefixes TEXT  Specify a custom non-breaking prefixes file,
                                 add prefixes to the default ones from the
                                 specified language.
  -h, --help                     Show this message and exit.
```

--------------------------------

### Sacremoses Truecase Help

Source: https://pypi.org/project/sacremoses/0.1.1

Shows the help information for the sacremoses truecase command, detailing its parameters.

```bash
$ sacremoses truecase --help
```

--------------------------------

### Sacremoses Detokenizer Help

Source: https://pypi.org/project/sacremoses

Displays the help message for the sacremoses detokenize command, listing available options for detokenization.

```bash
$ sacremoses detokenize --help
Usage: sacremoses detokenize [OPTIONS]

Options:
  -x, --xml-unescape  Unescape special characters for XML.
  -h, --help          Show this message and exit.
```

--------------------------------

### Tokenize and Detokenize Text

Source: https://pypi.org/project/sacremoses/0.1.1

Demonstrates basic tokenization and detokenization of English text using MosesTokenizer and MosesDetokenizer. Handles punctuation, symbols, and contractions.

```python
from sacremoses import MosesTokenizer, MosesDetokenizer
>>> mt = MosesTokenizer(lang='en')
>>> text = 'This, is a sentence with weird\xbb symbols\u2026 appearing everywhere\xbf'
>>> expected_tokenized = 'This , is a sentence with weird \xbb symbols \u2026 appearing everywhere \xbf'
>>> tokenized_text = mt.tokenize(text, return_str=True)
>>> tokenized_text == expected_tokenized
True
```

```python
>>> mt, md = MosesTokenizer(lang='en'), MosesDetokenizer(lang='en')
>>> sent = "This ain't funny. It's actually hillarious, yet double Ls. | [] < > [ ] & You're gonna shake it off? Don't?"
>>> expected_tokens = ['This', 'ain', '&apos;t', 'funny', '.', 'It', '&apos;s', 'actually', 'hillarious', ',', 'yet', 'double', 'Ls', '.', '&#124;', '&#91;', '&#93;', '&lt;', '&gt;', '&#91;', '&#93;', '&amp;', 'You', '&apos;re', 'gonna', 'shake', 'it', 'off', '?', 'Don', '&apos;t', '?']
>>> expected_detokens = "This ain't funny. It's actually hillarious, yet double Ls. | [] < > [] & You're gonna shake it off? Don't?"
>>> mt.tokenize(sent) == expected_tokens
True
>>> md.detokenize(tokens) == expected_detokens
True
```

--------------------------------

### Sacremoses Truecase Help

Source: https://pypi.org/project/sacremoses

Displays the help message for the sacremoses truecase command, detailing options for truecasing text, including model file management and ASR mode.

```bash
$ sacremoses truecase --help
Usage: sacremoses truecase [OPTIONS]

Options:
  -m, --modelfile TEXT            Filename to save/load the modelfile.
                                  [required]
  -a, --is-asr                    A flag to indicate that model is for ASR.
  -p, --possibly-use-first-token  Use the first token as part of truecase
                                  training.
  -h, --help                      Show this message and exit.
```

--------------------------------

### Sacremoses Normalize Help

Source: https://pypi.org/project/sacremoses

Displays the help message for the sacremoses normalize command, detailing options for text normalization, including quote/comma normalization, number normalization, and control character removal.

```bash
$ sacremoses normalize --help
Usage: sacremoses normalize [OPTIONS]

Options:
  -q, --normalize-quote-commas  Normalize quotations and commas.
  -d, --normalize-numbers       Normalize number.
  -p, --replace-unicode-puncts  Replace unicode punctuations BEFORE
                                  normalization.
  -c, --remove-control-chars    Remove control characters AFTER normalization.
  -h, --help                    Show this message and exit.
```

--------------------------------

### Sacremoses Detruecase Help

Source: https://pypi.org/project/sacremoses/0.1.1

Displays the help message for the sacremoses detruecase command, listing its options.

```bash
$ sacremoses detruecase --help
```

--------------------------------

### Tokenize Text File

Source: https://pypi.org/project/sacremoses

Tokenizes the content of 'big.txt' using default settings and saves the output to 'big.txt.tok'. Shows progress indication.

```bash
$ sacremoses -l en -j 4 tokenize  < big.txt > big.txt.tok
100%|██████████████████████████████████| 128457/128457 [00:05<00:00, 24363.39it/s
```

--------------------------------

### Truecase Text with Sacremoses

Source: https://pypi.org/project/sacremoses/0.1.1

Applies truecasing to tokenized text using a specified model file. The output is redirected to a file, and progress is shown.

```bash
$ sacremoses -j 4 truecase -m big.model < big.txt.tok > big.txt.tok.true
100%|██████████████████████████████████| 128457/128457 [00:09<00:00, 14257.27it/s]

```

--------------------------------

### Sacremoses Detruecase Help

Source: https://pypi.org/project/sacremoses

Displays the help message for the sacremoses detruecase command, outlining options for converting truecased text back to its original casing.

```bash
$ sacremoses detruecase --help
Usage: sacremoses detruecase [OPTIONS]

Options:
  -j, --processes INTEGER  No. of processes.
  -a, --is-headline        Whether the file are headlines.
  -e, --encoding TEXT      Specify encoding of file.
  -h, --help               Show this message and exit.
```

--------------------------------

### Apply Truecase Model to Tokenized Text

Source: https://pypi.org/project/sacremoses

Applies a truecase model (specified by '-m big.model') to the tokenized text in 'big.txt.tok', saving the output to 'big.txt.tok.true'. Shows progress indication.

```bash
$ sacremoses -j 4 truecase -m big.model < big.txt.tok > big.txt.tok.true
100%|██████████████████████████████████| 128457/128457 [00:09<00:00, 14257.27it/s]
```

--------------------------------

### Tokenize Text File with Protected Patterns

Source: https://pypi.org/project/sacremoses

Tokenizes 'big.txt' while protecting specific patterns defined in 'basic-protected-patterns', saving the result to 'big.txt.tok'. Requires downloading the patterns file first.

```bash
$ wget https://raw.githubusercontent.com/moses-smt/mosesdecoder/master/scripts/tokenizer/basic-protected-patterns
$ sacremoses -l en -j 4 tokenize -p basic-protected-patterns < big.txt > big.txt.tok
100%|██████████████████████████████████| 128457/128457 [00:05<00:00, 22183.94it/s
```

--------------------------------

### Tokenize Text with Sacremoses

Source: https://pypi.org/project/sacremoses/0.1.1

Tokenizes a text file using the sacremoses tool. The output is saved to a new file. Progress is shown during execution.

```bash
$ sacremoses -l en -j 4 tokenize  < big.txt > big.txt.tok
100%|██████████████████████████████████| 128457/128457 [00:05<00:00, 24363.39it/s

```

--------------------------------

### Normalize Text File

Source: https://pypi.org/project/sacremoses

Normalizes the text in 'big.txt' and saves the output to 'big.txt.norm'. Shows progress indication.

```bash
$ sacremoses -j 4 normalize < big.txt > big.txt.norm
100%|██████████████████████████████████| 128457/128457 [00:09<00:00, 13096.23it/s]
```

--------------------------------

### Convert Truecased Text Back to Original Casing

Source: https://pypi.org/project/sacremoses

Converts truecased text in 'big.txt.tok.true' back to its original casing and saves the result to 'big.txt.tok.true.detrue'. Shows progress indication.

```bash
$ sacremoses -j 4 detruecase  < big.txt.tok.true > big.txt.tok.true.detrue
100%|█████████████████████████████████| 128457/128457 [00:04<00:00, 26945.16it/s]
```

--------------------------------

### Normalize Punctuation

Source: https://pypi.org/project/sacremoses/0.1.1

Demonstrates using MosesPunctNormalizer to normalize punctuation in a given string. This is useful for standardizing text before further processing.

```python
from sacremoses import MosesPunctNormalizer
>>> mpn = MosesPunctNormalizer()
>>> mpn.normalize('THIS EBOOK IS OTHERWISE PROVIDED TO YOU "AS-IS."')
'THIS EBOOK IS OTHERWISE PROVIDED TO YOU "AS-IS."' 
```

--------------------------------

### Detokenize Text with Sacremoses

Source: https://pypi.org/project/sacremoses/0.1.1

Detokenizes a tokenized text file using sacremoses. The result is saved to a new file, reversing the tokenization process. Progress is displayed.

```bash
$ sacremoses -l en -j 4 detokenize < big.txt.tok > big.txt.tok.detok
100%|██████████████████████████████████| 128457/128457 [00:16<00:00, 7931.26it/s]

```

--------------------------------

### Detokenize Tokenized Text

Source: https://pypi.org/project/sacremoses

Detokenizes the content of 'big.txt.tok' and saves the result to 'big.txt.tok.detok'. Shows progress indication.

```bash
$ sacremoses -l en -j 4 detokenize < big.txt.tok > big.txt.tok.detok
100%|██████████████████████████████████| 128457/128457 [00:16<00:00, 7931.26it/s]
```

--------------------------------

### Chain Sacremoses Commands for Text Processing

Source: https://pypi.org/project/sacremoses/0.1.1

Chains normalize, tokenize, and truecase commands to process a text file. The output is redirected to a specified file. This is useful for a complete text normalization pipeline.

```bash
cat big.txt | sacremoses -l en -j 4 \
    normalize -c tokenize -a truecase -a -m big.truemodel \
    > big.txt.norm.tok.true

```

--------------------------------

### Tokenize Text with Protected Patterns

Source: https://pypi.org/project/sacremoses/0.1.1

Tokenizes text using sacremoses, specifying a file with patterns to be protected during tokenization. This ensures specific sequences are not split.

```bash
$ wget https://raw.githubusercontent.com/moses-smt/mosesdecoder/master/scripts/tokenizer/basic-protected-patterns
$ sacremoses -l en -j 4 tokenize -p basic-protected-patterns < big.txt > big.txt.tok
100%|██████████████████████████████████| 128457/128457 [00:05<00:00, 22183.94it/s

```

--------------------------------

### Normalize Text with Sacremoses

Source: https://pypi.org/project/sacremoses/0.1.1

Normalizes text using sacremoses, with control characters removed after normalization. The processed text is saved to a file, and progress is displayed.

```bash
$ sacremoses -j 4 normalize < big.txt > big.txt.norm
100%|██████████████████████████████████| 128457/128457 [00:09<00:00, 13096.23it/s]

```

--------------------------------

### Detruecase Text with Sacremoses

Source: https://pypi.org/project/sacremoses/0.1.1

Removes truecasing from text using sacremoses. The operation is performed with multiple processes for efficiency, and the output is saved to a file. Progress is indicated.

```bash
$ sacremoses -j 4 detruecase  < big.txt.tok.true > big.txt.tok.true.detrue
100%|█████████████████████████████████| 128457/128457 [00:04<00:00, 26945.16it/s]

```
Tokens:
Raw