### Install Sacremoses Source: https://pypi.org/project/sacremoses/0.1.1 Install the latest version of Sacremoses using pip. ```bash pip install -U sacremoses ``` -------------------------------- ### Chaining Sacremoses Commands in a Pipeline Source: https://pypi.org/project/sacremoses This example demonstrates how to chain multiple Sacremoses commands (normalize, tokenize, truecase) in a single pipeline for batch processing text files. It shows how to use options like removing control characters, aggressive dash splitting, and loading/training a truecase model. ```bash cat big.txt | sacremoses -l en -j 4 \ normalize -c tokenize -a truecase -a -m big.truemodel \ > big.txt.norm.tok.true ``` -------------------------------- ### Train and Use Truecaser Source: https://pypi.org/project/sacremoses/0.1.1 Shows how to train a truecase model from a text file, save it, and then use it to truecase strings. Also demonstrates loading a pre-trained model and using specific options. ```python from sacremoses import MosesTruecaser, MosesTokenizer # Train a new truecaser from a 'big.txt' file. >>> mtr = MosesTruecaser() >>> mtok = MosesTokenizer(lang='en') # Save the truecase model to 'big.truecasemodel' using `save_to` >> tokenized_docs = [mtok.tokenize(line) for line in open('big.txt')] >>> mtr.train(tokenized_docs, save_to='big.truecasemodel') ``` ```python # Save the truecase model to 'big.truecasemodel' after training # (just in case you forgot to use `save_to`) >>> mtr = MosesTruecaser() >>> mtr.train('big.txt') >>> mtr.save_model('big.truecasemodel') ``` ```python # Truecase a string after training a model. >>> mtr = MosesTruecaser() >>> mtr.train('big.txt') >>> mtr.truecase("THE ADVENTURES OF SHERLOCK HOLMES") ['the', 'adventures', 'of', 'Sherlock', 'Holmes'] ``` ```python # Loads a model and truecase a string using trained model. >>> mtr = MosesTruecaser('big.truecasemodel') >>> mtr.truecase("THE ADVENTURES OF SHERLOCK HOLMES") ['the', 'adventures', 'of', 'Sherlock', 'Holmes'] >>> mtr.truecase("THE ADVENTURES OF SHERLOCK HOLMES", use_known=True) ['the', 'ADVENTURES', 'OF', 'SHERLOCK', 'HOLMES'] >>> mtr.truecase("THE ADVENTURES OF SHERLOCK HOLMES", return_str=True) 'the adventures of Sherlock Holmes' ``` -------------------------------- ### Train and Use Truecaser Source: https://pypi.org/project/sacremoses Shows how to train a truecase model from a text file and then use it to truecase strings. Supports loading pre-trained models. ```python from sacremoses import MosesTruecaser, MosesTokenizer # Train a new truecaser from a 'big.txt' file. >>> mtr = MosesTruecaser() >>> mtok = MosesTokenizer(lang='en') # Save the truecase model to 'big.truecasemodel' using `save_to` >> tokenized_docs = [mtok.tokenize(line) for line in open('big.txt')] # doctest: +SKIP >>> mtr.train(tokenized_docs, save_to='big.truecasemodel') # Save the truecase model to 'big.truecasemodel' after training # (just in case you forgot to use `save_to`) >>> mtr = MosesTruecaser() >>> mtr.train('big.txt') # doctest: +SKIP >>> mtr.save_model('big.truecasemodel') # Truecase a string after training a model. >>> mtr = MosesTruecaser() >>> mtr.train('big.txt') # doctest: +SKIP >>> mtr.truecase("THE ADVENTURES OF SHERLOCK HOLMES") ['the', 'adventures', 'of', 'Sherlock', 'Holmes'] # Loads a model and truecase a string using trained model. >>> mtr = MosesTruecaser('big.truecasemodel') >>> mtr.truecase("THE ADVENTURES OF SHERLOCK HOLMES") ['the', 'adventures', 'of', 'Sherlock', 'Holmes'] >>> mtr.truecase("THE ADVENTURES OF SHERLOCK HOLMES", use_known=True) ['the', 'ADVENTURES', 'OF', 'SHERLOCK', 'HOLMES'] >>> mtr.truecase("THE ADVENTURES OF SHERLOCK HOLMES", return_str=True) 'the adventures of Sherlock Holmes' ``` -------------------------------- ### Sacremoses Normalize Help Source: https://pypi.org/project/sacremoses/0.1.1 Shows the help information for the sacremoses normalize command, detailing its normalization options. ```bash $ sacremoses normalize --help ``` -------------------------------- ### Sacremoses CLI Help Source: https://pypi.org/project/sacremoses/0.1.1 Displays the help message for the Sacremoses command-line interface, showing available global options and commands. ```bash $ sacremoses --help Usage: sacremoses [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]... Options: -l, --language TEXT Use language specific rules when tokenizing -j, --processes INTEGER No. of processes. -e, --encoding TEXT Specify encoding of file. -q, --quiet Disable progress bar. --version Show the version and exit. -h, --help Show this message and exit. Commands: detokenize detruecase normalize tokenize train-truecase truecase ``` -------------------------------- ### Sacremoses Tokenizer Help Source: https://pypi.org/project/sacremoses Displays the help message for the sacremoses tokenize command, outlining available options for tokenization. ```bash $ sacremoses tokenize --help Usage: sacremoses tokenize [OPTIONS] Options: -a, --aggressive-dash-splits Triggers dash split rules. -x, --xml-escape Escape special characters for XML. -p, --protected-patterns TEXT Specify file with patters to be protected in tokenisation. -c, --custom-nb-prefixes TEXT Specify a custom non-breaking prefixes file, add prefixes to the default ones from the specified language. -h, --help Show this message and exit. ``` -------------------------------- ### Sacremoses Truecase Help Source: https://pypi.org/project/sacremoses/0.1.1 Shows the help information for the sacremoses truecase command, detailing its parameters. ```bash $ sacremoses truecase --help ``` -------------------------------- ### Sacremoses Detokenizer Help Source: https://pypi.org/project/sacremoses Displays the help message for the sacremoses detokenize command, listing available options for detokenization. ```bash $ sacremoses detokenize --help Usage: sacremoses detokenize [OPTIONS] Options: -x, --xml-unescape Unescape special characters for XML. -h, --help Show this message and exit. ``` -------------------------------- ### Tokenize and Detokenize Text Source: https://pypi.org/project/sacremoses/0.1.1 Demonstrates basic tokenization and detokenization of English text using MosesTokenizer and MosesDetokenizer. Handles punctuation, symbols, and contractions. ```python from sacremoses import MosesTokenizer, MosesDetokenizer >>> mt = MosesTokenizer(lang='en') >>> text = 'This, is a sentence with weird\xbb symbols\u2026 appearing everywhere\xbf' >>> expected_tokenized = 'This , is a sentence with weird \xbb symbols \u2026 appearing everywhere \xbf' >>> tokenized_text = mt.tokenize(text, return_str=True) >>> tokenized_text == expected_tokenized True ``` ```python >>> mt, md = MosesTokenizer(lang='en'), MosesDetokenizer(lang='en') >>> sent = "This ain't funny. It's actually hillarious, yet double Ls. | [] < > [ ] & You're gonna shake it off? Don't?" >>> expected_tokens = ['This', 'ain', ''t', 'funny', '.', 'It', ''s', 'actually', 'hillarious', ',', 'yet', 'double', 'Ls', '.', '|', '[', ']', '<', '>', '[', ']', '&', 'You', ''re', 'gonna', 'shake', 'it', 'off', '?', 'Don', ''t', '?'] >>> expected_detokens = "This ain't funny. It's actually hillarious, yet double Ls. | [] < > [] & You're gonna shake it off? Don't?" >>> mt.tokenize(sent) == expected_tokens True >>> md.detokenize(tokens) == expected_detokens True ``` -------------------------------- ### Sacremoses Truecase Help Source: https://pypi.org/project/sacremoses Displays the help message for the sacremoses truecase command, detailing options for truecasing text, including model file management and ASR mode. ```bash $ sacremoses truecase --help Usage: sacremoses truecase [OPTIONS] Options: -m, --modelfile TEXT Filename to save/load the modelfile. [required] -a, --is-asr A flag to indicate that model is for ASR. -p, --possibly-use-first-token Use the first token as part of truecase training. -h, --help Show this message and exit. ``` -------------------------------- ### Sacremoses Normalize Help Source: https://pypi.org/project/sacremoses Displays the help message for the sacremoses normalize command, detailing options for text normalization, including quote/comma normalization, number normalization, and control character removal. ```bash $ sacremoses normalize --help Usage: sacremoses normalize [OPTIONS] Options: -q, --normalize-quote-commas Normalize quotations and commas. -d, --normalize-numbers Normalize number. -p, --replace-unicode-puncts Replace unicode punctuations BEFORE normalization. -c, --remove-control-chars Remove control characters AFTER normalization. -h, --help Show this message and exit. ``` -------------------------------- ### Sacremoses Detruecase Help Source: https://pypi.org/project/sacremoses/0.1.1 Displays the help message for the sacremoses detruecase command, listing its options. ```bash $ sacremoses detruecase --help ``` -------------------------------- ### Tokenize Text File Source: https://pypi.org/project/sacremoses Tokenizes the content of 'big.txt' using default settings and saves the output to 'big.txt.tok'. Shows progress indication. ```bash $ sacremoses -l en -j 4 tokenize < big.txt > big.txt.tok 100%|██████████████████████████████████| 128457/128457 [00:05<00:00, 24363.39it/s ``` -------------------------------- ### Truecase Text with Sacremoses Source: https://pypi.org/project/sacremoses/0.1.1 Applies truecasing to tokenized text using a specified model file. The output is redirected to a file, and progress is shown. ```bash $ sacremoses -j 4 truecase -m big.model < big.txt.tok > big.txt.tok.true 100%|██████████████████████████████████| 128457/128457 [00:09<00:00, 14257.27it/s] ``` -------------------------------- ### Sacremoses Detruecase Help Source: https://pypi.org/project/sacremoses Displays the help message for the sacremoses detruecase command, outlining options for converting truecased text back to its original casing. ```bash $ sacremoses detruecase --help Usage: sacremoses detruecase [OPTIONS] Options: -j, --processes INTEGER No. of processes. -a, --is-headline Whether the file are headlines. -e, --encoding TEXT Specify encoding of file. -h, --help Show this message and exit. ``` -------------------------------- ### Apply Truecase Model to Tokenized Text Source: https://pypi.org/project/sacremoses Applies a truecase model (specified by '-m big.model') to the tokenized text in 'big.txt.tok', saving the output to 'big.txt.tok.true'. Shows progress indication. ```bash $ sacremoses -j 4 truecase -m big.model < big.txt.tok > big.txt.tok.true 100%|██████████████████████████████████| 128457/128457 [00:09<00:00, 14257.27it/s] ``` -------------------------------- ### Tokenize Text File with Protected Patterns Source: https://pypi.org/project/sacremoses Tokenizes 'big.txt' while protecting specific patterns defined in 'basic-protected-patterns', saving the result to 'big.txt.tok'. Requires downloading the patterns file first. ```bash $ wget https://raw.githubusercontent.com/moses-smt/mosesdecoder/master/scripts/tokenizer/basic-protected-patterns $ sacremoses -l en -j 4 tokenize -p basic-protected-patterns < big.txt > big.txt.tok 100%|██████████████████████████████████| 128457/128457 [00:05<00:00, 22183.94it/s ``` -------------------------------- ### Tokenize Text with Sacremoses Source: https://pypi.org/project/sacremoses/0.1.1 Tokenizes a text file using the sacremoses tool. The output is saved to a new file. Progress is shown during execution. ```bash $ sacremoses -l en -j 4 tokenize < big.txt > big.txt.tok 100%|██████████████████████████████████| 128457/128457 [00:05<00:00, 24363.39it/s ``` -------------------------------- ### Normalize Text File Source: https://pypi.org/project/sacremoses Normalizes the text in 'big.txt' and saves the output to 'big.txt.norm'. Shows progress indication. ```bash $ sacremoses -j 4 normalize < big.txt > big.txt.norm 100%|██████████████████████████████████| 128457/128457 [00:09<00:00, 13096.23it/s] ``` -------------------------------- ### Convert Truecased Text Back to Original Casing Source: https://pypi.org/project/sacremoses Converts truecased text in 'big.txt.tok.true' back to its original casing and saves the result to 'big.txt.tok.true.detrue'. Shows progress indication. ```bash $ sacremoses -j 4 detruecase < big.txt.tok.true > big.txt.tok.true.detrue 100%|█████████████████████████████████| 128457/128457 [00:04<00:00, 26945.16it/s] ``` -------------------------------- ### Normalize Punctuation Source: https://pypi.org/project/sacremoses/0.1.1 Demonstrates using MosesPunctNormalizer to normalize punctuation in a given string. This is useful for standardizing text before further processing. ```python from sacremoses import MosesPunctNormalizer >>> mpn = MosesPunctNormalizer() >>> mpn.normalize('THIS EBOOK IS OTHERWISE PROVIDED TO YOU "AS-IS."') 'THIS EBOOK IS OTHERWISE PROVIDED TO YOU "AS-IS."' ``` -------------------------------- ### Detokenize Text with Sacremoses Source: https://pypi.org/project/sacremoses/0.1.1 Detokenizes a tokenized text file using sacremoses. The result is saved to a new file, reversing the tokenization process. Progress is displayed. ```bash $ sacremoses -l en -j 4 detokenize < big.txt.tok > big.txt.tok.detok 100%|██████████████████████████████████| 128457/128457 [00:16<00:00, 7931.26it/s] ``` -------------------------------- ### Detokenize Tokenized Text Source: https://pypi.org/project/sacremoses Detokenizes the content of 'big.txt.tok' and saves the result to 'big.txt.tok.detok'. Shows progress indication. ```bash $ sacremoses -l en -j 4 detokenize < big.txt.tok > big.txt.tok.detok 100%|██████████████████████████████████| 128457/128457 [00:16<00:00, 7931.26it/s] ``` -------------------------------- ### Chain Sacremoses Commands for Text Processing Source: https://pypi.org/project/sacremoses/0.1.1 Chains normalize, tokenize, and truecase commands to process a text file. The output is redirected to a specified file. This is useful for a complete text normalization pipeline. ```bash cat big.txt | sacremoses -l en -j 4 \ normalize -c tokenize -a truecase -a -m big.truemodel \ > big.txt.norm.tok.true ``` -------------------------------- ### Tokenize Text with Protected Patterns Source: https://pypi.org/project/sacremoses/0.1.1 Tokenizes text using sacremoses, specifying a file with patterns to be protected during tokenization. This ensures specific sequences are not split. ```bash $ wget https://raw.githubusercontent.com/moses-smt/mosesdecoder/master/scripts/tokenizer/basic-protected-patterns $ sacremoses -l en -j 4 tokenize -p basic-protected-patterns < big.txt > big.txt.tok 100%|██████████████████████████████████| 128457/128457 [00:05<00:00, 22183.94it/s ``` -------------------------------- ### Normalize Text with Sacremoses Source: https://pypi.org/project/sacremoses/0.1.1 Normalizes text using sacremoses, with control characters removed after normalization. The processed text is saved to a file, and progress is displayed. ```bash $ sacremoses -j 4 normalize < big.txt > big.txt.norm 100%|██████████████████████████████████| 128457/128457 [00:09<00:00, 13096.23it/s] ``` -------------------------------- ### Detruecase Text with Sacremoses Source: https://pypi.org/project/sacremoses/0.1.1 Removes truecasing from text using sacremoses. The operation is performed with multiple processes for efficiency, and the output is saved to a file. Progress is indicated. ```bash $ sacremoses -j 4 detruecase < big.txt.tok.true > big.txt.tok.true.detrue 100%|█████████████████████████████████| 128457/128457 [00:04<00:00, 26945.16it/s] ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.