### Dolma CLI Help

Source: https://github.com/allenai/dolma/blob/main/docs/getting-started.md

Display the available commands and options for the Dolma command-line interface.

```plain-text
$ dolma --help

usage: dolma [command] [options]

Command line interface for the DOLMa dataset processing toolkit

positional arguments:
  {dedupe,mix,tag,list,stat,tokens}
    dedupe              Deduplicate documents or paragraphs using a bloom filter.
    mix                 Mix documents from multiple streams.
    tag                 Tag documents or spans of documents using one or more taggers. For a
                        list of available taggers, run `dolma list`.
    list                List available taggers.
    stat                Analyze the distribution of attributes values in a dataset.
    tokens              Tokenize documents using the provided tokenizer.

options:
  -h, --help            show this help message and exit
  -c CONFIG, --config CONFIG
                        Path to configuration optional file

```

--------------------------------

### Install Wikipedia Extractor Dependencies

Source: https://github.com/allenai/dolma/blob/main/docs/getting-started.md

Install the necessary dependencies for the wikiextractor script.

```bash
pip install git+https://github.com/santhoshtr/wikiextractor.git requests smart_open tqdm
```

--------------------------------

### Invoke Taggers

Source: https://github.com/allenai/dolma/blob/main/docs/getting-started.md

Apply various taggers to the processed Wikipedia dataset.

```bash
dolma tag \
    --documents "wikipedia/v0/documents/*" \
    --experiment exp \
    --taggers random_number_v1 \
              cld2_en_paragraph_with_doc_score_v2 \
              ft_lang_id_en_paragraph_with_doc_score_v2 \
              char_length_with_paragraphs_v1 \
              whitespace_tokenizer_with_paragraphs_v1 \
    --processes 16
```

--------------------------------

### Config File Usage Example

Source: https://github.com/allenai/dolma/blob/main/docs/tokenize.md

Example of how to use the dolma command with a configuration file.

```bash
dolma -c config.json tokens
```