YaLM 100B

https://github.com/yandex/yalm-100b
Admin
YaLM 100B is a GPT-like neural network with 100 billion parameters, designed for generating and...
Tokens:8,342
Snippets:35
Trust Score:8.5
License:Apache-2.0
Update:1 year ago
Show doc for...
Tokens:
Raw
### Install Megatron-LM Library

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/README.md

Command to install the Megatron-LM library using pip. This requires Python 3 development headers to be installed.

```shell
pip install megatron-lm
```

--------------------------------

### Install Libraries and LSH

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/tools/openwebtext/README.md

Installs necessary Python packages for data processing and clones the LSH repository for duplicate detection. Ensure you have pip and git installed.

```shell
pip install ftfy langdetect numpy torch pandas nltk sentencepiece boto3 tqdm regex bs4 newspaper3k htmlmin tldextract 
git clone https://github.com/mattilyra/LSH
cd LSH
python setup.py install
```

--------------------------------

### Conditional Generation Input Example

Source: https://github.com/yandex/yalm-100b/blob/main/README.md

Example JSON Lines file format for providing input prompts to the conditional text generation scripts. Each line is a JSON object containing the prompt.

```json
{
  "prompt": "Write a short story about a robot learning to love."
}
{
  "prompt": "Explain the concept of quantum entanglement in simple terms."
}
```

--------------------------------

### BERT Pretraining Script

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/README.md

Executes single GPU 345M parameter BERT pretraining. It details arguments for model layers, hidden size, attention heads, sequence length, learning rate, training iterations, batch size, data splitting, and mixed precision. Logging, saving, and evaluation intervals are configurable. The script is intended for debugging distributed training setups.

```bash
examples/pretrain_bert.sh
```

```python
CHECKPOINT_PATH=checkpoints/bert_345m
VOCAB_FILE=bert-vocab.txt
DATA_PATH=my-bert_text_sentence

BERT_ARGS="--num-layers 24 \
           --hidden-size 1024 \
           --num-attention-heads 16 \
           --seq-length 512 \
           --max-position-embeddings 512 \
           --lr 0.0001 \
           --train-iters 2000000 \
           --min-lr 0.00001 \
           --lr-decay-iters 990000 \
           --warmup 0.01 \
           --batch-size 8 \
           --vocab-file $VOCAB_FILE \
           --split 949,50,1 \
           --fp16"

OUTPUT_ARGS="--log-interval 10 \
             --save-interval 500 \
             --eval-interval 100 \
             --eval-iters 10 \
             --checkpoint-activations"

python pretrain_bert.py \
       $BERT_ARGS \
       $OUTPUT_ARGS \
       --save $CHECKPOINT_PATH \
       --load $CHECKPOINT_PATH \
       --data-path $DATA_PATH
```

--------------------------------

### Download Model Checkpoint

Source: https://github.com/yandex/yalm-100b/blob/main/README.md

Script to download the YaLM 100B model weights and vocabulary. It specifies the default download location and mentions an alternative method using Hugging Face.

```bash
#!/bin/bash

# Run download.sh to download model weights and vocabulary.
# By default, weights will be downloaded to ./yalm100b_checkpoint/weights/, 
# and vocabulary will be downloaded to ./yalm100b_checkpoint/vocab/.
# Alternatively, clone the HF repo and pull the checkpoint.
./download/download.sh
```

--------------------------------

### Distributed GPT-2 Pretraining with PyTorch

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/README.md

Shows distributed GPT-2 pretraining using PyTorch's distributed launcher. It sets up environment variables, defines model/data paths, and includes parameters for model parallelism and DDP implementation. Requires PyTorch and specific GPT-2 model files.

```python
WORLD_SIZE=8
MP_SIZE=2

DISTRIBUTED_ARGS=<same as those directly above>

CHECKPOINT_PATH=checkpoints/gpt2_345m
VOCAB_FILE=gpt2-vocab.json
MERGE_FILE=gpt2-merges.txt
DATA_PATH=my-gpt2_text_document
GPT2_ARGS=<same as those in GPT-2 pretraining above>
OUTPUT_ARGS=<same as those in BERT pretraining above>

python -m torch.distributed.launch $DISTRIBUTED_ARGS ./pretrain_gpt2.py \
                $GPT2_ARGS \
                $OUTPUT_ARGS \
                --save $CHECKPOINT_PATH \
                --load $CHECKPOINT_PATH \
                --data-path $DATA_PATH \
                --model-parallel-size $MP_SIZE \
                --DDP-impl torch
```

--------------------------------

### Build Docker Image from Source

Source: https://github.com/yandex/yalm-100b/blob/main/README.md

Script to build the YaLM 100B Docker image from its source files. This is an alternative to pulling a pre-built image.

```bash
#!/bin/bash

# Build the Docker image from source using the Dockerfile.
docker/build.sh
```

--------------------------------

### Finetune BERT for RACE Dataset Evaluation

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/README.md

Finetunes a BERT model for evaluation on the RACE dataset, which consists of reading comprehension tasks. The script requires specifying training and validation data directories containing .txt files.

```python
TRAIN_DATA="data/RACE/train/middle"
VALID_DATA="data/RACE/dev/middle \
            data/RACE/dev/high"
VOCAB_FILE=bert-vocab.txt
PRETRAINED_CHECKPOINT=checkpoints/bert_345m
CHECKPOINT_PATH=checkpoints/bert_345m_race
COMMON_TASK_ARGS="--num-layers 24 \
                  --hidden-size 1024 \
                  --num-attention-heads 16 \
                  --seq-length 512 \
                  --max-position-embeddings 512 \
                  --fp16 \
                  --vocab-file $VOCAB_FILE"

COMMON_TASK_ARGS_EXT="--train-data $TRAIN_DATA \
                      --valid-data $VALID_DATA \
                      --pretrained-checkpoint $PRETRAINED_CHECKPOINT \
                      --checkpoint-activations \
                      --save-interval 10000 \
                      --save $CHECKPOINT_PATH \
                      --log-interval 100 \
                      --eval-interval 1000 \
                      --eval-iters 10 \
                      --weight-decay 1.0e-1"

python tasks/main.py \
       --task RACE \
       $COMMON_TASK_ARGS \
       $COMMON_TASK_ARGS_EXT \
       --tokenizer-type BertWordPieceLowerCase \
       --epochs 3 \
       --batch-size 4 \
       --lr 1.0e-5 \
       --warmup 0.06
```

--------------------------------

### Distributed BERT Pretraining with PyTorch

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/README.md

Demonstrates distributed BERT pretraining using PyTorch's distributed launcher. It configures environment variables for multinode training, specifies model and data paths, and sets parameters like model-parallel-size and DDP-impl. Requires PyTorch and specific BERT model files.

```python
WORLD_SIZE=8
MP_SIZE=2

DISTRIBUTED_ARGS="--nproc_per_node $WORLD_SIZE \
                  --nnodes 1 \
                  --node_rank 0 \
                  --master_addr localhost \
                  --master_port 6000"

CHECKPOINT_PATH=checkpoints/bert_345m
VOCAB_FILE=bert-vocab.txt
DATA_PATH=my-bert_text_sentence
BERT_ARGS=<same as those in BERT pretraining above>
OUTPUT_ARGS=<same as those in BERT pretraining above>

python -m torch.distributed.launch $DISTRIBUTED_ARGS ./pretrain_bert.py \
                $BERT_ARGS \
                $OUTPUT_ARGS \
                --save $CHECKPOINT_PATH \
                --load $CHECKPOINT_PATH \
                --data-path $DATA_PATH \
                --model-parallel-size $MP_SIZE \
                --DDP-impl torch
```

--------------------------------

### Download NGC Model Checkpoints

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/README.md

This command downloads pre-trained model checkpoints from the NVIDIA GPU Cloud (NGC) Registry. It requires specifying a destination directory and the model name and version. Ensure the NGC CLI is set up before execution.

```shell
ngc registry model download-version --dest <output_base_directory> nvidia/<model_name>:<version>
```

--------------------------------

### Data Preprocessing Script Usage

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/README.md

The `preprocess_data.py` script converts raw text data into a binary format suitable for training. It supports different dataset implementations (mmap, cached, lazy) and tokenizer types. Key arguments include input/output paths, vocabulary files, and dataset implementation.

```python
python tools/preprocess_data.py \
       --input my-corpus.json \
       --output-prefix my-bert \
       --vocab bert-vocab.txt \
       --dataset-impl mmap \
       --tokenizer-type BertWordPieceLowerCase \
       --split-sentences
```

```python
python tools/preprocess_data.py \
       --input my-corpus.json \
       --output-prefix my-gpt2 \
       --vocab gpt2-vocab.json \
       --dataset-impl mmap \
       --tokenizer-type GPT2BPETokenizer \
       --merge-file gpt2-merges.txt \
       --append-eod
```

--------------------------------

### OpenWebText Library for GPT-2 Webtext Data

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/README.md

This snippet refers to the use of the OpenWebText library for collecting GPT-2 Webtext data. It points to the library's GitHub repository for details on downloading URLs, which are then filtered, cleaned, and deduplicated.

```bash
# Clone the OpenWebText repository
git clone https://github.com/eukaryote31/openwebtext.git

# Follow instructions in the tools/openwebtext directory for downloading and processing
```

--------------------------------

### GPT-2 Pretraining Script

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/README.md

Executes single GPU 345M parameter GPT-2 pretraining, similar to BERT but with BPE tokenization, support for longer sequences, and cosine learning rate decay. It specifies arguments for model architecture, training parameters, vocabulary and merge files, and data paths. This script is also primarily for debugging distributed training.

```bash
examples/pretrain_gpt2.sh
```

```python
CHECKPOINT_PATH=checkpoints/gpt2_345m
VOCAB_FILE=gpt2-vocab.json
MERGE_FILE=gpt2-merges.txt
DATA_PATH=my-gpt2_text_document

GPT2_ARGS="--num-layers 24 \
           --hidden-size 1024 \
           --num-attention-heads 16 \
           --seq-length 1024 \
           --max-position-embeddings 1024 \
           --batch-size 4 \
           --lr 0.00015 \
           --train-iters 500000 \
           --lr-decay-iters 320000 \
           --lr-decay-style cosine \
           --vocab-file $VOCAB_FILE \
           --merge-file $MERGE_FILE \
           --warmup .01 \
           --fp16"

OUTPUT_ARGS=&#60;same as those in BERT pretraining&#62;

python pretrain_gpt2.py \
       $GPT2_ARGS \
       $OUTPUT_ARGS \
       --save $CHECKPOINT_PATH \
       --load $CHECKPOINT_PATH \
       --data-path $DATA_PATH
```

--------------------------------

### Preprocess Data for ICT Pretraining

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/README.md

This script preprocesses a corpus into indexed datasets, splitting text into sentences and tokenizing them using a specified vocabulary. It's a crucial first step for preparing data for ICT pretraining.

```python
python preprocess_data.py \
    --input /path/to/corpus.json \
    --json-keys text title \
    --split-sentences \
    --tokenizer-type BertWordPieceLowerCase \
    --vocab-file /path/to/vocab.txt \
    --output-prefix corpus_indexed \
    --workers 5
```

--------------------------------

### Create Document Index for Similarity Search

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/README.md

This script builds an index of block embeddings using a trained ICT model. It leverages multiple GPUs and nodes for efficient indexing and prepares data for fast similarity search in the retrieval pipeline.

```python
python tools/create_doc_index.py \
    --num-layers 12 \
    --hidden-size 768 \
    --ict-head-size 128 \
    --num-attention-heads 12 \
    --batch-size 128 \
    --checkpoint-activations \
    --seq-length 256 \
    --max-position-embeddings 256 \
    --ict-load /path/to/pretrained_ict \
    --data-path /path/to/indexed_dataset \
    --titles-data-path /path/to/titles_indexed_dataset \
    --block-data-path embedded_blocks.pkl \
    --indexer-log-interval 1000 \
    --indexer-batch-size 128 \
    --vocab-file /path/to/vocab.txt \
    --num-workers 2 \
    --fp16
```

--------------------------------

### Finetune BERT for MNLI Evaluation

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/README.md

Finetunes a BERT model for evaluation on the MultiNLI (Natural Language Inference) sentence pair corpus. The script can be adapted for other datasets like Quora Question Pairs (QQP).

```python

TRAIN_DATA="data/glue_data/MNLI/train.tsv"
VALID_DATA="data/glue_data/MNLI/dev_matched.tsv \
            data/glue_data/MNLI/dev_mismatched.tsv"
PRETRAINED_CHECKPOINT=checkpoints/bert_345m
VOCAB_FILE=bert-vocab.txt
CHECKPOINT_PATH=checkpoints/bert_345m_mnli
COMMON_TASK_ARGS="<same as those in RACE Evaluation>"
COMMON_TASK_ARGS_EXT="<same as those in RACE Evaluation>"

python tasks/main.py \
       --task MNLI \
       $COMMON_TASK_ARGS \
       $COMMON_TASK_ARGS_EXT \
       --tokenizer-type BertWordPieceLowerCase \
       --epochs 5 \
       --batch-size 8 \
       --lr 5.0e-5 \
       --warmup 0.065

```

--------------------------------

### Run Docker Container

Source: https://github.com/yandex/yalm-100b/blob/main/README.md

Script to run the YaLM 100B Docker container. It allows for customization of volumes, container name, and other parameters.

```bash
#!/bin/bash

# Run the container. Volumes, name, and other parameters can be changed.
docker/run.sh
```

--------------------------------

### Find Possible Duplicates using LSH

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/tools/openwebtext/README.md

Identifies potential duplicate documents using the LSH (Locality-Sensitive Hashing) algorithm. This step is computationally intensive and cannot be sharded, typically taking 12-24 hours for large datasets like OpenWebText. It requires the cleaned data file as input and outputs a file listing possible duplicate URLs.

```python
python find_duplicates.py <input cleaned data file> <output possible duplicate urls filename>
```

--------------------------------

### Pull Docker Image

Source: https://github.com/yandex/yalm-100b/blob/main/README.md

Command to pull the pre-built YaLM 100B Docker image from Docker Hub. The image is compatible with A100 and V100 GPUs.

```bash
#!/bin/bash

# Pull the published Docker image from Docker Hub.
# Compatible with A100 and V100 GPUs.
docker/pull.sh
```

--------------------------------

### WikiExtractor.py Usage for Wikipedia Data

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/README.md

This snippet demonstrates the recommended usage of WikiExtractor.py for processing Wikipedia dumps. It highlights the --json argument for outputting data in a line-delimited JSON format, which is more manageable and compatible with downstream processing pipelines.

```bash
python WikiExtractor.py --json enwiki-latest-pages-articles.xml.bz2
```

--------------------------------

### Group Similar URLs

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/tools/openwebtext/README.md

Groups URLs based on a similarity measure (default: 0.9) defined within the `is_similar` function. For each group of similar URLs, only one is kept, and the rest are marked for removal. This script takes the file of possible duplicate URLs and outputs a file containing groups of similar URLs.

```python
python group_duplicate_urls.py <possible duplicate urls file> <output file containing similar urls>
```

--------------------------------

### Generate Text Samples with GPT-2

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/README.md

This Python script generates text samples using a GPT-2 model. It supports both unconditional generation and conditional generation based on an input file. Users can control parameters like output sequence length, temperature, top-p sampling, and the number of samples.

```python
CHECKPOINT_PATH=checkpoints/gpt2_345m
VOCAB_FILE=gpt2-vocab.json
MERGE_FILE=gpt2-merges.txt
GPT2_ARGS=&#60;same as those in <a href="#gpt-2-pretraining">GPT-2 pretraining</a> above&#62;

MAX_OUTPUT_SEQUENCE_LENGTH=1024
TEMPERATURE=1.0
TOP_P=0.9
NUMBER_OF_SAMPLES=2
OUTPUT_FILE=samples.json

python tools/generate_samples_gpt2.py \
       $GPT2_ARGS \
       --load $CHECKPOINT_PATH \
       --out-seq-length $MAX_OUTPUT_SEQUENCE_LENGTH \
       --temperature $TEMPERATURE \
       --genfile $OUTPUT_FILE \
       --num-samples $NUMBER_OF_SAMPLES \
       --top_p $TOP_P \
       --recompute

```

--------------------------------

### Conditional Text Generation with Sampling

Source: https://github.com/yandex/yalm-100b/blob/main/README.md

Shell script for conditional text generation using sampling strategies like Top-p. Users can adjust temperature or use Top-k. Input is expected in JSON Lines format.

```bash
#!/bin/bash

# Execute the script for conditional generation with a sampling strategy.
# Top-p is used by default; temperature or top-k can be adjusted.
# Input is jsonlines (e.g., examples/example_cond_input.json),
# output will be the same jsonlines with a generated text field added.
./examples/generate_conditional_sampling.sh
```

--------------------------------

### Train ICT BERT Model

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/README.md

This script trains an ICTBertModel, which uses two BERT-based encoders for query and block encoding to perform retrieval. It requires a pretrained BERT model and specifies various training hyperparameters.

```python
python pretrain_ict.py \
    --num-layers 12 \
    --num-attention-heads 12 \
    --hidden-size 768 \
    --batch-size 128 \
    --seq-length 256 \
    --max-position-embeddings 256 \
    --ict-head-size 128 \
    --train-iters 100000 \
    --checkpoint-activations \
    --bert-load /path/to/pretrained_bert \
    --load checkpoints \
    --save checkpoints \
    --data-path /path/to/indexed_dataset \
    --titles-data-path /path/to/titles_indexed_dataset \
    --vocab-file /path/to/vocab.txt \
    --lr 0.0001 \
    --num-workers 2 \
    --lr-decay-style linear \
    --weight-decay 1e-2 \
    --clip-grad 1.0 \
    --warmup .01 \
    --save-interval 3000 \
    --query-in-block-prob 0.1 \
    --fp16
```

--------------------------------

### Interactive Text Generation

Source: https://github.com/yandex/yalm-100b/blob/main/README.md

Shell script for interactive text generation using the YaLM 100B model. This is the simplest method to test the model's capabilities.

```bash
#!/bin/bash

# Execute the script for interactive generation from the command line.
# This is the simplest way to try the model.
./examples/generate_interactive.sh
```

--------------------------------

### Remove Blacklisted URLs

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/tools/openwebtext/README.md

Filters a list of deduplicated URLs by removing those present in a blacklist. This script takes the path to the deduplicated URLs file and an output filename for the cleaned URLs.

```python
python blacklist_urls.py <path to the dowloaded deduplicated URLs> <filename for clean urls. e.g. clean_urls.txt>
```

--------------------------------

### Shuffle Dataset

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/tools/openwebtext/README.md

Randomly shuffles the lines of the deduplicated dataset file to ensure a random order for training. The `shuf` command is a standard Unix utility for this purpose.

```shell
shuf <cleaned deduped data file> -o train_data.json
```

--------------------------------

### Cleanup Dataset (ftfy, English Detection)

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/tools/openwebtext/README.md

Cleans the dataset by applying 'ftfy' for text fixing, detecting English content, and removing documents with fewer than 128 tokens. This process can be sharded for parallel processing.

```python
python cleanup_dataset.py <input data file> <output cleaned data filename>
```

--------------------------------

### Merge Model Parallel Partitions

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/README.md

This bash script executes a Python utility to merge model parallel partitions. It's used to combine models trained in parallel for efficient use on single GPUs in downstream tasks. Key parameters include model type, parallel size, and checkpoint path.

```bash
MODEL_PARALLEL_SIZE=2

VOCAB_FILE=bert-vocab.txt
CHECKPOINT_PATH=checkpoints/bert_345m

WORLD_SIZE=$MODEL_PARALLEL_SIZE python tools/merge_mp_partitions.py \
        --model-type BERT \
        --model-parallel-size $MODEL_PARALLEL_SIZE \
        --tokenizer-type BertWordPieceLowerCase \
        --vocab-file $VOCAB_FILE \
        --num-layers 24 \
        --hidden-size 1024 \
        --num-attention-heads 16 \
        --seq-length 512 \
        --max-position-embeddings 512 \
        --load $CHECKPOINT_PATH

```

--------------------------------

### Preprocessing Data with WikiExtractor and Sentence Splitting

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/README.md

This section describes preprocessing steps for Wikipedia data. It mentions using the --split-sentences flag in preprocess_data.py to add newlines between sentences, which is beneficial for BERT training. It also notes that this splitting should be omitted for GPT-2 training.

```python
# For BERT training:
# python preprocess_data.py --split-sentences ...

# For GPT-2 training (do NOT split sentences):
# python preprocess_data.py ...
```

--------------------------------

### Conditional Text Generation (Greedy)

Source: https://github.com/yandex/yalm-100b/blob/main/README.md

Shell script for conditional text generation using a greedy approach. This method is suitable for few-shot learning scenarios and requires JSON Lines input.

```bash
#!/bin/bash

# Execute the script for conditional generation using a greedy approach.
# Suitable for solving problems with few-shot learning.
# Input is jsonlines (e.g., examples/example_cond_input.json),
# output will be the same jsonlines with a generated text field added.
./examples/generate_conditional_greedy.sh
```

--------------------------------

### Evaluate GPT-2 WikiText Perplexity

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/README.md

This Python script evaluates a GPT-2 model's perplexity on the WikiText-103 dataset. It requires specifying the task, validation data path, vocabulary, merge file, and checkpoint. The script handles subword tokenization for accurate perplexity computation.

```python
TASK="WIKITEXT103"

VALID_DATA=&#60;wikitext path&#62;.txt
VOCAB_FILE=gpt2-vocab.json
MERGE_FILE=gpt2-merges.txt
CHECKPOINT_PATH=checkpoints/gpt2_345m

COMMON_TASK_ARGS="--num-layers 24 \
                  --hidden-size 1024 \
                  --num-attention-heads 16 \
                  --seq-length 1024 \
                  --max-position-embeddings 1024 \
                  --fp16 \
                  --vocab-file $VOCAB_FILE"

python tasks/main.py \
       --task $TASK \
       $COMMON_TASK_ARGS \
       --valid-data $VALID_DATA \
       --tokenizer-type GPT2BPETokenizer \
       --merge-file $MERGE_FILE \
       --load $CHECKPOINT_PATH \
       --batch-size 8 \
       --checkpoint-activations \
       --log-interval 10 \
       --no-load-optim \
       --no-load-rng

```

--------------------------------

### Run LAMBADA Evaluation with GPT-2 Tokenizer

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/README.md

Computes LAMBADA cloze accuracy by predicting the last token given preceding tokens. This script utilizes a detokenized, processed version of the LAMBADA dataset and requires specific GPT-2 tokenizer files.

```python
TASK="LAMBADA"

VALID_DATA="<lambada path>".json
VOCAB_FILE=gpt2-vocab.json
MERGE_FILE=gpt2-merges.txt
CHECKPOINT_PATH=checkpoints/gpt2_345m
COMMON_TASK_ARGS="<same as those in WikiText Perplexity Evaluation>"

python tasks/main.py \
       --task $TASK \
       $COMMON_TASK_ARGS \
       --valid-data $VALID_DATA \
       --tokenizer-type GPT2BPETokenizer \
       --strict-lambada \
       --merge-file $MERGE_FILE \
       --load $CHECKPOINT_PATH \
       --batch-size 8 \
       --checkpoint-activations \
       --log-interval 10 \
       --no-load-optim \
       --no-load-rng
```

--------------------------------

### Unconditional Text Generation

Source: https://github.com/yandex/yalm-100b/blob/main/README.md

Shell script for unconditional text generation. This script does not require any specific input and outputs generated text in JSON Lines format.

```bash
#!/bin/bash

# Execute the script for unconditional generation.
# No input is used, output will be in jsonlines format.
./examples/generate_unconditional.sh
```

--------------------------------

### Remove Group Duplicates

Source: https://github.com/yandex/yalm-100b/blob/main/megatron_lm/tools/openwebtext/README.md

Removes documents identified as duplicates in the previous grouping step. It takes the file containing similar URL groups, the cleaned data file, and outputs a new file with deduplicated data.

```python
python remove_group_duplicates.py <file containing simialr documents> <cleaned data file> <outputfile containing deduplicate data>
```
Tokens:
Raw