### Install Dependencies for trlx Summarization Example

Source: https://github.com/carperai/trlx/blob/main/examples/summarize_rlhf/README.md

Installs necessary Python packages listed in `requirements.txt` for the summarization example, including `evaluate` and `rouge-score`.

```Bash
pip install -r requirements.txt
```

--------------------------------

### Launching PPO Training for Helpful & Harmless (GPT-J) - Console

Source: https://github.com/carperai/trlx/blob/main/docs/source/examples.rst

This command launches the PPO training script for the Helpful & Harmless task using `accelerate`. It specifies 7 processes (GPUs) and uses a zero2-bf16 configuration file. This setup assumes one GPU is reserved for the reward model.

```console
accelerate launch --num_processes 7 --config_file ../../configs/accelerate/zero2-bf16.yaml ppo_hh.py
```

--------------------------------

### Install Apex Library - Bash

Source: https://github.com/carperai/trlx/blob/main/examples/llama_nemo/README.md

Clones the Apex repository and installs it using pip with specific build options (--cpp_ext, --cuda_ext) to ensure CUDA and C++ extensions are built, disabling cache and build isolation.

```bash
git clone https://github.com/NVIDIA/apex
cd apex
# if pip >= 23.1 (ref: https://pip.pypa.io/en/stable/news/#v23-1) which supports multiple `--config-settings` with the same key...
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" ./
```

--------------------------------

### Starting Triton Server with Singularity - Console

Source: https://github.com/carperai/trlx/blob/main/docs/source/examples.rst

This command starts the Triton Inference Server using the previously built Singularity image (`tritonserver-pyt.sif`). It sets `CUDA_VISIBLE_DEVICES` to 7 to dedicate a specific GPU, binds the `model_store` directory, and runs the `tritonserver` command pointing to the model repository. The `&` runs it in the background.

```console
SINGULARITYENV_CUDA_VISIBLE_DEVICES=7 singularity run --nv --bind model_store:/model_store tritonserver-pyt.sif tritonserver --model-repository=/model_store &
```

--------------------------------

### Install NeMo Megatron (v1.17.0) - Bash

Source: https://github.com/carperai/trlx/blob/main/examples/llama_nemo/README.md

Clones the NeMo repository, checks out a specific version (d3017e4), and installs it in editable mode with all dependencies using pip.

```bash
git clone https://github.com/NVIDIA/NeMo/
cd NeMo
git checkout d3017e4
pip install -e '.[all]'
```

--------------------------------

### Start Distributed Training - Bash

Source: https://github.com/carperai/trlx/blob/main/examples/llama_nemo/README.md

Submits a batch job script (dist_train.sh) to initiate the distributed training process using a job scheduler like Slurm.

```bash
sbatch dist_train.sh
```

--------------------------------

### Launch Hyperparameter Sweep with Ray Tune (Bash)

Source: https://github.com/carperai/trlx/blob/main/README.md

Starts a Ray head node and then initiates a trlX hyperparameter sweep using a specified configuration file, Accelerate configuration, number of GPUs, and an example script.

```bash
ray start --head --port=6379
python -m trlx.sweep --config configs/sweeps/ppo_sweep.yml --accelerate_config configs/accelerate/ddp.yaml --num_gpus 4 examples/ppo_sentiments.py
```

--------------------------------

### Install TRLX Library

Source: https://github.com/carperai/trlx/blob/main/examples/notebooks/trlx_simulacra.ipynb

Installs the TRLX library directly from its GitHub repository using pip. This is necessary to use the TRLX framework for reinforcement learning from human feedback.

```python
!pip install git+https://github.com/CarperAI/trlx
```

--------------------------------

### Launching PPO Training from Checkpoint (GPT-J) - Console

Source: https://github.com/carperai/trlx/blob/main/docs/source/examples.rst

This command is similar to the standard PPO launch but sets the `CONFIG_NAME` environment variable to `125M`. This is used to specify training from a predefined checkpoint configuration, still using 7 processes and the zero2-bf16 accelerate config.

```console
CONFIG_NAME=125M accelerate launch --num_processes 7 --config_file ../../configs/accelerate/zero2-bf16.yaml ppo_hh.py
```

--------------------------------

### Install trlX from source

Source: https://github.com/carperai/trlx/blob/main/docs/source/installation.rst

Commands to clone the trlX repository, navigate into the directory, install a specific version of PyTorch compatible with CUDA 11.8, and then install trlX in editable mode from the local source files.

```console
$ git clone https://github.com/CarperAI/trlx.git
$ cd trlx
$ pip install torch --extra-index-url https://download.pytorch.org/whl/cu118
$ pip install -e .
```

--------------------------------

### Install trlX with pip

Source: https://github.com/carperai/trlx/blob/main/docs/source/installation.rst

Command to install the latest version of the trlX library directly from its GitHub repository using pip. This is the simplest installation method.

```console
$ pip install -U git+https://github.com/CarperAI/trlx.git
```

--------------------------------

### Showcase Toy DSL Function Examples

Source: https://github.com/carperai/trlx/blob/main/examples/experiments/grounded_program_synthesis/README.md

Provides examples demonstrating the behavior of several functions defined in the toy list manipulation DSL, showing the input list and arguments and the resulting output list. Examples include `take`, `drop`, `reverse`, `sort_asc`, and `sort_des`.

```DSL Examples
take([1,2,3],2) -> [1,2]
drop([1,2,3],2) -> [1]
reverse([1,2,3]) -> [3,2,1]
sort_asc([10,5,6]) -> [5,6,10]
sort_des([10,5,6]) -> [10,6,5]

```

--------------------------------

### Install trlX Framework using Git and Pip

Source: https://github.com/carperai/trlx/blob/main/README.md

This snippet provides the necessary bash commands to clone the trlX repository from GitHub, navigate into the project directory, and install the framework along with a specific version of PyTorch (for CUDA 11.8) using pip in editable mode.

```Bash
git clone https://github.com/CarperAI/trlx.git
cd trlx
pip install torch --extra-index-url https://download.pytorch.org/whl/cu118
pip install -e .
```

--------------------------------

### Install Apex Dependency

Source: https://github.com/carperai/trlx/blob/main/docs/source/installation.rst

Commands to clone the NVIDIA Apex repository and install it with C++ and CUDA extensions. Apex is often used for performance optimizations, particularly with PyTorch and NeMo. The installation command includes options for newer pip versions.

```console
$ git clone https://github.com/NVIDIA/apex
$ cd apex
$ # if pip >= 23.1 (ref: https://pip.pypa.io/en/stable/news/#v23-1) which supports multiple `--config-settings` with the same key...
$ pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" ./
```

--------------------------------

### Building Singularity Image for Triton Server - Console

Source: https://github.com/carperai/trlx/blob/main/docs/source/examples.rst

This command uses Singularity to build a sandbox image (`tritonserver-pyt.sif`) from a specified NVIDIA Triton Server Docker image (`nvcr.io/nvidia/tritonserver:22.08-pyt-python-py3`). This is an alternative to using Docker directly.

```console
singularity build --sandbox tritonserver-pyt.sif docker://nvcr.io/nvidia/tritonserver:22.08-pyt-python-py3
```

--------------------------------

### Installing NeMo from Source

Source: https://github.com/carperai/trlx/blob/main/trlx/models/README.md

These commands clone the NeMo repository from GitHub, check out a specific release tag (r1.15.0), and install the library with all optional dependencies using pip.

```Shell
git clone https://github.com/NVIDIA/NeMo/
cd NeMo
git checkout r1.15.0
pip install '.[all]'

```

--------------------------------

### Launching PPO Training with Triton Reward Model - Console

Source: https://github.com/carperai/trlx/blob/main/docs/source/examples.rst

This command launches the PPO training script for the Helpful & Harmless task using `accelerate`, identical to the standard launch. However, because the `TRITON_HOST` environment variable was set, the script will connect to the Triton server at the specified address for reward calculations instead of loading the reward model locally.

```console
accelerate launch --num_processes 7 --config_file ../../configs/accelerate/zero2-bf16.yaml ppo_hh.py
```

--------------------------------

### Launch Distributed Training with Accelerate (Bash)

Source: https://github.com/carperai/trlx/blob/main/README.md

Configures Accelerate for distributed training (e.g., with DeepSpeed) and then launches a trlX example script using the configured environment.

```bash
accelerate config # choose DeepSpeed option
accelerate launch examples/simulacra.py
```

--------------------------------

### Install NeMo Dependency

Source: https://github.com/carperai/trlx/blob/main/docs/source/installation.rst

Commands to clone the NVIDIA NeMo repository, checkout a specific version (v1.17.0), and install it in editable mode with all optional dependencies. NeMo is an optional distributed backend for trlX.

```console
$ git clone https://github.com/NVIDIA/NeMo/
$ cd NeMo
$ git checkout d3017e4
$ pip install -e '.[all]'
```

--------------------------------

### Starting Triton Server with Singularity - Shell

Source: https://github.com/carperai/trlx/blob/main/examples/hh/README.md

Runs the Triton Server using the built Singularity image, binding the `model_store` directory and specifying CUDA device 7. The server is started in the background.

```Shell
SINGULARITYENV_CUDA_VISIBLE_DEVICES=7 singularity run --nv --bind model_store:/model_store tritonserver-pyt.sif tritonserver --model-repository=/model_store &
```

--------------------------------

### Launching TRLX PPO Training (Default RM) - Shell

Source: https://github.com/carperai/trlx/blob/main/examples/hh/README.md

Launches the TRLX PPO training script (`ppo_hh.py`) using `accelerate` with 7 processes and a specific configuration file. This setup assumes the 8th GPU will host the reward model instantiated by the script.

```Shell
accelerate launch --num_processes 7 --config_file ../../configs/accelerate/zero2-bf16.yaml ppo_hh.py
```

--------------------------------

### Installing NVIDIA Apex from Source

Source: https://github.com/carperai/trlx/blob/main/trlx/models/README.md

These commands clone the NVIDIA Apex repository and install it using pip with specific global options to enable C++ and CUDA extensions, fast layer normalization, and distributed Adam optimizers.

```Shell
git clone https://github.com/NVIDIA/apex/
cd apex
pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" --global-option="--fast_layer_norm" --global-option="--distributed_adam" --global-option="--deprecated_fused_adam" ./

```

--------------------------------

### Convert LLaMa Model to NeMo Format - Bash

Source: https://github.com/carperai/trlx/blob/main/examples/llama_nemo/README.md

Executes a Python script (convert_llama_to_nemo.py) to convert a LLaMa model from a specified path (NousResearch/Llama-2-7b-hf) into the NeMo format, saving the output to a folder (nemo_llama2_7b) and specifying tensor parallelism (--total_tp 4) and a name (--name 7b).

```bash
python convert_llama_to_nemo.py --model_path NousResearch/Llama-2-7b-hf --output_folder nemo_llama2_7b --total_tp 4  --name 7b
```

--------------------------------

### Launch Distributed Training with NeMo-Megatron (Bash)

Source: https://github.com/carperai/trlx/blob/main/README.md

Executes a trlX example script specifically designed for distributed training using the NeMo-Megatron framework.

```bash
python examples/nemo_ilql_sentiments.py
```

--------------------------------

### Configure Default PPO and Train with trlX (Python)

Source: https://github.com/carperai/trlx/blob/main/README.md

Initializes the default PPO configuration, sets model and tokenizer paths, adjusts sequence length, and starts training with a simple reward function based on sample length.

```python
from trlx.data.default_configs import default_ppo_config

config = default_ppo_config()
config.model.model_path = 'EleutherAI/gpt-neox-20b'
config.tokenizer.tokenizer_path = 'EleutherAI/gpt-neox-20b'
config.train.seq_length = 2048

trainer = trlx.train(config=config, reward_fn=lambda samples, **kwargs: [len(sample) for sample in samples])
```

--------------------------------

### Install Development Dependencies and Pre-commit Hooks

Source: https://github.com/carperai/trlx/blob/main/CONTRIBUTING.md

These Python commands install the project's development dependencies using pip and set up the pre-commit hooks. The hooks automatically run checks and formatting before each commit to ensure code quality and consistency.

```python
pip install -e ".[dev]"
pre-commit install
```

--------------------------------

### Launching TRLX PPO Training (Specific Config) - Shell

Source: https://github.com/carperai/trlx/blob/main/examples/hh/README.md

Launches the TRLX PPO training script (`ppo_hh.py`) using `accelerate` with 7 processes and a specific configuration file, overriding the default config with `CONFIG_NAME=125M`. Useful for training smaller models or starting from a supervised checkpoint.

```Shell
CONFIG_NAME=125M accelerate launch --num_processes 7 --config_file ../../configs/accelerate/zero2-bf16.yaml ppo_hh.py
```

--------------------------------

### Define Conda Environment - YAML

Source: https://github.com/carperai/trlx/blob/main/trlx/models/README.md

This snippet provides the full definition for a Conda environment. It specifies the environment name, installation prefix, package channels to use, and a detailed list of dependencies, including packages installed directly via Conda and those installed via Pip within the environment.

```YAML
name: nemo-113
prefix: /mnt/nvme/jobs/nemo/nemo-source
channels:
  - anaconda
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_gnu
  - bzip2=1.0.8=h7f98852_4
  - c-ares=1.18.1=h7f8727e_0
  - ca-certificates=2022.9.24=ha878542_0
  - curl=7.84.0=h5eee18b_0
  - expat=2.4.4=h295c915_0
  - gettext=0.21.1=h27087fc_0
  - git=2.34.1=pl5262hc120c5b_0
  - krb5=1.19.2=hac12032_0
  - lame=3.100=h166bdaf_1003
  - ld_impl_linux-64=2.39=hcc3a1bd_1
  - libcurl=7.84.0=h91b91d3_0
  - libedit=3.1.20210910=h7f8727e_0
  - libev=4.33=h7f8727e_1
  - libffi=3.2.1=he1b5a44_1007
  - libflac=1.4.2=h27087fc_0
  - libgcc-ng=12.2.0=h65d4601_19
  - libgomp=12.2.0=h65d4601_19
  - libnghttp2=1.46.0=hce63b2e_0
  - libnsl=2.0.0=h7f98852_0
  - libogg=1.3.4=h7f98852_1
  - libopus=1.3.1=h7f8727e_1
  - libsndfile=1.1.0=h27087fc_0
  - libsqlite=3.39.4=h753d276_0
  - libssh2=1.10.0=h8f2d780_0
  - libstdcxx-ng=12.2.0=h46fd767_19
  - libuuid=2.32.1=h7f98852_1000
  - libvorbis=1.3.7=h9c3ff4c_0
  - libzlib=1.2.12=h166bdaf_2
  - mpg123=1.30.2=h27087fc_1
  - ncurses=6.3=h27087fc_1
  - openssl=1.1.1q=h7f8727e_0
  - pcre2=10.37=he7ceb23_1
  - perl=5.26.2=h14c3975_0
  - pip=22.3.1=pyhd8ed1ab_0
  - python=3.8.2=he5300dc_7_cpython
  - readline=8.1.2=h0f457ee_0
  - sqlite=3.39.4=h4ff8645_0
  - tk=8.6.12=h1ccaba5_0
  - wheel=0.38.4=pyhd8ed1ab_0
  - xz=5.2.6=h166bdaf_0
  - zlib=1.2.12=h7f8727e_2
  - pip:
    - absl-py==1.3.0
    - aiohttp==3.8.3
    - aiosignal==1.3.1
    - alabaster==0.7.12
    - aniso8601==9.0.1
    - antlr4-python3-runtime==4.9.3
    - appdirs==1.4.4
    - asttokens==2.1.0
    - async-timeout==4.0.2
    - attrdict==2.0.1
    - attrs==22.1.0
    - audioread==3.0.0
    - babel==2.11.0
    - backcall==0.2.0
    - beautifulsoup4==4.11.1
    - black==19.10b0
    - boto3==1.26.13
    - botocore==1.29.13
    - braceexpand==0.1.7
    - cachetools==5.2.0
    - certifi==2022.9.24
    - cffi==1.15.1
    - charset-normalizer==2.1.1
    - click==8.0.2
    - colorama==0.4.6
    - commonmark==0.9.1
    - contourpy==1.0.6
    - cycler==0.11.0
    - cython==0.29.32
    - debugpy==1.6.3
    - decorator==5.1.1
    - distance==0.1.3
    - docker-pycreds==0.4.0
    - docopt==0.6.2
    - docutils==0.19
    - editdistance==0.6.1
    - einops==0.6.0
    - entrypoints==0.4
    - exceptiongroup==1.0.4
    - executing==1.2.0
    - faiss-cpu==1.7.3
    - fasttext==0.9.2
    - filelock==3.8.0
    - flask==2.2.2
    - flask-restful==0.3.9
    - fonttools==4.38.0
    - frozenlist==1.3.3
    - fsspec==2022.11.0
    - ftfy==6.1.1
    - g2p-en==2.1.0
    - gdown==4.5.3
    - gitdb==4.0.9
    - gitpython==3.1.29
    - google-auth==2.14.1
    - google-auth-oauthlib==0.4.6
    - grpcio==1.50.0
    - h5py==3.7.0
    - huggingface-hub==0.11.0
    - hydra-core==1.2.0
    - idna==3.4
    - ijson==3.1.4
    - imagesize==1.4.1
    - importlib-metadata==5.0.0
    - importlib-resources==5.10.0
    - inflect==6.0.2
    - iniconfig==1.1.1
    - ipadic==1.0.0
    - ipykernel==6.17.1
    - ipython==8.6.0
    - ipywidgets==8.0.2
    - isort==4.3.21
    - itsdangerous==2.1.2
    - jedi==0.18.1
    - jieba==0.42.1
    - jinja2==3.1.2
    - jiwer==2.5.1
    - jmespath==1.0.1
    - joblib==1.2.0
    - jupyter-client==7.4.7
    - jupyter-core==5.0.0
    - jupyterlab-widgets==3.0.3
    - kaldi-python-io==1.2.2
    - kaldiio==2.17.2
    - kiwisolver==1.4.4
    - latexcodec==2.0.1
    - levenshtein==0.20.2
    - librosa==0.9.2
    - llvmlite==0.39.1
    - loguru==0.6.0
    - lxml==4.9.1
    - markdown==3.4.1
    - markupsafe==2.1.1
    - marshmallow==3.19.0
    - matplotlib==3.6.2
    - matplotlib-inline==0.1.6
    - mecab-python3==1.0.5
    - mpmath==1.2.1
    - multidict==6.0.2
    - nest-asyncio==1.5.6
    - nltk==3.7
    - numba==0.56.4
    - numpy==1.23.4
    - nvidia-cublas-cu11==11.10.3.66
    - nvidia-cuda-nvrtc-cu11==11.7.99
    - nvidia-cuda-runtime-cu11==11.7.99
    - nvidia-cudnn-cu11==8.5.0.96
    - oauthlib==3.2.2
    - omegaconf==2.2.3
    - onnx==1.12.0
    - opencc==1.1.4
    - packaging==21.3
    - pandas==1.5.1
    - pangu==4.0.6.1
    - parameterized==0.8.1
    - parso==0.8.3
    - pathspec==0.10.2
    - pathtools==0.1.2
    - pesq==0.0.4
    - pexpect==4.8.0
    - pickleshare==0.7.5
    - pillow==9.3.0
    - pip-api==0.0.30
    - pipreqs==0.4.11
    - plac==1.3.5
    - platformdirs==2.5.4
    - pluggy==1.0.0
    - pooch==1.6.0
    - portalocker==2.6.0
    - progress==1.6
    - promise==2.3
    - prompt-toolkit==3.0.32
    - protobuf==3.20.1
    - psutil==5.9.4
    - ptyprocess==0.7.0
    - pure-eval==0.2.2
    - pyannote-core==4.5
    - pyannote-database==4.1.3
    - pyannote-metrics==3.2.1
    - pyasn1==0.4.8
    - pyasn1-modules==0.2.8
    - pybind11==2.10.1
    - pybtex==0.24.0
    - pybtex-docutils==1.0.2
    - pycparser==2.21
    - pydantic==1.10.2
    - pydeprecate==0.3.2
    - pydub==0.25.1
    - pygments==2.13.0
    - pynini==2.1.5
    - pyparsing==3.0.9
    - pypinyin==0.47.1
```

--------------------------------

### Finetuning GPT-J-6B on Alpaca Dataset (Bash)

Source: https://github.com/carperai/trlx/blob/main/examples/alpaca/README.md

This bash command executes the sft_alpaca.py script to perform supervised finetuning (SFT) of the specified model on the tatsu-lab/alpaca dataset. It requires Python and the necessary trlx dependencies to be installed.

```bash
python sft_alpaca.py --model_name EleutherAI/gpt-j-6B --dataset tatsu-lab/alpaca
```

--------------------------------

### Converting Reward Model for Triton Server - Python/Console

Source: https://github.com/carperai/trlx/blob/main/docs/source/examples.rst

This Python script command converts a specified reward model checkpoint (`Dahoas/gptj-rm-static`) based on a base model (`EleutherAI/gpt-j-6B`) into a format suitable for Triton Inference Server. It also creates the necessary configuration and directory structure (`model_store`).

```console
python to_triton.py --base_model EleutherAI/gpt-j-6B --checkpoint Dahoas/gptj-rm-static --revision 676bfd4d
```

--------------------------------

### Specifying Python Dependencies

Source: https://github.com/carperai/trlx/blob/main/examples/summarize_rlhf/requirements.txt

This snippet lists the external Python packages required for the project to run. Each line specifies a package name and a version constraint, ensuring that the installed version is at least the one specified.

```Python
evaluate>=0.4.0
nltk>=3.8.1
rouge-score>=0.1.2
```

--------------------------------

### Example of trlX Explicit Log Format (Bash)

Source: https://github.com/carperai/trlx/blob/main/README.md

Illustrates the format of log messages when explicit formatting is enabled, showing timestamp, log level, source file, line number, function, rank, and the message.

```bash
[2023-01-01 05:00:00,000] [INFO] [ppo_orchestrator.py:63:make_experience] [RANK 0] Message...
```

--------------------------------

### Starting Interactive Session on HPC Node

Source: https://github.com/carperai/trlx/blob/main/trlx/models/README.md

This SLURM command requests an interactive pseudo-terminal session on a compute node, typically used for setting up environments or running interactive jobs on an HPC cluster.

```Shell
srun --pty bash -i
```

--------------------------------

### Setting TRITON_HOST Environment Variable - Console

Source: https://github.com/carperai/trlx/blob/main/docs/source/examples.rst

This command sets the `TRITON_HOST` environment variable. This variable is used by the training script to specify the address and model name of the Triton Inference Server hosting the reward model, allowing the script to use the external server instead of instantiating the model locally.

```console
export TRITON_HOST=localhost:8001/gptj-rm-static
```

--------------------------------

### Clone Fork and Add Upstream Remote

Source: https://github.com/carperai/trlx/blob/main/CONTRIBUTING.md

This snippet provides the Bash commands to clone your fork of the trlX repository, navigate into the directory, and add the original CarperAI repository as an 'upstream' remote. This setup is standard practice for contributing via forks.

```bash
git clone https://github.com/<YOUR-USERNAME>/trlx.git
cd trlx
git remote add upstream https://github.com/CarperAI/trlx.git
```

--------------------------------

### Run PPO Training for Summarization

Source: https://github.com/carperai/trlx/blob/main/examples/summarize_rlhf/README.md

Launches the PPO training script `trlx_gptj_text_summarization.py` using `accelerate` with the specified configuration file `configs/default_accelerate_config.yaml`.

```Bash
accelerate launch --config_file configs/default_accelerate_config.yaml trlx_gptj_text_summarization.py
```

--------------------------------

### Create Directory for Reward Model Checkpoint

Source: https://github.com/carperai/trlx/blob/main/examples/summarize_rlhf/README.md

Creates the `rm_checkpoint` directory inside the `reward_model/` directory to store the downloaded reward model checkpoint.

```Bash
mkdir reward_model/rm_checkpoint
```

--------------------------------

### Configure and Launch Accelerate Training

Source: https://github.com/carperai/trlx/blob/main/docs/source/api.rst

Initialize the Accelerate configuration for your environment and then launch a trlX training script using Accelerate.

```console
$ accelerate config
$ accelerate launch examples/ppo_sentiments.py
```

--------------------------------

### Download Reward Model Checkpoint

Source: https://github.com/carperai/trlx/blob/main/examples/summarize_rlhf/README.md

Downloads the pre-trained Reward Model checkpoint file from Hugging Face and saves it as `pytorch_model.bin` in the `reward_model/rm_checkpoint` directory.

```Bash
wget https://huggingface.co/CarperAI/openai_summarize_tldr_rm_checkpoint/resolve/main/pytorch_model.bin -O reward_model/rm_checkpoint/pytorch_model.bin
```

--------------------------------

### Load Model from Hugging Face Hub

Source: https://github.com/carperai/trlx/blob/main/examples/notebooks/trlx_simulacra.ipynb

Loads the trained model directly from the specified repository on the Hugging Face Hub using `AutoModelForCausalLMWithILQLHeads` from TRLX.

```python
# Load the same model now stored on Hugging Face
from trlx.models.modeling_ilql import AutoModelForCausalLMWithILQLHeads
hf_model = AutoModelForCausalLMWithILQLHeads.from_pretrained(repo_id)
```

--------------------------------

### Train SFT Model for Summarization

Source: https://github.com/carperai/trlx/blob/main/examples/summarize_rlhf/README.md

Navigates to the `sft/` directory and runs the `train_gptj_summarize.py` script using `deepspeed` to train the Supervised Fine-Tuning (SFT) model.

```Bash
cd sft/ && deepspeed train_gptj_summarize.py
```

--------------------------------

### Launch NeMo Distributed Training with sbatch

Source: https://github.com/carperai/trlx/blob/main/docs/source/api.rst

Launch distributed training using NVIDIA NeMo via an sbatch script, suitable for cluster environments with multiple GPUs.

```console
$ sbatch examples/llama_nemo/dist_train.sh
```

--------------------------------

### Launch Accelerate Training with Config File

Source: https://github.com/carperai/trlx/blob/main/docs/source/api.rst

Launch trlX distributed training using Accelerate with a pre-defined configuration file, such as one provided in the trlX repository.

```console
$ accelerate launch --config_file configs/accelerate/zero2-bf16.yaml examples/ppo_sentiments.py
```

--------------------------------

### Train Reward Model for Summarization

Source: https://github.com/carperai/trlx/blob/main/examples/summarize_rlhf/README.md

Navigates to the `reward_model/` directory and runs the `train_reward_model_gptj.py` script using `deepspeed` to train the Reward Model.

```Bash
cd reward_model/ && deepspeed train_reward_model_gptj.py
```

--------------------------------

### Upload Model to Hugging Face Hub

Source: https://github.com/carperai/trlx/blob/main/examples/notebooks/trlx_simulacra.ipynb

Creates a new repository on the Hugging Face Hub (or uses an existing one) and uploads the locally saved model files to it.

```python
# Upload the model to <your_name>/gpt2-simulacra
from huggingface_hub import create_repo, HfApi

repo_id = create_repo("gpt2-simulacra", private=False, exist_ok=True).repo_id
HfApi().upload_folder(folder_path="gpt2-simulacra", repo_id=repo_id)
```

--------------------------------

### Train GPT-2 Model with TRLX ILQL

Source: https://github.com/carperai/trlx/blob/main/examples/notebooks/trlx_simulacra.ipynb

Initiates the training process using `trlx.train`. It specifies the base model ('gpt2'), the training configuration, the loaded prompt samples, the corresponding human feedback ratings, and evaluation prompts.

```python
model = trlx.train(
    "gpt2",
    config=config,
    samples=prompts,
    rewards=ratings,
    eval_prompts=["<|endoftext|>"] * 64
).model
```

--------------------------------

### Load and Process Simulacra Aesthetic Captions Data

Source: https://github.com/carperai/trlx/blob/main/examples/notebooks/trlx_simulacra.ipynb

Imports necessary libraries, downloads the Simulacra Aesthetic Captions SQLite database if not present, connects to the database, and queries prompt and rating data, storing them in separate lists for training.

```python
import os
import sqlite3
from urllib.request import urlretrieve

import trlx

url = "https://raw.githubusercontent.com/JD-P/simulacra-aesthetic-captions/main/sac_public_2022_06_29.sqlite"
dbpath = "sac_public_2022_06_29.sqlite"

if not os.path.exists(dbpath):
  print(f"fetching {dbpath}")
  urlretrieve(url, dbpath)

conn = sqlite3.connect(dbpath)
c = conn.cursor()
c.execute(
    "SELECT prompt, rating FROM ratings "
    "JOIN images ON images.id=ratings.iid "
    "JOIN generations ON images.gid=generations.id "
    "WHERE rating IS NOT NULL;"
)

prompts, ratings = tuple(map(list, zip(*c.fetchall())))
```

--------------------------------

### Login to Hugging Face Hub

Source: https://github.com/carperai/trlx/blob/main/examples/notebooks/trlx_simulacra.ipynb

Authenticates the user with the Hugging Face Hub using `notebook_login`. This is required before uploading models to the hub.

```python
# To upload the model to Hugging Face, login first
from huggingface_hub import notebook_login
notebook_login()
```

--------------------------------

### Launching TRLX PPO Training (External RM) - Shell

Source: https://github.com/carperai/trlx/blob/main/examples/hh/README.md

Launches the TRLX PPO training script (`ppo_hh.py`) using `accelerate` with 7 processes and a specific configuration file. This command is executed after setting the `TRITON_HOST` environment variable, indicating that an external Triton Server is hosting the reward model.

```Shell
accelerate launch --num_processes 7 --config_file ../../configs/accelerate/zero2-bf16.yaml ppo_hh.py
```

--------------------------------

### Configure TRLX ILQL Training

Source: https://github.com/carperai/trlx/blob/main/examples/notebooks/trlx_simulacra.ipynb

Imports the default Implicit Language Q-Learning (ILQL) configuration from TRLX and modifies specific training parameters, such as batch size and total steps, for the fine-tuning process.

```python
from trlx.data.default_configs import default_ilql_config
config = default_ilql_config().evolve(train=dict(batch_size=32, total_steps=300))
```

--------------------------------

### Benchmark trlX Fork Against Main Branch (Bash)

Source: https://github.com/carperai/trlx/blob/main/README.md

Runs a benchmark comparison of the current trlX fork against a specified reference branch (e.g., main) from a given repository.

```bash
python -m trlx.reference octocat/trlx-fork:fix-branch
```

--------------------------------

### Finetuning GPT-J-6B on Alpaca-Cleaned Dataset (Bash)

Source: https://github.com/carperai/trlx/blob/main/examples/alpaca/README.md

This bash command executes the sft_alpaca.py script to perform supervised finetuning (SFT) of the specified model on the yahma/alpaca-cleaned dataset. This is an alternative dataset to the original Alpaca for finetuning.

```bash
python sft_alpaca.py --model_name EleutherAI/gpt-j-6B --dataset yahma/alpaca-cleaned
```

--------------------------------

### Perform Inference with Trained Model

Source: https://github.com/carperai/trlx/blob/main/examples/notebooks/trlx_simulacra.ipynb

Loads the GPT-2 tokenizer, prepares input prompts, and uses the trained model to generate new text-to-image prompts. The generated output is then decoded and printed.

```python
# Infer the trained model
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")
output = model.generate(**tokenizer(["An astronaut riding a horse"] * 16, return_tensors="pt").to(0))
tokenizer.batch_decode(output, skip_special_tokens=True)
```

--------------------------------

### Building Triton Server Singularity Image - Shell

Source: https://github.com/carperai/trlx/blob/main/examples/hh/README.md

Builds a Singularity image (`tritonserver-pyt.sif`) from the specified NVIDIA Triton Server Docker image. This step is optional if using Docker directly.

```Shell
singularity build --sandbox tritonserver-pyt.sif docker://nvcr.io/nvidia/tritonserver:22.08-pyt-python-py3
```

--------------------------------

### Define Dataset JSON Format

Source: https://github.com/carperai/trlx/blob/main/examples/experiments/grounded_program_synthesis/README.md

Illustrates the expected JSON format for each data point in the training/testing dataset. Each entry contains an "input" field representing the prompt (input list and expected output list) and an "output" field containing the correct DSL program string that transforms the input to the output.

```JSON
    {"input": "Input: [4, -2, 0, 0, 5, 5] Output: [25, 25, 20, 0, 0, -10] Function:",
    "output": "sort_des(reverse(mul_n(sort_asc(sort_asc([4, -2, 0, 0, 5, 5])),5)))"}
```

--------------------------------

### Run Full Test Suite

Source: https://github.com/carperai/trlx/blob/main/CONTRIBUTING.md

Execute this command from the project root directory to run the entire test suite using pytest. This is crucial to ensure that your changes have not introduced any regressions or broken existing functionality.

```bash
pytest
```

--------------------------------

### Train trlX Model using a Prompt-Completion Dataset in Python

Source: https://github.com/carperai/trlx/blob/main/README.md

This Python snippet illustrates training a trlX model ('gpt2') using a dataset formatted as prompt-completion pairs. The dataset is provided as a list of lists, where each inner list contains the prompt string followed by its desired completion string.

```Python
trainer = trlx.train('gpt2', samples=[['Question: 1 + 2 Answer:', '3'], ['Question: Solve this equation: ∀n>0, s=2, sum(n ** -s). Answer:', '(pi ** 2)/ 6']])
```

--------------------------------

### Manually Run Pre-commit Hooks

Source: https://github.com/carperai/trlx/blob/main/CONTRIBUTING.md

This command allows you to manually trigger all pre-commit hooks on all files in the repository. It's useful for checking formatting and linting before attempting a commit.

```bash
pre-commit run --all-files
```

--------------------------------

### Save Trained Model Locally

Source: https://github.com/carperai/trlx/blob/main/examples/notebooks/trlx_simulacra.ipynb

Saves the fine-tuned model to a local directory named 'gpt2-simulacra'. This allows for later loading and use without retraining.

```python
# Save the model locally
model.save_pretrained("gpt2-simulacra")
```

--------------------------------

### Configuring NeMo Checkpoint Saving

Source: https://github.com/carperai/trlx/blob/main/trlx/models/README.md

This YAML configuration snippet sets up the NeMo experiment manager to create checkpoints during training and specifies the explicit directory for logs and checkpoints.

```YAML
exp_manager:
  explicit_log_dir: ilql_sentiments_logs
  create_checkpoint_callback: True
```

--------------------------------

### Save trlX Model to Hugging Face Format (Python)

Source: https://github.com/carperai/trlx/blob/main/README.md

Saves the trained trlX model to a specified local directory in a format compatible with Hugging Face pretrained language models, ready for uploading to the Hub.

```python
trainer.save_pretrained('/path/to/output/folder/')
```

--------------------------------

### Configuring NeMo Training Resumption

Source: https://github.com/carperai/trlx/blob/main/trlx/models/README.md

This YAML configuration snippet enables the NeMo experiment manager to automatically resume training from the latest checkpoint if one exists in the specified log directory.

```YAML
exp_manager:
  resume_if_exists: True
```

--------------------------------

### Running Inference with ILQL Trained NeMo Model

Source: https://github.com/carperai/trlx/blob/main/trlx/models/README.md

This command executes a Python script to perform inference using an ILQL-trained NeMo model checkpoint. It requires specifying the configuration file and the path to the checkpoint directory.

```Shell
python examples/nemo_ilql_inference.py configs/nemo_configs/megatron_20b.yaml "/path/to/ilql_sentiments_logs/checkpoints"
```

--------------------------------

### Converting Reward Model for Triton Server - Shell

Source: https://github.com/carperai/trlx/blob/main/examples/hh/README.md

Executes a Python script (`to_triton.py`) to convert a specified base model and checkpoint into a format suitable for Triton Inference Server, creating a `model_store` directory.

```Shell
python to_triton.py --base_model EleutherAI/gpt-j-6B --checkpoint Dahoas/gptj-rm-static --revision 676bfd4d
```

--------------------------------

### List of Required Python Packages

Source: https://github.com/carperai/trlx/blob/main/requirements.txt

This snippet contains the complete list of Python packages and their exact versions required for the project to run correctly. It also specifies an extra index URL where packages can be downloaded, often used for specific builds like CUDA-enabled PyTorch.

```Python Requirements
--extra-index-url https://download.pytorch.org/whl/cu118
accelerate==0.22.0
aiohttp==3.8.5
aiosignal==1.3.1
appdirs==1.4.4
async-timeout==4.0.3
attrs==23.1.0
cattrs==23.1.2
certifi==2023.7.22
charset-normalizer==3.2.0
click==8.1.7
cmake==3.25.0
datasets==2.14.4
deepspeed==0.10.1
dill==0.3.7
docker-pycreds==0.4.0
einops==0.6.1
exceptiongroup==1.1.3
filelock==3.9.0
frozenlist==1.4.0
fsspec==2023.6.0
gitdb==4.0.10
GitPython==3.1.32
grpcio==1.57.0
hjson==3.1.0
huggingface-hub==0.16.4
idna==3.4
Jinja2==3.1.2
jsonschema==4.19.0
jsonschema-specifications==2023.7.22
lit==15.0.7
markdown-it-py==3.0.0
MarkupSafe==2.1.2
mdurl==0.1.2
mpmath==1.2.1
msgpack==1.0.5
multidict==6.0.4
multiprocess==0.70.15
networkx==3.0
ninja==1.11.1
numpy==1.25.2
packaging==23.1
pandas==2.0.3
pathtools==0.1.2
peft==0.5.0
protobuf==4.24.2
psutil==5.9.5
py-cpuinfo==9.0.0
pyarrow==13.0.0
pydantic==1.10.12
Pygments==2.16.1
python-dateutil==2.8.2
python-rapidjson==1.10
pytz==2023.3
PyYAML==6.0.1
ray==2.6.3
referencing==0.30.2
regex==2023.8.8
requests==2.31.0
rich==13.5.2
rpds-py==0.9.2
safetensors==0.3.3
sentry-sdk==1.29.2
setproctitle==1.3.2
six==1.16.0
smmap==5.0.0
sympy==1.11.1
tabulate==0.9.0
tokenizers==0.13.3
torch==2.0.1+cu118
torchtyping==0.1.4
tqdm==4.66.1
transformers==4.32.0
triton==2.0.0
tritonclient==2.36.0
typeguard==4.1.3
typing_extensions==4.7.1
tzdata==2023.3
urllib3==2.0.4
wandb==0.15.8
xxhash==3.3.0
yarl==1.9.2
```

--------------------------------

### Run Specific Test File

Source: https://github.com/carperai/trlx/blob/main/CONTRIBUTING.md

If your changes have minimal scope, you can run tests for only a specific file using this command. Replace <TEST-FILE-NAME> with the name of the test file you want to execute. The -vv flag provides verbose output.

```bash
pytest -vv -k "<TEST-FILE-NAME>"
```

--------------------------------

### Generate Text using a Trained trlX Trainer in Python

Source: https://github.com/carperai/trlx/blob/main/README.md

This Python snippet demonstrates how to use the generate method of a trained trlX trainer object to produce text. It requires tokenized input (presumably from a Hugging Face tokenizer) and supports generation parameters like 'do_sample'.

```Python
trainer.generate(**tokenizer('Q: Who rules the world? A:', return_tensors='pt'), do_sample=True)
```

--------------------------------

### Train trlX Model using a Reward-Labeled Dataset in Python

Source: https://github.com/carperai/trlx/blob/main/README.md

This Python snippet shows how to train a trlX model ('EleutherAI/gpt-j-6B') directly using a pre-defined dataset consisting of input samples and their corresponding numerical reward labels. The samples and rewards are provided as separate lists.

```Python
trainer = trlx.train('EleutherAI/gpt-j-6B', samples=['dolphins', 'geese'], rewards=[1.0, 100.0])
```

--------------------------------

### Setting Triton Server Host Environment Variable - Shell

Source: https://github.com/carperai/trlx/blob/main/examples/hh/README.md

Exports the `TRITON_HOST` environment variable, setting the URL for the Triton Server. This variable is used by the training script to connect to the external reward model server.

```Shell
export TRITON_HOST=localhost:8001/gptj-rm-static
```

--------------------------------

### Extracting Pretrained NeMo Model

Source: https://github.com/carperai/trlx/blob/main/trlx/models/README.md

This command extracts the contents of a downloaded NeMo model file in .nemo format, typically containing model weights and configuration.

```Shell
tar xvf nemo_gpt20B_bf16_tp4.nemo
```