### Installing Picotron Development Environment (Shell)

Source: https://github.com/huggingface/picotron/blob/main/README.md

This command installs the Picotron project locally in editable mode using pip. This allows developers to make direct modifications to the source code without needing to reinstall the package. It requires Python and pip to be installed on the system.

```Shell
pip install -e .
```

--------------------------------

### Citing the Picotron Project (BibTeX)

Source: https://github.com/huggingface/picotron/blob/main/README.md

This BibTeX entry provides the standard format for citing the Picotron project in academic papers, reports, or other publications. It includes essential details such as authors, title, year, publisher, and a direct link to the GitHub repository.

```BibTeX
@misc{zhao2025picotron,
  author = {Haojun Zhao and Ferdinand Mom},
  title = {Picotron: Distributed training framework for education and research experimentation},
  year = {2025},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/huggingface/picotron}}
}
```

--------------------------------

### Creating Training Config for CPU with 3D Parallelism (Shell/Python)

Source: https://github.com/huggingface/picotron/blob/main/README.md

This command executes the create_config.py script to generate a JSON configuration file specifically for training on a CPU using 2-way Data, Tensor, and Pipeline Parallelism. It configures smaller model parameters suitable for CPU experimentation and requires the --use_cpu flag and a Hugging Face token.

```Shell
python create_config.py --out_dir tmp --exp_name llama-1B-cpu --dp 2 --tp 2 --pp 2 --pp_engine 1f1b --model_name HuggingFaceTB/SmolLM-1.7B --num_hidden_layers 5  --grad_acc_steps 2 --mbs 4 --seq_len 128 --hf_token <HF_TOKEN> --use_cpu
```

--------------------------------

### Creating Training Config with Data Parallelism (Shell/Python)

Source: https://github.com/huggingface/picotron/blob/main/README.md

This command runs the create_config.py script to generate a JSON configuration file for training a Llama-like model with 8-way Data Parallelism on GPUs. It sets parameters like output directory, experiment name, model source, layer count, gradient accumulation steps, micro batch size, sequence length, and requires a Hugging Face token for model download.

```Shell
python create_config.py --out_dir tmp --exp_name llama-1B --dp 8 --model_name HuggingFaceTB/SmolLM-1.7B --num_hidden_layers 15  --grad_acc_steps 32 --mbs 4 --seq_len 1024 --hf_token <HF_TOKEN>
```

--------------------------------

### Running Distributed Training on CPU Locally (Shell)

Source: https://github.com/huggingface/picotron/blob/main/README.md

This command initiates a distributed training process locally on the CPU using torchrun across 8 processes per node. It executes the train.py script, loading the CPU-specific configuration from the specified JSON file. This is suitable for multi-core CPU systems.

```Shell
torchrun --nproc_per_node 8 train.py --config tmp/llama-1B-cpu/config.json
```

--------------------------------

### Creating Training Config with 3D Parallelism (Shell/Python)

Source: https://github.com/huggingface/picotron/blob/main/README.md

This command executes the create_config.py script to generate a JSON configuration file for training a Llama-2-7B model using 4-way Data, 2-way Tensor, and 2-way Pipeline Parallelism (using the 1f1b engine). It sets various training hyperparameters and requires a Hugging Face token to access the model.

```Shell
python create_config.py --out_dir tmp --dp 4 --tp 2 --pp 2 --pp_engine 1f1b --exp_name llama-7B --model_name meta-llama/Llama-2-7b-hf  --grad_acc_steps 32 --mbs 4 --seq_len 1024 --hf_token <HF_TOKEN>
```

--------------------------------

### Running Distributed Training Locally on GPU (Shell)

Source: https://github.com/huggingface/picotron/blob/main/README.md

This command initiates a distributed training process locally using torchrun across 8 processes per node. It executes the train.py script, loading the training configuration from the specified JSON file. This requires a machine with 8 available GPUs and a correctly configured PyTorch distributed environment.

```Shell
torchrun --nproc_per_node 8 train.py --config tmp/llama-1B/config.json
```

--------------------------------

### Submitting Slurm Training Job (Shell/Python)

Source: https://github.com/huggingface/picotron/blob/main/README.md

This command runs the submit_slurm_jobs.py script to submit a training job to a Slurm cluster based on the configuration files found in the specified input directory. It requires a target Slurm Quality of Service (qos) and a Hugging Face token. This is intended for execution on a system with Slurm access.

```Shell
python submit_slurm_jobs.py --inp_dir tmp/llama-7B --qos high --hf_token <HF_TOKEN>
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.