Galvatron

https://hetu-galvatron.readthedocs.io/en/latest/
Admin
Galvatron is an automatic distributed training system for Transformer models and Large Language...
Tokens:35,737
Snippets:279
Trust Score:9.6
Update:1 month ago
Show doc for...
Tokens:
Raw
### Development Environment Setup

Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/6_developer_guide/contributing_guide.md.txt

Commands to fork, clone, and install the Hetu-Galvatron repository in development mode using Conda.

```bash
# Fork and clone the repository
git clone https://github.com/your-username/Hetu-Galvatron.git
cd Hetu-Galvatron

# Set up development environment
conda create -n galvatron-dev python=3.8
conda activate galvatron-dev

# Install in development mode
pip install -r requirements.txt
pip install -e .
```

--------------------------------

### Development Setup: Clone and Install Hetu-Galvatron

Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/contributing_guide.html

Sets up a local development environment for Hetu-Galvatron. Requires conda and Python 3.8. Installs dependencies and the package in editable mode.

```bash
# Fork and clone the repository
git clone https://github.com/your-username/Hetu-Galvatron.git
cd Hetu-Galvatron

# Set up development environment
conda create -n galvatron-dev python=3.8
conda activate galvatron-dev

# Install in development mode
pip install -r requirements.txt
pip install -e .

```

--------------------------------

### Start the development server

Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/7_visualization/visualization.md.txt

Launch the local development server to run the visualizer.

```bash
npm start
```

--------------------------------

### Start the development server

Source: https://hetu-galvatron.readthedocs.io/en/latest/7_visualization/visualization.html

Starts the development server for the Galvatron Visualizer. After running this, open http://localhost:3000 to view the application.

```bash
npm start

```

--------------------------------

### Install dependencies

Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/7_visualization/visualization.md.txt

Install the required project dependencies using npm.

```bash
npm install
```

--------------------------------

### Install Apex from source

Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/2_installation/installation.md.txt

Commands to clone and install NVIDIA Apex from source, with variations based on pip version.

```shell
git clone https://github.com/NVIDIA/apex
cd apex
# if pip >= 23.1 (ref: https://pip.pypa.io/en/stable/news/#v23-1) which supports multiple `--config-settings` with the same key... 
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" ./
# otherwise
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" ./
```

--------------------------------

### Install Galvatron from PyPI

Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/2_installation/installation.md.txt

Standard installation command for the Galvatron package via PyPI.

```shell
pip install hetu-galvatron
```

--------------------------------

### Install Galvatron from source

Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/2_installation/installation.md.txt

Commands to clone the repository and install Galvatron from the local source code.

```shell
git clone https://github.com/PKU-DAIR/Hetu-Galvatron.git
cd Hetu-Galvatron
pip install .
```

--------------------------------

### Install dependencies with npm

Source: https://hetu-galvatron.readthedocs.io/en/latest/7_visualization/visualization.html

Installs the required npm dependencies for the Galvatron Visualizer project.

```bash
npm install

```

--------------------------------

### Install Galvatron with FlashAttention-2

Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/2_installation/installation.md.txt

Command to install Galvatron with FlashAttention-2 enabled via environment variable.

```shell
GALVATRON_FLASH_ATTN_INSTALL=TRUE pip install hetu-galvatron
```

--------------------------------

### Install PyTorch 2.1.0 with CUDA 11.8

Source: https://hetu-galvatron.readthedocs.io/en/latest/2_installation/installation.html

Installs PyTorch 2.1.0, torchvision 0.16.0, and torchaudio 2.1.0 using the CUDA 11.8 index URL. The exact command depends on the CUDA version in your system; this is an example for CUDA 11.8.

```bash
pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu118

```

--------------------------------

### Install PyTorch

Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/2_installation/installation.md.txt

Command to install PyTorch 2.1.0 with CUDA 11.8 support.

```shell
pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu118
```

--------------------------------

### Clone the Galvatron repository

Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/7_visualization/visualization.md.txt

Clone the repository and navigate to the visualizer directory to prepare for local installation.

```bash
git clone https://github.com/PKU-DAIR/Hetu-Galvatron.git
cd Hetu-Galvatron
git checkout galvatron-visualizer
cd galvatron-visualizer
```

--------------------------------

### Memory Profiling Configuration Structure

Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/5_search_engine_usage/search_engine_usage.md.txt

Example structure for the memory profiling configuration file, defining activation memory usage for different parallel strategies.

```json
{
            "activation": {
                "1": {othe_pp_on_last_tp1_act},
                "2": {othe_pp_on_last_tp1_act},
                ...
            }
        }
    }
}
```

--------------------------------

### Run parallelism search script

Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/3_quick_start/quick_start.md.txt

Run the search script to automatically optimize the parallelism strategy. The script runs in the background and generates search logs in files starting with 'Search'. The search concludes when the marker '========================= Galvatron Search Engine End Searching =========================' appears in the log file.

```shell
sh scripts/search_dist.sh
```

--------------------------------

### Build Documentation Locally

Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/6_developer_guide/contributing_guide.md.txt

Commands to build and view the English or Chinese documentation locally.

```bash
# English documentation
cd docs/en
make html
open _build/html/index.html

# Chinese documentation
cd docs/zh_CN
make html
open _build/html/index.html
```

--------------------------------

### Build Documentation Locally: make html for docs/en and docs/zh_CN

Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/contributing_guide.html

Builds the documentation locally for both English and Chinese versions. Requires Sphinx and the documentation dependencies. After building, opens the generated HTML index in the default browser.

```bash
# English documentation
cd docs/en
make html
open _build/html/index.html

# Chinese documentation
cd docs/zh_CN
make html
open _build/html/index.html

```

--------------------------------

### Initialize and execute strategy search in search_dist.py

Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html

This script initializes the Galvatron search engine, sets model configuration, and performs parallelism optimization.

```python
if __name__ == '__main__':
    args = initialize_galvatron(model_args, mode='search')
    config = get_mymodel_config(args, overwrite_args=True)
    path = os.path.dirname(os.path.abspath(__file__))
    print(args)
    print(config)
    # Create an instance of the strategy search engine
    search_engine = GalvatronSearchEngine(args)
    
    # Set basic information for the search engine
    search_engine.set_search_engine_info(path, model_layer_configs(config), model_name(config))
    
    # Initialize the search engine
    search_engine.initialize_search_engine()

    # Perform strategy search
    search_engine.parallelism_optimization()
```

--------------------------------

### Clone the Galvatron Visualizer repository

Source: https://hetu-galvatron.readthedocs.io/en/latest/7_visualization/visualization.html

Commands to clone the Hetu-Galvatron repository, switch to the galvatron-visualizer branch, and navigate into the galvatron-visualizer directory.

```bash
git clone https://github.com/PKU-DAIR/Hetu-Galvatron.git
cd Hetu-Galvatron
git checkout galvatron-visualizer
cd galvatron-visualizer

```

--------------------------------

### Commit Message Format

Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/6_developer_guide/contributing_guide.md.txt

Template and example for writing conventional commit messages in the project.

```text
[Modified Module]<type>(<scope>): <description>

Modified Module: Runtime, Search Engine, Profiler, Misc
Types: feat, fix, docs, style, refactor, test, chore
Example: feat(profiler): add GPU memory profiling support
```

--------------------------------

### Run Galvatron Training

Source: https://hetu-galvatron.readthedocs.io/en/latest/4_galvatron_model_usage/galvatron_model_usage.html

The basic command to initiate model training with Galvatron.

```bash
sh scripts/train_dist.sh
```

--------------------------------

### Initialize Profiling in profiler.py

Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/6_developer_guide/adding_a_new_model_in_galvatron.md.txt

This script initializes the Galvatron profiling environment, sets the launcher, and processes collected data.

```python
if __name__ == '__main__':
    # Initialize Galvatron profiling environment
    args = initialize_galvatron(model_args, mode='profile')
    
    # Load model configuration
    config = get_mymodel_config(args, overwrite_args=False)
    
    # Create profiler instance
    profiler = GalvatronProfiler(args)
    
    # Get the directory path of the current file
    path = os.path.dirname(os.path.abspath(__file__))
    
    # Set profiler launcher
    profiler.set_profiler_launcher(path, layernum_arg_names(), model_name(config))
    
    # Launch profiling scripts
    profiler.launch_profiling_scripts()
    
    # Process collected profiling data
    profiler.process_profiled_data()
```

--------------------------------

### Create and activate a Python 3.8 conda environment

Source: https://hetu-galvatron.readthedocs.io/en/latest/2_installation/installation.html

Creates a new conda environment named 'galvatron' with Python 3.8 and activates it. This is the recommended preparation step before installing dependencies.

```bash
conda create -n galvatron python=3.8
conda activate galvatron

```

--------------------------------

### get_mymodel_config: Get model configuration

Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html

Use this function to retrieve the model configuration. It builds the config from the model size metadata, applies training arguments, and prints the config on the local rank 0 process.

```python
def get_mymodel_config(args, overwrite_args=True):
    config = config_from_meta(args.model_size)
    config = set_model_config(config, args, overwrite_args)
    if hasattr(args, 'local_rank') and args.local_rank == 0:
        print(config)
    return config

```

--------------------------------

### Create and activate conda environment

Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/2_installation/installation.md.txt

Recommended steps to create a Python 3.8 virtual environment using conda.

```shell
conda create -n galvatron python=3.8
conda activate galvatron
```

--------------------------------

### construct_tensor_parallel_model function for fa tensor parallel model

Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/6_developer_guide/adding_a_new_model_in_galvatron.md.txt

Constructs the tensor parallel model for the 'fa' model format. It replaces the Transformer Layer with MyModelLayer_tp, the embedding layer with VocabParallelEmbedding, and the lm_head with ColumnParallelLinear. Use this as a template when implementing your own tensor parallel model, referring to the gpt_fa example for details.

```python
def construct_tensor_parallel_model(model, config, tp_groups_enc, sp_groups_enc):
    """
    Convert the model to a tensor parallel version
    
    Parameters:
        model: Original model instance
        config: Model configuration object, TransformerConfig
        tp_groups_enc: List of tensor parallel communication groups for each layer, List[CommGroup]
        sp_groups_enc: List of sequence parallel communication groups for each layer, List[CommGroup]
        
    Returns:
        Converted tensor parallel model
    """
    # ...
    pass
```

--------------------------------

### Profiling: GalvatronProfiler launch and process

Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html

Profiling entry point for a custom model in Galvatron. It initializes the Galvatron profiling environment, loads the model configuration, creates a GalvatronProfiler instance, sets the profiler launcher with the current file path and model-specific arguments, launches profiling scripts, and processes the collected data.

```python
if __name__ == '__main__':
    # Initialize Galvatron profiling environment
    args = initialize_galvatron(model_args, mode='profile')
    
    # Load model configuration
    config = get_mymodel_config(args, overwrite_args=False)
    
    # Create profiler instance
    profiler = GalvatronProfiler(args)
    
    # Get the directory path of the current file
    path = os.path.dirname(os.path.abspath(__file__))
    
    # Set profiler launcher
    profiler.set_profiler_launcher(path, layernum_arg_names(), model_name(config))
    
    # Launch profiling scripts
    profiler.launch_profiling_scripts()
    
    # Process collected profiling data
    profiler.process_profiled_data()

```

--------------------------------

### Random data loading: random_collate_fn and distributed_dataloader

Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html

Random data loading example for a custom model in Galvatron. The first block defines the random dataset, collate function, and attention mask builder; the second block creates the distributed train loader using Galvatron's distributed_dataloader. The dataset generates random token sequences and the collate function returns tokens, a dictionary with attention_mask and labels, and None. The trainloader uses the model's data-parallel group and the random_collate_fn.

```python
def random_get_ltor_masks_and_position_ids(data):
"""Build masks and position id for left to right model."""
    micro_batch_size, seq_length = data.size()
    att_mask_batch = 1
    attention_mask = torch.tril(torch.ones(
        (att_mask_batch, seq_length, seq_length), device=data.device)).view(
            att_mask_batch, 1, seq_length, seq_length)
    attention_mask = (attention_mask < 0.5)

    return attention_mask

def random_collate_fn(batch):
    # Stack data in the batch and return data in the corresponding format
    tokens_ = torch.stack(batch, dim=0)
    labels = tokens_[:, 1:].contiguous()
    tokens = tokens_[:, :-1].contiguous()
    args = get_args()
    if not args.use_flash_attn:
        attention_mask = random_get_ltor_masks_and_position_ids(tokens)
    else:
        attention_mask = None
    return tokens, {"attention_mask":attention_mask, "labels" : labels}, None

class DataLoaderForMyModel(Dataset):
    def __init__(self, args, device, dataset_size = 2560 * 16):
        self.vocab_size = args.vocab_size
        self.sentence_length = args.seq_length
        self.dataset_size = dataset_size
        # Randomly generate the actual length of each sample (between 1 and the maximum length)
        self.data_length = np.random.randint(1,self.sentence_length+1,(self.dataset_size,))
        self.device = device

        # Generate random input data
        self.input_ids = []
        for i in range(self.dataset_size):
            sentence = np.random.randint(0,self.vocab_size,(self.sentence_length,))
            sentence[self.data_length[i]:] = 0
            mask = np.ones((self.sentence_length,))
            mask[self.data_length[i]:] = 0

            padding_sentence = np.zeros(self.sentence_length + 1, dtype=sentence.dtype)
            padding_sentence[:self.sentence_length] = sentence
            self.input_ids.append(padding_sentence)

        self.input_ids = np.array(self.input_ids)

    def __len__(self):
        return self.dataset_size

    def __getitem__(self, idx):
        if idx >= self.dataset_size:
            raise IndexError
        input_ids = torch.LongTensor(self.input_ids[idx]).to(self.device)
        return input_ids

```

```python
trainloader = distributed_dataloader(
    dataset=DataLoaderForGPT(args, device),
    global_bsz=args.global_train_batch_size,
    shuffle=True,
    args=args,
    group = model.dp_groups_whole[0].group,
    collate_fn = random_collate_fn
)

```

--------------------------------

### Profile hardware environment with scripts/profile_hardware.sh

Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/3_quick_start/quick_start.md.txt

Profiles the hardware environment (communication bandwidth) using nccl-tests or PyTorch profiler. Requires setting NUM_NODES, NUM_GPUS_PER_NODE, MPI_PATH in scripts/profile_hardware.sh and writing host addresses into hostfile.

```shell
sh scripts/profile_hardware.sh
```
Tokens:
Raw