### Development Environment Setup Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/6_developer_guide/contributing_guide.md.txt Commands to fork, clone, and install the Hetu-Galvatron repository in development mode using Conda. ```bash # Fork and clone the repository git clone https://github.com/your-username/Hetu-Galvatron.git cd Hetu-Galvatron # Set up development environment conda create -n galvatron-dev python=3.8 conda activate galvatron-dev # Install in development mode pip install -r requirements.txt pip install -e . ``` -------------------------------- ### Development Setup: Clone and Install Hetu-Galvatron Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/contributing_guide.html Sets up a local development environment for Hetu-Galvatron. Requires conda and Python 3.8. Installs dependencies and the package in editable mode. ```bash # Fork and clone the repository git clone https://github.com/your-username/Hetu-Galvatron.git cd Hetu-Galvatron # Set up development environment conda create -n galvatron-dev python=3.8 conda activate galvatron-dev # Install in development mode pip install -r requirements.txt pip install -e . ``` -------------------------------- ### Start the development server Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/7_visualization/visualization.md.txt Launch the local development server to run the visualizer. ```bash npm start ``` -------------------------------- ### Start the development server Source: https://hetu-galvatron.readthedocs.io/en/latest/7_visualization/visualization.html Starts the development server for the Galvatron Visualizer. After running this, open http://localhost:3000 to view the application. ```bash npm start ``` -------------------------------- ### Install dependencies Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/7_visualization/visualization.md.txt Install the required project dependencies using npm. ```bash npm install ``` -------------------------------- ### Install Apex from source Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/2_installation/installation.md.txt Commands to clone and install NVIDIA Apex from source, with variations based on pip version. ```shell git clone https://github.com/NVIDIA/apex cd apex # if pip >= 23.1 (ref: https://pip.pypa.io/en/stable/news/#v23-1) which supports multiple `--config-settings` with the same key... pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" ./ # otherwise pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" ./ ``` -------------------------------- ### Install Galvatron from PyPI Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/2_installation/installation.md.txt Standard installation command for the Galvatron package via PyPI. ```shell pip install hetu-galvatron ``` -------------------------------- ### Install Galvatron from source Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/2_installation/installation.md.txt Commands to clone the repository and install Galvatron from the local source code. ```shell git clone https://github.com/PKU-DAIR/Hetu-Galvatron.git cd Hetu-Galvatron pip install . ``` -------------------------------- ### Install dependencies with npm Source: https://hetu-galvatron.readthedocs.io/en/latest/7_visualization/visualization.html Installs the required npm dependencies for the Galvatron Visualizer project. ```bash npm install ``` -------------------------------- ### Install Galvatron with FlashAttention-2 Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/2_installation/installation.md.txt Command to install Galvatron with FlashAttention-2 enabled via environment variable. ```shell GALVATRON_FLASH_ATTN_INSTALL=TRUE pip install hetu-galvatron ``` -------------------------------- ### Install PyTorch 2.1.0 with CUDA 11.8 Source: https://hetu-galvatron.readthedocs.io/en/latest/2_installation/installation.html Installs PyTorch 2.1.0, torchvision 0.16.0, and torchaudio 2.1.0 using the CUDA 11.8 index URL. The exact command depends on the CUDA version in your system; this is an example for CUDA 11.8. ```bash pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu118 ``` -------------------------------- ### Install PyTorch Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/2_installation/installation.md.txt Command to install PyTorch 2.1.0 with CUDA 11.8 support. ```shell pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu118 ``` -------------------------------- ### Clone the Galvatron repository Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/7_visualization/visualization.md.txt Clone the repository and navigate to the visualizer directory to prepare for local installation. ```bash git clone https://github.com/PKU-DAIR/Hetu-Galvatron.git cd Hetu-Galvatron git checkout galvatron-visualizer cd galvatron-visualizer ``` -------------------------------- ### Memory Profiling Configuration Structure Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/5_search_engine_usage/search_engine_usage.md.txt Example structure for the memory profiling configuration file, defining activation memory usage for different parallel strategies. ```json { "activation": { "1": {othe_pp_on_last_tp1_act}, "2": {othe_pp_on_last_tp1_act}, ... } } } } ``` -------------------------------- ### Commit Message Format Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/6_developer_guide/contributing_guide.md.txt Template and example for writing conventional commit messages in the project. ```text [Modified Module](): Modified Module: Runtime, Search Engine, Profiler, Misc Types: feat, fix, docs, style, refactor, test, chore Example: feat(profiler): add GPU memory profiling support ``` -------------------------------- ### Run parallelism search script Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/3_quick_start/quick_start.md.txt Run the search script to automatically optimize the parallelism strategy. The script runs in the background and generates search logs in files starting with 'Search'. The search concludes when the marker '========================= Galvatron Search Engine End Searching =========================' appears in the log file. ```shell sh scripts/search_dist.sh ``` -------------------------------- ### Create and activate a Python 3.8 conda environment Source: https://hetu-galvatron.readthedocs.io/en/latest/2_installation/installation.html Creates a new conda environment named 'galvatron' with Python 3.8 and activates it. This is the recommended preparation step before installing dependencies. ```bash conda create -n galvatron python=3.8 conda activate galvatron ``` -------------------------------- ### get_mymodel_config: Get model configuration Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html Use this function to retrieve the model configuration. It builds the config from the model size metadata, applies training arguments, and prints the config on the local rank 0 process. ```python def get_mymodel_config(args, overwrite_args=True): config = config_from_meta(args.model_size) config = set_model_config(config, args, overwrite_args) if hasattr(args, 'local_rank') and args.local_rank == 0: print(config) return config ``` -------------------------------- ### Random data loading: random_collate_fn and distributed_dataloader Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html Random data loading example for a custom model in Galvatron. The first block defines the random dataset, collate function, and attention mask builder; the second block creates the distributed train loader using Galvatron's distributed_dataloader. The dataset generates random token sequences and the collate function returns tokens, a dictionary with attention_mask and labels, and None. The trainloader uses the model's data-parallel group and the random_collate_fn. ```python def random_get_ltor_masks_and_position_ids(data): """Build masks and position id for left to right model.""" micro_batch_size, seq_length = data.size() att_mask_batch = 1 attention_mask = torch.tril(torch.ones( (att_mask_batch, seq_length, seq_length), device=data.device)).view( att_mask_batch, 1, seq_length, seq_length) attention_mask = (attention_mask < 0.5) return attention_mask def random_collate_fn(batch): # Stack data in the batch and return data in the corresponding format tokens_ = torch.stack(batch, dim=0) labels = tokens_[:, 1:].contiguous() tokens = tokens_[:, :-1].contiguous() args = get_args() if not args.use_flash_attn: attention_mask = random_get_ltor_masks_and_position_ids(tokens) else: attention_mask = None return tokens, {"attention_mask":attention_mask, "labels" : labels}, None class DataLoaderForMyModel(Dataset): def __init__(self, args, device, dataset_size = 2560 * 16): self.vocab_size = args.vocab_size self.sentence_length = args.seq_length self.dataset_size = dataset_size # Randomly generate the actual length of each sample (between 1 and the maximum length) self.data_length = np.random.randint(1,self.sentence_length+1,(self.dataset_size,)) self.device = device # Generate random input data self.input_ids = [] for i in range(self.dataset_size): sentence = np.random.randint(0,self.vocab_size,(self.sentence_length,)) sentence[self.data_length[i]:] = 0 mask = np.ones((self.sentence_length,)) mask[self.data_length[i]:] = 0 padding_sentence = np.zeros(self.sentence_length + 1, dtype=sentence.dtype) padding_sentence[:self.sentence_length] = sentence self.input_ids.append(padding_sentence) self.input_ids = np.array(self.input_ids) def __len__(self): return self.dataset_size def __getitem__(self, idx): if idx >= self.dataset_size: raise IndexError input_ids = torch.LongTensor(self.input_ids[idx]).to(self.device) return input_ids ``` ```python trainloader = distributed_dataloader( dataset=DataLoaderForGPT(args, device), global_bsz=args.global_train_batch_size, shuffle=True, args=args, group = model.dp_groups_whole[0].group, collate_fn = random_collate_fn ) ``` -------------------------------- ### construct_tensor_parallel_model function for fa tensor parallel model Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/6_developer_guide/adding_a_new_model_in_galvatron.md.txt Constructs the tensor parallel model for the 'fa' model format. It replaces the Transformer Layer with MyModelLayer_tp, the embedding layer with VocabParallelEmbedding, and the lm_head with ColumnParallelLinear. Use this as a template when implementing your own tensor parallel model, referring to the gpt_fa example for details. ```python def construct_tensor_parallel_model(model, config, tp_groups_enc, sp_groups_enc): """ Convert the model to a tensor parallel version Parameters: model: Original model instance config: Model configuration object, TransformerConfig tp_groups_enc: List of tensor parallel communication groups for each layer, List[CommGroup] sp_groups_enc: List of sequence parallel communication groups for each layer, List[CommGroup] Returns: Converted tensor parallel model """ # ... pass ``` -------------------------------- ### Build Documentation Locally Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/6_developer_guide/contributing_guide.md.txt Commands to build and view the English or Chinese documentation locally. ```bash # English documentation cd docs/en make html open _build/html/index.html # Chinese documentation cd docs/zh_CN make html open _build/html/index.html ``` -------------------------------- ### Build Documentation Locally: make html for docs/en and docs/zh_CN Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/contributing_guide.html Builds the documentation locally for both English and Chinese versions. Requires Sphinx and the documentation dependencies. After building, opens the generated HTML index in the default browser. ```bash # English documentation cd docs/en make html open _build/html/index.html # Chinese documentation cd docs/zh_CN make html open _build/html/index.html ``` -------------------------------- ### Initialize and execute strategy search in search_dist.py Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html This script initializes the Galvatron search engine, sets model configuration, and performs parallelism optimization. ```python if __name__ == '__main__': args = initialize_galvatron(model_args, mode='search') config = get_mymodel_config(args, overwrite_args=True) path = os.path.dirname(os.path.abspath(__file__)) print(args) print(config) # Create an instance of the strategy search engine search_engine = GalvatronSearchEngine(args) # Set basic information for the search engine search_engine.set_search_engine_info(path, model_layer_configs(config), model_name(config)) # Initialize the search engine search_engine.initialize_search_engine() # Perform strategy search search_engine.parallelism_optimization() ``` -------------------------------- ### Clone the Galvatron Visualizer repository Source: https://hetu-galvatron.readthedocs.io/en/latest/7_visualization/visualization.html Commands to clone the Hetu-Galvatron repository, switch to the galvatron-visualizer branch, and navigate into the galvatron-visualizer directory. ```bash git clone https://github.com/PKU-DAIR/Hetu-Galvatron.git cd Hetu-Galvatron git checkout galvatron-visualizer cd galvatron-visualizer ``` -------------------------------- ### Run Galvatron Training Source: https://hetu-galvatron.readthedocs.io/en/latest/4_galvatron_model_usage/galvatron_model_usage.html The basic command to initiate model training with Galvatron. ```bash sh scripts/train_dist.sh ``` -------------------------------- ### Initialize Profiling in profiler.py Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/6_developer_guide/adding_a_new_model_in_galvatron.md.txt This script initializes the Galvatron profiling environment, sets the launcher, and processes collected data. ```python if __name__ == '__main__': # Initialize Galvatron profiling environment args = initialize_galvatron(model_args, mode='profile') # Load model configuration config = get_mymodel_config(args, overwrite_args=False) # Create profiler instance profiler = GalvatronProfiler(args) # Get the directory path of the current file path = os.path.dirname(os.path.abspath(__file__)) # Set profiler launcher profiler.set_profiler_launcher(path, layernum_arg_names(), model_name(config)) # Launch profiling scripts profiler.launch_profiling_scripts() # Process collected profiling data profiler.process_profiled_data() ``` -------------------------------- ### Create and activate conda environment Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/2_installation/installation.md.txt Recommended steps to create a Python 3.8 virtual environment using conda. ```shell conda create -n galvatron python=3.8 conda activate galvatron ``` -------------------------------- ### Profiling: GalvatronProfiler launch and process Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html Profiling entry point for a custom model in Galvatron. It initializes the Galvatron profiling environment, loads the model configuration, creates a GalvatronProfiler instance, sets the profiler launcher with the current file path and model-specific arguments, launches profiling scripts, and processes the collected data. ```python if __name__ == '__main__': # Initialize Galvatron profiling environment args = initialize_galvatron(model_args, mode='profile') # Load model configuration config = get_mymodel_config(args, overwrite_args=False) # Create profiler instance profiler = GalvatronProfiler(args) # Get the directory path of the current file path = os.path.dirname(os.path.abspath(__file__)) # Set profiler launcher profiler.set_profiler_launcher(path, layernum_arg_names(), model_name(config)) # Launch profiling scripts profiler.launch_profiling_scripts() # Process collected profiling data profiler.process_profiled_data() ``` -------------------------------- ### Profile hardware environment with scripts/profile_hardware.sh Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/3_quick_start/quick_start.md.txt Profiles the hardware environment (communication bandwidth) using nccl-tests or PyTorch profiler. Requires setting NUM_NODES, NUM_GPUS_PER_NODE, MPI_PATH in scripts/profile_hardware.sh and writing host addresses into hostfile. ```shell sh scripts/profile_hardware.sh ``` -------------------------------- ### Run hardware profiling script Source: https://hetu-galvatron.readthedocs.io/en/latest/3_quick_start/quick_start.html Run the hardware profiling script from the galvatron/profile_hardware directory. Before running, configure the hostfile and set NUM_NODES, NUM_GPUS_PER_NODE, and MPI_PATH in scripts/profile_hardware.sh. ```bash sh scripts/profile_hardware.sh ``` -------------------------------- ### train_dist.py: Galvatron distributed training loop Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html Main training entry point for a Galvatron model. It initializes the distributed environment, builds the model, sets up the dataset, optimizer, and profiler, then runs the training loop with forward/backward, gradient clipping, optimizer steps, and checkpoint saving. The `__main__` block initializes Galvatron with `mode='train_dist'`, sets a random seed, and calls `train(args)`. ```python def train(args): # Initialize the distributed training environment local_rank = args.local_rank rank = torch.distributed.get_rank() torch.cuda.set_device(local_rank) device = torch.device("cuda", local_rank) world_size = torch.distributed.get_world_size() config = get_mymodel_config(args) model = mymodel_model_hp(config, args) # Create dataset if local_rank == 0: print("Creating Dataset...") # Set dataset-related parameters set_megatron_args_for_dataset(args, model, model.sp_groups_whole[0] if args.vocab_sp else model.tp_groups_whole[0], model.dp_groups_whole[0]) if local_rank == 0: _print_args("arguments", args) # Get data iterators train_data_iterator, valid_data_iterator, test_data_iterator = get_train_valid_test_data_iterators() # Create optimizer and learning rate scheduler optimizer, opt_param_scheduler = get_optimizer_and_param_scheduler(model, args) # Set profiler path = os.path.dirname(os.path.abspath(__file__)) profiler = GalvatronProfiler(args) profiler.set_profiler_dist(path, model_layer_configs(config), model_name(config), start_iter=0) # Record memory usage after model creation profiler.profile_memory(0, "After creating model") if local_rank == 0: print("Start training...") # Training loop for iter in range(args.iteration, args.train_iters): # Get a batch of data tokens, kwargs, loss_func = get_batch(train_data_iterator) # Record start time and memory usage profiler.profile_time_start(iter) profiler.profile_memory(iter, "Before Forward") # Prepare input data input_ids = tokens batch = [input_ids] # Forward and backward propagation loss = model.forward_backward(batch, iter, profiler, loss_func=loss_func, **kwargs) # Record memory usage after backward propagation profiler.profile_memory(iter, "After Backward") # Gradient clipping total_norm = clip_grad_norm(model, args.clip_grad) # Optimizer step optimizer.step() # Learning rate scheduler step opt_param_scheduler.step(increment=args.global_batch_size) # Record memory usage after optimizer step profiler.profile_memory(iter, "After optimizer_step") # Zero gradients optimizer.zero_grad() # Update profiler statistics profiler.post_profile_memory(iter) # Get current learning rate for param_group in optimizer.param_groups: learning_rate = param_group['lr'] # Record performance metrics for this iteration profiler.profile_time_end(iter, loss, learning_rate, total_norm) # Synchronize all processes torch.distributed.barrier() # Periodically save model checkpoints if args.save != None and (iter + 1) % args.save_interval == 0: save_llama_module(args.save, model, optimizer, opt_param_scheduler, iter + 1, args) if __name__ == '__main__': # Initialize Galvatron training environment args = initialize_galvatron(model_args, mode='train_dist') # Set random seed for reproducibility set_seed() # Start training train(args) ``` -------------------------------- ### Run training script Source: https://hetu-galvatron.readthedocs.io/en/latest/3_quick_start/quick_start.html Run the training script to train the model with Galvatron. Set the environment variables NUM_NODES, NUM_GPUS_PER_NODE, MASTER_ADDR, MASTER_PORT, and NODE_RANK before running. ```shell sh scripts/train_dist_random.sh ``` -------------------------------- ### Run training script Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/3_quick_start/quick_start.md.txt Run the training script to train the model with Galvatron. Set NUM_NODES, NUM_GPUS_PER_NODE, MASTER_ADDR, MASTER_PORT, and NODE_RANK before running. Use the --galvatron_config_path parameter to apply the searched parallel strategy. ```shell sh scripts/train_dist_random.sh ``` -------------------------------- ### First Contribution Workflow: Branch, Test, Commit, Push Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/contributing_guide.html Walks through the standard workflow for making a first contribution: create a feature branch, edit files, run tests with pytest, commit with a conventional message, and push to open a pull request. ```bash # Create a new branch for your feature git checkout -b feature/your-awesome-feature # Make your changes # ... edit files ... # Test your changes python -m pytest tests/ # Commit with clear message git add . git commit -m "[Runtime] feat: add awesome new feature" # Push and create PR git push origin feature/your-awesome-feature ``` -------------------------------- ### Load Checkpoint Arguments Source: https://hetu-galvatron.readthedocs.io/en/latest/4_galvatron_model_usage/galvatron_model_usage.html Arguments to include in the training script to load a converted checkpoint. ```bash --initialize_on_meta 1 \ --load ${OUTPUT_PATH} ``` -------------------------------- ### First Contribution Workflow Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/6_developer_guide/contributing_guide.md.txt Standard workflow for creating a feature branch, testing changes, and pushing a contribution. ```bash # Create a new branch for your feature git checkout -b feature/your-awesome-feature # Make your changes # ... edit files ... # Test your changes python -m pytest tests/ # Commit with clear message git add . git commit -m "[Runtime] feat: add awesome new feature" # Push and create PR git push origin feature/your-awesome-feature ``` -------------------------------- ### Convert Huggingface Checkpoint to Galvatron Source: https://hetu-galvatron.readthedocs.io/en/latest/4_galvatron_model_usage/galvatron_model_usage.html Execute this script to convert Huggingface model checkpoints for use with Galvatron. Ensure INPUT_PATH and OUTPUT_PATH are set correctly in the script. ```bash cd tools bash convert_{MODEL_TYPE}_h2g.sh ``` -------------------------------- ### Implement get_hybrid_parallel_configs Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html This function retrieves hybrid parallel strategies by calling the Galvatron API and requires no modifications. ```python def get_hybrid_parallel_configs(model_config, training_args): hybrid_parallel_configs = get_hybrid_parallel_configs_api(model_config, training_args, MyModelModelInfo) return hybrid_parallel_configs ``` -------------------------------- ### Construct Sequential Model with construct_sequential_model Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html Convert a model into a PipeSequential container for pipeline parallelism. ```python def construct_sequential_model(model, config): """ Convert the model to a sequential version Parameters: model: Original model instance config: Model configuration object, TransformerConfig Returns: Converted sequential model """ model_ = PipeSequential() # ... ``` -------------------------------- ### Galvatron Model Directory Structure Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html The standard directory layout for a new model in Galvatron, including configuration, scripts, and core implementation files. ```text MyModel/ ├── meta_configs/ # Directory for model configuration files │ ├── __init__.py │ ├── config_utils.py # Configuration utility functions │ ├── MyModel-{MODEL_SIZE}b.json # Model configuration │ └── ... # Other model size configurations │ ├── scripts/ # Directory for running scripts │ ├── profile.sh # Profiling script │ ├── train.sh # Training script │ └── search.sh # Parallel strategy search script │ ├── __init__.py ├── arguments.py # Argument definitions ├── dataloader.py # Data loading implementation ├── profiler.py # Profiling entry point ├── search_dist.py # Parallel strategy search entry point ├── train.py # Single-machine training entry point ├── train_dist.py # Distributed training entry point ├── train_dist_random.py # Random data training entry point │ ├── MyModelModel_checkpoint.py # Checkpoint save/load ├── MyModelModel_hybrid_parallel.py # Hybrid parallel implementation ├── MyModelModel_sequential.py # Sequential model implementation └── MyModelModel_tensor_parallel.py # Tensor parallel implementation ``` -------------------------------- ### Profile model computation time Source: https://hetu-galvatron.readthedocs.io/en/latest/4_galvatron_model_usage/galvatron_model_usage.html Execute this script to profile the forward computation time of the model. ```bash sh scripts/profile_computation.sh ``` -------------------------------- ### p2p_bandwidth JSON format Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/5_search_engine_usage/search_engine_usage.md.txt Format for the p2p_bandwidth_{num_nodes}nodes_{num_gpus}gpus_per_node.json file. Keys use the pattern pp_size_{stage_num} with bandwidth values. ```json { "pp_size_{stage_num}": {bandwidth} ... } ``` -------------------------------- ### Save Checkpoint Arguments Source: https://hetu-galvatron.readthedocs.io/en/latest/4_galvatron_model_usage/galvatron_model_usage.html Arguments to include in the training script to save checkpoints during training. ```bash --save ${OUTPUT_PATH} --save-interval ${SAVE_INTERVAL} ``` -------------------------------- ### Profile model computation and memory usage Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/3_quick_start/quick_start.md.txt Profiles model computation time and memory usage. Run from the galvatron/models/model_name directory. ```shell sh scripts/profile_computation.sh sh scripts/profile_memory.sh ``` -------------------------------- ### Implement construct_hybrid_parallel_model Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html This function constructs the hybrid parallel model by invoking the Galvatron API. ```python def construct_hybrid_parallel_model(model, model_config, training_args, hybrid_parallel_configs): # ... hp_model = construct_hybrid_parallel_model_api(...) return hp_model ``` -------------------------------- ### Galvatron JSON Configuration Format Source: https://hetu-galvatron.readthedocs.io/en/latest/4_galvatron_model_usage/galvatron_model_usage.html The structure of the JSON configuration file used for defining hybrid parallel training strategies. ```json { // Pipeline parallelism configuration "pp_deg": , "pp_division": ",,...", "pipeline_type": "pipedream_flush", // or "gpipe" "chunks": , // Tensor parallelism configuration (per-layer) "tp_sizes_enc": ",,...,", "tp_consecutive_flags": ",,...,", // Data parallelism configuration (per-layer) "dp_types_enc": ",,...,", "default_dp_type": "zero2", // or "ddp", "zero3" // Sequence parallelism configuration (per-layer) "use_sp": ",,...,", // Memory optimization configuration (per-layer) "checkpoint": ",,...,", // Global training configuration "global_bsz": , // Vocabulary parallelism configuration "vtp": , "vsp": , "embed_sdp": } ``` -------------------------------- ### Implement MyModelAttention_tp for hf format Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html Implements the attention layer using Galvatron's modified ParallelAttention from Megatron-TP. It requires tp_group, sp_group, and use_ulysses parameters. ```python class MyModelAttention_tp(nn.Module): def __init__(self, config, layer_number, tp_group=None, sp_group=None): """ Parameters: config: Model configuration object, TransformerConfig layer_number: Index number of the current layer tp_group: Tensor parallel communication group, CommGroup sp_group: Sequence parallel communication group, CommGroup """ super().__init__() # ... megatron_config = core_transformer_config_from_args(args) self.attention = ParallelAttention(megatron_config, ...) # ... def forward(self, hidden_states, attention_mask): # ... pass ``` -------------------------------- ### Retrieving Model Configuration Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/6_developer_guide/adding_a_new_model_in_galvatron.md.txt The get_mymodel_config function retrieves and sets the model configuration based on provided arguments. ```python def get_mymodel_config(args, overwrite_args=True): config = config_from_meta(args.model_size) config = set_model_config(config, args, overwrite_args) if hasattr(args, 'local_rank') and args.local_rank == 0: print(config) return config ``` -------------------------------- ### mymodel_model_hp: Build hybrid parallel model Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html Use this function to build a hybrid parallel model. It obtains hybrid parallel configurations, creates the Hugging Face model, and constructs the hybrid parallel model. The model must include the output layer. ```python def mymodel_model_hp(config, args): hybrid_parallel_configs = get_hybrid_parallel_configs(model_config=config, training_args=args) if args.local_rank == 0: print("Creating Model...") mymodel_model = MyModelModel_huggingface(config) model = construct_hybrid_parallel_model( model=mymodel_model, model_config=config, training_args=args, hybrid_parallel_configs=hybrid_parallel_configs ) return model ``` -------------------------------- ### Convert Galvatron Checkpoint to Huggingface Source: https://hetu-galvatron.readthedocs.io/en/latest/4_galvatron_model_usage/galvatron_model_usage.html Command to convert a saved distributed Galvatron checkpoint back into the Hugging Face format. ```bash cd tools bash convert_{MODEL_TYPE}_g2h.sh ``` -------------------------------- ### Define Model Information with MyModelModelInfo Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html Implement the model information class to define layer counts, shapes, data types, and module types. ```python class MyModelModelInfo(ModelInfo): def __init__(self, config, args): super(MyModelModelInfo, self).__init__() # ... self.set_layernums(layernum_list) self.set_shapes(layer_shapes_list) self.set_dtypes(layer_dtypes_list) self.set_module_types(module_types) ``` -------------------------------- ### Galvatron construct_hybrid_parallel_model_api signature Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html The API signature for constructing a hybrid parallel model, including optional parameters for FSDP wrapping, checkpointing, and weight tying. ```python def construct_hybrid_parallel_model_api( model, # Original model instance model_config, # Model configuration object training_args, # Training parameters hybrid_parallel_configs, # Hybrid parallel configuration model_info, # Model information class construct_sequential_model, # Function to construct sequential model construct_tensor_parallel_model, # Function to construct tensor parallel model wrap_block_name=None, # List of module names to wrap with FSDP wrap_checkpoint_block_name=None, # List of module names to add checkpoints wrap_other_block_name=None, # List of other module names to wrap with FSDP tied_wte_attr_names=None, # List of attribute names for weight tying layernorm_name = [], # List of layer normalization names all_block_name = None, # List of all module names load_module_func = None, # Function to load module ): # ... pass ``` -------------------------------- ### allreduce_bandwidth JSON format Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/5_search_engine_usage/search_engine_usage.md.txt Format for the allreduce_bandwidth_{num_nodes}nodes_{num_gpus}gpus_per_node.json file. Keys use the pattern allreduce_size_{group_size}_consec_[0/1] with bandwidth values. ```json { "allreduce_size_{group_size}_consec_[0/1]": {bandwidth} ... } ``` -------------------------------- ### Define Output Layer with MyModelCls_ Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html Implement the model output layer by extending nn.Module. ```python class MyModelCls_(nn.Module): def __init__(self, model): """ Parameters: model: Model instance """ super().__init__() # ... def forward(self, hidden_states, **kwargs): # ... pass ``` -------------------------------- ### sp_time JSON format Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/5_search_engine_usage/search_engine_usage.md.txt Format for the sp_time_{num_nodes}nodes_{num_gpus}gpus_per_node.json file, required when sp_space is set to tp+sp. Keys use patterns allreduce_size_{group_size}_{message_size}MB_time and all2all_size_{group_size}_{message_size}MB_time. ```json { "allreduce_size_{group_size}_{message_size}MB_time": {time}, "all2all_size_{group_size}_{message_size}MB_time": {time}, ... } ``` -------------------------------- ### Profile model memory cost Source: https://hetu-galvatron.readthedocs.io/en/latest/4_galvatron_model_usage/galvatron_model_usage.html Required for user-customized models to profile the memory cost of the model. ```bash sh scripts/profile_memory.sh ``` -------------------------------- ### computation_profiling JSON format Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/5_search_engine_usage/search_engine_usage.md.txt Format for computation_profiling_[bf16/fp16/fp32]_hidden_{hidden_size}_head_{head_num}.json files. Keys use the pattern layertype_{layer_type}_bsz{batch_size}_seq{sequence_length} with forward computation time per layer. ```json { "layertype_{layer_type}_bsz{batch_size}_seq{sequence_length}": {time}, } ``` -------------------------------- ### Define Normalization Layer with MyModelPreNorm_ Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html Implement the normalization layer before the model output layer by extending nn.Module. ```python class MyModelPreNorm_(nn.Module): def __init__(self, model): """ Parameters: model: Model instance """ super().__init__() # ... def forward(self, hidden_states, **kwargs): # ... pass ``` -------------------------------- ### construct_tensor_parallel_model for hf model format Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html Converts a model to a tensor parallel version by replacing the Transformer layer, embedding layer, and lm_head with Galvatron's tensor parallel implementations. The function takes the original model, configuration, and communication groups for tensor and sequence parallelism. The communication groups are Galvatron's CommGroup class; access underlying torch groups via tp_group.group and sp_group.group. ```python def construct_tensor_parallel_model(model, config, tp_groups_enc, sp_groups_enc): """ Convert the model to a tensor parallel version Parameters: model: Original model instance config: Model configuration object, TransformerConfig tp_groups_enc: List of tensor parallel communication groups for each layer, List[CommGroup] sp_groups_enc: List of sequence parallel communication groups for each layer, List[CommGroup] Returns: Converted tensor parallel model """ # ... pass ``` -------------------------------- ### Implement MyModelMLP_tp for hf format Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html Implements the feedforward neural network layer using Galvatron's modified ParallelMLP from Megatron-TP, incorporating the tp_group parameter. ```python class MyModelMLP_tp(nn.Module): def __init__(self, config, tp_group=None): """ Parameters: config: Model configuration object, TransformerConfig tp_group: Tensor parallel communication group, CommGroup """ super().__init__() # ... megatron_config = core_transformer_config_from_args(get_args()) self.mlp = ParallelMLP(megatron_config, tp_group = self.tp_group) # ... def forward(self, hidden_states): # ... pass ``` -------------------------------- ### Commit Message Format: [Modified Module](): Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/contributing_guide.html Defines the required commit message format, similar to Conventional Commits. The modified module must be one of Runtime, Search Engine, Profiler, or Misc, and the type must be one of feat, fix, docs, style, refactor, test, or chore. ```text [Modified Module](): Modified Module: Runtime, Search Engine, Profiler, Misc Types: feat, fix, docs, style, refactor, test, chore Example: feat(profiler): add GPU memory profiling support ``` -------------------------------- ### Implement MyModelLayer_tp for hf format Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html Defines the Transformer layer structure, including attention and MLP components. The layer_number index is required for layer identification. ```python class MyModelLayer_tp(nn.Module): def __init__(self, config, layer_number, tp_group=None, sp_group=None): """ Parameters: config: Model configuration object, TransformerConfig layer_number: Index number of the current layer tp_group: Tensor parallel communication group, CommGroup sp_group: Sequence parallel communication group, CommGroup """ super().__init__() self.attention = MyModelAttention_tp(config, layer_number, tp_group, sp_group) self.mlp = MyModelMLP_tp(config, tp_group) self.idx = layer_number def forward(self, hidden_states, attention_mask=None): # ... pass ``` -------------------------------- ### MyModelEmbeddings_ class for sequential embedding layer Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html Defines the embedding layer for the sequential model, including word and position embeddings. The model passed to __init__ must be obtained directly from transformers or flash-attn. For the hf backend, define separate embedding classes for different embedding types (e.g., vocab vs position embeddings) as shown in gpt_hf. ```python class MyModelEmbeddings_(nn.Module): def __init__(self, model): """ Parameters: model: Model instance """ super().__init__() # ... def forward(self, tokens, **kwargs): # ... pass ``` -------------------------------- ### memory_profiling JSON format Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/5_search_engine_usage/search_engine_usage.md.txt Format for memory_profiling_[bf16/fp16/fp32]_hidden_{hidden_size}_head_{head_num}.json files. Contains nested structures for layer parameters, activations per tensor parallel size, and other memory usage with and without pipeline parallelism. ```json { "layertype_{layer_type}[/_sp]": { "{sequence_length}": { "parameter_size": {layer_parameter}, "tp_activation_per_bsz_dict": { "checkpoint": {layer_ckpt_act}, "1": {layer_tp1_act}, "2": {layer_tp2_act}, ... } } ... } "other_memory_pp_off[/_sp]": { "{sequence_length}": { "model_states": { "1": {othe_pp_off_tp1_ms}, "2": {othe_pp_off_tp2_ms}, ... }, "activation": { "1": {othe_pp_off_tp1_act}, "2": {othe_pp_off_tp2_act}, ... } } } "other_memory_pp_on_first[/_sp]": { "{sequence_length}": { "model_states": { "1": {othe_pp_on_first_tp1_ms}, "2": {othe_pp_on_first_tp1_ms}, ... }, "activation": { "1": {othe_pp_on_first_tp1_act}, "2": {othe_pp_on_first_tp1_act}, ... } } } "other_memory_pp_on_last[/_sp]": { "{sequence_length}": { "model_states": { "1": {othe_pp_on_last_tp1_ms}, "2": {othe_pp_on_last_tp1_ms}, ``` -------------------------------- ### Memory Profiling JSON Structure Source: https://hetu-galvatron.readthedocs.io/en/latest/5_search_engine_usage/search_engine_usage.html The structure for memory_profiling files, capturing parameter sizes, activation memory for checkpointing and tensor parallel strategies, and memory usage for non-layer modules like embeddings. ```json { "layertype_{layer_type}[/_sp]": { "{sequence_length}": { "parameter_size": {layer_parameter}, "tp_activation_per_bsz_dict": { "checkpoint": {layer_ckpt_act}, "1": {layer_tp1_act}, "2": {layer_tp2_act}, ... } } ... } "other_memory_pp_off[/_sp]": { "{sequence_length}": { "model_states": { "1": {othe_pp_off_tp1_ms}, "2": {othe_pp_off_tp2_ms}, ... }, "activation": { "1": {othe_pp_off_tp1_act}, "2": {othe_pp_off_tp2_act}, ... } } } "other_memory_pp_on_first[/_sp]": { "{sequence_length}": { "model_states": { "1": {othe_pp_on_first_tp1_ms}, "2": {othe_pp_on_first_tp1_ms}, ... }, "activation": { "1": {othe_pp_on_first_tp1_act}, "2": {othe_pp_on_first_tp1_act}, ... } } } "other_memory_pp_on_last[/_sp]": { "{sequence_length}": { "model_states": { "1": {othe_pp_on_last_tp1_ms}, "2": {othe_pp_on_last_tp1_ms}, ... }, "activation": { "1": {othe_pp_on_last_tp1_act}, "2": {othe_pp_on_last_tp1_act}, ... } } } } ``` -------------------------------- ### Search completion marker Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/3_quick_start/quick_start.md.txt This marker in the search log file indicates that the search has concluded. No other commands need to be executed before this point. ```text ========================= Galvatron Search Engine End Searching ========================= ``` -------------------------------- ### MyModelLayers_ class for Transformer layer Source: https://hetu-galvatron.readthedocs.io/en/latest/6_developer_guide/adding_a_new_model_in_galvatron.html Defines the Transformer layer for the sequential model, including self-attention and feedforward neural network layers. For the fa backend, decide whether to add residuals and dropout based on the actual model structure. ```python class MyModelLayers_(nn.Module): def __init__(self, model, layer_idx): """ Parameters: model: Model instance layer_idx: Index number of the current layer """ super().__init__() # ... def forward(self, hidden_states, **kwargs): # ... pass ``` -------------------------------- ### overlap_coefficient JSON format Source: https://hetu-galvatron.readthedocs.io/en/latest/_sources/5_search_engine_usage/search_engine_usage.md.txt Format for the overlap_coefficient.json file. Contains a single key overlap_coe with a value representing the slowdown ratio when computation and communication overlap, typically between 1.1 and 1.3. ```json { "overlap_coe": {coe} } ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.