### Install and Run Simumax App

Source: https://github.com/moorethreads/simumax/blob/main/app/README.md

This snippet shows the commands to navigate to the application directory, execute the installation script, and start the Streamlit application. It assumes a Unix-like environment.

```shell
cd app
bash install.sh
streamlit run streamlit_app.py
```

--------------------------------

### Bash Command to Run Example Script

Source: https://github.com/moorethreads/simumax/blob/main/docs/tutorial.md

This bash command navigates to the examples directory and executes a Python script for performance analysis of Llama3 8B. The results are stored in a specific directory, which can be used for further inspection.

```bash
cd ./examples
python perf_llama3_8b_tp1_pp2.py
```

--------------------------------

### JSON Example: Intra-node PCIe 2x Bandwidth Configuration

Source: https://github.com/moorethreads/simumax/blob/main/docs/system.md

This JSON snippet demonstrates a detailed configuration for continuous two-card communication bandwidth for A100_PCIE using the intra-node PCIe 2x setup. It includes parameters for processor usage, overall bandwidth efficiency, and specific operation bandwidth efficiencies.

```json
{
            "intra_node_pcie_2x": {
                "processor_usage": 0.00,
                "bandwidth": {
                    "efficient_factor": 0.5,
                    "gbps": 30,
                    "latency_us": 10
                },
                "op": {
                    "all_reduce": {
                        "scale": 2,
                        "offset": -1,
                        "efficient_factor": 0.6965,
                        "latency_us": 15.51
                    },
                    "all_gather": {
                        "scale": 1,
                        "offset": -1,
                        "efficient_factor": 0.6866,
                        "latency_us": 24.84
                    },
                    "reduce_scatter": {
                        "scale": 1,
                        "offset": -1,
                        "efficient_factor": 0.6419,
                        "latency_us": 131.30
                    },
                    "p2p": {
                        "scale": 1,
                        "offset": 0
                    },
                    "all2all": {
                        "scale": 1,
                        "offset": -1,
                        "efficient_factor": 0.6969,
                        "latency_us": 24.07
                    }
                }

            }

        }
```

--------------------------------

### Run Full Recompute Simulation (Shell)

Source: https://github.com/moorethreads/simumax/blob/main/docs/tutorial.md

This shell command executes a Python script for a full recomputation simulation of the Llama3 model. It navigates to the examples directory and starts the performance test, directing the output to a specific directory.

```shell
python perf_llama3_70b_layer12_tp2_full_recompute.py
```

--------------------------------

### Network Configuration with PCIe (JSON Example)

Source: https://github.com/moorethreads/simumax/blob/main/docs/system.md

This JSON snippet shows network bandwidth configurations when intra-node connections utilize PCIe. It includes settings for different PCIe lane configurations (8x, 4x, 2x) and inter-node communication.

```json
{
    "intra_with_pcie": true,
    "networks": {
        "intra_node_pcie_8x": {},
        "intra_node_pcie_4x": {},
        "intra_node_pcie_2x": {},
        "inter_node": {}
    }
}
```

--------------------------------

### Network Configuration with NVLink (JSON Example)

Source: https://github.com/moorethreads/simumax/blob/main/docs/system.md

This JSON snippet outlines network bandwidth configurations when intra-node connections use high-speed NVLink. It specifies settings for low and high intra-node bandwidth, as well as inter-node communication.

```json
{
    "intra_with_pcie": false,
    "networks": {
        "low_intra_node": {},
        "high_intra_node": {},
        "inter_node": {}
    }
}
```

--------------------------------

### Install Flash-Attention (Bash)

Source: https://github.com/moorethreads/simumax/blob/main/simu_tools/megatron_scripts/README.md

Clones and installs a specific branch of the flash-attention repository. This is necessary for using the community version of flash-mla on A100 GPUs. It sets the MAX_JOBS environment variable for parallel compilation.

```Bash
git clone -b head_dim_not_equal https://github.com/defei-coder/flash-attention.git
cd flash-attention
MAX_JOBS=8 python setup.py install --verbose > install.log
```

--------------------------------

### Initialize and Run SimuMax Performance Model

Source: https://context7.com/moorethreads/simumax/llms.txt

Initializes the PerfLLM class, configures it with strategy, model, and system parameters loaded from JSON files, runs the performance estimation, and performs an analysis to get results. This is the main workflow for simulating LLM training performance.

```python
from simumax.core.config import ModelConfig, StrategyConfig, SystemConfig
from simumax.core.perf_llm import PerfLLM

# Initialize the performance model
perf_model = PerfLLM()

# Configure with model, strategy, and system parameters
perf_model.configure(
    strategy_config=StrategyConfig.init_from_config_file('configs/strategy/tp2_pp1_dp4_mbs1.json'),
    model_config=ModelConfig.init_from_config_file('configs/models/llama3-8b.json'),
    system_config=SystemConfig.init_from_config_file('configs/system/a100_pcie.json'),
)

# Run the simulation
perf_model.run_estimate()

# Analyze and get results (saves to directory and prints summary)
result = perf_model.analysis(save_path='llama3_8b_a100_pcie_bf16')
```

--------------------------------

### StrategyConfig - Training Strategy Configuration

Source: https://context7.com/moorethreads/simumax/llms.txt

Defines distributed training strategies, including parallelism dimensions (TP, PP, EP, ETP), recomputation settings (granularity, layers, attention/MLP recomputation), and optimization flags like sequence parallelism and Flash-SDP. Configurations can be loaded from JSON files or created programmatically, allowing detailed control over training setup and enabling calculation of derived properties like data parallel size and global batch size.

```python
from simumax.core.config import StrategyConfig

# Method 1: Load from JSON file
strategy = StrategyConfig.init_from_config_file('configs/strategy/tp2_pp1_dp4_mbs1.json')

# Method 2: Create programmatically
strategy = StrategyConfig(
    seq_len=4096,
    micro_batch_size=1,
    micro_batch_num=8,
    dtype='bf16',
    world_size=8,
    tp_size=2,
    pp_size=1,
    ep_size=1,
    etp_size=1,
    enable_sequence_parallel=True,
    zero_state=1,
    enable_recompute=True,
    recompute_granularity='selective_recompute',  # or 'full_block'
    recompute_layer_num=16,
    attn_recompute=True,
    mlp_recompute=True,
    use_flash_sdp=True,
    mem_factor=0.94,
)

# Access derived properties
print(f"Data parallel size: {strategy.dp_size}")  # Computed: world_size / (tp_size * pp_size)
print(f"Global batch size: {strategy.global_batch_size}")  # Computed: mbs * mbc * dp_size
print(f"Parallelism summary: {strategy.parallelism}")
```

--------------------------------

### Run Selective Recompute Simulation (Shell)

Source: https://github.com/moorethreads/simumax/blob/main/docs/tutorial.md

This shell command executes a Python script to run a simulation with selective recomputation for the DeepseekV2 model. It navigates to the examples directory and initiates the performance test, saving results in a specified directory.

```shell
cd ./examples
python perf_deepseekv2_layer4_ep4_pp2_selective_recompute.py
```

--------------------------------

### Operator Performance Metrics (JSON Example)

Source: https://github.com/moorethreads/simumax/blob/main/docs/system.md

This JSON snippet illustrates how operator performance is defined, including TFLOPS, efficient_factor, and accurate_efficient_factor for different operator types and configurations. It covers default settings for unsupported operators and specific metrics for 'fp8_group_matmul'.

```json
{
    "default": {
        "ng=40, M=616, N=3072, K=5120, dtype=bf16, out_dtype=bf16, main_grad_dtype=fp32, stage=bwd_grad_w, grad=True, accumulate=True, use_split_accumulator=True, single_output=False": 0.5196854178569805
    }
 },
 "fp8_group_matmul" : {
    "tflops": 312,
    "efficient_factor": 0.75,
    "accurate_efficient_factor": {}
 }
}
```

--------------------------------

### Memory Bandwidth Configuration (JSON Example)

Source: https://github.com/moorethreads/simumax/blob/main/docs/system.md

This JSON structure defines memory access bandwidth parameters, including default efficiency, GBps, and latency. It also specifies operator-specific fine-tuned efficiencies for memory-bound operations like permute forward/backward passes and cross-entropy (ce).

```json
{
    "bandwidth": {
        "default" : {
            "efficient_factor": 0.91,
            "gbps": 1600,
            "latency_us": 40
        },
        "permute_fwd":{
            "efficient_factor": 0.1879,
            "gbps": 1600,
            "latency_us": 40
        },
        "permute_bwd":{
            "efficient_factor": 0.1879,
            "gbps": 1600,
            "latency_us": 40
        },
        "ce":{
            "efficient_factor": 0.808,
            "gbps": 1600,
            "latency_us": 40
        }
    }
}
```

--------------------------------

### Execute System Generation Script (Shell)

Source: https://github.com/moorethreads/simumax/blob/main/simu_tools/efficency_test/README.md

This command executes the main shell script responsible for generating the system.json file. It assumes that the environment variables and configuration files (like run_params.json) have been set up correctly in the preceding steps.

```shell
bash run.sh
```

--------------------------------

### Configure and Run Operator Efficiency Tests (Shell)

Source: https://github.com/moorethreads/simumax/blob/main/simu_tools/efficency_test/README.md

This script sets environment variables to define hardware characteristics like nominal computing power and system name, then executes Python scripts to test and combine operator efficiencies. It generates an initial system.json file. Ensure run_params.json is correctly configured.

```shell
export MAX_TFLOPS=312 # Specify the machine's nominal computing power
export SYS_NAME="A100_PCIE" # Specify the system name
export PICE_INTRA_LINK=1 # Specify whether intra-machine cards are connected via PCIE
export FC8_MODE=0 # Specify whether intra-machine communication connections are in FC8 mode
export PARAM_FILE="./run_params.json" # Test hyperparameters, specify the list of test models, mbs interval, seq_len interval, etc. If not specified, the default hyperparameters will be used.
python test_gemm_efficency.py
python test_grouped_gemm_efficency.py
python test_fa_efficency.py
python combine_efficency.py
```

--------------------------------

### Run nccl_test for Communication Performance Benchmarking (Shell)

Source: https://github.com/moorethreads/simumax/blob/main/simu_tools/efficency_test/README.md

This script executes various nccl_test performance benchmarks for different communication operators like all_reduce, all_gather, reduce_scatter, alltoall, and sendrecv. It tests bandwidths ranging from 1MB to 8GB using bfloat16 data type and outputs the results to text files for further analysis.

```shell
end=8G
echo "run all_reduce_perf"
./build/all_reduce_perf -n 10 -b 1M -e $end -f 2 -g 8 -w 2 -d bfloat16 > perf_all_reduce.txt
echo "run all_gather_perf"
./build/all_gather_perf -n 10 -b 1M -e $end -f 2 -g 8 -w 2 -d bfloat16 > perf_all_gather.txt
echo "run reduce_scatter_perf"
./build/reduce_scatter_perf -n 10 -b 1M -e $end -f 2 -g 8 -w 2 -d bfloat16 > perf_reduce_scatter.txt
echo "run alltoall_perf"
./build/alltoall_perf -n 10 -b 1M -e $end -f 2 -g 8 -w 2 -d bfloat16 > perf_alltoall_perf.txt
echo "run sendrecv_perf"
./build/sendrecv_perf -n 1 -b 1M -e $end -f 2 -g 1 -t 1  -d bfloat16 > send_recv.txt
```

--------------------------------

### Python Performance Simulation and Analysis

Source: https://github.com/moorethreads/simumax/blob/main/docs/tutorial.md

This Python script demonstrates how to set up and run performance simulations using the PerfLLM class. It configures the model, runs an estimation, and then performs memory and cost analysis. Dependencies include configuration files for system, strategy, and model parameters.

```python
# Define the system、strategy、model config
system_config_file = ...
strategy_config_file = ...
model_config_file = ...
# Setup perf model and config
perf_model = PerfLLM()
perf_model.configure(
    strategy_config=strategy_config_file, 
    model_config=model_config_file, 
    system_config=system_config_file
)

# Run simulate
perf_model.run_estimate()

# Based simulate result, run memory analysis
mem_result = perf_model.analysis_mem()

# Based simulate result, run cost analysis
cost_result = perf_model.analysis_cost()
```

--------------------------------

### Utility Functions - Configuration Helpers

Source: https://context7.com/moorethreads/simumax/llms.txt

Provides utility functions for discovering and loading model, strategy, and system configurations. It includes functions to display available configurations and retrieve configuration file paths by name, which can then be used to initialize simulation components.

```python
from simumax.utils import (
    get_simu_model_config,
    get_simu_strategy_config,
    get_simu_system_config,
    show_simu_models,
    show_simu_strategy,
    show_simu_system,
)

# Show all available configurations
show_simu_models()     # Prints table of available model configs
show_simu_strategy()   # Prints table of available strategy configs
show_simu_system()     # Prints table of available system configs

# Get config file paths by name
model_path = get_simu_model_config('llama3-8b')
strategy_path = get_simu_strategy_config('tp2_pp1_dp4_mbs1')
system_path = get_simu_system_config('a100_pcie')

print(f"Model config: {model_path}")
print(f"Strategy config: {strategy_path}")
print(f"System config: {system_path}")

# Use in simulation
from simumax.core.config import ModelConfig, StrategyConfig, SystemConfig
from simumax.core.perf_llm import PerfLLM

perf_model = PerfLLM()
perf_model.configure(
    strategy_config=StrategyConfig.init_from_config_file(get_simu_strategy_config('tp2_pp1_dp4_mbs1')),
    model_config=ModelConfig.init_from_config_file(get_simu_model_config('llama3-8b')),
    system_config=SystemConfig.init_from_config_file(get_simu_system_config('a100_pcie')),
)
perf_model.run_estimate()
perf_model.analysis(save_path='./results')
```

--------------------------------

### Search Best Parallel Strategy with Recompute (Python)

Source: https://github.com/moorethreads/simumax/blob/main/docs/tutorial.md

This Python code snippet demonstrates how to use the `search_best_parallel_strategy_with_recompute` method from the `PerfLLM` class. It configures the model, system, and strategy, then searches for optimal parallel strategies across different recomputation types and execution partitions.

```python
perf_model = PerfLLM()
perf_model.configure(
    strategy_config=StrategyConfig.init_from_config_file(strategy_config_file),
    model_config=ModelConfig.init_from_config_file(model_config_file),
    system_config=SystemConfig.init_from_config_file(system_config_file),
)
perf_model.search_best_parallel_strategy_with_recompute(
        world_size=2048,
        gmi_error=6, 
        micro_batch_size=1,
        global_batch_size= 2048*8,
        all_search_result=all_search_result,
        tp_search_list=[1],
        ep_search_list=[8, 16, 32, 64],
        recompute_search_type=['no_recompute', 'full_block', 'selective_recompute'],
        use_reserved_memory=False,
        dump_path=f"search_{{perf_model.model_config.model_name}}_{{perf_model.system.sys_name}}"
    )
```

--------------------------------

### Define Test Hyperparameters for Efficiency Tests (JSON)

Source: https://github.com/moorethreads/simumax/blob/main/simu_tools/efficency_test/README.md

This JSON file specifies the parameters for the operator efficiency testing process. It includes lists of models to test, micro-batch sizes, sequence lengths, tensor parallelism (TP), and expert parallelism (EP) configurations.

```json
{
    "model_list":["deepseekv2",
                  "deepseekv3",
                  "deepseek-32b",
                  "deepseek-16b",
                  "deepseek-1b",
                  "llama3-8b",
                  "llama3-70b",
                  "qwen3-32b",
                  "kimi-1T",
                  "mixtral-8x7b"],
    "mbs_list": [1, 2, 4],
    "seq_len_list":[4096],
    "tp_list": [1, 2, 4, 8],
    "ep_list": [1, 2, 4, 8, 16, 64]
}
```

--------------------------------

### Access Model Properties

Source: https://context7.com/moorethreads/simumax/llms.txt

Demonstrates how to access and print key properties of a model, such as total parameters and FLOPs per token. This is useful for understanding model scale and computational cost.

```python
print(f"Total parameters: {model.param_numel / 1e9:.2f}B")
print(f"FLOPs per token: {model.flops_per_token(context_seq_len=4096, with_attn=True):.2e}")
```

--------------------------------

### Search Best Parallel Strategy with Recompute

Source: https://context7.com/moorethreads/simumax/llms.txt

Searches for the optimal combination of parallel strategies (TP/EP/PP) and recomputation configurations to maximize training efficiency under memory constraints. It takes world size, memory error tolerance, micro-batch size, global batch size, and search spaces for parallelism and recomputation types as input. Outputs are saved to CSV files, and the best strategy configuration is returned.

```python
from simumax.core.perf_llm import PerfLLM
from simumax.core.config import StrategyConfig, ModelConfig, SystemConfig

perf_model = PerfLLM()
perf_model.configure(
    strategy_config=StrategyConfig.init_from_config_file('configs/strategy/tp1_pp2_dp4_mbs1.json'),
    model_config=ModelConfig.init_from_config_file('configs/models/deepseekv3.json'),
    system_config=SystemConfig.init_from_config_file('configs/system/a100_pcie.json'),
)

# Optional: Set Megatron-compatible configurations
perf_model.model_config.moe_pad_expert_input_to_capacity = True
perf_model.model_config.capacity = 1
perf_model.strategy.dispatch_probs = True

all_search_results = {}
best_strategy = perf_model.search_best_parallel_strategy_with_recompute(
    world_size=2048,
    gmi_error=6,  # 6GB memory reserved for GPU memory interface overhead
    micro_batch_size=1,
    global_batch_size=2048 * 8,
    all_search_result=all_search_results,
    tp_search_list=[1],  # Search space for tensor parallelism
    ep_search_list=[8, 16, 32, 64],  # Search space for expert parallelism
    recompute_search_type=['no_recompute', 'full_block', 'selective_recompute'],
    use_reserved_memory=False,
    dump_path="search_deepseekv3_a100_pcie"
)

# Results saved to CSV files in dump_path directory
# Returns best_strategy dict with optimal configuration
print(f"Best MFU: {best_strategy['mfu']}")
print(f"Best parallelism: {best_strategy['parallelism']}")
```

--------------------------------

### JSON Configuration for Pipeline Parallelism

Source: https://github.com/moorethreads/simumax/blob/main/docs/tutorial.md

This JSON snippet illustrates how to configure pipeline parallelism in a strategy file. It specifies the number of layers for the first and last pipeline stages, which is crucial when `pp_size` is greater than 1. The remaining layers are distributed among the middle stages.

```json
{
    "seq_len": 4096,
    "micro_batch_size": 1,
    "world_size": 8,
    "tp_size": 1,
    "pp_size": 4,
    "ep_size": 2,
    ...
    "num_layers_in_first_pipeline_stage": 9,
    "num_layers_in_last_pipeline_stage": 11,
}
```

--------------------------------

### System Configuration JSON for Hardware Specifications

Source: https://context7.com/moorethreads/simumax/llms.txt

This JSON defines the hardware specifications for a system, including accelerator details like memory, operations (tflops, efficient_factor), bandwidth, and network configurations. It's used to model hardware for performance estimations.

```json
{
    "sys_name": "a100_pcie_bf16",
    "num_per_node": 8,
    "accelerator": {
        "backend": "cuda",
        "mem_gbs": 80,
        "op": {
            "default": {"tflops": 312, "efficient_factor": 0.75},
            "matmul": {"tflops": 312, "efficient_factor": 0.75, "accurate_efficient_factor": {}},
            "sdp_fwd": {"tflops": 312, "efficient_factor": 0.75, "accurate_efficient_factor": {}}
        },
        "bandwidth": {
            "default": {"efficient_factor": 0.91, "gbps": 1600, "latency_us": 40}
        },
        "mode": "roofline"
    },
    "networks": {
        "intra_with_pcie": true,
        "intra_node_pcie_8x": {
            "processor_usage": 0.0,
            "bandwidth": {"efficient_factor": 0.5, "gbps": 30, "latency_us": 10},
            "op": {
                "all_reduce": {"scale": 2, "offset": -1, "efficient_factor": 0.42, "latency_us": 9},
                "all_gather": {"scale": 1, "offset": -1, "efficient_factor": 0.45, "latency_us": 5},
                "reduce_scatter": {"scale": 1, "offset": -1, "efficient_factor": 0.37, "latency_us": 16}
            }
        }
    }
}
```

--------------------------------

### analysis_cost() - Cost and Performance Analysis

Source: https://context7.com/moorethreads/simumax/llms.txt

Analyzes compute costs, communication overhead, and overall training performance metrics including MFU, TFLOPS, and throughput.

```APIDOC
## analysis_cost() - Cost and Performance Analysis

### Description
Analyzes compute costs, communication overhead, and overall training performance metrics including MFU, TFLOPS, and throughput.

### Method
`analysis_cost()`

### Endpoint
N/A (Class method of PerfLLM)

### Parameters
None

### Request Example
```python
from simumax.core.perf_llm import PerfLLM
from simumax.core.config import StrategyConfig, ModelConfig, SystemConfig

perf_model = PerfLLM()
perf_model.configure(
    strategy_config=StrategyConfig.init_from_config_file('configs/strategy/tp2_pp1_dp4_mbs1.json'),
    model_config=ModelConfig.init_from_config_file('configs/models/llama3-8b.json'),
    system_config=SystemConfig.init_from_config_file('configs/system/a100_pcie.json'),
)
perf_model.run_estimate()

# Get cost analysis results
cost_result = perf_model.analysis_cost()
```

### Response
#### Success Response (200)
- **`cost_result.data`** (dict) - A dictionary containing cost and performance metrics. Key fields include:
    - `mfu_6nd_with_attn` (float): Model FLOPs Utilization.
    - `throughput per GPU (TFLOP/s/GPU)` (float): TFLOPS per GPU.
    - `throughput_per_accelerator` (float): Tokens per GPU per second.
    - `duration_time_per_iter` (float): Iteration time in milliseconds.

#### Response Example
```
MFU: 0.48
TFLOPS per GPU: 151.01
Tokens per GPU per second: 2933.66
Iteration time: 5584.8 ms
```
```

--------------------------------

### SystemConfig - Hardware System Configuration

Source: https://context7.com/moorethreads/simumax/llms.txt

Defines and configures GPU cluster hardware specifications using SystemConfig. It allows loading configurations from JSON files and accessing properties like memory, bandwidth, and compute capabilities. It also provides methods to compute operation times for both compute and network operations.

```python
from simumax.core.config import SystemConfig

# Load from JSON file (recommended - includes detailed efficiency factors)
system = SystemConfig.init_from_config_file('configs/system/a100_pcie.json')

# Access system properties
print(f"System name: {system.sys_name}")
print(f"GPUs per node: {system.num_per_node}")
print(f"GPU memory: {system.accelerator.mem_gbs} GB")
print(f"Default TFLOPS: {system.accelerator.op['default'].tflops}")
print(f"Memory bandwidth: {system.accelerator.bandwidth['default'].gbps} GB/s")

# Compute operation times
compute_time = system.compute_op_accuracy_time(
    op_name='matmul',
    flops=1e12,
    shape_desc='b=1, m=4096, k=4096, n=4096, layout=TN, accumulate=False, out_dtype=bf16'
)
print(f"Compute time: {compute_time:.4f} ms")

# Compute network operation time
net_time = system.compute_net_op_time(
    op_name='all_reduce',
    size=1024*1024*100,  # 100MB
    comm_num=8,
    net='intra_node_pcie_8x'
)
print(f"Network time: {net_time:.4f} ms")
```

--------------------------------

### Strategy Configuration JSON

Source: https://context7.com/moorethreads/simumax/llms.txt

Defines distributed training parameters for a model. This JSON file specifies settings like sequence length, batch sizes, data type, world size, and tensor/pipeline/expert parallelism sizes. It also includes options for sequence parallelism, recomputation, and attention/MLP optimizations.

```json
{
    "seq_len": 4096,
    "micro_batch_size": 1,
    "micro_batch_num": 8,
    "dtype": "bf16",
    "world_size": 8,
    "tp_size": 2,
    "pp_size": 1,
    "ep_size": 1,
    "etp_size": 1,
    "moe_dispatcher_policy": "all2all",
    "enable_sequence_parallel": true,
    "zero_state": 1,
    "enable_recompute": true,
    "recompute_granularity": "selective_recompute",
    "recompute_layer_num": 16,
    "attn_recompute": true,
    "mlp_recompute": true,
    "use_flash_sdp": true,
    "mem_factor": 0.94
}
```

--------------------------------

### Configure Full Recompute in Strategy File (JSON)

Source: https://github.com/moorethreads/simumax/blob/main/docs/tutorial.md

This JSON configuration enables full recomputation for a layer. It sets the `recompute_granularity` to 'full_block' and specifies the number of layers to recompute using `recompute_layer_num`. This is useful for memory-intensive training scenarios.

```json
{
    "seq_len": 4096,
    "micro_batch_size": 1,
    "world_size": 8,
    "tp_size": 1,
    "pp_size": 1,
    "ep_size": 8,
    "recompute_granularity" : "full_block", 
    "recompute_layer_num": 1
}
```

--------------------------------

### Run Llama3 Performance Test (Bash)

Source: https://github.com/moorethreads/simumax/blob/main/simu_tools/megatron_scripts/README.md

Executes performance tests for Llama3 models. Requires specifying batch size, model configuration, tensor/pipeline parallelism, model type, test type (test/profile), and number of layers. The 'test' type uses mock data for 10 iterations, while 'profile' uses torch.profiler for trace file generation over 6 iterations with 1 warm-up.

```Bash
bash run_llama3.sh ${mbs} ${mbc} ${tp_size} ${pp_size} ${model_type} ${test_type} ${num_layers}
```

```Bash
bash run_llama3.sh 1 32 1 2 llama3_70b test 12
```

--------------------------------

### Configure Selective Recompute in Strategy File (JSON)

Source: https://github.com/moorethreads/simumax/blob/main/docs/tutorial.md

This JSON configuration enables selective recomputation, allowing fine-grained control over which components are recomputed. It activates recomputation for attention, MLP, and RMS layers by setting corresponding boolean flags and specifies the number of layers to recompute.

```json
{
    "seq_len": 4096,
    "micro_batch_size": 1,
    "world_size": 8,
    "tp_size": 1,
    "pp_size": 1,
    "ep_size": 8,
    "recompute_granularity" : "selective_recompute", 
    "recompute_layer_num": 1,
    "attn_recompute":true,
    "mla_rms_recompute":true,   
    "mlp_recompute":true,
    "mlp_rms_recompute":true
}
```

--------------------------------

### Run Deepseek Performance Test (Bash)

Source: https://github.com/moorethreads/simumax/blob/main/simu_tools/megatron_scripts/README.md

Executes performance tests for Deepseek models (v2/v3). Requires specifying batch size, model configuration, expert/pipeline parallelism, model type, test type (test/profile), and number of layers. Similar to Llama3, 'test' uses mock data, and 'profile' uses torch.profiler.

```Bash
bash run_deepseekv2.sh ${mbs} ${mbc} ${ep_size} ${pp_size} ${model_type} ${test_type} ${num_layers}
```

```Bash
bash run_deepseekv2.sh 1 32 8 1 deepseek_v2 test 4
```

```Bash
bash run_deepseekv2.sh 1 32 8 1 deepseek_v2 profile 4
```

--------------------------------

### analysis_mem() - Memory Analysis

Source: https://context7.com/moorethreads/simumax/llms.txt

Analyzes memory usage during training, returning detailed breakdowns of weight, gradient, state, and activation memory for each pipeline stage.

```APIDOC
## analysis_mem() - Memory Analysis

### Description
Analyzes memory usage during training, returning detailed breakdowns of weight, gradient, state, and activation memory for each pipeline stage.

### Method
`analysis_mem()`

### Endpoint
N/A (Class method of PerfLLM)

### Parameters
None

### Request Example
```python
from simumax.core.perf_llm import PerfLLM
from simumax.core.config import StrategyConfig, ModelConfig, SystemConfig

perf_model = PerfLLM()
perf_model.configure(
    strategy_config=StrategyConfig.init_from_config_file('configs/strategy/tp1_pp2_dp4_mbs1.json'),
    model_config=ModelConfig.init_from_config_file('configs/models/llama3-70b.json'),
    system_config=SystemConfig.init_from_config_file('configs/system/a100_pcie.json'),
)
perf_model.run_estimate()

# Get memory analysis results
mem_result = perf_model.analysis_mem()
```

### Response
#### Success Response (200)
- **`mem_result.data`** (dict) - A dictionary containing memory usage details. For PP=1, it returns single stage results. For PP>1, it returns a dictionary with keys like 'first_stage', 'middle_stage' (if PP>2), and 'last_stage'. Each stage contains fields like 'model_mem', 'fwd_activation_cache_per_micro_batch', 'peak_mem', 'peak_mem_with_reserved', etc.

#### Response Example
```json
{
  "first_stage": {
    "model_mem": "45.2 GB",
    "fwd_activation_cache_per_micro_batch": "2.5 GB",
    "peak_mem": "58.3 GB",
    "peak_mem_with_reserved": "62.0 GB"
  },
  "last_stage": {...}
}
```
```

--------------------------------

### Analyze Compute Cost and Performance with SimuMax

Source: https://context7.com/moorethreads/simumax/llms.txt

Analyzes the compute cost and performance metrics of an LLM training simulation using the PerfLLM class. After configuring and running the simulation, `analysis_cost()` is called to retrieve metrics like MFU, TFLOPS, throughput, and iteration time.

```python
from simumax.core.perf_llm import PerfLLM
from simumax.core.config import StrategyConfig, ModelConfig, SystemConfig

perf_model = PerfLLM()
perf_model.configure(
    strategy_config=StrategyConfig.init_from_config_file('configs/strategy/tp2_pp1_dp4_mbs1.json'),
    model_config=ModelConfig.init_from_config_file('configs/models/llama3-8b.json'),
    system_config=SystemConfig.init_from_config_file('configs/system/a100_pcie.json'),
)
perf_model.run_estimate()

# Get cost analysis results
cost_result = perf_model.analysis_cost()

print(f"MFU: {cost_result.data['mfu_6nd_with_attn']:.2f}")
print(f"TFLOPS per GPU: {cost_result.data['throughput per GPU (TFLOP/s/GPU)']:.2f}")
print(f"Tokens per GPU per second: {cost_result.data['throughput_per_accelerator']:.2f}")
print(f"Iteration time: {cost_result.data['duration_time_per_iter']:.2f} ms")
```

--------------------------------

### ModelConfig - Model Architecture Configuration

Source: https://context7.com/moorethreads/simumax/llms.txt

Defines transformer model architecture parameters for both dense and Mixture-of-Experts (MoE) models. Users can load configurations from JSON files or define them programmatically, specifying parameters such as hidden size, number of heads, layers, vocabulary size, and MoE-specific parameters like expert count and top-k routing. This class supports various attention types and LoRA configurations.

```python
from simumax.core.config import ModelConfig

# Method 1: Load from JSON file
model = ModelConfig.init_from_config_file('configs/models/llama3-8b.json')

# Method 2: Create for dense model
dense_model = ModelConfig(
    model_type='dense',
    model_name='llama3_8b',
    hidden_size=4096,
    head_num=32,
    kv_head_num=8,
    head_size=128,
    intermediate_size=14336,
    layer_num=32,
    vocab_size=128256,
    use_swiglu=True,
)

# Method 3: Create for MoE model (e.g., DeepSeek-V2 style)
moe_model = ModelConfig(
    model_type='moe',
    model_name='deepseek_v2',
    hidden_size=5120,
    head_num=128,
    kv_head_num=128,
    layer_num=60,
    vocab_size=102400,
    use_swiglu=True,
    expert_num=160,
    topk=6,
    moe_ffn_hidden_size=1536,
    moe_shared_expert_intermediate_size=3072,
    attention_type='mla',  # Multi-head Latent Attention
    v_head_dim=128,
    qk_head_dim=192,
    qk_pos_emb_head_dim=64,
    q_lora_rank=1536,
    kv_lora_rank=512,
    dense_layers=1,  # Number of dense layers in MoE model
)
```

--------------------------------

### Default Operator Efficiency Configuration (JSON)

Source: https://github.com/moorethreads/simumax/blob/main/docs/system.md

This JSON snippet defines the default computing power (TFLOPS) and an efficient factor for operators. It also includes specific configurations for 'matmul' and 'sdp_fwd' operators, detailing their accurate computational efficiency under various shapes and parameters.

```json
"op": {
    "default" : {
        "tflops": 312,
        "efficient_factor": 0.75
    },
     "matmul" : {
                "tflops": 312,
                "efficient_factor": 0.75,
                "accurate_efficient_factor": {
                    "b=1, m=4096, k=5120, n=1536, layout=TN, accumulate=False, out_dtype=bf16": 0.7876672065615554,
                    "b=1, m=4096, k=1536, n=5120, layout=NN, accumulate=False, out_dtype=bf16": 0.737505124681297
                }
    },
     "fp8_matmul" : {
                "tflops": 312,
                "efficient_factor": 0.75,
                "accurate_efficient_factor": {}
     },
     "sdp_fwd" : {
                "tflops": 312,
                "efficient_factor": 0.75,
                "accurate_efficient_factor": {
                    "batch=1, seq_len=4096, head_num=128, kv_head_num=128, qk_head_dim=192, v_head_dim=128, qkv_contiguous=True": 1.0729673001633662,
                    "batch=1, seq_len=4096, head_num=64, kv_head_num=64, qk_head_dim=192, v_head_dim=128, qkv_contiguous=True": 1.0544285429372056
                }
     },
     "sdp_bwd" : {
                "tflops": 312,
                "efficient_factor": 0.75,
                "accurate_efficient_factor": {
                    "batch=1, seq_len=4096, head_num=128, kv_head_num=128, qk_head_dim=192, v_head_dim=128, qkv_contiguous=True": 0.8018473732899901,
                    "batch=1, seq_len=4096, head_num=64, kv_head_num=64, qk_head_dim=192, v_head_dim=128, qkv_contiguous=True": 0.7942592665301026
                }
     },
     "group_matmul" : {
                "tflops": 312,
                "efficient_factor": 0.75,
                "accurate_efficient_factor": {
                    "ng=40, M=616, N=3072, K=5120, dtype=bf16, out_dtype=bf16, main_grad_dtype=fp32, stage=fwd, grad=False, accumulate=False, use_split_accumulator=False, single_output=True": 0.6313438865579614,
                    "ng=40, M=616, N=3072, K=5120, dtype=bf16, out_dtype=bf16, main_grad_dtype=fp32, stage=bwd_grad_act, grad=True, accumulate=False, use_split_accumulator=True, single_output=False": 0.6790978664070304
                }
     }
}
```

--------------------------------

### SimuMax System File Template (JSON)

Source: https://github.com/moorethreads/simumax/blob/main/docs/system.md

A basic JSON template for the SimuMax system file. It outlines the structure for defining system name, number of GPUs per node, accelerator properties (backend, memory, operations, bandwidth, mode), and FC8 support.

```json
{
    "sys_name": "A100",
    "num_per_node": 8,
    "accelerator": {
        "backend": "cuda",
        "mem_gbs": 80,
        "op" : {

        },
        "bandwidth": {
            
        },
        "mode": "roofline"
    },
    "FC8":true,
}
```

--------------------------------

### Model Configuration JSON

Source: https://context7.com/moorethreads/simumax/llms.txt

Defines the parameters for a transformer model architecture. This JSON file specifies details such as model type, name, hidden size, number of heads, intermediate size, layer count, vocabulary size, and activation function.

```json
{
    "model_type": "dense",
    "model_name": "llama3_8b",
    "hidden_size": 4096,
    "head_num": 32,
    "kv_head_num": 8,
    "head_size": 128,
    "intermediate_size": 14336,
    "layer_num": 32,
    "vocab_size": 128256,
    "use_swiglu": true
}
```