### Install Dependencies using Conda Environment Source: https://github.com/casys-kaist/llmservingsim/blob/main/README.md Sets up the project's dependencies using a Conda environment file. This is the recommended method for a consistent installation. It creates a new Conda environment named 'env' in the current directory. ```bash conda env create -p ./env -f ./environment.yml conda activate ./env ``` -------------------------------- ### Install Protobuf Compiler Source: https://github.com/casys-kaist/llmservingsim/blob/main/README.md This error indicates a version mismatch between the protoc compiler and Protocol Buffer headers. Regenerating the file with a newer version of protoc is recommended. This snippet provides a common command to install or update the protobuf-compiler. ```bash sudo apt-get update sudo apt-get install protobuf-compiler ``` -------------------------------- ### Output Analysis with Pandas Source: https://context7.com/casys-kaist/llmservingsim/llms.txt Shows how to load and analyze simulation output using the pandas library. The example demonstrates reading a CSV file containing request performance metrics such as latency, queuing delay, and time to first token (TTFT), and displays the first few rows of the data. ```python import pandas as pd # Load simulation output results = pd.read_csv('output/example_run.csv') # Columns: request id, model, input, output, arrival, end_time, # latency, queuing_delay, TTFT, TPOT print(results.head()) # request id model input output arrival end_time latency queuing_delay TTFT TPOT # 0 0 meta-llama/Llama-3.1-8B-Instruct 128 256 50000000 500000000 450000000 50000000 100000000 3515625 # 1 1 meta-llama/Llama-3.1-8B-Instruct 256 384 75000000 600000000 525000000 75000000 125000000 4101562 ``` -------------------------------- ### Configure Multi-NPU Simulation with Tensor Parallelism Source: https://context7.com/casys-kaist/llmservingsim/llms.txt Configures and runs a distributed simulation across multiple NPUs, supporting hybrid parallelism strategies. This example uses 16 NPUs with specific bandwidth, latency, and batching parameters. Key parameters include npu_num, npu_group for defining parallelism, and max_batch for batch size. ```bash python main.py \ --model_name 'meta-llama/Llama-3.1-8B-Instruct' \ --hardware 'RTX3090' \ --npu_num 16 \ --npu_group 4 \ --npu_mem 40 \ --local_bw 1024 \ --remote_bw 512 \ --link_bw 256 \ --link_latency 0 \ --fp 16 \ --block_size 8 \ --max_batch 32 \ --dataset 'dataset/share-gpt-req100-rate10.tsv' \ --output 'output/multi_npu_results.csv' \ --req_num 100 \ --log_interval 0.5 \ --verbose ``` -------------------------------- ### Build ASTRA-Sim and Chakra Source: https://github.com/casys-kaist/llmservingsim/blob/main/README.md Builds the ASTRA-Sim simulation framework and installs Chakra, a component likely used for graph processing or frontend tasks. It requires navigating into specific directories and running build scripts. ```bash cd astra-sim ./build/astra_analytical/build.sh cd extern/graph_frontend/chakra pip install . cd ../../../.. ``` -------------------------------- ### Clean Conda Installation Source: https://github.com/casys-kaist/llmservingsim/blob/main/README.md Provides an alternative method for installing dependencies using a clean Conda environment. It creates a new environment named 'env_name' and installs specific versions of GCC, G++, and CMake, along with Protocol Buffers. ```bash conda create -n env_name python=3.9 conda activate env_name conda install -c conda-forge gcc_linux-64=7.5.0 gxx_linux-64=7.5.0 libprotobuf=3.6.1 cmake=3.22 ``` -------------------------------- ### Model Configuration Management (JSON and Python) Source: https://context7.com/casys-kaist/llmservingsim/llms.txt Illustrates how model configurations are defined in JSON format and loaded using Python. This includes essential parameters like model type, hidden size, number of layers, attention heads, and vocabulary size, which are crucial for simulation setup. ```json { "model_type": "llama", "hidden_size": 4096, "intermediate_size": 14336, "max_position_embeddings": 131072, "num_attention_heads": 32, "num_hidden_layers": 32, "num_key_value_heads": 8, "vocab_size": 128256 } ``` ```python from inference_serving.utils import get_config # Load model configuration config = get_config('meta-llama/Llama-3.1-8B-Instruct') print(f"Model: {config['model_type']}") print(f"Hidden size: {config['hidden_size']}") print(f"Layers: {config['num_hidden_layers']}") print(f"Attention heads: {config['num_attention_heads']}") print(f"KV heads: {config['num_key_value_heads']}") print(f"Vocab size: {config['vocab_size']}") print(f"FFN dimension: {config['intermediate_size']}") # Save as: model_configs/meta-llama/Llama-3.1-8B-Instruct.json ``` -------------------------------- ### Clone LLMServingSim Repository Source: https://github.com/casys-kaist/llmservingsim/blob/main/README.md Clones the LLMServingSim Git repository and its submodules, then navigates into the project directory. This is the first step to setting up the simulation environment. ```bash git clone --recurse-submodules https://github.com/casys-kaist/LLMServingSim.git cd LLMServingSim ``` -------------------------------- ### Run Basic LLM Serving Simulation Source: https://context7.com/casys-kaist/llmservingsim/llms.txt Executes a basic simulation of LLM inference serving using specified model, hardware, and network configuration parameters. It requires the 'main.py' script and outputs performance metrics to a CSV file. Verbose output provides real-time progress. ```bash python main.py \ --model_name 'meta-llama/Llama-3.1-8B-Instruct' \ --hardware 'RTX3090' \ --npu_num 1 \ --npu_group 1 \ --npu_mem 40 \ --remote_bw 512 \ --link_bw 256 \ --fp 16 \ --block_size 4 \ --dataset 'dataset/share-gpt-req100-rate10.tsv' \ --output 'output/example_run.csv' \ --verbose \ --req_num 10 ``` -------------------------------- ### Add Custom Models and Hardware to LLMServingSim (Bash) Source: https://context7.com/casys-kaist/llmservingsim/llms.txt This bash script outlines the steps to integrate new LLM models and hardware profiles into the LLMServingSim framework. It involves profiling hardware using `llm-profile`, creating a model configuration JSON file, and then running the simulation with the new model and hardware. ```bash # Step 1: Profile hardware using llm-profile tool # See: https://github.com/casys-kaist/llm-profile cd llm-profile python profile.py \ --model 'meta-llama/Llama-3.1-70B-Instruct' \ --hardware 'A100' \ --output '../perf_model/A100.csv' # Step 2: Create model config cat > model_configs/meta-llama/Llama-3.1-70B-Instruct.json << 'EOF' { "model_type": "llama", "hidden_size": 8192, "intermediate_size": 28672, "max_position_embeddings": 131072, "num_attention_heads": 64, "num_hidden_layers": 80, "num_key_value_heads": 8, "vocab_size": 128256 } EOF # Step 3: Run simulation with new configuration python main.py \ --model_name 'meta-llama/Llama-3.1-70B-Instruct' \ --hardware 'A100' \ --npu_num 8 \ --npu_group 8 \ --npu_mem 80 \ --remote_bw 1024 \ --link_bw 600 \ --fp 16 \ --block_size 16 \ --dataset 'dataset/share-gpt-req100-rate10.tsv' \ --output 'output/llama70b_a100.csv' \ --req_num 50 # Performance model CSV format (perf_model/A100.csv): # model,hardware,layer_name,input,kv_cache,latency(ns) # meta-llama/Llama-3.1-70B-Instruct,A100,embedding,128,0,1234567 # meta-llama/Llama-3.1-70B-Instruct,A100,q_proj,128,0,2345678 ``` -------------------------------- ### Run LLMServingSim Test Case Source: https://github.com/casys-kaist/llmservingsim/blob/main/README.md Executes a test run of the LLMServingSim with specified parameters for model, hardware, network, and dataset. This command demonstrates how to configure and launch a simulation. ```bash python main.py --model_name 'meta-llama/Llama-3.1-8B-Instruct' --hardware 'RTX3090' --npu_num 1 --npu_group 1 --npu_mem 40 \ --remote_bw 512 --link_bw 256 --fp 16 --block_size 4 \ --dataset 'dataset/share-gpt-req100-rate10.tsv' --output 'output/example_run.csv' \ --verbose --req_num 10 ``` -------------------------------- ### Initialize Scheduler for Request Batching Source: https://context7.com/casys-kaist/llmservingsim/llms.txt Initializes the Scheduler API from the 'inference_serving' library to manage and batch LLM inference requests. This code snippet demonstrates the import of necessary classes, 'Scheduler' and 'Request', which are fundamental for implementing memory-aware batching strategies. ```python from inference_serving.scheduler import Scheduler from inference_serving.request import Request ``` -------------------------------- ### Generate Network Configuration with Python Source: https://context7.com/casys-kaist/llmservingsim/llms.txt Automatically generates network topology configurations for NPU interconnects using the `create_network_config` function. It also demonstrates how to set remote memory bandwidth using `set_remote_bandwidth`. The generated network configuration is saved to a file, and the memory configuration is updated. ```python from inference_serving.config_generator import create_network_config, set_remote_bandwidth # Create network configuration network_file = create_network_config( astra_sim='./astra-sim', npu_nums=16, # Total NPUs npu_group=4, # NPUs per group link_bw=256, # Link bandwidth in GB/s link_latency=0 # Link latency in ns ) print(f"Network config written to: {network_file}") # Generated YAML structure: # topology: [FullyConnected, FullyConnected, FullyConnected] # npus_count: [4, 2, 2] # bandwidth: [256.0, 256.0, 256.0] # latency: [0.0, 0.0, 0.0] # Configure remote (host) memory bandwidth memory_file = set_remote_bandwidth( remote='astra-sim/inputs/remote_memory/per_npu_memory_expansion.json', remote_bw=512 # GB/s ) print(f"Memory config updated: {memory_file}") # Modifies: remote-mem-latency: 0, remote-mem-bw: 512 ``` -------------------------------- ### Add Requests to Batch and Generate Trace Source: https://context7.com/casys-kaist/llmservingsim/llms.txt Demonstrates how to create Request objects, add them to a batch, and then generate a simulation trace file. The trace file contains detailed information about layer operations, memory transactions, and communication operations for the specified hardware and batch. ```python req1 = Request(0, 'meta-llama/Llama-3.1-8B-Instruct', 64, 128, 0, is_init=True) req2 = Request(1, 'meta-llama/Llama-3.1-8B-Instruct', 66, 132, 0, is_init=True) batch.requests.extend([req1, req2]) generate_trace( batch=batch, hardware='RTX3090', npu_num=4, npu_group=2, fp=16 ) # Output: inputs/trace/RTX3090_meta-llama/Llama-3.1-8B-Instruct_batch0.txt # Trace includes: layer operations, memory transactions, communication ops first_arrival_ns = 100000000 # 100ms generate_event(alarm=first_arrival_ns) # Creates: inputs/trace/event_handler.txt ``` -------------------------------- ### Initialize and Run LLM Scheduler Source: https://context7.com/casys-kaist/llmservingsim/llms.txt Initializes a scheduler with model parameters and then loads requests from a dataset or adds them programmatically. It schedules the next batch of requests, prints details about the scheduled batch, marks the batch as complete, and prints the completion results. Finally, it checks if all requests are processed and prints the simulation results. ```python scheduler = Scheduler( model='meta-llama/Llama-3.1-8B-Instruct', max_batch=32, # Maximum batch size npu_num=4, # Number of NPUs npu_group=2, # NPU groups for parallelism npu_mem=40, # NPU memory in GB fp=16, # Floating point bits block_size=8, # KV cache block size req_num=100, # Total requests to process verbose=True ) # Load requests from dataset scheduler.generate( path='dataset/share-gpt-req100-rate10.tsv', is_init=True ) # Manually add requests programmatically # Format: [model, input_length, output_length, arrival_time_ns] scheduler.add_request([ 'meta-llama/Llama-3.1-8B-Instruct', 128, # input tokens 256, # total output tokens (input + generated) 0 # arrival time in nanoseconds ], is_init=True) # Schedule next batch at current time current_time_ns = 1000000000 # 1 second sys_id = 0 # System/NPU ID batch = scheduler.schedule(current_time_ns, sys_id) if batch: print(f"Scheduled batch {batch.batch_id}") print(f" Requests: {[req.id for req in batch.requests]}") print(f" Total input length: {batch.input}") print(f" KV cache size: {batch.kv_size} bytes") print(f" Init requests: {batch.init_cnt}") # Mark batch as complete finish_time_ns = 1050000000 prompt_tokens, gen_tokens, completed_reqs = scheduler.add_done( id=batch.batch_id, sys=sys_id, finish=finish_time_ns ) print(f"Completed: {completed_reqs} requests") print(f"Prompt tokens: {prompt_tokens}, Generated: {gen_tokens}") # Check if all requests processed if scheduler.is_request_empty(): scheduler.print_result() scheduler.save_output('output/results.csv') ``` -------------------------------- ### Create Conda Activation/Deactivation Directories Source: https://github.com/casys-kaist/llmservingsim/blob/main/README.md These commands create the necessary directories for activation and deactivation scripts within the conda environment. `$CONDA_PREFIX` is an environment variable that points to the current conda environment's root directory. ```bash mkdir -p $CONDA_PREFIX/etc/conda/activate.d mkdir -p $CONDA_PREFIX/etc/conda/deactivate.d ``` -------------------------------- ### Execute LLMServingSim via Run Script Source: https://github.com/casys-kaist/llmservingsim/blob/main/README.md A simplified command to run LLMServingSim using a provided shell script. This is a convenient way to launch the simulation with default or pre-configured settings. ```bash ./run.sh ``` -------------------------------- ### Request and Batch Data Structures Source: https://context7.com/casys-kaist/llmservingsim/llms.txt Defines and demonstrates the usage of core data structures for managing inference requests and batches. It shows how to create individual requests, track their lifecycle, and construct batch objects, including details on their properties and how to extend them with requests. ```python from inference_serving.request import Request, Batch # Create individual request request = Request( id=0, model='meta-llama/Llama-3.1-8B-Instruct', input=128, # Input sequence length output=256, # Total output length (input + generated) arrival=50000000, # Arrival time in ns is_init=True # True for prefill, False for decode ) # Track request lifecycle request.set_que_delay(current=100000000) # When scheduling starts request.set_ttft(current=150000000) # Time to first token request.add_latency(end_time=500000000) # When request completes print(request) # {'id': 0, 'model': 'meta-llama/Llama-3.1-8B-Instruct', # 'input': 128, 'output': 256, 'arrival': 50000000, # 'end_time': 500000000, 'latency': 450000000, # 'queuing_delay': 50000000, 'ttft': 100000000, 'tpot': 3515625} # Create batch batch = Batch( batch_id=0, model='meta-llama/Llama-3.1-8B-Instruct', input=384, # Total tokens across batch init_cnt=3, # Requests in prefill phase batch_size='1', batch_time=100000000, kv_size=6144000, # Total KV cache bytes evict=0, load=0, is_orca=True ) # Track execution across NPUs batch.fired.append(0) # NPU 0 started batch.fired.append(1) # NPU 1 started batch.requests.extend([request]) # Add requests to batch batch.end.append(0) # NPU 0 completed print(f"Batch {batch.batch_id}: {len(batch.requests)} requests, " f"{len(batch.fired)} NPUs fired, {len(batch.end)} NPUs done") ``` -------------------------------- ### Generate Execution Trace for Hardware Simulation Source: https://context7.com/casys-kaist/llmservingsim/llms.txt Generates execution traces for ASTRA-Sim from batched requests. This snippet shows how to create a `Batch` object with essential parameters like batch ID, model name, input length, number of prefill requests, batch size, creation time, KV cache size, and evicted/loaded KV size. ```python from inference_serving.generate_trace import generate_trace, generate_event from inference_serving.request import Batch, Request # Create batch with requests batch = Batch( batch_id=0, model='meta-llama/Llama-3.1-8B-Instruct', input=130, # Total input length across batch init_cnt=2, # Number of prefill requests batch_size='1', # Batch size string batch_time=0, # Batch creation time kv_size=2048000, # KV cache size in bytes evict=0, # Evicted KV size load=0, # Loaded KV size is_orca=True ) ``` -------------------------------- ### Create Custom Request Datasets in Python Source: https://context7.com/casys-kaist/llmservingsim/llms.txt Generates custom TSV datasets for LLM serving simulations using Python with Pandas and NumPy. This function creates datasets with specified numbers of requests and average arrival rates, defining input/output token counts and arrival times in nanoseconds. The output is saved to a TSV file. ```python import pandas as pd import numpy as np # Create custom request dataset with Poisson arrival pattern def create_dataset(num_requests=100, avg_rate_per_sec=10): """ num_requests: Total number of requests avg_rate_per_sec: Average arrival rate (requests per second) """ # Generate inter-arrival times (exponential distribution) inter_arrival_times = np.random.exponential( scale=1.0/avg_rate_per_sec, size=num_requests ) * 1e9 # Convert to nanoseconds arrival_times = np.cumsum(inter_arrival_times).astype(int) # Generate token counts (customize distributions as needed) input_tokens = np.random.randint(10, 512, size=num_requests) output_tokens = np.random.randint(20, 200, size=num_requests) df = pd.DataFrame({ 'input_toks': input_tokens, 'output_toks': output_tokens, 'arrival_time_ns': arrival_times }) df.to_csv('dataset/custom_workload.tsv', sep='\t', index=False) return df # Example: Create dataset with 200 requests at 15 req/s dataset = create_dataset(num_requests=200, avg_rate_per_sec=15) print(dataset.head()) # input_toks output_toks arrival_time_ns # 0 45 102 67234561 # 1 128 89 189456732 # 2 67 145 345678901 ``` -------------------------------- ### Activate Conda Environment Source: https://github.com/casys-kaist/llmservingsim/blob/main/README.md This command activates a specified conda environment. Replace 'your_env_name' with the actual name of your conda environment. ```bash conda activate your_env_name ``` -------------------------------- ### Calculate KV Cache Memory Requirements with Python Source: https://context7.com/casys-kaist/llmservingsim/llms.txt Initializes a memory model to calculate and manage memory requirements for KV cache using vLLM's block management. It prints model weight size, available memory, and the KV cache size for a given sequence length. It also checks memory availability and demonstrates calculating layer tensor sizes for different phases. ```python from inference_serving.memory_model import MemoryModel, calculate_sizes # Initialize memory model memory = MemoryModel( model='meta-llama/Llama-3.1-8B-Instruct', npu_num=4, npu_mem=40, # GB per NPU block_size=8, # Tokens per KV cache block fp=16, # Bits per parameter verbose=True ) print(f"Model weight size: {memory.weight / 1024**3:.2f} GB per NPU") print(f"Available memory: {(memory.npu_mem - memory.used_mem) / 1024**3:.2f} GB") # Calculate KV cache size for sequence seq_length = 128 kv_size = memory.get_kv(seq_length) print(f"KV cache for {seq_length} tokens: {kv_size / 1024**2:.2f} MB") # Check memory availability if memory.mem_avail(kv_size): memory.mem_load(kv_size) print("KV cache loaded successfully") # Calculate layer tensor sizes input_size, weight_size, output_size = calculate_sizes( model='meta-llama/Llama-3.1-8B-Instruct', layer_name='q_proj', length=128, init=True, # Prefill phase fp=2 # Bytes per element ) print(f"Q Projection - Input: {input_size/1024:.2f}KB, " f"Weight: {weight_size/1024**2:.2f}MB, " f"Output: {output_size/1024:.2f}KB") # Supported layer names: embedding, input_layernorm, q_proj, k_proj, v_proj, # rope, attn, o_proj, gate_proj, up_proj, down_proj, act_fn, # post_layernorm, final_layernorm, lm_head ``` -------------------------------- ### Calculate Inference Statistics and Throughput (Python) Source: https://context7.com/casys-kaist/llmservingsim/llms.txt This Python script calculates average latency, TTFT, TPOT, and queuing delay from simulation results. It also computes throughput based on tokens generated and latency, and analyzes throughput distribution across different input request sizes using pandas. ```python import pandas as pd # Assuming 'results' is a pandas DataFrame with columns like 'latency', 'TTFT', 'TPOT', 'queuing_delay', 'input', 'output' # Calculate statistics avg_latency = results['latency'].mean() / 1e9 # Convert to seconds avg_ttft = results['TTFT'].mean() / 1e9 avg_tpot = results['TPOT'].mean() / 1e9 avg_queue_delay = results['queuing_delay'].mean() / 1e9 print(f"Average latency: {avg_latency:.3f}s") print(f"Average TTFT: {avg_ttft:.3f}s") print(f"Average TPOT: {avg_tpot:.6f}s") print(f"Average queuing delay: {avg_queue_delay:.3f}s") # Analyze throughput by request size results['tokens_generated'] = results['output'] - results['input'] results['throughput'] = results['tokens_generated'] / (results['latency'] / 1e9) size_bins = pd.cut(results['input'], bins=[0, 100, 200, 500]) print("\nThroughput by input size:") print(results.groupby(size_bins)['throughput'].agg(['mean', 'std', 'count'])) ``` -------------------------------- ### Activate Conda Environment and Set CMAKE_PREFIX_PATH Source: https://github.com/casys-kaist/llmservingsim/blob/main/README.md This method sets the conda environment for CMake to use by explicitly exporting the CMAKE_PREFIX_PATH environment variable. This is useful when dealing with build systems that need to find libraries within a specific conda environment. ```bash conda activate your_env_name export CMAKE_PREFIX_PATH=$CONDA_PREFIX:$CMAKE_PREFIX_PATH ``` -------------------------------- ### Generate Chakra Graph for ASTRA-Sim Source: https://context7.com/casys-kaist/llmservingsim/llms.txt This section details how to convert execution traces into Chakra execution graphs for ASTRA-Sim. It covers generating both the main workload graph and event handler graphs using the `generate_graph` function and retrieving the workload path. ```python from inference_serving.generate_graph import generate_graph from inference_serving.utils import get_workload # Generate Chakra graph from trace generate_graph( batch=batch, hardware='RTX3090', total_num=4, # Total NPUs event=False ) # Creates graph at: inputs/workload/RTX3090_meta-llama/Llama-3.1-8B-Instruct_batch0/llm # Generate event handler graph generate_graph( batch=None, hardware='RTX3090', total_num=4, event=True ) # Get workload path for simulator workload_path = get_workload(batch, 'RTX3090', event=False) print(f"Workload: {workload_path}") # Returns: /path/to/inputs/workload/RTX3090_meta-llama/Llama-3.1-8B-Instruct_batch0/llm ``` -------------------------------- ### Set CMAKE_PREFIX_PATH on Conda Activation Source: https://github.com/casys-kaist/llmservingsim/blob/main/README.md This bash script is placed in the `activate.d` directory of a conda environment. It prepends the conda environment's prefix to the `CMAKE_PREFIX_PATH` and saves the original value. ```bash #!/bin/bash export OLD_CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH export CMAKE_PREFIX_PATH=$CONDA_PREFIX:$CMAKE_PREFIX_PATH ``` -------------------------------- ### Set Script Permissions Source: https://github.com/casys-kaist/llmservingsim/blob/main/README.md These commands ensure that the activation and deactivation scripts have execute permissions, allowing them to run when the conda environment is activated or deactivated. ```bash chmod +x $CONDA_PREFIX/etc/conda/activate.d/set_cmake_prefix.sh chmod +x $CONDA_PREFIX/etc/conda/deactivate.d/unset_cmake_prefix.sh ``` -------------------------------- ### Reset CMAKE_PREFIX_PATH on Conda Deactivation Source: https://github.com/casys-kaist/llmservingsim/blob/main/README.md This bash script is placed in the `deactivate.d` directory of a conda environment. It restores the `CMAKE_PREFIX_PATH` to its value before activation and unsets the temporary variable. ```bash #!/bin/bash export CMAKE_PREFIX_PATH=$OLD_CMAKE_PREFIX_PATH unset OLD_CMAKE_PREFIX_PATH ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.