LLM Wiki Agent

https://github.com/zhdhnsfdx/wiki-agent
Admin
The LLM Wiki Agent is a coding agent skill that ingests source documents, extracts knowledge, and...
Tokens:25,989
Snippets:277
Trust Score:2.9
License:MIT
Update:4 months ago
Show doc for...
Tokens:
Raw
### Example Activation Value Calculation

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/vllm推理框架并发帧存消耗分析.md

Example calculation of activation value memory consumption for 100 tokens using Qwen2.5-32B parameters.

```plaintext
激活值 = 1 × 100 × (5×5120  + 2×27648) × 2  # bfloat16
       = 100 × (25,600  + 55,296) × 2
       = 100 × 80,896  × 2
       = 16,179,200  字节 ≈  15.43 MB
```

--------------------------------

### Example Concurrency Calculation

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/vllm推理框架并发帧存消耗分析.md

Example calculation for maximum concurrency on a 4090D with a Qwen 32B Q8 model, assuming 4096 tokens.

```plaintext
# 参数
total_vram = 48  # 2张显卡，一张24GB
gpu_util = 0.85  # 85%利用率
model_size = 33 GB   # DeepSeek R1 Distill Qwen 32B Q8模型
token_kv = 0.00025 # 每个token的KV缓存(约0.25MB)
number_tokens = 4096 # 假设新增4096 tokens
# 计算
usable_vram = 48 * 0.85 = 40.8 GB
kv_per_request = number_tokens * token_kv = 1024 Mb # 总共算4096 tokens
activate_val = 632.5 Mb
tmp_val = 420 Mb
middle_val = activate_val + tmp_val
max_concurrent = (usable_vram - model_size) / （(kv_per_request + middle_val)/1024）
               = 4.33
```

--------------------------------

### Install Rust using Official Script

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/vllm部署与使用.md

Installs Rust and Cargo, the package manager for Rust, using the official installation script. This is a prerequisite for installing vLLM if Cargo is not already present.

```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```

--------------------------------

### Install Specific vLLM Dependencies

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/vllm部署与使用.md

Install specific versions of PyTorch, Transformers, and xformers using wheel files as required by vLLM. This ensures compatibility and optimal performance.

```bash
pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118
pip install transformers==4.31.0
pip install xformers==0.0.20
pip install vllm==0.2.1.post1
```

--------------------------------

### Configure Rustup Mirrors

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/vllm部署与使用.md

Sets environment variables to use Tsinghua University's mirrors for Rustup. This significantly speeds up the Rust installation process, which can be slow with the default servers.

```bash
export RUSTUP_UPDATE_ROOT=https://mirrors.tuna.tsinghua.edu.cn/rustup/rustup
export RUSTUP_DIST_SERVER=https://mirrors.tuna.tsinghua.edu.cn/rustup
```

--------------------------------

### Install Specific vLLM Version and Dependencies

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/vllm部署与使用.md

Use these commands to install a specific version of vLLM and its required Python packages. Ensure that the necessary wheel files are available locally or on the specified network path.

```bash
pip install vllm-0.4.0+cu116-cp38-cp38-linux_x86_64.whl
```

```bash
pip install outlines==0.0.44
```

```bash
pip install xformers-0.0.22+hc2.23.0.23.wbtorch2.0-cp38-cp38-linux_x86_64.whl
```

--------------------------------

### Llama Context Constructor and Graph Reservation

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/大模型在ollama框架下帧存消耗分析.md

Illustrates the steps involved in constructing a llama_context, including parameter initialization, backend creation, KV cache setup, and the crucial computation graph reservation phase. This phase reserves memory for both post-processing and token generation graphs under worst-case conditions to ensure consistent frame buffer allocation during inference.

```plain
llama_context构造函数
├── 初始化参数（n_ctx, n_batch, n_ubatch等）
├── 创建GPU/CPU后端
├── 构建KV Cache
├── **计算图预留阶段（worst-case）**
│ ├── 预留pp图（n_tokens=512, n_seqs=1）
│ │ └── 调用graph_build/graph_reserve分配CUDA0/CUDA1显存
│ ├── 预留tg图（n_tokens=1, n_seqs=1）
│ │ └── 调用graph_build/graph_reserve分配CUDA0/CUDA1显存
│ └── 再次预留pp图（n_tokens=512, n_seqs=1）
│ └── 确保推理时帧存分配一致性
└── 同步与性能统计初始化
```

--------------------------------

### Launch vLLM OpenAI API Server

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/vllm部署与使用.md

Starts a vLLM service compatible with the OpenAI API. Ensure the model path is correct and adjust parameters like port and model length as needed.

```plain
python -m vllm.entrypoints.openai.api_server --port 6600 --trust-remote-code --host 0.0.0.0 --model  /data/ftp/compute/model-data/models/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B  --served-model-name jjw 	--max-model-len 16384 --max-num-seqs 4
```

--------------------------------

### Install vLLM using Pip or Conda

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/vllm部署与使用.md

Installs the vLLM library using either pip or conda. Note that the official version does not support quantization and is intended for NVIDIA GPUs only.

```bash
pip install vllm
// 或者
conda install vllm
```

--------------------------------

### Run vLLM OpenAI API Server

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/vllm推理框架并发帧存消耗分析.md

This command starts the vLLM OpenAI API server with specific configurations for model loading, network settings, and resource utilization. It's useful for setting up a high-performance inference endpoint.

```plain
root@8bcd6a5704ba:/# python -m vllm.entrypoints.openai.api_server --port 6600 --trust-remote-code --host 0.0.0.0 --model /data/ftp/compute/model-data/models/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B-GPTQ-Int8 --served-model-name jjw --max-model-len 4096 --max-num-seqs 3 --tensor-parallel-size 2 --disable-custom-all-reduce --gpu-memory-utilization 0.85
INFO 06-03 01:09:55 [init.py:239] Automatically detected platform cuda.
INFO 06-03 01:09:56 [api_server.py:981] vLLM API server version 0.8.2
INFO 06-03 01:09:56 [api_server.py:982] args: Namespace(host='0.0.0.0', port=6600, uvicorn_log_level='info', disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=[''], allowed_methods=[''], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[ ], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='/data/ftp/compute/model-data/models/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B-GPTQ-Int8', task='auto', tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=4096, guided_decoding_backend='xgrammar', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=2, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=None, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.85, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=3, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=True, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, use_tqdm_on_load=True, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_config=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[ ], preemption_mode=None, served_model_name=['jjw'], qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_reasoning=False, reasoning_parser=None, disable_cascade_attn=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False)
INFO 06-03 01:10:02 [config.py:585] This model supports multiple tasks: {'reward', 'generate', 'score', 'embed', 'classify'}. Defaulting to 'generate'.
INFO 06-03 01:10:03 [gptq_marlin.py:143] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
INFO 06-03 01:10:03 [config.py:1519] Defaulting to use mp for distributed inference
INFO 06-03 01:10:03 [config.py:1697] Chunked prefill is enabled with max_num_batched_tokens=2048.

```

--------------------------------

### Verify Rust and Cargo Installation

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/vllm部署与使用.md

Checks if Rust and Cargo have been installed successfully by printing their version numbers. This command should be run after restarting the shell or sourcing the configuration file.

```bash
rustc --version
 cargo --version
```

--------------------------------

### Run Docker Container for vLLM

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/vllm部署与使用.md

Starts a Docker container with vLLM environment, mapping ports, devices, and volumes. This command is used to set up the isolated environment for vLLM operations.

```plain
docker run -it --name vllm-jaynean -p 8000:8000 --privileged=true  --device=/dev/htcd --device=/dev/dri --mount type=bind,source=/home/jingjia/jaynean,target=/jaynean -v /home/jingjia/tanyang/models/:/jaynean/models image.marsgpu.com/library/hpcc-m200-pytorch:2.23.0.23-wb-ubuntu20.04-amd64 /bin/bash
```

--------------------------------

### Example KV Cache Size Calculation

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/vllm推理框架并发帧存消耗分析.md

Example calculation of KV cache size per token for a specific model (DeepSeek R1 Distill Qwen 32B Q8).

```plaintext
n_layer = 64 
num_kv_heads = 8
head_dim: 128 其中，head_dim = hidden_size / num_heads = 5120 / 40 = 128 （注意：这里num_heads是查询头的数量，而num_kv_heads是键值头的数量）
数据类型为 f16（2 字节）
每个token KV 缓存=2×64×8x128×2 B=256 kb
```

--------------------------------

### Install Common Tools in Docker

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/vllm部署与使用.md

Updates package lists and installs essential tools like build-essential, vim, and inetutils-ping within the Docker container. These tools are useful for development and debugging.

```bash
apt upgrade
apt update
apt install build-essential
apt install vim
apt install inetutils-ping
```

--------------------------------

### KV Cache Size Calculation Example

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/大模型在ollama框架下帧存消耗分析.md

Calculates the total KV cache size based on context length, model layers, KV heads, head dimension, and data type size. Assumes a context length of 4096 tokens, 64 layers, 8 KV heads, a head dimension of 128, and f16 data type (2 bytes).

```text
n_ctx = 4096（实际加载的上下文长度）
n_layer = 64 
num_kv_heads = 8
head_dim: 128 其中，head_dim = hidden_size / num_heads
数据类型为 f16（2 字节）
总 KV 缓存=2×64×4096×8x128×2 B=1024 MiB
```

--------------------------------

### Verify Deployment Success

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/vllm部署与使用.md

Check the /etc/ld.so.conf.d/hpcc.conf file to ensure the deployment was successful. This step is relevant if issues arise later.

```shell
echo "/opt/hpcc/lib"             > /etc/ld.so.conf.d/hpcc.conf && \

```

--------------------------------

### llm_build_qwen2 Class Constructor and Graph Building

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/大模型在ollama框架下帧存消耗分析.md

The `llm_build_qwen2` class, derived from `llm_graph_context`, handles the specific graph construction for Qwen2 models. Its constructor initializes parameters and the subsequent `build` process involves calling specific functions for input embeddings, position encoding, and self-attention.

```cpp
struct llm_build_qwen2 : public llm_graph_context {
 llm_build_qwen2(const llama_model & model, const llm_graph_params & params, ggml_cgraph * gf) : llm_graph_context(params) {
 const int64_t n_embd_head = hparams.n_embd_head_v;

 GGML_ASSERT(n_embd_head == hparams.n_embd_head_k);
 GGML_ASSERT(n_embd_head == hparams.n_rot);

 ggml_tensor * cur;
 ggml_tensor * inpL;

 inpL = build_inp_embd(model.tok_embd);

 // inp_pos - contains the positions
 ggml_tensor * inp_pos = build_inp_pos();

 auto * inp_attn = build_attn_inp_kv_unified();

 for (int il = 0; il < n_layer; ++il) {
 ggml_tensor * inpSA = inpL;

 // norm
 cur = build_norm(inpL,
 model.layers[il].attn_norm, NULL,
 LLM_NORM_RMS, il);
 cb(cur, "attn_norm", il);

 // self-attention
 {
 // compute Q and K and RoPE them
 ggml_tensor * Qcur = build_lora_mm(model.layers[il].wq, cur);
 Qcur = ggml_add(ctx0, Qcur, model.layers[il].bq);
 cb(Qcur, "Qcur", il);

 ggml_tensor * Kcur = build_lora_mm(model.layers[il].wk, cur);
 Kcur = ggml_add(ctx0, Kcur, model.layers[il].bk);
 cb(Kcur, "Kcur", il);

 ggml_tensor * Vcur = build_lora_mm(model.layers[il].wv, cur);
 Vcur = ggml_add(ctx0, Vcur, model.layers[il].bv);
 cb(Vcur, "Vcur", il);

 Qcur = ggml_reshape_3d(ctx0, Qcur, n_embd_head, n_head, n_tokens);
 Kcur = ggml_reshape_3d(ctx0, Kcur, n_embd_head, n_head_kv, n_tokens);
 Vcur = ggml_reshape_3d(ctx0, Vcur, n_embd_head, n_head_kv, n_tokens);

 Qcur = ggml_rope_ext(
 ctx0, Qcur, inp_pos, nullptr,
 n_rot, rope_type, n_ctx_orig, freq_base, freq_scale,
 ext_factor, attn_factor, beta_fast, beta_slow
 );

 Kcur = ggml_rope_ext(

```

--------------------------------

### Run vLLM OpenAI API Server

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/vllm推理框架并发帧存消耗分析.md

This command launches the vLLM OpenAI API server with specific configurations for model loading, port, host, and resource utilization. It's used for testing concurrency limits with a particular model.

```bash
sudo docker run --gpus all -p 6600:6600 --mount type=bind,source=/home,target=/home   --mount type=bind,source=/data,target=/data  --security-opt seccomp=unconfined --security-opt apparmor=unconfined --shm-size '100gb' --ulimit memlock=-1 -it --rm  7fd824bc1487 /bin/bash
python -m vllm.entrypoints.openai.api_server --port 6600 --trust-remote-code --host 0.0.0.0 --model  /data/ftp/compute/model-data/models/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B-GPTQ-Int8 --served-model-name jjw --max-model-len 4096 --max-num-seqs 4 --tensor-parallel-size 2  --disable-custom-all-reduce  --gpu-memory-utilization 0.85
```

--------------------------------

### Run vLLM Performance Benchmark

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/vllm推理框架并发帧存消耗分析.md

This script initiates a performance benchmark for the vLLM inference framework. It requires specifying the model URL, parallel requests, total requests, model name, timeouts, API type, dataset path, and stream setting.

```plain
from evalscope.perf.main import run_perf_benchmark
task_cfg = {"url": "http://127.0.0.1:6600/v1/chat/completions",
            "parallel":8,
            "number":64,
            "model":"jjw",
            "connect_timeout": 600,
            "read_timeout": 600,
            "api": "openai",
            "dataset_path": "/home/jingjia/jaynean/mussycodes/testcodes/data/local64qa.jsonl",
            "stream":True }
run_perf_benchmark(task_cfg)
```

--------------------------------

### MOE Model Configuration and Data Sources

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/MOE模型分析.md

Defines the configuration parameters and data sources used in the MOE model, including model parameters, inference settings, and GPU specifications.

```text
配置模块：定义模型参数、推理配置、GPU参数
数据源：严格基于 DeepSeek-V3.2-Exp-main/inference/config_671B_v3.2.json 和 model.py
```

--------------------------------

### Python Client for vLLM OpenAI API

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/vllm部署与使用.md

Connects to a vLLM service using the OpenAI Python client. This script demonstrates how to send prompts, receive responses, and maintain conversation history.

```python
from openai import OpenAI
import time

# 配置 OpenAI API 密钥和 Base URL 以连接 vLLM 服务
openai_api_key = "EMPTY"  # vLLM 服务不需要 API 密钥，可以使用任意字符串
openai_api_base = "http://localhost:8630/v1"  # 确保端口号与启动 vLLM 服务时一致

# 创建 OpenAI 客户端
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

# 初始化历史记录
history = []

# 定义一个循环请求的函数
def loop_requests_with_history(model_name, delay=1):
    """
    循环发送请求并打印模型的回答，同时维护对话历史记录. 
    
    :param model_name: 模型名称
    :param delay: 每次请求之间的延迟时间（秒）
    """
    while True:
        # 手动输入提示内容
        prompt = input("请输入你的问题（输入 'exit' 结束）：")
        
        # 如果输入 'exit'，则退出循环
        if prompt.lower() == 'exit':
            print("退出程序。")
            break
        
        # 将当前提示添加到历史记录
        history.append({"role": "user", "content": prompt})
        
        # 构造完整的提示内容（包含历史记录）
        full_prompt = "\n".join([f"{item['role']}: {item['content']}" for item in history])
        
        try:
            # 调用模型生成回答
            response = client.completions.create(
                model=model_name,
                prompt=full_prompt,
                stream=False,  # 设置为 False 表示不使用流式输出
            )
            # 获取模型的回答内容
            response_text = response.choices[0].text.strip()
            
            # 将模型的回答添加到历史记录
            history.append({"role": "assistant", "content": response_text})
            
            # 输出模型的回答内容
            print("模型回答：")
            print(response_text)
        except Exception as e:
            print(f"处理请求时出错：{e}")
        finally:
            # 等待一段时间再发送下一个请求
            time.sleep(delay)

# 调用循环请求函数
loop_requests_with_history(model_name="DeepSeek-R1-Distill-Qwen-1.5B", delay=1)
```

--------------------------------

### Print LLM Outputs

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/vllm部署与使用.md

Iterates through a list of outputs and prints the prompt and generated text for each. This is useful for inspecting the results of LLM generations.

```python
for output in outputs:
  prompt = output.prompt
  generated_text = output.outputs[0].text
  print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
```

--------------------------------

### Launch vLLM API Server with Specific Model

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/vllm部署与使用.md

Launches a vLLM API server using a specific local model path and port. The `served-model-name` parameter is used by the client to identify the model.

```plain
CUDA_VISIBLE_DEVICES=0 python -m vllm.entrypoints.openai.api_server     --model ./DeepSeek-R1-Distill-Qwen-1_5B/     --port 8630     --max-model-len 16384     --served-model-name DeepSeek-R1-Distill-Qwen-1.5B
```

--------------------------------

### Indexer Module Query Generation

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/MOE模型分析.md

Shows how the Indexer module generates its Query vectors from the compressed Query latent and projection weights.

```text
1. 生成 Indexer Query
      q_latent @ indexer_wq_b ──► idx_q_combined: [b, s, index_n_heads * index_head_dim]
      reshape ──► [b, s, index_n_heads, index_head_dim]   # 多头 Query
```

--------------------------------

### Indexer Module Weights

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/MOE模型分析.md

Lists the weights used by the Indexer module for Query projection and Key projection.

```text
权重:
  - indexer_wq_b: [r_q, index_n_heads * index_head_dim]   # Query 投影
  - indexer_wk: [d, index_head_dim]                        # Key 投影（共享头）
  - indexer_weights_proj: [d, index_n_heads]               # （可选）多头得分融合
```

--------------------------------

### Run vLLM Model Inference

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/vllm部署与使用.md

Executes a vLLM model for text generation. It sets the CUDA visible devices, initializes the LLM with a specified model path, and generates outputs based on provided prompts and sampling parameters.

```python
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"

from vllm import LLM, SamplingParams

llm = LLM('../models/Qwen/Qwen2-7B-Instruct')

prompts = [
  "Hello, my name is",
  "The president of the United States is",
  "The capital of France is",
  "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

outputs = llm.generate(prompts, sampling_params)
```

--------------------------------

### 并发数对中间层显存消耗的影响

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/大模型在ollama框架下帧存消耗分析.md

使用DeepSeek R1 Distill Qwen 32B CTX 4096模型，分析不同并发数下中间层显存的理论计算值与实际消耗值对比。

```text
并发数|n_ubatch|计算公式|中间结果（MiB）|总显存消耗（MiB）|实际显存消耗（MiB）
1|512|(65 * 512 * 5120 * 1.25 * 2) / (1024²) * 2|406.25|812.5|（432*2）
2|1024|(65 * 1024 * 5120 * 1.25 * 2) / (1024²) * 2|812.5|1,625|（784*2）
    4|2048|(65 * 2048 * 5120 * 1.25 * 2) / (1024²) * 2|   1624|        3248|2758（1406+1352）
```

--------------------------------

### Dense FFN (SwiGLU) Implementation

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/MOE模型分析.md

Implements a standard SwiGLU activation within a Feed-Forward Network layer. This involves column-parallel and row-parallel linear transformations.

```python
h1 = x_norm @ w1  # [b, s, inter_dim] (整体维度，但各卡持有部分)
h3 = x_norm @ w3  # [b, s, inter_dim]
act = Swish(h1)  # [b, s, inter_dim]
gated = act * h3  # [b, s, inter_dim]  (逐元素乘)
ffn_out = gated @ w2  # [b, s, d]  (需跨卡 all-reduce 求和)
```

--------------------------------

### 模型参数、KV缓存和计算缓冲区显存占用分析

Source: https://github.com/zhdhnsfdx/wiki-agent/blob/main/raw/大模型在ollama框架下帧存消耗分析.md

展示了DeepSeek R1 Distill Qwen 32B CTX 4096模型在不同GPU设备上的显存占用明细，包括模型参数、KV缓存和计算缓冲区。

```text
GPU设备|模型参数|KV缓存|计算缓冲区|总计
GPU0|16.3 GB|0.52 GB|0.43 GB|**~17.25 GB**
GPU1|16.1 GB|0.50 GB|0.43 GB|**~17.03 GB**
```
Tokens:
Raw