### Minimal RLHF Stack Example (7B Model)

Source: https://github.com/wj-mcat/agent-handbook/blob/master/llms/04-post-training-alignment/03-rlhf/01-rlhf-pipeline.md

Provides a concrete example of a minimal RLHF setup for a 7B parameter model, including the sequence of training stages and estimated hardware requirements.

```text
7B: SFT → 7B RM → `trl` PPO，4×A100 级（随实现波动）。
```

--------------------------------

### Common npm Commands

Source: https://github.com/wj-mcat/agent-handbook/blob/master/CLAUDE.md

Provides essential npm commands for managing the Docusaurus project, including installation, starting the development server, building the static site, serving the build, and type checking.

```bash
npm install
npm run start
npm run build
npm run serve
npm run typecheck
npm run clear
```

--------------------------------

### Quantization Training Flow

Source: https://github.com/wj-mcat/agent-handbook/blob/master/llms/05-inference-deployment/03-quantization/05-extreme-low-bit.md

A Mermaid diagram illustrating the workflow for quantization-aware training, starting from optional FP pre-training, proceeding to quantization-friendly training with STE, and finally utilizing specialized 1.58-bit kernels for low-power edge deployment.

```mermaid
flowchart LR
  fp[FP 预训练可选] --> qtrain[量化友好训练 STE]
  qtrain --> kernel[专用 1.58bit kernel]
  kernel --> edge[边缘低功耗]
```

--------------------------------

### Markdown Code Block Example

Source: https://github.com/wj-mcat/agent-handbook/blob/master/CLAUDE.md

Demonstrates the correct syntax for embedding code blocks in Markdown, requiring the programming language to be specified.

```markdown
```python
print("hello")
```
```

--------------------------------

### DeepSeek-R1 Training Flow

Source: https://github.com/wj-mcat/agent-handbook/blob/master/llms/08-technical-reports/01-deepseek/02-deepseek-r1.md

A simplified diagram illustrating the training pipeline for DeepSeek-R1, starting from a base model and progressing through cold-start SFT, GRPO-based RL, and finally distillation into smaller models.

```mermaid
flowchart LR
  v3[DeepSeek-V3 基座] --> cold[冷启动 SFT 少量长 CoT]
cold --> rl[GRPO + 规则/模型奖励]
rl --> r1[DeepSeek-R1]
r1 --> distill[蒸馏到小模型]
```

--------------------------------

### PEFT Library Configuration for Prompt and Prefix Tuning

Source: https://github.com/wj-mcat/agent-handbook/blob/master/llms/04-post-training-alignment/06-peft/02-prefix-prompt-p-tuning.md

Use PromptTuningConfig or PrefixTuningConfig from the 'peft' library to configure these PEFT methods. These configurations specify how the learnable prompts or prefixes are integrated into the model.

```python
from peft import PromptTuningConfig, PrefixTuningConfig
```

--------------------------------

### How to Use This Syllabus

Source: https://github.com/wj-mcat/agent-handbook/blob/master/llms/09-frontier-future/05-conclusion/01-panorama-review.md

Recommended learning paths through the syllabus for different roles, from beginners to inference engineers.

```markdown
- **入门**：1 → 2 → 3.1–3.3 → 4.1 → 5.1。
- **训练工程师**：3 全 + 4.3–4.4 + 3.5–3.6。
- **推理/部署**：5 全 + 8 技术报告选读。
- **Agent 应用**：第六部分 + `docs` + [9.2 记忆](../02-memory-continual-learning/)
```

--------------------------------

### Decision Flowchart for Preference Optimization Methods

Source: https://github.com/wj-mcat/agent-handbook/blob/master/llms/04-post-training-alignment/04-preference-optimization/04-methods-comparison.md

A flowchart to guide the selection of preference optimization methods based on data availability and computational constraints.

```mermaid
flowchart TD
  start[有成对偏好数据?]
  start -->|否| kto[考虑 KTO / 隐式反馈]
  start -->|是| gpu[GPU 紧且要简单?]
  gpu -->|是| simpo[SimPO / ORPO 或 DPO+LoRA]
  gpu -->|否| goal[要最强在线迭代?]
  goal -->|是| rlhf[RLHF 或 RLAIF+RM]
  goal -->|否| dpo[DPO / IPO 离线]
```

--------------------------------

### Self-Study / Implementation Checklist

Source: https://github.com/wj-mcat/agent-handbook/blob/master/llms/09-frontier-future/05-conclusion/01-panorama-review.md

A checklist for self-study and practical implementation of LLMs, covering reading sources, baselining, and documentation.

```markdown
| 步骤 | 动作 |
| --- | --- |
| 1 | 阅读官方 primary source（报告、博客、模型卡） |
| 2 | 固定 prompt 与解码参数，在自有验证集上建基线 |
| 3 | 记录延迟、成本、上下文长度与是否启用思考模式 |
| 4 | 与相邻章节对照，画出与上下游模块的数据流 |
| 5 | 在 [paper-reading](/paper-reading/) 或本大纲相关节做深度笔记 |
```

--------------------------------

### llama.cpp Server Configuration

Source: https://github.com/wj-mcat/agent-handbook/blob/master/llms/05-inference-deployment/06-inference-serving/04-edge-deployment.md

Configure llama.cpp server for edge deployment by setting GPU layers and context size. Ensure sufficient memory for weights and KV cache.

```bash
llama-server -ngl 35 -c 4096
```

--------------------------------

### Build Dataset for ORPO

Source: https://github.com/wj-mcat/agent-handbook/blob/master/paper-reading/rl-algo/orpo.md

This function constructs the dataset for ORPO training by loading data and applying a chat template. It processes prompts, chosen responses, and rejected responses, preparing them in a format suitable for preference optimization. Ensure the 'trl' library is installed and the dataset is accessible.

```python
def build_dataset(tokenizer):
    ds_train = load_dataset(
        script_args.data_name, split="train", cache_dir=script_args.cache_dir
    )

    def chat_template_to_text(sample):
        sample["prompt"] = [
            tokenizer.apply_chat_template(
                [{"role": "user", "content": item_prompt}],
                tokenize=False,
                add_generation_prompt=True,
            )
            for item_prompt in sample["prompt"]
        ]
        sample["chosen"] = [
            item_chosen[1]["content"] for item_chosen in sample["chosen"]
        ]
        sample["rejected"] = [
            item_rejected[1]["content"] for item_rejected in sample["rejected"]
        ]
        return sample

    ds_train = ds_train.map(chat_template_to_text, batched=True, num_proc=8)  # type: ignore

    return ds_train
```

--------------------------------

### MLX Model Loading on Apple Silicon

Source: https://github.com/wj-mcat/agent-handbook/blob/master/llms/05-inference-deployment/06-inference-serving/04-edge-deployment.md

Load quantized models like Qwen or Llama using MLX on Apple Silicon. Utilizes the `mx.metal` backend for optimized performance.

```python
import mlx.core as mx
from mlx_lm import load, generate

# Load model weights for Qwen/Llama
model, tokenizer = load("mlx-community/Qwen-1_8B-Chat-4bit")

# Generate text using the model
template = "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: {prompt} ASSISTANT:"
prompt = "What is the capital of France?"

response = generate(model, tokenizer, template.format(prompt=prompt), max_tokens=100)
print(response)
```

--------------------------------

### Pruning Process Flowchart

Source: https://github.com/wj-mcat/agent-handbook/blob/master/llms/05-inference-deployment/04-model-compression/01-pruning.md

Visualizes the typical workflow for pruning a pre-trained model, from importance scoring to exporting compact weights.

```mermaid
flowchart TD
  train[预训练模型] --> score[重要性评分 Wanda/Magnitude]
  score --> mask[生成 mask]
  mask --> finetune[稀疏微调恢复]
  finetune --> export[导出紧凑权重或稀疏格式]
```

--------------------------------

### Model Synthesis Pipeline for Instruction Data

Source: https://github.com/wj-mcat/agent-handbook/blob/master/llms/04-post-training-alignment/02-instruction-tuning/03-high-quality-instruction-data.md

This diagram illustrates a pipeline for generating instruction data using a language model. It involves a prompt library, generating multiple responses, scoring them with a reward model or rules, and selecting the top response or preference pairs.

```mermaid
flowchart LR
  prompt[场景 prompt 库] --> gen[多采样 N 条回复]
  gen --> score[RM/规则/执行器打分]
  score --> top[取 top-1 或构造 preference pair]
```

--------------------------------

### RLHF Pipeline Stages

Source: https://github.com/wj-mcat/agent-handbook/blob/master/llms/04-post-training-alignment/03-rlhf/01-rlhf-pipeline.md

Illustrates the flow of the RLHF pipeline using a Mermaid diagram, showing the progression from pre-training through SFT, RM, RL (PPO), and deployment.

```mermaid
flowchart LR
  pt[预训练] --> sft[SFT]
  sft --> rm[训练 RM]
  rm --> ppo[PPO / 其他 RL]
  ppo --> deploy[部署]
  sft -.->|作 pi_ref| ppo
```

--------------------------------

### RL Training with Verified Data

Source: https://github.com/wj-mcat/agent-handbook/blob/master/llms/06-reasoning-test-time-compute/03-rl-reasoning/04-self-play.md

Mathematical representation of the reinforcement learning training process using data generated and verified by the current policy.

```latex
\pi_{\theta_{t+1}} \leftarrow \text{Train}\big(\{ (x, y) : \text{Verify}(y)=1,\ y \sim \pi_{\theta_t}(x) \}ig)
```

--------------------------------

### BitNet b1.58 Core Concept

Source: https://github.com/wj-mcat/agent-handbook/blob/master/llms/05-inference-deployment/03-quantization/05-extreme-low-bit.md

Illustrates the mathematical formulation for BitNet b1.58, where weights are constrained to {-1, 0, +1} and forward passes use integer accumulation. This is a core concept for understanding the model's arithmetic.

```latex
Y \approx \sum_j W_j X_j, \quad W_j \in \{-1,0,+1\}
```

--------------------------------

### Open Source vs. Closed Source LLM Landscape (2025 Outlook)

Source: https://github.com/wj-mcat/agent-handbook/blob/master/llms/09-frontier-future/05-conclusion/01-panorama-review.md

A summary of the open-source vs. closed-source LLM landscape, focusing on model weights, training details, and product-level offerings.

```markdown
- **权重开源**：DeepSeek、Qwen、Llama、Mistral 等逼近闭源旗舰。
- **数据/训练细节**：仍多保密；**OLMo** 走全开放科研路线。
- **产品层**：API 聚合、Agent 平台、合规 **仍壁垒**。
```

--------------------------------

### Extended Exercises for LLMs

Source: https://github.com/wj-mcat/agent-handbook/blob/master/llms/09-frontier-future/05-conclusion/01-panorama-review.md

Suggestions for further practice, including reproducing key conclusions, running evaluation scripts, and documenting open questions.

```markdown
- 复现表中 **一行关键结论**（ablation 或小型对照实验）。
- 用 [附录 D 工具](../../10-appendix/04-d-tools-ecosystem) 或 [lm-eval](https://github.com/EleutherAI/lm-evaluation-harness) 跑通评测脚本。
- 将未知参数整理进 [9.5.3 开放问题](../05-conclusion/03-open-questions) 个人笔记。
```

--------------------------------

### 1-bit LLM Core Concept

Source: https://github.com/wj-mcat/agent-handbook/blob/master/llms/05-inference-deployment/03-quantization/05-extreme-low-bit.md

Explains the mathematical formulation for 1-bit LLMs, where weights are constrained to {-1, +1} and require a scaling factor alpha. This is fundamental for understanding binary weight models.

```latex
\hat{W} = \alpha \cdot \text{sign}(W)
```

--------------------------------

### Agent Task Flow Diagram

Source: https://github.com/wj-mcat/agent-handbook/blob/master/llms/07-evaluation/01-benchmarks/05-agent-benchmarks.md

Illustrates the typical workflow of an LLM agent interacting with tools and environments like WebArena or OSWorld.

```mermaid
flowchart TD
  agent[LLM Agent] --> plan[规划]
  plan --> tool[浏览器/终端/API]
  tool --> env[WebArena/OSWorld]
  env --> obs[观察]
  obs --> agent
```

--------------------------------

### Continuous Batching Gantt Chart

Source: https://github.com/wj-mcat/agent-handbook/blob/master/llms/05-inference-deployment/06-inference-serving/02-continuous-batching.md

Illustrates the iteration-level scheduling of requests in continuous batching. It shows how requests are processed, new ones are inserted, and completed ones are released at each GPU iteration.

```mermaid
gantt
  title 连续批处理示意
  section GPU iteration
  请求A decode步 :a1, 0, 1
  请求B decode步 :b1, 1, 2
  请求C 插入 :c1, 2, 3
  请求A 完成释放 :a2, 3, 4
```

--------------------------------

### LLM Inference Deployment Architecture

Source: https://github.com/wj-mcat/agent-handbook/blob/master/llms/05-inference-deployment/06-inference-serving/03-scheduling-load-balancing.md

A Mermaid diagram illustrating the flow of requests from a client through an API Gateway and Scheduler to multiple vLLM replicas.

```mermaid
flowchart TB
  client[Client] --> gw[API Gateway]
  gw --> router[Scheduler]
  router --> r1[Replica 1 vLLM]
  router --> r2[Replica 2 vLLM]
  router --> r3[Replica 3 vLLM]
```