### Stage 1: Supervised Fine-Tuning (SFT)

Source: https://github.com/freedomintelligence/huatuogpt-o1/blob/main/README.md

Command to initiate supervised fine-tuning for the model on an 8-GPU setup. Ensure DeepSpeed Zero3 configuration is used.

```bash
accelerate launch --config_file ./configs/deepspeed_zero3.yaml \
    --num_processes 8  \
    --num_machines 1 \
    --machine_rank 0 \
    --deepspeed_multinode_launcher standard SFT_stage1.py \
    --model_path [meta-llama/Llama-3.1-8B-Instruct] \
    --data_path [FreedomIntelligence/medical-o1-reasoning-SFT] 
```

--------------------------------

### Stage 2: Reinforcement Learning (RL) with PPO

Source: https://github.com/freedomintelligence/huatuogpt-o1/blob/main/README.md

Command to start reinforcement learning using PPO for an 8B model on an 8-GPU A100 machine. Requires a pre-downloaded reward model.

```bash
accelerate launch \
	--num_processes 8 \
	--num_machines 1 \
	--machine_rank 0 \
    --config_file ./configs/deepspeed_zero3.yaml \
	--deepspeed_multinode_launcher standard RL_stage2.py \
    --model_name_or_path [FreedomIntelligence/HuatuoGPT-o1-8B] \
    --reward_model_path [FreedomIntelligence/medical_o1_verifier_3B] \
    --value_model_path [meta-llama/Llama-3.2-3B-Instruct] \
    --dataset_name  [FreedomIntelligence/medical-o1-verifiable-problem]\
    --response_length 1300 \
    --temperature 0.5 \
    --local_rollout_forward_batch_size 8 \
    --num_ppo_epochs 3 \
    --num_mini_batches 1 \
    --total_episodes 20000 \
    --per_device_train_batch_size 1 \
    --gradient_accumulation_steps 16 \
    --bf16 True \
    --output_dir ./ckpts \
    --save_strategy steps \
    --save_step 20 \
    --save_total_limit 1 \
    --eval_strategy steps \
    --eval_steps 20 \
    --kl_coef 0.03 \
    --learning_rate 5e-7 \
    --warmup_ratio 0.05 \
    --gradient_checkpointing True \
    --dataloader_num_workers 4 \
    --run_name ppo_medical_o1_8B \
    --num_sample_generations -1 \
    --report_to wandb
```

--------------------------------

### Deploy Model with Sglang

Source: https://github.com/freedomintelligence/huatuogpt-o1/blob/main/README.md

Deploys a specified model using Sglang. Ensure Sglang is installed and the model path is correct. Logs are redirected to a file.

```bash
log_num=0
model_name="FreedomIntelligence/HuatuoGPT-o1-8B" # Path to the model you are deploying
port=28${log_num}35
CUDA_VISIBLE_DEVICES=0  python -m sglang.launch_server --model-path $model_name --port $port --mem-fraction-static 0.8 --dp 1 --tp 1  > sglang${log_num}.log 2>&1 &
```

--------------------------------

### Direct Inference with HuatuoGPT-o1

Source: https://github.com/freedomintelligence/huatuogpt-o1/blob/main/README.md

This Python snippet demonstrates how to load and perform direct inference using a HuatuoGPT-o1 model with the Hugging Face Transformers library. Ensure you have the library installed and the model name is correct.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("FreedomIntelligence/HuatuoGPT-o1-8B",torch_dtype="auto",device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("FreedomIntelligence/HuatuoGPT-o1-8B")

input_text = "How to stop a cough?"
messages = [{"role": "user", "content": input_text}]

inputs = tokenizer(tokenizer.apply_chat_template(messages, tokenize=False,add_generation_prompt=True
), return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=2048)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

--------------------------------

### Search for Complex Reasoning Paths for SFT

Source: https://github.com/freedomintelligence/huatuogpt-o1/blob/main/README.md

This script searches for complex reasoning paths suitable for SFT. Adjust search parameters like max attempts and depth as needed. Requires data path, model name, and API key.

```bash
python search_for_complex_reasoning_path.py --data_path  data/demo_data.json --efficient_search True  --max_search_attempts 1 --max_search_depth 2 --model_name gpt-4o --api_key [your api key]
```

--------------------------------

### Run Model Evaluation

Source: https://github.com/freedomintelligence/huatuogpt-o1/blob/main/README.md

Executes the evaluation script for the deployed model. Requires the model name, evaluation file, and port number. Can use `--strict_prompt` for more precise extraction.

```bash
python evaluation/eval.py --model_name $model_name  --eval_file evaluation/data/eval_data.json --port $port 
```

--------------------------------

### Construct Verifiable Medical Problems

Source: https://github.com/freedomintelligence/huatuogpt-o1/blob/main/README.md

Use this script to construct verifiable medical problems from multi-choice questions. Ensure you have the necessary data path and API key configured.

```bash
python construct_verifiable_medical_problems.py --data_path  data/demo_data.json --filter_data --model_name gpt-4o --api_key [your api key]
```

--------------------------------

### Stop Sglang Server

Source: https://github.com/freedomintelligence/huatuogpt-o1/blob/main/README.md

Stops the Sglang service and releases GPU memory after evaluation is complete.

```bash
bash evaluation/kill_sglang_server.sh
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.