HEP·Xiwu LLM

https://github.com/zhangzhengde0225/xiwu
Admin
HEP·Xiwu LLM is the first Large Language Model specifically designed for High Energy Physics,...
Tokens:8,658
Snippets:28
Trust Score:9.2
Update:1 year ago
Show doc for...
Tokens:
Raw
### Deploy CLI for Interactive Inference

Source: https://context7.com/zhangzhengde0225/xiwu/llms.txt

Starts a command-line interface for direct model interaction. Supports streaming output, conversation history, and various model/hardware configurations. It allows specifying model paths, quantization (8-bit), temperature, max tokens, and display style.

```bash
# Start CLI with default Xiwu model
python run_cli.py \
  --model_path xiwu/xiwu-13b-16k-20240417 \
  --load_8bit False \
  --temperature 0.7 \
  --max_new_tokens 512 \
  --style rich \
  --stream True

# Use Vicuna base model
python run_cli.py \
  --model_path lmsys/vicuna-7b-v1.5 \
  --device cuda \
  --num_gpus 1 \
  --dtype float16

# Multi-GPU with quantization
python run_cli.py \
  --model_path xiwu/xiwu-13b-16k-20240417 \
  --num_gpus 2 \
  --max_gpu_memory 13GiB \
  --load_8bit True
```

--------------------------------

### Fine-tuning Models with XTrainer (Python)

Source: https://context7.com/zhangzhengde0225/xiwu/llms.txt

Illustrates how to configure and start model fine-tuning using the XTrainer Python API. It involves parsing arguments for model, data, and training configurations, then initiating the training process.

```python
# Python training script
from xiwu.modules.trainer.xtrainer import XTrainer, train
from xiwu.configs import ModelArgs, DataArgs, TrainingArgs
import hepai

# Parse arguments
model_args, data_args, training_args = hepai.parse_args(
    (ModelArgs, DataArgs, TrainingArgs)
)

# Configure training
model_args.model_name_or_path = "lmsys/vicuna-13b-v1.5-16k"
data_args.data_path = "/data/datasets/hep_text_v1.0"
training_args.output_dir = "/data/runs/xiwu_finetune"
training_args.num_train_epochs = 2
training_args.per_device_train_batch_size = 4
training_args.learning_rate = 2e-5

# Start training
train()
```

--------------------------------

### Install Project Dependencies with Pip

Source: https://github.com/zhangzhengde0225/xiwu/blob/main/README.md

Installs all necessary Python packages listed in the 'requirements.txt' file. This is a prerequisite for running the Xiwu LLM project.

```bash
pip install -r requirements.txt
```

--------------------------------

### Data Format Conversion

Source: https://context7.com/zhangzhengde0225/xiwu/llms.txt

Convert datasets from various formats into the Xiwu training format. Includes an example for converting Alpaca format.

```APIDOC
## Data Format Conversion

### Description
Convert datasets from various formats to Xiwu training format.

### Method
Python script execution

### Endpoint
`xiwu.data.data_anno_tools.alpaca_52k_to_xiwu_format.convert`

### Parameters
#### Request Body (Implicitly through function arguments)
- `input_file` (string) - Path to the input dataset file (e.g., Alpaca format).
- `output_file` (string) - Path where the converted Xiwu format dataset will be saved.

### Request Example
```python
# Convert Alpaca format to Xiwu format
from xiwu.data.data_anno_tools.alpaca_52k_to_xiwu_format import convert

convert(
    input_file="/data/alpaca-data-conversation.json",
    output_file="/data/alpaca-52k-xiwu-format.json"
)
```

### Response
Converts data to Xiwu format (JSONL).

#### Xiwu Format Example
```json
{
  "id": "identity_0",
  "conversations": [
    {"from": "human", "value": "What is the Higgs boson?"},
    {"from": "gpt", "value": "The Higgs boson is..."}
  ]
}
```

#### Multi-turn Conversation Format Example
```json
{
  "id": "conv_123",
  "conversations": [
    {"from": "human", "value": "Question 1"},
    {"from": "gpt", "value": "Answer 1"},
    {"from": "human", "value": "Question 2"},
    {"from": "gpt", "value": "Answer 2"}
  ]
}
```
```

--------------------------------

### Deploy Xiwu LLM as an API Worker

Source: https://github.com/zhangzhengde0225/xiwu/blob/main/README.md

Starts a worker process to host the Xiwu LLM as an API server. This allows external applications to interact with the model over a network. The model path must be specified.

```bash
python run_worker.py \
  --model_path xiwu/xiwu-13b-16k-20240417
```

--------------------------------

### WorkerModel: Custom Model Integration

Source: https://context7.com/zhangzhengde0225/xiwu/llms.txt

Integrate custom models with the HepAI worker infrastructure for API deployment. This involves defining model and worker arguments and starting the worker server.

```APIDOC
## WorkerModel: Custom Model Integration

### Description
Integrate custom models with HepAI worker infrastructure for API deployment.

### Method
Python script execution

### Endpoint
`hepai.worker.start`

### Parameters
#### Request Body (Implicitly through Python objects)
- `model_args` (ModelArgs)
  - `model_path` (string) - Path to the model weights.
  - `device` (string) - Device to run the model on (e.g., `cuda`).
  - `num_gpus` (int) - Number of GPUs to use.
  - `max_gpu_memory` (string) - Maximum GPU memory per GPU (e.g., `13GiB`).
  - `temperature` (float) - Sampling temperature.
  - `max_new_tokens` (int) - Maximum number of new tokens to generate.
  - `debug` (bool) - Debug mode flag.
- `worker_args` (WorkerArgs)
  - `host` (string) - Host address for the worker.
  - `port` (string) - Port for the worker (`auto` for automatic).
  - `limit_model_concurrency` (int) - Limit for model concurrency.
  - `stream_interval` (float) - Stream interval.
  - `no_register` (bool) - Whether to skip registration.
  - `permissions` (string) - Worker permissions.
  - `test` (bool) - Flag to test the model before deployment.

### Request Example
```python
from xiwu.modules.deployer.worker_model import WorkerModel, ModelArgs, WorkerArgs
from xiwu.configs import BaseArgs
import hepai

# Define model configuration
model_args = ModelArgs(
    model_path="hepai/xiwu-13b-16k-20240417",
    device="cuda",
    num_gpus=2,
    max_gpu_memory="13GiB",
    temperature=0.7,
    max_new_tokens=512,
    debug=False
)

# Define worker configuration
worker_args = WorkerArgs(
    host="0.0.0.0",
    port="auto",
    limit_model_concurrency=5,
    stream_interval=0.0,
    no_register=True,
    permissions="groups: public,PAYG; owner: hepai@ihep.ac.cn",
    test=True
)

# Create worker model
model = WorkerModel(args=model_args)

# Auto-fill metadata
worker_args.description = model.xmodel.get_description()
worker_args.author = model.xmodel.get_author()

# Test before deployment
if worker_args.test:
    from xiwu.utils import general
    general.test_model(model, stream=True)

# Start worker server
hepai.worker.start(model=model, worker_args=worker_args)
```

### Response
Server available at `http://0.0.0.0:{port}/v1/chat/completions`.
```

--------------------------------

### WorkerModel: Custom Model Integration with HepAI

Source: https://context7.com/zhangzhengde0225/xiwu/llms.txt

Integrate custom models with the HepAI worker infrastructure for API deployment. This involves defining model and worker arguments, creating a WorkerModel instance, and starting the worker server. Includes optional model testing before deployment.

```python
from xiwu.modules.deployer.worker_model import WorkerModel, ModelArgs, WorkerArgs
from xiwu.configs import BaseArgs
import hepai

# Define model configuration
model_args = ModelArgs(
    model_path="hepai/xiwu-13b-16k-20240417",
    device="cuda",
    num_gpus=2,
    max_gpu_memory="13GiB",
    temperature=0.7,
    max_new_tokens=512,
    debug=False
)

# Define worker configuration
worker_args = WorkerArgs(
    host="0.0.0.0",
    port="auto",
    limit_model_concurrency=5,
    stream_interval=0.0,
    no_register=True,
    permissions="groups: public,PAYG; owner: hepai@ihep.ac.cn",
    test=True
)

# Create worker model
model = WorkerModel(args=model_args)

# Auto-fill metadata
worker_args.description = model.xmodel.get_description()
worker_args.author = model.xmodel.get_author()

# Test before deployment
if worker_args.test:
    from xiwu.utils import general
    general.test_model(model, stream=True)

# Start worker server
hepai.worker.start(model=model, worker_args=worker_args)

# Server available at http://0.0.0.0:{port}/v1/chat/completions
```

--------------------------------

### XAssembler: Model Loading and Adapter Management

Source: https://context7.com/zhangzhengde0225/xiwu/llms.txt

Core assembler for automatic model detection, adapter registration, and unified model loading.

```APIDOC
## XAssembler: Model Loading and Adapter Management

### Description
The core assembler responsible for automatic model detection, adapter registration, and providing a unified interface for loading models and conversation templates.

### Method
Python SDK

### Endpoint
Not applicable (internal framework component)

### Parameters
#### `ASSEMBLER.load_model` Arguments
- **`model_path`** (string) - Required - Path to the model.
- **`device`** (string) - Optional - Device to use (e.g., 'cuda').
- **`num_gpus`** (integer) - Optional - Number of GPUs to use.
- **`load_8bit`** (boolean) - Optional - Whether to load in 8-bit.
- **`dtype`** (string) - Optional - Data type (e.g., 'float16').

#### `ASSEMBLER.get_conversation_template` Arguments
- **`model_path`** (string) - Required - Path to the model to get the conversation template for.

### Request Example
```python
from xiwu import ASSEMBLER
from xiwu.configs import BaseArgs

# Load model with adapter auto-detection
args = BaseArgs(
    model_path="lmsys/vicuna-7b-v1.5",
    device="cuda",
    num_gpus=1,
    load_8bit=False,
    dtype="float16"
)

model, tokenizer = ASSEMBLER.load_model(
    model_path=args.model_path,
    device=args.device,
    num_gpus=args.num_gpus,
    load_8bit=args.load_8bit,
    dtype=args.dtype
)

# Get conversation template for model
conv = ASSEMBLER.get_conversation_template(args.model_path)
conv.append_message(conv.roles[0], "What is the Higgs boson?")
conv.append_message(conv.roles[1], None)
prompt = conv.get_prompt()
```

### Response
#### Success Response
- **`model`** - The loaded language model.
- **`tokenizer`** - The tokenizer associated with the model.
- **`conv`** - The conversation template object.
- **`prompt`** - The formatted prompt string.
```

--------------------------------

### Deploy CLI for Interactive Inference

Source: https://context7.com/zhangzhengde0225/xiwu/llms.txt

Command-line interface for direct model interaction with streaming output and conversation history.

```APIDOC
## Deploy CLI for Interactive Inference

### Description
Command-line interface for direct model interaction with streaming output and conversation history.

### Method
CLI Commands (Bash)

### Endpoint
Not applicable (CLI)

### Parameters
#### CLI Arguments
- **`--model_path`** (string) - Required - Path to the model.
- **`--load_8bit`** (boolean) - Optional - Whether to load in 8-bit.
- **`--temperature`** (float) - Optional - Sampling temperature.
- **`--max_new_tokens`** (integer) - Optional - Maximum new tokens to generate.
- **`--style`** (string) - Optional - Output style (e.g., 'rich').
- **`--stream`** (boolean) - Optional - Whether to stream output.
- **`--device`** (string) - Optional - Device to use (e.g., 'cuda').
- **`--num_gpus`** (integer) - Optional - Number of GPUs to use.
- **`--dtype`** (string) - Optional - Data type (e.g., 'float16').
- **`--max_gpu_memory`** (string) - Optional - Maximum GPU memory per GPU.

### Request Example
```bash
# Start CLI with default Xiwu model
python run_cli.py \
  --model_path xiwu/xiwu-13b-16k-20240417 \
  --load_8bit False \
  --temperature 0.7 \
  --max_new_tokens 512 \
  --style rich \
  --stream True

# Use Vicuna base model
python run_cli.py \
  --model_path lmsys/vicuna-7b-v1.5 \
  --device cuda \
  --num_gpus 1 \
  --dtype float16

# Multi-GPU with quantization
python run_cli.py \
  --model_path xiwu/xiwu-13b-16k-20240417 \
  --num_gpus 2 \
  --max_gpu_memory 13GiB \
  --load_8bit True
```

### Response
#### Success Response
Interactive CLI output with model responses.

#### Response Example
```
User: Hello!
AI: Hello! How can I help you today?
```
```

--------------------------------

### Prepare Model Weights

Source: https://context7.com/zhangzhengde0225/xiwu/llms.txt

Download and manage pretrained model weights from HuggingFace or custom sources using the prepare_weights.sh script. Specify the model to download or list all available models. Configured via PRETRAINED_WEIGHTS_DIR.

```bash
# List all available models
./prepare_weights.sh --list_all

# Download specific model
./prepare_weights.sh --model lmsys/vicuna-7b-v1.5

# Download Xiwu model
./prepare_weights.sh --model xiwu/xiwu-13b-16k-20240417

# Models are stored in: /data/<USERNAME>/weights/
# Configure path via PRETRAINED_WEIGHTS_DIR in xiwu/configs/constant.py
```

--------------------------------

### Prepare Model Weights

Source: https://context7.com/zhangzhengde0225/xiwu/llms.txt

This section details how to download and manage pretrained model weights from HuggingFace or custom sources using the `prepare_weights.sh` script.

```APIDOC
## Prepare Model Weights

### Description
Download and manage pretrained model weights from HuggingFace or custom sources.

### Method
Shell script execution

### Endpoint
`./prepare_weights.sh`

### Parameters
#### Query Parameters
- `--list_all` (flag) - Lists all available models.
- `--model` (string) - Specifies the model to download (e.g., `lmsys/vicuna-7b-v1.5`, `xiwu/xiwu-13b-16k-20240417`).

### Request Example
```bash
# List all available models
./prepare_weights.sh --list_all

# Download specific model
./prepare_weights.sh --model lmsys/vicuna-7b-v1.5

# Download Xiwu model
./prepare_weights.sh --model xiwu/xiwu-13b-16k-20240417
```

### Response
Models are stored in: `/data/<USERNAME>/weights/`. Configure path via `PRETRAINED_WEIGHTS_DIR` in `xiwu/configs/constant.py`.
```

--------------------------------

### Model Evaluation with XiwuEval

Source: https://context7.com/zhangzhengde0225/xiwu/llms.txt

Shows how to set up and run model evaluation using the XiwuEval class. This involves defining evaluation arguments such as model path, system prompt, and file paths for questions and answers, then executing the evaluation process to generate model responses.

```python
from xiwu.eval.gen_model_answer import XiwuEval
from dataclasses import dataclass, field

@dataclass
class EvalArgs:
    model: str = "xiwu/xiwu-13b-16k-20240417"
    system_prompt: str = "You are a helpful physics assistant."
    question_file: str = "/data/eval/hep_questions.jsonl"
    answer_file: str = "/data/eval/xiwu_answers.jsonl"
    question_begin: int = 0
    question_end: int = None
    num_choices: int = 1

args = EvalArgs()
evaluator = XiwuEval(args)

# Generate answers for questions
evaluator(
    question_file=args.question_file,
    question_begin=args.question_begin,
    question_end=args.question_end,
    num_choices=args.num_choices,
    answer_file=args.answer_file
)

# Question file format (JSONL):
# {"question_id": 1, "text": "What is quantum chromodynamics?", "category": "QCD"}
# {"question_id": 2, "text": "Explain CP violation", "category": "theory"}

# Answer file format (JSONL):

```

--------------------------------

### XAssembler: Model Loading and Adapter Management

Source: https://context7.com/zhangzhengde0225/xiwu/llms.txt

The core assembler for automatic model detection, adapter registration, and unified model loading. It utilizes configuration arguments to specify model paths, device, number of GPUs, quantization, and data type for loading.

```python
from xiwu import ASSEMBLER
from xiwu.configs import BaseArgs

# Load model with adapter auto-detection
args = BaseArgs(
    model_path="lmsys/vicuna-7b-v1.5",
    device="cuda",
    num_gpus=1,
    load_8bit=False,
    dtype="float16"
)

model, tokenizer = ASSEMBLER.load_model(
    model_path=args.model_path,
    device=args.device,
    num_gpus=args.num_gpus,
    load_8bit=args.load_8bit,
    dtype=args.dtype
)

# Get conversation template for model
conv = ASSEMBLER.get_conversation_template(args.model_path)
conv.append_message(conv.roles[0], "What is the Higgs boson?")
conv.append_message(conv.roles[1], None)
prompt = conv.get_prompt()

```

--------------------------------

### Deploy API Worker Server

Source: https://context7.com/zhangzhengde0225/xiwu/llms.txt

Host model as OpenAI-compatible API server with streaming support and automatic port allocation.

```APIDOC
## Deploy API Worker Server

### Description
Host model as an OpenAI-compatible API server with streaming support and automatic port allocation.

### Method
CLI Commands (Bash)

### Endpoint
Not applicable (CLI)

### Parameters
#### CLI Arguments
- **`--model_path`** (string) - Required - Path to the model.
- **`--host`** (string) - Optional - Host address to bind the server to (default: '0.0.0.0').
- **`--port`** (string or integer) - Optional - Port to bind the server to. 'auto' will select a random available port (default: 'auto').
- **`--limit_model_concurrency`** (integer) - Optional - Limit on concurrent model requests.
- **`--test`** (boolean) - Optional - Enable test mode.
- **`--stream_interval`** (float) - Optional - Interval for streaming responses.
- **`--no_register`** (boolean) - Optional - Disable registration with a controller.
- **`--controller_address`** (string) - Optional - Address of the controller.

### Request Example
```bash
# Start worker with automatic port (42902-42999)
python run_worker.py \
  --model_path xiwu/xiwu-13b-16k-20240417 \
  --host 0.0.0.0 \
  --port auto \
  --limit_model_concurrency 5 \
  --test True

# Production deployment with custom port
python run_worker.py \
  --model_path lmsys/vicuna-13b-v1.5 \
  --port 8080 \
  --stream_interval 0.1 \
  --no_register False \
  --controller_address http://localhost:42901
```

### Response
#### Success Response
API server starts listening on the specified or auto-selected port.

#### Response Example
(Server output indicating the port it's listening on, e.g., 'Uvicorn running on http://0.0.0.0:42902')
```

--------------------------------

### Distributed Training with XTrainer (Bash)

Source: https://context7.com/zhangzhengde0225/xiwu/llms.txt

Provides a bash command for initiating multi-GPU training using XTrainer, leveraging FSDP and gradient checkpointing. It specifies model, data paths, batch sizes, learning rates, and FSDP configurations.

```bash
# Multi-GPU training with FSDP
torchrun --nproc_per_node=4 --master_port=20001 xiwu/train/fine_tuning/train_mem.py \
    --model_name_or_path lmsys/vicuna-13b-v1.5-16k \
    --data_path /data/datasets/hep_text_v1.0 \
    --bf16 True \
    --num_train_epochs 2 \
    --per_device_train_batch_size 4 \
    --gradient_accumulation_steps 16 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 1200 \
    --learning_rate 2e-5 \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --fsdp "full_shard auto_wrap offload" \
    --fsdp_config "./xiwu/configs/fsdp_config.json" \
    --model_max_length 2048 \
    --gradient_checkpointing True \
    --lazy_preprocess True
```

--------------------------------

### Configuration Management

Source: https://context7.com/zhangzhengde0225/xiwu/llms.txt

Provides details on the configuration classes for managing model, training, and deployment parameters within the Xiwu project.

```APIDOC
## Configuration Management

### Description
Centralized configuration for model, training, and deployment parameters.

### Method
Python class instantiation and usage

### Endpoint
N/A (Configuration classes)

### Parameters
#### Base Inference Arguments (`BaseArgs` and its subclasses like `CustomArgs`)
- `model_path` (str) - Path to the model weights.
- `device` (str) - Device to run the model on (e.g., `cuda`).
- `num_gpus` (int) - Number of GPUs to use.
- `max_gpu_memory` (str) - Maximum GPU memory per GPU (e.g., `13GiB`).
- `load_8bit` (bool) - Whether to load the model in 8-bit precision.
- `dtype` (str) - Data type for model computation (e.g., `float16`).
- `temperature` (float) - Sampling temperature.
- `top_p` (float) - Top-p sampling parameter.
- `max_new_tokens` (int) - Maximum number of new tokens to generate.
- `stream` (bool) - Whether to enable streaming output.
- `conv_template` (str) - Conversation template to use.
- `gptq_wbits` (int) - Number of bits for GPTQ quantization.
- `awq_wbits` (int) - Number of bits for AWQ quantization.
- `enable_exllama` (bool) - Whether to enable Exllama for inference.

#### Training Arguments (`TrainingArgs`)
- `output_dir` (str) - Directory to save training outputs.
- `num_train_epochs` (int) - Number of training epochs.
- `per_device_train_batch_size` (int) - Batch size per device for training.
- `gradient_accumulation_steps` (int) - Number of steps for gradient accumulation.
- `learning_rate` (float) - Learning rate for the optimizer.
- `warmup_ratio` (float) - Ratio of training steps for learning rate warmup.
- `lr_scheduler_type` (str) - Type of learning rate scheduler.
- `model_max_length` (int) - Maximum sequence length for the model.
- `bf16` (bool) - Whether to use bfloat16 mixed precision.
- `gradient_checkpointing` (bool) - Whether to enable gradient checkpointing.
- `save_strategy` (str) - Strategy for saving checkpoints (e.g., `steps`).
- `save_steps` (int) - Number of steps between saving checkpoints.

#### Data Arguments (`DataArgs`)
- `data_path` (str) - Path to the training dataset.
- `eval_data_path` (str) - Path to the evaluation dataset.
- `lazy_preprocess` (bool) - Whether to use lazy preprocessing for data.

### Request Example
```python
from xiwu.configs.configs import BaseArgs, ModelArgs, DataArgs, TrainingArgs
from xiwu.configs.constant import RUNS_DIR, DATASETS_DIR, PRETRAINED_WEIGHTS_DIR
from dataclasses import dataclass, field

# Base inference arguments (example using a custom derived class)
@dataclass
class CustomArgs(BaseArgs):
    model_path: str = "xiwu/xiwu-13b-16k-20240417"
    device: str = "cuda"
    num_gpus: int = 2
    max_gpu_memory: str = "13GiB"
    load_8bit: bool = False
    dtype: str = "float16"
    temperature: float = 0.7
    top_p: float = 0.9
    max_new_tokens: int = 512
    stream: bool = True
    conv_template: str = "xiwu"
    gptq_wbits: int = 4
    awq_wbits: int = 4
    enable_exllama: bool = False

# Training arguments
training_args = TrainingArgs(
    output_dir=RUNS_DIR,
    num_train_epochs=2,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=16,
    learning_rate=2e-5,
    warmup_ratio=0.03,
    lr_scheduler_type="cosine",
    model_max_length=2048,
    bf16=True,
    gradient_checkpointing=True,
    save_strategy="steps",
    save_steps=1200
)

# Data arguments
data_args = DataArgs(
    data_path=f"{DATASETS_DIR}/hep_text_v1.0",
    eval_data_path=f"{DATASETS_DIR}/hep_eval.json",
    lazy_preprocess=True
)
```

### Response
These classes define structures for organizing and passing configuration parameters throughout the Xiwu project.
```

--------------------------------

### XModel: Unified Model Interface for Inference

Source: https://context7.com/zhangzhengde0225/xiwu/llms.txt

Shows how to use the XModel class for high-level model interaction, including lazy loading, automatic adapter selection, and performing inference with streaming output. It also demonstrates how to retrieve model metadata.

```python
from xiwu.modules.base.xmodel import XModel
from xiwu.configs.configs import BaseArgs

# Initialize with lazy loading
args = BaseArgs(
    model_path="xiwu/xiwu-13b-16k-20240417",
    lazy_loading=True,
    temperature=0.7,
    top_p=0.9,
    max_new_tokens=512,
    stream=True
)

xmodel = XModel(args=args)

# Model loads automatically on first use
messages = [
    {"role": "user", "content": "Generate BOSS code for particle tracking"}
]

# Convert messages to prompt
prompt = xmodel.oai_messages2prompt(messages=messages)

# Run inference with streaming
for response in xmodel.inference(
    prev_text=prompt,
    temperature=0.7,
    max_new_tokens=512
):
    if response.get("text"):
        print(response["text"], end="", flush=True)

# Get model metadata
print(f"Description: {xmodel.get_description()}")
print(f"Author: {xmodel.get_author()}")
print(f"Context length: {xmodel.context_len}")
```

--------------------------------

### Configuration Management for Xiwu

Source: https://context7.com/zhangzhengde0225/xiwu/llms.txt

Defines configuration classes for Xiwu, including base inference arguments, training arguments, and data arguments. These dataclasses allow for centralized management of model, training, and deployment parameters. Constants for directories are also provided.

```python
from xiwu.configs.configs import BaseArgs, ModelArgs, DataArgs, TrainingArgs
from xiwu.configs.constant import RUNS_DIR, DATASETS_DIR, PRETRAINED_WEIGHTS_DIR
from dataclasses import dataclass, field

# Base inference arguments
@dataclass
class CustomArgs(BaseArgs):
    model_path: str = "xiwu/xiwu-13b-16k-20240417"
    device: str = "cuda"
    num_gpus: int = 2
    max_gpu_memory: str = "13GiB"
    load_8bit: bool = False
    dtype: str = "float16"
    temperature: float = 0.7
    top_p: float = 0.9
    max_new_tokens: int = 512
    stream: bool = True
    conv_template: str = "xiwu"

    # Quantization
    gptq_wbits: int = 4
    awq_wbits: int = 4
    enable_exllama: bool = False

# Training arguments
training_args = TrainingArgs(
    output_dir=RUNS_DIR,
    num_train_epochs=2,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=16,
    learning_rate=2e-5,
    warmup_ratio=0.03,
    lr_scheduler_type="cosine",
    model_max_length=2048,
    bf16=True,
    gradient_checkpointing=True,
    save_strategy="steps",
    save_steps=1200
)

# Data arguments
data_args = DataArgs(
    data_path=f"{DATASETS_DIR}/hep_text_v1.0",
    eval_data_path=f"{DATASETS_DIR}/hep_eval.json",
    lazy_preprocess=True
)
```

--------------------------------

### Deploy API Worker Server

Source: https://context7.com/zhangzhengde0225/xiwu/llms.txt

Hosts a model as an OpenAI-compatible API server, supporting streaming output and automatic port allocation within a specified range. It can be configured with host, port, model concurrency limits, and testing flags.

```bash
# Start worker with automatic port (42902-42999)
python run_worker.py \
  --model_path xiwu/xiwu-13b-16k-20240417 \
  --host 0.0.0.0 \
  --port auto \
  --limit_model_concurrency 5 \
  --test True

# Production deployment with custom port
python run_worker.py \
  --model_path lmsys/vicuna-13b-v1.5 \
  --port 8080 \
  --stream_interval 0.1 \
  --no_register False \
  --controller_address http://localhost:42901
```

--------------------------------

### Request API with HepAI Client

Source: https://context7.com/zhangzhengde0225/xiwu/llms.txt

OpenAI-compatible API client for chat completions with streaming and non-streaming modes.

```APIDOC
## Request API with HepAI Client

### Description
An OpenAI-compatible API client for performing chat completions, supporting both streaming and non-streaming responses.

### Method
Python SDK

### Endpoint
`http://localhost:PORT/v1` (where PORT is the port the API worker server is running on)

### Parameters
#### Initialization
- **`api_key`** (string) - API key (default: 'default').
- **`base_url`** (string) - The base URL of the API server.

#### Chat Completion Parameters
- **`model`** (string) - Required - The model to use for completion (e.g., 'default').
- **`messages`** (array) - Required - A list of message objects representing the conversation history.
  - **`role`** (string) - 'system', 'user', or 'assistant'.
  - **`content`** (string) - The message content.
- **`temperature`** (float) - Optional - Sampling temperature (default: 0.7).
- **`max_tokens`** (integer) - Optional - Maximum number of tokens to generate.
- **`stream`** (boolean) - Optional - Whether to stream the response (default: False).

### Request Example
```python
from hepai import HepAI, Stream

# Initialize client
client = HepAI(
    api_key="default",
    base_url="http://localhost:42902/v1"
)

# Non-streaming request
messages = [
    {"role": "system", "content": "You are a physics expert."},
    {"role": "user", "content": "Explain quantum entanglement"},
    {"role": "assistant", "content": "Quantum entanglement is..."},
    {"role": "user", "content": "How is it used in particle physics?"}
]

response = client.chat.completions.create(
    model="default",
    messages=messages,
    temperature=0.7,
    max_tokens=512,
    stream=False
)
content = response.choices[0].message.content
print(content)

# Streaming request
stream_response = client.chat.completions.create(
    model="default",
    messages=messages,
    stream=True
)

if isinstance(stream_response, Stream):
    for chunk in stream_response:
        if chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end='', flush=True)
    print()
```

### Response
#### Success Response (200)
- **`choices`** (array) - List of completion choices.
  - **`message`** (object)
    - **`content`** (string) - The generated message content.
- **`model`** (string) - The model used.
- **`usage`** (object) - Token usage statistics.

#### Response Example (Non-streaming)
```json
{
  "choices": [
    {
      "message": {
        "content": "In particle physics, quantum entanglement is utilized in various experiments and theoretical frameworks, such as..."
      }
    }
  ],
  "model": "default",
  "usage": {
    "prompt_tokens": 100,
    "completion_tokens": 50,
    "total_tokens": 150
  }
}
```

#### Response Example (Streaming)
```
In particle physics, quantum entanglement is utilized in various experiments and theoretical frameworks, such as...
```
```

--------------------------------

### Register Custom Model Adapter with Xiwu

Source: https://context7.com/zhangzhengde0225/xiwu/llms.txt

Demonstrates how to register a custom model adapter by subclassing BaseModelAdapter and implementing match and get_default_conv_template methods. This allows Xiwu to recognize and use custom models.

```python
from xiwu.modules.base.base_adapter import BaseModelAdapter
from xiwu.apis.fastchat_api import Conversation

class CustomAdapter(BaseModelAdapter):
    def match(self, model_path):
        return "custom" in model_path.lower()

    def get_default_conv_template(self):
        return Conversation(
            name="custom",
            system_template="[SYSTEM]\n{system_message}",
            roles=("USER", "ASSISTANT"),
            sep="\n"
        )

ASSEMBLER.register_model_adapter(CustomAdapter, index=0)
```

--------------------------------

### Seed Fission for QA Dataset Generation

Source: https://context7.com/zhangzhengde0225/xiwu/llms.txt

Demonstrates the use of the SeedFission class to generate synthetic Question-Answering datasets. It involves initializing the system with domain and language, then running the generation process from seed questions with specified parameters for quantity and saving intervals.

```python
from xiwu.data.data_acquisition_tools.seed_fission.seed_fission import SeedFission

# Initialize seed fission system
sf = SeedFission(
    domain="high_energy_physics",
    language="zh",
    output_file="/data/datasets/hep_qa_generated.json"
)

# Generate QA pairs from seed
seed_question = "What is the Standard Model of particle physics?"

sf.run(
    seed=seed_question,
    max_entities=2000,  # Generate up to 2000 QA pairs
    num_to_gen=100,     # Generate 100 pairs in this run
    save_interval=10    # Save every 10 generations
)

# Exponential growth: 10 fissions → 2^11 - 1 = 2047 QA pairs
# Access generated data
print(f"Total QA pairs: {sf.num_enrities}")
print(f"Questions: {sf.exist_questions[:5]}")
```

--------------------------------

### Data Format Conversion to Xiwu Training Format

Source: https://context7.com/zhangzhengde0225/xiwu/llms.txt

Convert datasets from formats like Alpaca to the Xiwu training format (JSONL). This function takes an input file and an output file path for the conversion. The Xiwu format supports multi-turn conversations.

```python
# Convert Alpaca format to Xiwu format
from xiwu.data.data_anno_tools.alpaca_52k_to_xiwu_format import convert

convert(
    input_file="/data/alpaca-data-conversation.json",
    output_file="/data/alpaca-52k-xiwu-format.json"
)

# Xiwu format (JSONL):
# {
#   "id": "identity_0",
#   "conversations": [
#     {"from": "human", "value": "What is the Higgs boson?"},
#     {"from": "gpt", "value": "The Higgs boson is..."}
#   ]
# }

# Multi-turn conversation format:
# {
#   "id": "conv_123",
#   "conversations": [
#     {"from": "human", "value": "Question 1"},
#     {"from": "gpt", "value": "Answer 1"},
#     {"from": "human", "value": "Question 2"},
#     {"from": "gpt", "value": "Answer 2"}
#   ]
# }
```

--------------------------------

### Run Xiwu LLM via Command Line Interface

Source: https://github.com/zhangzhengde0225/xiwu/blob/main/README.md

Launches the Command Line Interface (CLI) for interacting with the Xiwu LLM. Users can specify the model path and whether to load in 8-bit precision. The model assembler automatically locates weights in the configured directory.

```bash
python run_cli.py \
  --model_path xiwu/xiwu-13b-16k-20240417 \
  --load_8bit False
```

--------------------------------

### Request API with HepAI Client

Source: https://context7.com/zhangzhengde0225/xiwu/llms.txt

Provides an OpenAI-compatible client for chat completions, supporting both streaming and non-streaming requests. It requires initialization with an API key and base URL, and accepts messages in a standard format.

```python
from hepai import HepAI, Stream

# Initialize client
client = HepAI(
    api_key="default",
    base_url="http://localhost:42902/v1"
)

# Non-streaming request
messages = [
    {"role": "system", "content": "You are a physics expert."},
    {"role": "user", "content": "Explain quantum entanglement"},
    {"role": "assistant", "content": "Quantum entanglement is..."},
    {"role": "user", "content": "How is it used in particle physics?"}
]

response = client.chat.completions.create(
    model="default",
    messages=messages,
    temperature=0.7,
    max_tokens=512,
    stream=False
)
content = response.choices[0].message.content
print(content)

# Streaming request
stream_response = client.chat.completions.create(
    model="default",
    messages=messages,
    stream=True
)

if isinstance(stream_response, Stream):
    for chunk in stream_response:
        if chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end='', flush=True)
    print()
```

--------------------------------

### Download Trained Model Weights

Source: https://github.com/zhangzhengde0225/xiwu/blob/main/README.md

Downloads specific trained weights for a given model, such as 'lmsys/vicuna-7b-v1.5'. The script allows listing all available weights before downloading. Weights are typically stored in a configured directory.

```bash
./prepare_weights.sh --list_all 
./prepare_weights.sh --model lmsys/vicuna-7b-v1.5
```

--------------------------------

### Train Xiwu LLM on Custom Data

Source: https://github.com/zhangzhengde0225/xiwu/blob/main/README.md

Initiates the training process for the Xiwu LLM using custom data. This script likely involves a comprehensive training pipeline, including data preparation and model fine-tuning, to create a new model variant.

```bash
bash scripts/train_xiwu.sh
```

--------------------------------

### Request Data from Xiwu LLM API

Source: https://github.com/zhangzhengde0225/xiwu/blob/main/README.md

A client script to send requests to a running Xiwu LLM API worker. It requires specifying the 'base_url' of the worker and supports streaming API responses. This script demonstrates how to access the deployed model.

```bash
python request_api.py
```
Tokens:
Raw