ChatGLM3

https://github.com/zai-org/chatglm3
Admin
ChatGLM3 is a series of pre-trained dialogue models jointly released by Zhipu AI and Tsinghua KEG,...
Tokens:19,388
Snippets:237
Trust Score:7.8
License:Apache-2.0
Update:3 months ago
Show doc for...
Tokens:
Raw
### Install Python Dependencies for ChatGLM3 Example

Source: https://github.com/zai-org/chatglm3/blob/main/tensorrt_llm_demo/README.md

Navigates to the ChatGLM3 example directory and installs required Python packages. Also ensures git-lfs is available.

```bash
cd ./examples/chatglm
pip install -r requirements.txt
apt-get update
apt-get install git-lfs
```

--------------------------------

### Install Dependencies

Source: https://github.com/zai-org/chatglm3/blob/main/finetune_demo/README.md

Installs the necessary dependencies for running the fine-tuning examples. Ensure you have Python 3.10 or higher.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install TensorRT-LLM and Dependencies

Source: https://github.com/zai-org/chatglm3/blob/main/tensorrt_llm_demo/README.md

Installs git and git-lfs, clones the TensorRT-LLM repository, checks out a specific version, and initializes submodules.

```bash
apt-get update && apt-get -y install git git-lfs

git clone https://github.com/NVIDIA/TensorRT-LLM.git
cd TensorRT-LLM

# 本流程将使用 v0.7.0 Release 版本
git checkout tags/v0.7.0 -b release/0.7.0
git submodule update --init --recursive
git lfs install
git lfs pull
```

--------------------------------

### Start Gradio Web Demo

Source: https://github.com/zai-org/chatglm3/blob/main/README.md

Launch the Gradio-based web demo for ChatGLM3 by running the provided Python script.

```shell
python web_demo_gradio.py
```

--------------------------------

### Start Streamlit Web Demo

Source: https://github.com/zai-org/chatglm3/blob/main/README.md

Launch the Streamlit-based web demo for ChatGLM3 by running the provided Python script.

```shell
streamlit run web_demo_streamlit.py
```

--------------------------------

### Install Dependencies with Conda

Source: https://github.com/zai-org/chatglm3/blob/main/composite_demo/README.md

Use these commands to create a Conda environment and install necessary dependencies for the ChatGLM3 demo. Ensure you have Python 3.10 or higher.

```bash
conda create -n chatglm3-demo python=3.10
conda activate chatglm3-demo
pip install -r requirements.txt
```

--------------------------------

### Dataset Format Example

Source: https://github.com/zai-org/chatglm3/blob/main/finetune_demo/README.md

Example of a single data instance in the required JSON format for fine-tuning. Ensure your dataset includes both training and validation files.

```json
{"conversations": [{"role": "user", "content": "类型#裙*裙长#半身裙"}, {"role": "assistant", "content": "这款百搭时尚的仙女半身裙，整体设计非常的飘逸随性，穿上之后每个女孩子都能瞬间变成小仙女啦。料子非常的轻盈，透气性也很好，穿到夏天也很舒适。"}]}
```

--------------------------------

### Launch Conversational Demo with TensorRT-LLM

Source: https://github.com/zai-org/chatglm3/blob/main/tensorrt_llm_demo/README.md

Start the command-line interface (CLI) conversational demo using `tensorrt_llm_cli_demo.py`. This script requires the tokenizer and engine directories.

```bash
python3 tensorrt_llm_cli_demo.py --tokenizer_dir chatglm3_6b --engine_dir trt_engines/chatglm3_6b/fp16/1-gpu
```

--------------------------------

### Define Tools for ChatGLM3-6B

Source: https://github.com/zai-org/chatglm3/blob/main/tools_using_demo/README.md

Prepare tool definitions including name, description, and parameters. Ensure the format matches the example for optimal performance.

```python
tools = [
    {
        "name": "track",
        "description": "追踪指定股票的实时价格",
        "parameters": {
            "type": "object",
            "properties": {
                "symbol": {
                    "description": "需要追踪的股票代码"
                }
            },
            "required": ['symbol']
        }
    },
    {
        "name": "text-to-speech",
        "description": "将文本转换为语音",
        "parameters": {
            "type": "object",
            "properties": {
                "text": {
                    "description": "需要转换成语音的文本"
                },
                "voice": {
                    "description": "要使用的语音类型（男声、女声等）"
                },
                "speed": {
                    "description": "语音的速度（快、中等、慢等）"
                }
            },
            "required": ['text']
        }
    }
]
system_info = {"role": "system", "content": "Answer the following questions as best as you can. You have access to the following tools:", "tools": tools}
```

--------------------------------

### Clone Repository and Install Dependencies

Source: https://github.com/zai-org/chatglm3/blob/main/Intel_device_demo/openvino_demo/README.md

Clone the OpenVINO GLM3 inference repository and install necessary dependencies. It's recommended to use a virtual environment.

```bash
git clone https://github.com/OpenVINO-dev-contest/chatglm3.openvino.git
cd chatglm3.openvino
```

```bash
python3 -m venv openvino_env
source openvino_env/bin/activate
python3 -m pip install --upgrade pip
pip install wheel setuptools
pip install -r requirements.txt
```

--------------------------------

### Python API Request Example

Source: https://github.com/zai-org/chatglm3/blob/main/README_en.md

Example of how to test the API calls using a Python script.

```APIDOC
## Python API Request Test

### Description
This example shows how to run a Python script to test the API. If the script executes successfully, the model should return a story.

### Command
```bash
cd openai_api_demo
python openai_api_request.py
```
```

--------------------------------

### Dataset Format Example

Source: https://github.com/zai-org/chatglm3/blob/main/finetune_demo/README_en.md

This is an example of the conversation format expected for the dataset. Ensure your data adheres to this structure for successful fine-tuning.

```json
{
"conversations": [{"role": "user", "content": "Type#skirt*skirt length#skirt"}, {"role": "assistant", "content": "This is versatile Fashionable fairy skirt, the overall design is very elegant and casual. Every girl can instantly turn into a fairy after wearing it. The material is very light and breathable, making it very comfortable to wear in summer."}]
}
```

--------------------------------

### Finetune ChatGLM3 with DeepSpeed (Multi-GPU)

Source: https://github.com/zai-org/chatglm3/blob/main/finetune_demo/README.md

Execute finetuning on a single machine with multiple GPUs using DeepSpeed. Ensure DeepSpeed is installed.

```bash
cd finetune_demo
OMP_NUM_THREADS=1 torchrun --standalone --nnodes=1 --nproc_per_node=8  finetune_hf.py  data/AdvertiseGen/  THUDM/chatglm3-6b  configs/lora.yaml
```

--------------------------------

### Run TensorRT-LLM Docker Image

Source: https://github.com/zai-org/chatglm3/blob/main/tensorrt_llm_demo/README.md

Starts the TensorRT-LLM Docker container in release mode.

```bash
make -C docker release_run
```

--------------------------------

### Define Tools for ChatGLM3

Source: https://github.com/zai-org/chatglm3/blob/main/tools_using_demo/README_en.md

Prepare the description information for the tools that ChatGLM3 can use. Ensure the format matches the example for optimal performance.

```python
tools = [
    {
        "name": "track",
        "description": "Track the real-time price of a specified stock",
        "parameters": {
            "type": "object",
            "properties": {
                "symbol": {
                    "description": "The stock code that needs to be tracked"
                }
            },
            "required": ['symbol']
        }
    },
    {
        "name": "text-to-speech",
        "description": "Convert text to speech",
        "parameters": {
            "type": "object",
            "properties": {
                "text": {
                    "description": "The text that needs to be converted into speech"
                },
                "voice": {
                    "description": "The type of voice to use (male, female, etc.)"
                },
                "speed": {
                    "description": "The speed of the speech (fast, medium, slow, etc.)"
                }
            },
            "required": ['text']
        }
    }
]
system_info = {"role": "system", "content": "Answer the following questions as best as you can. You have access to the following tools:", "tools": tools}
```

--------------------------------

### Initialize and Use ChatGLM3 Model

Source: https://github.com/zai-org/chatglm3/blob/main/README_en.md

Initialize the ChatGLM3 model and tokenizer from Hugging Face, and start a conversation. Ensure 'cuda' is available if specifying the device.

```python
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True)
model = AutoModel.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True, device='cuda')
model = model.eval()
response, history = model.chat(tokenizer, "你好", history=[])
print(response)

response, history = model.chat(tokenizer, "What should I do if I can't sleep at night?", history=history)
print(response)
```

--------------------------------

### Load ChatGLM3 Across Multiple GPUs

Source: https://github.com/zai-org/chatglm3/blob/main/DEPLOYMENT_en.md

Distribute the model across multiple GPUs when individual GPU VRAM is insufficient. Install the 'accelerate' library first. This example loads the model onto 2 GPUs.

```python
from utils import load_model_on_gpus

model = load_model_on_gpus("THUDM/chatglm3-6b", num_gpus=2)
```

--------------------------------

### Load ChatGLM3 on Mac with MPS Backend

Source: https://github.com/zai-org/chatglm3/blob/main/DEPLOYMENT_en.md

Utilize the MPS backend for GPU acceleration on Macs with Apple Silicon or AMD GPUs. Ensure you have the correct PyTorch-Nightly version installed. This example assumes the model is loaded locally.

```python
model = AutoModel.from_pretrained("your local path", trust_remote_code=True).to('mps')
```

--------------------------------

### Install Jupyter Kernel for Code Interpreter

Source: https://github.com/zai-org/chatglm3/blob/main/composite_demo/README.md

Install the Jupyter kernel for the 'chatglm3-demo' environment to enable the Code Interpreter functionality.

```bash
ipython kernel install --name chatglm3-demo --user
```

--------------------------------

### Run Command-Line Chat Demo

Source: https://github.com/zai-org/chatglm3/blob/main/README.md

Start an interactive command-line chat session with ChatGLM3 using the provided Python script. Use 'clear' to reset history and 'stop' to exit.

```shell
python cli_demo.py
```

--------------------------------

### Register a Custom Tool

Source: https://github.com/zai-org/chatglm3/blob/main/composite_demo/README.md

Example of registering a new tool for the ChatGLM3 model. Use the '@register_tool' decorator and define parameters using 'Annotated' for type, description, and required status.

```python
@register_tool
def get_weather(
    city_name: Annotated[str, 'The name of the city to be queried', True],
) -> str:
    """
    Get the weather for `city_name` in the following week
    """
    ...
```

--------------------------------

### ChatGLM3 Multi-turn Conversation Structure

Source: https://github.com/zai-org/chatglm3/blob/main/PROMPT.md

This example shows the overall structure of a multi-turn conversation in ChatGLM3, including system, user, and assistant roles. Ensure proper formatting for each turn.

```text
<|system|>
You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.
<|user|>
Hello
<|assistant|>
Hello, I'm ChatGLM3. What can I assist you today?
```

--------------------------------

### Standard OpenAI Interface Agent-Chat Curl Test

Source: https://github.com/zai-org/chatglm3/blob/main/README_en.md

Example of testing the Chat Completions API with tool definitions using curl.

```APIDOC
## Curl Test for Agent-Chat with Tools

### Description
This example shows how to use curl to interact with the Chat Completions API, including defining tools that the model can utilize for specific tasks like tracking stock prices or performing calculations.

### Method
POST

### Endpoint
http://127.0.0.1:8000/v1/chat/completions

### Request Example
```bash
curl -X POST "http://127.0.0.1:8000/v1/chat/completions" \
-H "Content-Type: application/json" \
-d "{\"model\": \"chatglm3-6b\", \"messages\": [{\"role\": \"user\", \"content\": \"37乘以8加7除2等于多少？\"}], \"tools\": [{\"name\": \"track\", \"description\": \"追踪指定股票的实时价格\",
          \"parameters\": {\"type\": \"object\", \"properties\": {\"symbol\": {\"description\": \"需要追踪的股票代码\"}},
                         \"required\": []}},
         {\"name\": \"Calculator\", \"description\": \"数学计算器，计算数学问题\",
          \"parameters\": {\"type\": \"object\", \"properties\": {\"symbol\": {\"description\": \"要计算的数学公式\"}},
                         \"required\": []}}
         ], \"stream\": true, \"max_tokens\": 100, \"temperature\": 0.8, \"top_p\": 0.8}"
```
```

--------------------------------

### Test Chat Completions with cURL

Source: https://github.com/zai-org/chatglm3/blob/main/README.md

Use cURL to send a POST request to the API for chat completions. This example demonstrates a basic story generation request.

```shell
curl -X POST "http://127.0.0.1:8000/v1/chat/completions" \
-H "Content-Type: application/json" \
-d "{\"model\": \"chatglm3-6b\", \"messages\": [{\"role\": \"system\", \"content\": \"You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.\"}, {\"role\": \"user\", \"content\": \"你好，给我讲一个故事，大概100字\"}], \"stream\": false, \"max_tokens\": 100, \"temperature\": 0.8, \"top_p\": 0.8}"
```

--------------------------------

### Register a Custom Tool

Source: https://github.com/zai-org/chatglm3/blob/main/composite_demo/README_en.md

Example of registering a new tool named 'get_weather' using the '@register_tool' decorator. The function name serves as the tool name, and its docstring describes the tool. Parameters are annotated using 'Annotated'.

```python
from typing import Annotated

@register_tool
def get_weather(
    city_name: Annotated[str, 'The name of the city to be queried', True],
) -> str:
    """
    Get the weather for `city_name` in the following week
    """
    ...
```

--------------------------------

### Load and Use ChatGLM3 Model for Chat

Source: https://github.com/zai-org/chatglm3/blob/main/README.md

Load the ChatGLM3 model and tokenizer from Hugging Face and use the chat method for dialogue. Ensure PyTorch is installed according to official documentation.

```python
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True)
model = AutoModel.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True, device='cuda')
model = model.eval()
response, history = model.chat(tokenizer, "你好", history=[])
print(response)
response, history = model.chat(tokenizer, "晚上睡不着应该怎么办", history=history)
print(response)
```

--------------------------------

### Clone ChatGLM3 Model Locally

Source: https://github.com/zai-org/chatglm3/blob/main/README.md

Clone the ChatGLM3 model parameters from Hugging Face Hub to your local machine for offline use. This requires Git LFS to be installed.

```shell
git clone https://huggingface.co/THUDM/chatglm3-6b
```

--------------------------------

### Test Agent Chat with cURL (Tool Use)

Source: https://github.com/zai-org/chatglm3/blob/main/README.md

Test agent chat completions using cURL, including the definition of tools for the agent to use. This example shows a mathematical calculation request.

```shell
curl -X POST "http://127.0.0.1:8000/v1/chat/completions" \
-H "Content-Type: application/json" \
-d "{\"model\": \"chatglm3-6b\", \"messages\": [{\"role\": \"user\", \"content\": \"37乘以8加7除2等于多少？\"}], \"tools\": [{\"name\": \"track\", \"description\": \"追踪指定股票的实时价格\",
          \"parameters\": {\"type\": \"object\", \"properties\": {\"symbol\": {\"description\": \"需要追踪的股票代码\"}},
                         \"required\": []}},
         {\"name\": \"Calculator\", \"description\": \"数学计算器，计算数学问题\",
          \"parameters\": {\"type\": \"object\", \"properties\": {\"symbol\": {\"description\": \"要计算的数学公式\"}},
                         \"required\": []}}
         ], \"stream\": true, \"max_tokens\": 100, \"temperature\": 0.8, \"top_p\": 0.8}"
```

--------------------------------

### Finetune ChatGLM3 (Single GPU)

Source: https://github.com/zai-org/chatglm3/blob/main/finetune_demo/README.md

Execute finetuning on a single machine with a single GPU.

```bash
cd finetune_demo
python finetune_hf.py  data/AdvertiseGen/  THUDM/chatglm3-6b  configs/lora.yaml
```

--------------------------------

### Run the ChatGLM3 Streamlit Demo

Source: https://github.com/zai-org/chatglm3/blob/main/composite_demo/README.md

Launch the local Streamlit web demo for ChatGLM3. The application will load the model and provide a URL to access it in your browser. Initial model loading may take time.

```bash
streamlit run main.py
```

--------------------------------

### Execute Single Machine Single Card Fine-tuning

Source: https://github.com/zai-org/chatglm3/blob/main/finetune_demo/README_en.md

Use this command for fine-tuning on a single GPU. Navigate to the finetune_demo directory before execution.

```bash
cd finetune_demo
python finetune_hf.py data/AdvertiseGen/ THUDM/chatglm3-6b configs/lora.yaml
```

--------------------------------

### Chat Curl Test

Source: https://github.com/zai-org/chatglm3/blob/main/README_en.md

Example of how to test the Chat Completions API using curl.

```APIDOC
## Curl Test for Chat Completions

### Description
This example demonstrates how to make a POST request to the `/v1/chat/completions` endpoint using curl to get a story from the model.

### Method
POST

### Endpoint
http://127.0.0.1:8000/v1/chat/completions

### Request Example
```bash
curl -X POST "http://127.0.0.1:8000/v1/chat/completions" \
-H "Content-Type: application/json" \
-d "{\"model\": \"chatglm3-6b\", \"messages\": [{\"role\": \"system\", \"content\": \"You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.\"}, {\"role\": \"user\", \"content\": \"你好，给我讲一个故事，大概100字\"}], \"stream\": false, \"max_tokens\": 100, \"temperature\": 0.8, \"top_p\": 0.8}"
```
```

--------------------------------

### LoRA Finetuning Command

Source: https://github.com/zai-org/chatglm3/blob/main/finetune_demo/lora_finetune.ipynb

This command initiates the LoRA finetuning process. Ensure all necessary arguments are provided for your specific dataset and model configuration.

```bash
python finetune_demo/lora_finetune.py \
    --model_name_or_path "bigscience/bloom-560m" \
    --dataset_name "./my_dataset" \
    --output_dir "./output" \
    --per_device_train_batch_size 1 \
    --gradient_accumulation_steps 1 \
    --per_device_eval_batch_size 1 \
    --do_train \
    --do_eval \
    --logging_steps 10 \
    --save_steps 100 \
    --eval_steps 100 \
    --learning_rate 2e-5 \
    --num_train_epochs 3 \
    --fp16 \
    --use_lora \
    --lora_rank 8 \
    --lora_alpha 16 \
    --lora_dropout 0.05 \
    --lora_target_modules ""
```

--------------------------------

### Resume Finetuning from Checkpoint

Source: https://github.com/zai-org/chatglm3/blob/main/finetune_demo/README.md

Continue finetuning from the last saved checkpoint or a specific checkpoint number. Use 'yes' to resume from the latest, or a number like '600' for a specific checkpoint.

```bash
cd finetune_demo
python finetune_hf.py  data/AdvertiseGen/  THUDM/chatglm3-6b  configs/lora.yaml yes
```

--------------------------------

### Verify Finetuned Model with Inference Script

Source: https://github.com/zai-org/chatglm3/blob/main/finetune_demo/README.md

Test your finetuned model using the inference script. Replace 'your_finetune_path' with the path to your model and 'your prompt' with your desired input.

```bash
python inference_hf.py your_finetune_path --prompt your prompt
```

--------------------------------

### OpenAI Style Custom Interface Agent-Chat Curl Test

Source: https://github.com/zai-org/chatglm3/blob/main/README_en.md

Example of testing the Chat Completions API for custom toolboxes using curl.

```APIDOC
## Curl Test for Custom Tool Interface

### Description
This example demonstrates how to test the Chat Completions API for custom toolboxes using curl. This interface is designed for autonomous scheduling of custom toolkits and does not require an API key.

### Method
POST

### Endpoint
http://127.0.0.1:8000/v1/chat/completions

### Request Example
```bash
curl -X POST "http://127.0.0.1:8000/v1/chat/completions" \
-H "Content-Type: application/json" \
-d "{\"model\": \"chatglm3-6b\", \"messages\": [{\"role\": \"user\", \"content\": \"37乘以8加7除2等于多少？\"}], \"stream\": true, \"max_tokens\": 100, \"temperature\": 0.8, \"top_p\": 0.8}"
```
```

--------------------------------

### Check Current Directory

Source: https://github.com/zai-org/chatglm3/blob/main/finetune_demo/lora_finetune.ipynb

Verifies the current working directory to ensure it is within the 'finetune_demo' directory.

```python
!pwd
```

--------------------------------

### Execute Single Machine Multiple Cards/Multiple Machines Multiple Cards Fine-tuning

Source: https://github.com/zai-org/chatglm3/blob/main/finetune_demo/README_en.md

Use this command for distributed fine-tuning across multiple GPUs on a single machine or across multiple machines. Ensure you are in the finetune_demo directory.

```bash
cd finetune_demo
OMP_NUM_THREADS=1 torchrun --standalone --nnodes=1 --nproc_per_node=8  finetune_hf.py  data/AdvertiseGen/  THUDM/chatglm3-6b  configs/lora.yaml 
```

--------------------------------

### Continue Fine-tuning from Specific Checkpoint

Source: https://github.com/zai-org/chatglm3/blob/main/finetune_demo/README_en.md

To resume fine-tuning from a specific checkpoint number, provide the checkpoint number (e.g., '600') as the fourth parameter. Ensure you are in the finetune_demo directory.

```bash
cd finetune_demo
python finetune_hf.py  data/AdvertiseGen/  THUDM/chatglm3-6b  configs/lora.yaml 600
```

--------------------------------

### Load and Display Metadata

Source: https://github.com/zai-org/chatglm3/blob/main/PROMPT_en.md

Loads a JSONL file and displays the first five records to understand its structure. Assumes the file is already uploaded.

```text
<|system|>
You are an intelligent AI assistant, named ChatGLM3. You are connected to a computer, but please note that you cannot connect to the internet. When using Python to solve tasks, you can run the code and get the results. If there are errors in the running results, you need to improve the code as soon as possible. You can process files uploaded by users to your computer, and the default storage path for files is /mnt/data/.
<|user|>
#File: /mnt/data/metadata.jsonl
#Size: 35380
#File uploaded
Are there missing or abnormal values in the file?
<|assistant|>
Firstly, I will read the file you provided and review its contents to determine if there are missing or abnormal values.

We start by loading the file:
<|assistant|>interpreter

```

```python
import json

# Load the uploaded file
data = []
with open("/mnt/data/metadata.jsonl", 'r') as file:
    for line in file:
        data.append(json.loads(line))

# Display the first few records to get a sense of the data
data[:5]

```

--------------------------------

### Download ChatGLM3 Models from Huggingface

Source: https://github.com/zai-org/chatglm3/blob/main/tensorrt_llm_demo/README.md

Clones different versions of the ChatGLM3 model from Huggingface. Choose the specific model you wish to deploy.

```bash
# 您可以选择具体想部署的模型下载
git clone https://huggingface.co/THUDM/chatglm3-6b      chatglm3_6b
git clone https://huggingface.co/THUDM/chatglm3-6b-base chatglm3_6b_base
git clone https://huggingface.co/THUDM/chatglm3-6b-32k  chatglm3_6b_32k
```

--------------------------------

### Run LoRA Finetuning Inference

Source: https://github.com/zai-org/chatglm3/blob/main/finetune_demo/lora_finetune.ipynb

Execute inference using the finetuned weights from the last training checkpoint. Ensure to specify the correct output directory and prompt.

```shell
!CUDA_VISIBLE_DEVICES=0 NCCL_P2P_DISABLE="1" NCCL_IB_DISABLE="1" python inference_hf.py output/checkpoint-4000/ --prompt "类型#裙*版型#显瘦*材质#网纱*风格#性感*裙型#百褶*裙下摆#压褶*裙长#连衣裙*裙衣门襟#拉链*裙衣门襟#套头*裙款式#拼接*裙款式#拉链*裙款式#木耳边*裙款式#抽褶*裙款式#不规则"
```

--------------------------------

### Load ChatGLM3 for CPU Inference

Source: https://github.com/zai-org/chatglm3/blob/main/DEPLOYMENT_en.md

Deploy the model on a CPU if GPU hardware is unavailable. Note that CPU inference will be significantly slower and requires approximately 32GB of memory.

```python
model = AutoModel.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True).float()
```

--------------------------------

### Query Model for Tool Invocation

Source: https://github.com/zai-org/chatglm3/blob/main/tools_using_demo/README_en.md

Initiate a chat with the model, providing the system information and a query that requires tool usage. The model's response will indicate the tool to be called and its parameters.

```python
history = [system_info]
query = "Help me inquire the price of stock 10111"
response, history = model.chat(tokenizer, query, history=history)
print(response)
```

--------------------------------

### Deploy ChatGLM3 API Server

Source: https://github.com/zai-org/chatglm3/blob/main/README.md

Run the Python script to deploy the API server locally. This server is compatible with OpenAI and ZhipuAI API formats.

```shell
cd openai_api_demo
python api_server.py
```

--------------------------------

### Specify Local Model Path

Source: https://github.com/zai-org/chatglm3/blob/main/composite_demo/README.md

If you have already downloaded the model, set the MODEL_PATH environment variable to specify the local path for loading.

```bash
export MODEL_PATH=/path/to/model
```

--------------------------------

### ChatGLM3 Tool Calling Format

Source: https://github.com/zai-org/chatglm3/blob/main/PROMPT_en.md

Illustrates a ChatGLM3 dialogue involving tool calling. This format includes system instructions with tool definitions, user queries, assistant responses, tool calls, and observations.

```text
<|system|>
Answer the following questions as best as you can. You have access to the following tools:
[
    {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                    "description": "The city and state, e.g. San Francisco, CA",
                },
                "unit": {"type": "string"},
            },
            "required": ["location"],
        },
    }
]
<|user|>
What's the weather in Beijing today?
<|assistant|>
Okay, let's look up the weather in Bejing today.
<|assistant|>get_current_weather
```python
tool_call(location="beijing", unit="celsius")
```
<|observation|>
{"temperature": 22}
<|assistant|>
According to the query results, the temperature in Beijing today is 22 degrees Celsius.
```

--------------------------------

### Preprocessing Sanity Check Output

Source: https://github.com/zai-org/chatglm3/blob/main/finetune_demo/README_en.md

This log output shows the tokenization details for the first training data sample. It displays detokenized strings, token IDs, and target IDs, indicating which tokens participate in loss calculation.

```log
Sanity
Check >> >> >> >> >> >> >
'[gMASK]': 64790 ->   -100
'sop': 64792 ->   -100
'<|system|>': 64794 ->   -100
'': 30910 ->   -100
'\n': 13 ->   -100
'Answer': 20115 ->   -100
'the': 267 ->   -100
'following': 1762 ->   -100
...
'know': 683 ->   -100
'the': 267 ->   -100
'response': 3010 ->   -100
'details': 3296 ->   -100
'.': 30930 ->   -100
'<|assistant|>': 64796 ->   -100
'': 30910 ->  30910
'\n': 13 ->     13
'I': 307 ->    307
'need': 720 ->    720
'to': 289 ->    289
'use': 792 ->    792
...
<< << << << << << < Sanity
Check
```

--------------------------------

### Build TensorRT-LLM Docker Image

Source: https://github.com/zai-org/chatglm3/blob/main/tensorrt_llm_demo/README.md

Builds the release version of the TensorRT-LLM Docker image.

```bash
make -C docker release_build
```

--------------------------------

### Load Fine-tuned Model and Tokenizer (Python)

Source: https://github.com/zai-org/chatglm3/blob/main/finetune_demo/README_en.md

This Python function loads a model and tokenizer, automatically detecting if it's a PEFT (LoRA) model by checking for 'adapter_config.json'. It sets device_map to 'auto' for efficient device utilization.

```python
from typing import Union
from pathlib import Path
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import AutoPeftModelForCausalLM

ModelType = ""
TokenizerType = ""

def load_model_and_tokenizer(
        model_dir: Union[str, Path], trust_remote_code: bool = True
) -> tuple[ModelType, TokenizerType]:
    model_dir = _resolve_path(model_dir)
    if (model_dir / 'adapter_config.json').exists():
        model = AutoPeftModelForCausalLM.from_pretrained(
            model_dir, trust_remote_code=trust_remote_code, device_map='auto'
        )
        tokenizer_dir = model.peft_config['default'].base_model_name_or_path
    else:
        model = AutoModelForCausalLM.from_pretrained(
            model_dir, trust_remote_code=trust_remote_code, device_map='auto'
        )
        tokenizer_dir = model_dir
    tokenizer = AutoTokenizer.from_pretrained(
        tokenizer_dir, trust_remote_code=trust_remote_code
    )
    return model, tokenizer
```

--------------------------------

### Query Model for Tool Call

Source: https://github.com/zai-org/chatglm3/blob/main/tools_using_demo/README.md

Initiate a tool call by sending a query to the model's chat method. This method only supports `chat`, not `stream_chat`.

```python
history = [system_info]
query = "帮我查询股票10111的价格"
response, history = model.chat(tokenizer, query, history=history)
print(response)
```

--------------------------------

### Gradient Checkpointing Configuration

Source: https://github.com/zai-org/chatglm3/blob/main/finetune_demo/README_en.md

This Python code snippet shows how gradient checkpointing is configured in the ChatGLM3 model. Adjusting 'use_reentrant' can help manage video memory during training.

```python
if self.gradient_checkpointing and self.training:
                layer_ret = torch.utils.checkpoint.checkpoint(
                    layer,
                    hidden_states,
                    attention_mask,
                    rotary_pos_emb,
                    kv_caches[index],
                    use_cache,
                    use_reentrant=False
                )
```

--------------------------------

### Build TensorRT-LLM Engine with Weight-Only Quantization

Source: https://github.com/zai-org/chatglm3/blob/main/tensorrt_llm_demo/README.md

Builds an inference engine using weight-only quantization (defaulting to INT8) to accelerate inference and reduce memory footprint.

```bash
python3 build.py -m chatglm3_6b --use_weight_only --output_dir trt_engines/chatglm3_6b/weight_only/1-gpu
```

--------------------------------

### Load ChatGLM3 with 4-bit Quantization

Source: https://github.com/zai-org/chatglm3/blob/main/DEPLOYMENT_en.md

Use this snippet to load the model with 4-bit quantization when GPU VRAM is limited. This reduces VRAM requirements but may incur a slight performance loss.

```python
model = AutoModel.from_pretrained("THUDM/chatglm3-6b",trust_remote_code=True).quantize(4).cuda()
```

--------------------------------

### Call Tool and Generate Response

Source: https://github.com/zai-org/chatglm3/blob/main/tools_using_demo/README.md

After receiving a tool call request, implement the logic to call the tool and then send the result back to the model using `role="observation"` to generate a final response.

```python
result = json.dumps({"price": 12412}, ensure_ascii=False)
response, history = model.chat(tokenizer, result, history=history, role="observation")
print(response)
```

--------------------------------

### Test API with Python Script

Source: https://github.com/zai-org/chatglm3/blob/main/README.md

Execute a Python script to test the API calls. If successful, the model should return a story.

```shell
cd openai_api_demo
python openai_api_request.py
```

--------------------------------

### Multi-turn Dialogue and Tool Use Fine-tuning Data Format

Source: https://github.com/zai-org/chatglm3/blob/main/finetune_demo/README.md

Use this JSON format to fine-tune both the dialogue and tool-use capabilities of the model. It includes tool definitions and conversation turns with tool interactions.

```json
[
  {
    "tools": [
      // available tools, format is not restricted
    ],
    "conversations": [
      {
        "role": "system",
        "content": "<system prompt text>"
      },
      {
        "role": "user",
        "content": "<user prompt text>"
      },
      {
        "role": "assistant",
        "content": "<assistant thought to text>"
      },
      {
        "role": "tool",
        "name": "<name of the tool to be called",
        "parameters": {
          "<parameter_name>": "<parameter_value>"
        },
        "observation": "<observation>"
        // don't have to be string
      },
      {
        "role": "assistant",
        "content": "<assistant response to observation>"
      },
      // ... Muti Turn
      {
        "role": "user",
        "content": "<user prompt text>"
      },
      {
        "role": "assistant",
        "content": "<assistant response text>"
      }
    ]
  }
  // ...
]
```

--------------------------------

### Run ChatGLM3 Model Inference

Source: https://github.com/zai-org/chatglm3/blob/main/Intel_device_demo/openvino_demo/README.md

Execute the ChatGLM3 model for inference using OpenVINO. Specify the model path, maximum sequence length, and target device (e.g., CPU).

```python
python3 chat.py --model_path {your_path}/chatglm3-6b --max_sequence_length 4096 --device CPU
```

--------------------------------

### Run Summarization Task with TensorRT-LLM

Source: https://github.com/zai-org/chatglm3/blob/main/tensorrt_llm_demo/README.md

Execute the `summarize.py` script for article summarization using TensorRT-LLM. This requires specifying the Hugging Face model directory and the TensorRT engine directory.

```bash
python3 ../summarize.py --test_trt_llm \
                        --hf_model_dir chatglm3_6b \
                        --engine_dir trt_engines/chatglm3_6b/fp16/1-gpu
```

--------------------------------

### Build TensorRT-LLM Engine for chatglm3_6b-32k Model

Source: https://github.com/zai-org/chatglm3/blob/main/tensorrt_llm_demo/README.md

Builds a default FP16 inference engine for the chatglm3_6b-32k model, which supports a longer context length.

```bash
python3 build.py -m chatglm3_6b_32k --output_dir trt_engines/chatglm3_6b-32k/fp16/1-gpu
```

--------------------------------

### Build TensorRT-LLM Engine for Multi-GPU Deployment

Source: https://github.com/zai-org/chatglm3/blob/main/tensorrt_llm_demo/README.md

Builds an FP16 inference engine configured to run across two GPUs, suitable for larger models or higher throughput.

```bash
python3 build.py -m chatglm3_6b --world_size 2 --output_dir trt_engines/chatglm3_6b/fp16/2-gpu
```

--------------------------------

### Build TensorRT-LLM Engine for chatglm3_6b_base Model

Source: https://github.com/zai-org/chatglm3/blob/main/tensorrt_llm_demo/README.md

Builds a default FP16 inference engine specifically for the chatglm3_6b_base model.

```bash
python3 build.py -m chatglm3_6b_base --output_dir trt_engines/chatglm3_6b_base/fp16/1-gpu
```
Tokens:
Raw