### Environment Installation

Source: https://github.com/thudm/chatglm-6b/blob/main/README.md

Instructions for installing necessary dependencies for ChatGLM-6B.

```APIDOC
## Environment Installation

Use pip to install dependencies:
`pip install -r requirements.txt`

Recommended `transformers` library version is `4.27.1`, but any version not lower than `4.23.1` should theoretically work.

For running quantized models on CPU, you also need to install `gcc` and `openmp`. Most Linux distributions have these installed by default. For Windows, you can select `openmp` during the installation of [TDM-GCC](https://jmeubank.github.io/tdm-gcc/). The tested `gcc` version on Windows is `TDM-GCC 10.3.0`, and on Linux it is `gcc 11.3.0`. For MacOS, please refer to [Q1](FAQ.md#q1).
```

--------------------------------

### Install Dependencies

Source: https://github.com/thudm/chatglm-6b/blob/main/README.md

Install the required Python packages for the project.

```shell
pip install -r requirements.txt
```

--------------------------------

### Install Software Dependencies

Source: https://github.com/thudm/chatglm-6b/blob/main/ptuning/README.md

Required packages for running the fine-tuning process.

```shell
pip install rouge_chinese nltk jieba datasets
```

--------------------------------

### Install Dependencies for P-Tuning v2

Source: https://context7.com/thudm/chatglm-6b/llms.txt

Installs necessary Python packages for P-Tuning v2 fine-tuning, including libraries for evaluation metrics and data handling.

```bash
# 安装依赖
pip install rouge_chinese nltk jieba datasets
```

--------------------------------

### Install OpenMP on Mac

Source: https://github.com/thudm/chatglm-6b/blob/main/FAQ.md

Run these commands to install the OpenMP library on macOS. This is necessary to resolve the 'clang: error: unsupported option '-fopenmp'' error. Ensure your GCC (clang) version matches the downloaded file.

```bash
# 参考`https://mac.r-project.org/openmp/`
## 假设: gcc(clang)是14.x版本，其他版本见R-Project提供的表格
curl -O https://mac.r-project.org/openmp/openmp-14.0.6-darwin20-Release.tar.gz
sudo tar fvxz openmp-14.0.6-darwin20-Release.tar.gz -C /
```

--------------------------------

### P-Tuning v2 Training Data Format

Source: https://context7.com/thudm/chatglm-6b/llms.txt

Example JSON format for training data used with P-Tuning v2. Requires 'content' and 'summary' fields.

```json
# 训练数据格式 (train.json)
# {"content": "类型#上衣*版型#宽松*图案#线条", "summary": "这件衬衫的款式非常的宽松..."}
```

--------------------------------

### Load and Interact with ChatGLM-6B

Source: https://context7.com/thudm/chatglm-6b/llms.txt

Basic setup for loading the model and tokenizer to perform single and multi-turn conversations with optional generation parameters.

```python
from transformers import AutoTokenizer, AutoModel

# 加载分词器和模型
tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)
model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).half().cuda()
model = model.eval()

# 单轮对话
response, history = model.chat(tokenizer, "你好", history=[])
print(response)
# 输出: 你好👋!我是人工智能助手 ChatGLM-6B,很高兴见到你,欢迎问我任何问题。

# 多轮对话（传入历史记录）
response, history = model.chat(tokenizer, "晚上睡不着应该怎么办", history=history)
print(response)
# 输出: 晚上睡不着可能会让你感到焦虑或不舒服,但以下是一些可以帮助你入睡的方法:
# 1. 制定规律的睡眠时间表...
# 2. 创造一个舒适的睡眠环境...

# 可选参数控制生成行为
response, history = model.chat(
    tokenizer,
    "请用100字介绍人工智能",
    history=[],
    max_length=2048,      # 最大生成长度
    top_p=0.7,            # 核采样概率
    temperature=0.95      # 温度参数
)
```

--------------------------------

### Dialogue Dataset Format

Source: https://github.com/thudm/chatglm-6b/blob/main/ptuning/README.md

Example of a three-turn dialogue dataset in JSON format.

```json
{"prompt": "长城h3风扇不转。继电器好的。保险丝好的传感器新的风扇也新的这是为什么。就是继电器缺一个信号线", "response": "用电脑能读数据流吗？水温多少", "history": []}
{"prompt": "95", "response": "上下水管温差怎么样啊？空气是不是都排干净了呢？", "history": [["长城h3风扇不转。继电器好的。保险丝好的传感器新的风扇也新的这是为什么。就是继电器缺一个信号线", "用电脑能读数据流吗？水温多少"]]}
{"prompt": "是的。上下水管都好的", "response": "那就要检查线路了，一般风扇继电器是由电脑控制吸合的，如果电路存在断路，或者电脑坏了的话会出现继电器不吸合的情况！", "history": [["长城h3风扇不转。继电器好的。保险丝好的传感器新的风扇也新的这是为什么。就是继电器缺一个信号线", "用电脑能读数据流吗？水温多少"], ["95", "上下水管温差怎么样啊？空气是不是都排干净了呢？"]]}
```

--------------------------------

### Code Usage: Generating Dialogue

Source: https://github.com/thudm/chatglm-6b/blob/main/README.md

Example Python code demonstrating how to load and use the ChatGLM-6B model for dialogue generation.

```APIDOC
## Code Usage

You can use the following code to call the ChatGLM-6B model for dialogue generation:

```python
>>> from transformers import AutoTokenizer, AutoModel
>>> tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)
>>> model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).half().cuda()
>>> model = model.eval()
>>> response, history = model.chat(tokenizer, "你好", history=[])
>>> print(response)
你好👋!我是人工智能助手 ChatGLM-6B,很高兴见到你,欢迎问我任何问题。
>>> response, history = model.chat(tokenizer, "晚上睡不着应该怎么办", history=history)
>>> print(response)
晚上睡不着可能会让你感到焦虑或不舒服,但以下是一些可以帮助你入睡的方法:

1. 制定规律的睡眠时间表:保持规律的睡眠时间表可以帮助你建立健康的睡眠习惯,使你更容易入睡。尽量在每天的相同时间上床,并在同一时间起床。
2. 创造一个舒适的睡眠环境:确保睡眠环境舒适,安静,黑暗且温度适宜。可以使用舒适的床上用品,并保持房间通风。
3. 放松身心:在睡前做些放松的活动,例如泡个热水澡,听些轻柔的音乐,阅读一些有趣的书籍等,有助于缓解紧张和焦虑,使你更容易入睡。
4. 避免饮用含有咖啡因的饮料:咖啡因是一种刺激性物质,会影响你的睡眠质量。尽量避免在睡前饮用含有咖啡因的饮料,例如咖啡,茶和可乐。
5. 避免在床上做与睡眠无关的事情:在床上做些与睡眠无关的事情,例如看电影,玩游戏或工作等,可能会干扰你的睡眠。
6. 尝试呼吸技巧:深呼吸是一种放松技巧,可以帮助你缓解紧张和焦虑,使你更容易入睡。试着慢慢吸气,保持几秒钟,然后缓慢呼气。

如果这些方法无法帮助你入睡,你可以考虑咨询医生或睡眠专家,寻求进一步的建议。
```

Model implementations are subject to change. To ensure compatibility, you can add the `revision="v1.1.0"` parameter to the `from_pretrained` call. `v1.1.0` is the latest version number. For a complete list of versions, see [Change Log](https://huggingface.co/THUDM/chatglm-6b#change-log).
```

--------------------------------

### Model Quantization

Source: https://github.com/thudm/chatglm-6b/blob/main/README.md

Guide on using model quantization to reduce GPU memory usage.

```APIDOC
## Model Quantization

By default, the model is loaded with FP16 precision, requiring approximately 13GB of GPU memory. If your GPU memory is limited, you can try loading the model in a quantized format. Usage is as follows:

```python
# Modify as needed; currently only supports 4/8 bit quantization
model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).quantize(8).half().cuda()
```

After 2-3 rounds of dialogue, 8-bit quantization uses about 10GB of GPU memory, and 4-bit quantization requires only 6GB. As the number of dialogue rounds increases, the corresponding memory consumption also grows. Due to the use of relative positional encoding, ChatGLM-6B theoretically supports infinite context length, but performance gradually degrades beyond 2048 tokens (training length).

Model quantization incurs some performance loss. Tested results show that ChatGLM-6B can still generate natural and fluent responses with 4-bit quantization. Using quantization schemes like [GPT-Q](https://arxiv.org/abs/2210.17323) can further compress quantization precision/improve model performance at the same quantization precision. Pull Requests for this are welcome.

Quantization requires loading the FP16 format model into memory first, consuming about 13GB of RAM. If your RAM is insufficient, you can directly load the quantized model. The INT4 quantized model only requires about 5.2GB of RAM:

```python
# For INT8 quantization, change "THUDM/chatglm-6b-int4" to "THUDM/chatglm-6b-int8"
model = AutoModel.from_pretrained("THUDM/chatglm-6b-int4", trust_remote_code=True).half().cuda()
```

Quantized model parameter files can also be manually downloaded from [here](https://cloud.tsinghua.edu.cn/d/674208019e314311ab5c/).
```

--------------------------------

### ADGEN Dataset Format

Source: https://github.com/thudm/chatglm-6b/blob/main/ptuning/README.md

Example of the expected JSON format for the ADGEN dataset.

```json
{
    "content": "类型#上衣*版型#宽松*版型#显瘦*图案#线条*衣样式#衬衫*衣袖型#泡泡袖*衣款式#抽绳",
    "summary": "这件衬衫的款式非常的宽松，利落的线条可以很好的隐藏身材上的小缺点，穿在身上有着很好的显瘦效果。领口装饰了一个可爱的抽绳，漂亮的绳结展现出了十足的个性，配合时尚的泡泡袖型，尽显女性甜美可爱的气息。"
}
```

--------------------------------

### Send POST request to ChatGLM API

Source: https://context7.com/thudm/chatglm-6b/llms.txt

Example cURL commands to interact with the running ChatGLM API. Demonstrates basic requests and requests with advanced parameters like max_length, top_p, and temperature.

```bash
# 启动服务
python api.py

# 发送请求
curl -X POST "http://127.0.0.1:8000" \
     -H 'Content-Type: application/json' \
     -d '{"prompt": "你好", "history": []}'

# 响应示例
# {
#   "response": "你好👋！我是人工智能助手 ChatGLM-6B，很高兴见到你，欢迎问我任何问题。",
#   "history": [["你好", "你好👋！我是人工智能助手 ChatGLM-6B，很高兴见到你，欢迎问我任何问题。"]],
#   "status": 200,
#   "time": "2023-03-23 21:38:40"
# }

# 带参数的请求
curl -X POST "http://127.0.0.1:8000" \
     -H 'Content-Type: application/json' \
     -d '{
       "prompt": "请介绍一下机器学习",
       "history": [],
       "max_length": 4096,
       "top_p": 0.8,
       "temperature": 0.9
     }'
```

--------------------------------

### VisualGLM-6B Multimodal Dialogue

Source: https://context7.com/thudm/chatglm-6b/llms.txt

Engage in image understanding and multimodal conversations using VisualGLM-6B. Requires installation of SwissArmyTransformer and torchvision. Uses stream_chat for interactive responses.

```python
from transformers import AutoTokenizer, AutoModel

# 加载 VisualGLM-6B 模型
tokenizer = AutoTokenizer.from_pretrained("THUDM/visualglm-6b", trust_remote_code=True)
model = AutoModel.from_pretrained("THUDM/visualglm-6b", trust_remote_code=True).half().cuda()
model = model.eval()

# 图像对话
image_path = "example.jpg"
history = []

# 首轮对话：描述图片
for response, history in model.stream_chat(tokenizer, image_path, "描述这张图片。", history=history):
    print(response)

# 后续对话：基于图片内容提问
for response, history in model.stream_chat(tokenizer, image_path, "图片中有什么颜色？", history=history):
    print(response)
```

--------------------------------

### Run Web Demo

Source: https://github.com/thudm/chatglm-6b/blob/main/README.md

Launch the Gradio-based web interface.

```shell
pip install gradio
python web_demo.py
```

--------------------------------

### Run Web Demo

Source: https://github.com/thudm/chatglm-6b/blob/main/ptuning/README.md

Execute the web demo script for P-Tuning v2 checkpoints.

```shell
bash web_demo.sh
```

--------------------------------

### Run CLI Demo

Source: https://github.com/thudm/chatglm-6b/blob/main/README.md

Launch the interactive command-line interface.

```shell
python cli_demo.py
```

--------------------------------

### Run Training Scripts

Source: https://github.com/thudm/chatglm-6b/blob/main/ptuning/README.md

Commands to initiate P-Tuning v2 or full-parameter fine-tuning.

```shell
bash train.sh
```

```shell
bash ds_train_finetune.sh
```

--------------------------------

### CPU Deployment

Source: https://github.com/thudm/chatglm-6b/blob/main/README.md

Instructions for running ChatGLM-6B inference on a CPU.

```APIDOC
## CPU Deployment

If you do not have GPU hardware, you can also perform inference on a CPU, but the inference speed will be slower. Usage is as follows (requires approximately 32GB of RAM):

```python
model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).float()
```

If your RAM is insufficient, you can directly load the quantized model:
```

--------------------------------

### Clone Project Repository

Source: https://github.com/thudm/chatglm-6b/blob/main/README.md

Download the project source code to run demos.

```shell
git clone https://github.com/THUDM/ChatGLM-6B
cd ChatGLM-6B
```

--------------------------------

### Run Training Script

Source: https://github.com/thudm/chatglm-6b/blob/main/ptuning/README.md

Execute the training script for chat fine-tuning.

```shell
bash train_chat.sh
```

--------------------------------

### Loading Model from Local

Source: https://github.com/thudm/chatglm-6b/blob/main/README.md

Instructions on how to download and load the ChatGLM-6B model from a local directory.

```APIDOC
## Loading Model from Local

The code above automatically downloads the model implementation and parameters via `transformers`. The complete model implementation can be found on [Hugging Face Hub](https://huggingface.co/THUDM/chatglm-6b). If you have a poor network connection, downloading model parameters may take a long time or fail. In such cases, you can download the model locally and then load it.

To download the model from Hugging Face Hub, first [install Git LFS](https://docs.github.com/zh/repositories/working-with-files/managing-large-files/installing-git-large-file-storage) and then run:
```Shell
git clone https://huggingface.co/THUDM/chatglm-6b
```

If the download speed of checkpoints from Hugging Face Hub is slow, you can download only the model implementation:
```Shell
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/THUDM/chatglm-6b
```
Then, manually download the model parameter files from [here](https://cloud.tsinghua.edu.cn/d/fb9f16d6dc8f482596c2/) and replace the files in the local `chatglm-6b` directory.

After downloading the model locally, replace `"THUDM/chatglm-6b"` in the code above with the path to your local `chatglm-6b` folder to load the model from local.

**Optional** Model implementations are subject to change. To ensure compatibility, you can execute:
```Shell
git checkout v1.1.0
```
```

--------------------------------

### Deploy API Service

Source: https://github.com/thudm/chatglm-6b/blob/main/README.md

Run the FastAPI server and test it with a curl request.

```shell
pip install fastapi uvicorn
python api.py
```

```shell
curl -X POST "http://127.0.0.1:8000" \
     -H 'Content-Type: application/json' \
     -d '{"prompt": "你好", "history": []}'
```

--------------------------------

### Deploy on CPU

Source: https://context7.com/thudm/chatglm-6b/llms.txt

Run inference on CPU-only environments, optionally using quantized models to save memory.

```python
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)

# FP32 精度 CPU 推理（需要约32GB内存）
model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).float()

# 使用量化模型减少内存占用
model = AutoModel.from_pretrained("THUDM/chatglm-6b-int4", trust_remote_code=True).float()

model = model.eval()
response, history = model.chat(tokenizer, "你好，这是CPU推理测试", history=[])
print(response)
```

--------------------------------

### API Deployment

Source: https://github.com/thudm/chatglm-6b/blob/main/README.md

Instructions for deploying the ChatGLM-6B model as an API using FastAPI.

```APIDOC
## API Deployment

First, install additional dependencies: `pip install fastapi uvicorn`. Then, run `api.py` from the repository:

```shell
python api.py
```

By default, it is deployed on the local port 8000 and can be called using the POST method:

```shell
curl -X POST "http://127.0.0.1:8000" \
     -H 'Content-Type: application/json' \
     -d '{"prompt": "你好", "history": []}'
```

The returned value is:

```shell
{
  "response":"你好👋！我是人工智能助手 ChatGLM-6B，很高兴见到你，欢迎问我任何问题。",
  "history": [["你好","你好👋！我是人工智能助手 ChatGLM-6B，很高兴见到你，欢迎问我任何问题。"]],
  "status":200,
  "time":"2023-03-23 21:38:40"
}
```
```

--------------------------------

### Experimental Settings

Source: https://github.com/thudm/chatglm-6b/blob/main/ptuning/README.md

Hyperparameter configurations for different training methods.

```text
max_source_length=64
max_target_length=64
max_steps=3000
```

```text
pre_seq_len=128
learning_rate=2e-2
quantization_bit=4
per_device_train_batch_size=16
gradient_accumulation_steps=1
```

```text
learning_rate=1e-4
fp16
num_gpus=4
per_device_train_batch_size=4
gradient_accumulation_steps=1
```

```text
learning_rate=5e-4
per_device_train_batch_size=16
gradient_accumulation_steps=1
```

--------------------------------

### Train ChatGLM-6B with P-Tuning v2

Source: https://context7.com/thudm/chatglm-6b/llms.txt

Use this script to train the ChatGLM-6B model with P-Tuning v2. Ensure necessary files like AdvertiseGen/train.json and AdvertiseGen/dev.json are available.

```bash
# train.sh 训练脚本
PRE_SEQ_LEN=128
LR=2e-2

CUDA_VISIBLE_DEVICES=0 python3 main.py \
    --do_train \
    --train_file AdvertiseGen/train.json \
    --validation_file AdvertiseGen/dev.json \
    --prompt_column content \
    --response_column summary \
    --overwrite_cache \
    --model_name_or_path THUDM/chatglm-6b \
    --output_dir output/adgen-chatglm-6b-pt-$PRE_SEQ_LEN-$LR \
    --overwrite_output_dir \
    --max_source_length 64 \
    --max_target_length 64 \
    --per_device_train_batch_size 1 \
    --per_device_eval_batch_size 1 \
    --gradient_accumulation_steps 16 \
    --predict_with_generate \
    --max_steps 3000 \
    --logging_steps 10 \
    --save_steps 1000 \
    --learning_rate $LR \
    --pre_seq_len $PRE_SEQ_LEN \
    --quantization_bit 4
```

--------------------------------

### Load Tokenizer

Source: https://github.com/thudm/chatglm-6b/blob/main/ptuning/README.md

Initialize the tokenizer for the ChatGLM-6B model.

```python
tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)
```

--------------------------------

### Run ChatGLM-6B Inference

Source: https://github.com/thudm/chatglm-6b/blob/main/README.md

Initialize the model and tokenizer to generate conversational responses.

```python
>>> from transformers import AutoTokenizer, AutoModel
>>> tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)
>>> model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).half().cuda()
>>> model = model.eval()
>>> response, history = model.chat(tokenizer, "你好", history=[])
>>> print(response)
你好👋!我是人工智能助手 ChatGLM-6B,很高兴见到你,欢迎问我任何问题。
>>> response, history = model.chat(tokenizer, "晚上睡不着应该怎么办", history=history)
>>> print(response)
晚上睡不着可能会让你感到焦虑或不舒服,但以下是一些可以帮助你入睡的方法:

1. 制定规律的睡眠时间表:保持规律的睡眠时间表可以帮助你建立健康的睡眠习惯,使你更容易入睡。尽量在每天的相同时间上床,并在同一时间起床。
2. 创造一个舒适的睡眠环境:确保睡眠环境舒适,安静,黑暗且温度适宜。可以使用舒适的床上用品,并保持房间通风。
3. 放松身心:在睡前做些放松的活动,例如泡个热水澡,听些轻柔的音乐,阅读一些有趣的书籍等,有助于缓解紧张和焦虑,使你更容易入睡。
4. 避免饮用含有咖啡因的饮料:咖啡因是一种刺激性物质,会影响你的睡眠质量。尽量避免在睡前饮用含有咖啡因的饮料,例如咖啡,茶和可乐。
5. 避免在床上做与睡眠无关的事情:在床上做些与睡眠无关的事情,例如看电影,玩游戏或工作等,可能会干扰你的睡眠。
6. 尝试呼吸技巧:深呼吸是一种放松技巧,可以帮助你缓解紧张和焦虑,使你更容易入睡。试着慢慢吸气,保持几秒钟,然后缓慢呼气。

如果这些方法无法帮助你入睡,你可以考虑咨询医生或睡眠专家,寻求进一步的建议。
```

--------------------------------

### Create ChatGLM Gradio Web UI

Source: https://context7.com/thudm/chatglm-6b/llms.txt

Builds an interactive web interface for ChatGLM using Gradio. Supports real-time streaming output and includes sliders for controlling generation parameters.

```python
# web_demo.py
from transformers import AutoModel, AutoTokenizer
import gradio as gr
import mdtex2html

tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)
model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).half().cuda()
model = model.eval()

def postprocess(self, y):
    if y is None:
        return []
    for i, (message, response) in enumerate(y):
        y[i] = (
            None if message is None else mdtex2html.convert((message)),
            None if response is None else mdtex2html.convert(response),
        )
    return y

gr.Chatbot.postprocess = postprocess

def predict(input, chatbot, max_length, top_p, temperature, history):
    chatbot.append((input, ""))
    for response, history in model.stream_chat(
        tokenizer, input, history,
        max_length=max_length,
        top_p=top_p,
        temperature=temperature
    ):
        chatbot[-1] = (input, response)
        yield chatbot, history

def reset_state():
    return [], []

with gr.Blocks() as demo:
    gr.HTML("<h1 align=\"center\">ChatGLM</h1>")
    chatbot = gr.Chatbot()
    with gr.Row():
        with gr.Column(scale=4):
            user_input = gr.Textbox(show_label=False, placeholder="输入内容...", lines=10)
            submitBtn = gr.Button("提交", variant="primary")
        with gr.Column(scale=1):
            emptyBtn = gr.Button("清空历史")
            max_length = gr.Slider(0, 4096, value=2048, step=1.0, label="最大长度")
            top_p = gr.Slider(0, 1, value=0.7, step=0.01, label="Top P")
            temperature = gr.Slider(0, 1, value=0.95, step=0.01, label="Temperature")

    history = gr.State([])
    submitBtn.click(predict, [user_input, chatbot, max_length, top_p, temperature, history], [chatbot, history])
    emptyBtn.click(reset_state, outputs=[chatbot, history])

demo.queue().launch(share=False, inbrowser=True)
```

--------------------------------

### Inference Configuration

Source: https://github.com/thudm/chatglm-6b/blob/main/ptuning/README.md

Parameters for loading the model and checkpoints during inference.

```shell
--model_name_or_path THUDM/chatglm-6b
--ptuning_checkpoint $CHECKPOINT_PATH
```

```shell
--model_name_or_path $CHECKPOINT_PATH
```

--------------------------------

### Run Model on CPU

Source: https://github.com/thudm/chatglm-6b/blob/main/README.md

Load the model for CPU inference.

```python
model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).float()
```

--------------------------------

### Checkout Specific Model Version

Source: https://github.com/thudm/chatglm-6b/blob/main/README.md

Switch to a specific version tag to ensure compatibility.

```shell
git checkout v1.1.0
```

--------------------------------

### Download Model Repository

Source: https://github.com/thudm/chatglm-6b/blob/main/README.md

Clone the model repository from Hugging Face using Git LFS.

```shell
git clone https://huggingface.co/THUDM/chatglm-6b
```

```shell
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/THUDM/chatglm-6b
```

--------------------------------

### Evaluate ChatGLM-6B with P-Tuning v2

Source: https://context7.com/thudm/chatglm-6b/llms.txt

Evaluate the performance of a fine-tuned ChatGLM-6B model using P-Tuning v2. This script uses Rouge and BLEU-4 metrics. Ensure the checkpoint path is correct.

```bash
# evaluate.sh 评估脚本
PRE_SEQ_LEN=128
CHECKPOINT=adgen-chatglm-6b-pt-128-2e-2
STEP=3000

CUDA_VISIBLE_DEVICES=0 python3 main.py \
    --do_predict \
    --validation_file AdvertiseGen/dev.json \
    --test_file AdvertiseGen/dev.json \
    --overwrite_cache \
    --prompt_column content \
    --response_column summary \
    --model_name_or_path THUDM/chatglm-6b \
    --ptuning_checkpoint ./output/$CHECKPOINT/checkpoint-$STEP \
    --output_dir ./output/$CHECKPOINT \
    --overwrite_output_dir \
    --max_source_length 64 \
    --max_target_length 64 \
    --per_device_eval_batch_size 1 \
    --predict_with_generate \
    --pre_seq_len $PRE_SEQ_LEN \
    --quantization_bit 4
```

--------------------------------

### Load Full Checkpoint

Source: https://github.com/thudm/chatglm-6b/blob/main/ptuning/README.md

Load an old checkpoint or a full-parameter fine-tuned model.

```python
model = AutoModel.from_pretrained(CHECKPOINT_PATH, trust_remote_code=True)
```

--------------------------------

### Deploy ChatGLM-6B on MacOS

Source: https://github.com/thudm/chatglm-6b/blob/main/README.md

Loads a local model using the MPS backend for Apple Silicon or AMD GPU acceleration.

```python
model = AutoModel.from_pretrained("your local path", trust_remote_code=True).half().to('mps')
```

--------------------------------

### Run ChatGLM-6B API with FastAPI

Source: https://context7.com/thudm/chatglm-6b/llms.txt

Launches a FastAPI application to serve the ChatGLM-6B model. Requires model and tokenizer to be loaded. Handles POST requests for chat interactions.

```python
from fastapi import FastAPI, Request
from transformers import AutoTokenizer, AutoModel
import uvicorn, json, datetime
import torch

DEVICE = "cuda"
DEVICE_ID = "0"
CUDA_DEVICE = f"{DEVICE}:{DEVICE_ID}" if DEVICE_ID else DEVICE

def torch_gc():
    if torch.cuda.is_available():
        with torch.cuda.device(CUDA_DEVICE):
            torch.cuda.empty_cache()
            torch.cuda.ipc_collect()

app = FastAPI()

@app.post("/")
async def create_item(request: Request):
    global model, tokenizer
    json_post_raw = await request.json()
    json_post = json.dumps(json_post_raw)
    json_post_list = json.loads(json_post)
    prompt = json_post_list.get('prompt')
    history = json_post_list.get('history')
    max_length = json_post_list.get('max_length')
    top_p = json_post_list.get('top_p')
    temperature = json_post_list.get('temperature')

    response, history = model.chat(
        tokenizer,
        prompt,
        history=history,
        max_length=max_length if max_length else 2048,
        top_p=top_p if top_p else 0.7,
        temperature=temperature if temperature else 0.95
    )

    now = datetime.datetime.now()
    time = now.strftime("%Y-%m-%d %H:%M:%S")
    answer = {
        "response": response,
        "history": history,
        "status": 200,
        "time": time
    }
    torch_gc()
    return answer

if __name__ == '__main__':
    tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)
    model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).half().cuda()
    model.eval()
    uvicorn.run(app, host='0.0.0.0', port=8000, workers=1)
```

--------------------------------

### Quantize and Run Inference

Source: https://github.com/thudm/chatglm-6b/blob/main/ptuning/README.md

Apply 4-bit quantization and run a chat inference session.

```python
# Comment out the following line if you don't use quantization
model = model.quantize(4)
model = model.half().cuda()
model.transformer.prefix_encoder.float()
model = model.eval()

response, history = model.chat(tokenizer, "你好", history=[])
```

--------------------------------

### Deploy on Mac MPS

Source: https://context7.com/thudm/chatglm-6b/llms.txt

Utilize Apple Silicon GPU acceleration via the MPS backend.

```python
from transformers import AutoTokenizer, AutoModel

# 需要从本地加载模型
tokenizer = AutoTokenizer.from_pretrained("your_local_path/chatglm-6b", trust_remote_code=True)

# 使用 MPS 后端（需要 PyTorch-Nightly 2.1.0.dev）
model = AutoModel.from_pretrained("your_local_path/chatglm-6b", trust_remote_code=True).half().to('mps')

# 如果内存不足，使用CPU运行量化模型
model = AutoModel.from_pretrained("THUDM/chatglm-6b-int4", trust_remote_code=True).float()

model = model.eval()
response, history = model.chat(tokenizer, "Mac上的推理测试", history=[])
print(response)
```

--------------------------------

### Load Tokenizer

Source: https://github.com/thudm/chatglm-6b/blob/main/ptuning/README.md

Initial code for loading the tokenizer and model components.

```python
from transformers import AutoConfig, AutoModel, AutoTokenizer
```

--------------------------------

### Quantize Model for GPU

Source: https://github.com/thudm/chatglm-6b/blob/main/README.md

Load the model with 4-bit or 8-bit quantization to reduce VRAM usage.

```python
# 按需修改，目前只支持 4/8 bit 量化
model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).quantize(8).half().cuda()
```

```python
# INT8 量化的模型将"THUDM/chatglm-6b-int4"改为"THUDM/chatglm-6b-int8"
model = AutoModel.from_pretrained("THUDM/chatglm-6b-int4", trust_remote_code=True).half().cuda()
```

--------------------------------

### Citation

Source: https://github.com/thudm/chatglm-6b/blob/main/ptuning/README.md

BibTeX citation for the P-tuning paper.

```bibtex
@inproceedings{liu2022p,
  title={P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks},
  author={Liu, Xiao and Ji, Kaixuan and Fu, Yicheng and Tam, Weng and Du, Zhengxiao and Yang, Zhilin and Tang, Jie},
  booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  pages={61--68},
  year={2022}
}
```

--------------------------------

### Apply INT4 and INT8 Model Quantization

Source: https://context7.com/thudm/chatglm-6b/llms.txt

Reduce memory footprint using runtime quantization or by loading pre-quantized model weights.

```python
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)

# 方法1: 运行时量化（需要约13GB内存加载FP16模型后量化）
model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).quantize(8).half().cuda()
# 或 INT4 量化
model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).quantize(4).half().cuda()

# 方法2: 直接加载预量化模型（仅需约5.2GB内存）
# INT4 量化模型
model = AutoModel.from_pretrained("THUDM/chatglm-6b-int4", trust_remote_code=True).half().cuda()
# INT8 量化模型
model = AutoModel.from_pretrained("THUDM/chatglm-6b-int8", trust_remote_code=True).half().cuda()

model = model.eval()
response, history = model.chat(tokenizer, "量化后的模型运行测试", history=[])
print(response)
```

--------------------------------

### Load PrefixEncoder Checkpoint

Source: https://github.com/thudm/chatglm-6b/blob/main/ptuning/README.md

Load a new checkpoint containing only PrefixEncoder parameters.

```python
config = AutoConfig.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True, pre_seq_len=128)
model = AutoModel.from_pretrained("THUDM/chatglm-6b", config=config, trust_remote_code=True)
prefix_state_dict = torch.load(os.path.join(CHECKPOINT_PATH, "pytorch_model.bin"))
new_prefix_state_dict = {}
for k, v in prefix_state_dict.items():
    if k.startswith("transformer.prefix_encoder."):
        new_prefix_state_dict[k[len("transformer.prefix_encoder."):]] = v
model.transformer.prefix_encoder.load_state_dict(new_prefix_state_dict)
```

--------------------------------

### Distribute Model Across Multiple GPUs

Source: https://github.com/thudm/chatglm-6b/blob/main/README.md

Utilizes the accelerate library to split the model across multiple GPUs. Requires the utils module.

```python
from utils import load_model_on_gpus
model = load_model_on_gpus("THUDM/chatglm-6b", num_gpus=2)
```

--------------------------------

### Load Fine-tuned PrefixEncoder for ChatGLM-6B

Source: https://context7.com/thudm/chatglm-6b/llms.txt

Load fine-tuned PrefixEncoder parameters for inference with ChatGLM-6B. This method loads only the PrefixEncoder weights, not the full model.

```python
import os
import torch
from transformers import AutoConfig, AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)
CHECKPOINT_PATH = "./output/adgen-chatglm-6b-pt-128-2e-2/checkpoint-3000"

# 方法1: 加载新版Checkpoint（仅包含PrefixEncoder参数）
config = AutoConfig.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True, pre_seq_len=128)
model = AutoModel.from_pretrained("THUDM/chatglm-6b", config=config, trust_remote_code=True)

# 加载 PrefixEncoder 权重
prefix_state_dict = torch.load(os.path.join(CHECKPOINT_PATH, "pytorch_model.bin"))
new_prefix_state_dict = {}
for k, v in prefix_state_dict.items():
    if k.startswith("transformer.prefix_encoder."):
        new_prefix_state_dict[k[len("transformer.prefix_encoder."):]] = v
model.transformer.prefix_encoder.load_state_dict(new_prefix_state_dict)

# 可选量化
model = model.quantize(4)
model = model.half().cuda()
model.transformer.prefix_encoder.float()
model = model.eval()

# 方法2: 加载完整Checkpoint（全参数微调）
model = AutoModel.from_pretrained(CHECKPOINT_PATH, trust_remote_code=True)
model = model.half().cuda()
model = model.eval()

# 推理测试
response, history = model.chat(tokenizer, "类型#上衣*版型#宽松*图案#线条", history=[])
print(response)
```

--------------------------------

### Fine-tune ChatGLM-6B for Multi-turn Dialogue

Source: https://context7.com/thudm/chatglm-6b/llms.txt

Train ChatGLM-6B on multi-turn dialogue data. This script requires specific JSON formats for training and validation data, including a 'history_column'.

```bash
# 多轮对话数据格式 (train_chat.json)
# {"prompt": "长城h3风扇不转", "response": "用电脑能读数据流吗？", "history": []}
# {"prompt": "95", "response": "上下水管温差怎么样？", "history": [["长城h3风扇不转", "用电脑能读数据流吗？"]]}

# train_chat.sh
PRE_SEQ_LEN=128
LR=2e-2

CUDA_VISIBLE_DEVICES=0 python3 main.py \
    --do_train \
    --train_file train_chat.json \
    --validation_file dev_chat.json \
    --prompt_column prompt \
    --response_column response \
    --history_column history \
    --overwrite_cache \
    --model_name_or_path THUDM/chatglm-6b \
    --output_dir output/chat-chatglm-6b-pt-$PRE_SEQ_LEN-$LR \
    --overwrite_output_dir \
    --max_source_length 256 \
    --max_target_length 256 \
    --per_device_train_batch_size 1 \
    --gradient_accumulation_steps 16 \
    --predict_with_generate \
    --max_steps 3000 \
    --logging_steps 10 \
    --save_steps 1000 \
    --learning_rate $LR \
    --pre_seq_len $PRE_SEQ_LEN \
    --quantization_bit 4
```

--------------------------------

### Implement Streaming Output

Source: https://context7.com/thudm/chatglm-6b/llms.txt

Use stream_chat to generate responses incrementally for a typewriter-like effect.

```python
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)
model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).half().cuda()
model = model.eval()

# 流式生成
history = []
query = "请详细介绍一下Python编程语言"
for response, history in model.stream_chat(
    tokenizer,
    query,
    history=history,
    max_length=2048,
    top_p=0.7,
    temperature=0.95
):
    # 每次迭代返回当前已生成的完整响应
    print(f"\r{response}", end="", flush=True)

print()  # 最终换行
# history 包含完整的对话历史
```

--------------------------------

### Load Quantized ChatGLM-6B Model

Source: https://github.com/thudm/chatglm-6b/blob/main/README.md

Loads a quantized version of the model. Change the model identifier to 'THUDM/chatglm-6b-int8' for INT8 quantization.

```python
model = AutoModel.from_pretrained("THUDM/chatglm-6b-int4",trust_remote_code=True).float()
```

```python
# INT8 量化的模型将"THUDM/chatglm-6b-int4"改为"THUDM/chatglm-6b-int8"
model = AutoModel.from_pretrained("THUDM/chatglm-6b-int4",trust_remote_code=True).float()
```

--------------------------------

### JSON Lines Data Format for Feedback

Source: https://github.com/thudm/chatglm-6b/blob/main/improve/README.md

Use this format for submitting feedback data. Each line must be a JSON object containing a 'prompt' and a 'response' field.

```json
{"prompt": "请根据以下标签为商品编写一段广告\n类型#裤*版型#宽松*风格#性感*图案#线条*裤型#阔腿x s裤", "response": "宽松的阔腿裤这两年真的吸粉不少，明星时尚达人的心头爱。毕竟好穿时尚，谁都能穿出腿长2米的效果宽松的裤腿，当然是遮肉小能手啊。上身随性自然不拘束，面料亲肤舒适贴身体验感棒棒哒。系带部分增加设计看点，还让单品的设计感更强。腿部线条若隐若现的，性感撩人。颜色敲温柔的，与裤子本身所呈现的风格有点反差萌。"}
```