Yi

https://github.com/01-ai/yi
Admin
Yi is an open-source series of bilingual large language models trained by 01.AI that delivers...
Tokens:82,222
Snippets:582
Trust Score:9.1
License:Apache-2.0
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install SWIFT via git clone and pip

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/fine_tune/finetune-yi-with-swift.md

Start by cloning the SWIFT repository and installing it with the LLM extras. This is required before fine-tuning the Yi model.

```bash
git clone https://github.com/modelscope/swift.git
cd swift
pip install -e '.[llm]'
```

--------------------------------

### Set up environment for Yi-VL

Source: https://github.com/01-ai/yi/blob/main/VL/README.md

Clones the Yi repository, navigates to the VL directory, sets the PYTHONPATH, and installs required packages. Use this to set up the environment before running any Yi-VL examples.

```bash
git clone https://github.com/01-ai/Yi.git
cd Yi/VL
export PYTHONPATH=$PYTHONPATH:$(pwd)
pip install -r requirements.txt

```

--------------------------------

### Installation output

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/api/rag/yi_rag_llamaindex.ipynb

The output from running the installation commands, showing that all required packages are already satisfied. This confirms the environment is ready for the RAG example.

```text
Requirement already satisfied: llama-index in /usr/local/lib/python3.10/dist-packages (0.10.58)
Requirement already satisfied: llama-index-agent-openai<0.3.0,>=0.1.4 in /usr/local/lib/python3.10/dist-packages (from llama-index) (0.2.9)
Requirement already satisfied: llama-index-cli<0.2.0,>=0.1.2 in /usr/local/lib/python3.10/dist-packages (from llama-index) (0.1.13)
Requirement already satisfied: llama-index-core==0.10.58 in /usr/local/lib/python3.10/dist-packages (from llama-index) (0.10.58)
Requirement already satisfied: llama-index-embeddings-openai<0.2.0,>=0.1.5 in /usr/local/lib/python3.10/dist-packages (from llama-index) (0.1.11)
Requirement already satisfied: llama-index-indices-managed-llama-cloud>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from llama-index) (0.2.7)
Requirement already satisfied: llama-index-legacy<0.10.0,>=0.9.48 in /usr/local/lib/python3.10/dist-packages (from llama-index) (0.9.48)
Requirement already satisfied: llama-index-llms-openai<0.2.0,>=0.1.27 in /usr/local/lib/python3.10/dist-packages (from llama-index) (0.1.27)
Requirement already satisfied: llama-index-multi-modal-llms-openai<0.2.0,>=0.1.3 in /usr/local/lib/python3.10/dist-packages (from llama-index) (0.1.8)
Requirement already satisfied: llama-index-program-openai<0.2.0,>=0.1.3 in /usr/local/lib/python3.10/dist-packages (from llama-index) (0.1.7)
Requirement already satisfied: llama-index-question-gen-openai<0.2.0,>=0.1.2 in /usr/local/lib/python3.10/dist-packages (from llama-index) (0.1.3)
Requirement already satisfied: llama-index-readers-file<0.2.0,>=0.1.4 in /usr/local/lib/python3.10/dist-packages (from llama-index) (0.1.31)
Requirement already satisfied: llama-index-readers-llama-parse>=0.1.2 in /usr/local/lib/python3.10/dist-packages (from llama-index) (0.1.6)
Requirement already satisfied: PyYAML>=6.0.1 in /usr/local/lib/python3.10/dist-packages (from llama-index-core==0.10.58->llama-index) (6.0.1)
Requirement already satisfied: SQLAlchemy>=1.4.49 in /usr/local/lib/python3.10/dist-packages (from SQLAlchemy[asyncio]>=1.4.49->llama-index-core==0.10.58->llama-index) (2.0.31)
Requirement already satisfied: aiohttp<4.0.0,>=3.8.6 in /usr/local/lib/python3.10/dist-packages (from llama-index-core==0.10.58->llama-index) (3.9.5)
Requirement already satisfied: dataclasses-json in /usr/local/lib/python3.10/dist-packages (from llama-index-core==0.10.58->llama-index) (0.6.7)
Requirement already satisfied: deprecated>=1.2.9.3 in /usr/local/lib/python3.10/dist-packages (from llama-index-core==0.10.58->llama-index) (1.2.14)
Requirement already satisfied: dirtyjson<2.0.0,>=1.0.8 in /usr/local/lib/python3.10/dist-packages (from llama-index-core==0.10.58->llama-index) (1.0.8)
Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from llama-index-core==0.10.58->llama-index) (2023.6.0)
Requirement already satisfied: httpx in /usr/local/lib/python3.10/dist-packages (from llama-index-core==0.10.58->llama-index) (0.27.0)
Requirement already satisfied: nest-asyncio<2.0.0,>=1.5.8 in /usr/local/lib/python3.10/dist-packages (from llama-index-core==0.10.58->llama-index) (1.6.0)
Requirement already satisfied: networkx>=3.0 in /usr/local/lib/python3.10/dist-packages (from llama-index-core==0.10.58->llama-index) (3.3)
Requirement already satisfied: nltk<4.0.0,>=3.8.1 in /usr/local/lib/python3.10/dist-packages (from llama-index-core==0.10.58->llama-index) (3.8.1)
Requirement already satisfied: numpy<2.0.0 in /usr/local/lib/python3.10/dist-packages (from llama-index-core==0.10.58->llama-index) (1.25.2)
Requirement already satisfied: openai>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from llama-index-core==0.10.58->llama-index) (1.37.1)
Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from llama-index-core==0.10.58->llama-index) (2.0.3)
Requirement already satisfied: pillow>=9.0.0 in /usr/local/lib/python3.10/dist-packages (from llama-index-core==0.10.58->llama-index) (9.4.0)
Requirement already satisfied: requests>=2.31.0 in /usr/local/lib/python3.10/dist-packages (from llama-index-core==0.10.58->llama-index) (2.31.0)
Requirement already satisfied: tenacity!=8.4.0,<9.0.0,>=8.2.0 in /usr/local/lib/python3.10/dist-packages (from llama-index-core==0.10.58->llama-index) (8.5.0)

```

--------------------------------

### Install transformers and torch

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/function_calling/function_calling.ipynb

Installs the transformers and torch libraries required to run the function call example. The source warns to be mindful of GPU memory.

```python
!pip install transformers torch
```

--------------------------------

### Install Miniconda

Source: https://github.com/01-ai/yi/blob/main/finetune/README.md

Installs Miniconda and initializes conda in the bash shell. This is a prerequisite for the local server setup.

```bash
mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh
~/miniconda3/bin/conda init bash
source ~/.bashrc
```

--------------------------------

### Install ms-swift and set pip mirror

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/Inference/Inference_using_swift.ipynb

Optional pip mirror setup and installation of the ms-swift package with LLM extras. Run these commands in a Colab or Jupyter notebook.

```python
!pip config set global.index-url https://mirrors.aliyun.com/pypi/simple/
```

```python
!pip install 'ms-swift[llm]' -U
```

--------------------------------

### Install AutoGPTQ from source

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/quantization/autogptq-yi-quantization.md

Clone the AutoGPTQ repository and install it via pip. Recommended for the latest features.

```shell
git clone https://github.com/AutoGPTQ/AutoGPTQ
cd AutoGPTQ
pip install .
```

--------------------------------

### Run Yi-VL CLI quick example

Source: https://github.com/01-ai/yi/blob/main/VL/README.md

Quick example of the command-line interface using the Yi-VL-34B model, showing an interactive dialogue about the image content.

```python
CUDA_VISIBLE_DEVICES=0 python cli.py --model-path ../model/Yi-VL-34B --image-file images/cats.jpg

Human: what is in this image
Assistant: cats
Human: what are they doing
Assistant: eating
Human: what are they eating
Assistant: cat food

```

--------------------------------

### JSONL training example format

Source: https://github.com/01-ai/yi/blob/main/finetune/README.md

Example of a single training example in the JSONL format used for finetuning. The prompt field contains the instruction and the chosen field contains the desired response.

```json
{ "prompt": "Human: Who are you? Assistant:", "chosen": "I'm Yi." }
```

--------------------------------

### Run OpenWebUI Container with Docker

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/local/local-ollama.md

Starts an OpenWebUI container using Docker. Requires ollama and Docker to be installed beforehand. The container maps port 3000 to 8080 and includes a host-gateway route for connecting to the host's ollama service. The restart policy ensures the container stays running.

```bash
docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/ecosystem/Mastering_Street_Fighter_III_with_the_Yi_Language_Model.md

Installs all project dependencies listed in the requirements.txt file.

```bash
pip install -r requirements.txt
```

--------------------------------

### Expected output from quick_start.py

Source: https://github.com/01-ai/yi/blob/main/README.md

Example output showing the model's response to the prompt 'hi'.

```bash
Hello! How can I assist you today?
```

--------------------------------

### Example dataset directory structure

Source: https://github.com/01-ai/yi/blob/main/finetune/README.md

Shows the directory structure for the example datasets in finetune/yi_example_dataset, which are modified from BAAI/COIG.

```bash
|-- $DATA_PATH
    |--data
        |-- train.jsonl
        |-- eval.jsonl
```

--------------------------------

### Start Web Demo Service

Source: https://github.com/01-ai/yi/blob/main/README.md

Run the web demo locally by specifying the path to your chat model.

```bash
python demo/web_demo.py -c <your-model-path>
```

--------------------------------

### Install llama-index-llms-yi and llama-index-embeddings-huggingface

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/api/rag/yi_rag_llamaindex.ipynb

Install the required packages for using Yi with LlamaIndex. The output shows that llama-index-llms-yi is already installed and llama-index-embeddings-huggingface is being downloaded.

```bash
!pip install llama-index-llms-yi llama-index-embeddings-huggingface
```

--------------------------------

### Launch web demo for Yi-VL

Source: https://github.com/01-ai/yi/blob/main/VL/README.md

Starts a web UI demo for Yi-VL models. Requires a model path as argument.

```python
python web_demo.py --model-path path-to-yi-vl-model

```

--------------------------------

### Clone repository and install dependencies

Source: https://github.com/01-ai/yi/blob/main/README.md

Commands to clone the Yi repository and install required Python packages. Run these in your terminal to set up the environment.

```bash
git clone https://github.com/01-ai/Yi.git
cd yi
pip install -r requirements.txt
```

--------------------------------

### Start fine-tuning with llamafactory-cli

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/fine_tune/finetune-yi-with-llamafactory.md

Runs the fine-tuning process using the created configuration file. The process may take around 10 minutes.

```bash
llamafactory-cli train examples/train_qlora/yi_lora_sft_bitsandbytes.yaml
```

--------------------------------

### Install LLaMA-Factory dependencies

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/fine_tune/finetune-yi-with-llamafactory.md

Navigates into the LLaMA-Factory directory and installs the package with torch and metrics extras.

```bash
cd LLaMA-Factory
pip install -e ".[torch,metrics]"
```

--------------------------------

### Output of LangChain installation

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/rag/yi_rag_langchain.ipynb

Shows the terminal output from installing LangChain, including download progress and requirement checks. This output is provided as a reference to verify the installation succeeded.

```text
Output:
Collecting langchain
  Downloading langchain-0.2.11-py3-none-any.whl.metadata (7.1 kB)
Requirement already satisfied: PyYAML>=5.3 in /usr/local/lib/python3.10/dist-packages (from langchain) (6.0.1)
Requirement already satisfied: SQLAlchemy<3,>=1.4 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.0.31)
Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /usr/local/lib/python3.10/dist-packages (from langchain) (3.9.5)
Requirement already satisfied: async-timeout<5.0.0,>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from langchain) (4.0.3)
Collecting langchain-core<0.3.0,>=0.2.23 (from langchain)
  Downloading langchain_core-0.2.23-py3-none-any.whl.metadata (6.2 kB)
Collecting langchain-text-splitters<0.3.0,>=0.2.0 (from langchain)
  Downloading langchain_text_splitters-0.2.2-py3-none-any.whl.metadata (2.1 kB)
Collecting langsmith<0.2.0,>=0.1.17 (from langchain)
  Downloading langsmith-0.1.93-py3-none-any.whl.metadata (13 kB)
Requirement already satisfied: numpy<2,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain) (1.25.2)
Requirement already satisfied: pydantic<3,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.8.2)
Requirement already satisfied: requests<3,>=2 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.31.0)
Requirement already satisfied: tenacity!=8.4.0,<9.0.0,>=8.1.0 in /usr/local/lib/python3.10/dist-packages (from langchain) (8.5.0)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.3.1)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (23.2.0)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.4.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (6.0.5)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.9.4)
Collecting jsonpatch<2.0,>=1.33 (from langchain-core<0.3.0,>=0.2.23->langchain)
  Downloading jsonpatch-1.33-py2.py3-none-any.whl.metadata (3.0 kB)
Requirement already satisfied: packaging<25,>=23.2 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.3.0,>=0.2.23->langchain) (24.1)
Collecting orjson<4.0.0,>=3.9.14 (from langsmith<0.2.0,>=0.1.17->langchain)
  Downloading orjson-3.10.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (50 kB)
[2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m50.4/50.4 kB[0m [31m1.1 MB/s[0m eta [36m0:00:00[0m
[?25hRequirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain) (0.7.0)
Requirement already satisfied: pydantic-core==2.20.1 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain) (2.20.1)
Requirement already satisfied: typing-extensions>=4.6.1 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain) (4.12.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain) (2024.7.4)
Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/dist-packages (from SQLAlchemy<3,>=1.4->langchain) (3.0.3)
Collecting jsonpointer>=1.9 (from jsonpatch<2.0,>=1.33->langchain-core<0.3.0,>=0.2.23->langchain)
  Downloading jsonpointer-3.0.0-py2.py3-none-any.whl.metadata (2.3 kB)
Downloading langchain-0.2.11-py3-none-any.whl (990 kB)
[2K   [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m990.3/990.3 kB[0m [31m15.5 MB/s[0m eta [36m0:00:00[0m
[?25hDownloading langchain_core-0.2.23-py3-none-any.whl (374 kB)
[2K   [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m374.2/374.2 kB[0m [31m22.7 MB/s[0m eta [36m0:00:00[0m
```

--------------------------------

### Copy Environment File

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/ecosystem/Mastering_Street_Fighter_III_with_the_Yi_Language_Model.md

Copies the example environment file to create the .env file, which will be edited to set the YI_API_KEY.

```bash
cp .env.example .env
```

--------------------------------

### Install Homebrew and Python on macOS

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/ecosystem/Building_a_Mind_Map_Generator_Powered_by_Yi-Large.md

Commands to install Homebrew, then Python via Homebrew, and verify the installation by checking the Python version.

```bash
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
```

```bash
brew install python
```

```bash
python3 --version
```

--------------------------------

### Test the Q&A system with sample queries

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/ecosystem/Building_an_Intelligent_Q&A_System_Based_on_LlamaIndex_and_Yi-large.md

Example user queries and the corresponding responses from the Yi-large model, demonstrating the system's ability to answer questions about LlamaIndex capabilities.

```shell
user>>Can llama-index query data in SQL and CSV?
```

```shell
Yi-large>> Yes, llama-index can query data in SQL and CSV formats.
It supports text-to-SQL operations for querying SQL databases and 
text-to-Pandas operations for working with CSV files. This allows 
users to interact with structured data sources using natural language
queries, making it easier to extract information from these types of data.
```

```shell
user>>I want to use the Yi-large model in python, what code do I need?
```

```shell
Yi-large>> To use the Yi-large model in Python, you can use the following code as a guide. This code snippet demonstrates how to set up the OpenAI SDK to work with the Yi-large model:
```

--------------------------------

### Install quantization dependencies (awq, gptq, bnb, hqq, eetq)

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/quantization/swift-yi-quantization.md

Installs the quantization method of choice: awq, gptq, bnb, hqq, or eetq. Note that autoawq and auto_gptq require matching CUDA versions, and hqq/eetq may need transformers>=4.41.

```bash
# Use awq quantization:
# autoawq and cuda versions have a correspondence, please select the version according to `https://github.com/casper-hansen/AutoAWQ`
pip install autoawq -U

# Use gptq quantization:
# auto_gptq and cuda versions have a correspondence, please select the version according to `https://github.com/PanQiWei/AutoGPTQ#quick-installation`
pip install auto_gptq -U

# Use bnb quantization:
pip install bitsandbytes -U

# Use hqq quantization:
# pip install transformers>=4.41
pip install hqq

# Use eetq quantization:
# pip install transformers>=4.41

# Refer to https://github.com/NetEase-FuXi/EETQ
git clone https://github.com/NetEase-FuXi/EETQ.git
cd EETQ/
git submodule update --init --recursive
pip install .
```

--------------------------------

### Install huggingface_hub via pip

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/local/local-llama.cpp.md

Install the huggingface_hub library to download models from Hugging Face using the CLI. This step can be skipped if you have a locally fine-tuned or quantized model.

```bash
pip install huggingface_hub
```

--------------------------------

### Install git-lfs

Source: https://github.com/01-ai/yi/blob/main/docs/README_llama.cpp.md

Installs git-lfs via Homebrew on macOS.

```bash
brew install git-lfs
```

--------------------------------

### Install LangChain

Source: https://github.com/01-ai/yi/blob/main/Cookbook/cn/api/rag/yi_rag_langchain.ipynb

Installs the core LangChain package using pip.

```python
!pip install langchain
```

--------------------------------

### Install AutoAWQ from source for CUDA 11.8, ROCm 5.6, and ROCm 5.7

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/quantization/autoawq-yi-quantization.md

For CUDA 11.8, ROCm 5.6, and ROCm 5.7, install AutoAWQ from source by cloning the repository and running pip install -e .

```shell
git clone https://github.com/casper-hansen/AutoAWQ.git
cd AutoAWQ
pip install -e .
```

--------------------------------

### Install Dependencies with Micromamba

Source: https://github.com/01-ai/yi/blob/main/README.md

Create a conda environment named 'yi' and install dependencies from the lock file.

```bash
micromamba install -y -n yi -f conda-lock.yml
```

--------------------------------

### Load and generate with MLX-LM

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/local/local-mlx.md

Python example that loads the 'mlx-community/Yi-1.5-6B-Chat-8bit' model and generates a response to the prompt 'hello'. You can replace the model with others like 'mlx-community/Yi-1.5-34B-Chat-4bit'.

```python
from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Yi-1.5-6B-Chat-8bit")

response = generate(model, tokenizer, prompt="hello", verbose=True)
```

--------------------------------

### Install Flask, OpenAI, and PyPDF2 with pip3

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/ecosystem/Building_a_Mind_Map_Generator_Powered_by_Yi-Large.md

Installs the Python libraries needed for the application: Flask for the web framework, OpenAI for the AI interface, and PyPDF2 for PDF processing.

```bash
pip3 install flask openai PyPDF2
```

--------------------------------

### Install langchain-openai package

Source: https://github.com/01-ai/yi/blob/main/Cookbook/cn/api/rag/yi_rag_langchain.ipynb

Installs the langchain-openai integration package required to use ChatOpenAI with a custom base URL.

```python
!pip install -qU langchain-openai
```

--------------------------------

### Start Yi Docker Container

Source: https://github.com/01-ai/yi/blob/main/README.md

Run the Yi-34B-chat model locally using Docker with GPU support.

```bash
docker run -it --gpus all \
-v <your-model-path>: /models
ghcr.io/01-ai/yi:latest
```

--------------------------------

### Install LlamaIndex dependencies for function calling

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/api/function_calling/function_calling_llamaindex.ipynb

Install the required packages: llama-index, llama-index-llms-huggingface, and llama-index-llms-yi. Run these commands in a notebook or terminal before proceeding.

```python
!pip install llama-index
!pip install llama-index-llms-huggingface
!pip install llama-index-llms-yi
```

--------------------------------

### app.py - Flask setup and Yi-Large API configuration

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/ecosystem/Building_a_Mind_Map_Generator_Powered_by_Yi-Large.md

Imports required libraries, creates the Flask app instance, configures the upload folder and allowed extensions, and sets up the OpenAI client with the Yi-Large API base URL. Replace 'Your API Key' with your actual API key.

```python
from flask import Flask, render_template, request, jsonify, send_from_directory, Response
import os
import json
from werkzeug.utils import secure_filename
from openai import OpenAI
import openai
from PyPDF2 import PdfReader

app = Flask(__name__)
app.config['UPLOAD_FOLDER'] = 'uploads'
app.config['ALLOWED_EXTENSIONS'] = {'txt', 'pdf', 'doc', 'docx'}

API_BASE = "https://api.01.ai/v1"
API_KEY = "Your API Key"  # Please replace with your actual API key
client = OpenAI(
    api_key=API_KEY,
    base_url=API_BASE
)
```

--------------------------------

### Example inference output

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/Inference/Inference_using_swift.ipynb

Example output from the inference code above, showing the model's response to the query 'Hello!'.

```text
query: Hello!
response: Hi! How can I help you today?
```

--------------------------------

### Run Yi-VL-34B single inference example

Source: https://github.com/01-ai/yi/blob/main/VL/README.md

Quick example of single inference using the Yi-VL-34B model on a local image. The output is not deterministic due to temperature 0.2.

```python
CUDA_VISIBLE_DEVICES=0 python single_inference.py --model-path ../model/Yi-VL-34B --image-file images/cats.jpg --question "Describe the cats and what they are doing in detail."

```

--------------------------------

### Launch API server for Yi-1.5-6B model

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/Inference/Inference_using_lmdeploy.ipynb

Starts the lmdeploy 'server' command with the Qwen2ForCausalLM model. Sets cache directory, server port 23333, tensor parallel size 1, and custom session length.

```bash
lmdeploy serve api_server /path/to/yi-1.5-6b-chat \
    --cache-directory /workspace/lmdeploy --server-port 23333 --tp 1 --session-len 12352
```

--------------------------------

### Quantize and save the Yi model

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/quantization/autogptq-yi-quantization.md

Prepare example data as a list of dictionaries with input_ids and attention_mask, then quantize and save the model. Ensure the data format matches the required structure.

```python
import torch
examples = []
messages = [
    {"role": "user", "content": "hi"},
    {"role": "assistant", "content": "Hello! It's great to see you today. How can I assist you"}
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)
model_inputs = tokenizer([text])
input_ids = torch.tensor(model_inputs.input_ids[:max_len], dtype=torch.int)
examples.append(dict(input_ids=input_ids, attention_mask=input_ids.ne(tokenizer.pad_token_id)))
model.quantize(examples)

model.save_quantized(quant_path, use_safetensors=True)
tokenizer.save_pretrained(quant_path)
```

--------------------------------

### Install LlamaIndex and Yi dependencies

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/ecosystem/Building_an_Intelligent_Q&A_System_Based_on_LlamaIndex_and_Yi-large.md

Install the required Python packages for LlamaIndex, the Yi LLM integration, and supporting libraries for file reading and embeddings.

```shell
pip install llama-index
pip install llama-index-llms-yi
pip install llama-index-core llama-index-readers-file llama-index-embeddings-huggingface
```

--------------------------------

### Initialize web chatbot server

Source: https://github.com/01-ai/yi/blob/main/README.md

Starts the llama.cpp server to host a web-based chatbot interface. The server listens on port 8080 by default.

```bash
cd llama.cpp
./server --ctx-size 2048 --host 0.0.0.0 --n-gpu-layers 64 --model /Users/yu/yi-chat-6B-GGUF/yi-chat-6b.Q2_K.gguf
```

```bash
...\n\nllama_new_context_with_model: n_ctx      = 2048\nllama_new_context_with_model: freq_base  = 5000000.0\nllama_new_context_with_model: freq_scale = 1\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M2 Pro\nggml_metal_init: picking default device: Apple M2 Pro\nggml_metal_init: ggml.metallib not found, loading from source\nggml_metal_init: GGML_METAL_PATH_RESOURCES = nil\nggml_metal_init: loading '/Users/yu/llama.cpp/ggml-metal.metal'\nggml_metal_init: GPU name:   Apple M2 Pro\nggml_metal_init: GPU family: MTLGPUFamilyApple8 (1008)\nggml_metal_init: hasUnifiedMemory              = true\nggml_metal_init: recommendedMaxWorkingSetSize  = 11453.25 MB\nggml_metal_init: maxTransferRate               = built-in GPU\nggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =   128.00 MiB, ( 2629.44 / 10922.67)\nllama_new_context_with_model: KV self size  =  128.00 MiB, K (f16):   64.00 MiB, V (f16):   64.00 MiB\nggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =     0.02 MiB, ( 2629.45 / 10922.67)\nllama_build_graph: non-view tensors processed: 676/676\nllama_new_context_with_model: compute buffer total size = 159.19 MiB\nggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =   156.02 MiB, ( 2785.45 / 10922.67)\nAvailable slots:\n-> Slot 0 - max context: 2048\n\nllama server listening at http://0.0.0.0:8080
```

--------------------------------

### Deploy Yi-1.5-6B with lmdeploy using backend and batch size

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/Inference/Inference_using_lmdeploy.ipynb

Starts 'lmdeploy serve api_server' specifying the model path, backend 'pytorch', server port 5000, cache directory, and batch size 8.

```bash
lmdeploy serve api_server path/to/yi-1.5-6b-chat \
    --backend pytorch --server-port 5000 --cache-directory /workspace/lmdeploy --batch-size 8
```

--------------------------------

### Run Yi-VL command line interface

Source: https://github.com/01-ai/yi/blob/main/VL/README.md

Starts a command-line interface for conversational inference with Yi-VL. Provide a model path and an image path.

```python
python cli.py --model-path path-to-yi-vl-model --image path-to-image

```

--------------------------------

### Fine-tuning configuration (yi_lora_sft_bitsandbytes.yaml)

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/fine_tune/finetune-yi-with-llamafactory.md

Example YAML configuration for QLoRA fine-tuning of Yi model with LLaMA-Factory. Uses the identity dataset, sets quantization to 4 bits, and specifies training parameters.

```yaml
### model
model_name_or_path: <Path to your downloaded model, e.g., ../Yi-1.5-6B-Chat>
quantization_bit: 4

### method
stage: sft
do_train: true
finetuning_type: lora
lora_target: all

### dataset
dataset: identity
template: yi
cutoff_len: 1024
max_samples: 1000
overwrite_cache: true
preprocessing_num_workers: 16

### output
output_dir: saves/yi-6b/lora/sft
logging_steps: 10
save_steps: 500
plot_loss: true
overwrite_output_dir: true

### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 1.0e-4
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
fp16: true

### eval
val_size: 0.1
per_device_eval_batch_size: 1
eval_strategy: steps
eval_steps: 500
```

--------------------------------

### Usage example for function calling with Yi-1.5-9B-Chat

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/function_calling/function_calling.ipynb

Shows an example interaction where the user asks 'What is 5 plus 3?' and the model responds with a function call, which is then executed to produce the result 8.

```python
Human: What is 5 plus 3?
Model response: Here's the function call to perform the addition:
{"function": "plus", "arguments": {"a": 5, "b": 3}}
Assistant: The result of plus({'a': 5, 'b': 3}) is 8
```

--------------------------------

### SWIFT dataset format example

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/ecosystem/Enhancing_the_Mathematical_and_Logical_Reasoning_Abilities_of_Yi-1.5-6B-Chat.md

Example of the JSON Lines format required by SWIFT for training datasets. Each line is a training sample with optional system, query, and response fields. Use this format for custom datasets like HJ_math.jsonl.

```json
{"system": "You are a math expert", "query": "What is the result of (1+3)*10+99?", "response": "First, calculate 1+3=4, then 4*10=40, finally 40+99=139, so the result is 139"}
{"query": "Solve the following equation: x^2 - 4x + 4 = 0", "response": "The solution to the equation x^2 - 4x + 4 = 0 is x = 2 (double root)."}
{"query": "Solve the following system of equations: x + y = 5 and 2x - y = 1", "response": "Solving this system of equations, we get x = 2, y = 3."}
```

--------------------------------

### Install required libraries for Yi inference

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/Inference/Inference_using_transformers.ipynb

Install the necessary Python packages including transformers, gradio, torch, accelerate, and sentencepiece. This step is required before loading the Yi model.

```python
!pip install transformers>=4.36.2
!pip install gradio>=4.13.0
!pip install torch>=2.0.1,<=2.3.0
!pip install accelerate
!pip install sentencepiece
```

--------------------------------

### Example agent conversation output

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/api/function_calling/function_calling_llamaindex.ipynb

Shows the agent's reasoning and tool calls when solving (1+2)*10. The agent uses the plus tool to add 1+2, then the multiply tool to multiply 3*10, and finally answers with the result 30.

```text
user>>(1+2)*10
Thought: The user's question is in English. I need to use a combination of the 'plus' and 'multiply' tools to solve the mathematical expression (1+2)*10.
Action: plus
Action Input: {'a': 1, 'b': 2}
Observation: 3
Thought: I have the result of the 'plus' operation, which is 3. Now I need to multiply this result by 10 to complete the expression (1+2)*10.
Action: multiply
Action Input: {'a': 3, 'b': 10}
Observation: 30
Thought: I have the result of the 'multiply' operation, which is 30. This is the final result of the expression (1+2)*10.
Thought: I can answer without using any more tools. I'll use the user's language to answer.
Answer: The result of the expression (1+2)*10 is 30.
```

--------------------------------

### Yi-9B base model inference output

Source: https://github.com/01-ai/yi/blob/main/README.md

Example output from the Yi-9B model showing a quick sort implementation in Python.

```bash
# write the quick sort algorithm
def quick_sort(arr):
    if len(arr) <= 1:
        return arr
    pivot = arr[len(arr) // 2]
    left = [x for x in arr if x < pivot]
    middle = [x for x in arr if x == pivot]
    right = [x for x in arr if x > pivot]
    return quick_sort(left) + middle + quick_sort(right)

# test the quick sort algorithm
print(quick_sort([3, 6, 8, 10, 1, 2, 1]))
```

--------------------------------

### Fine-tune Yi-1.5-6B-Chat with LoRA using SWIFT

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/ecosystem/Enhancing_the_Mathematical_and_Logical_Reasoning_Abilities_of_Yi-1.5-6B-Chat.md

Run this bash command to start LoRA fine-tuning of the Yi-1.5-6B-Chat model with SWIFT. It uses three datasets (two remote with 5000 samples each and one local), sets training epochs to 1, max sequence length to 1024, and auto-selects data type. Fine-tuning may take several hours depending on hardware.

```bash
CUDA_VISIBLE_DEVICES=0 swift sft \
    --model_id_or_path 01ai/Yi-1.5-6B-Chat \
    --dataset ruozhiba#5000 AI-ModelScope/blossom-math-v2#5000 /content/HJ_math.jsonl \
    --output_dir output \
    --sft_type lora \
    --num_train_epochs 1 \
    --max_length 1024 \
    --dtype AUTO
```

--------------------------------

### Example Output from swift infer

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/ecosystem/yi-vl-best-practice.md

Shows the interactive output from the swift infer command. The model accepts local paths or URLs for images and responds to queries about the image content.

```python
"""
<<< Describe this type of image
Input a media path or URL <<< http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png
The image shows a kitten sitting on the floor, eyes open, staring at the camera. The kitten looks very cute, with gray and white fur, and blue eyes. It seems to be looking at the camera, possibly curious about the surroundings.
--------------------------------------------------
<<< How many sheep are in the picture
Input a media path or URL <<< http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png
There are four sheep in the image.
--------------------------------------------------
<<< clear
<<< What is the calculation result
Input a media path or URL <<< http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/math.png
1452 + 45304 = 46756
--------------------------------------------------
<<< clear
<<< Write a poem based on the content in the image
Input a media path or URL <<< http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/poem.png
Night falls, starlight twinkles,
A small boat drifts on the river,
A bright lantern hangs on the bow,
Illuminating the surrounding darkness.

Two people are on the boat,
One at the bow, the other at the stern,
They seem to be talking,
Enjoying a tranquil moment under the starlight.

On the riverbank, trees stand in the dark,
Casting long shadows in the starlight.
The scene is so peaceful,
Reminiscent of an ancient legend.

The boat, the people, and the starlight,
Form a beautiful picture,
Evoking a feeling of serenity,
Beyond the hustle and bustle of city life.
"""
```

--------------------------------

### Use Yi-large with OpenAI SDK

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/ecosystem/Building_an_Intelligent_Q&A_System_Based_on_LlamaIndex_and_Yi-large.md

A code example showing how to use the Yi-large model via the OpenAI SDK, including setting the API base URL and making a chat completion request.

```python
import openai
from openai import OpenAI

API_BASE = "https://api.01.ai/v1"
API_KEY = "your key"

client = OpenAI(
    api_key=API_KEY,
    base_url=API_BASE
)

completion = client.chat.completions.create(
    model="yi-large",
    messages=[{"role": "user", "content": "Hi, who are you?"}]
)

print(completion)
```

--------------------------------

### Run quick_start.py

Source: https://github.com/01-ai/yi/blob/main/README.md

Command to execute the quick_start.py script for Yi chat model inference.

```bash
python quick_start.py
```

--------------------------------

### Verify Conda Installation

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/ecosystem/Mastering_Street_Fighter_III_with_the_Yi_Language_Model.md

Verifies that Conda is installed correctly. If a version number is displayed, the installation was successful.

```bash
conda --version
```

--------------------------------

### Install llama.cpp Python dependencies

Source: https://github.com/01-ai/yi/blob/main/docs/README_llama.cpp.md

Installs Python dependencies required by llama.cpp.

```bash
cd llama.cpp
python3 -m pip install -r requirements.txt
```

--------------------------------

### Build a knowledge base with VectorStoreIndex

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/api/rag/yi_rag_llamaindex.ipynb

Load documents from the 'data' directory and create an index. Then create a query engine to query the index.

```python
documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
```

--------------------------------

### Install MLX-LM with pip

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/local/local-mlx.md

Install the MLX-LM package using pip. Note that MLX-LM is only compatible with Mac OS.

```bash
pip install mlx-lm
```

--------------------------------

### Show llama-quantize help

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/local/local-llama.cpp.md

Displays the help and usage information for llama-quantize, including available quantization types and options.

```bash
./llama-quantize -h
```

--------------------------------

### Install lmdeploy with pip

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/Inference/Inference_using_lmdeploy.ipynb

Installs the lmdeploy library and its dependencies. Output shows version 0.5.1 being downloaded for Python 3.10.

```python
!pip install lmdeploy
```

--------------------------------

### Install AutoAWQ via pip for CUDA >= 12.1

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/quantization/autoawq-yi-quantization.md

Install AutoAWQ using pip. This method requires CUDA version 12.1 or higher.

```shell
pip install autoawq
```

--------------------------------

### Create ReAct agent with tools

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/api/function_calling/function_calling_llamaindex.ipynb

Instantiate a ReActAgent with the three tools and the configured Yi model. The verbose=True flag enables detailed output during agent reasoning.

```python
agent = ReActAgent.from_tools([multiply_tool,plus_tool,minus_tool], llm=llm, verbose=True)
```

--------------------------------

### Initialize Yi API and set up LlamaIndex settings

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/api/rag/yi_rag_llamaindex.ipynb

Import the necessary modules and initialize the Yi model with your API key. Set the global LlamaIndex settings to use the Yi model and a HuggingFace embedding model.

```python
from llama_index.llms.yi import Yi
from llama_index.readers.web import SimpleWebPageReader
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings

llm = Yi(model="yi-large", api_key="api_key")
Settings.llm = llm
Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-base-zh-v1.5")
```

--------------------------------

### Check PyTorch version for AutoAWQ compatibility

Source: https://github.com/01-ai/yi/blob/main/Cookbook/en/opensource/quantization/autoawq-yi-quantization.md

Before installing AutoAWQ, check the torch version to ensure compatibility. The installation via pip requires CUDA >= 12.1.

```python
import torch
print(torch.__version__)
```

--------------------------------

### Run Yi-6B finetuning script

Source: https://github.com/01-ai/yi/blob/main/finetune/README.md

Runs the finetuning script for the Yi-6B base model. Setting training_debug_steps=20 and num_train_epochs=4 can output a chat model in about 20 minutes.

```bash
cd finetune/scripts

bash run_sft_Yi_6b.sh
```

--------------------------------

### Run the model

Source: https://github.com/01-ai/yi/blob/main/docs/README_llama.cpp.md

Runs the quantized model with the --chatml flag to interact via prompts.

```bash
./main -m $MODEL_PATH/ggml-model-q4_0.gguf --chatml
```

--------------------------------

### Create conda environment and install dependencies

Source: https://github.com/01-ai/yi/blob/main/finetune/README.md

Creates a conda environment named dev_env with Python 3.10 and installs the required packages for finetuning, including torch, deepspeed, and transformers.

```bash
conda create -n dev_env python=3.10 -y
conda activate dev_env
pip install torch==2.0.1 deepspeed==0.10 tensorboard transformers datasets sentencepiece accelerate ray==2.7
```
Tokens:
Raw