### Running the Voice Assistant Example

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/GPU/HuggingFace/Multimodal/voiceassistant/README.md

Execute this command to start the voice assistant. Ensure you have the necessary model paths for Llama 2 and Whisper.

```bash
python generate.py --llama2-repo-id-or-model-path /mnt/windows/demo/models/Llama-2-7b-chat-hf --whisper-repo-id-or-model-path /mnt/windows/demo/models/whisper-medium
```

--------------------------------

### Install lm-evaluation-harness

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/dev/benchmark/harness/README.md

Clone the repository, checkout a specific version, and install the harness locally. Ensure IPEX-LLM is installed beforehand.

```bash
git clone https://github.com/EleutherAI/lm-evaluation-harness.git
cd lm-evaluation-harness
git checkout b281b09
pip install -e .
```

--------------------------------

### Install IPEX-LLM and Transformers on Windows

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/GPU/HuggingFace/LLM/qwen2/README.md

Installs IPEX-LLM with XPU support and the transformers library for Qwen2 models on Windows. Includes optional ModelScope installation.

```bash
conda create -n llm python=3.11 libuv
conda activate llm
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
pip install transformers==4.37.0

pip install modelscope==1.11.0
```

--------------------------------

### Start Docker Container for LLM Fine-Tuning

Source: https://github.com/intel/ipex-llm/blob/main/docker/llm/README_backup.md

Prepare your environment by setting model and data paths, proxy settings, and then launch the Docker container with necessary device and volume mounts.

```bash
export BASE_MODE_PATH=your_downloaded_base_model_path
export DATA_PATH=your_downloaded_data_path
export HTTP_PROXY=your_http_proxy
export HTTPS_PROXY=your_https_proxy
export CONTAINER_NAME=my_container
export DOCKER_IMAGE=intelanalytics/ipex-llm-finetune-xpu:2.2.0-SNAPSHOT

docker run -itd \
   --net=host \
   --device=/dev/dri \
   --memory="32G" \
   --name=$CONTAINER_NAME \
   -e http_proxy=${HTTP_PROXY} \
   -e https_proxy=${HTTPS_PROXY} \
   -v $BASE_MODE_PATH:/model \
   -v $DATA_PATH:/data/alpaca-cleaned \
   --shm-size="16g" \
   $DOCKER_IMAGE
```

--------------------------------

### Install IPEX-LLM and Dependencies (Linux)

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/CPU/HF-Transformers-AutoModels/Model/phi-1_5/README.md

Installs the IPEX-LLM library with all options, along with einops and a specific version of transformers, within a conda environment on Linux. This setup is required to run the generation example.

```bash
conda create -n llm python=3.11
conda activate llm

pip install --pre --upgrade ipex-llm[all] --extra-index-url https://download.pytorch.org/whl/cpu
pip install einops
pip install "transformers>=4.37.0,<4.42.0"
```

--------------------------------

### Install IPEX 2.1.0

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/CPU/Speculative-Decoding/Self-Speculation/README.md

Install a specific version of Intel Extension for PyTorch (IPEX). This version is recommended for the examples.

```bash
pip install intel_extension_for_pytorch==2.1.0
```

--------------------------------

### Install IPEX-LLM and Initialize Environment

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/CPU/HF-Transformers-AutoModels/Model/README.md

Install the IPEX-LLM library with all extras and configure the environment variables for optimal performance on Linux systems.

```bash
pip install --pre --upgrade ipex-llm[all] --extra-index-url https://download.pytorch.org/whl/cpu
source ipex-llm-init
```

--------------------------------

### Install LangChain Dependencies

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/CPU/LangChain/README.md

Installs necessary LangChain and related community packages. Run this before executing LangChain examples.

```bash
pip install -U langchain langchain-community
```

--------------------------------

### Install Helm Chart for IPEX-LLM QLoRA Finetuning

Source: https://github.com/intel/ipex-llm/blob/main/docker/llm/finetune/qlora/cpu/kubernetes/README.md

Submit the finetuning job to Kubernetes after preparing parameters in the values.yaml file. Ensure you are in the ./kubernetes directory.

```bash
cd ./kubernetes
helm install ipex-llm-qlora-finetuning .
```

--------------------------------

### Install LangChain and Transformers for Voice Assistant

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/CPU/LangChain/README.md

Installs LangChain, community, and a specific version of transformers required for the voice assistant example.

```bash
pip install -U langchain langchain-community
pip install transformers==4.36.2
```

--------------------------------

### Start QLoRA Fine-Tuning Script

Source: https://github.com/intel/ipex-llm/blob/main/docker/llm/finetune/qlora/cpu/docker/README.md

Navigates to the IPEX LLM directory and executes the script to start QLoRA fine-tuning on CPU.

```bash
cd /ipex_llm
bash start-qlora-finetuning-on-cpu.sh
```

--------------------------------

### Set Up Environment and Run llama.cpp Inference

Source: https://github.com/intel/ipex-llm/blob/main/docs/mddocs/DockerGuides/docker_cpp_xpu_quickstart.md

Navigate to the scripts directory, source the IPEX-LLM initialization script for GPU, and then execute the start script for llama.cpp inference. Ensure model paths are correctly configured.

```bash
cd /llm/scripts/
source ipex-llm-init --gpu --device $DEVICE
bash start-llama-cpp.sh
```

--------------------------------

### Install LangChain Dependencies for RAG

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/CPU/LangChain/README.md

Installs LangChain, community, Chroma, and sentence-transformers for the RAG example. Ensure version 3.0.1 for sentence-transformers.

```bash
pip install -U langchain langchain-community langchain-chroma sentence-transformers==3.0.1
```

--------------------------------

### Verify ipex-llm Installation

Source: https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quickstart/install_linux_gpu.md

Verify the installation by sourcing oneAPI environment variables, starting a Python interpreter, and importing necessary classes from ipex_llm.transformers.

```bash
source /opt/intel/oneapi/setvars.sh

python

> from ipex_llm.transformers import AutoModel, AutoModelForCausalLM
```

--------------------------------

### Start QLoRA Fine-Tuning on XPU

Source: https://github.com/intel/ipex-llm/blob/main/docker/llm/finetune/xpu/README.md

Execute this script to initiate the QLoRA fine-tuning process on the XPU. Monitor the output for training metrics.

```bash
bash start-qlora-finetuning-on-xpu.sh
```

--------------------------------

### Install Required Packages for MiniCPM-o-2_6

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/GPU/HuggingFace/Multimodal/MiniCPM-o-2_6/README.md

Install the necessary Python packages for running MiniCPM-o-2_6 examples with IPEX-LLM. Ensure you have the correct conda environment activated.

```bash
conda activate llm

# refer to: https://huggingface.co/openbmb/MiniCPM-o-2_6#usage
pip install transformers==4.44.2 trl
pip install librosa==0.9.0
pip install soundfile==0.12.1
pip install moviepy
```

--------------------------------

### Run Qwen2-Audio Example

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/GPU/HuggingFace/Multimodal/qwen2-audio/README.md

Execute the generate.py script to run the Qwen2-Audio example. Specify the model repository ID or path using the --repo-id-or-model-path argument.

```bash
python ./generate.py --repo-id-or-model-path REPO_ID_OR_MODEL_PATH
```

--------------------------------

### Install IPEX-LLM for XPU on Windows

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/GPU/PyTorch-Models/Model/bark/README.md

Install IPEX-LLM with XPU support and other necessary libraries for running examples on Windows. It is recommended to use conda for environment management.

```bash
conda create -n llm python=3.11 libuv
conda activate llm

# below command will install intel_extension_for_pytorch==2.1.10+xpu as default
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/

pip install scipy
```

--------------------------------

### Launch QLoRA Finetuning with Accelerate

Source: https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quickstart/axolotl_quickstart.md

Execute the finetuning process using `accelerate launch` with the prepared `llama3-qlora.yml` configuration file. The `finetune.py` script is used for this purpose.

```bash
accelerate launch finetune.py llama3-qlora.yml
```

--------------------------------

### Install IPEX-LLM and Dependencies

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/CPU/Speculative-Decoding/EAGLE/README.md

Installs the necessary libraries for running IPEX-LLM examples, including PyTorch, Intel Extension for PyTorch, and EAGLE-LLM. Ensure you are in a conda environment with Python 3.11.

```bash
conda create -n llm python=3.11 # recommend to use Python 3.11
conda activate llm

pip install --pre --upgrade ipex-llm[all] --extra-index-url https://download.pytorch.org/whl/cpu
pip install intel_extension_for_pytorch==2.1.0 
pip install -r requirements.txt
pip install transformers==4.36.2
pip install gradio==3.50.2
pip install eagle-llm
```

--------------------------------

### Launch Finetuning with train.py

Source: https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quickstart/axolotl_quickstart.md

Alternatively, launch the finetuning process using `accelerate launch` with the `train.py` script and the `llama3-qlora.yml` configuration file.

```bash
accelerate launch train.py llama3-qlora.yml
```

--------------------------------

### Install IPEX-LLM and Dependencies (Windows)

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/CPU/HF-Transformers-AutoModels/Model/phi-3-vision/README.md

Installs IPEX-LLM and other required packages for running Phi-3-Vision examples on Windows. It's recommended to use Python 3.11 and manage the environment with conda.

```cmd
conda create -n llm python=3.11
conda activate llm
pip install --pre --upgrade ipex-llm[all]
pip install pillow torchvision
pip install "transformers>=4.37.0,<4.42.3"
```

--------------------------------

### Run Lightweight Serving Example

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/GPU/Lightweight-Serving/README.md

Executes the lightweight serving script with specified model repository, low-bit optimization, and port. Default values are provided for each argument.

```bash
python ./lightweight_serving.py --repo-id-or-model-path REPO_ID_OR_MODEL_PATH --low-bit LOW_BIT --port PORT
```

--------------------------------

### Run Qwen-VL Chat Example (Client)

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/CPU/HF-Transformers-AutoModels/Model/qwen-vl/README.md

Executes the chat.py example for Qwen-VL models on client Windows machines, utilizing all available cores for optimal performance.

```cmd
python ./chat.py
```

--------------------------------

### Install IPEX-LLM for Baichuan on Windows

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/CPU/HF-Transformers-AutoModels/Model/baichuan/README.md

Installs the IPEX-LLM library and the transformers_stream_generator package on a Windows environment using conda. This setup is necessary for running Baichuan models with INT4 optimization.

```cmd
conda create -n llm python=3.11
conda activate llm

pip install --pre --upgrade ipex-llm[all]
pip install transformers_stream_generator
```

--------------------------------

### Install IPEX-LLM and Dependencies (Linux)

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/CPU/HF-Transformers-AutoModels/Model/phi-3-vision/README.md

Installs IPEX-LLM with all options and other required packages for running Phi-3-Vision examples on Linux. It's recommended to use Python 3.11 and manage the environment with conda.

```bash
conda create -n llm python=3.11
conda activate llm
pip install --pre --upgrade ipex-llm[all] --extra-index-url https://download.pytorch.org/whl/cpu
pip install pillow torchvision
pip install "transformers>=4.37.0,<4.42.3"
```

--------------------------------

### Run Docker Container for Finetuning

Source: https://github.com/intel/ipex-llm/blob/main/docker/llm/finetune/lora/cpu/docker/README.md

Start a Docker container, mounting your base model and data paths. Configure CPU, worker count, and other environment variables as needed. Replace placeholders like 'your_expected_range_of_cpu_numbers', 'your_worker_count', 'your_downloaded_base_model_path', and 'your_downloaded_data_path'.

```bash
docker run -itd \
 --name=ipex-llm-fintune-lora-cpu \
 --cpuset-cpus="your_expected_range_of_cpu_numbers" \
 -e STANDALONE_DOCKER=TRUE \
 -e WORKER_COUNT_DOCKER=your_worker_count \
 -v your_downloaded_base_model_path:/ipex_llm/model \
 -v your_downloaded_data_path:/ipex_llm/data/alpaca_data_cleaned_archive.json \
 intelanalytics/ipex-llm-finetune-lora-cpu:2.2.0-SNAPSHOT \
 bash
```

--------------------------------

### Setup FastChat and AutoGen Environment

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/GPU/Applications/autogen/README.md

Clones the FastChat repository and installs it along with the AutoGen library.

```bash
# clone the FastChat
git clone https://github.com/lm-sys/FastChat.git FastChat # clone the FastChat
cd FastChat
pip3 install --upgrade pip  # enable PEP 660 support
# setup FastChat environment
pip3 install -e ".[model_worker,webui]"

# setup AutoGen environment
pip install pyautogen==0.2.7
```

--------------------------------

### Install IPEX-LLM and Initialize Environment

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/CPU/PyTorch-Models/Model/README.md

Install the IPEX-LLM library and initialize the environment variables for optimal performance on Linux systems. This is a prerequisite for running accelerated models.

```bash
pip install ipex-llm
source ipex-llm-init
```

--------------------------------

### Install Dependencies for AWQ Models (Windows)

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/CPU/HF-Transformers-AutoModels/Advanced-Quantizations/AWQ/README.md

Installs required packages for AWQ model inference on Windows, including autoawq, ipex-llm, transformers, accelerate, and einops. Conda is recommended for environment setup.

```cmd
conda create -n llm python=3.11
conda activate llm

pip install autoawq==0.1.8 --no-deps
pip install --pre --upgrade ipex-llm[all]
pip install transformers==4.35.0
pip install accelerate==0.25.0
pip install einops
```

--------------------------------

### RAGFlow Server Startup Log Example

Source: https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quickstart/ragflow_quickstart.md

This output indicates that the RAGFlow server has started successfully and is accessible.

```text
    ____                 ______ __
   / __ \ ____ _ ____ _ / ____// /____  _      __
  / /_/ // __ `// __ `// /_   / // __ \| | /| / /
 / _, _// /_/ // /_/ // __/  / // /_/ /| |/ |/ /
/_/ |_| \__,_/ \__, //_/    /_/ \____/ |__/|__/
              /____/

* Running on all addresses (0.0.0.0)
* Running on http://127.0.0.1:9380
* Running on http://x.x.x.x:9380
INFO:werkzeug:Press CTRL+C to quit
```

--------------------------------

### Initialize IPEX-LLM Environment and Start Ollama

Source: https://github.com/intel/ipex-llm/blob/main/docker/llm/inference-cpp/README.md

Set up the necessary environment variables for IPEX-LLM with GPU support and start the Ollama service in the background. Logs are available at /root/ollama/ollama.log.

```bash
cd /llm/scripts/
source ipex-llm-init --gpu --device $DEVICE
bash start-ollama.sh # ctrl+c to exit
```

--------------------------------

### Install IPEX-LLM and Transformers on Windows

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/GPU/HuggingFace/Multimodal/qwen2-audio/README.md

Install necessary libraries for running Qwen2-Audio with IPEX-LLM on Windows. Ensure you have the correct transformers version from GitHub.

```bash
conda create -n llm python=3.11 libuv
conda activate llm

# below command will install intel_extension_for_pytorch==2.1.10+xpu as default
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/

pip install librosa
pip install git+https://github.com/huggingface/transformers
```

--------------------------------

### Dify Server Startup Logs

Source: https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quickstart/dify_quickstart.md

Example log output indicating the Dify server has started successfully and is accessible.

```bash
INFO:werkzeug:
* Running on all addresses (0.0.0.0)
* Running on http://127.0.0.1:5001
* Running on http://10.239.44.83:5001
INFO:werkzeug:Press CTRL+C to quit
INFO:werkzeug: * Restarting with stat
WARNING:werkzeug: * Debugger is active!
INFO:werkzeug: * Debugger PIN: 227-697-894
```

--------------------------------

### Setup IPEX-LLM Environment

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/GPU/Applications/autogen/README.md

Installs necessary packages for IPEX-LLM, transformers, and chromadb within a new conda environment.

```bash
mkdir autogen
cd autogen

conda create -n autogen python=3.11
conda activate autogen

pip install --pre --upgrade ipex-llm[xpu,serving] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/

pip install transformers==4.36.2

pip install chromadb==0.4.22
```

--------------------------------

### Run Math Example

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/CPU/LangChain/README.md

Executes the math-focused chat pipeline example. Requires the path to the model and optionally a question.

```bash
python llm_math.py -m <path_to_model> [-q <your_question>]
```

--------------------------------

### Start QLoRA Fine-Tuning Script

Source: https://github.com/intel/ipex-llm/blob/main/docker/llm/finetune/qlora/cpu/docker/README.md

Execute this script after entering the Docker container to initiate the QLoRA fine-tuning process on CPU.

```bash
bash start-qlora-finetuning-on-cpu.sh
```

--------------------------------

### Environment Setup with Conda on Windows

Source: https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Overview/install_cpu.md

Create and activate a Conda environment with Python 3.11, then install IPEX-LLM for CPU on Windows.

```cmd
conda create -n llm python=3.11
conda activate llm

pip install --pre --upgrade ipex-llm[all]
```

--------------------------------

### Fine-Tuning Output Example

Source: https://github.com/intel/ipex-llm/blob/main/docker/llm/finetune/qlora/cpu/docker/README.md

Illustrates the expected output during the QLoRA fine-tuning process, showing loss, learning rate, and epoch progression.

```text
{'loss': 2.0251, 'learning_rate': 0.0002, 'epoch': 0.02}
{'loss': 1.2389, 'learning_rate': 0.00017777777777777779, 'epoch': 0.03}
{'loss': 1.032, 'learning_rate': 0.00015555555555555556, 'epoch': 0.05}
{'loss': 0.9141, 'learning_rate': 0.00013333333333333334, 'epoch': 0.06}
{'loss': 0.8505, 'learning_rate': 0.00011111111111111112, 'epoch': 0.08}
{'loss': 0.8713, 'learning_rate': 8.888888888888889e-05, 'epoch': 0.09}
{'loss': 0.8635, 'learning_rate': 6.666666666666667e-05, 'epoch': 0.11}
{'loss': 0.8853, 'learning_rate': 4.4444444444444447e-05, 'epoch': 0.12}
{'loss': 0.859, 'learning_rate': 2.2222222222222223e-05, 'epoch': 0.14}
{'loss': 0.8608, 'learning_rate': 0.0, 'epoch': 0.15}
{'train_runtime': xxxx, 'train_samples_per_second': xxxx, 'train_steps_per_second': xxxx, 'train_loss': 1.0400420665740966, 'epoch': 0.15}
100%|███████████████████████████████████████████████████████████████████████████████████| 200/200 [07:16<00:00,  2.18s/it]
TrainOutput(global_step=200, training_loss=1.0400420665740966, metrics={'train_runtime': xxxx, 'train_samples_per_second': xxxx, 'train_steps_per_second': xxxx, 'train_loss': 1.0400420665740966, 'epoch': 0.15})
```

--------------------------------

### Environment Setup with Conda on Linux

Source: https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Overview/install_cpu.md

Create and activate a Conda environment with Python 3.11, then install IPEX-LLM for CPU on Linux.

```bash
conda create -n llm python=3.11
conda activate llm

pip install --pre --upgrade ipex-llm[all] --extra-index-url https://download.pytorch.org/whl/cpu
```

--------------------------------

### Run AutoGen Example Script

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/CPU/Applications/autogen/README.md

Executes the teachability example script using AutoGen and the configured IPEX-LLM environment. This script demonstrates the agent's ability to learn and retain new information.

```bash
# activate conda environment
conda activate autogen

# go to the autogen folder
cd autogen

# run the autogen example
python teachability_new_knowledge.py
```

--------------------------------

### Setup Python Environment with Miniforge

Source: https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quickstart/bmg_quickstart.md

Downloads and installs Miniforge for managing Python environments. Activates the 'llm' environment with Python 3.11.

```bash
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
bash Miniforge3-Linux-x86_64.sh
source ~/.bashrc
conda create -n llm python=3.11
conda activate llm
```

--------------------------------

### Start FastChat Service within Docker

Source: https://github.com/intel/ipex-llm/blob/main/docs/mddocs/DockerGuides/vllm_docker_quickstart.md

Installs Gradio, starts the FastChat controller, configures IPEX-LLM and CCL environment variables, and launches the vLLM worker and Gradio web server. This assumes the model directory is correctly mapped inside the container.

```bash
#/bin/bash

# This command assumes that you have mapped the host model directory to the container
# and the model directory is /llm/models/
# we take Yi-1.5-34B as an example, and you can replace it with your own model

ps -ef | grep "fastchat" | awk '{print $2}' | xargs kill -9
pip install -U gradio==4.43.0

# start controller
python -m fastchat.serve.controller &

export USE_XETLA=OFF
export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=2

export TORCH_LLM_ALLREDUCE=0
export CCL_DG2_ALLREDUCE=1
# CCL needed environment variables
export CCL_WORKER_COUNT=4
#pin ccl worker to cores
# export CCL_WORKER_AFFINITY=32,33,34,35
export FI_PROVIDER=shm
export CCL_ATL_TRANSPORT=ofi
export CCL_ZE_IPC_EXCHANGE=sockets
export CCL_ATL_SHM=1

source /opt/intel/1ccl-wks/setvars.sh

python -m ipex_llm.serving.fastchat.vllm_worker \
--model-path /llm/models/Yi-1.5-34B \
--device xpu \
--enforce-eager \
--disable-async-output-proc \
--distributed-executor-backend ray \
--dtype float16 \
--load-in-low-bit fp8 \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.9 \
--max-model-len 4096 \
--max-num-batched-tokens 8000 &

sleep 120

python -m fastchat.serve.gradio_web_server &

```

--------------------------------

### Enter Docker Container and Start Fine-Tuning

Source: https://github.com/intel/ipex-llm/blob/main/docker/llm/README_backup.md

Access the running Docker container and execute the QLoRA fine-tuning script. The `use_gradient_checkpointing=True` option can be set if memory is insufficient.

```bash
docker exec -it ipex-llm-fintune-qlora-cpu bash

cd /ipex_llm
bash start-qlora-finetuning-on-cpu.sh
```

--------------------------------

### Launch QLoRA Finetuning

Source: https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quickstart/axolotl_quickstart.md

Launch the QLoRA finetuning process using Axolotl. This command uses the 'finetune.py' script.

```bash
accelerate launch finetune.py qlora.yml
```

--------------------------------

### Finetune ChatGLM3-6B on Two Arc A770 with DeepSpeed

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/GPU/LLM-Finetuning/LoRA/README.md

Installs DeepSpeed and its Intel extension, then starts finetuning ChatGLM3-6B on two Intel Arc A770 GPUs using DeepSpeed Zero3. Ensure OneAPI environment variables are sourced before installing DeepSpeed.

```bash
# install deepspeed dependencies
source /opt/intel/oneapi/setvars.sh # necessary to run before installing deepspeed
pip install git+https://github.com/microsoft/DeepSpeed.git@78c518e
pip install git+https://github.com/intel/intel-extension-for-deepspeed.git@ec33277

#start finetuning
bash lora_deepspeed_zero3_finetune_chatglm3_6b_arc_2_card.sh
```

--------------------------------

### Sample Output for Qwen/Qwen2-1.5B-Instruct (English Prompt)

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/GPU/HuggingFace/LLM/qwen2/README.md

Example of inference output for the Qwen/Qwen2-1.5B-Instruct model with an English prompt.

```log
Inference time: xxxx s
-------------------- Prompt --------------------
What is AI?
-------------------- Output --------------------
Artificial Intelligence (AI) refers to the simulation of human intelligence in machines that are programmed to think and work like humans. It involves creating computer programs, algorithms
```

--------------------------------

### Start Multi-Process Fine-Tuning with Docker

Source: https://github.com/intel/ipex-llm/blob/main/docker/llm/finetune/qlora/cpu/docker/README.md

Use this command to launch a Docker container for multi-process QLoRA fine-tuning on CPU. Ensure to replace placeholder values for CPU ranges, worker counts, and model/data paths.

```bash
docker run -itd \
 --name=ipex-llm-fintune-qlora-cpu \
 --cpuset-cpus="your_expected_range_of_cpu_numbers" \
 -e STANDALONE_DOCKER=TRUE \
 -e WORKER_COUNT_DOCKER=your_worker_count \
 -v your_downloaded_base_model_path:/ipex_llm/model \
 -v your_downloaded_data_path:/ipex_llm/data/alpaca_data_cleaned_archive.json \
 intelanalytics/ipex-llm-finetune-qlora-cpu-standalone:2.2.0-SNAPSHOT
```

--------------------------------

### Setup FastChat and AutoGen Environment

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/CPU/Applications/autogen/README.md

Clones the FastChat repository and installs it along with the pyautogen library. This step is crucial for setting up the communication layer for the agents.

```bash
# clone the FastChat in the autogen folder
git clone https://github.com/lm-sys/FastChat.git FastChat # clone the FastChat
cd FastChat
pip3 install --upgrade pip  # enable PEP 660 support
pip3 install -e ".[model_worker,webui]"

# setup AutoGen environment
pip install pyautogen==0.2.7
```

--------------------------------

### Setup IPEX-LLM Environment

Source: https://github.com/intel/ipex-llm/blob/main/python/llm/example/CPU/Applications/autogen/README.md

Installs necessary packages for running AutoGen with IPEX-LLM, including FastChat, transformers, and chromadb. Ensure you are in a conda environment with Python 3.11.

```bash
mkdir autogen
cd autogen
conda create -n autogen python=3.11
conda activate autogen
pip install --pre --upgrade ipex-llm[serving]
pip install transformers==4.36.2
pip install chromadb==0.4.22
```

--------------------------------

### Configure User Permissions and Verify GPU Setup

Source: https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quickstart/install_linux_gpu.md

Ensure the current user has the necessary permissions to access the GPU by adding them to the 'render' group and then verifying the GPU driver setup using the `clinfo` command.

```bash
sudo gpasswd -a ${USER} render
newgrp render
```

```bash
clinfo | grep "Device Name"
```

--------------------------------

### Initialize llama.cpp with IPEX-LLM (Windows)

Source: https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quickstart/llama_cpp_quickstart.md

Execute this command with administrator privileges in a Miniforge Prompt on Windows to initialize llama.cpp with IPEX-LLM.

```cmd
init-llama-cpp.bat
```