### Setup Development Environment

Source: https://github.com/espere-1119-song/videonsa/blob/main/lmms-eval/CLAUDE.md

Commands to initialize the project environment, including activating the conda environment and installing dependencies via uv.

```bash
# IMPORTANT: Always activate msswift environment first
conda activate msswift

# For development
git clone https://github.com/EvolvingLMMs-Lab/lmms-eval
cd lmms-eval
uv venv dev
source dev/bin/activate
uv pip install -e .
```

--------------------------------

### Install SWIFT from Source Code

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source/GetStarted/SWIFT安装.md

Clone the repository and install SWIFT using pip in editable mode. The '[all]' option installs all capabilities.

```shell
git clone https://github.com/modelscope/ms-swift.git
cd ms-swift
pip install -e .
```

```shell
pip install -e '.[all]'
```

--------------------------------

### Install SWIFT from source

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/GetStarted/SWIFT-installation.md

Clone the repository and perform an editable installation for development purposes.

```shell
# pip install git+https://github.com/modelscope/ms-swift.git

# Full capabilities
# pip install "git+https://github.com/modelscope/ms-swift.git#egg=ms-swift[all]"

git clone https://github.com/modelscope/ms-swift.git
cd ms-swift
pip install -e .

# Full capabilities
# pip install -e '.[all]'
```

--------------------------------

### Install ms-swift from GitHub

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/README.md

Install the ms-swift library using pip by cloning the GitHub repository and performing a local installation.

```shell
pip install git+https://github.com/modelscope/ms-swift.git
git clone https://github.com/modelscope/ms-swift.git
cd ms-swift
pip install -e .
```

--------------------------------

### Install ms-swift using pip

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/README.md

Install the ms-swift package using pip for quick setup. Ensure you use the -U flag for the latest version.

```shell
pip install ms-swift -U
```

--------------------------------

### Agent Training Examples

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/Instruction/Pre-training-and-Fine-tuning.md

Examples for training agent models using SWIFT.

```shell
bash
# Example script for agent training
# Refer to https://github.com/modelscope/ms-swift/blob/main/examples/train/agent for details.
```

--------------------------------

### Install ms-swift

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/examples/notebook/qwen2vl-ocr/ocr-sft.ipynb

Command to install the ms-swift library.

```python
# # install ms-swift
# pip install ms-swift -U
```

--------------------------------

### Install VideoNSA Repository and Dependencies

Source: https://github.com/espere-1119-song/videonsa/blob/main/README.md

Clone the VideoNSA repository and install its core dependencies, including ms-swift and flash-linear-attention. Ensure you are in the correct directories for installation.

```bash
git clone https://github.com/Espere-1119-Song/VideoNSA.git
cd VideoNSA

cd ms-swift
pip install -e .

pip uninstall fla-core flash-linear-attention -y
pip install -U git+https://github.com/fla-org/flash-linear-attention

cd ..
```

--------------------------------

### Install VideoNSA Training Dependencies

Source: https://context7.com/espere-1119-song/videonsa/llms.txt

Clones the repository and installs the required ms-swift and flash-linear-attention packages.

```bash
# Clone the repository
git clone https://github.com/Espere-1119-Song/VideoNSA.git
cd VideoNSA

# Install ms-swift
cd ms-swift
pip install -e .

# Install flash-linear-attention
pip uninstall fla-core flash-linear-attention -y
pip install -U git+https://github.com/fla-org/flash-linear-attention

cd ..
```

--------------------------------

### Install SWIFT using pip

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source/GetStarted/SWIFT安装.md

Use pip to install the ms-swift package. Install with evaluation capabilities using '[eval]' or all capabilities using '[all]'.

```shell
pip install 'ms-swift'
```

```shell
pip install 'ms-swift[eval]' -U
```

```shell
pip install 'ms-swift[all]' -U
```

--------------------------------

### Multi-node Multi-GPU Training Launch Scripts

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/Instruction/Pre-training-and-Fine-tuning.md

Example shell scripts for launching multi-node training runs using various frameworks like swift, torchrun, dlc, deepspeed, and accelerate. Scripts need to be started on all nodes except dlc and deepspeed.

```shell
bash
# Example scripts for multi-node multi-GPU training
# Refer to https://github.com/modelscope/ms-swift/blob/main/examples/train/multi-node for details.
```

--------------------------------

### Setup NPU Environment

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/BestPractices/NPU-support.md

Commands to create a conda environment and install the required packages for NPU support.

```shell
# Create a new conda virtual environment (optional)
conda create -n swift-npu python=3.10 -y
conda activate swift-npu

# Set pip global mirror (optional, to speed up downloads)
pip config set global.index-url https://mirrors.aliyun.com/pypi/simple/
pip install ms-swift -U

# Install torch-npu
pip install torch-npu decorator
```

--------------------------------

### Any-to-Any Model Training Examples

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/Instruction/Pre-training-and-Fine-tuning.md

Examples for training any-to-any models using SWIFT.

```shell
bash
# Example script for any-to-any model training
# Refer to https://github.com/modelscope/ms-swift/blob/main/examples/train/all_to_all for details.
```

--------------------------------

### Multi-modal Model Packing Example

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/Instruction/Pre-training-and-Fine-tuning.md

Example demonstrating multimodal model packing to increase training speed.

```shell
bash
# Example script for multimodal model packing
# Refer to https://github.com/modelscope/ms-swift/tree/main/examples/train/packing for details.
```

--------------------------------

### Install SWIFT environment

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/Instruction/Sample.md

Provides commands to install the SWIFT package via pip or from the source repository.

```shell
pip install ms-swift[llm] -U
```

```shell
git clone https://github.com/modelscope/ms-swift.git
cd ms-swift
pip install -e '.[llm]'
```

--------------------------------

### Lightweight Fine-tuning Examples

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/Instruction/Pre-training-and-Fine-tuning.md

Examples of lightweight fine-tuning supported by SWIFT. These methods can also be used for pre-training.

```shell
bash
# Example script for lightweight fine-tuning
# Refer to https://github.com/modelscope/ms-swift/blob/main/examples/train/tuners for details.
```

--------------------------------

### Install ms-swift from source

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/README.md

Install ms-swift directly from its source code. This method is useful for development or when needing the absolute latest changes.

```shell
git clone https://github.com/modelscope/ms-swift.git
cd ms-swift
pip install -e .
```

--------------------------------

### Install Required Libraries

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source/BestPractices/Qwen3最佳实践.md

Install necessary Python packages for training, including ms-swift, transformers, deepspeed, liger-kernel, and flash-attn.

```bash
pip install ms-swift -U
pip install transformers

pip install deepspeed # 多GPU训练
pip install liger-kernel # 节约显存资源
pip install flash-attn --no-build-isolation  # packing需要
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/espere-1119-song/videonsa/blob/main/lmms-eval/lmms_eval/tasks/plm_videobench/README.md

Commands to install required packages including ffmpeg, torchcodec, and the local project environment.

```bash
conda install ffmpeg -c conda-forge
pip install torchcodec==0.1 --index-url=https://download.pytorch.org/whl/cu124

pip install -e .
```

--------------------------------

### Install Perception Models Environment

Source: https://github.com/espere-1119-song/videonsa/blob/main/lmms-eval/lmms_eval/tasks/plm_videobench/README.md

Sets up a Python 3.12 environment for perception models and installs PyTorch with CUDA support.

```shell
git clone https://github.com/facebookresearch/perception_models.git
cd perception_models

conda create --name perception-env python=3.12
conda activate perception-env

pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 xformers --index-url https://download.pytorch.org/whl/cu124
```

--------------------------------

### Install ms-swift from source

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source/Instruction/采样.md

Install ms-swift from its source code repository. This method is useful for development or when needing the latest unreleased features.

```shell
git clone https://github.com/modelscope/ms-swift.git
cd ms-swift
pip install -e '.[llm]'
```

--------------------------------

### Install pdsh from source

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/examples/train/multi-node/deepspeed/README.md

Compiles and installs pdsh with SSH support. Requires root privileges and build tools.

```shell
# https://code.google.com/archive/p/pdsh/downloads
# For example, download to /root:
cd /root
wget https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/pdsh/pdsh-2.29.tar.bz2
tar -xvf pdsh-2.29.tar.bz2
cd pdsh-2.29
./configure --prefix=/root/pdsh-2.29 --with-ssh --without-rsh --with-exec --with-timeout=60 --with-nodeupdown --with-rcmd-rank-list=ssh
make
make install
```

--------------------------------

### Install Datasets and Pillow Libraries

Source: https://github.com/espere-1119-song/videonsa/blob/main/lmms-eval/tools/make_image_hf_dataset.ipynb

Installs the necessary libraries for creating and handling datasets, including image processing capabilities.

```python
!pip install datasets Pillow
```

--------------------------------

### Install ms-swift with LLM support

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source/Instruction/采样.md

Install the ms-swift library with LLM capabilities using pip. This is a prerequisite for using SWIFT's sampling features.

```shell
pip install ms-swift[llm] -U
```

--------------------------------

### Import Libraries and Initialize Environment

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/examples/notebook/qwen2vl-ocr/ocr-sft.ipynb

Setup environment variables and import necessary modules from ms-swift.

```python
# import some libraries
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0'

from swift.llm import (
    get_model_tokenizer, load_dataset, get_template, EncodePreprocessor, get_model_arch,
    get_multimodal_target_regex, LazyLLMDataset
)
from swift.utils import get_logger, get_model_parameter_info, plot_images, seed_everything
from swift.tuners import Swift, LoraConfig
from swift.trainers import Seq2SeqTrainer, Seq2SeqTrainingArguments
from functools import partial

logger = get_logger()
seed_everything(42)
```

--------------------------------

### Build documentation

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/README.md

Execute this command from the project root directory to generate documentation.

```shell
# in root directory:
make docs
```

--------------------------------

### Training Script Example for Qwen2.5 Agent

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/Instruction/Agent支持.md

This is a command-line example for training an agent with the Qwen2.5 model. It shows how to specify the model and points to a shell script for detailed configuration.

```bash
--model
```

--------------------------------

### Run Official Magic Mirror Image with Docker

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/Instruction/Frequently-asked-questions.md

Use this command to start the official Magic Mirror Docker container. Ensure you have Docker and NVIDIA Container Toolkit installed. This command maps port 8000 and allocates all available GPUs.

```bash
docker run --gpus all -p 8000:8000 -it -d --name ms modelscope-registry.cn-beijing.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.4.0-py311-torch2.6.0-1.26.0-LLM /bin/bash
```

--------------------------------

### Start SWIFT Web UI

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/GetStarted/Web-UI.md

Use this command to launch the web interface for training and inference. Specify the language with `--lang`.

```shell
swift web-ui --lang zh
```

```shell
# or en
swift web-ui --lang en
```

--------------------------------

### Initialize Environment and Imports

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/examples/notebook/qwen2vl-ocr/infer.ipynb

Set up the CUDA environment and import necessary modules from the SWIFT library.

```python
# import some libraries
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0'

from swift.llm import (
    InferEngine, InferRequest, PtEngine, RequestConfig, get_template, load_dataset, load_image
)
from swift.utils import get_model_parameter_info, get_logger, seed_everything
logger = get_logger()
seed_everything(42)
```

--------------------------------

### Install Apex

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/Instruction/Megatron-SWIFT-Training.md

Clone the Apex repository, checkout a specific commit, and install it with C++ and CUDA extensions enabled. This process requires specific versioning as indicated by the issue link.

```shell
git clone https://github.com/NVIDIA/apex
cd apex
# https://github.com/modelscope/ms-swift/issues/4176
git checkout e13873debc4699d39c6861074b9a3b2a02327f92
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" ./
```

--------------------------------

### Install lmms-eval from Git

Source: https://github.com/espere-1119-song/videonsa/blob/main/lmms-eval/README.md

Use this command to install the package directly from its Git repository. It sets up a virtual environment and installs the package.

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv eval
uv venv --python 3.12
source eval/bin/activate
uv pip install git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git
```

--------------------------------

### Launch TensorBoard

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/examples/notebook/qwen2_5-self-cognition/self-cognition-sft.ipynb

Start the TensorBoard server to visualize logs from the specified output directory.

```bash
tensorboard --logdir '{output_dir}/runs'
```

--------------------------------

### Install Megatron-LM Core

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/Instruction/Megatron-SWIFT-Training.md

Install the core Megatron-LM library from its GitHub repository using pip. This command installs the specified version of the core library.

```shell
pip install git+https://github.com/NVIDIA/Megatron-LM.git@core_r0.13.0
```

--------------------------------

### Initialize environment and imports

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/examples/notebook/qwen2_5-self-cognition/self-cognition-sft.ipynb

Set up CUDA environment variables and import necessary modules from ms-swift.

```python
# import some libraries
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0'

from swift.llm import get_model_tokenizer, load_dataset, get_template, EncodePreprocessor
from swift.utils import get_logger, find_all_linears, get_model_parameter_info, plot_images, seed_everything
from swift.tuners import Swift, LoraConfig
from swift.trainers import Seq2SeqTrainer, Seq2SeqTrainingArguments
from functools import partial

logger = get_logger()
seed_everything(42)
```

--------------------------------

### RLHF Training Examples

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/Instruction/Pre-training-and-Fine-tuning.md

Examples for Reinforcement Learning from Human Feedback (RLHF) training. Includes specific examples for multi-modal models and GRPO training.

```shell
bash
# Example scripts for RLHF training
# Refer to https://github.com/modelscope/ms-swift/tree/main/examples/train/rlhf for details.
```

```shell
bash
# Example scripts for multi-modal RLHF training
# Refer to https://github.com/modelscope/ms-swift/tree/main/examples/train/multimodal/rlhf for details.
```

```shell
bash
# Example scripts for GRPO training
# Refer to https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/internal for details.
```

```shell
bash
# Example scripts for reinforcement fine-tuning (RFT)
# Refer to https://github.com/modelscope/ms-swift/tree/main/examples/train/rft for details.
```

--------------------------------

### Implement Image Preprocessing

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/Instruction/GRPO/AdvancedResearch/deepeyes.md

Example showing how to access raw images before preprocessing in the Qwen2.5-VL series.

```python
from qwen_vl_utils import fetch_image
# At this point, images have not yet been preprocessed
infer_request.images
```

--------------------------------

### Install lmms-eval

Source: https://github.com/espere-1119-song/videonsa/blob/main/lmms-eval/lmms_eval/tasks/plm_videobench/README.md

Command to install the lmms-eval evaluation framework.

```bash
pip install lmms-eval
```

--------------------------------

### Inference Output Example

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/Instruction/Megatron-SWIFT训练.md

Example output from the model inference.

```default
<<< who are you?
I am a language model developed by swift, you can call me swift-robot. How can I assist you?
```

--------------------------------

### Clone Repository and Install Dependencies

Source: https://github.com/espere-1119-song/videonsa/blob/main/lmms-eval/lmms_eval/tasks/mme_cot/README.md

Clone the MME-CoT repository and install necessary Python packages using pip. Ensure you are in the cloned directory before running the install command.

```bash
git clone https://github.com/CaraJ7/MME-CoT.git
cd MME-CoT
pip install -r requirements.txt
```

--------------------------------

### Run Web-UI

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/README.md

Launch the Web-UI for training and deployment with Gradio. Set the language for the UI using `SWIFT_UI_LANG`.

```shell
SWIFT_UI_LANG=en swift web-ui
```

--------------------------------

### Install lmms-eval and Dependencies for SliME

Source: https://github.com/espere-1119-song/videonsa/blob/main/lmms-eval/docs/run_examples.md

Installs lmms-eval, clones the SliME repository, and installs SliME in editable mode. This prepares the environment for running SliME with specified model arguments.

```bash
cd /path/to/lmms-eval
python3 -m pip install -e .;

git clone https://github.com/yfzhang114/SliME.git

cd SliME
pip install --upgrade pip  # enable PEP 660 support
pip install -e .
cd ..

TASK=$1
echo $TASK
TASK_SUFFIX="${TASK//,/_}"
echo $TASK_SUFFIX

accelerate launch --num_processes 8 --main_process_port 12345 -m lmms_eval \
    --model slime \
    --tasks $TASK \
    --model_args pretrained="yifanzhang114/SliME-Llama3-8B,conv_template=llama3,model_name=slime" \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $TASK_SUFFIX \
    --output_path ./logs/ 

```

--------------------------------

### Initialize Development Environment

Source: https://github.com/espere-1119-song/videonsa/blob/main/lmms-eval/docs/model_guide.md

Commands to clone the repository, create a feature branch, and install the project in editable mode.

```sh
# After forking...
git clone https://github.com/<YOUR-USERNAME>/lmms-eval.git
cd lmms-eval
git checkout -b <model-type>
pip install -e .
```

--------------------------------

### Install PyTorch and Transformer Engine

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/Instruction/Megatron-SWIFT-Training.md

Install PyTorch and the transformer_engine library. Ensure you use a compatible PyTorch version. If installation errors occur for transformer_engine, refer to the provided GitHub issue.

```shell
pip install pybind11

# transformer_engine
# If an installation error occurs, you can refer to this issue for resolution: https://github.com/modelscope/ms-swift/issues/3793
pip install --no-build-isolation transformer_engine[pytorch]
# Or install using the following command
# pip install --no-build-isolation git+https://github.com/NVIDIA/TransformerEngine.git@release_v2.5#egg=transformer_engine[pytorch]
```

--------------------------------

### Install RLHF dependencies

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/BestPractices/Qwen3-Best-Practice.md

Installs required packages for reinforcement learning tasks.

```shell
pip install "math_verify==0.5.2"
pip install vllm==0.8.5.post1
```

--------------------------------

### Execute Training

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/examples/notebook/qwen2vl-ocr/ocr-sft.ipynb

Initialize the trainer and start the training process.

```python
# Get the trainer and start the training.
model.enable_input_require_grads()  # Compatible with gradient checkpointing
trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    data_collator=template.data_collator,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    template=template,
)
trainer.train()

last_model_checkpoint = trainer.state.last_model_checkpoint
logger.info(f'last_model_checkpoint: {last_model_checkpoint}')
```

--------------------------------

### Simulate SWIFT Command Line Execution

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/GetStarted/Web-UI.md

This pseudo-code demonstrates how the web UI initiates tasks via the command line, similar to running SWIFT commands directly.

```python
import os
os.system('swift sft --model xxx --dataset xxx')
```

--------------------------------

### QLoRA Quantization Training Examples

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/Instruction/Pre-training-and-Fine-tuning.md

Examples for QLoRA training using quantization techniques like GPTQ, AWQ, AQLM, BNB, HQQ, and EETQ. Enables fine-tuning large models with minimal memory.

```shell
bash
# Example scripts for QLoRA quantization training
# Refer to https://github.com/modelscope/ms-swift/tree/main/examples/train/qlora for details.
```

--------------------------------

### Embedding Model Training Examples

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/Instruction/Pre-training-and-Fine-tuning.md

Examples for training embedding models using SWIFT.

```shell
bash
# Example scripts for embedding model training
# Refer to https://github.com/modelscope/ms-swift/tree/main/examples/train/embedding for details.
```

--------------------------------

### Install SGLang for SRT API

Source: https://github.com/espere-1119-song/videonsa/blob/main/lmms-eval/docs/commands.md

Commands to clone and install the SGLang repository and required dependencies.

```bash
git clone https://github.com/sgl-project/sglang.git
cd sglang;
pip install -e "python[srt]"
python3 -m pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.3/
```

--------------------------------

### Run InstructBLIP Evaluation

Source: https://github.com/espere-1119-song/videonsa/blob/main/lmms-eval/docs/run_examples.md

Configures the environment and launches the InstructBLIP model evaluation using accelerate.

```bash
cd /path/to/lmms-eval
python3 -m pip install -e .;

python3 -m pip install transformers --upgrade;

CKPT_PATH=$1
TASK=$2
echo $TASK
TASK_SUFFIX="${TASK//,/_}"
echo $TASK_SUFFIX

accelerate launch --num_processes 8 --main_process_port 12345 -m lmms_eval \
    --model instructblip \
    --model_args pretrained=$CKPT_PATH \
    --tasks $TASK \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix instructblip \
    --output_path ./logs/ 
```

--------------------------------

### Install SWIFT and DeepSpeed

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/Instruction/Pre-training-and-Fine-tuning.md

Install the core SWIFT library and optional DeepSpeed dependencies for distributed training.

```shell
pip install ms-swift -U

# If using deepspeed zero2/zero3
pip install deepspeed -U
```

--------------------------------

### Install SWIFT from Source with Eval Support

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/Instruction/Evaluation.md

Clone the ms-swift repository and install it from source with evaluation dependencies. This is useful for development or when needing the latest unreleased features.

```shell
git clone https://github.com/modelscope/ms-swift.git
cd ms-swift
pip install -e '.[eval]'
```

--------------------------------

### Sequence Classification Model Training Examples

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/Instruction/Pre-training-and-Fine-tuning.md

Examples for training sequence classification models using SWIFT.

```shell
bash
# Example scripts for sequence classification model training
# Refer to https://github.com/modelscope/ms-swift/tree/main/examples/train/seq_cls for details.
```

--------------------------------

### Install ms-swift and Dependencies

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/BestPractices/Qwen3-Best-Practice.md

These commands install the ms-swift library and its essential dependencies, including transformers, deepspeed for multi-GPU training, liger-kernel for memory optimization, and flash-attn for efficient attention mechanisms.

```bash
pip install ms-swift -U
pip install transformers

pip install deepspeed # for multi-GPU training
pip install liger-kernel # to save GPU memory resources
pip install flash-attn --no-build-isolation  # required for packing
```

--------------------------------

### Install Evalscope Dependencies and Patch

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/Instruction/Frequently-asked-questions.md

Commands to install evalscope and apply a necessary patch for Python 3.11 environments.

```bash
pip3 install evalscope[all]
```

```bash
pip3 install https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/package/evalscope-0.5.3.post1-py3-none-any.whl
```

--------------------------------

### Prepare Model and LoRA Configuration

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/examples/notebook/qwen2vl-ocr/ocr-sft.ipynb

Initialize the model, template, and apply LoRA adapters.

```python
# Obtain the model and template
model, processor = get_model_tokenizer(model_id_or_path)
logger.info(f'model_info: {model.model_info}')
template = get_template(model.model_meta.template, processor, default_system=system, max_length=max_length)
template.set_mode('train')
if template.use_model:
    template.model = model

# Get target_modules and add trainable LoRA modules to the model.
target_modules = get_multimodal_target_regex(model, freeze_llm=freeze_llm, freeze_vit=freeze_vit, 
                            freeze_aligner=freeze_aligner)
lora_config = LoraConfig(task_type='CAUSAL_LM', r=lora_rank, lora_alpha=lora_alpha,
                         target_modules=target_modules)
model = Swift.prepare_model(model, lora_config)
logger.info(f'lora_config: {lora_config}')

# Print model structure and trainable parameters.
logger.info(f'model: {model}')
model_parameter_info = get_model_parameter_info(model)
logger.info(f'model_parameter_info: {model_parameter_info}')
```

--------------------------------

### Install Evaluation Dependencies

Source: https://github.com/espere-1119-song/videonsa/blob/main/README.md

Install the necessary packages for evaluation, including the UV package manager, lmms-eval, flash-attn, qwen_vl_utils, and accelerate. Remember to log in to Hugging Face.

```bash
pip install uv

uv pip install git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git

pip install flash-attn --no-build-isolation
pip install qwen_vl_utils
pip install accelerate

huggingface-cli login
```

--------------------------------

### Install VideoNSA Evaluation Dependencies

Source: https://context7.com/espere-1119-song/videonsa/llms.txt

Installs the UV package manager, lmms-eval, and necessary runtime dependencies for evaluation.

```bash
# Install UV package manager
pip install uv

# Install lmms-eval
uv pip install git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git

# Install additional dependencies
pip install flash-attn --no-build-isolation
pip install qwen_vl_utils
pip install accelerate

# Login to Hugging Face
huggingface-cli login
```

--------------------------------

### Agent Data Format Examples

Source: https://github.com/espere-1119-song/videonsa/blob/main/ms-swift/docs/source_en/Customization/Custom-dataset.md

JSONL samples demonstrating tool definitions, message sequences, and multimodal input handling for agent training.

```jsonl
{"tools": "[{\"type\": \"function\", \"function\": {\"name\": \"realtime_aqi\", \"description\": \"Weather forecast. Get real-time air quality, including current air quality, PM2.5, and PM10 information.\", \"parameters\": {\"type\": \"object\", \"properties\": {\"city\": {\"type\": \"string\", \"description\": \"City name, e.g., Shanghai\"}}, \"required\": [\"city\"]}}}", "messages": [{"role": "user", "content": "What is the weather like in Beijing and Shanghai today?"}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Beijing\"}}"}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Shanghai\"}}"}, {"role": "tool_response", "content": "{\"city\": \"Beijing\", \"aqi\": \"10\", \"unit\": \"celsius\"}"}, {"role": "tool_response", "content": "{\"city\": \"Shanghai\", \"aqi\": \"72\", \"unit\": \"fahrenheit\"}"}, {"role": "assistant", "content": "According to the weather forecast tool, the air quality index (AQI) in Beijing is 10, which indicates good air quality; whereas in Shanghai, the AQI is 72, indicating mild pollution."}]}
{"tools": "[{\"type\": \"function\", \"function\": {\"name\": \"click\", \"description\": \"Click on a position on the screen\", \"parameters\": {\"type\": \"object\", \"properties\": {\"x\": {\"type\": \"integer\", \"description\": \"X-coordinate representing the horizontal position on the screen\"}, \"y\": {\"type\": \"integer\", \"description\": \"Y-coordinate representing the vertical position on the screen\"}}, \"required\": [\"x\", \"y\"]}}}", "messages": [{"role": "user", "content": "<image>What time is it now?"}, {"role": "assistant", "content": "<think>\nI can check the current time by opening the calendar app.\n</think>\n"}, {"role": "tool_call", "content": "{\"name\": \"click\", \"arguments\": {\"x\": 105, \"y\": 132}}"}, {"role": "tool_response", "content": "{\"images\": \"<image>\", \"status\": \"success\"}"}, {"role": "assistant", "content": "Successfully opened the calendar app. The current time is 11 o'clock in the morning."}], "images": ["desktop.png", "calendar.png"]}
```