### Library Installation Output

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_models_transformer_4_35_2.ipynb

Shows the output of the pip installation command, detailing the download and setup process for the specified libraries.

```text
Collecting tiktoken\n  Downloading tiktoken-0.5.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.0 MB)\n\u0010\u0010 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 10.4 MB/s eta \u0010\u0010 0:00:00\u0010\u0010\n\u0010\u0010[?25hCollecting einops\n  Downloading einops-0.7.0-py3-none-any.whl (44 kB)\n\u0010\u0010 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.6/44.6 kB 6.0 MB/s eta \u0010\u0010 0:00:00\u0010\u0010\n\u0010\u0010Collecting transformers_stream_generator\n  Downloading transformers-stream-generator-0.0.4.tar.gz (12 kB)\n  Preparing metadata (setup.py) ... \u0010\u0010[?25l\u0010\u0010[?25hdone
```

--------------------------------

### Install airllm, Transformers, and Torch

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_models_transformer_4_36_2_torch_2_1_2.ipynb

Use this command to install or upgrade airllm, transformers, and torch. Ensure you have pip installed.

```python
!pip install -U airllm transformers torch
```

--------------------------------

### Install and Show AirLLM Package

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_compression.ipynb

Installs the airllm package and then displays its version and metadata. Ensure you have the necessary permissions to install packages.

```python
!pip show airllm
```

--------------------------------

### Basic Text Output Example

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/examples/run_all_types_of_models.ipynb

A minimal example showing a string output. This might be used for simple demonstrations or placeholder content.

```python
Result: 
' <s>I like to think of'
```

--------------------------------

### Install Dependencies and Run DPO Training

Source: https://github.com/lyogavin/airllm/blob/main/rlhf/README.md

This script installs necessary dependencies and then initiates the DPO training process. Ensure you have prepared your dataset and reference model beforehand. Key parameters include dataset path, reference model path, and beta for the DPO loss.

```bash
# 1. install dependencies
pip install -r requirements.txt
# 2. run DPO training
cd rlhf
./run_dpo_training.sh
```

--------------------------------

### Install AirLLM and Transformers

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_mixtral.ipynb

Use this command to install the AirLLM library and the Transformers package, which are essential for running LLM models.

```bash
!pip install -U airllm transformers
```

--------------------------------

### Dependency Installation Output

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/examples/run_all_types_of_models.ipynb

Shows the output from the pip installation command, detailing the download and installation process of specified packages.

```shell
Collecting tiktoken
  Downloading tiktoken-0.5.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 10.4 MB/s eta 0:00:00
[?25hCollecting einops
  Downloading einops-0.7.0-py3-none-any.whl (44 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.6/44.6 kB 6.0 MB/s eta 0:00:00
[?25hCollecting transformers_stream_generator
  Downloading transformers-stream-generator-0.0.4.tar.gz (12 kB)
  Preparing metadata (setup.py) ... [?25l[?25hdone
```

--------------------------------

### Model Comparison Setup with System Prompt

Source: https://github.com/lyogavin/airllm/blob/main/eval/elo_tournanment_all_models_on_translated_vicuna.ipynb

Shows the system prompt and question setup for comparing two AI assistants (Anima and CN Vicuna) on a Chinese language question about detecting signs of pretended understanding.

```python
competing: (31, 'anima', 'cn_vicuna')
PROMPT: 
sys_prompt:
You are a helpful and precise assistant for checking the quality of the answer.
prompt:
[Question]
有哪些不易察觉的线索，暗示某人在假装理解一个话题或对话，而实际上他们却很困惑或无知？

[The Start of Assistant 1's Answer]
```

--------------------------------

### Install Dependencies and Train Anima Model

Source: https://github.com/lyogavin/airllm/blob/main/training/README_en.md

Installs project dependencies and then executes the training script for the Anima model. Ensure you are in the 'training' directory before running the script.

```bash
pip install -r requirements.txt
cd training
./run_Amina_training.sh
```

--------------------------------

### bitsandbytes Installation Confirmation

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_compression.ipynb

Confirms that the bitsandbytes library is installed, which is necessary for using 4-bit compression.

```text
>>>> bitsandbytes installed
```

--------------------------------

### Install bitsandbytes Package

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_compression.ipynb

Installs the bitsandbytes library, which is often required for efficient model loading and quantization in deep learning tasks.

```python
!pip install bitsandbytes
```

--------------------------------

### Dijkstra's Algorithm using NetworkX in Python

Source: https://github.com/lyogavin/airllm/blob/main/training/README.md

This example demonstrates how to use the NetworkX library in Python to find the shortest path between two nodes using Dijkstra's algorithm. It requires the NetworkX library to be installed.

```python
import networkx as nx
G = nx.DiGraph() # DiGraph表示有向图
source = 'A'
target = 'F'
weight = {('A', 'B'): 4, ('A', 'C'): 7, ('B', 'D'): 2, ('C', 'D'): 6, ('D', 'E'): 10, ('E', 'F'): 8}
for node in G.nodes():
    G[node][source] = float("inf")
G.add_edge(*tuple(weight))
path = nx.dijkstra_path(G, source=source, target=target)
print path
```

--------------------------------

### Load and Run QWen Model

Source: https://github.com/lyogavin/airllm/blob/main/README.md

Shows how to load the QWen-7B model, prepare input tokens, and generate output. This example assumes CUDA is available.

```python
from airllm import AutoModel
MAX_LENGTH = 128
model = AutoModel.from_pretrained("Qwen/Qwen-7B")
input_text = ['What is the capital of China?',]
input_tokens = model.tokenizer(input_text,
    return_tensors="pt", 
    return_attention_mask=False, 
    truncation=True, 
    max_length=MAX_LENGTH)
generation_output = model.generate(
    input_tokens['input_ids'].cuda(), 
    max_new_tokens=5,
    use_cache=True,
    return_dict_in_generate=True)
model.tokenizer.decode(generation_output.sequences[0])
```

--------------------------------

### Install AirLLM

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_compression.ipynb

Installs the AirLLM library using pip. This is a prerequisite for running the tests.

```python
!pip install -U airllm
```

--------------------------------

### Install Dependencies and Fine-tune Train Based on Anima

Source: https://github.com/lyogavin/airllm/blob/main/training/README_en.md

Installs project dependencies and then executes the fine-tuning script based on the Anima model. Remember to modify the --dataset and --dataset_format arguments to point to your specific dataset.

```bash
pip install -r requirements.txt
cd training
./run_finetune_raining_based_on_Anima.sh
```

--------------------------------

### Example Text Output

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_models_transformer_4_35_2.ipynb

This snippet shows an example of text output, potentially from a model's generation process.

```text
Result:
' <s>I like to think of'
```

--------------------------------

### Install Dependencies

Source: https://github.com/lyogavin/airllm/blob/main/training/README.md

Install the required dependencies for the Anima model using pip. Ensure you have a stable internet connection.

```bash
pip install -r https://github.com/lyogavin/Anima/blob/main/requirements.txt?raw=true
```

--------------------------------

### Load and Run Baichuan, InternLM, or Mistral Models

Source: https://github.com/lyogavin/airllm/blob/main/README.md

Provides an example for loading models like Baichuan2, InternLM, or Mistral. Uncomment the desired model to use it. Requires CUDA for generation.

```python
from airllm import AutoModel
MAX_LENGTH = 128
model = AutoModel.from_pretrained("baichuan-inc/Baichuan2-7B-Base")
#model = AutoModel.from_pretrained("internlm/internlm-20b")
#model = AutoModel.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
input_text = ['What is the capital of China?',]
input_tokens = model.tokenizer(input_text,
    return_tensors="pt", 
    return_attention_mask=False, 
    truncation=True, 
    max_length=MAX_LENGTH)
generation_output = model.generate(
    input_tokens['input_ids'].cuda(), 
    max_new_tokens=5,
    use_cache=True,
    return_dict_in_generate=True)
model.tokenizer.decode(generation_output.sequences[0])
```

--------------------------------

### Python Quick Sort Implementation

Source: https://github.com/lyogavin/airllm/blob/main/training/README_en.md

This example provides a Python implementation of the Quick Sort algorithm. It includes a brief explanation of the code's logic and parameters. Useful for understanding sorting algorithms in Python.

```python
def quicksort(arr):
    if len(arr) <= 1:
        return arr
    else:
        pivot = arr[len(arr)/2]
        left = [x for x in arr if x < pivot]
        right = [x for x in arr if x >= pivot]
        return quicksort(left)+[pivot]+quicksort(right)
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/lyogavin/airllm/blob/main/examples/inferrence.ipynb

Installs all required dependencies for the Anima project from its GitHub repository. This command should be run in a Python environment.

```python
!pip install -r https://github.com/lyogavin/Anima/blob/main/requirements.txt?raw=true

```

--------------------------------

### Example GPT-4 call with backoff

Source: https://github.com/lyogavin/airllm/blob/main/eval/elo_tournanment_all_models_on_translated_vicuna.ipynb

Demonstrates calling the 'run_gpt4_backoff' function with an empty system prompt and a Chinese user prompt, retrieving the first result.

```python
run_gpt4_backoff('', '中国的首都是哪里？')[0]
```

--------------------------------

### Initialize Model with 4-bit Compression

Source: https://github.com/lyogavin/airllm/blob/main/README.md

Use this code to initialize a model with 4-bit block-wise quantization for faster inference. Ensure bitsandbytes and AirLLM are installed.

```python
model = AutoModel.from_pretrained("garage-bAInd/Platypus2-70B-instruct",
                     compression='4bit' # specify '8bit' for 8-bit block-wise quantization 
                    )
```

--------------------------------

### Verify CUDA Installation

Source: https://github.com/lyogavin/airllm/blob/main/training/README.md

Run this command to check if your CUDA installation is successful. This helps diagnose 'Half' tensor runtime errors.

```bash
nvidia-smi
```

--------------------------------

### Generated Text Example

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/examples/run_all_types_of_models.ipynb

Shows an example of generated text output from a model. The '[gMASK]sop' prefix suggests a specific prompt format or control token usage.

```text
Result:
'[gMASK]sop I like a bird on'
```

--------------------------------

### Inference Result Example

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_compression.ipynb

Displays a sample output string from a model inference, likely after compression or loading.

```text
Result:
'<s> I like to think of'
```

--------------------------------

### Install wonderwords Package

Source: https://github.com/lyogavin/airllm/blob/main/anima_100k/gen_longchat_lines_retrieval_eval_dataset.ipynb

Installs the 'wonderwords' package, which might be used for generating text or other related functionalities. This is a prerequisite for certain operations within the notebook.

```python
!pip install wonderwords
```

--------------------------------

### AirLLM Inference Example

Source: https://github.com/lyogavin/airllm/blob/main/README.md

Initialize AirLLM and perform inference using a Hugging Face model repository ID or a local path. Ensure sufficient disk space for layer-wise model saving.

```python
from airllm import AutoModel

MAX_LENGTH = 128
# could use hugging face model repo id:
model = AutoModel.from_pretrained("garage-bAInd/Platypus2-70B-instruct")

# or use model's local path...
#model = AutoModel.from_pretrained("/home/ubuntu/.cache/huggingface/hub/models--garage-bAInd--Platypus2-70B-instruct/snapshots/b585e74bcaae02e52665d9ac6d23f4d0dbc81a0f")

input_text = [
        'What is the capital of United States?',
        #'I like',
    ]

input_tokens = model.tokenizer(input_text,
    return_tensors="pt", 
    return_attention_mask=False, 
    truncation=True, 
    max_length=MAX_LENGTH, 
    padding=False)
           
generation_output = model.generate(
    input_tokens['input_ids'].cuda(), 
    max_new_tokens=20,
    use_cache=True,
    return_dict_in_generate=True)

output = model.tokenizer.decode(generation_output.sequences[0])

print(output)
```

--------------------------------

### Xformers Installation Warning

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_models_transformer_4_35_2.ipynb

A warning message indicating that Xformers is not installed correctly and providing instructions for installation to accelerate training.

```text
WARNING:transformers_modules.70f0215b61f05d7200408dac35466aaf447d1660.modeling_baichuan:Xformers is not installed correctly. If you want to use memory_efficient_attention to accelerate training use the following command to install Xformers\npip install xformers.
```

--------------------------------

### Install AirLLM

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/examples/run_on_macos.ipynb

Installs the AirLLM package and its dependencies using pip. This command should be run in a terminal or a notebook cell.

```bash
!pip install -U  airllm
```

--------------------------------

### Model Bin File (Part 2) Download Progress

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_compression.ipynb

Shows the download progress for the second part of the model's binary weights.

```text
pytorch_model-00002-of-00002.bin:   0%|          | 0.00/3.50G [00:00<?, ?B/s]
```

--------------------------------

### Download Model Assets (NOTICE)

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_models_transformer_4_36_2_torch_2_1_2.ipynb

Downloads the NOTICE file for the Qwen model. This file may contain legal notices or attributions.

```shell
NOTICE:   0%|          | 0.00/15.3k [00:00<?, ?B/s]
```

--------------------------------

### Install Required Python Packages

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_models_transformer_4_36_2_torch_2_1_2.ipynb

Installs essential Python libraries for working with language models, including tiktoken, einops, and transformers_stream_generator.

```python
!pip install tiktoken einops transformers_stream_generator
```

--------------------------------

### FlashAttention Dependency Warning

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/examples/run_all_types_of_models.ipynb

This warning indicates that FlashAttention is not installed, which might lead to lower efficiency. Consider installing it for performance gains.

```text
WARNING:transformers_modules.ffe04dd57f85293043ba999a2c0daa788d6182e9.modeling_qwen:Warning: import flash_attn fail, please install FlashAttention to get higher efficiency https://github.com/Dao-AILab/flash-attention
```

--------------------------------

### Initialize ELO Tournament Parameters

Source: https://github.com/lyogavin/airllm/blob/main/eval/elo_tournanment_all_models_on_translated_vicuna.ipynb

Sets up the parameters for the ELO tournament, including the number of rounds, K-factor, initial rating, scale, base, and seeds the random number generator.

```python
rounds = 300
K=32
seed=42
INIT_RATING=1000
SCALE=400
BASE=10
random.seed(seed)

records = []
rating = defaultdict(lambda: INIT_RATING)
```

--------------------------------

### Model Bin File (Part 1) Download Progress

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_compression.ipynb

Indicates the download progress for the first part of the model's binary weights.

```text
pytorch_model-00001-of-00002.bin:   0%|          | 0.00/9.98G [00:00<?, ?B/s]
```

--------------------------------

### Install and Copy AirLLM Code

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_models_transformer_4_36_2_torch_2_1_2.ipynb

Commands to show installed airllm package details and copy local code for debugging purposes.

```bash
#!pip show airllm
```

```bash
#!cp ./*.py /usr/local/lib/python3.10/dist-packages/airllm/
```

```bash
#!rm ./airllm.py
```

--------------------------------

### Model Output Example

Source: https://github.com/lyogavin/airllm/blob/main/eval/elo_tournanment_all_models_on_translated_vicuna.ipynb

An example of GPT-4 output for a given question, including the original question, translation, model answers, and extracted answers.

```python
gpt4 output: {'question_id': 12, 'text': 'What factors would you consider when designing an inclusive and accessible public transportation system?', 'category': 'knowledge', 'translation': '在设计一个包容性和无障碍的公共交通系统时，您会考虑哪些因素？', 'cn_vicuna_answer': 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n在设计一个包容性和无障碍的公共交通系统时，您会考虑哪些因素？\n\n### Response:\n当你为了创造一种可以接受所有人的公共交通体系时，你应该考虑如何使用这种体系。 你应该考虑如何将这种体系与其他交通工具结合起来。 还要考虑是否能提供安全、快速和方便的服务。\n\n### Explanation:\nWhen designing a public transport system, you will consider which factors?\nYou should think about how to use this kind of system when creating one for everyone. You need to think about whether it can be combined with other means of transport and also if it provides safe, fast and convenient services.', 'extracted_cn_vicuna_answer': '当你为了创造一种可以接受所有人的公共交通体系时，你应该考虑如何使用这种体系。 你应该考虑如何将这种体系与其他交通工具结合起来。 还要考虑是否能提供安全、快速和方便的服务。', 'belle_answer': 'Human: \n在设计一个包容性和无障碍的公共交通系统时，您会考虑哪些因素？\n\nAssistant:\n1. 人口密度：需要考虑城市或地区的人口密度，以确定需要多少公共交通工具和路线。\n2. 可达性：需要让更多的人能够方便地使用公共交通，包括不同年龄、身体状况和经济能力的人群。\n3. 可持续性：需要考虑公共交通对环境的影响，并采取措施减少碳排放和其他污染物。\n4. 安全性：需要保证乘客和司机的安全，包括提供必要的紧急出口和安全设备。\n5. 可靠性：需要确保公共交通的准点率和可靠性，以便乘客能够按时到达目的地。\n6. 便利性：需要考虑乘客的需求，例如提供无障碍服务、充电站等。\n7. 合理性：需要考虑公共交通的成本和运营效率，以确保其可持续发展。', 'extracted_belle_answer': '1. 人口密度：需要考虑城市或地区的人口密度，以确定需要多少公共交通工具和路线。\n2. 可达性：需要让更多的人能够方便地使用公共交通，包括不同年龄、身体状况和经济能力的人群。\n3. 可持续性：需要考虑公共交通对环境的影响，并采取措施减少碳排放和其他污染物。\n4. 安全性：需要保证乘客和司机的安全，包括提供必要的紧急出口和安全设备。\n5. 可靠性：需要确保公共交通的准点率和可靠性，以便乘客能够按时到达目的地。\n6. 便利性：需要考虑乘客的需求，例如提供无障碍服务、充电站等。\n7. 合理性：需要考虑公共交通的成本和运营效率，以确保其可持续发展。', 'score': [7.0, 9.0], 'row_id': 11, 'first_model': 'cn_vicuna', 'second_model': 'belle'}
```

--------------------------------

### Download Model Python Script for Modeling

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_models_transformer_4_36_2_torch_2_1_2.ipynb

Downloads the main Python script for modeling the Qwen architecture. This file contains the model's class definitions.

```shell
modeling_qwen.py:   0%|          | 0.00/55.6k [00:00<?, ?B/s]
```

--------------------------------

### Download Qwen Generation Utilities

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_models_transformer_4_36_2_torch_2_1_2.ipynb

Downloads a Python script containing utility functions for Qwen model generation. This aids in controlling the generation process.

```shell
qwen_generation_utils.py:   0%|          | 0.00/14.6k [00:00<?, ?B/s]
```

--------------------------------

### Download Tokenizer Configuration

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_models_transformer_4_36_2_torch_2_1_2.ipynb

Downloads the configuration file for the Qwen tokenizer. This file specifies tokenizer parameters and settings.

```shell
tokenizer_config.json:   0%|          | 0.00/174 [00:00<?, ?B/s]
```

--------------------------------

### Check PyTorch and Related Libraries Versions

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_models_transformer_4_36_2_torch_2_1_2.ipynb

This command lists all installed PyTorch-related packages and their versions. Verify that PyTorch and its associated libraries are installed correctly and are compatible with each other.

```bash
!pip list | grep torch
```

--------------------------------

### Download Model Configuration File

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_models_transformer_4_36_2_torch_2_1_2.ipynb

Downloads the configuration file for a Qwen model. This is part of the model loading process.

```shell
config.json:   0%|          | 0.00/911 [00:00<?, ?B/s]
```

--------------------------------

### Download Model README File

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_models_transformer_4_36_2_torch_2_1_2.ipynb

Downloads the README file for the Qwen model. This file provides information about the model.

```shell
README.md:   0%|          | 0.00/20.1k [00:00<?, ?B/s]
```

--------------------------------

### Model Loading Performance Metrics (Second Instance)

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_models_transformer_4_35_2.ipynb

Detailed timing metrics for model loading, similar to the first instance, with slightly different values.

```text
total time for load_safe_tensor: 1.0051381284403647
total time for compression_time: 0.000952750999942964
total time for pin_memory_time: 59.13254952430725
total time for load_safe_tensor_cpu_wait: 49.493255853652954
total time for create_layer_from_state_dict: 3.094468593597412
total time for kick_off_load_cpu: 0.001931905746459961
total infer process time(including all above plus gpu compute): 24.3226
total infer wall time(including all above plus gpu compute): 66.1063
new version of transfomer, no need to use BetterTransformer, setting attn impl to sdpa...
either BetterTransformer or attn_implementation='sdpa' is available, creating model without flashattention
```

--------------------------------

### Assert macOS Platform

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/examples/run_on_macos.ipynb

Ensures the script is executed on a macOS environment. This is a prerequisite for the following installation steps.

```python
from sys import platform

assert platform == "darwin", "this example is supposed to be run on mac os"
```

--------------------------------

### Initialize Progress Bar and Results List

Source: https://github.com/lyogavin/airllm/blob/main/eval/elo_tournanment_all_models_on_translated_vicuna.ipynb

Initializes a progress bar to track the tournament rounds and an empty list to store the results of each comparison.

```python
pbar = tqdm(total=rounds)
result_list = []
```

--------------------------------

### Download Tokenizer Vocabulary File

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_models_transformer_4_36_2_torch_2_1_2.ipynb

Downloads the vocabulary file for the Qwen tokenizer. This file maps tokens to their IDs.

```shell
qwen.tiktoken:   0%|          | 0.00/2.56M [00:00<?, ?B/s]
```

--------------------------------

### Test Case for Longest Common Subsequence

Source: https://github.com/lyogavin/airllm/blob/main/eval/elo_tournanment_all_models_on_translated_vicuna.ipynb

Demonstrates how to use the `longest_common_subsequence` function with example strings and prints the result.

```python
s1 = "abcde"
s2 = "ace"
print(longest_common_subsequence(s1, s2))  # 3
```

--------------------------------

### Example Output String

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_models_transformer_4_35_2.ipynb

A simple string output, likely from a print statement or a model's generated text.

```text
Result:\n'I like to think that'
```

--------------------------------

### Inference Timing Metrics (First Run)

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_models_transformer_4_36_2_torch_2_1_2.ipynb

Details the time taken for various stages of inference, including loading, CPU wait, and GPU computation.

```text
total time for load_safe_tensor: 1.1101220474658362
total time for compression_time: 0.0007654799999841089
total time for pin_memory_to_trigger_load: 59.261908531188965
total time for load_safe_tensor_cpu_wait: 47.84440994262695
total time for create_layer_from_state_dict: 3.110097646713257
total time for kick_off_load_cpu: 0.002074718475341797
total infer process time(including all above plus gpu compute): 24.9389
total infer wall time(including all above plus gpu compute): 65.7340
new version of transfomer, no need to use BetterTransformer, try setting attn impl to sdpa...
attn imp: <class 'transformers.models.llama.modeling_llama.LlamaSdpaAttention'>
```

--------------------------------

### FlashAttention Rotary Warning

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_models_transformer_4_35_2.ipynb

This warning indicates that flash_attn rotary could not be imported. Install FlashAttention rotary for improved efficiency.

```python
WARNING:transformers_modules.ffe04dd57f85293043ba999a2c0daa788d6182e9.modeling_qwen:Warning: import flash_attn rotary fail, please install FlashAttention rotary to get higher efficiency https://github.com/Dao-AILab/flash-attention/tree/main/csrc/rotary
```

--------------------------------

### Import AutoModel from AirLLM

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_mixtral.ipynb

Imports the AutoModel class from the airllm library. This is a common starting point for using AirLLM models.

```python
from airllm import AutoModel
```

--------------------------------

### Download Model License File

Source: https://github.com/lyogavin/airllm/blob/main/air_llm/tests/test_notebooks/test_models_transformer_4_36_2_torch_2_1_2.ipynb

Downloads the LICENSE file for the Qwen model. This file contains licensing information.

```shell
LICENSE:   0%|          | 0.00/6.90k [00:00<?, ?B/s]
```

--------------------------------

### Load and Run ChatGLM Model

Source: https://github.com/lyogavin/airllm/blob/main/README.md

Demonstrates how to load the ChatGLM-6B model using AutoModel, tokenize input text, and generate a response. Ensure CUDA is available for GPU acceleration.

```python
from airllm import AutoModel
MAX_LENGTH = 128
model = AutoModel.from_pretrained("THUDM/chatglm3-6b-base")
input_text = ['What is the capital of China?',]
input_tokens = model.tokenizer(input_text,
    return_tensors="pt", 
    return_attention_mask=False, 
    truncation=True, 
    max_length=MAX_LENGTH, 
    padding=True)
generation_output = model.generate(
    input_tokens['input_ids'].cuda(), 
    max_new_tokens=5,
    use_cache= True,
    return_dict_in_generate=True)
model.tokenizer.decode(generation_output.sequences[0])
```