LLM Related

https://github.com/wyf3/llm_related
Admin
This project aims to reproduce various algorithms related to large language models (LLMs).
Tokens:24,536
Snippets:162
Trust Score:7.5
Update:1 year ago
Show doc for...
Tokens:
Raw
### Initializing OpenAI Client (Local) - Python

Source: https://github.com/wyf3/llm_related/blob/main/all_to_tool_call/test.ipynb

This snippet initializes the OpenAI client with a placeholder API key and a local base URL, indicating a setup for local development or a self-hosted model.

```Python
client = OpenAI(
    api_key='yyy',
    base_url="http://10.250.2.24:8888/v1",
)
```

--------------------------------

### Initializing OpenAI Client (Commented) - Python

Source: https://github.com/wyf3/llm_related/blob/main/all_to_tool_call/test.ipynb

This snippet imports necessary modules and shows a commented-out example of initializing the OpenAI client with a specific API key and base URL, likely for a production or external API endpoint.

```Python
from openai import OpenAI
from datetime import datetime
import json
import os
import random

# client = OpenAI(
#     api_key='sk-zfaguzfmjrruybpjgwaxabwytcdgwrvrcsldmxigrsmolpyt',
#     base_url="https://api.siliconflow.cn/v1",
# )
```

--------------------------------

### Accelerated MOE Pre-training with DeepSpeed

Source: https://github.com/wyf3/llm_related/blob/main/train_moe_from_scratch/README.md

Launches accelerated pre-training of the MOE model using DeepSpeed. The `--include 'localhost:0,1'` argument specifies using GPUs 0 and 1 on the local machine for distributed training with DeepSpeed's optimizations.

```bash
deepspeed --include 'localhost:0,1' moe_train.py
```

--------------------------------

### Loading Image from File (Python)

Source: https://github.com/wyf3/llm_related/blob/main/train_multimodal_from_scratch/test.ipynb

This code loads an image from a specified file path using the `PIL.Image.open` function. This image will serve as the visual input for the Vision-Language Model.

```python
image = Image.open('/home/user/wyf/th.png')
```

--------------------------------

### Running MOE Pre-training Directly (Python)

Source: https://github.com/wyf3/llm_related/blob/main/train_moe_from_scratch/README.md

Executes the MOE pre-training script directly using the Python interpreter. This is suitable for single-node, single-GPU, or CPU training environments.

```bash
python moe_train.py
```

--------------------------------

### Distributed MOE SFT Training with torchrun

Source: https://github.com/wyf3/llm_related/blob/main/train_moe_from_scratch/README.md

Initiates distributed Supervised Fine-Tuning (SFT) of the MOE model using `torchrun`. This command enables multi-GPU SFT training by utilizing 2 processes per node.

```bash
torchrun --nproc_per_node=2 moe_sft_train.py
```

--------------------------------

### Initializing Image Processor (Python)

Source: https://github.com/wyf3/llm_related/blob/main/train_multimodal_from_scratch/test.ipynb

This code initializes an `AutoProcessor` from a specified local path. The processor is crucial for preparing image inputs for the Vision-Language Model, handling tasks like resizing and normalization.

```python
processor = AutoProcessor.from_pretrained("/home/user/wyf/siglip-base-patch16-224")
```

--------------------------------

### Accelerated MOE SFT Training with DeepSpeed

Source: https://github.com/wyf3/llm_related/blob/main/train_moe_from_scratch/README.md

Launches accelerated Supervised Fine-Tuning (SFT) of the MOE model using DeepSpeed. This command leverages DeepSpeed's capabilities for efficient multi-GPU SFT training on local GPUs 0 and 1.

```bash
deepspeed --include 'localhost:0,1' moe_sft_train.py
```

--------------------------------

### Preparing Text Input with Chat Template (Python)

Source: https://github.com/wyf3/llm_related/blob/main/train_multimodal_from_scratch/test.ipynb

This snippet constructs the input text for the model using the tokenizer's chat template. It includes a system prompt and a user query with an `<image>` placeholder, which is then replaced by multiple `<|image_pad|>` tokens to reserve space for image embeddings. It also tokenizes the prepared text.

```python
q_text = tokenizer.apply_chat_template([{"role":"system", "content":'You are a helpful assistant.'}, {"role":"user", "content":'描述图片内容\n<image>'}], \
            tokenize=False, \
            add_generation_prompt=True).replace('<image>', '<|image_pad|>'*49)

print(q_text)
input_ids = tokenizer(q_text, return_tensors='pt')['input_ids']
print(input_ids)
```

--------------------------------

### Distributed MOE Pre-training with torchrun

Source: https://github.com/wyf3/llm_related/blob/main/train_moe_from_scratch/README.md

Initiates distributed pre-training of the MOE model using `torchrun`. The `--nproc_per_node=2` argument specifies running on 2 processes per node, typically mapping to 2 GPUs for multi-GPU training.

```bash
torchrun --nproc_per_node=2 moe_train.py
```

--------------------------------

### Running MOE SFT Training Directly (Python)

Source: https://github.com/wyf3/llm_related/blob/main/train_moe_from_scratch/README.md

Executes the MOE Supervised Fine-Tuning (SFT) script directly using the Python interpreter. This method is for single-node SFT training.

```bash
python moe_sft_train.py
```

--------------------------------

### Displaying Loaded Image (Python)

Source: https://github.com/wyf3/llm_related/blob/main/train_multimodal_from_scratch/test.ipynb

This snippet, typically used in interactive environments like Jupyter notebooks, displays the image object that was previously loaded. It's a quick way to verify the image content.

```python
image
```

--------------------------------

### Loading Pre-trained Vision-Language Model (Python)

Source: https://github.com/wyf3/llm_related/blob/main/train_multimodal_from_scratch/test.ipynb

This code loads a pre-trained `AutoModelForCausalLM` instance from a specified local directory. This model is the core Vision-Language Model that will be used for inference, combining both text and image understanding capabilities.

```python
model = AutoModelForCausalLM.from_pretrained('/home/user/wyf/train_multimodal_from_scratch/save/sft')
```

--------------------------------

### Importing Libraries and Custom VLM Components (Python)

Source: https://github.com/wyf3/llm_related/blob/main/train_multimodal_from_scratch/test.ipynb

This snippet imports essential classes from the `transformers` library for model and tokenizer handling, `PIL` for image processing, and custom `VLMConfig`, `VLM` from a local `train` module. These imports are prerequisites for setting up and running a Vision-Language Model.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM, AutoProcessor, AutoConfig
from PIL import Image
from train import VLMConfig, VLM
```

--------------------------------

### Initializing LangChain ChatOpenAI Model in Python

Source: https://github.com/wyf3/llm_related/blob/main/deepseek_learn/test.ipynb

This snippet imports the `ChatOpenAI` class from `langchain_openai` and creates an instance of it. Similar to the raw OpenAI client, it configures the model to use a custom `base_url` and `api_key`, and sets `temperature` to 0 for consistent responses. This setup integrates an OpenAI-compatible model into the LangChain framework.

```python
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(temperature=0, base_url='http://10.250.2.24:8222/v1', api_key='nn', model='ddd')
```

--------------------------------

### Example Call to Chat Completion Function - Python

Source: https://github.com/wyf3/llm_related/blob/main/all_to_tool_call/test.ipynb

This snippet demonstrates how to call the `get_response` function with a user query, triggering the chat completion process and potentially invoking one of the defined tools based on the query.

```Python
get_response('北京天气怎么样')
```

--------------------------------

### Initializing Text Tokenizer and Registering Custom Model (Python)

Source: https://github.com/wyf3/llm_related/blob/main/train_multimodal_from_scratch/test.ipynb

This snippet initializes an `AutoTokenizer` from a local path for text processing. It also registers `VLMConfig` and `VLM` with `AutoConfig` and `AutoModelForCausalLM` respectively, allowing the `transformers` library to recognize and load the custom VLM architecture.

```python
tokenizer = AutoTokenizer.from_pretrained('/home/user/Downloads/Qwen2.5-0.5B-Instruct')
AutoConfig.register("vlm_model", VLMConfig)
AutoModelForCausalLM.register(VLMConfig, VLM)
```

--------------------------------

### Testing MOE Model (Python)

Source: https://github.com/wyf3/llm_related/blob/main/train_moe_from_scratch/README.md

Executes the MOE model testing script directly using the Python interpreter. This script is used to evaluate the performance or functionality of the trained model.

```bash
python moe_test.py
```

--------------------------------

### Defining System Prompt Format in Python

Source: https://github.com/wyf3/llm_related/blob/main/deepseek_learn/test.ipynb

This snippet defines a `SYSTEM_PROMPT` constant, which is a multi-line string. It specifies a desired output format for an AI model, requiring the response to be enclosed within `<think>` and `<answer>` XML-like tags. This prompt is typically used to guide the model's behavior and output structure.

```python
SYSTEM_PROMPT = """
按照如下格式生成：
<think>
...
</think>
<answer>
...
</answer>
"""
```

--------------------------------

### Performing Model Inference and Text Generation (Python)

Source: https://github.com/wyf3/llm_related/blob/main/train_multimodal_from_scratch/test.ipynb

This comprehensive snippet sets the model to evaluation mode and implements a greedy or top-k sampling text generation loop. It iteratively feeds `input_ids` and `pixel_values` to the model, processes logits, and appends the generated token until a maximum length is reached or an EOS token is generated. It includes logic for temperature and top-k sampling.

```python
model.eval()
import torch
from torch.nn import functional as F
max_new_tokens = 20
temperature = 0.0
eos = tokenizer.eos_token_id
top_k = None
s = input_ids.shape[1]
while input_ids.shape[1] < s + max_new_tokens - 1:  
    inference_res = model(input_ids, None, pixel_values)  
    logits = inference_res.logits 
    logits = logits[:, -1, :] 

    for token in set(input_ids.tolist()[0]):  
        logits[:, token] /= 1.0

    if temperature == 0.0: 
        _, idx_next = torch.topk(logits, k=1, dim=-1)
    else:
        logits = logits / temperature  
        if top_k is not None:  
            v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
            logits[logits < v[:, [-1]]] = -float('Inf') 

        probs = F.softmax(logits, dim=-1)  
        idx_next = torch.multinomial(probs, num_samples=1, generator=None)  

    if idx_next == eos:  
        break

    input_ids = torch.cat((input_ids, idx_next), dim=1)  
print(input_ids[:, s:])
```

--------------------------------

### Decoding Generated Text (Python)

Source: https://github.com/wyf3/llm_related/blob/main/train_multimodal_from_scratch/test.ipynb

This snippet decodes the newly generated token IDs (from the `s` index onwards) back into a human-readable string using the tokenizer. This displays the model's generated response based on the input text and image.

```python
print(tokenizer.decode(input_ids[:, s:][0]))
```

--------------------------------

### Re-initializing Final Output Tensor (Demonstration)

Source: https://github.com/wyf3/llm_related/blob/main/train_moe_from_scratch/1.ipynb

This snippet re-initializes the `final_outputs` tensor to zeros, demonstrating a fresh start for accumulating expert outputs. This is typically done once before the expert processing loop, but shown here for clarity of the aggregation step.

```python
final_outputs = torch.zeros_like(x)
final_outputs
```

--------------------------------

### Preparing Input Data for Pre-trained Model - Python

Source: https://github.com/wyf3/llm_related/blob/main/train_llm_from_scratch/test_llm.ipynb

This code prepares input data for the language model. It starts with the beginning-of-sequence (BOS) token ID from the tokenizer, then encodes the Chinese phrase '1+1等于' (1+1 equals) and concatenates them. The resulting list of token IDs is then printed.

```Python
input_data = [t.bos_token_id] + t.encode('1+1等于')
print(input_data)
```

--------------------------------

### Direct Tool Call Example with Qwen API (Python)

Source: https://github.com/wyf3/llm_related/blob/main/all_to_tool_call/README.md

This snippet demonstrates a direct tool call response from a large language model API, specifically showing how the 'tool_calls' field contains the function name ('get_current_weather') and its arguments ('{"location": "北京市"}'). It illustrates the structured output of an API response when a tool call is triggered, indicating the model's decision to invoke an external function.

```Python
ChatCompletion(id='0196bea6713a7620552a143e3aa91f93', choices=[Choice(finish_reason='tool_calls', index=0, logprobs=None, message=ChatCompletionMessage(content='', refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='0196bea67535a49350b2ab4b41a7e588', function=Function(arguments='{"location": "北京市"}', name='get_current_weather'), type='function')]))], created=1746955301, model='Qwen/Qwen2.5-7B-Instruct', object='chat.completion', service_tier=None, system_fingerprint='', usage=CompletionUsage(completion_tokens=22, prompt_tokens=273, total_tokens=295, completion_tokens_details=None, prompt_tokens_details=None))
```

--------------------------------

### Processing Image into Pixel Values (Python)

Source: https://github.com/wyf3/llm_related/blob/main/train_multimodal_from_scratch/test.ipynb

This code uses the previously initialized `processor` to convert the loaded `PIL` image into a tensor of pixel values. The `text=None` argument indicates that only image processing is being performed. The resulting `pixel_values` are in the correct format for model input.

```python
pixel_values = processor(text=None, images=image).pixel_values
print(pixel_values.shape)
```

--------------------------------

### Creating Expert Routing Mask (Example for Expert 0)

Source: https://github.com/wyf3/llm_related/blob/main/train_moe_from_scratch/1.ipynb

This snippet generates a boolean mask `expert_mask` indicating which input elements (batch, sequence position) are routed to expert 0. It checks if expert 0's index is present in the `indices` (top-k selected experts) for any given input element, demonstrating a routing mechanism.

```python
expert_mask = (indices == 0).any(-1)
expert_mask.shape
```

--------------------------------

### Defining MoE Gating Network (Linear Layer)

Source: https://github.com/wyf3/llm_related/blob/main/train_moe_from_scratch/1.ipynb

This snippet defines a simple linear layer `gate` using `torch.nn.Linear`. This layer acts as the gating network in an MoE setup, mapping the input feature dimension (10) to the number of experts (4), which will determine the expert weights for each input.

```python
gate = nn.Linear(10, 4)
```

--------------------------------

### Re-initializing ChatOpenAI LLM with Specific Base URL (Python)

Source: https://github.com/wyf3/llm_related/blob/main/date_modify.ipynb

This snippet re-initializes the ChatOpenAI instance, similar to the first snippet. It uses the same configuration (temperature, model, API key) but specifies a different, more concrete base_url for the LLM API endpoint. This suggests a potential change in the LLM service location or a specific environment setup.

```python
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(temperature=0, model="qwen2", api_key="nn", base_url='http://10.250.2.23:8600/v1')
```

--------------------------------

### Defining LLM Prompt for Date Transformation (Python)

Source: https://github.com/wyf3/llm_related/blob/main/date_modify.ipynb

This snippet defines a multi-line string that serves as a prompt for an LLM. The prompt instructs the LLM to act as a "date conversion assistant" and provides examples of how to transform a natural language query containing a date into a query with explicit start and end timestamps. This is used to standardize date formats for subsequent processing.

```python
date_modify_prompt = '''
你是一个日期转换助手，请按照如下格式对用户输入进行转化。

用户输入：
2023年端午节的订单销量是多少

日期：
['2023-06-22 00:00:00', '2023-06-22 23:59:59']

输出：
2023-06-22 00:00:00到2023-06-22 23:59:59的订单销量是多少

用户输入：
2022年中秋节的营收多少

日期：
['2022-09-10 00:00:00', '2022-09-10 23:59:59']

输出：
2022-09-10 00:00:00到2022-09-10 23:59:59营收多少

用户输入：
{}

日期：
{}

输出：
'''
```

--------------------------------

### Defining Tool Function Schema for LLM (Python)

Source: https://github.com/wyf3/llm_related/blob/main/date_modify.ipynb

This code defines a list of tool functions, specifically one named get_item_info. This function is designed to retrieve information about a product within a specified time range. It includes a JSON schema for its parameters: item (product name), start (start time), and end (end time), all of which are required. This schema is crucial for enabling the LLM to understand and call this function.

```python
functions = [{
        'name': 'get_item_info',
        'description': '获取时间范围内某个产品的信息',
        'parameters': {
            'type': 'object',
            'properties': {
                'item': {
                    'type': 'string',
                    'description':
                    '产品名称'
                },
                'start': {
                    'type': 'string',
                    'description':
                    '时间范围的起始时间'
                },
                'end': {
                    'type': 'string',
                    'description':
                    '时间范围的结束时间'
                }
            },
            'required': ['event', 'start', 'end']
        }
    }]
```

--------------------------------

### Initializing OpenAI Client in Python

Source: https://github.com/wyf3/llm_related/blob/main/deepseek_learn/test.ipynb

This snippet imports the `OpenAI` class and creates an instance of the client. It configures the client to connect to a local or custom API endpoint specified by `base_url` and uses a placeholder `api_key`. This is essential for making subsequent API calls.

```python
from openai import OpenAI
client = OpenAI(base_url='http://10.250.2.24:8222/v1', api_key='nn')
```

--------------------------------

### Setting Up and Running Hugging Face Trainer (Python)

Source: https://github.com/wyf3/llm_related/blob/main/train_llm_from_scratch/train.ipynb

This code block sets up and initiates training using the Hugging Face `Trainer` API. It configures `TrainingArguments` for output, epochs, batch size, and logging, initializes a tokenizer and a `LLMDataset`, and then creates and runs the `Trainer` instance with a pre-defined model.

```python
data_collator = DefaultDataCollator()
tokenizer = AutoTokenizer.from_pretrained("./tokenizer")
args = TrainingArguments(output_dir='./results', 
                        num_train_epochs=20, 
                        do_train=True, 
                        per_device_train_batch_size=2,
                        gradient_accumulation_steps=1,
                        group_by_length=False,
                        max_steps=10,
                        logging_steps=10,
                        report_to = 'none')            
dataset = LLMDataset('./dataset/pretrain_data.jsonl', tokenizer=tokenizer, max_seq_len=512)
trainer = Trainer(model=model, args=args, train_dataset=dataset, tokenizer=tokenizer, data_collator=data_collator)
trainer.train()
```

--------------------------------

### Instantiating LLM Model

Source: https://github.com/wyf3/llm_related/blob/main/train_llm_from_scratch/train.ipynb

This snippet demonstrates how to create an instance of the `LLM` model. It first initializes a `Config` object (presumably a custom configuration class) and then passes it to the `LLM` constructor to build the model.

```Python
config = Config()
model = LLM(config)
```

--------------------------------

### Initializing SFTDataset Instance (Python)

Source: https://github.com/wyf3/llm_related/blob/main/train_multimodal_from_scratch/trainer.ipynb

This snippet initializes an instance of the SFTDataset class. It specifies the paths for image and data files, and passes pre-configured tokenizer, processor, and config objects, preparing the dataset for use in training or evaluation.

```Python
images_path = '/home/user/wyf/train_multimodal_from_scratch/sft_images'
data_path = '/home/user/wyf/llava_instruct_230k.json'
ds = SFTDataset(images_path, data_path, tokenizer, processor, config)
```

--------------------------------

### Printing Document Content Example in Python

Source: https://github.com/wyf3/llm_related/blob/main/table_rag.ipynb

This snippet demonstrates how to access and print the `page_content` of a specific `Document` object from the `splits` list. It's used for debugging or inspecting the processed text chunks before they are embedded.

```python
print(splits[1].page_content)
```

--------------------------------

### Loading Text Datasets from JSONL in Python

Source: https://github.com/wyf3/llm_related/blob/main/train_siglip_from_scratch/data_process.ipynb

This code snippet reads training, validation, and test text data from separate JSON Lines (`.jsonl`) files. Each file is opened in read mode with UTF-8 encoding, and its contents are read line by line into distinct lists: `train_texts`, `valid_texts`, and `test_texts`. The final line prints the number of lines loaded for each dataset, indicating their sizes.

```python
import json
with open('/home/user/wyf/train_siglip_from_scratch/MUGE/train_texts.jsonl', 'r', encoding='utf-8') as f:
    train_texts = f.readlines()

with open('/home/user/wyf/train_siglip_from_scratch/MUGE/valid_texts.jsonl', 'r', encoding='utf-8') as f:
    valid_texts = f.readlines()
    
with open('/home/user/wyf/train_siglip_from_scratch/MUGE/test_texts.jsonl', 'r', encoding='utf-8') as f:
    test_texts = f.readlines()
    
len(train_texts), len(valid_texts), len(test_texts)
```

--------------------------------

### Configuring Hugging Face Training Arguments

Source: https://github.com/wyf3/llm_related/blob/main/train_llm_from_scratch/train.ipynb

This snippet initializes `TrainingArguments` from the Hugging Face `transformers` library. It sets up various training parameters such as output directory, number of epochs, batch size, gradient accumulation steps, and maximum steps, which are used to configure the `Trainer`.

```Python
args = TrainingArguments(output_dir='./results', 
                         num_train_epochs=20, 
                         do_train=True, 
                         per_device_train_batch_size=1, 
                         gradient_accumulation_steps=1,
                         group_by_length=False,
                         max_steps=100)
```

--------------------------------

### Initializing Qwen2.5 Tokenizer - Python

Source: https://github.com/wyf3/llm_related/blob/main/train_multimodal_from_scratch/trainer.ipynb

This code initializes a tokenizer using `AutoTokenizer.from_pretrained`, loading it from a specified local path. This tokenizer is crucial for converting text into numerical input IDs for the language model, ensuring compatibility with the Qwen2.5-0.5B-Instruct model. It's a prerequisite for any text processing or generation tasks.

```Python
tokenizer = AutoTokenizer.from_pretrained('/home/user/Downloads/Qwen2.5-0.5B-Instruct')
```

--------------------------------

### Initializing Multimodal Model Components - Python

Source: https://github.com/wyf3/llm_related/blob/main/train_siglip_from_scratch/test.ipynb

This snippet initializes the necessary components (image processor, text tokenizer, and the multimodal model) for a vision-language task. It loads a pre-trained image processor, a Chinese RoBERTa tokenizer, and a custom-trained AutoModel from specified local file paths, preparing them for inference.

```python
from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel, AutoTokenizer
import torch
processor = AutoProcessor.from_pretrained("/home/user/wyf/train_siglip_from_scratch/vit-base-patch16-224")
tokenizer = AutoTokenizer.from_pretrained('/home/user/wyf/chinese-roberta-wwm-ext')
model = AutoModel.from_pretrained('/home/user/wyf/train_siglip_from_scratch/outputs')
```

--------------------------------

### Initializing SFTDataset for Fine-tuning (Python)

Source: https://github.com/wyf3/llm_related/blob/main/train_llm_from_scratch/train.ipynb

This snippet initializes an instance of `SFTDataset`, specifically for supervised fine-tuning. It provides the path to the SFT data, a tokenizer object, and a maximum sequence length, preparing the dataset for use in a training loop.

```python
dataset = SFTDataset('/home/user/wyf/sft_data_zh.jsonl', tokenizer=tokenizer, max_seq_len=2048)
```

--------------------------------

### Flattening Expert Routing Mask

Source: https://github.com/wyf3/llm_related/blob/main/train_moe_from_scratch/1.ipynb

This snippet flattens the `expert_mask` into a 1D tensor `expert_mask_flat`. This flattened mask is then used to select the specific input elements from `x_flat` that are designated for processing by a particular expert (e.g., expert 0 in this example).

```python
expert_mask_flat = expert_mask.view(-1)
expert_mask_flat
```

--------------------------------

### Invoking LLM for Question Answering in Python

Source: https://github.com/wyf3/llm_related/blob/main/table_rag.ipynb

This snippet demonstrates how to use the initialized LLM to answer a specific `query`. It formats the predefined `prompt` template with the `text` (retrieved context) and the `query`, then invokes the LLM and prints its generated content.

```python
query = '斑小将美白隔离防晒乳的检验结果'
print(llm.invoke(prompt.format(text,query)).content)
```

--------------------------------

### Initializing BPE Trainer with Custom Settings in Python

Source: https://github.com/wyf3/llm_related/blob/main/train_llm_from_scratch/train_tokenizer.ipynb

This code initializes a `BpeTrainer` instance, configuring it with a specified vocabulary size (6400), the previously defined special tokens, and enabling a progress display during training. It also sets the initial alphabet using `pre_tokenizers.ByteLevel.alphabet()` to ensure all possible byte values are considered.

```Python
# 初始化训练器
trainer = trainers.BpeTrainer(
    vocab_size=6400,
    special_tokens = special_tokens,
    show_progress=True,
    initial_alphabet = pre_tokenizers.ByteLevel.alphabet()
    
)
```

--------------------------------

### Invoking LLM with RAG Prompt - Python

Source: https://github.com/wyf3/llm_related/blob/main/rag_demo/rag.ipynb

This snippet initializes a `ChatOpenAI` model, configured to connect to a local Qwen2-7B-Instruct model via a specified base URL. It then invokes the model with the RAG prompt, formatting it with the user's question and the RRF-re-ranked retrieved documents, and prints the model's response.

```python
from langchain_openai import ChatOpenAI
model = ChatOpenAI(model='Qwen2-7B-Instruct', base_url='http://localhost:8000/v1', api_key='n')
res = model.invoke(prompt.format('骨折了应该怎么办', ''.join(rrf_res)))
print(res.content)
```

--------------------------------

### Loading Pre-trained Causal Language Model (Python)

Source: https://github.com/wyf3/llm_related/blob/main/train_multimodal_from_scratch/trainer.ipynb

This line loads a pre-trained causal language model, specifically 'Qwen2.5-0.5B-Instruct', from a local path using AutoModelForCausalLM.from_pretrained. This prepares the model for further fine-tuning or inference tasks.

```Python
model = AutoModelForCausalLM.from_pretrained('/home/user/Downloads/Qwen2.5-0.5B-Instruct')
```

--------------------------------

### Extracting Dates Using Jionlp Library (Python)

Source: https://github.com/wyf3/llm_related/blob/main/date_modify.ipynb

This snippet demonstrates how to extract date information from a natural language query using the jionlp library. It defines a date_extract function that parses the input query and returns the extracted time range. The example shows extracting the date for "last year's Mid-Autumn Festival revenue".

```python
import jionlp
query = '去年中秋节的营收多少'
def date_extract(query):
    date = jionlp.parse_time(query)
    return date['time']
date = date_extract(query)
print(date)
```

--------------------------------

### Defining LLM Prompt Template in Python

Source: https://github.com/wyf3/llm_related/blob/main/table_rag.ipynb

This snippet defines a multi-line string representing a prompt template for the LLM. It includes placeholders for `知识` (knowledge/context) and `用户问题` (user question), along with instructions for the LLM to answer based only on the provided knowledge and to respond '我不知道' if the answer is not found.

```python
prompt = '''任务目标：根据给定的知识，回答用户提出的问题。
任务要求：
1、不得脱离给定的知识进行回答。
2、如果给定的知识不包含问题的答案，请回答我不知道。

知识：
{}

用户问题：
{}
'''
```

--------------------------------

### Calling Chat Completions API with OpenAI Client in Python

Source: https://github.com/wyf3/llm_related/blob/main/deepseek_learn/test.ipynb

This snippet demonstrates how to make a chat completion request using the previously initialized `OpenAI` client. It sends a list of messages, including a system prompt and a user query, to the specified `model`. The `temperature` is set to 0 for deterministic output, and the response content is then printed.

```python
res = client.chat.completions.create(
    model="ddd",
    messages=[{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": '树下一只猴？树上骑个猴？请问一共几只猴？'}],
    temperature=0
)
print(res.choices[0].message.content)
```

--------------------------------

### Tool Calling Implementation Flow Diagram (Mermaid)

Source: https://github.com/wyf3/llm_related/blob/main/all_to_tool_call/README.md

This Mermaid diagram illustrates the high-level implementation flow for tool calling within a large language model system. It outlines the sequential steps from a user's initial request to the final structured result, encompassing model inference, decision generation, parsing by a smaller model, execution of the tool call, and the return of structured output.

```Mermaid
graph TD
    A[用户请求] --> B{大模型推理}
    B --> C[生成调用决策]
    C --> D{小模型解析}
    D --> E[执行工具调用]
    E --> F[返回结构化结果]
```

--------------------------------

### Initializing Input Tensor for MoE Gate

Source: https://github.com/wyf3/llm_related/blob/main/train_moe_from_scratch/1.ipynb

This snippet initializes a random PyTorch tensor `x` with dimensions (2, 4, 10). This tensor simulates the input features that would be fed into an MoE gating network, where the last dimension (10) represents the feature size.

```python
x = torch.randn(2, 4, 10)
```

--------------------------------

### Initializing ChatOpenAI LLM with Custom Base URL (Python)

Source: https://github.com/wyf3/llm_related/blob/main/date_modify.ipynb

This snippet initializes a ChatOpenAI instance from langchain_openai. It configures the LLM with a temperature of 0 for deterministic output, specifies "qwen2" as the model, provides an API key, and sets a custom base_url for the API endpoint. This is typically used when interacting with a self-hosted or custom LLM service.

```python
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(temperature=0, model="qwen2", api_key="nn", base_url='http://***/v1')
```

--------------------------------

### Importing Core Libraries for VLM Development - Python

Source: https://github.com/wyf3/llm_related/blob/main/train_multimodal_from_scratch/trainer.ipynb

This snippet imports essential modules for constructing a Vision-Language Model. It includes components from Hugging Face Transformers for pre-trained models and tokenizers, PIL for image handling, PyTorch for neural network operations, and `typing` for type hints. These imports provide the foundational building blocks for defining the VLM's architecture and data processing.

```Python
from transformers import PreTrainedModel, PretrainedConfig, AutoTokenizer, AutoModelForCausalLM
from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch
import torch.nn as nn
import torch.nn.functional as F
from transformers.modeling_outputs import CausalLMOutputWithPast
from typing import List, Dict, Any
```

--------------------------------

### Saving Tokenizer Configuration File in Python

Source: https://github.com/wyf3/llm_related/blob/main/train_llm_from_scratch/train_tokenizer.ipynb

This comprehensive snippet defines a configuration dictionary for the tokenizer, including settings like special tokens, tokenization behavior, and a chat template. It then saves this configuration as `tokenizer_config.json` in the tokenizer directory, which is essential for loading the tokenizer with libraries like Hugging Face Transformers.

```Python
config = {
        "add_bos_token": False,
        "add_eos_token": False,
        "add_prefix_space": True,
        "added_tokens_decoder": {
            "0": {
                "content": "<unk>",
                "lstrip": False,
                "normalized": False,
                "rstrip": False,
                "single_word": False,
                "special": True
            },
            "1": {
                "content": "<s>",
                "lstrip": False,
                "normalized": False,
                "rstrip": False,
                "single_word": False,
                "special": True
            },
            "2": {
                "content": "</s>",
                "lstrip": False,
                "normalized": False,
                "rstrip": False,
                "single_word": False,
                "special": True
            }
        },
        "additional_special_tokens": [],
        "bos_token": "<s>",
        "clean_up_tokenization_spaces": False,
        "eos_token": "</s>",
        "legacy": True,
        "model_max_length": 100000,
        "pad_token": None,
        "sp_model_kwargs": {},
        "spaces_between_special_tokens": False,
        "tokenizer_class": "PreTrainedTokenizerFast",
        "unk_token": "<unk>",
        "use_default_system_prompt": False,
        "chat_template": "{% if messages[0]['role'] == 'system' %}{% set system_message = messages[0]['content'] %}{% endif %}{% if system_message is defined %}{{ system_message }}{% endif %}{% for message in messages %}{% set content = message['content'] %}{% if message['role'] == 'user' %}{{ '<s>user\\n' + content + '</s>\\n<s>assistant\\n' }}{% elif message['role'] == 'assistant' %}{{ content + '</s>' + '\\n' }}{% endif %}{% endfor %}"
    }

# 保存配置文件
with open(os.path.join(tokenizer_dir, "tokenizer_config.json"), "w", encoding="utf-8") as config_file:
    json.dump(config, config_file, ensure_ascii=False, indent=4)
```

--------------------------------

### Initializing FAISS Retriever in Python

Source: https://github.com/wyf3/llm_related/blob/main/table_rag.ipynb

This snippet configures a retriever from the FAISS vector store. The `as_retriever` method creates an object capable of fetching relevant documents based on a query, with `search_kwargs={'k':3}` specifying that it should retrieve the top 3 most similar documents.

```python
retriever = vectorstore.as_retriever(search_kwargs={'k':3})
```

--------------------------------

### Preparing Chat Input for Fine-tuned Model - Python

Source: https://github.com/wyf3/llm_related/blob/main/train_llm_from_scratch/test_llm.ipynb

This code prepares input data for the fine-tuned model using the tokenizer's `apply_chat_template` method. It formats a user message '1+1等于' into a chat-specific format, which is common for instruction-tuned or chat-tuned models. The resulting token IDs are then printed.

```Python
input_data = t.apply_chat_template([{'role':'user', 'content':'1+1等于'}])
print(input_data)
```

--------------------------------

### Importing Libraries for RAG System - Python

Source: https://github.com/wyf3/llm_related/blob/main/rag_demo/rag.ipynb

This snippet imports necessary classes and functions from `langchain_community`, `langchain`, `jieba`, `langchain_huggingface`, and `typing` for building a Retrieval-Augmented Generation (RAG) system. These include components for retrievers, text splitting, vector stores, document loading, and embeddings.

```python
from langchain_community.retrievers import BM25Retriever
from typing import List
import jieba
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.vectorstores import FAISS
from langchain.document_loaders import TextLoader
from langchain_huggingface import HuggingFaceEmbeddings
```

--------------------------------

### Initializing Tokenizer and Registering Custom Model/Config - Python

Source: https://github.com/wyf3/llm_related/blob/main/train_llm_from_scratch/test_llm.ipynb

This code initializes a tokenizer from a pre-trained path. It then registers a custom `Config` class under the name 'small_model' and associates the custom `LLM` model class with this `Config` for `AutoModelForCausalLM`, enabling the `from_pretrained` method to load models based on these custom definitions.

```Python
t = AutoTokenizer.from_pretrained('/home/user/wyf/train_model_from_scratch/saves/pretrain')
AutoConfig.register("small_model", Config)
AutoModelForCausalLM.register(Config, LLM)
```

--------------------------------

### Instantiating VLMConfig Object (Python)

Source: https://github.com/wyf3/llm_related/blob/main/train_multimodal_from_scratch/trainer.ipynb

This line initializes a VLMConfig object, which is expected to hold configuration parameters for a Vision-Language Model. This object would typically define model architecture, training parameters, or other settings relevant to the VLM.

```Python
config = VLMConfig()
```

--------------------------------

### Importing PyTorch Modules for MoE Implementation

Source: https://github.com/wyf3/llm_related/blob/main/train_moe_from_scratch/1.ipynb

This snippet imports essential PyTorch modules: `torch` for tensor operations, `torch.nn` for neural network layers, and `torch.nn.functional` for functional operations like softmax. These imports are prerequisites for building and operating a Mixture-of-Experts (MoE) layer.

```python
import torch
import torch.nn as nn
import torch.nn.functional as F
```

--------------------------------

### Loading and Testing Tokenizer with Hugging Face Transformers in Python

Source: https://github.com/wyf3/llm_related/blob/main/train_llm_from_scratch/train_tokenizer.ipynb

This snippet demonstrates how to load the previously trained and saved tokenizer using `AutoTokenizer.from_pretrained` from the Hugging Face `transformers` library. It then immediately tests the tokenizer by encoding the Chinese phrase '您好' (Hello), verifying its functionality.

```Python
# 测试
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("./tokenizer")
tokenizer.encode("您好")
```

--------------------------------

### Initializing BPE Tokenizer with ByteLevel Pre-tokenizer in Python

Source: https://github.com/wyf3/llm_related/blob/main/train_llm_from_scratch/train_tokenizer.ipynb

This snippet initializes a new Byte-Pair Encoding (BPE) tokenizer using `models.BPE()`. It then configures the pre-tokenizer to `pre_tokenizers.ByteLevel`, which handles byte-level pre-tokenization, ensuring that spaces are not automatically prefixed.

```Python
# BPE分词器
tokenizer = Tokenizer(models.BPE())
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)
```

--------------------------------

### Consolidating Text Datasets to JSONL in Python

Source: https://github.com/wyf3/llm_related/blob/main/train_siglip_from_scratch/data_process.ipynb

This snippet concatenates the `train_texts`, `valid_texts`, and `test_texts` lists and writes their combined content to a new file named `all_texts.jsonl`. The file is opened in write mode (`'w'`) with UTF-8 encoding, ensuring all text data from the individual datasets is consolidated into a single JSON Lines file for unified access.

```python
with open('/home/user/wyf/train_siglip_from_scratch/MUGE/all_texts.jsonl', 'w', encoding='utf-8') as f:
    f.writelines(train_texts + valid_texts + test_texts)
```

--------------------------------

### Importing Libraries for Tokenizer Training in Python

Source: https://github.com/wyf3/llm_related/blob/main/train_llm_from_scratch/train_tokenizer.ipynb

This snippet imports the necessary modules from the `tokenizers` library, including `decoders`, `models`, `pre_tokenizers`, `trainers`, and `Tokenizer`, along with `os` and `json` for file operations and data handling.

```Python
from tokenizers import decoders, models, pre_tokenizers, trainers, Tokenizer
import os 
import json
```

--------------------------------

### Defining Custom Tools for OpenAI Chatbot - Python

Source: https://github.com/wyf3/llm_related/blob/main/all_to_tool_call/test.ipynb

This snippet defines a list of tools that the OpenAI model can use. It includes `get_current_time` (for fetching the current time) and `get_current_weather` (for querying weather by location), specifying their descriptions and required parameters.

```Python
tools = [
    # 工具1 获取当前时刻的时间
    {
        "type": "function",
        "function": {
            "name": "get_current_time",
            "description": "当你想知道现在的时间时非常有用。",
            "parameters": {},
        },
    },
    # 工具2 获取指定城市的天气
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "当你想查询指定城市的天气时非常有用。",
            "parameters": {
                "type": "object",
                "properties": {
                    # 查询天气时需要提供位置，因此参数设置为location
                    "location": {
                        "type": "string",
                        "description": "城市或县区，比如北京市、杭州市、余杭区等。",
                    }
                },
                "required": ["location"],
            },
        },
    },
]
```

--------------------------------

### Initializing LLMDataset for Pre-training (Python)

Source: https://github.com/wyf3/llm_related/blob/main/train_llm_from_scratch/train.ipynb

This snippet initializes an instance of `LLMDataset`, likely used for pre-training large language models. It specifies the path to the pre-training data, a tokenizer instance, and the maximum sequence length for processing.

```python
dataset = LLMDataset('./dataset/pretrain_data.jsonl', tokenizer=tokenizer, max_seq_len=512)
```

--------------------------------

### Importing Transformers and Custom Model Components - Python

Source: https://github.com/wyf3/llm_related/blob/main/train_llm_from_scratch/test_llm.ipynb

This snippet imports core classes from the Hugging Face `transformers` library for tokenization, model loading, and configuration. It also imports `torch` for tensor operations and custom `LLM` and `Config` classes, likely defining the model architecture and its configuration, from a local `train` module.

```Python
from transformers import AutoTokenizer, AutoModelForCausalLM, AutoConfig
import torch
from train import LLM, Config
```

--------------------------------

### Initializing Embeddings, LLM, and FAISS Vector Store in Python

Source: https://github.com/wyf3/llm_related/blob/main/table_rag.ipynb

This snippet initializes a `HuggingFaceEmbeddings` model for generating document embeddings and a `ChatOpenAI` instance for the Large Language Model (LLM). It then creates a FAISS vector store from the previously loaded `splits` using the initialized embeddings, preparing the data for efficient similarity search.

```python
from langchain_community.vectorstores import FAISS
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_openai import ChatOpenAI
embeddings = HuggingFaceEmbeddings(model_name='/home/user/wyf/fastchat/bge-large-zh-v1.5', model_kwargs = {'device': 'cuda:1'})
llm = ChatOpenAI(temperature=0, model='/home/user/Downloads/Qwen2-7B-Instruct/', base_url='http://10.250.2.23:8600/v1', api_key='ww')
vectorstore = FAISS.from_documents(splits, embeddings)
```

--------------------------------

### Defining SFTDataset for Supervised Fine-tuning (Python)

Source: https://github.com/wyf3/llm_related/blob/main/train_llm_from_scratch/train.ipynb

This class defines a PyTorch `Dataset` for Supervised Fine-Tuning (SFT) of language models. It loads data from a JSONL file, processes each entry to construct conversational prompts and answers using a tokenizer's chat template, and then prepares `input_ids` and `labels` with appropriate padding/truncation for training.

```python
class SFTDataset(Dataset):
    def __init__(self, data_path, tokenizer, max_seq_len):
        super().__init__()
        self.data_path = data_path
        self.tokenizer = tokenizer
        self.max_seq_len = max_seq_len
        
        with open(self.data_path, 'r', encoding='utf-8') as f:
            self.data = f.readlines()
            
    def __len__(self):
        return len(self.data)    
    
    def __getitem__(self, index):
        line = self.data[index]
        line = json.loads(line)
        instruction_text = line['instruction']
        input_text = line['input']
        output_text = line['output']
        history = line['history']
        query = instruction_text + input_text
        answer = output_text + self.tokenizer.eos_token
        messages = []
        if history:
            for i in history:
                messages.append({'role': 'user', 'content': i[0]})
                messages.append({'role': 'assistant', 'content': i[1]})
        
        messages.append({'role': 'user', 'content': query})   
        prompt = self.tokenizer.apply_chat_template(messages, tokenize=False) 
        prompt_input_ids = self.tokenizer.encode(prompt)
        answer_input_ids = self.tokenizer.encode(answer)
        input_ids = prompt_input_ids + answer_input_ids
        labels = [0] * len(prompt_input_ids) + answer_input_ids
        text_len = len(input_ids)
        if text_len > self.max_seq_len:
            input_ids = input_ids[:self.max_seq_len]
            labels = labels[:self.max_seq_len]
        else:
            input_ids = input_ids + [0] * (self.max_seq_len - text_len)
            labels = labels + [0] * (self.max_seq_len - text_len)
        
        input_ids = input_ids[:-1]
        labels = labels[1:]
        return {'input_ids': torch.tensor(input_ids), 'labels': torch.tensor(labels)}
```

--------------------------------

### Loading and Visualizing MoE Training Loss with Plotly

Source: https://github.com/wyf3/llm_related/blob/main/train_moe_from_scratch/1.ipynb

This snippet loads training history data from a JSON file, extracts 'step' and 'loss' values, and then visualizes the loss progression over steps using a Plotly line chart. It uses pandas for data structuring and plotly.express for plotting, providing a clear view of the model's training performance.

```python
import json
import pandas as pd
with open('/home/user/wyf/train_moe_from_scratch/20000step_trainer_state.json' , 'r', encoding='utf-8') as f:
    data = json.load(f)
    log_history = data['log_history']
    steps = []
    losses = []
    for i in log_history[:-1]:
        step = i['step']
        loss = i['loss']
        steps.append(step)
        losses.append(loss)

import seaborn as sns
import plotly.express as px

# 创建一个DataFrame
df = pd.DataFrame({
    'Step': steps,
    'Loss': losses
})

# 使用Plotly绘制散点图
fig = px.line(df, x='Step', y='Loss', title='MoE预训练损失变化',
                 labels={'Step': 'step', 'Loss': 'loss'},
                 )  # 添加趋势线可选

fig.update_layout(width=600, height=400)
# 显示图表
fig.show()
```

--------------------------------

### Initializing BM25 Retriever with Jieba Preprocessing - Python

Source: https://github.com/wyf3/llm_related/blob/main/rag_demo/rag.ipynb

This code defines a `preprocessing_func` using `jieba` for Chinese text segmentation, which is then passed to the `BM25Retriever`. It initializes a BM25 retriever with the processed document chunks and sets `k` to 10, indicating it will retrieve the top 10 most relevant documents.

```python
def preprocessing_func(text: str) -> List[str]:
    return list(jieba.cut(text))
bm25 = BM25Retriever(docs=docs,k=10)
print(bm25.k)
retriever = bm25.from_documents(docs,preprocess_func=preprocessing_func)
```

--------------------------------

### Invoking LLM Directly (Without RAG) - Python

Source: https://github.com/wyf3/llm_related/blob/main/rag_demo/rag.ipynb

This snippet demonstrates invoking the initialized `ChatOpenAI` model directly with a user question '骨折了应该怎么办' (What should I do if I have a fracture?), without providing any retrieved documents. This can be used to compare the model's performance with and without the RAG context.

```python
res = model.invoke('骨折了应该怎么办')
print(res.content)
```
Tokens:
Raw