### Setup Environment with Bash Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/README.md Clones the repository and sets up the project environment by installing dependencies using `uv`. It also provides instructions to activate the virtual environment. ```bash git clone https://github.com/QwenLM/Qwen3-VL-Embedding.git cd Qwen3-VL-Embedding bash scripts/setup_environment.sh source .venv/bin/activate ``` -------------------------------- ### Setup: Import Libraries and Load Model Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/embedding.ipynb Imports necessary libraries for model operation and data handling. It then loads the Qwen3-VL embedding model from a specified path and initializes the embedder with configuration parameters like max_length, num_frames, and fps. ```python import os import sys import json import torch import numpy as np import tempfile import pickle import requests import matplotlib.pyplot as plt from pathlib import Path from collections import Counter from datasets import load_dataset, Video, load_from_disk from PIL import Image from io import BytesIO from decord import VideoReader, cpu sys.path.append(str(Path("..").resolve())) from src.models.qwen3_vl_embedding import Qwen3VLEmbedder MODEL_PATH = r"./models/Qwen3-VL-Embedding-2B" # "your model path" OUTPUT_DIR = "./retrieval_results" os.makedirs(OUTPUT_DIR, exist_ok=True) embedder = Qwen3VLEmbedder( model_name_or_path=MODEL_PATH, max_length=8192, num_frames=16, fps=1 ) print("Embedding model loaded successfully!") ``` -------------------------------- ### Setup Imports for Qwen3-VL Reranker Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/reranker.ipynb Imports necessary libraries for the Qwen3-VL reranker, including system utilities, data handling (pickle, numpy), plotting, file path manipulation, and image processing (PIL). These are standard imports for machine learning and data processing tasks. ```python import os import sys import pickle import numpy as np import matplotlib.pyplot as plt from pathlib import Path from PIL import Image ``` -------------------------------- ### Install Poppler Utilities for PDF Processing Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/Qwen3VL_Multimodal_RAG.ipynb Installs the poppler-utils package on Ubuntu systems, which is required by pdf2image for converting PDF documents into images. This step ensures that the pipeline can correctly process PDF inputs. ```bash !sudo apt-get install poppler-utils ``` -------------------------------- ### Install Qwen3-VL-Embedding Source: https://context7.com/qwenlm/qwen3-vl-embedding/llms.txt Clones the repository, sets up the environment, and downloads pre-trained models from Hugging Face. Requires Git, Bash, and Hugging Face CLI. ```bash # Clone the repository git clone https://github.com/QwenLM/Qwen3-VL-Embedding.git cd Qwen3-VL-Embedding # Setup environment bash scripts/setup_environment.sh source .venv/bin/activate # Download models from Hugging Face uv pip install huggingface-hub huggingface-cli download Qwen/Qwen3-VL-Embedding-2B --local-dir ./models/Qwen3-VL-Embedding-2B huggingface-cli download Qwen/Qwen3-VL-Reranker-2B --local-dir ./models/Qwen3-VL-Reranker-2B ``` -------------------------------- ### Download Model from Hugging Face with Bash Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/README.md Installs the `huggingface-hub` library and downloads a specified Qwen3-VL-Embedding model to a local directory using the `huggingface-cli`. ```bash uv pip install huggingface-hub huggingface-cli download Qwen/Qwen3-VL-Embedding-2B --local-dir ./models/Qwen3-VL-Embedding-2B ``` -------------------------------- ### Install Python Dependencies for Qwen3-VL RAG Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/Qwen3VL_Multimodal_RAG.ipynb Installs necessary Python packages for the multimodal RAG pipeline, including pdf2image, qwen-vl-utils, flash-attn, and bitsandbytes. These libraries are essential for image processing, model utilities, and performance optimizations. ```python !pip install -q pdf2image>=1.16.0 qwen-vl-utils flash-attn bitsandbytes ``` -------------------------------- ### Progress Bar Output Example Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/reranker_vllm.ipynb These snippets showcase various stages of a progress bar, likely used to indicate the status of operations such as adding requests or processing prompts. They are purely visual indicators and do not contain executable logic. ```text Adding requests: 0%| | 0/3 [00:00system Retrieve images or text relevant to the user's query.<|im_end|> <|im_start|>user A woman playing with her dog on a beach at sunset.<|im_end|> <|im_start|>assistant ... ``` -------------------------------- ### Engine Initialization and Task Information Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/embedding_vllm.ipynb These snippets show log messages related to engine initialization and the supported tasks for the language model. They indicate the time taken for initialization and list available tasks like 'embed' and 'token_embed'. ```text (EngineCore_DP0 pid=82232) ``` ```text INFO 01-16 07:48:14 [gpu_model_runner.py:4852] Graph capturing finished in 3 secs, took -0.76 GiB ``` ```text (EngineCore_DP0 pid=82232) ``` ```text INFO 01-16 07:48:14 [core.py:272] init engine (profile, create kv cache, warmup model) took 27.63 seconds ``` ```text INFO 01-16 07:48:21 [llm.py:347] Supported tasks: ['embed', 'token_embed'] ``` -------------------------------- ### CUDA Graph Capture Progress Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/embedding_vllm.ipynb These snippets show the progress of capturing CUDA graphs for mixed prefill-decode operations. They indicate the percentage completion and the number of iterations completed out of the total. ```text Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/51 [00:00= 2: break # Display queries fig, axes = plt.subplots(1, 2, figsize=(8, 4)) for i, (ax, q) in enumerate(zip(axes, image_qa_queries)): ax.imshow(q['image']) ax.set_title(f"Q: {q['question']}\nA: {q['gt_answer']}", fontsize=9) ax.axis('off') plt.tight_layout() plt.show() ``` -------------------------------- ### Initialize and Use Reranker Model in Python Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/Qwen3VL_Multimodal_RAG.ipynb This snippet initializes a reranker model, prepares inputs including query and document embeddings, and processes them to get reranking scores. It then identifies the best-ranked document based on these scores. Dependencies include numpy and the reranker model itself. ```python query_for_reranking = queries[0] top_indices, _ = retrieve_top_k(query_embeddings[0], document_embeddings, k=3) print(f"\nReranking results for: {query_for_reranking['text']}") reranker_inputs = { "instruction": "Retrieve pages relevant to the user's query about climate change.", "query": query_for_reranking, "documents": [{"image": f"temp_page_{idx}.png"} for idx in top_indices], "fps": 1.0 } reranker_scores = reranker.process(reranker_inputs) for rank, (page_idx, score) in enumerate(zip(top_indices, reranker_scores), 1): print(f" {rank}. Page {page_idx + 1} (reranker score: {score:.4f})") best_page_idx = top_indices[np.argmax(reranker_scores)] print(f"\nBest page after reranking: Page {best_page_idx + 1}") ``` -------------------------------- ### Summarize Reranker Examples and Metrics in Python Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/reranker.ipynb This Python script generates a summary of reranker examples, displaying performance metrics (MRR, R@1, R@5) for both embedding and reranked results for text and image retrieval tasks. It calculates the change in performance and prints the results in a formatted table, along with the paths to saved rerank output files. Dependencies include a `compute_metrics` function and predefined result variables. ```python print("=" * 80) print("RERANKER EXAMPLES SUMMARY") print("=" * 80) print() tasks = [ ("Text Retrieval", text_emb_results, text_rerank_results), ("Image Retrieval", image_emb_results, image_rerank_results) ] print(f"{ 'Task':<18} {'Metric':<6} {'Embedding':<10} {'Reranked':<10} {'Change'}") print("-" * 60) for task_name, emb_res, rerank_res in tasks: metrics = compute_metrics(emb_res, rerank_res) for i, m in enumerate(['mrr', 'r@1', 'r@5']): emb_val = metrics['emb'][m] rerank_val = metrics['rerank'][m] change = rerank_val - emb_val change_str = f"{'+{change:.2%}' if change >= 0 else '{change:.2%}'}" task_display = task_name if i == 0 else "" print(f"{task_display:<18} {m.upper():<6} {emb_val:<10.2%} {rerank_val:<10.2%} {change_str}") print("-" * 60) print() print("Saved rerank outputs:") print(f"- {OUTPUT_DIR}/text_rerank_results.pkl") print(f"- {OUTPUT_DIR}/image_rerank_results.pkl") print("=" * 80) ``` -------------------------------- ### Initialize Qwen3-VL-Reranker Model Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/reranker_vllm.ipynb Initializes the Qwen3-VL-Reranker model with a specified path, runner type, data type, and Hugging Face overrides. It requires the 'LLM' class and sets 'trust_remote_code' to True. The output confirms successful model initialization. ```python from vllm import LLM llm = LLM( model='/cpfs01/user/linqi.lmx/models/finetune/qwen/qwen3-vl/Qwen3-VL-Embedding/Qwen3-VL-Reranker-2B', runner='pooling', dtype='bfloat16', trust_remote_code=True, hf_overrides={ "architectures": ["Qwen3VLForSequenceClassification"], "classifier_from_token": ["no", "yes"], "is_original_qwen3_reranker": True, }, ) print("Model initialized successfully!") ``` -------------------------------- ### Prepare Image Classification Queries and Display Images Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/embedding.ipynb Selects image samples from the dataset to use as queries for classification. It also displays these query images along with their ground truth labels. ```python # Select 2 queries image_cls_query_indices = [0, 50] image_cls_queries = [{ "image": image_cls_dataset[i]['img'], "gt_label": IMAGE_CLS_LABELS[image_cls_dataset[i]['label']], "gt_idx": image_cls_dataset[i]['label'] } for i in image_cls_query_indices] # Display query images fig, axes = plt.subplots(1, 2, figsize=(6, 3)) for i, (ax, q) in enumerate(zip(axes, image_cls_queries)): ax.imshow(q['image']) ax.set_title(f"Query {i+1}: {q['gt_label']}") ax.axis('off') plt.tight_layout() plt.show() ``` -------------------------------- ### Generate Embeddings using Qwen3-VL-Embedding Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/embedding_vllm.ipynb This Python code snippet demonstrates how to generate embeddings using the Qwen3-VL-Embedding model. It assumes that `llm` is an initialized model object and `vllm_inputs` is a list of input samples. ```python # Generate embeddings outputs = llm.embed(vllm_inputs) ``` -------------------------------- ### Load VQAv2 Dataset and Build Answer Corpus Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/embedding.ipynb Loads the validation split of the VQAv2 dataset and constructs a corpus of the 50 most common answers. This corpus will be used for visual question answering. ```python # Load dataset - use validation split which has answers image_qa_dataset = load_dataset("lmms-lab/VQAv2", split="validation[:200]") # Build answer corpus all_answers = [] for item in image_qa_dataset: if item['answers'] is not None: # 添加安全检查 all_answers.extend([ans['answer'] for ans in item['answers']]) image_qa_corpus = [ans for ans, _ in Counter(all_answers).most_common(50)] print(f"Dataset size: {len(image_qa_dataset)}") print(f"Answer corpus size: {len(image_qa_corpus)}") ``` -------------------------------- ### Load CIFAR-10 Dataset and Define Labels Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/embedding.ipynb Loads the first 100 images from the CIFAR-10 test set and defines the corresponding image class labels. This is a prerequisite for image classification tasks. ```python # Load dataset image_cls_dataset = load_dataset("cifar10", split="test[:100]") IMAGE_CLS_LABELS = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'] print(f"Dataset size: {len(image_cls_dataset)}") print(f"Labels: {IMAGE_CLS_LABELS}") ``` -------------------------------- ### Load Qwen3-VL Reranker Model Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/reranker.ipynb Loads the Qwen3-VL reranker model from a specified path. It configures model parameters such as max_length, max_frames, and fps. The code also sets up input and output directories and prints a success message upon loading. ```python sys.path.append(str(Path("..")..resolve())) from src.models.qwen3_vl_reranker import Qwen3VLReranker MODEL_PATH = r"./models/Qwen3-VL-Reranker-2B" # "your model path" INPUT_DIR = "./retrieval_results" OUTPUT_DIR = "./rerank_results" os.makedirs(OUTPUT_DIR, exist_ok=True) reranker = Qwen3VLReranker( model_name_or_path=MODEL_PATH, max_length=8192, max_frames=16, fps=1 ) print("Reranker model loaded successfully!") ``` -------------------------------- ### Prepare Input Data for Reranking in Python Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/reranker_vllm.ipynb This Python code defines a dictionary structure for input data, including instructions, a query with text, and a list of candidate documents. Each document can contain text, an image URL, or both. This structure is used for reranking tasks. ```python # Define query and candidate documents for reranking inputs = { "instruction": "Retrieve images or text relevant to the user's query.", "query": { "text": "A woman playing with her dog on a beach at sunset." }, "documents": [ { "text": "A woman shares a joyful moment with her golden retriever on a sun-drenched beach at sunset, as the dog offers its paw in a heartwarming display of companionship and trust." }, { "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg" }, { "text": "A woman shares a joyful moment with her golden retriever on a sun-drenched beach at sunset, as the dog offers its paw in a heartwarming display of companionship and trust.", "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg" } ] } print(f"Prepared query with {len(inputs['documents'])} candidate documents") ``` -------------------------------- ### Load and Process MS MARCO Dataset - Python Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/embedding.ipynb This Python code loads a subset of the MS MARCO dataset (v1.1) for passage retrieval and builds a unique corpus from the passages. It prints the dataset size and the final corpus size. ```python # Load dataset text_ret_dataset = load_dataset("ms_marco", "v1.1", split="validation[:100]") # Build corpus text_ret_corpus = [] for item in text_ret_dataset: for passage in item['passages']['passage_text']: if passage not in text_ret_corpus: text_ret_corpus.append(passage) print(f"Dataset size: {len(text_ret_dataset)}") print(f"Corpus size: {len(text_ret_corpus)}") ``` -------------------------------- ### Download Qwen3-VL Embedding and Reranker Scripts Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/Qwen3VL_Multimodal_RAG.ipynb Downloads the Python scripts for Qwen3-VL embedding and reranking models from Hugging Face. These scripts are essential for preparing the models and integrating them into the RAG pipeline. The downloaded scripts are then moved into a 'scripts' directory. ```bash !wget https://huggingface.co/Qwen/Qwen3-VL-Embedding-8B/resolve/main/scripts/qwen3_vl_embedding.py !wget https://huggingface.co/Qwen/Qwen3-VL-Reranker-8B/resolve/main/scripts/qwen3_vl_reranker.py !mkdir scripts !mv qwen3_vl_embedding.py scripts/ !mv qwen3_vl_reranker.py scripts/ ``` -------------------------------- ### Load and Prepare SQuAD Dataset for QA (Python) Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/embedding.ipynb This code snippet loads the SQuAD validation dataset and extracts unique contexts to form a corpus for question answering. It prints the dataset size and the number of unique contexts found. ```python # Load dataset text_qa_dataset = load_dataset("squad", split="validation[:100]") # Build corpus of unique contexts text_qa_corpus = list(set(text_qa_dataset["context"])) print(f"Dataset size: {len(text_qa_dataset)}") print(f"Unique contexts: {len(text_qa_corpus)}") ``` -------------------------------- ### Select and Format Queries for Text QA (Python) Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/embedding.ipynb This code selects specific queries from the SQuAD dataset and formats them for question answering. It extracts the question, answer, ground truth context, and the index of the ground truth context within the corpus. ```python # Select 2 queries text_qa_query_indices = [0, 99] text_qa_queries = [{ "question": text_qa_dataset[i]["question"], "answer": text_qa_dataset[i]["answers"]["text"] [0], "gt_context": text_qa_dataset[i]["context"], "gt_idx": text_qa_corpus.index(text_qa_dataset[i]["context"]) } for i in text_qa_query_indices] print("Selected queries:") for i, q in enumerate(text_qa_queries): print(f" {i+1}. Q: {q['question']}") print(f" A: {q['answer']}") ``` -------------------------------- ### Select Queries for Image Retrieval (Python) Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/embedding.ipynb This code selects a specified number of queries from the image captions and pairs them with their ground truth indices. It prepares the queries for the retrieval process. The output is a list of dictionaries, each containing a caption and its corresponding ground truth index. ```python # Select 2 queries image_ret_query_indices = [0, 9] image_ret_queries = [{ "caption": image_ret_captions[i], "gt_idx": i } for i in image_ret_query_indices] print("Selected queries:") for i, q in enumerate(image_ret_queries): print(f" {i+1}. {q['caption']}") ``` -------------------------------- ### Load Text Retrieval Results for Reranking Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/reranker.ipynb Loads the pre-generated text retrieval results from a pickle file. It extracts the text corpus and embedding results, then prints the number of queries and corpus size for verification. This step is essential before performing text reranking. ```python # Load embedding results text_path = os.path.join(INPUT_DIR, "text_retrieval_results.pkl") text_data = safe_load_pkl(text_path) text_corpus = text_data['corpus'] text_emb_results = text_data['results'] print(f"Loaded {len(text_emb_results)} text queries for reranking") print(f"Corpus size: {len(text_corpus)}") ``` -------------------------------- ### Initialize Qwen3-VL-Embedding Model and Prepare Inputs (Python) Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/embedding_vllm.ipynb This Python code initializes the Qwen3-VL-Embedding model using vLLM with specified parameters. It then defines a list of sample inputs, each containing text and optionally an image URL or path, demonstrating how to prepare data for multimodal inference. ```python # Initialize model llm = LLM( model="Qwen/Qwen3-VL-Embedding-2B", runner="pooling", dtype='bfloat16', trust_remote_code=True, ) # Prepare input samples inputs = [ { "text": "A woman playing with her dog on a beach at sunset.", "instruction": "Retrieve images or text relevant to the user's query.", }, { "text": "A woman shares a joyful moment with her golden retriever on a sun-drenched beach at sunset, as the dog offers its paw in a heartwarming display of companionship and trust." }, { "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg" }, { "text": "A woman shares a joyful moment with her golden retriever on a sun-drenched beach at sunset, as the dog offers its paw in a heartwarming display of companionship and trust.", "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg" } ] print(f"Prepared {len(inputs)} input samples") ``` -------------------------------- ### Calculate Similarity Matrix using Python Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/embedding_vllm.ipynb This code calculates the similarity matrix between embedding vectors using matrix multiplication. It takes the embeddings array, computes its transpose, and then performs the dot product to get the similarity scores. The resulting matrix is then printed. This requires the embeddings to be a NumPy array. ```python # Calculate similarity matrix similarity_scores = embeddings @ embeddings.T print("Similarity Score Matrix:") print(similarity_scores) ``` -------------------------------- ### Prepare vLLM Inputs for Qwen3-VL Embedding Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/embedding_vllm.ipynb This Python code snippet demonstrates how to prepare inputs for the vLLM model, specifically for the Qwen3-VL Embedding. It iterates through a list of inputs, applies a preparation function, and then prints a preview of the first processed prompt. This is crucial for ensuring data is in the correct format for the vLLM engine. ```python vllm_inputs = [prepare_vllm_inputs(inp, llm) for inp in inputs] print("Input conversion completed!") print(f"\nPreview of the first input prompt:") print(vllm_inputs[0]["prompt"][:200] + "...") ``` -------------------------------- ### Missing Package Notification for Rich Notebook Output Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/embedding_vllm.ipynb This warning message informs the user about a missing package, `ipywidgets`, which is required for rich notebook output. It provides instructions to install the package using pip and suggests restarting the notebook server for the changes to take effect. This is a common dependency issue in interactive Python environments. ```log 2026-01-16 07:47:30,017 INFO util.py:154 -- Missing packages: ['ipywidgets']. Run `pip install -U ipywidgets`, then restart the notebook server for rich notebook output. ``` -------------------------------- ### vLLM Engine Status and Configuration Details Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/embedding_vllm.ipynb This log entry provides a comprehensive overview of the vLLM engine's status and its detailed configuration. It includes information about the engine version, model and tokenizer paths, parallelism settings, data types, sequence lengths, and various compilation and scheduling configurations such as chunked prefill and asynchronous scheduling. This is useful for debugging and performance tuning. ```log INFO 01-16 07:47:33 [core.py:96] Initializing a V1 LLM engine (v0.14.0rc2.dev90+gbcf2333cd) with config: model='/cpfs01/user/linqi.lmx/models/finetune/qwen/qwen3-vl/Qwen3-VL-Embedding/Qwen3-VL-Embedding-2B', speculative_config=None, tokenizer='/cpfs01/user/linqi.lmx/models/finetune/qwen/qwen3-vl/Qwen3-VL-Embedding/Qwen3-VL-Embedding-2B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/cpfs01/user/linqi.lmx/models/finetune/qwen/qwen3-vl/Qwen3-VL-Embedding/Qwen3-VL-Embedding-2B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=PoolerConfig(pooling_type=None, seq_pooling_type='LAST', tok_pooling_type='ALL', normalize=None, dimensions=None, enable_chunked_processing=None, max_embed_len=None, softmax=None, activation=None, use_activation=None, logit_bias=None, step_tag_id=None, returned_token_ids=None), compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [16384], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_con'} ``` -------------------------------- ### vLLM Engine Initialization and Configuration Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/embedding_vllm.ipynb This section displays detailed information logged during the initialization of the vLLM engine for the Qwen3-VL Embedding model. It includes configurations such as the model path, tokenizer settings, parallelism, data types, sequence length, and compilation settings like `cudagraph_mode` and `inductor_compile_config`. This information is vital for understanding the runtime environment and potential performance optimizations. ```log INFO 01-16 07:47:29 [utils.py:267] non-default args: {'runner': 'pooling', 'trust_remote_code': True, 'dtype': 'bfloat16', 'disable_log_stats': True, 'model': '/cpfs01/user/linqi.lmx/models/finetune/qwen/qwen3-vl/Qwen3-VL-Embedding/Qwen3-VL-Embedding-2B' } INFO 01-16 07:47:29 [model.py:859] Resolved `--convert auto` to `--convert embed`. Pass the value explicitly to silence this message. INFO 01-16 07:47:29 [model.py:530] Resolved architecture: Qwen3VLForConditionalGeneration INFO 01-16 07:47:29 [model.py:1547] Using max model len 262144 INFO 01-16 07:47:30 [scheduler.py:229] Chunked prefill is enabled with max_num_batched_tokens=16384. INFO 01-16 07:47:30 [vllm.py:618] Asynchronous scheduling is enabled. INFO 01-16 07:47:30 [vllm.py:625] Disabling NCCL for DP synchronization when using async scheduling. WARNING 01-16 07:47:30 [vllm.py:732] Pooling models do not support full cudagraphs. Overriding cudagraph_mode to PIECEWISE. INFO 01-16 07:47:33 [core.py:96] Initializing a V1 LLM engine (v0.14.0rc2.dev90+gbcf2333cd) with config: model='/cpfs01/user/linqi.lmx/models/finetune/qwen/qwen3-vl/Qwen3-VL-Embedding/Qwen3-VL-Embedding-2B', speculative_config=None, tokenizer='/cpfs01/user/linqi.lmx/models/finetune/qwen/qwen3-vl/Qwen3-VL-Embedding/Qwen3-VL-Embedding-2B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/cpfs01/user/linqi.lmx/models/finetune/qwen/qwen3-vl/Qwen3-VL-Embedding/Qwen3-VL-Embedding-2B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=PoolerConfig(pooling_type=None, seq_pooling_type='LAST', tok_pooling_type='ALL', normalize=None, dimensions=None, enable_chunked_processing=None, max_embed_len=None, softmax=None, activation=None, use_activation=None, logit_bias=None, step_tag_id=None, returned_token_ids=None), compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [16384], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_con'} ``` -------------------------------- ### Qwen3-VL Embedding Model Initialization Parameters Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/README.md Shows the parameters available when initializing the Qwen3VLEmbedder model. These parameters control aspects like model path, context length, image/video pixel limits, frame sampling rates, and attention implementations for performance optimization. ```python Qwen3VLEmbedder( model_name_or_path="./models/Qwen3-VL-Embedding-2B", max_length=8192, # Default context length min_pixels=4096, # Minimum pixels for input images max_pixels=1843200, # Maximum pixels for input images (equivalent to 1280×1440 resolution) total_pixels=7864320, # Maximum total pixels for input videos (multiplied by 2 in model) # For a 16-frame video, each frame can have up to 983040 pixels (1280×768 resolution) fps=1.0, # Default sampling frame rate for video files (frames per second) max_frames=64, # Maximum number of frames for video input torch_dtype=torch.bfloat16, attn_implementation="flash_attention_2" ) ``` -------------------------------- ### Encode Image Classification Labels Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/embedding.ipynb Generates embeddings for the predefined image class labels using the Qwen3-VL-Embedding model. These embeddings form the corpus for classification. ```python # Encode label corpus image_cls_corpus_inputs = [{"text": label} for label in IMAGE_CLS_LABELS] image_cls_corpus_embeddings = embedder.process(image_cls_corpus_inputs) print(f"Label embeddings shape: {image_cls_corpus_embeddings.shape}") ``` -------------------------------- ### Prepare and Generate Answer with Qwen3-VL in Python Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/Qwen3VL_Multimodal_RAG.ipynb This snippet prepares the input messages for the Qwen3-VL model, including an image and a text prompt. It then applies a chat template, moves the inputs to the model's device, and generates an answer. Finally, it decodes and prints the generated text. Dependencies include the Qwen3-VL model, processor, and torch. ```python print(f"Generating answer for: {query_for_reranking['text']}") print(f"Using retrieved page: {best_page_idx + 1}") messages = [ { "role": "user", "content": [ { "type": "image", "image": f"temp_page_{best_page_idx}.png", }, { "type": "text", "text": f"Based on this page from a climate change document, please answer the following question: {query_for_reranking['text']}" }, ], } ] inputs = processor.apply_chat_template( messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt" ) inputs = inputs.to(vlm_model.device) generated_ids = vlm_model.generate(**inputs, max_new_tokens=256) generated_ids_trimmed = [ out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids) ] output_text = processor.batch_decode( generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False ) print(f"\nQuery: {query_for_reranking['text']}") print(f"\nAnswer:\n{output_text[0]}") ``` -------------------------------- ### Encode Queries and Retrieve Answers for VQA Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/embedding.ipynb Encodes the selected image-question pairs for visual question answering. It then retrieves the most relevant answers from the answer corpus based on the embeddings. ```python # Encode queries and retrieve image_qa_query_inputs = [{ "image": q['image'], "text": q['question'], "instruction": "Find the answer to this question about the image." } for q in image_qa_queries] image_qa_query_embeddings = embedder.process(image_qa_query_inputs) image_qa_results = retrieve_topk(image_qa_query_embeddings, image_qa_corpus_embeddings, TOP_K) ``` -------------------------------- ### Encode Visual Question Answering Answer Corpus Source: https://github.com/qwenlm/qwen3-vl-embedding/blob/main/examples/embedding.ipynb Generates embeddings for the constructed answer corpus using the Qwen3-VL-Embedding model. These embeddings represent the possible answers to visual questions. ```python # Encode answer corpus image_qa_corpus_inputs = [{"text": ans} for ans in image_qa_corpus] image_qa_corpus_embeddings = encode_corpus(image_qa_corpus_inputs, batch_size=16) print(f"Answer embeddings shape: {image_qa_corpus_embeddings.shape}") ```