### Custom Quantizer Example for Qwen2 VL Model with AutoAWQ

Source: https://github.com/casper-hansen/autoawq/blob/main/docs/examples.md

Demonstrates using a custom quantizer class, extending `AwqQuantizer`, to quantize the Qwen2 VL model. This approach allows for specialized processing of multimodal examples, enabling effective quantization for vision-language tasks.

```python
import torch
import torch.nn as nn

from awq import AutoAWQForCausalLM
from awq.utils.qwen_vl_utils import process_vision_info
from awq.quantize.quantizer import AwqQuantizer, clear_memory, get_best_device

# Specify paths and hyperparameters for quantization
model_path = "Qwen/Qwen2-VL-7B-Instruct"
quant_path = "qwen2-vl-7b-instruct"
quant_config = {"zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM"}

model = AutoAWQForCausalLM.from_pretrained(
    model_path, attn_implementation="flash_attention_2"
)

# We define our own quantizer by extending the AwqQuantizer.
# The main difference is in how the samples are processed when

```

--------------------------------

### Install AutoAWQ via pip

Source: https://context7.com/casper-hansen/autoawq/llms.txt

Commands to install the AutoAWQ library with optional kernel extensions for CUDA or Intel CPU/XPU support.

```bash
# Default installation with Triton-based inference
pip install autoawq

# With optimized CUDA kernels (requires matching torch version)
pip install autoawq[kernels]

# For Intel CPU/XPU optimized performance
pip install autoawq[cpu]
```

--------------------------------

### Qwen2 VL Inference with AutoAWQ

Source: https://github.com/casper-hansen/autoawq/blob/main/docs/examples.md

Provides an example for running inference with the Qwen2 VL (Vision-Language) model quantized with AutoAWQ. It shows how to load the model and processor, and set up a conversational structure with image input.

```python
from awq import AutoAWQForCausalLM
from awq.utils.qwen_vl_utils import process_vision_info
from transformers import AutoProcessor, TextStreamer

# Load model
quant_path = "Qwen/Qwen2-VL-7B-Instruct-AWQ"
model = AutoAWQForCausalLM.from_quantized(quant_path)
processor = AutoProcessor.from_pretrained(quant_path)
streamer = TextStreamer(processor, skip_prompt=True)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]
```

--------------------------------

### Inference with AWQ Model using vLLM

Source: https://github.com/casper-hansen/autoawq/blob/main/docs/examples.md

Runs inference with an AWQ model using the vLLM library for efficient serving. This example demonstrates setting up the AsyncLLMEngine, defining sampling parameters, and handling asynchronous generation with streaming output.

```python
import asyncio
from transformers import AutoTokenizer, PreTrainedTokenizer
from vllm import AsyncLLMEngine, SamplingParams, AsyncEngineArgs

model_path = "casperhansen/mixtral-instruct-awq"

# prompting
prompt = "You're standing on the surface of the Earth. "
         "You walk one mile south, one mile west and one mile north. "
         "You end up exactly where you started. Where are you?",

prompt_template = "[INST] {prompt} [/INST]"

# sampling params
sampling_params = SamplingParams(
    repetition_penalty=1.1,
    temperature=0.8,
    max_tokens=512
)

# tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_path)

# async engine args for streaming
engine_args = AsyncEngineArgs(
    model=model_path,
    quantization="awq",
    dtype="float16",
    max_model_len=512,
    enforce_eager=True,
    disable_log_requests=True,
    disable_log_stats=True,
)

async def generate(model: AsyncLLMEngine, tokenizer: PreTrainedTokenizer):
    tokens = tokenizer(prompt_template.format(prompt=prompt)).input_ids

    outputs = model.generate(
        prompt=prompt,
        sampling_params=sampling_params,
        request_id=1,
        prompt_token_ids=tokens,
    )

    print("\n** Starting generation!\n")
    last_index = 0

    async for output in outputs:
        print(output.outputs[0].text[last_index:], end="", flush=True)
        last_index = len(output.outputs[0].text)
    
    print("\n\n** Finished generation!\n")

if __name__ == '__main__':
    model = AsyncLLMEngine.from_engine_args(engine_args)
    asyncio.run(generate(model, tokenizer))
```

--------------------------------

### Vision-Language Model Inference (LLaVa)

Source: https://context7.com/casper-hansen/autoawq/llms.txt

Run inference on multimodal vision-language models like LLaVa using quantized weights. This example shows how to load a quantized LLaVa model and generate descriptions for images.

```APIDOC
## Vision-Language Model Inference (LLaVa)

Run inference on multimodal vision-language models like LLaVa using quantized weights.

### Method
```python
model.generate(
    **inputs,
    max_new_tokens=512,
    streamer=streamer
)
```

### Parameters
#### Request Body
- **inputs** (dict) - Processed inputs containing both text prompt and image data, typically from the model's processor.
- **max_new_tokens** (int) - The maximum number of new tokens to generate.
- **streamer** (transformers.TextStreamer) - An optional streamer object to output tokens as they are generated.

### Request Example
```python
import torch
import requests
from PIL import Image
from awq import AutoAWQForCausalLM
from transformers import AutoProcessor, TextStreamer

# Load quantized LLaVa model
quant_path = "casperhansen/llama3-llava-next-8b-awq"
model = AutoAWQForCausalLM.from_quantized(quant_path)
processor = AutoProcessor.from_pretrained(quant_path)
streamer = TextStreamer(processor, skip_prompt=True)

# Define multimodal prompt
prompt = """
<|im_start|>system
Answer the questions.<|im_end|>
<|im_start|>user
<image>
What is shown in this image?<|im_end|>
<|im_start|>assistant
"""

# Load image from URL
url = "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"
image = Image.open(requests.get(url, stream=True).raw)

# Process inputs
inputs = processor(prompt, image, return_tensors='pt').to(0, torch.float16)

# Generate description
generation_output = model.generate(
    **inputs,
    max_new_tokens=512,
    streamer=streamer
)
```

### Response
Returns the generated text describing the image. If a `streamer` is provided, output will be streamed to the console.
```

--------------------------------

### Configure AutoAWQ for CPU Inference

Source: https://github.com/casper-hansen/autoawq/blob/main/docs/index.md

This snippet shows how to load a quantized model for CPU inference using the Intel Extension for PyTorch (IPEX). It requires the ipex library to be installed and the use_ipex flag enabled.

```python
model = AutoAWQForCausalLM.from_quantized(
    ...,
    use_ipex=True
)
```

--------------------------------

### Quantization for Coding Models

Source: https://github.com/casper-hansen/autoawq/blob/main/docs/examples.md

Example of quantizing a coding-specific model, DeepSeek-Coder-V2-Lite-Instruct, using AutoAWQ. This snippet sets up the model path, quantization configuration, and prepares for the quantization process.

```python
from tqdm import tqdm
from datasets import load_dataset
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = 'deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct'
quant_path = 'deepseek-coder-v2-lite-instruct-awq'
quant_config = { "zero_point": True, "q_group_size": 64, "w_bit": 4, "version": "GEMM" }


```

--------------------------------

### Generate Text with Quantized Models

Source: https://context7.com/casper-hansen/autoawq/llms.txt

Provides an example of running inference on a quantized model with streaming support. It utilizes Hugging Face chat templates and the AutoAWQ inference engine.

```python
import torch
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer, TextStreamer
from awq.utils.utils import get_best_device

device = get_best_device()
model_id = "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4"

model = AutoAWQForCausalLM.from_quantized(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain quantum computing in simple terms."},
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(device)

outputs = model.generate(
    **inputs,
    do_sample=True,
    max_new_tokens=256,
    temperature=0.7,
    top_p=0.9,
    streamer=streamer,
)
```

--------------------------------

### Quantization with Custom Calibration Data

Source: https://context7.com/casper-hansen/autoawq/llms.txt

Quantize models using custom calibration datasets for domain-specific optimization, especially useful for specialized models. This example demonstrates loading custom data and quantizing with memory-efficient settings for long-context models.

```APIDOC
## Quantization with Custom Calibration Data

Quantize using custom calibration datasets for domain-specific optimization. This is particularly useful for specialized models like coding assistants or long-context applications.

### Method
```python
model.quantize(
    tokenizer,
    quant_config=quant_config,
    calib_data=load_cosmopedia(),
    n_parallel_calib_samples=32,    # Process samples in batches to save GPU memory
    max_calib_samples=128,          # Total calibration samples
    max_calib_seq_len=4096          # Support long context
)
```

### Parameters
#### Request Body
- **tokenizer** (transformers.AutoTokenizer) - The tokenizer for the model.
- **quant_config** (dict) - Configuration for quantization (e.g., `{"zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM"}`).
- **calib_data** (list) - A list of text samples for calibration.
- **n_parallel_calib_samples** (int) - Number of samples to process in parallel during calibration to save GPU memory.
- **max_calib_samples** (int) - Total number of calibration samples to use.
- **max_calib_seq_len** (int) - Maximum sequence length supported for calibration.

### Request Example
```python
from datasets import load_dataset
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = 'Qwen/Qwen2-7B-Instruct'
quant_path = 'qwen2-7b-awq'
quant_config = {"zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM"}

model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

# Custom calibration data loader for long-context models
def load_cosmopedia():
    data = load_dataset('HuggingFaceTB/cosmopedia-100k', split="train")
    data = data.filter(lambda x: x["text_token_length"] >= 2048)
    return [text for text in data["text"]]

model.quantize(
    tokenizer,
    quant_config=quant_config,
    calib_data=load_cosmopedia(),
    n_parallel_calib_samples=32,
    max_calib_samples=128,
    max_calib_seq_len=4096
)

model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
```

### Response
This method does not return a value directly but saves the quantized model and tokenizer to disk.
```

--------------------------------

### Inference with AWQ Model using Transformers

Source: https://github.com/casper-hansen/autoawq/blob/main/docs/examples.md

Performs text generation using an AWQ model loaded via Hugging Face Transformers. It includes tokenization, model loading with specific dtypes, and generation using a TextStreamer for real-time output. Requires installation from a specific PR.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

# NOTE: Must install from PR until merged
# pip install --upgrade git+https://github.com/younesbelkada/transformers.git@add-awq
model_id = "casperhansen/mistral-7b-instruct-v0.1-awq"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    torch_dtype=torch.float16, 
    low_cpu_mem_usage=True,
)
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

# Convert prompt to tokens
text = "[INST] What are the basic steps to use the Huggingface transformers library? [/INST]"

tokens = tokenizer(
    text, 
    return_tensors='pt'
).input_ids.cuda()

# Generate output
generation_output = model.generate(
    tokens, 
    streamer=streamer,
    max_new_tokens=512
)
```

--------------------------------

### Load Pretrained Model for Quantization

Source: https://context7.com/casper-hansen/autoawq/llms.txt

Initializes a model from a Hugging Face path or local directory in FP16 format, preparing it for the AWQ quantization process.

```python
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = 'mistralai/Mistral-7B-Instruct-v0.2'

# Load pretrained model for quantization
model = AutoAWQForCausalLM.from_pretrained(
    model_path,
    torch_dtype="auto",
    trust_remote_code=True,
    safetensors=True,
    device_map="auto",
    low_cpu_mem_usage=True
)

tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

print(f"Model loaded: {model.model_type}")
print(f"Is quantized: {model.is_quantized")
```

--------------------------------

### Transformers Integration

Source: https://context7.com/casper-hansen/autoawq/llms.txt

Load AWQ models directly through Hugging Face Transformers AutoModelForCausalLM when AutoAWQ is installed.

```APIDOC
## Transformers Integration

Load AWQ models directly through Hugging Face Transformers AutoModelForCausalLM when AutoAWQ is installed.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

model_id = "casperhansen/mistral-7b-instruct-v0.1-awq"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    low_cpu_mem_usage=True,
    device_map="auto",
)
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

# Generate text
text = "[INST] What are the basic steps to use the Huggingface transformers library? [/INST]"
tokens = tokenizer(text, return_tensors='pt').input_ids.cuda()

generation_output = model.generate(
    tokens,
    streamer=streamer,
    max_new_tokens=512
)
```
```

--------------------------------

### Load AWQ Models with vLLM for Production Serving

Source: https://context7.com/casper-hansen/autoawq/llms.txt

Shows how to load AWQ models using vLLM for high-throughput production serving. It configures the vLLM engine with AWQ quantization and demonstrates asynchronous text generation with streaming support.

```python
import asyncio
from transformers import AutoTokenizer
from vllm import AsyncLLMEngine, SamplingParams, AsyncEngineArgs

model_path = "casperhansen/mixtral-instruct-awq"

# Sampling parameters
sampling_params = SamplingParams(
    repetition_penalty=1.1,
    temperature=0.8,
    max_tokens=512
)

# Engine configuration for AWQ models
engine_args = AsyncEngineArgs(
    model=model_path,
    quantization="awq",
    dtype="float16",
    max_model_len=4096,
    enforce_eager=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_path)

async def generate(model, prompt):
    prompt_template = "[INST] {prompt} [/INST]"
    formatted_prompt = prompt_template.format(prompt=prompt)
    tokens = tokenizer(formatted_prompt).input_ids

    outputs = model.generate(
        prompt=formatted_prompt,
        sampling_params=sampling_params,
        request_id="1",
        prompt_token_ids=tokens,
    )

    async for output in outputs:
        print(output.outputs[0].text, end="", flush=True)

if __name__ == '__main__':
    model = AsyncLLMEngine.from_engine_args(engine_args)
    asyncio.run(generate(model, "Explain the theory of relativity."))
```

--------------------------------

### Quantization with Custom Datasets (Dolly/WikiText)

Source: https://github.com/casper-hansen/autoawq/blob/main/docs/examples.md

Shows how to quantize a model using custom calibration data, specifically loading from the Dolly or WikiText datasets. It includes functions to load and preprocess these datasets for quantization.

```python
from datasets import load_dataset
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = 'lmsys/vicuna-7b-v1.5'
quant_path = 'vicuna-7b-v1.5-awq'
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }

# Load model
model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

# Define data loading methods
def load_dolly():
    data = load_dataset('databricks/databricks-dolly-15k', split="train")

    # concatenate data
    def concatenate_data(x):
        return {"text": x['instruction'] + '\n' + x['context'] + '\n' + x['response']}
    
    concatenated = data.map(concatenate_data)
    return [text for text in concatenated["text"]]

def load_wikitext():
    data = load_dataset('wikitext', 'wikitext-2-raw-v1', split="train")
    return [text for text in data["text"] if text.strip() != '' and len(text.split(' ')) > 20]

# Quantize
model.quantize(tokenizer, quant_config=quant_config, calib_data=load_wikitext())

# Save quantized model
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

print(f'Model is quantized and saved at "{quant_path}"')

```

--------------------------------

### Configure Benchmark and Run Inference with AutoAWQ

Source: https://context7.com/casper-hansen/autoawq/llms.txt

This Python code snippet demonstrates how to set up benchmark parameters such as context length and the number of new tokens to generate. It utilizes `torch` for tensor operations and `GenerationConfig` for generation parameters. A custom `TimeMeasuringLogitsProcessor` is employed to log prefill and decode durations, which are then used to calculate and print inference speeds in tokens/s and maximum allocated GPU memory.

```python
context_length = 512
n_generate = 128
input_ids = torch.randint(0, tokenizer.vocab_size, (1, context_length)).cuda()

generation_config = GenerationConfig(
    min_new_tokens=n_generate,
    max_new_tokens=n_generate,
    use_cache=True,
)

time_processor = TimeMeasuringLogitsProcessor()
model.generate(
    input_ids,
    generation_config=generation_config,
    logits_processor=LogitsProcessorList([time_processor]),
)

prefill_time = time_processor.get_prefill_duration()
decode_times = time_processor.get_decode_durations()

print(f"Prefill: {context_length / prefill_time:.2f} tokens/s")
print(f"Decode: {1 / (sum(decode_times) / len(decode_times)):.2f} tokens/s")
print(f"Memory: {torch.cuda.max_memory_allocated() / 1024**3:.2f} GB")
```

--------------------------------

### Run Vision-Language Model Inference

Source: https://context7.com/casper-hansen/autoawq/llms.txt

Demonstrates multimodal inference using quantized vision-language models like LLaVa. It covers image loading, processor input preparation, and generation.

```python
import torch
import requests
from PIL import Image
from awq import AutoAWQForCausalLM
from transformers import AutoProcessor, TextStreamer

quant_path = "casperhansen/llama3-llava-next-8b-awq"
model = AutoAWQForCausalLM.from_quantized(quant_path)
processor = AutoProcessor.from_pretrained(quant_path)
streamer = TextStreamer(processor, skip_prompt=True)

prompt = """<|im_start|>system\nAnswer the questions.<|im_end|>\n<|im_start|>user\n<image>\nWhat is shown in this image?<|im_end|>\n<|im_start|>assistant\n"""

url = "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(prompt, image, return_tensors='pt').to(0, torch.float16)

generation_output = model.generate(
    **inputs,
    max_new_tokens=512,
    streamer=streamer
)
```

--------------------------------

### Load AWQ Models with Hugging Face Transformers

Source: https://context7.com/casper-hansen/autoawq/llms.txt

Demonstrates how to load AWQ-quantized models directly using Hugging Face Transformers' AutoModelForCausalLM. This allows for easy integration and inference with AWQ models within the Transformers ecosystem.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

model_id = "casperhansen/mistral-7b-instruct-v0.1-awq"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    low_cpu_mem_usage=True,
    device_map="auto",
)
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

# Generate text
text = "[INST] What are the basic steps to use the Huggingface transformers library? [/INST]"
tokens = tokenizer(text, return_tensors='pt').input_ids.cuda()

generation_output = model.generate(
    tokens,
    streamer=streamer,
    max_new_tokens=512
)
```

--------------------------------

### GPU Inference with AutoAWQ Mistral Model

Source: https://github.com/casper-hansen/autoawq/blob/main/docs/examples.md

This Python code snippet shows how to perform inference with a quantized Mistral-7B-Instruct model using AutoAWQ. It loads the model with `fuse_layers=True` for potential speedup and uses a TextStreamer for output. The example includes prompt formatting and generation using CUDA.

```python
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer, TextStreamer

quant_path = "TheBloke/Mistral-7B-Instruct-v0.2-AWQ"

# Load model
model = AutoAWQForCausalLM.from_quantized(quant_path, fuse_layers=True)
tokenizer = AutoTokenizer.from_pretrained(quant_path, trust_remote_code=True)
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

# Convert prompt to tokens
prompt_template = "[INST] {prompt} [/INST]"

prompt = "You're standing on the surface of the Earth. "
        "You walk one mile south, one mile west and one mile north. "
        "You end up exactly where you started. Where are you?"

tokens = tokenizer(
    prompt_template.format(prompt=prompt), 
    return_tensors='pt'
).input_ids.cuda()

# Generate output
generation_output = model.generate(
    tokens, 
    streamer=streamer,
    max_new_tokens=512
)

```

--------------------------------

### Quantize a Causal Language Model with AutoAWQ

Source: https://github.com/casper-hansen/autoawq/blob/main/README.md

This snippet demonstrates how to load a pre-trained model, apply AWQ quantization with specific configuration parameters, and save the resulting quantized model to disk.

```python
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = 'mistralai/Mistral-7B-Instruct-v0.2'
quant_path = 'mistral-instruct-v0.2-awq'
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }

# Load model
model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

# Quantize
model.quantize(tokenizer, quant_config=quant_config)

# Save quantized model
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

print(f'Model is quantized and saved at "{quant_path}"') 
```

--------------------------------

### Perform Multimodal Inference with AutoAWQ

Source: https://github.com/casper-hansen/autoawq/blob/main/docs/examples.md

This snippet demonstrates how to prepare text, image, and video inputs using a processor and pass them to the AutoAWQ model for generation. It includes moving inputs to the GPU and setting generation parameters like max_new_tokens.

```python
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to("cuda")

generation_output = model.generate(
    **inputs,
    max_new_tokens=512,
    streamer=streamer
)
```

--------------------------------

### Prepare Calibration Dataset for Vision-Language Models

Source: https://github.com/casper-hansen/autoawq/blob/main/docs/examples.md

Demonstrates how to structure chat-based datasets for vision-language models. It includes a helper function to load and format caption datasets into the expected list-of-dictionaries format.

```python
def prepare_dataset(n_sample: int = 8) -> list[list[dict]]:
    from datasets import load_dataset
    dataset = load_dataset("laion/220k-GPT4Vision-captions-from-LIVIS", split=f"train[:{n_sample}]")
    return [
        [
            {
                "role": "user",
                "content": [
                    {"type": "image", "image": sample["url"]},
                    {"type": "text", "text": "generate a caption for this image"},
                ],
            },
            {"role": "assistant", "content": sample["caption"]},
        ]
        for sample in dataset
    ]
```

--------------------------------

### Execute Quantization and Save Model

Source: https://github.com/casper-hansen/autoawq/blob/main/docs/examples.md

Processes the prepared dataset into tensors using the model's processor, runs the quantization process, and saves the quantized model to disk using safetensors.

```python
text = model.processor.apply_chat_template(dataset, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(dataset)
inputs = model.processor(text=text, images=image_inputs, videos=video_inputs, padding=True, return_tensors="pt")
model.quantize(calib_data=inputs, quant_config=quant_config, quantizer_cls=Qwen2VLAwqQuantizer)
model.model.config.use_cache = model.model.generation_config.use_cache = True
model.save_quantized(quant_path, safetensors=True, shard_size="4GB")
```

--------------------------------

### Run Multimodal Inference with Qwen2-VL

Source: https://context7.com/casper-hansen/autoawq/llms.txt

Demonstrates how to load a quantized Qwen2-VL model and process multimodal inputs including images and text. It utilizes the AutoAWQ library alongside Hugging Face transformers to generate streaming responses.

```python
from awq import AutoAWQForCausalLM
from awq.utils.qwen_vl_utils import process_vision_info
from transformers import AutoProcessor, TextStreamer

quant_path = "Qwen/Qwen2-VL-7B-Instruct-AWQ"
model = AutoAWQForCausalLM.from_quantized(quant_path)
processor = AutoProcessor.from_pretrained(quant_path)
streamer = TextStreamer(processor, skip_prompt=True)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
            },
            {"type": "text", "text": "Describe this image in detail."},
        ],
    }
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
).to("cuda")

generation_output = model.generate(
    **inputs,
    max_new_tokens=512,
    streamer=streamer
)
```

--------------------------------

### Basic Quantization with AutoAWQ

Source: https://github.com/casper-hansen/autoawq/blob/main/docs/examples.md

Demonstrates the basic usage of AutoAWQ for quantizing a causal language model to 4-bit precision. It loads a pre-trained model and tokenizer, applies quantization with specified configuration, and saves the quantized model.

```python
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = 'mistralai/Mistral-7B-Instruct-v0.2'
quant_path = 'mistral-instruct-v0.2-awq'
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }

# Load model
model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

# Quantize
model.quantize(tokenizer, quant_config=quant_config)

# Save quantized model
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

print(f'Model is quantized and saved at "{quant_path}"')

```

--------------------------------

### Export AWQ Models to GGUF Format

Source: https://context7.com/casper-hansen/autoawq/llms.txt

Shows the process of applying AWQ scales to a model without full quantization, saving it in a compatible format, and using llama.cpp scripts to convert it into GGUF for efficient inference.

```python
import os
import subprocess
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = 'mistralai/Mistral-7B-v0.1'
quant_path = 'mistral-awq-gguf'
llama_cpp_path = '/workspace/llama.cpp'

quant_config = {"zero_point": True, "q_group_size": 128, "w_bit": 6, "version": "GEMM"}

model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

model.quantize(
    tokenizer,
    quant_config=quant_config,
    export_compatible=True
)

model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

convert_cmd_path = os.path.join(llama_cpp_path, "convert.py")
quantize_cmd_path = os.path.join(llama_cpp_path, "quantize")

subprocess.run([
    f"python {convert_cmd_path} {quant_path} --outfile {quant_path}/model.gguf"
], shell=True, check=True)

subprocess.run([
    f"{quantize_cmd_path} {quant_path}/model.gguf {quant_path}/model_q4_K_M.gguf q4_K_M"
], shell=True, check=True)
```

--------------------------------

### Configure AutoAWQ for AMD GPU Inference

Source: https://github.com/casper-hansen/autoawq/blob/main/docs/index.md

This snippet demonstrates how to initialize a quantized model for AMD GPUs. It disables fused layers and enables ExLlamaV2 kernels to ensure compatibility.

```python
model = AutoAWQForCausalLM.from_quantized(
    ...,
    fuse_layers=False,
    use_exllama_v2=True
)
```

--------------------------------

### Quantize and Save MiniCPM3 Model with Custom Quantizer

Source: https://github.com/casper-hansen/autoawq/blob/main/docs/examples.md

This Python script demonstrates how to load a pre-trained MiniCPM3 model, apply the custom CPM3AwqQuantizer with a specified quantization configuration, and save the quantized model and tokenizer. It utilizes the AutoAWQForCausalLM and AutoTokenizer classes from the awq library.

```python
model_path = 'openbmb/MiniCPM3-4B'
quant_path = 'minicpm3-4b-awq'
quant_config = { "zero_point": True, "q_group_size": 64, "w_bit": 4, "version": "GEMM" }

# Load model
model = AutoAWQForCausalLM.from_pretrained(model_path, safetensors=False)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

# Quantize
model.quantize(tokenizer, quant_config=quant_config, quantizer_cls=CPM3AwqQuantizer)

# Save quantized model
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

print(f'Model is quantized and saved at "{quant_path}"')

```

--------------------------------

### Save Quantized Model to Disk

Source: https://context7.com/casper-hansen/autoawq/llms.txt

Shows how to save a quantized model using safetensors or PyTorch formats. It supports sharding for large models to facilitate easier storage and distribution.

```python
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = 'Qwen/Qwen2.5-14B-Instruct'
quant_path = 'Qwen2.5-14B-Instruct-awq'
quant_config = {"zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM"}

model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model.quantize(tokenizer, quant_config=quant_config)

model.save_quantized(
    quant_path,
    safetensors=True,
    shard_size="5GB"
)
tokenizer.save_pretrained(quant_path)
```

--------------------------------

### LLaVa Multimodal Inference with AutoAWQ

Source: https://github.com/casper-hansen/autoawq/blob/main/docs/examples.md

Demonstrates how to perform inference with LLaVa, a multimodal model, using AutoAWQ. It involves loading both the AWQ model and a processor, defining a prompt with an image placeholder, and processing an image for input.

```python
import torch
import requests
from PIL import Image
from awq import AutoAWQForCausalLM
from transformers import AutoProcessor, TextStreamer

# Load model
quant_path = "casperhansen/llama3-llava-next-8b-awq"
model = AutoAWQForCausalLM.from_quantized(quant_path)
processor = AutoProcessor.from_pretrained(quant_path)
streamer = TextStreamer(processor, skip_prompt=True)

# Define prompt
prompt = """
<|im_start|>system
Answer the questions.<|im_end|>
<|im_start|>user
<image>
What is shown in this image?<|im_end|>
<|im_start|>assistant
"

# Define image
url = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"
image = Image.open(requests.get(url, stream=True).raw)

# Load inputs
inputs = processor(prompt, image, return_tensors='pt').to(0, torch.float16)

generation_output = model.generate(
    **inputs,
    max_new_tokens=512,
    streamer=streamer
)
```

--------------------------------

### Implement Custom Vision-Language Quantizer

Source: https://context7.com/casper-hansen/autoawq/llms.txt

Extends the AwqQuantizer class to handle custom initialization logic for vision-language models. This is useful for capturing activation data from specific model layers during the calibration process.

```python
import torch
import torch.nn as nn
from awq import AutoAWQForCausalLM
from awq.utils.qwen_vl_utils import process_vision_info
from awq.quantize.quantizer import AwqQuantizer, clear_memory, get_best_device

class Qwen2VLAwqQuantizer(AwqQuantizer):
    def init_quant(self, n_samples=None, max_seq_len=None):
        modules = self.awq_model.get_model_layers(self.model)
        samples = self.calib_data
        inps = []
        layer_kwargs = {}
        best_device = get_best_device()

        modules[0] = modules[0].to(best_device)
        self.awq_model.move_embed(self.model, best_device)

        class Catcher(nn.Module):
            def __init__(self, module):
                super().__init__()
                self.module = module

            def forward(self, *args, **kwargs):
                if len(args) > 0:
                    hidden_states = args[0]
                    del args
                else:
                    first_key = list(kwargs.keys())[0]
                    hidden_states = kwargs.pop(first_key)
                inps.append(hidden_states)
                layer_kwargs.update(kwargs)
                raise ValueError

        modules[0] = Catcher(modules[0])
        for k, v in samples.items():
            if isinstance(v, (torch.Tensor, nn.Module)):
                samples[k] = v.to(best_device)
        try:
            self.model(**samples)
        except ValueError:
            pass
        modules[0] = modules[0].module

        del samples
        inps = inps[0]
        modules[0] = modules[0].cpu()
        self.awq_model.move_embed(self.model, "cpu")
        clear_memory()

        return modules, layer_kwargs, inps
```

--------------------------------

### vLLM Integration

Source: https://context7.com/casper-hansen/autoawq/llms.txt

Load AWQ models in vLLM for high-throughput production serving with async streaming support.

```APIDOC
## vLLM Integration

Load AWQ models in vLLM for high-throughput production serving with async streaming support.

```python
import asyncio
from transformers import AutoTokenizer
from vllm import AsyncLLMEngine, SamplingParams, AsyncEngineArgs

model_path = "casperhansen/mixtral-instruct-awq"

# Sampling parameters
sampling_params = SamplingParams(
    repetition_penalty=1.1,
    temperature=0.8,
    max_tokens=512
)

# Engine configuration for AWQ models
engine_args = AsyncEngineArgs(
    model=model_path,
    quantization="awq",
    dtype="float16",
    max_model_len=4096,
    enforce_eager=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_path)

async def generate(model, prompt):
    prompt_template = "[INST] {prompt} [/INST]"
    formatted_prompt = prompt_template.format(prompt=prompt)
    tokens = tokenizer(formatted_prompt).input_ids

    outputs = model.generate(
        prompt=formatted_prompt,
        sampling_params=sampling_params,
        request_id="1",
        prompt_token_ids=tokens,
    )

    async for output in outputs:
        print(output.outputs[0].text, end="", flush=True)

if __name__ == '__main__':
    model = AsyncLLMEngine.from_engine_args(engine_args)
    asyncio.run(generate(model, "Explain the theory of relativity."))
```
```

--------------------------------

### Prepare Multimodal Calibration Data

Source: https://context7.com/casper-hansen/autoawq/llms.txt

Prepares multimodal calibration data from the LAION dataset for use with vision-language models. It loads a specified number of samples, formats them into a chat-like structure with image and text content, and returns the processed dataset.

```python
from datasets import load_dataset
def prepare_dataset(n_sample=8):
    dataset = load_dataset("laion/220k-GPT4Vision-captions-from-LIVIS", split=f"train[:{n_sample}]")
    return [
        [
            {"role": "user", "content": [
                {"type": "image", "image": sample["url"]},
                {"type": "text", "text": "generate a caption for this image"},
            ]},
            {"role": "assistant", "content": sample["caption"]},
        ]
        for sample in dataset
    ]

dataset = prepare_dataset()
text = model.processor.apply_chat_template(dataset, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(dataset)
inputs = model.processor(text=text, images=image_inputs, videos=video_inputs, padding=True, return_tensors="pt")

# Quantize with custom quantizer
model.quantize(calib_data=inputs, quant_config=quant_config, quantizer_cls=Qwen2VLAwqQuantizer)
model.save_quantized(quant_path, safetensors=True, shard_size="4GB")
```

--------------------------------

### Quantize LLM with Custom Calibration Data

Source: https://context7.com/casper-hansen/autoawq/llms.txt

Demonstrates how to perform model quantization using a specific dataset for domain-specific optimization. It includes a custom data loader and memory-efficient configuration for long-context models.

```python
from datasets import load_dataset
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = 'Qwen/Qwen2-7B-Instruct'
quant_path = 'qwen2-7b-awq'
quant_config = {"zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM"}

model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

def load_cosmopedia():
    data = load_dataset('HuggingFaceTB/cosmopedia-100k', split="train")
    data = data.filter(lambda x: x["text_token_length"] >= 2048)
    return [text for text in data["text"]]

model.quantize(
    tokenizer,
    quant_config=quant_config,
    calib_data=load_cosmopedia(),
    n_parallel_calib_samples=32,
    max_calib_samples=128,
    max_calib_seq_len=4096
)

model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
```

--------------------------------

### Export Model to GGUF Format using AutoAWQ and llama.cpp

Source: https://github.com/casper-hansen/autoawq/blob/main/docs/examples.md

Quantizes a model with AWQ scales applied (without full quantization) and saves it in FP16. It then uses llama.cpp's `convert.py` and `quantize` tools to convert the model to GGUF format, enabling compatibility with other frameworks. This process involves downloading llama.cpp if not present and compiling it.

```python
import os
import subprocess
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = 'mistralai/Mistral-7B-v0.1'
quant_path = 'mistral-awq'
llama_cpp_path = '/workspace/llama.cpp'
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 6, "version": "GEMM" }

# Load model
model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

# Quantize
# NOTE: We avoid packing weights, so you cannot use this model in AutoAWQ
# after quantizing. The saved model is FP16 but has the AWQ scales applied.
model.quantize(
    tokenizer,
    quant_config=quant_config,
    export_compatible=True
)

# Save quantized model
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
print(f'Model is quantized and saved at "{quant_path}"')

# GGUF conversion
print('Converting model to GGUF...')
llama_cpp_method = "q4_K_M"
convert_cmd_path = os.path.join(llama_cpp_path, "convert.py")
quantize_cmd_path = os.path.join(llama_cpp_path, "quantize")

if not os.path.exists(llama_cpp_path):
    cmd = f"git clone https://github.com/ggerganov/llama.cpp.git {llama_cpp_path} && cd {llama_cpp_path} && make LLAMA_CUBLAS=1 LLAMA_CUDA_F16=1"
    subprocess.run([cmd], shell=True, check=True)

subprocess.run([
    f"python {convert_cmd_path} {quant_path} --outfile {quant_path}/model.gguf"
], shell=True, check=True)

subprocess.run([
    f"{quantize_cmd_path} {quant_path}/model.gguf {quant_path}/model_{llama_cpp_method}.gguf {llama_cpp_method}"
], shell=True, check=True)

```

--------------------------------

### CPU Inference with AutoAWQ Model using IPEX

Source: https://github.com/casper-hansen/autoawq/blob/main/docs/examples.md

This Python code snippet demonstrates how to load a quantized AutoAWQ model for CPU inference. It specifies `use_ipex=True` to enable Intel Extension for PyTorch (IPEX) for optimized CPU performance. This is useful for environments where GPU is not available.

```python
from awq import AutoAWQForCausalLM

quant_path = "TheBloke/Mistral-7B-Instruct-v0.2-AWQ"
# Load model for CPU inference with IPEX
# model = AutoAWQForCausalLM.from_quantized(quant_path, use_ipex=True)
# tokenizer = AutoTokenizer.from_pretrained(quant_path, trust_remote_code=True)
# ... rest of the inference code ...

```

--------------------------------

### Initialize Qwen2-VL AWQ Quantizer

Source: https://github.com/casper-hansen/autoawq/blob/main/docs/examples.md

Defines a custom quantizer class that intercepts layer inputs and kwargs for calibration. It uses a Catcher module to capture hidden states from the first layer before restoring the model state.

```python
class Qwen2VLAwqQuantizer(AwqQuantizer):
    def init_quant(self, n_samples=None, max_seq_len=None):
        modules = self.awq_model.get_model_layers(self.model)
        samples = self.calib_data
        inps = []
        layer_kwargs = {}
        best_device = get_best_device()
        modules[0] = modules[0].to(best_device)
        self.awq_model.move_embed(self.model, best_device)
        class Catcher(nn.Module):
            def __init__(self, module):
                super().__init__()
                self.module = module
            def forward(self, *args, **kwargs):
                if len(args) > 0:
                    hidden_states = args[0]
                    del args
                else:
                    first_key = list(kwargs.keys())[0]
                    hidden_states = kwargs.pop(first_key)
                inps.append(hidden_states)
                layer_kwargs.update(kwargs)
                raise ValueError
        modules[0] = Catcher(modules[0])
        try:
            self.model(**samples)
        except ValueError:
            pass
        finally:
            modules[0] = modules[0].module
        return modules, layer_kwargs, inps[0]
```