### Custom Quantizer for Qwen2 VL Example Source: https://casper-hansen.github.io/AutoAWQ/examples Demonstrates using a custom quantizer by extending `AwqQuantizer` for specific models like Qwen2 VL. This allows for specialized processing of multimodal examples. ```python import torch import torch.nn as nn from awq import AutoAWQForCausalLM from awq.utils.qwen_vl_utils import process_vision_info from awq.quantize.quantizer import AwqQuantizer, clear_memory, get_best_device # Specify paths and hyperparameters for quantization model_path = "Qwen/Qwen2-VL-7B-Instruct" quant_path = "qwen2-vl-7b-instruct" quant_config = {"zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM"} model = AutoAWQForCausalLM.from_pretrained( model_path, attn_implementation="flash_attention_2" ) # We define our own quantizer by extending the AwqQuantizer. # The main difference is in how the samples are processed when ``` -------------------------------- ### Run Qwen2 VL AWQ Model Source: https://casper-hansen.github.io/AutoAWQ/examples Example of running inference with the Qwen2 VL AWQ model. It demonstrates loading the model and processor, and setting up messages for multimodal input. ```python from awq import AutoAWQForCausalLM from awq.utils.qwen_vl_utils import process_vision_info from transformers import AutoProcessor, TextStreamer # Load model quant_path = "Qwen/Qwen2-VL-7B-Instruct-AWQ" model = AutoAWQForCausalLM.from_quantized(quant_path) processor = AutoProcessor.from_pretrained(quant_path) streamer = TextStreamer(processor, skip_prompt=True) messages = [ { "role": "user", "content": [ { "type": "image", "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg", }, {"type": "text", "text": "Describe this image."}, ], } ] ``` -------------------------------- ### Load AWQ Model with vLLM Source: https://casper-hansen.github.io/AutoAWQ/examples Load and run AWQ models using vLLM for efficient inference. This example demonstrates setting up the async engine and generating text with specified sampling parameters. ```python import asyncio from transformers import AutoTokenizer, PreTrainedTokenizer from vllm import AsyncLLMEngine, SamplingParams, AsyncEngineArgs model_path = "casperhansen/mixtral-instruct-awq" # prompting prompt = "You're standing on the surface of the Earth. " "You walk one mile south, one mile west and one mile north. " "You end up exactly where you started. Where are you?", prompt_template = "[INST] {prompt} [/INST]" # sampling params sampling_params = SamplingParams( repetition_penalty=1.1, temperature=0.8, max_tokens=512 ) # tokenizer tokenizer = AutoTokenizer.from_pretrained(model_path) # async engine args for streaming engine_args = AsyncEngineArgs( model=model_path, quantization="awq", dtype="float16", max_model_len=512, enforce_eager=True, disable_log_requests=True, disable_log_stats=True, ) async def generate(model: AsyncLLMEngine, tokenizer: PreTrainedTokenizer): tokens = tokenizer(prompt_template.format(prompt=prompt)).input_ids outputs = model.generate( prompt=prompt, sampling_params=sampling_params, request_id=1, prompt_token_ids=tokens, ) print("\n** Starting generation!\n") last_index = 0 async for output in outputs: print(output.outputs[0].text[last_index:], end="", flush=True) last_index = len(output.outputs[0].text) print("\n\n** Finished generation!\n") if __name__ == '__main__': model = AsyncLLMEngine.from_engine_args(engine_args) asyncio.run(generate(model, tokenizer)) ``` -------------------------------- ### Quantize Vision-Language Model with AutoAWQ Source: https://casper-hansen.github.io/AutoAWQ/examples Quantizes a vision-language model like LLaVa. This example uses LLaMa3-LLaVa-Next. Ensure `low_cpu_mem_usage=True` for large models. ```python from awq import AutoAWQForCausalLM from transformers import AutoTokenizer model_path = 'llava-hf/llama3-llava-next-8b-hf' quant_path = 'llama3-llava-next-8b-awq' quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" } # Load model model = AutoAWQForCausalLM.from_pretrained( model_path, low_cpu_mem_usage=True, ) tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) # Quantize model.quantize(tokenizer, quant_config=quant_config) # Save quantized model model.save_quantized(quant_path) tokenizer.save_pretrained(quant_path) print(f'Model is quantized and saved at "{quant_path}"') ``` -------------------------------- ### Load LLaVa AWQ Model for Multimodal Inference Source: https://casper-hansen.github.io/AutoAWQ/examples Load a LLaVa AWQ model and use AutoProcessor to handle multimodal inputs (text and image). This example shows processing a prompt with an image for generation. ```python import torch import requests from PIL import Image from awq import AutoAWQForCausalLM from transformers import AutoProcessor, TextStreamer # Load model quant_path = "casperhansen/llama3-llava-next-8b-awq" model = AutoAWQForCausalLM.from_quantized(quant_path) processor = AutoProcessor.from_pretrained(quant_path) streamer = TextStreamer(processor, skip_prompt=True) # Define prompt prompt = """ <|im_start|>system Answer the questions.<|im_end|> <|im_start|>user What is shown in this image?<|im_end|> <|im_start|>assistant " # Define image url = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true" image = Image.open(requests.get(url, stream=True).raw) # Load inputs inputs = processor(prompt, image, return_tensors='pt').to(0, torch.float16) generation_output = model.generate( **inputs, max_new_tokens=512, streamer=streamer ) ``` -------------------------------- ### Load AWQ Model with Transformers Source: https://casper-hansen.github.io/AutoAWQ/examples Load an AWQ model using Hugging Face's AutoModelForCausalLM. Ensure AutoAWQ is installed and note that fused modules might not be available for all models. ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer # NOTE: Must install from PR until merged # pip install --upgrade git+https://github.com/younesbelkada/transformers.git@add-awq model_id = "casperhansen/mistral-7b-instruct-v0.1-awq" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.float16, low_cpu_mem_usage=True, ) streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True) # Convert prompt to tokens text = "[INST] What are the basic steps to use the Huggingface transformers library? [/INST]" tokens = tokenizer( text, return_tensors='pt' ).input_ids.cuda() # Generate output generation_output = model.generate( tokens, streamer=streamer, max_new_tokens=512 ) ``` -------------------------------- ### Initialize Coding Model Quantization Source: https://casper-hansen.github.io/AutoAWQ/examples Sets up the environment for quantizing specialized coding models like DeepSeek-Coder. ```python from tqdm import tqdm from datasets import load_dataset from awq import AutoAWQForCausalLM from transformers import AutoTokenizer model_path = 'deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct' quant_path = 'deepseek-coder-v2-lite-instruct-awq' quant_config = { "zero_point": True, "q_group_size": 64, "w_bit": 4, "version": "GEMM" } ``` -------------------------------- ### Initialize AutoAWQForCausalLM Source: https://casper-hansen.github.io/AutoAWQ/reference Direct instantiation is not supported; use factory methods instead. ```python AutoAWQForCausalLM() ``` ```python def __init__(self): raise EnvironmentError( "You must instantiate AutoAWQForCausalLM with\n" "AutoAWQForCausalLM.from_quantized or AutoAWQForCausalLM.from_pretrained" ) ``` -------------------------------- ### Initialize Qwen2-VL AWQ Quantizer Source: https://casper-hansen.github.io/AutoAWQ/examples Custom quantizer class extending AwqQuantizer to handle model layer initialization and input catching for calibration. ```python class Qwen2VLAwqQuantizer(AwqQuantizer): def init_quant(self, n_samples=None, max_seq_len=None): modules = self.awq_model.get_model_layers(self.model) samples = self.calib_data inps = [] layer_kwargs = {} best_device = get_best_device() modules[0] = modules[0].to(best_device) self.awq_model.move_embed(self.model, best_device) # get input and kwargs to layer 0 # with_kwargs is only supported in PyTorch 2.0 # use this Catcher hack for now class Catcher(nn.Module): def __init__(self, module): super().__init__() self.module = module def forward(self, *args, **kwargs): # assume first input to forward is hidden states if len(args) > 0: hidden_states = args[0] del args else: first_key = list(kwargs.keys())[0] hidden_states = kwargs.pop(first_key) inps.append(hidden_states) layer_kwargs.update(kwargs) raise ValueError # early exit to break later inference def move_to_device(obj: torch.Tensor | nn.Module, device: torch.device): def get_device(obj: torch.Tensor | nn.Module): if isinstance(obj, torch.Tensor): return obj.device return next(obj.parameters()).device if get_device(obj) != device: obj = obj.to(device) return obj # patch layer 0 to catch input and kwargs modules[0] = Catcher(modules[0]) for k, v in samples.items(): if isinstance(v, (torch.Tensor, nn.Module)): samples[k] = move_to_device(v, best_device) try: self.model(**samples) except ValueError: # work with early exit pass finally: for k, v in samples.items(): if isinstance(v, (torch.Tensor, nn.Module)): samples[k] = move_to_device(v, "cpu") modules[0] = modules[0].module # restore del samples inps = inps[0] modules[0] = modules[0].cpu() self.awq_model.move_embed(self.model, "cpu") clear_memory() return modules, layer_kwargs, inps ``` -------------------------------- ### Initialize Quantized Model with AutoAWQ Source: https://casper-hansen.github.io/AutoAWQ/reference Use this method to initialize a quantized model. It handles loading configurations, weights, and preparing the model for inference across different devices and backends. ```python def __init__( self, model_path: Union[str, Path], model_filename: Optional[str] = None, use_safetensors: bool = True, trust_remote_code: bool = False, max_seq_len: Optional[int] = None, torch_dtype: torch.dtype = torch.float16, device_map: Optional[Dict[str, Union[str, int, torch.device]]] = None, max_memory: Optional[Dict[Union[int, str], Union[int, str]]], offload_folder: Optional[str] = None, download_kwargs: Optional[Dict] = None, **config_kwargs: ): """A method for initialization of a quantized model, usually in INT4.""" # [STEP 1-2] Load weights path and configs model_weights_path, config, quant_config = self._load_config( self, model_path, model_filename, safetensors, trust_remote_code, max_seq_len=max_seq_len, download_kwargs=download_kwargs, **config_kwargs, ) target_cls_name = TRANSFORMERS_AUTO_MAPPING_DICT[config.model_type] target_cls = getattr(transformers, target_cls_name) # [STEP 3] Load model with init_empty_weights(): model = target_cls.from_config( config=config, torch_dtype=torch_dtype, trust_remote_code=trust_remote_code, ) best_device = get_best_device() if best_device == "cpu" or (best_device == "xpu:0" and not triton_available): use_ipex = True if use_ipex and not ipex_available: raise ImportError( "Please install intel_extension_for_pytorch with " "`pip install intel_extension_for_pytorch` for 'ipex' kernel!" ) # Prepare WQLinear layers, replace nn.Linear self._load_quantized_modules( self, model, quant_config, quant_config.version, use_exllama=use_exllama, use_exllama_v2=use_exllama_v2, use_ipex=use_ipex, ) model.tie_weights() # loads the weights into modules and distributes # across available devices automatically load_checkpoint_and_dispatch( model, checkpoint=model_weights_path, device_map=device_map, max_memory=max_memory, no_split_module_classes=[self.layer_type], offload_folder=offload_folder, dtype=torch_dtype, ) # Dispath to devices awq_ext, msg = try_import("awq_ext") if fuse_layers: if best_device in ["mps", "cuda:0"] and awq_ext is None: warnings.warn("Skipping fusing modules because AWQ extension is not installed." + msg) else: self.fuse_layers(model) if use_ipex: # repack qweight to match the ipex kernel. model = ipex_post_init(model) elif quant_config.version == "marlin": model = marlin_post_init(model) elif use_exllama: # creates q4 handle model = exllama_post_init(model) elif use_exllama_v2: # creates q4 handle and allocates scratch spaces wrt max_input_len and max_batch_size model = exllamav2_post_init( model, max_input_len=max_seq_len or 2048, max_batch_size=int(os.getenv("AWQ_BATCH_SIZE", 1)), ) model.eval() return self( model, model_type, is_quantized=True, config=config, quant_config=quant_config, processor=None, ) ``` -------------------------------- ### Method: to Source: https://casper-hansen.github.io/AutoAWQ/reference Moves the model to a specified device. ```APIDOC ## Method: to ### Description A utility function for moving the model to a device. ### Parameters #### Query Parameters - **device** (str) - Required - The device to move your model to. ``` -------------------------------- ### Constructor: BaseAWQForCausalLM Source: https://casper-hansen.github.io/AutoAWQ/reference Initializes the base model for AutoAWQ with configuration and processor details. ```APIDOC ## Constructor: BaseAWQForCausalLM ### Description Initializes the base model for all AutoAWQ models. ### Parameters #### Request Body - **model** (PreTrainedModel) - Required - The pretrained or quantized model. - **model_type** (str) - Required - The model type, found in config.json. - **is_quantized** (bool) - Required - Indicates if the current model is quantized. - **config** (PretrainedConfig) - Required - The config of the model. - **quant_config** (AwqConfig) - Required - The quantization config of the model. - **processor** (BaseImageProcessor) - Required - An optional processor, e.g. for vision models. ``` -------------------------------- ### Initialize BaseAWQForCausalLM Source: https://casper-hansen.github.io/AutoAWQ/reference Initializes the base model for AutoAWQ. Requires pretrained model, model type, quantization status, configuration, quantization configuration, and an optional processor. ```python def __init__( self, model: Annotated[PreTrainedModel, Doc("The pretrained or quantized model.")], model_type: Annotated[str, Doc("The model type, found in config.json.")], is_quantized: Annotated[ bool, Doc("Indicates if the current model is quantized.") ], config: Annotated[PretrainedConfig, Doc("The config of the model.")], quant_config: Annotated[ AwqConfig, Doc("The quantization config of the model.") ], processor: Annotated[ BaseImageProcessor, Doc("An optional processor, e.g. for vision models.") ], ): """The base model for all AutoAWQ models.""" super().__init__() self.model: PreTrainedModel = model self.model_type: str = model_type self.is_quantized: bool = is_quantized self.search_result = None self.config: PretrainedConfig = config self.quant_config: AwqConfig = quant_config self.processor: ProcessorMixin = processor ``` -------------------------------- ### Process Inputs and Run Quantization Source: https://casper-hansen.github.io/AutoAWQ/examples Tokenize the dataset, process vision information, and execute the AWQ quantization process. ```python # process the dataset into tensors text = model.processor.apply_chat_template(dataset, tokenize=False, add_generation_prompt=True) image_inputs, video_inputs = process_vision_info(dataset) inputs = model.processor(text=text, images=image_inputs, videos=video_inputs, padding=True, return_tensors="pt") # Then just run the calibration process by one line of code model.quantize(calib_data=inputs, quant_config=quant_config, quantizer_cls=Qwen2VLAwqQuantizer) # Save the model model.model.config.use_cache = model.model.generation_config.use_cache = True model.save_quantized(quant_path, safetensors=True, shard_size="4GB") ``` -------------------------------- ### Perform Basic Model Quantization Source: https://casper-hansen.github.io/AutoAWQ/examples Quantizes a model using default settings and saves the result to disk. ```python from awq import AutoAWQForCausalLM from transformers import AutoTokenizer model_path = 'mistralai/Mistral-7B-Instruct-v0.2' quant_path = 'mistral-instruct-v0.2-awq' quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" } # Load model model = AutoAWQForCausalLM.from_pretrained(model_path) tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) # Quantize model.quantize(tokenizer, quant_config=quant_config) # Save quantized model model.save_quantized(quant_path) tokenizer.save_pretrained(quant_path) print(f'Model is quantized and saved at "{quant_path}"') ``` -------------------------------- ### Custom Quantizer for MiniCPM3 with AutoAWQ Source: https://casper-hansen.github.io/AutoAWQ/examples Demonstrates how to implement and use a custom quantizer class, CPM3AwqQuantizer, for quantizing MiniCPM3 models with AutoAWQ. This involves modifying the weight clipping mechanism. ```python import torch from transformers import AutoTokenizer from awq import AutoAWQForCausalLM from awq.quantize.quantizer import AwqQuantizer, clear_memory class CPM3AwqQuantizer(AwqQuantizer): @torch.no_grad() def _compute_best_clip( self, w: torch.Tensor, input_feat: torch.Tensor, n_grid=20, max_shrink=0.5, n_sample_token=512, ): assert w.dim() == 2 org_w_shape = w.shape # w [co, ci] -> [co, 1, n_group, group size] # input_feat [n_token, ci] -> [1, n_token, n_group, group size] group_size = self.group_size if self.group_size > 0 else org_w_shape[1] input_feat = input_feat.view(-1, input_feat.shape[-1]) input_feat = input_feat.reshape(1, input_feat.shape[0], -1, group_size) # Compute input feature step size (minimum 1) step_size = max(1, input_feat.shape[1] // n_sample_token) input_feat = input_feat[:, ::step_size] w = w.reshape(org_w_shape[0], 1, -1, group_size) oc_batch_size = 256 if org_w_shape[0] % 256 == 0 else 64 # prevent OOM if org_w_shape[0] % oc_batch_size != 0: oc_batch_size = org_w_shape[0] assert org_w_shape[0] % oc_batch_size == 0 w_all = w best_max_val_all = [] for i_b in range(org_w_shape[0] // oc_batch_size): w = w_all[i_b * oc_batch_size : (i_b + 1) * oc_batch_size] org_max_val = w.abs().amax(dim=-1, keepdim=True) # co, 1, n_group, 1 best_max_val = org_max_val.clone() min_errs = torch.ones_like(org_max_val) * 1e9 input_feat = input_feat.to(w.device) org_out = (input_feat * w).sum(dim=-1) # co, n_token, n_group for i_s in range(int(max_shrink * n_grid)): max_val = org_max_val * (1 - i_s / n_grid) min_val = -max_val cur_w = torch.clamp(w, min_val, max_val) q_w = self.pseudo_quantize_tensor(cur_w)[0] cur_out = (input_feat * q_w).sum(dim=-1) # co, 1, n_group, 1 err = (cur_out - org_out).pow(2).mean(dim=1).view(min_errs.shape) del cur_w del cur_out cur_best_idx = err < min_errs min_errs[cur_best_idx] = err[cur_best_idx] best_max_val[cur_best_idx] = max_val[cur_best_idx] best_max_val_all.append(best_max_val) best_max_val = torch.cat(best_max_val_all, dim=0) clear_memory(input_feat) clear_memory(org_out) return best_max_val.squeeze(1) model_path = 'openbmb/MiniCPM3-4B' quant_path = 'minicpm3-4b-awq' quant_config = { "zero_point": True, "q_group_size": 64, "w_bit": 4, "version": "GEMM" } # Load model model = AutoAWQForCausalLM.from_pretrained(model_path, safetensors=False) tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) # Quantize model.quantize(tokenizer, quant_config=quant_config, quantizer_cls=CPM3AwqQuantizer) # Save quantized model model.save_quantized(quant_path) tokenizer.save_pretrained(quant_path) print(f'Model is quantized and saved at "{quant_path}"') ``` -------------------------------- ### Configure AutoAWQ for CPU inference Source: https://casper-hansen.github.io/AutoAWQ Enable IPEX support when running quantized models on CPU devices. ```python model = AutoAWQForCausalLM.from_quantized( ..., use_ipex=True ) ``` -------------------------------- ### Process inputs and generate model output Source: https://casper-hansen.github.io/AutoAWQ/examples Prepares chat templates and vision inputs for the model, then executes generation with a specified token limit and streamer. ```python text = processor.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) image_inputs, video_inputs = process_vision_info(messages) inputs = processor( text=[text], images=image_inputs, videos=video_inputs, padding=True, return_tensors="pt", ) inputs = inputs.to("cuda") generation_output = model.generate( **inputs, max_new_tokens=512, streamer=streamer ) ``` -------------------------------- ### classmethod from_pretrained Source: https://casper-hansen.github.io/AutoAWQ/reference Initializes pretrained models, typically in FP16, from a specified model path. ```APIDOC ## classmethod from_pretrained ### Description A method for initialization of pretrained models, usually in FP16. ### Method classmethod ### Endpoint N/A (Class method) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```python # Example usage (conceptual, actual instantiation depends on context) model = AutoModelClass.from_pretrained( model_path="/path/to/your/model", model_type="llama", torch_dtype=torch.float16, trust_remote_code=True, safetensors=True, device_map="auto" ) ``` ### Response #### Success Response (200) - **model** (object) - The initialized model instance. #### Response Example ```json { "message": "Model initialized successfully" } ``` ### Parameters Table | PARAMETER | DESCRIPTION | |---|---| | `model_path` | A Huggingface path or local path to a model. **TYPE:** `str` | | `model_type` | The model type, loaded from config.json. **TYPE:** `str` | | `torch_dtype` | The dtype to load the model as. May not work with other values than float16. **TYPE:** `dtype` **DEFAULT:** `float16` | | `trust_remote_code` | Useful for Huggingface repositories that have not been integrated into transformers yet. **TYPE:** `bool` **DEFAULT:** `True` | | `safetensors` | Whether to download/load safetensors instead of torch weights. **TYPE:** `bool` **DEFAULT:** `True` | | `device_map` | A device map that will be passed onto the model loading method from transformers. **TYPE:** `Union[str, Dict]` **DEFAULT:** `'auto'` | | `download_kwargs` | Used for configure download model **TYPE:** `Dict` **DEFAULT:** `None` | | `low_cpu_mem_usage` | Use low_cpu_mem_usage when loading from transformers. **TYPE:** `bool` **DEFAULT:** `True` | | `use_cache` | Use use_cache argument in transformers **TYPE:** `bool` **DEFAULT:** `False` | | `**model_init_kwargs` | Additional kwargs that are passed to the model during initialization. **TYPE:** `Dict` **DEFAULT:** `{}` | ``` -------------------------------- ### Prepare Calibration Dataset Source: https://casper-hansen.github.io/AutoAWQ/examples Function to load and format a dataset for calibration, mapping samples into the required chat template structure. ```python def prepare_dataset(n_sample: int = 8) -> list[list[dict]]: from datasets import load_dataset dataset = load_dataset("laion/220k-GPT4Vision-captions-from-LIVIS", split=f"train[:{n_sample}]") return [ [ { "role": "user", "content": [ {"type": "image", "image": sample["url"]}, {"type": "text", "text": "generate a caption for this image"}, ], }, {"role": "assistant", "content": sample["caption"]}, ] for sample in dataset ] dataset = prepare_dataset() ``` -------------------------------- ### Prepare Model for GGUF Export with AutoAWQ Source: https://casper-hansen.github.io/AutoAWQ/examples Quantizes a model with AWQ scales applied but without actual quantization, making it compatible with other frameworks like llama.cpp for GGUF conversion. This process involves saving the FP16 model with scales, then using `convert.py` and `quantize` from llama.cpp. ```python import os import subprocess from awq import AutoAWQForCausalLM from transformers import AutoTokenizer model_path = 'mistralai/Mistral-7B-v0.1' quant_path = 'mistral-awq' llama_cpp_path = '/workspace/llama.cpp' quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 6, "version": "GEMM" } # Load model model = AutoAWQForCausalLM.from_pretrained(model_path) tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) # Quantize # NOTE: We avoid packing weights, so you cannot use this model in AutoAWQ # after quantizing. The saved model is FP16 but has the AWQ scales applied. model.quantize( tokenizer, quant_config=quant_config, export_compatible=True ) # Save quantized model model.save_quantized(quant_path) tokenizer.save_pretrained(quant_path) print(f'Model is quantized and saved at "{quant_path}"') # GGUF conversion print('Converting model to GGUF...') llama_cpp_method = "q4_K_M" convert_cmd_path = os.path.join(llama_cpp_path, "convert.py") quantize_cmd_path = os.path.join(llama_cpp_path, "quantize") if not os.path.exists(llama_cpp_path): cmd = f"git clone https://github.com/ggerganov/llama.cpp.git {llama_cpp_path} && cd {llama_cpp_path} && make LLAMA_CUBLAS=1 LLAMA_CUDA_F16=1" subprocess.run([cmd], shell=True, check=True) subprocess.run([ f"python {convert_cmd_path} {quant_path} --outfile {quant_path}/model.gguf" ], shell=True, check=True) subprocess.run([ f"{quantize_cmd_path} {quant_path}/model.gguf {quant_path}/model_{llama_cpp_method}.gguf {llama_cpp_method}" ], shell=True, check=True) ``` -------------------------------- ### POST /api/models/from_quantized Source: https://casper-hansen.github.io/AutoAWQ/reference Initializes a quantized model, typically in INT4 format, from a specified path. This method allows for fine-grained control over model loading, including sequence length, data types, and optimization kernels. ```APIDOC ## POST /api/models/from_quantized ### Description Initializes a quantized model, typically in INT4 format, from a specified path. This method allows for fine-grained control over model loading, including sequence length, data types, and optimization kernels. ### Method POST ### Endpoint /api/models/from_quantized ### Parameters #### Request Body - **model_path** (str) - Required - A Huggingface path or local path to a model. - **model_type** (str) - Required - The model type, loaded from config.json. - **model_filename** (str) - Optional - Load a specific model's filename by specifying this argument. Default: '' - **max_seq_len** (int) - Optional - The maximum sequence cached sequence length of the model. Larger values may increase loading time and memory usage. Default: None - **torch_dtype** (dtype) - Optional - The dtype to load the model as. May not work with other values than float16. Default: float16 - **trust_remote_code** (bool) - Optional - Useful for Huggingface repositories that have not been integrated into transformers yet. Default: True - **safetensors** (bool) - Optional - Whether to download/load safetensors instead of torch weights. Default: True - **fuse_layers** (bool) - Optional - Whether to use fused/optimized combination of layers for increased speed. Default: True - **use_exllama** (bool) - Optional - Whether to map the weights to ExLlamaV1 kernels. Default: False - **use_exllama_v2** (bool) - Optional - Whether to map the weights to ExLlamaV2 kernels. Default: False - **use_ipex** (bool) - Optional - Whether to map the weights to ipex kernels for CPU and XPU device. Default: False - **device_map** (Union[str, Dict]) - Optional - A device map that will be passed onto the model loading method from transformers. Default: 'balanced' - **max_memory** (Dict[Union[int, str], Union[int, str]]) - Optional - A dictionary device identifier to maximum memory which will be passed onto the model loading method from transformers. For example：{0: "4GB",1: "10GB" Default: None - **offload_folder** (str) - Optional - The folder ot offload the model to. Default: None - **download_kwargs** (Dict) - Optional - Used for configure download model. Default: None - **config_kwargs** (Dict) - Optional - Additional kwargs that are passed to the config during initialization. Default: {} ### Request Example { "model_path": "/path/to/your/model", "model_type": "llama", "max_seq_len": 2048, "torch_dtype": "float16", "device_map": "auto" } ### Response #### Success Response (200) - **model** (object) - The loaded quantized model object. #### Response Example { "model": "" } ``` -------------------------------- ### from_pretrained Implementation Source: https://casper-hansen.github.io/AutoAWQ/reference The full implementation of the from_pretrained method located in awq/models/base.py. ```python @classmethod def from_pretrained( self, model_path: Annotated[str, Doc("A Huggingface path or local path to a model.")], model_type: Annotated[str, Doc("The model type, loaded from config.json.")], torch_dtype: Annotated[ torch.dtype, Doc( "The dtype to load the model as. May not work with other values than float16." ), ] = torch.float16, trust_remote_code: Annotated[ bool, Doc( "Useful for Huggingface repositories that have not been integrated into transformers yet." ), ] = True, safetensors: Annotated[ bool, Doc("Whether to download/load safetensors instead of torch weights.") ] = True, device_map: Annotated[ Union[str, Dict], Doc( "A device map that will be passed onto the model loading method from transformers." ), ] = "auto", download_kwargs: Annotated[ Dict, Doc("Used for configure download model"), ] = None, low_cpu_mem_usage: Annotated[ bool, Doc("Use low_cpu_mem_usage when loading from transformers.") ] = True, use_cache: Annotated[ bool, Doc("Use use_cache argument in transformers") ] = False, **model_init_kwargs: Annotated[ Dict, Doc( "Additional kwargs that are passed to the model during initialization." ), ], ): """A method for initialization of pretrained models, usually in FP16.""" # Get weights path and quant config model_weights_path, config, quant_config = self._load_config( self, model_path, "", safetensors, trust_remote_code=trust_remote_code, download_kwargs=download_kwargs, ) target_cls_name = TRANSFORMERS_AUTO_MAPPING_DICT[config.model_type] target_cls = getattr(transformers, target_cls_name) processor = None if target_cls_name == "AutoModelForVision2Seq" or target_cls_name == "AutoModelForTextToWaveform": processor = AutoProcessor.from_pretrained(model_weights_path) if model_init_kwargs.get("low_cpu_mem_usage") is None: model_init_kwargs["low_cpu_mem_usage"] = low_cpu_mem_usage if model_init_kwargs.get("use_cache") is None and not ((target_cls_name == "AutoModelForVision2Seq") or (target_cls_name == "AutoModelForTextToWaveform")): model_init_kwargs["use_cache"] = use_cache # If not quantized, must load with AutoModelForCausalLM model = target_cls.from_pretrained( model_weights_path, trust_remote_code=trust_remote_code, torch_dtype=torch_dtype, use_safetensors=safetensors, device_map=device_map, **model_init_kwargs, ) model.eval() return self( model, model_type, is_quantized=False, config=config, quant_config=quant_config, processor=processor, ) ``` -------------------------------- ### Load quantized model with from_quantized Source: https://casper-hansen.github.io/AutoAWQ/reference Loads a previously quantized model. Configures environment variables and model parameters for inference. ```python from_quantized(quant_path, quant_filename='', max_seq_len=2048, trust_remote_code=True, fuse_layers=True, use_exllama=False, use_exllama_v2=False, use_ipex=False, batch_size=1, safetensors=True, device_map='balanced', max_memory=None, offload_folder=None, download_kwargs=None, **config_kwargs) ``` ```python @classmethod def from_quantized( self, quant_path, quant_filename="", max_seq_len=2048, trust_remote_code=True, fuse_layers=True, use_exllama=False, use_exllama_v2=False, use_ipex=False, batch_size=1, safetensors=True, device_map="balanced", max_memory=None, offload_folder=None, download_kwargs=None, **config_kwargs, ) -> BaseAWQForCausalLM: os.environ["AWQ_BATCH_SIZE"] = str(batch_size) model_type = check_and_get_model_type(quant_path, trust_remote_code) if config_kwargs.get("max_new_tokens") is not None: max_seq_len = config_kwargs["max_new_tokens"] logging.warning( "max_new_tokens argument is deprecated... gracefully " "setting max_seq_len=max_new_tokens." ) return AWQ_CAUSAL_LM_MODEL_MAP[model_type].from_quantized( quant_path, model_type, quant_filename, max_seq_len, trust_remote_code=trust_remote_code, fuse_layers=fuse_layers, use_exllama=use_exllama, use_exllama_v2=use_exllama_v2, use_ipex=use_ipex, safetensors=safetensors, device_map=device_map, max_memory=max_memory, offload_folder=offload_folder, download_kwargs=download_kwargs, **config_kwargs, ) ``` -------------------------------- ### Load pre-trained model with from_pretrained Source: https://casper-hansen.github.io/AutoAWQ/reference Loads a model from a path. Requires using the class method directly. ```python from_pretrained(model_path, torch_dtype='auto', trust_remote_code=True, safetensors=True, device_map=None, download_kwargs=None, low_cpu_mem_usage=True, use_cache=False, **model_init_kwargs) ``` ```python @classmethod def from_pretrained( self, model_path, torch_dtype="auto", trust_remote_code=True, safetensors=True, device_map=None, download_kwargs=None, low_cpu_mem_usage=True, use_cache=False, **model_init_kwargs, ) -> BaseAWQForCausalLM: model_type = check_and_get_model_type( model_path, trust_remote_code, **model_init_kwargs ) return AWQ_CAUSAL_LM_MODEL_MAP[model_type].from_pretrained( model_path, model_type, torch_dtype=torch_dtype, trust_remote_code=trust_remote_code, safetensors=safetensors, device_map=device_map, download_kwargs=download_kwargs, low_cpu_mem_usage=low_cpu_mem_usage, use_cache=use_cache, **model_init_kwargs, ) ``` -------------------------------- ### Configure AutoAWQ for AMD GPUs Source: https://casper-hansen.github.io/AutoAWQ Use these parameters when loading a quantized model on AMD hardware to ensure compatibility with ExLlamaV2 kernels. ```python model = AutoAWQForCausalLM.from_quantized( ..., fuse_layers=False, use_exllama_v2=True ) ``` -------------------------------- ### Quantize with Custom Calibration Data Source: https://casper-hansen.github.io/AutoAWQ/examples Uses specific datasets like wikitext or dolly for calibration during the quantization process. Samples exceeding 512 tokens are discarded. ```python from datasets import load_dataset from awq import AutoAWQForCausalLM from transformers import AutoTokenizer model_path = 'lmsys/vicuna-7b-v1.5' quant_path = 'vicuna-7b-v1.5-awq' quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" } # Load model model = AutoAWQForCausalLM.from_pretrained(model_path) tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) # Define data loading methods def load_dolly(): data = load_dataset('databricks/databricks-dolly-15k', split="train") # concatenate data def concatenate_data(x): return {"text": x['instruction'] + '\n' + x['context'] + '\n' + x['response']} concatenated = data.map(concatenate_data) return [text for text in concatenated["text"]] def load_wikitext(): data = load_dataset('wikitext', 'wikitext-2-raw-v1', split="train") return [text for text in data["text"] if text.strip() != '' and len(text.split(' ')) > 20] # Quantize model.quantize(tokenizer, quant_config=quant_config, calib_data=load_wikitext()) # Save quantized model model.save_quantized(quant_path) tokenizer.save_pretrained(quant_path) print(f'Model is quantized and saved at "{quant_path}"') ``` -------------------------------- ### AutoAWQForCausalLM Class Source: https://casper-hansen.github.io/AutoAWQ/reference The `AutoAWQForCausalLM` class serves as an entry point for loading causal language models quantized with AutoAWQ. It requires instantiation through its `from_pretrained` or `from_quantized` class methods. ```APIDOC ## AutoAWQForCausalLM Class ### Description This class is used to load causal language models quantized with AutoAWQ. It cannot be instantiated directly and must be used with its `from_pretrained` or `from_quantized` class methods. ### Methods - `from_pretrained(model_path, torch_dtype='auto', trust_remote_code=True, safetensors=True, device_map=None, download_kwargs=None, low_cpu_mem_usage=True, use_cache=False, **model_init_kwargs)`: Class method to load a pre-trained model. - `from_quantized(quant_path, quant_filename='', max_seq_len=2048, trust_remote_code=True, fuse_layers=True, use_exllama=False, use_exllama_v2=False, use_ipex=False, batch_size=1, safetensors=True, device_map='balanced', max_memory=None, offload_folder=None, download_kwargs=None, **config_kwargs)`: Class method to load a quantized model. ### Initialization ```python AutoAWQForCausalLM() ``` ### Initialization Error ```python raise EnvironmentError( "You must instantiate AutoAWQForCausalLM with\n" "AutoAWQForCausalLM.from_quantized or AutoAWQForCausalLM.from_pretrained" ) ``` ``` -------------------------------- ### Quantize Source Implementation Source: https://casper-hansen.github.io/AutoAWQ/reference The internal implementation of the quantize method in the base model class. ```python @torch.no_grad() def quantize( self, tokenizer: Annotated[ PreTrainedTokenizer, Doc("The tokenizer to use for quantization.") ] = None, quant_config: Annotated[ Dict, Doc("The quantization config you want to use.") ] = {}, calib_data: Annotated[ Union[str, List[str]], Doc( "The calibration dataset. Either a string pointing to Huggingface or a list of preloaded examples." ), ] = "pileval", split: Annotated[str, Doc("The split of calib_data.")] = "train", text_column: Annotated[str, Doc("The text column of calib_data.")] = "text", duo_scaling: Annotated[ bool, Doc("Whether to scale using both w/x or just x.") ] = True, export_compatible: Annotated[ bool, Doc( "This argument avoids real quantization by only applying the scales without quantizing down to FP16." ), ] = False, apply_clip: Annotated[ bool, Doc( "Whether to apply clipping to the model during quantization. Some models may perform better with this set to False." ), ] = True, n_parallel_calib_samples: Annotated[ int, Doc( "The number of parallel samples to run through the model. " "A high number of parallel samples can result in OOM during quantization if max_calib_samples is high enough. " "If None, runs through all samples at the same time. " "You can set this to a low number for more memory efficient quantization." ), ] = None, max_calib_samples: Annotated[ ``` -------------------------------- ### Method: forward Source: https://casper-hansen.github.io/AutoAWQ/reference Mimics the torch forward function for the model. ```APIDOC ## Method: forward ### Description A forward function that mimics the torch forward. ``` -------------------------------- ### CPU Inference with AutoAWQ Source: https://casper-hansen.github.io/AutoAWQ/examples Enables CPU inference by setting `use_ipex=True`. This utilizes Intel Extension for PyTorch (IPEX) for optimized CPU kernels. ```python from awq import AutoAWQForCausalLM quant_path = "TheBloke/Mistral-7B-Instruct-v0.2-AWQ" ```