PolarQuant Skills

https://huggingface.co/caiovicentino1/polarquant-skills
Admin
PolarQuant Skills provides 14 slash commands for Claude Code and 13 CLI commands for quantizing,...
Tokens:158,828
Snippets:1,725
Trust Score:10
Update:5 months ago
Show doc for...
Tokens:
Raw
### PolarQuant Skills Workflow Example

Source: https://huggingface.co/caiovicentino1/polarquant-skills/commit/d553a927b27914564539f79a9632a6cbbb6fc5e6

Demonstrates a typical workflow using various PolarQuant skills, starting from identifying opportunities to deploying a quantized model.

```bash
/polarquant-monitor opportunities # Find hot models  
/polarquant google/gemma-5-48B-it # Quantize  
/polarquant-bench google/gemma-5-48B # Benchmark vs alternatives  
/polarquant-inference caiovicentino1/... # Create user notebook  
/polarquant-gguf caiovicentino1/... # GGUF for ollama users  
/polarquant-serve caiovicentino1/... # Deploy API  
/polarquant-collection sync # Update HF collection  
```

--------------------------------

### PolarQuant Workflow Example

Source: https://huggingface.co/caiovicentino1/polarquant-skills/commit/c9877219da59a1a55495a2b3b32d697b96899e34

This sequence demonstrates a typical workflow using PolarQuant CLI commands, starting from identifying models to deploy.

```bash
/polarquant-monitor opportunities # Find hot models
/polarquant google/gemma-5-48B-it # Quantize
/polarquant-bench google/gemma-5-48B # Benchmark vs alternatives
/polarquant-inference caiovicentino1/...
/polarquant-gguf caiovicentino1/...
/polarquant-serve caiovicentino1/...
/polarquant-collection sync # Update HF collection
```

--------------------------------

### Install PolarQuant CLI Commands

Source: https://huggingface.co/caiovicentino1/polarquant-skills/commit/950ea2635a8821f0cb5a7264d55184d9c8721f6c

Installs PolarQuant and provides examples of common CLI commands for model quantization, chat, serving, and monitoring.

```bash
pip install git+https://github.com/caiovicentino/polarengine-vllm.git[all]
```

```bash
polarquant quantize google/gemma-4-31B-it --upload
```

```bash
polarquant serve caiovicentino1/model --port 8000
```

```bash
polarquant monitor --stats
```

--------------------------------

### Documentation Installation Instruction

Source: https://huggingface.co/caiovicentino1/polarquant-skills/raw/main/polarquant-vllm-kv.md

Command to install PolarQuant vLLM with KV cache support.

```bash
pip install polarengine-vllm[kv-cache]

```

--------------------------------

### Install Dependencies

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blob/main/polarquant-bench.md

Installs necessary libraries for the benchmark, including transformers, datasets, accelerate, safetensors, sentencepiece, scipy, torchao, and bitsandbytes.

```bash
!pip install git+https://github.com/huggingface/transformers.git --force-reinstall -q
!pip install -q datasets accelerate safetensors sentencepiece scipy torchao bitsandbytes

```

--------------------------------

### Colab Notebook Setup

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blob/main/polarquant-serve.md

Code snippet for setting up the PolarQuant serving environment in a Google Colab notebook. Includes installing ngrok and starting the server with a tunnel for public API access.

```python
!pip install pyngrok fastapi uvicorn
from pyngrok import ngrok
# ... start server + ngrok tunnel ...
print(f'API endpoint: {ngrok_url}/v1/chat/completions')

```

--------------------------------

### Ollama Quick Start Command

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blob/main/polarquant-gguf.md?code=true

This command provides a quick way to run the PolarQuant GGUF model using Ollama. Ensure you have Ollama installed and replace `{REPO}` and `{short_name}` with the appropriate values.

```bash
ollama run hf.co/{REPO}:{short_name}-Q4_K_M
```

--------------------------------

### Install Dependencies and Clone llama.cpp

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blame/main/polarquant-gguf.md

Install necessary libraries including the Transformers library with force-reinstall, accelerate, safetensors, and sentencepiece. It also clones the llama.cpp repository and installs its conversion requirements.

```python
!pip install git+https://github.com/huggingface/transformers.git --force-reinstall -q
!pip install -q accelerate safetensors sentencepiece scipy

# Clone llama.cpp for conversion tools
!git clone https://github.com/ggerganov/llama.cpp.git /content/llama.cpp
!cd /content/llama.cpp && pip install -r requirements/requirements-convert_hf_to_gguf.txt -q
```

--------------------------------

### PolarQuant CLI Usage Examples

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blame/main/README.md

Examples demonstrating common PolarQuant CLI commands for interacting with models, including info, chat, quantization, and serving.

```bash
polarquant info google/gemma-4-31B-it
```

```bash
polarquant chat google/gemma-4-31B-it --vision
```

```bash
polarquant quantize google/gemma-4-31B-it --upload
```

```bash
polarquant serve caiovicentino1/model --port 8000
```

```bash
polarquant monitor --stats
```

--------------------------------

### Install and Use PolarQuant CLI

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blob/main/README.md?code=true

Install the polarquant-vllm package with all extras and demonstrates basic CLI commands for model interaction, quantization, and serving.

```bash
pip install git+https://github.com/caiovicentino/polarengine-vllm.git[all]

polarquant info google/gemma-4-31B-it
polarquant chat google/gemma-4-31B-it --vision
polarquant quantize google/gemma-4-31B-it --upload
polarquant serve caiovicentino1/model --port 8000
polarquant monitor --stats
```

--------------------------------

### Install PolarQuant CLI

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blame/main/README.md

Install the PolarQuant command-line interface by cloning the repository and installing with all extras. This enables various command-line operations.

```bash
pip install git+https://github.com/caiovicentino/polarengine-vllm.git[all]
```

--------------------------------

### Install Dependencies for Benchmarking

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blame/main/polarquant-bench.md

Install necessary libraries for benchmarking, including transformers, datasets, accelerate, safetensors, sentencepiece, scipy, torchao, and bitsandbytes. The transformers library is reinstalled to ensure the latest version.

```bash
!pip install git+https://github.com/huggingface/transformers.git --force-reinstall -q
!pip install -q datasets accelerate safetensors sentencepiece scipy torchao bitsandbytes
```

--------------------------------

### Install Transformers Library

Source: https://huggingface.co/caiovicentino1/polarquant-skills/colab

Installs or upgrades the Hugging Face transformers library. This is a prerequisite for using pre-trained models.

```bash
!pip install -U transformers
```

--------------------------------

### Install vLLM Fork for Expert Offloading

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blob/main/nemotron-offload.md?code=true

Installs the necessary vLLM fork with expert offloading capabilities and the transformers library. Ensure the VLLM_USE_PRECOMPILED environment variable is set.

```python
# Install vLLM fork with expert offloading
import subprocess, sys, os
os.environ["VLLM_USE_PRECOMPILED"] = "1"
subprocess.run([sys.executable, "-m", "pip", "install", 
 "git+https://github.com/caiovicentino/vllm-expert-offload.git@nemotron-expert-offload", 
 "-q"], check=True)
!pip install transformers -q
print("Installed!")
```

--------------------------------

### Install vLLM Fork with Expert Offloading

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blame/main/nemotron-offload.md

Installs the vLLM fork with expert offloading capabilities and the transformers library. Sets an environment variable to use precompiled vLLM components.

```python
# Install vLLM fork with expert offloading
import subprocess, sys, os
os.environ["VLLM_USE_PRECOMPILED"] = "1"
subprocess.run([sys.executable, "-m", "pip", "install",
    "git+https://github.com/caiovicentino/vllm-expert-offload.git@nemotron-expert-offload",
    "-q"], check=True)
!pip install transformers -q
print("Installed!")
```

--------------------------------

### Install PolarQuant CLI

Source: https://huggingface.co/caiovicentino1/polarquant-skills/raw/main/README.md

Install the PolarQuant package with all extras. Use these commands to interact with models via the CLI.

```bash
pip install git+https://github.com/caiovicentino/polarengine-vllm.git[all]
```

```bash
polarquant info google/gemma-4-31B-it
```

```bash
polarquant chat google/gemma-4-31B-it --vision
```

```bash
polarquant quantize google/gemma-4-31B-it --upload
```

```bash
polarquant serve caiovicentino1/model --port 8000
```

```bash
polarquant monitor --stats
```

--------------------------------

### Install Dependencies

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blob/main/polarquant.md?code=true

Installs necessary libraries for model quantization and processing. Use --force-reinstall for transformers to ensure the latest version is used. Avoid flash-attn.

```python
!pip install git+https://github.com/huggingface/transformers.git --force-reinstall -q
!pip install -q datasets accelerate safetensors sentencepiece tiktoken scipy torchao
```

--------------------------------

### Client Example for PolarQuant API

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blob/main/polarquant-serve.md?code=true

Example Python script demonstrating how to use the OpenAI client to interact with the locally served PolarQuant API. Shows how to create a chat completion request and stream the response.

```python
from openai import OpenAI  
  
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")  
  
response = client.chat.completions.create(  
 model="{MODEL}",  
 messages=[{"role": "user", "content": "Hello!"}],  
 stream=True,  
)  
for chunk in response:  
 print(chunk.choices[0].delta.content, end="")  
```

--------------------------------

### OpenAI Client Example with PolarQuant

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blame/main/polarquant-serve.md

Demonstrates how to use the OpenAI client to interact with a PolarQuant server. Ensure the server is running at 'http://localhost:8000/v1' and uses the specified model. This example shows how to send a message and stream the response.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="{MODEL}",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
)
for chunk in response:
    print(chunk.choices[0].delta.content, end="")
```

--------------------------------

### OpenAI Client Example

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blob/main/polarquant-serve.md

Example Python script demonstrating how to use the OpenAI client to interact with the locally served PolarQuant model. Connects to the local API endpoint and streams responses.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="{MODEL}",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
)
for chunk in response:
    print(chunk.choices[0].delta.content, end="")

```

--------------------------------

### Install Dependencies for Fine-tuning

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blame/main/polarquant-finetune.md

Install necessary libraries for fine-tuning, including transformers, accelerate, safetensors, sentencepiece, scipy, torchao, peft, trl, datasets, and bitsandbytes. Use --force-reinstall for transformers to ensure the latest version.

```python
!pip install git+https://github.com/huggingface/transformers.git --force-reinstall -q
!pip install -q accelerate safetensors sentencepiece scipy torchao
!pip install -q peft trl datasets bitsandbytes
```

--------------------------------

### Install Dependencies for Evaluation Harness

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blame/main/polarquant-arena.md

Install necessary libraries for running benchmarks, including transformers, accelerate, safetensors, sentencepiece, scipy, torchao, and the lm-eval library. The transformers library is reinstalled with force-reinstall to ensure the latest version.

```python
!pip install git+https://github.com/huggingface/transformers.git --force-reinstall -q
!pip install -q accelerate safetensors sentencepiece scipy torchao
!pip install -q lm-eval  # EleutherAI evaluation harness
```

--------------------------------

### Python Client Example for PolarQuant API

Source: https://huggingface.co/caiovicentino1/polarquant-skills/commit/d553a927b27914564539f79a9632a6cbbb6fc5e6

Example Python script demonstrating how to interact with the PolarQuant serving API using the OpenAI client library. It shows how to create a client, send a chat completion request, and stream the response.

```python
from openai import OpenAI  
  
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")  
  
response = client.chat.completions.create(  
model="{MODEL}",  
messages=[{"role": "user", "content": "Hello!"}],  
stream=True,  
)  
for chunk in response:  
print(chunk.choices[0].delta.content, end="")  
```

--------------------------------

### Initialize HuggingFace Hub API

Source: https://huggingface.co/caiovicentino1/polarquant-skills/raw/main/polarquant-collection.md

Instantiate the HfApi client to interact with the HuggingFace Hub. Ensure the `huggingface_hub` library is installed.

```python
from huggingface_hub import HfApi
api = HfApi()
```

--------------------------------

### Example Workflow with PolarQuant Skills

Source: https://huggingface.co/caiovicentino1/polarquant-skills/commit/4cf0dc38b6b3651d67ca0a44e12ee4a29b52264d

Demonstrates a typical workflow using PolarQuant skills for monitoring, quantization, benchmarking, inference, and collection synchronization.

```bash
/polarquant-monitor opportunities # Find hot models
```

```bash
/polarquant google/gemma-5-48B-it # Quantize
```

```bash
/polarquant-bench google/gemma-5-48B # Benchmark vs alternatives
```

```bash
/polarquant-inference caiovicentino1/... # Create user notebook
```

```bash
/polarquant-gguf caiovicentino1/... # GGUF for ollama users
```

```bash
/polarquant-serve caiovicentino1/... # Deploy API
```

```bash
/polarquant-collection sync # Update HF collection
```

--------------------------------

### Configure and Run SFTTrainer

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blame/main/polarquant-finetune.md

Sets up the training arguments for SFTTrainer, specifying output directory, epochs, batch size, learning rate, and quantization settings (bf16). The trainer is then initialized with the model, arguments, dataset, and tokenizer, followed by starting the training process.

```python
from trl import SFTConfig, SFTTrainer

training_args = SFTConfig(
    output_dir='/content/lora_output',
    num_train_epochs=1,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=8,
    learning_rate=2e-4,
    warmup_ratio=0.1,
    logging_steps=10,
    save_steps=100,
    bf16=True,
    max_seq_length=2048,
    dataset_text_field='text',
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    tokenizer=tokenizer,
)

trainer.train()
```

--------------------------------

### Dockerfile for PolarQuant Serving

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blob/main/polarquant-serve.md

Docker image definition for serving PolarQuant models. Installs necessary libraries and copies the serving script. The CMD starts the server with a specified model and port.

```dockerfile
FROM nvidia/cuda:12.4.0-runtime-ubuntu22.04

RUN pip install torch transformers torchao safetensors scipy sentencepiece \
    fastapi uvicorn polarengine-vllm

COPY serve.py /app/serve.py
WORKDIR /app

EXPOSE 8000
CMD ["python", "serve.py", "--model", "{MODEL}", "--port", "8000"]

```

--------------------------------

### Initialize Environment and Load Model

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blame/main/polarquant-mlx.md

Sets up environment variables for Hugging Face cache and temporary directories, logs into Hugging Face Hub, and loads the model in BF16 format onto the CPU. Ensure HF_TOKEN is replaced with your actual token.

```python
#!/usr/bin/env python3
"""PolarQuant MLX Converter — {MODEL_NAME}"""

import os
os.environ['HF_HOME'] = '/Volumes/SSD Major/fish/.hf_cache'
os.environ['TMPDIR'] = '/Volumes/SSD Major/fish/.tmp'
os.makedirs(os.environ['HF_HOME'], exist_ok=True)
os.makedirs(os.environ['TMPDIR'], exist_ok=True)

import torch, torch.nn as nn, torch.nn.functional as F
import numpy as np, math, gc, time, json, shutil
from transformers import AutoModelForCausalLM, AutoTokenizer
from scipy.stats import norm
from huggingface_hub import HfApi, login

MODEL = '{owner/model-name}'
HF_TOKEN = 'YOUR_HF_TOKEN'
WORK_DIR = '/Volumes/SSD Major/fish/polar_mlx_{short_name}'
DEQUANT_DIR = os.path.join(WORK_DIR, 'dequanted_bf16')
MLX_DIR = os.path.join(WORK_DIR, 'mlx_4bit')
REPO_ID = 'caiovicentino1/{Model-Name}-PolarQuant-MLX-4bit'
BS = 128

os.makedirs(DEQUANT_DIR, exist_ok=True)
os.makedirs(MLX_DIR, exist_ok=True)
login(token=HF_TOKEN)
```

```python
model = AutoModelForCausalLM.from_pretrained(
    MODEL, dtype=torch.bfloat16, device_map='cpu', trust_remote_code=True
)
```

--------------------------------

### Initialize Environment and Load Model

Source: https://huggingface.co/caiovicentino1/polarquant-skills/resolve/main/polarquant-mlx.md?download=true

Sets up environment variables for HuggingFace cache and temporary directories, then loads the model in BF16 format onto the CPU.

```python
#!/usr/bin/env python3
"""PolarQuant MLX Converter — {MODEL_NAME}"""

import os
os.environ['HF_HOME'] = '/Volumes/SSD Major/fish/.hf_cache'
os.environ['TMPDIR'] = '/Volumes/SSD Major/fish/.tmp'
os.makedirs(os.environ['HF_HOME'], exist_ok=True)
os.makedirs(os.environ['TMPDIR'], exist_ok=True)

import torch, torch.nn as nn, torch.nn.functional as F
import numpy as np, math, gc, time, json, shutil
from transformers import AutoModelForCausalLM, AutoTokenizer
from scipy.stats import norm
from huggingface_hub import HfApi, login

MODEL = '{owner/model-name}'
HF_TOKEN = 'YOUR_HF_TOKEN'
WORK_DIR = '/Volumes/SSD Major/fish/polar_mlx_{short_name}'
DEQUANT_DIR = os.path.join(WORK_DIR, 'dequanted_bf16')
MLX_DIR = os.path.join(WORK_DIR, 'mlx_4bit')
REPO_ID = 'caiovicentino1/{Model-Name}-PolarQuant-MLX-4bit'
BS = 128

os.makedirs(DEQUANT_DIR, exist_ok=True)
os.makedirs(MLX_DIR, exist_ok=True)
login(token=HF_TOKEN)

model = AutoModelForCausalLM.from_pretrained(
    MODEL, dtype=torch.bfloat16, device_map='cpu', trust_remote_code=True
)

```

--------------------------------

### Configure and Run SFTTrainer

Source: https://huggingface.co/caiovicentino1/polarquant-skills/commit/d553a927b27914564539f79a9632a6cbbb6fc5e6

Sets up training arguments for SFTTrainer, including output directory, epochs, batch size, learning rate, and sequence length. The trainer then proceeds with the fine-tuning process using the prepared model and dataset. Ensure `trl` and `transformers` are imported.

```python
training_args = SFTConfig(
output_dir='/content/lora_output',
num_train_epochs=1,
per_device_train_batch_size=1,
gradient_accumulation_steps=8,
learning_rate=2e-4,
warmup_ratio=0.1,
logging_steps=10,
save_steps=100,
bf16=True,
max_seq_length=2048,
dataset_text_field='text',
)

trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=dataset,
tokenizer=tokenizer,
)

trainer.train()
```

--------------------------------

### Install Dependencies for GGUF Conversion

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blob/main/polarquant-gguf.md?code=true

Installs necessary libraries including transformers, accelerate, safetensors, sentencepiece, and scipy. It also clones the llama.cpp repository and installs its conversion requirements.

```python
!pip install git+https://github.com/huggingface/transformers.git --force-reinstall -q
!pip install -q accelerate safetensors sentencepiece scipy  
  
# Clone llama.cpp for conversion tools  
!git clone https://github.com/ggerganov/llama.cpp.git /content/llama.cpp  
!cd /content/llama.cpp && pip install -r requirements/requirements-convert_hf_to_gguf.txt -q  
```

--------------------------------

### Configure and Run SFT Trainer

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blob/main/polarquant-finetune.md?code=true

Sets up training arguments for the SFTTrainer, including output directory, epochs, batch size, learning rate, and sequence length. Ensure `model`, `dataset`, and `tokenizer` are properly initialized.

```python
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer

training_args = SFTConfig(
 output_dir='/content/lora_output',
 num_train_epochs=1,
 per_device_train_batch_size=1,
 gradient_accumulation_steps=8,
 learning_rate=2e-4,
 warmup_ratio=0.1,
 logging_steps=10,
 save_steps=100,
 bf16=True,
 max_seq_length=2048,
 dataset_text_field='text',
)

trainer = SFTTrainer(
 model=model,
 args=training_args,
 train_dataset=dataset,
 tokenizer=tokenizer,
)

trainer.train()
```

--------------------------------

### Configure and Run SFTTrainer

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blob/main/polarquant-finetune.md

Sets up the SFTConfig with training parameters such as output directory, epochs, batch size, learning rate, and sequence length. Initializes and runs the SFTTrainer with the model, arguments, dataset, and tokenizer.

```python
training_args = SFTConfig(
    output_dir='/content/lora_output',
    num_train_epochs=1,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=8,
    learning_rate=2e-4,
    warmup_ratio=0.1,
    logging_steps=10,
    save_steps=100,
    bf16=True,
    max_seq_length=2048,
    dataset_text_field='text',
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    tokenizer=tokenizer,
)

trainer.train()
```

--------------------------------

### Install Dependencies for PolarQuant

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blame/main/polarquant-video.md

Installs the necessary libraries for using PolarQuant with Hugging Face Transformers and Diffusers.

```python
!pip install git+https://github.com/huggingface/transformers.git --force-reinstall -q
!pip install -q diffusers accelerate safetensors scipy torchao imageio[ffmpeg] opencv-python pillow
```

--------------------------------

### Install mlx-lm Package

Source: https://huggingface.co/caiovicentino1/polarquant-skills/commit/d553a927b27914564539f79a9632a6cbbb6fc5e6

Install the mlx-lm library using pip. This is a prerequisite for using the PolarQuant models with MLX.

```bash
pip install mlx-lm
```

--------------------------------

### Install Dependencies for PolarQuant Inference

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blame/main/polarquant-inference.md

Installs the required Hugging Face Transformers library and other essential packages. Ensure to avoid flash-attn.

```python
!pip install git+https://github.com/huggingface/transformers.git --force-reinstall -q
!pip install -q accelerate safetensors sentencepiece scipy torchao gradio
```

--------------------------------

### Start Speed Benchmark

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blob/main/nemotron-offload.md

Initiates a speed benchmark for the loaded Nemotron model by preparing a long prompt for generation. This snippet is the beginning of a performance test.

```python
import time

p = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Write a detailed essay about artificial intelligence."}],
    tokenize=False, add_generation_prompt=True)

```

--------------------------------

### Install Dependencies for PolarQuant Notebook

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blame/main/polarquant.md

Installs necessary Hugging Face libraries and other dependencies like `datasets`, `accelerate`, `safetensors`, `sentencepiece`, `tiktoken`, `scipy`, and `torchao`. Use `--force-reinstall` for transformers to ensure the latest version is used. Avoid installing `flash-attn` as SDPA offers similar performance.

```python
!pip install git+https://github.com/huggingface/transformers.git --force-reinstall -q
!pip install -q datasets accelerate safetensors sentencepiece tiktoken scipy torchao

```

--------------------------------

### Colab Notebook - Restart Check

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blob/main/polarquant.md

Verifies the installed versions of the transformers and torchao libraries. This cell is crucial after installations to ensure the environment is set up correctly.

```python
import transformers; print(f'transformers: {transformers.__version__}')
import torchao; print(f'torchao: {torchao.__version__}')


```

--------------------------------

### Build and Run Llama.cpp with PQ3_0

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blame/main/polarquant-llamacpp.md

Compile Llama.cpp with CPU-only support and run inference using PQ3_0 for KV cache types.

```bash
cmake -B build -DGGML_CUDA=OFF  # CPU-only first
cmake --build build -j$(nproc)
./build/bin/llama-cli -m model.gguf --cache-type-k pq3_0 --cache-type-v pq3_0 -c 65536
```

--------------------------------

### Install Dependencies for GGUF Conversion

Source: https://huggingface.co/caiovicentino1/polarquant-skills/blob/main/polarquant-gguf.md

Installs necessary libraries including transformers, accelerate, safetensors, sentencepiece, scipy, and clones the llama.cpp repository for conversion tools.

```python
!pip install git+https://github.com/huggingface/transformers.git --force-reinstall -q
!pip install -q accelerate safetensors sentencepiece scipy

# Clone llama.cpp for conversion tools
!git clone https://github.com/ggerganov/llama.cpp.git /content/llama.cpp
!cd /content/llama.cpp && pip install -r requirements/requirements-convert_hf_to_gguf.txt -q

```
Tokens:
Raw