### Install Dependencies

Source: https://github.com/ist-daslab/sparsegpt/blob/master/demo.ipynb

Installs the required libraries for SparseGPT and transformers.

```python
!pip install -q datasets
!pip install -q transformers
```

--------------------------------

### Prune OPT Models with `opt.py` Script

Source: https://context7.com/ist-daslab/sparsegpt/llms.txt

Command-line examples for pruning OPT models using the `opt.py` script. Covers baseline, unstructured sparsity, structured sparsity, combined sparsity and quantization, and GMP comparison.

```bash
# Run dense baseline (no pruning) to get baseline perplexity
python opt.py facebook/opt-125m c4

# Prune OPT-125M to 50% unstructured sparsity
python opt.py facebook/opt-125m c4 --sparsity 0.5

# Prune to 2:4 structured sparsity (50% with hardware acceleration)
python opt.py facebook/opt-125m c4 --prunen 2 --prunem 4

# Combined 50% sparsity + 4-bit quantization
python opt.py facebook/opt-125m c4 --sparsity 0.5 --wbits 4

# Run magnitude pruning baseline (GMP) for comparison
python opt.py facebook/opt-125m c4 --sparsity 0.5 --gmp

# Prune larger OPT models
python opt.py facebook/opt-1.3b c4 --sparsity 0.5
python opt.py facebook/opt-6.7b c4 --sparsity 0.5
python opt.py facebook/opt-30b c4 --sparsity 0.5

# Save pruned model checkpoint
python opt.py facebook/opt-125m c4 --sparsity 0.5 --save ./opt-125m-sparse

# Advanced: Prune specific layers only
python opt.py facebook/opt-125m c4 --sparsity 0.5 --minlayer 0 --maxlayer 6

# Log results to Weights & Biases
python opt.py facebook/opt-125m c4 --sparsity 0.5 --log_wandb

# Custom calibration settings
python opt.py facebook/opt-125m c4 --sparsity 0.5 \
    --nsamples 256 \
    --seed 42 \
    --percdamp 0.01 \
    --blocksize 128
```

--------------------------------

### Load Data Loaders and Tokenizer

Source: https://context7.com/ist-daslab/sparsegpt/llms.txt

Utilizes `datautils.py` to get data loaders and tokenizers for various datasets and models. Ensure the correct model name is provided for tokenizer selection.

```python
from datautils import get_loaders, get_tokenizer, set_seed

# Set reproducibility seed
set_seed(42)

# Get appropriate tokenizer for model
tokenizer = get_tokenizer("facebook/opt-125m")  # AutoTokenizer
tokenizer = get_tokenizer("meta-llama/Llama-2-7b")  # LlamaTokenizer

# Load calibration and test data
# Returns (trainloader, testloader) tuple
dataloader, testloader = get_loaders(
    name="c4",              # Dataset: 'wikitext2', 'ptb', or 'c4'
    nsamples=128,           # Number of calibration samples
    seed=0,                 # Random seed
    seqlen=2048,            # Sequence length
    model="facebook/opt-125m"  # Model name for tokenizer
)

# Iterate through calibration batches
for inp, tar in dataloader:
    # inp: input token ids [1, seqlen]
    # tar: target ids (same as inp with -100 masking)
    print(inp.shape)  # torch.Size([1, 2048])

# Test data for perplexity evaluation
test_ids = testloader.input_ids  # Full tokenized test set
```

--------------------------------

### Initialize and Prune with SparseGPT Core Class

Source: https://context7.com/ist-daslab/sparsegpt/llms.txt

Demonstrates initializing the SparseGPT wrapper for a linear layer, accumulating calibration data, and performing unstructured or structured pruning. Includes steps for freeing GPU memory.

```python
import torch
import torch.nn as nn
from sparsegpt import SparseGPT
from quant import Quantizer

# Initialize SparseGPT wrapper for a linear layer
linear_layer = nn.Linear(768, 768).cuda()
sparse_gpt = SparseGPT(linear_layer)

# Collect calibration data (128 samples typical)
# Each batch should be input/output pairs from the layer
for batch_idx in range(128):
    # inp: layer input tensor, out: layer output tensor
    inp = torch.randn(1, 2048, 768).cuda()  # [batch, seq_len, hidden_dim]
    out = linear_layer(inp)
    sparse_gpt.add_batch(inp, out)

# Prune to 50% sparsity (unstructured)
sparse_gpt.fasterprune(
    sparsity=0.5,      # Target 50% zeros
    prunen=0,          # N for N:M pruning (0 = unstructured)
    prunem=0,          # M for N:M pruning (0 = unstructured)
    blocksize=128,     # Block size for processing
    percdamp=0.01      # Hessian dampening factor
)

# For 2:4 structured sparsity (NVIDIA Ampere optimized)
sparse_gpt_structured = SparseGPT(linear_layer)
# ... add_batch calls ...
sparse_gpt_structured.fasterprune(
    sparsity=0,        # Ignored when using N:M
    prunen=2,          # Keep 2 weights
    prunem=4           # Out of every 4 weights
)

# For combined sparsity + quantization
sparse_gpt_quant = SparseGPT(linear_layer)
sparse_gpt_quant.quantizer = Quantizer()
sparse_gpt_quant.quantizer.configure(
    bits=4,            # 4-bit quantization
    perchannel=True,   # Per-channel scales
    sym=False,         # Asymmetric quantization
    mse=False          # Use min-max (not MSE optimization)
)
# ... add_batch calls ...
sparse_gpt_quant.fasterprune(sparsity=0.5)

# Free GPU memory when done
sparse_gpt.free()
```

--------------------------------

### Configure and Use Quantizer Module

Source: https://context7.com/ist-daslab/sparsegpt/llms.txt

Configures and uses the `Quantizer` class for weight quantization, supporting various options like per-channel, asymmetric, and MSE optimization. This module is designed for integration with SparseGPT.

```python
import torch
from quant import Quantizer, quantize

# Quantizer module for integration with SparseGPT
quantizer = Quantizer()

# Configure 4-bit asymmetric per-channel quantization
quantizer.configure(
    bits=4,             # Number of bits (4, 8, etc.)
    perchannel=True,    # Per-channel vs per-tensor scales
    sym=False,          # Symmetric vs asymmetric
    mse=False,          # MSE optimization for scales
    norm=2.4,           # Norm for MSE optimization
    grid=100,           # Grid search resolution
    maxshrink=0.8,      # Max scale shrinkage for MSE
    grouprows=1         # Group rows for sub-channel quant
)

# Find quantization parameters from weights
weight = torch.randn(768, 768).cuda()
quantizer.find_params(weight, weight=True)

print(f"Scale shape: {quantizer.scale.shape}")
print(f"Zero point shape: {quantizer.zero.shape}")
print(f"Max quantized value: {quantizer.maxq}")

# Apply quantization
quantized_weight = quantizer.quantize(weight)

# Check if quantizer is ready
print(f"Quantizer ready: {quantizer.ready()}")
print(f"Quantizer enabled: {quantizer.enabled()}")
```

--------------------------------

### Navigate to SparseGPT Directory

Source: https://github.com/ist-daslab/sparsegpt/blob/master/demo.ipynb

Changes the current working directory to the cloned sparsegpt directory.

```python
%cd sparsegpt
```

--------------------------------

### Create Directory for Pruned Models

Source: https://github.com/ist-daslab/sparsegpt/blob/master/demo.ipynb

Creates a directory named 'sparse_opt' to store the pruned model files.

```bash
!mkdir -p sparse_opt
```

--------------------------------

### Load Dense and Sparse Models

Source: https://github.com/ist-daslab/sparsegpt/blob/master/demo.ipynb

Loads the original dense OPT model and the pruned sparse OPT model, along with the tokenizer. Models are moved to the specified device.

```python
# load dense model
model_dn = OPTForCausalLM.from_pretrained('facebook/opt-125m', torch_dtype='auto').to(device)
# load sparse model
model_sp = OPTForCausalLM.from_pretrained('sparse_opt/opt-125m', torch_dtype='auto').to(device)
# init tokenizer
tokenizer = AutoTokenizer.from_pretrained('facebook/opt-125m')
```

--------------------------------

### Run GMP Baseline Comparison

Source: https://context7.com/ist-daslab/sparsegpt/llms.txt

Compares the pruning results against the GMP baseline. This is useful for evaluating the effectiveness of SparseGPT.

```bash
python bloom.py bigscience/bloom-560m c4 --sparsity 0.5 --gmp
```

--------------------------------

### Clone SparseGPT Repository

Source: https://github.com/ist-daslab/sparsegpt/blob/master/demo.ipynb

Clones the SparseGPT repository from GitHub to the local environment.

```bash
!git clone https://github.com/IST-DASLab/sparsegpt
```

--------------------------------

### Prune OPT Model with SparseGPT

Source: https://github.com/ist-daslab/sparsegpt/blob/master/demo.ipynb

Applies SparseGPT to prune the 'facebook/opt-125m' model to 0.5 unstructured sparsity. The pruned model is saved to 'sparse_opt/opt-125m'. This command also prints perplexity on specified benchmarks.

```bash
!python opt.py facebook/opt-125m c4 --sparsity 0.5 --save sparse_opt/opt-125m
```

--------------------------------

### Combine Sparsity and 4-bit Quantization for LLaMA

Source: https://context7.com/ist-daslab/sparsegpt/llms.txt

Applies 50% sparsity and 4-bit quantization to LLaMA models. This combines two compression techniques for greater efficiency.

```bash
python llama.py /path/to/llama-7b-hf c4 --sparsity 0.5 --wbits 4
```

--------------------------------

### Prune OPT Models with SparseGPT (50% Sparsity + 4-bit Quantization)

Source: https://github.com/ist-daslab/sparsegpt/blob/master/README.md

Combine 50% sparsity with 4-bit weight quantization for OPT models using SparseGPT on the C4 dataset. This showcases a more aggressive compression strategy.

```python
python opt.py facebook/opt-125m c4 --sparsity .5 --wbits 4
```

--------------------------------

### Set Device

Source: https://github.com/ist-daslab/sparsegpt/blob/master/demo.ipynb

Sets the computation device to 'cuda' for GPU acceleration.

```python
device = 'cuda'
```

--------------------------------

### Run Dense Baseline for OPT Models

Source: https://github.com/ist-daslab/sparsegpt/blob/master/README.md

Execute the dense baseline model for OPT on the C4 dataset. This serves as a reference point before applying pruning techniques.

```python
python opt.py facebook/opt-125m c4
```

--------------------------------

### Define Input Text

Source: https://github.com/ist-daslab/sparsegpt/blob/master/demo.ipynb

Sets the input text prompt for text generation.

```python
input_text = "It takes a great deal of bravery"
```

--------------------------------

### Apply 2:4 Structured Sparsity to BLOOM

Source: https://context7.com/ist-daslab/sparsegpt/llms.txt

Apply structured sparsity with a 2:4 ratio to BLOOM models using the `bloom.py` script. This is useful for hardware-aware sparsity.

```bash
python bloom.py bigscience/bloom-560m c4 --prunen 2 --prunem 4
```

--------------------------------

### Prune OPT Models with SparseGPT (2:4 Sparsity)

Source: https://github.com/ist-daslab/sparsegpt/blob/master/README.md

Achieve full 2:4 sparsity in OPT models using SparseGPT on the C4 dataset. This specific configuration targets a structured sparsity pattern.

```python
python opt.py facebook/opt-125m c4 --prunen 2 --prunem 4
```

--------------------------------

### Prune OPT Models with SparseGPT (50% Sparsity)

Source: https://github.com/ist-daslab/sparsegpt/blob/master/README.md

Prune OPT models to 50% uniform sparsity using the SparseGPT algorithm on the C4 dataset. This demonstrates the core pruning functionality.

```python
python opt.py facebook/opt-125m c4 --sparsity .5
```

--------------------------------

### Run Magnitude Baseline for OPT Models

Source: https://github.com/ist-daslab/sparsegpt/blob/master/README.md

Apply the magnitude baseline pruning to OPT models on the C4 dataset with 50% sparsity. This method prunes weights based on their magnitude.

```python
python opt.py facebook/opt-125m c4 --sparsity .5 --gmp
```

--------------------------------

### Load Transformers and Model Classes

Source: https://github.com/ist-daslab/sparsegpt/blob/master/demo.ipynb

Imports necessary classes from the transformers library for loading models and tokenizers.

```python
from transformers import AutoTokenizer, OPTForCausalLM
```

--------------------------------

### Log Evaluation Results to W&B

Source: https://github.com/ist-daslab/sparsegpt/blob/master/README.md

Enable logging of evaluation results to Weights & Biases (W&B) by using the `--log_wandb` flag. This is useful for tracking experiments and comparing results.

```python
python opt.py facebook/opt-125m c4 --sparsity .5 --log_wandb
```

--------------------------------

### Sparsify LLaMA Models with SparseGPT

Source: https://github.com/ist-daslab/sparsegpt/blob/master/README.md

Use SparseGPT to sparsify LLaMA models, with the script accepting the HuggingFace weights location and the C4 dataset, targeting 50% sparsity.

```python
python llama.py LLAMA_HF_WEIGHTS_LOCATION c4 --sparsity 0.5
```

--------------------------------

### Apply 2:4 Structured Sparsity to LLaMA

Source: https://context7.com/ist-daslab/sparsegpt/llms.txt

Applies structured sparsity with a 2:4 ratio to LLaMA models. This is useful for hardware-aware sparsity.

```bash
python llama.py /path/to/llama-7b-hf c4 --prunen 2 --prunem 4
```

--------------------------------

### Generate Text with Dense Model

Source: https://github.com/ist-daslab/sparsegpt/blob/master/demo.ipynb

Generates text using the original dense OPT model.

```python
output_ids = model_dn.generate(input_ids)
```

--------------------------------

### Generate Text with Sparse Model

Source: https://github.com/ist-daslab/sparsegpt/blob/master/demo.ipynb

Generates text using the pruned sparse OPT model.

```python
output_ids = model_sp.generate(input_ids)
```

--------------------------------

### Prune Full BLOOM-176B Model

Source: https://context7.com/ist-daslab/sparsegpt/llms.txt

This command prunes the full BLOOM-176B model to 50% sparsity. Ensure sufficient resources are available for this large model.

```bash
python bloom.py bigscience/bloom c4 --sparsity 0.5
```

--------------------------------

### Sparsify BLOOM Models with SparseGPT

Source: https://github.com/ist-daslab/sparsegpt/blob/master/README.md

Apply SparseGPT to sparsify BLOOM models, demonstrated here with BLOOM-176B on the C4 dataset with 50% sparsity. Note that some features might be OPT-specific.

```python
python bloom.py bigscience/bloom c4 --sparsity .5
```

--------------------------------

### Prune BLOOM Models to 50% Sparsity

Source: https://context7.com/ist-daslab/sparsegpt/llms.txt

Use the `bloom.py` script to prune BLOOM models to a specified sparsity level. This command is suitable for general sparsity application.

```bash
python bloom.py bigscience/bloom-560m c4 --sparsity 0.5
```

```bash
python bloom.py bigscience/bloom-1b1 c4 --sparsity 0.5
```

```bash
python bloom.py bigscience/bloom-3b c4 --sparsity 0.5
```

```bash
python bloom.py bigscience/bloom-7b1 c4 --sparsity 0.5
```

--------------------------------

### Enable True Sequential Mode for LLaMA Pruning

Source: https://context7.com/ist-daslab/sparsegpt/llms.txt

Enables true sequential processing for LLaMA pruning, respecting the internal layer structure of attention and MLP blocks. This ensures more accurate pruning.

```bash
python llama.py /path/to/llama-7b-hf c4 --sparsity 0.5 --true-sequential
```

--------------------------------

### Print Dense Model Generation

Source: https://github.com/ist-daslab/sparsegpt/blob/master/demo.ipynb

Decodes and prints the generated text from the dense model, skipping special tokens.

```python
print(tokenizer.decode(output_ids[0].cpu(), skip_special_tokens=True))
```

--------------------------------

### Find Prunable Layers in PyTorch

Source: https://context7.com/ist-daslab/sparsegpt/llms.txt

Identifies prunable layers within a PyTorch model. Useful for understanding model structure before pruning.

```python
layers = find_layers(layer)
for name, module in layers.items():
    print(f"{name}: {module.weight.shape}")
```

--------------------------------

### Basic Weight Quantization

Source: https://context7.com/ist-daslab/sparsegpt/llms.txt

Performs basic quantization of a weight tensor using provided scale, zero point, and maximum quantized value. This is a fundamental operation for quantization.

```python
import torch
from quant import Quantizer, quantize

# Basic quantization function
weight = torch.randn(768, 768)
scale = torch.tensor([0.01])
zero = torch.tensor([128.0])
maxq = torch.tensor([255])  # 8-bit

quantized = quantize(weight, scale, zero, maxq)
```

--------------------------------

### Save Pruned LLaMA Model

Source: https://context7.com/ist-daslab/sparsegpt/llms.txt

Saves the pruned LLaMA model checkpoint to a specified directory. Use this after applying sparsity to persist the changes.

```bash
python llama.py /path/to/llama-7b-hf c4 --sparsity 0.5 --save ./llama-7b-sparse
```

--------------------------------

### Save Sparsified Model Checkpoint

Source: https://github.com/ist-daslab/sparsegpt/blob/master/README.md

Optionally save the sparsified model checkpoint by specifying a path using the `--save` flag. This allows for later use or further analysis of the pruned model.

```python
python opt.py facebook/opt-125m c4 --sparsity .5 --save /path/to/save/checkpoint
```

--------------------------------

### Tokenize Input Text

Source: https://github.com/ist-daslab/sparsegpt/blob/master/demo.ipynb

Converts the input text into token IDs and moves them to the specified device.

```python
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to(device)
```

--------------------------------

### Save Pruned BLOOM Checkpoint

Source: https://context7.com/ist-daslab/sparsegpt/llms.txt

Saves the pruned BLOOM model checkpoint to a specified directory. Use this after applying sparsity to persist the changes.

```bash
python bloom.py bigscience/bloom-560m c4 --sparsity 0.5 --save ./bloom-sparse
```

--------------------------------

### Prune LLaMA Models

Source: https://context7.com/ist-daslab/sparsegpt/llms.txt

Prunes LLaMA models using the `llama.py` script. Requires converted HuggingFace weights. This command applies 50% sparsity.

```bash
python llama.py /path/to/llama-7b-hf c4 --sparsity 0.5
```

```bash
python llama.py /path/to/llama-13b-hf c4 --sparsity 0.5
```

```bash
python llama.py /path/to/llama-30b-hf c4 --sparsity 0.5
```

```bash
python llama.py /path/to/llama-65b-hf c4 --sparsity 0.5
```

--------------------------------

### Identify Model Layers

Source: https://context7.com/ist-daslab/sparsegpt/llms.txt

Uses `modelutils.py` to find all Linear and Conv2d layers within a given PyTorch module. This is a utility for identifying prunable layers in transformer models.

```python
import torch
import torch.nn as nn
from modelutils import find_layers, DEV
from transformers import OPTForCausalLM

# Default CUDA device
print(f"Device: {DEV}")  # cuda:0

# Find all Linear and Conv2d layers in a module
model = OPTForCausalLM.from_pretrained("facebook/opt-125m")
layer = model.model.decoder.layers[0]  # First transformer block

# Example usage (assuming find_layers is called elsewhere to get layers)
```

--------------------------------

### SparseGPT Paper Citation

Source: https://github.com/ist-daslab/sparsegpt/blob/master/README.md

BibTeX entry for citing the SparseGPT paper. Include this in your academic work if you found the research useful.

```bibtex
@article{frantar-sparsegpt,
  title={{SparseGPT}: Massive Language Models Can Be Accurately Pruned in One-Shot},
  author={Elias Frantar and Dan Alistarh},
  year={2023},
  journal={arXiv preprint arXiv:2301.00774}
}
```

--------------------------------

### Find Specific Layer Types in PyTorch

Source: https://context7.com/ist-daslab/sparsegpt/llms.txt

Filters layers by type (e.g., nn.Linear) and name prefix. Useful for targeted pruning or analysis.

```python
custom_layers = find_layers(
    layer,
    layers=[nn.Linear],  # Only Linear layers
    name=''
)
```

--------------------------------

### Prune Specific Layer Range in BLOOM

Source: https://context7.com/ist-daslab/sparsegpt/llms.txt

Prunes a specific range of layers within a BLOOM model. Use `minlayer` and `maxlayer` to define the target range.

```bash
python bloom.py bigscience/bloom-560m c4 --sparsity 0.5 \
    --minlayer 0 --maxlayer 12
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.