### Install Example Requirements

Source: https://github.com/huggingface/optimum-intel/blob/main/README.md

Navigate to an example's directory and install its specific requirements using pip. This is necessary before running any of the provided examples.

```sh
cd <example-folder>
pip install -r requirements.txt
```

--------------------------------

### Install Optimum-Intel and Evaluate

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/sentence_transformer_quantization.ipynb

Install the necessary libraries for Optimum-Intel with OpenVINO support and the evaluate library.

```python
# %pip install "optimum-intel[openvino]" evaluate
```

--------------------------------

### Install Optimum Intel from Source

Source: https://github.com/huggingface/optimum-intel/blob/main/docs/source/installation.mdx

Install Optimum Intel directly from its GitHub repository for the latest development version.

```bash
pip install optimum-intel@git+https://github.com/huggingface/optimum-intel.git
```

--------------------------------

### Install Optimum with OpenVINO and NNCF

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/demos/quantized_generation_demo.ipynb

Installs the necessary libraries for using Optimum with OpenVINO and NNCF. Ensure you have the latest GPU drivers installed.

```python
# ! pip install optimum[openvino,nncf] torch==2.2.2
```

--------------------------------

### Install Optimum Intel

Source: https://github.com/huggingface/optimum-intel/blob/main/docs/source/installation.mdx

Install the latest release of Optimum Intel with its required dependencies using pip.

```bash
pip install -U optimum-intel
```

--------------------------------

### Install Optimum Intel with OpenVINO and NNCF

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/question_answering_quantization.ipynb

Installs the necessary libraries for quantizing Hugging Face models with OpenVINO and NNCF. Use this command to set up your environment.

```python
# %pip install "optimum-intel[openvino,nncf]" datasets evaluate[evaluator] ipywidgets
```

--------------------------------

### Install Gradio Dependency

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/demos/quantized_generation_demo.ipynb

Run this command to install the Gradio library, which is required for the chatbot interface.

```python
# ! pip install gradio
```

--------------------------------

### Install Diffusers

Source: https://github.com/huggingface/optimum-intel/blob/main/docs/source/openvino/inference.mdx

Install the 🤗 Diffusers library using pip.

```bash
pip install diffusers
```

--------------------------------

### Install Optimum with OpenVINO support

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/optimum_openvino_inference.ipynb

Install the necessary dependencies for using Optimum with OpenVINO. This command includes optional packages like ipywidgets, pillow, torchaudio, soundfile, and librosa for specific functionalities.

```python
# %pip install optimum[openvino] ipywidgets pillow torchaudio soundfile librosa
```

--------------------------------

### Install Optimum Intel for OpenVINO

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/stable_diffusion_hybrid_quantization.ipynb

Installs the necessary Optimum Intel package with OpenVINO support, along with datasets and ipywidgets for the notebook environment.

```python
# %pip install "optimum-intel[openvino]" datasets ipywidgets
```

--------------------------------

### Install Dependencies for Optimum Intel

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/vision_language_quantization.ipynb

Installs necessary libraries including Optimum Intel with OpenVINO support, datasets, num2words, torchvision, transformers, and the optimum-benchmark package.

```python
! pip install "optimum-intel[openvino]" datasets num2words torchvision transformers==4.52.*
! pip install git+https://github.com/huggingface/optimum-benchmark.git
```

--------------------------------

### Launch the Gradio Demo

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/demos/quantized_generation_demo.ipynb

This command starts the Gradio application, making the chatbot interface accessible.

```python
demo.launch()
```

--------------------------------

### Tokenizer Output Example

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/question_answering_quantization.ipynb

Demonstrates the output of the tokenizer for the input string 'hello world!', showing the generated input IDs, token type IDs, and attention mask.

```text
Output:
{'input_ids': [101, 7592, 2088, 999, 102], 'token_type_ids': [0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1]}
```

--------------------------------

### Prepare Context and Question for QA

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/question_answering_quantization.ipynb

Define the context and question for the question-answering task. This setup is used to test the inference pipelines.

```python
context = validation_examples[200]["context"]
question = "Who won the game?"
print(context)
```

--------------------------------

### Load Dataset and Initialize Recorder

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/demos/quantized_generation_demo.ipynb

Loads the HumanEval dataset and initializes the AcceptanceRateRecorder for the stateless model. This setup is used to evaluate code generation performance.

```python
from datasets import load_dataset
from tqdm import tqdm


dataset_name = "openai_humaneval"
dataset_subset_name = None
field_name = "prompt"
prompt_template = "{text}"
dataset = load_dataset(dataset_name, dataset_subset_name, split="test")[field_name]
samples_number = 30
with AcceptanceRateRecorder(stateless_model) as ar_recorder:
```

--------------------------------

### Dataset Item Example

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/question_answering_quantization.ipynb

An example of a single item from the SQuAD dataset, showing its structure including id, title, context, question, and answers.

```python
{
 'id': '570e53690b85d914000d7e3c',
 'title': 'Melbourne',
 'context': "Melbourne is experiencing high population growth, generating high demand for housing. This housing boom has increased house prices and rents, as well as the availability of all types of housing. Subdivision regularly occurs in the outer areas of Melbourne, with numerous developers offering house and land packages. However, after 10 years[when?] of planning policies to encourage medium-density and high-density development in existing areas with greater access to public transport and other services, Melbourne's middle and outer-ring suburbs have seen significant brownfields redevelopment.",
 'question': 'What effect has the housing boom had on house prices and rents?',
 'answers': {'text': ['increased'], 'answer_start': [108]}
}
```

--------------------------------

### Question Answering with OVModelForQuestionAnswering

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/optimum_openvino_inference.ipynb

This example shows how to set up a question-answering pipeline using `OVModelForQuestionAnswering`. For models requiring static shapes, like BERT-based architectures, specify `max_seq_len`, `padding`, and `truncation` in the pipeline call.

```python
from transformers import AutoTokenizer, pipeline

from optimum.intel import OVModelForQuestionAnswering


# Load the model and tokenizer saved in Part 1 of this notebook. Or use the line below to load them from the hub
```

--------------------------------

### Text-to-Image Generation with OVStableDiffusionXLPipeline

Source: https://github.com/huggingface/optimum-intel/blob/main/docs/source/openvino/tutorials/diffusers.mdx

Generates an image from a text prompt using the OVStableDiffusionXLPipeline. This is a basic text-to-image example for SDXL models.

```python
from optimum.intel import OVStableDiffusionXLPipeline

model_id = "stabilityai/stable-diffusion-xl-base-1.0"
base = OVStableDiffusionXLPipeline.from_pretrained(model_id)
prompt = "train station by Caspar David Friedrich"
image = base(prompt).images[0]
image.save("train_station.png")
```

--------------------------------

### Example Docstring for Model Inference

Source: https://github.com/huggingface/optimum-intel/blob/main/docs/README.md

Demonstrates how to use a Wav2Vec2 model for speech transcription within a docstring. Ensure all necessary libraries are imported and the audio is processed correctly.

```python
>>> from transformers import Wav2Vec2Processor, Wav2Vec2ForCTC
>>> from datasets import load_dataset
>>> import torch

>>> dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation")
>>> dataset = dataset.sort("id")
>>> sampling_rate = dataset.features["audio"].sampling_rate

>>> processor = Wav2Vec2Processor.from_pretrained("facebook/wav2vec2-base-960h")
>>> model = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-base-960h")

>>> # audio file is decoded on the fly
>>> inputs = processor(dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_ids = torch.argmax(logits, dim=-1)

>>> # transcribe speech
>>> transcription = processor.batch_decode(predicted_ids)
>>> transcription[0]
'MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL'
```

--------------------------------

### Load and Save Quantized Diffusion Pipeline (INT8)

Source: https://github.com/huggingface/optimum-intel/blob/main/docs/source/openvino/optimization.mdx

Load a diffusion pipeline with INT8 quantization and save it. This example uses OVWeightQuantizationConfig for weight quantization.

```python
OVDiffusionPipeline.from_pretrained('dreamlike-art/dreamlike-anime-1.0', quantization_config=OVWeightQuantizationConfig(bits=8)).save_pretrained('save_dir')
```

--------------------------------

### Load and Inspect Dataset

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/question_answering_quantization.ipynb

Loads a dataset from the Hugging Face Hub and displays a random item. Ensure the 'datasets' library is installed.

```python
dataset = datasets.load_dataset(DATASET_NAME)
dataset["train"][31415]
```

--------------------------------

### Load and Inspect Dataset

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/stable_diffusion_hybrid_quantization.ipynb

Loads a streaming dataset from the Hugging Face Hub and prints a random item to inspect its structure. Ensure the 'datasets' library is installed.

```python
import datasets

DATASET_NAME = "jxie/coco_captions"
dataset = datasets.load_dataset(DATASET_NAME, split="train", streaming=True).shuffle(seed=42)
print(next(iter(dataset)))
```

--------------------------------

### Upgrade Pip

Source: https://github.com/huggingface/optimum-intel/blob/main/docs/source/installation.mdx

Upgrade pip to the latest version within a virtual environment before installing Optimum Intel.

```bash
python -m pip install --upgrade pip
```

--------------------------------

### Token Classification Pipeline with OVModel

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/optimum_openvino_inference.ipynb

Use this snippet to initialize and run a token classification pipeline with an OpenVINO-optimized model. Ensure you have the necessary transformers and optimum.intel libraries installed.

```python
from transformers import AutoTokenizer, pipeline

from optimum.intel import OVModelForTokenClassification


model_id = "helenai/dslim-bert-base-NER-ov-fp32"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = OVModelForTokenClassification.from_pretrained(model_id)
ov_pipe = pipeline("token-classification", model=model, tokenizer=tokenizer)
result = ov_pipe("My name is Wolfgang and I live in Berlin")
for item in result:
    print(item)
```

--------------------------------

### Prepare Inputs for Static Shapes

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/optimum_openvino_inference.ipynb

When using static shapes, ensure inputs match the defined shape by padding shorter sequences or truncating longer ones. This example shows how to configure the tokenizer for this purpose.

```python
inputs = tokenizer(question, text, return_tensors="pt", padding="max_length", max_length=128, truncation=True)
```

--------------------------------

### Compare FP32 and INT8 QA Predictions

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/question_answering_quantization.ipynb

Iterates through validation examples, generates answers using FP32 and INT8 QA pipelines, computes F1 scores, and stores results where scores differ. Requires 'evaluate', 'pandas', and pre-defined 'hf_qa_pipeline', 'ov_qa_pipeline_ptq', 'validation_examples', 'VERSION_2_WITH_NEGATIVE'.

```python
results = []
metric = evaluate.load("squad_v2" if VERSION_2_WITH_NEGATIVE else "squad")

for item in validation_examples:
    id, title, context, question, answers = item.values()
    fp32_answer = hf_qa_pipeline(question, context)["answer"]
    int8_answer = ov_qa_pipeline_ptq(question, context)["answer"]

    references = [{"id": id, "answers": answers}]
    fp32_predictions = [{"id": id, "prediction_text": fp32_answer}]
    int8_predictions = [{"id": id, "prediction_text": int8_answer}]

    fp32_score = round(metric.compute(references=references, predictions=fp32_predictions)["f1"], 2)
    int8_score = round(metric.compute(references=references, predictions=int8_predictions)["f1"], 2)

    if int8_score != fp32_score:
        results.append((question, answers["text"], fp32_answer, fp32_score, int8_answer, int8_score))
```

```python
pd.set_option("display.max_colwidth", None)
pd.DataFrame(
    results,
    columns=["Question", "Answer", "FP32 answer", "FP32 F1", "INT8 answer", "INT8 F1"],
)
```

--------------------------------

### Generate Documentation

Source: https://github.com/huggingface/optimum-intel/blob/main/docs/README.md

Run this command from the root of the optimum-intel repository to generate HTML documentation files. The BUILD_DIR argument can be adapted to any temporary folder.

```bash
make doc BUILD_DIR=intel-doc-build VERSION=main
```

--------------------------------

### Benchmark Configuration and Launch

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/vision_language_quantization.ipynb

Sets up benchmark configurations for PyTorch and OpenVINO (including 8-bit quantization) and launches the benchmarks. Requires `optimum-benchmark` and `openvino` libraries. Ensure `model_id` is defined.

```python
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from huggingface_hub import create_repo, upload_file
from optimum_benchmark import (
    Benchmark,
    BenchmarkConfig,
    BenchmarkReport,
    InferenceConfig,
    OpenVINOConfig,
    ProcessConfig,
    PyTorchConfig,
)
from optimum_benchmark.logging_utils import setup_logging
from openvino import Core

setup_logging(level="INFO", prefix="MAIN-PROCESS")

launcher_config = ProcessConfig()
scenario_config = InferenceConfig(
    memory=True,
    latency=True,
    generate_kwargs={"max_new_tokens": 16, "min_new_tokens": 16},
    input_shapes={"batch_size": 1, "sequence_length": 16, "num_images": 1},
)

configs = {
    "pytorch": PyTorchConfig(device="cpu", model=model_id, no_weights=True),
    "openvino": OpenVINOConfig(device="cpu", model=model_id, no_weights=True),
    "openvino-8bit-woq": OpenVINOConfig(
        device="cpu",
        model=model_id,
        no_weights=True,
        quantization_config={"bits": 8, "num_samples": 1, "weight_only": True},
    ),
}

for config_name, backend_config in configs.items():
    benchmark_config = BenchmarkConfig(
        name=f"{config_name}",
        launcher=launcher_config,
        scenario=scenario_config,
        backend=backend_config,
    )
    benchmark_report = Benchmark.launch(benchmark_config)
    benchmark_report.save_json(f"{config_name}_report.json")
    benchmark_config.save_json(f"{config_name}_config.json")

reports = {}
for config_name in configs.keys():
    reports[config_name] = BenchmarkReport.from_json(f"{config_name}_report.json")

# Plotting results
_, ax = plt.subplots()
ax.boxplot(
    [reports[config_name].prefill.latency.values for config_name in reports.keys()],
    tick_labels=reports.keys(),
    showfliers=False,
)
plt.xticks(rotation=10)
ax.set_ylabel("Latency (s)")
ax.set_xlabel("Configurations")
ax.set_title("Prefill Latencies")
plt.savefig("prefill_latencies_boxplot.png")

_, ax = plt.subplots()
ax.bar(
    list(reports.keys()),
    [reports[config_name].decode.throughput.value for config_name in reports.keys()],
    color=["C0", "C1", "C2", "C3", "C4", "C5"],
)
plt.xticks(rotation=10)
ax.set_xlabel("Configurations")
ax.set_title("Decoding Throughput")
ax.set_ylabel("Throughput (tokens/s)")
plt.savefig("decode_throughput_barplot.png")
```

--------------------------------

### Token Classification Output Example

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/optimum_openvino_inference.ipynb

This is an example output from the token classification pipeline, showing detected entities, their scores, and positions within the text.

```python
{'entity': 'B-PER', 'score': np.float32(0.99901474), 'index': 4, 'word': 'Wolfgang', 'start': 11, 'end': 19}
{'entity': 'B-LOC', 'score': np.float32(0.999645), 'index': 9, 'word': 'Berlin', 'start': 34, 'end': 40}
```

--------------------------------

### Load Draft Model with Optimum Intel

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/demos/quantized_generation_demo.ipynb

Configure loading parameters including device, OpenVINO runtime configuration, and quantization settings. It checks if the model was previously exported and loads or exports it accordingly.

```python
# Load kwargs
load_kwargs = {
    "device": device,
    "ov_config": {
        "PERFORMANCE_HINT": "LATENCY",
        "INFERENCE_PRECISION_HINT": precision,
        "CACHE_DIR": os.path.join(save_name, "model_cache"),  # OpenVINO will use this directory as cache
    },
    "compile": False,
    "quantization_config": quantization_config
}

# Check whether the model was already exported
saved = os.path.exists(save_name)

asst_model = OVModelForCausalLM.from_pretrained(
    model_name if not saved else save_name,
    export=not saved,
    stateful=False,
    **load_kwargs,
)
```

--------------------------------

### Optimum CLI OpenVINO Export Help

Source: https://github.com/huggingface/optimum-intel/blob/main/docs/source/openvino/export.mdx

This displays the help message for the `optimum-cli export openvino` command, detailing all available arguments for model export and quantization.

```text
usage: optimum-cli export openvino [-h] -m MODEL [--task TASK] [--framework {pt}]
                                   [--trust-remote-code] [--weight-format {fp32,fp16,int8,int4,mxfp4,nf4,cb4}]
                                   [--quant-mode {int8,f8e4m3,f8e5m2,cb4_f8e4m3,int4_f8e4m3,int4_f8e5m2}]
                                   [--library {transformers,diffusers,timm,sentence_transformers,open_clip}]
                                   [--cache_dir CACHE_DIR] [--pad-token-id PAD_TOKEN_ID] [--ratio RATIO] [--sym]
                                   [--group-size GROUP_SIZE] [--backup-precision {none,int8_sym,int8_asym}]
                                   [--dataset DATASET] [--all-layers] [--awq] [--scale-estimation] [--gptq]
                                   [--lora-correction] [--sensitivity-metric SENSITIVITY_METRIC]
                                   [--quantization-statistics-path QUANTIZATION_STATISTICS_PATH]
                                   [--num-samples NUM_SAMPLES] [--disable-stateful] [--disable-convert-tokenizer]
                                   [--smooth-quant-alpha SMOOTH_QUANT_ALPHA]
                                   output

optional arguments:
  -h, --help            show this help message and exit

Required arguments:
  -m MODEL, --model MODEL
                        Model ID on huggingface.co or path on disk to load model from.
  output                Path indicating the directory where to store the generated OV model.

Optional arguments:
  --task TASK           The task to export the model for. If not specified, the task will be auto-inferred based on
                        the model. Available tasks depend on the model, but are among: ['image-to-image',
                        'image-segmentation', 'inpainting', 'sentence-similarity', 'text-to-audio', 'image-to-text',
                        'automatic-speech-recognition', 'token-classification', 'text-to-image', 'audio-classification',
                        'feature-extraction', 'semantic-segmentation', 'masked-im', 'audio-xvector',
                        'audio-frame-classification', 'text2text-generation', 'multiple-choice', 'depth-estimation',
                        'image-classification', 'fill-mask', 'zero-shot-object-detection', 'object-detection',
                        'question-answering', 'zero-shot-image-classification', 'mask-generation', 'text-generation',
                        'text-classification']. For decoder models, use 'xxx-with-past' to export the model using past
                        key values in the decoder.
  --framework {pt}   The framework to use for the export. Defaults to 'pt' for PyTorch.
  --trust-remote-code   Allows to use custom code for the modeling hosted in the model repository. This option should
                        only be set for repositories you trust and in which you have read the code, as it will execute
                        on your local machine arbitrary code present in the model repository.
  --weight-format {fp32,fp16,int8,int4,mxfp4,nf4,cb4}
                        The weight format of the exported model. Option 'cb4' represents a codebook with 16
                        fixed fp8 values in E4M3 format.
  --quant-mode {int8,f8e4m3,f8e5m2,cb4_f8e4m3,int4_f8e4m3,int4_f8e5m2}
                        Quantization precision mode. This is used for applying full model quantization including
                        activations.
  --library {transformers,diffusers,timm,sentence_transformers,open_clip}
                        The library used to load the model before export. If not provided, will attempt to infer the
                        local checkpoint's library
  --cache_dir CACHE_DIR
                        The path to a directory in which the downloaded model should be cached if the standard cache
                        should not be used.
```

--------------------------------

### Full Quantization (8-bit, Data-Aware) CLI

Source: https://github.com/huggingface/optimum-intel/blob/main/docs/source/openvino/optimization.mdx

Use this CLI command for 8-bit full quantization with data-aware calibration. The dataset is required for this process.

```bash
optimum-cli export openvino -m TinyLlama/TinyLlama-1.1B-Chat-v1.0 --quant-mode int8 --dataset wikitext2 ./save_dir
```

--------------------------------

### Write Multi-line Code Blocks

Source: https://github.com/huggingface/optimum-intel/blob/main/docs/README.md

Use triple backticks to enclose multi-line code examples. This format is compatible with doctest for automatic testing.

```markdown
```
# first line of code
# second line
# etc
```
```

--------------------------------

### Configure model parameters for export and quantization

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/demos/quantized_generation_demo.ipynb

Sets up configuration variables including the model name, save directory, precision, quantization parameters (bits, symmetry, group size, ratio), and target device.

```python
model_name = "microsoft/phi-2"
save_name = model_name.split("/")[-1] + "_openvino"
precision = "f32"
quantization_config = OVWeightQuantizationConfig(
    bits=4,
    sym=False,
    group_size=128,
    ratio=0.8,
)
device = "gpu"
```

--------------------------------

### Initialize Model for Full Quantization

Source: https://github.com/huggingface/optimum-intel/blob/main/docs/source/openvino/optimization.mdx

Load a model and tokenizer for post-training full quantization. The `export=True` argument ensures the model is exported to OpenVINO format.

```python
from transformers import AutoTokenizer
from optimum.intel import OVQuantizer, OVModelForSequenceClassification, OVConfig, OVQuantizationConfig

model_id = "distilbert-base-uncased-finetuned-sst-2-english"
model = OVModelForSequenceClassification.from_pretrained(model_id, export=True)
tokenizer = AutoTokenizer.from_pretrained(model_id)
save_dir = "ptq_model"

quantizer = OVQuantizer.from_pretrained(model)

```

--------------------------------

### Perform Inference with INT8 Pipeline

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/question_answering_quantization.ipynb

Get the answer to a question using the OpenVINO INT8 model pipeline. This is used to compare results with the FP32 model.

```python
ov_qa_pipeline_ptq({"question": question, "context": context})["answer"]
```

--------------------------------

### Perform Inference with FP32 Pipeline

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/question_answering_quantization.ipynb

Get the answer to a question using the Hugging Face FP32 model pipeline. This demonstrates a basic inference call.

```python
hf_qa_pipeline({"question": question, "context": context})["answer"]
```

--------------------------------

### Load and Export Model On-the-Fly (PyTorch to OpenVINO)

Source: https://github.com/huggingface/optimum-intel/blob/main/docs/source/openvino/export.mdx

Load a PyTorch model and convert it to OpenVINO format during loading by setting `export=True`. The model and tokenizer are then saved to a specified directory.

```python
from transformers import AutoModelForCausalLM
from optimum.intel import OVModelForCausalLM
from transformers import AutoTokenizer

model_id = "meta-llama/Meta-Llama-3-8B"
model = OVModelForCausalLM.from_pretrained(model_id, export=True)
tokenizer = AutoTokenizer.from_pretrained(model_id)

save_directory = "ov_model"
model.save_pretrained(save_directory)
tokenizer.save_pretrained(save_directory)
```

--------------------------------

### Compile Model on GPU Using .to() Method

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/optimum_openvino_inference.ipynb

If a GPU is available, this snippet shows how to move the model to the GPU using the `.to("gpu")` method, compile it, and then print the device it's running on.

```python
# Use `model.to()` to compile the model on GPU if a GPU is found
if "GPU" in Core().available_devices:
    model.reshape(1, 28)
    model.to("gpu")
    model.compile()
    print(ov_pipe.model._device)
```

--------------------------------

### Dataset Preprocessing Function

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/question_answering_quantization.ipynb

Defines a function to preprocess dataset examples by tokenizing questions and contexts. It uses padding and truncation to a maximum length of 384 tokens.

```python
def preprocess_fn(examples, tokenizer):
    """convert the text from the dataset into tokens in the format that the model expects"""
    return tokenizer(
        examples["question"],
        examples["context"],
        padding=True,
        truncation=True,
        max_length=384,
    )
```

--------------------------------

### Prepare Draft Model Configuration

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/demos/quantized_generation_demo.ipynb

Define the model name, save path, precision, quantization configuration, and device for the draft model.

```python
model_name = "Salesforce/codegen-350M-multi"
save_name = model_name.split("/")[-1] + "_openvino_stateless"
precision = "f32"
quantization_config = OVWeightQuantizationConfig(
    bits=4,
    sym=False,
    group_size=128,
    ratio=0.8,
)
device = "cpu"
```

--------------------------------

### Create Question Answering Pipelines

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/question_answering_quantization.ipynb

Create separate pipelines for a Hugging Face FP32 model and an OpenVINO INT8 model to facilitate comparison. Ensure the model paths and tokenizer are correctly specified.

```python
quantized_model_ptq = OVModelForQuestionAnswering.from_pretrained(int8_ptq_model_path)
original_model = AutoModelForQuestionAnswering.from_pretrained(MODEL_ID)
ov_qa_pipeline_ptq = pipeline("question-answering", model=quantized_model_ptq, tokenizer=tokenizer)
hf_qa_pipeline = pipeline("question-answering", model=original_model, tokenizer=tokenizer)
```

--------------------------------

### List Available Devices and Full Device Names

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/optimum_openvino_inference.ipynb

Query the OpenVINO Core to list all available devices on the system and retrieve their full names. This is useful for understanding hardware capabilities.

```python
from openvino import Core


for device in Core().available_devices:
    print(device, Core().get_property(device, "FULL_DEVICE_NAME"))
```

--------------------------------

### Calculate Model Size in MB

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/question_answering_quantization.ipynb

Use this function to get the size of PyTorch or OpenVINO models in megabytes. Ensure the correct framework is specified and model files exist in the provided directory.

```python
def get_model_size(model_folder, framework):
    """
    Return OpenVINO or PyTorch model size in Mb.
    Arguments:
        model_folder:
            Directory containing a model.safetensors for a PyTorch model, and an openvino_model.xml/.bin for an OpenVINO model.
        framework:
            Define whether the model is a PyTorch or an OpenVINO model.
    """
    if framework.lower() == "openvino":
        model_path = Path(model_folder) / "openvino_model.xml"
        model_size = model_path.stat().st_size + model_path.with_suffix(".bin").stat().st_size
    elif framework.lower() == "pytorch":
        model_path = Path(model_folder) / "model.safetensors"
        model_size = model_path.stat().st_size
    model_size /= 1000 * 1000
    return model_size


model.save_pretrained(fp32_model_path)

fp32_model_size = get_model_size(fp32_model_path, "pytorch")
int8_model_size = get_model_size(int8_ptq_model_path, "openvino")
print(f"FP32 model size: {fp32_model_size:.2f} MB")
print(f"INT8 model size: {int8_model_size:.2f} MB")
print(f"INT8 size decrease: {fp32_model_size / int8_model_size:.2f}x")
```

--------------------------------

### Initialize Gradio Chatbot Interface

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/demos/quantized_generation_demo.ipynb

Sets up the Gradio interface with a chatbot, message input, status display, and advanced options for generation parameters. Use this to create the interactive UI.

```python
import gradio as gr

try:
    demo.close()
except:
    pass


EXAMPLES = [
    ["What is OpenVINO?"],
    ["Can you explain to me briefly what is Python programming language?"],
    ["Explain the plot of Cinderella in a sentence."],
    ["Write a Python function to perform binary search over a sorted list. Use markdown to write code"],
    ["Lily has a rubber ball that she drops from the top of a wall. The wall is 2 meters tall. How long will it take for the ball to reach the ground?"],
]


def add_user_text(message, history):
    """
    Add user's message to chatbot history

    Params:
      message: current user message
      history: conversation history
    Returns:
      Updated history, clears user message and status
    """
    # Append current user message to history with a blank assistant message which will be generated by the model
    history.append([message, None])
    return ('', history)


def prepare_for_regenerate(history):
    """
    Delete last assistant message to prepare for regeneration

    Params:
      history: conversation history
    Returns:
      updated history
    """
    history[-1][1] = None
    return history


with gr.Blocks(theme=gr.themes.Soft()) as demo:
    gr.Markdown('<h1 style="text-align: center;">Chat with Phi-2 on Meteor Lake iGPU</h1>')
    chatbot = gr.Chatbot()
    with gr.Row():
        assisted = gr.Checkbox(value=False, label="Assisted Generation", scale=10)
        msg = gr.Textbox(placeholder="Enter message here...", show_label=False, autofocus=True, scale=75)
        status = gr.Textbox("Status: Idle", show_label=False, max_lines=1, scale=15)
    with gr.Row():
        submit = gr.Button("Submit", variant='primary')
        regenerate = gr.Button("Regenerate")
        clear = gr.Button("Clear")
    with gr.Accordion("Advanced Options:", open=False):
        with gr.Row():
            with gr.Column():
                temperature = gr.Slider(
                    label="Temperature",
                    value=0.0,
                    minimum=0.0,
                    maximum=1.0,
                    step=0.05,
                    interactive=True,
                )
                max_new_tokens = gr.Slider(
                    label="Max new tokens",
                    value=128,
                    minimum=0,
                    maximum=512,
                    step=32,
                    interactive=True,
                )
            with gr.Column():
                top_p = gr.Slider(
                    label="Top-p (nucleus sampling)",
                    value=1.0,
                    minimum=0.0,
                    maximum=1.0,
                    step=0.05,
                    interactive=True,
                )
                repetition_penalty = gr.Slider(
                    label="Repetition penalty",
                    value=1.0,
                    minimum=1.0,
                    maximum=2.0,
                    step=0.1,
                    interactive=True,
                )
    gr.Examples(
        EXAMPLES, inputs=msg, label="Click on any example and press the 'Submit' button"
    )

    # Sets generate function to be triggered when the user submit a new message
    gr.on(
        triggers=[submit.click, msg.submit],
        fn=add_user_text,
        inputs=[msg, chatbot],
        outputs=[msg, chatbot],
        queue=False,
    ).then(
        fn=generate,
        inputs=[chatbot, temperature, max_new_tokens, top_p, repetition_penalty, assisted],
        outputs=[chatbot, status, msg, submit, regenerate, clear],
        concurrency_limit=1,
        queue=True
    )
    regenerate.click(
        fn=prepare_for_regenerate,
        inputs=chatbot,
        outputs=chatbot,
        queue=True,
        concurrency_limit=1
    ).then(
        fn=generate,
        inputs=[chatbot, temperature, max_new_tokens, top_p, repetition_penalty, assisted],
        outputs=[chatbot, status, msg, submit, regenerate, clear],
        concurrency_limit=1,
        queue=True
    )
    clear.click(fn=lambda: (None, "Status: Idle"), inputs=None, outputs=[chatbot, status], queue=False)
```

--------------------------------

### Load Model Directly on GPU with .from_pretrained()

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/optimum_openvino_inference.ipynb

Load a model directly onto the GPU by specifying the device name in the `.from_pretrained()` method. This is an alternative to using `.to("gpu")` after loading.

```python
# Set the device directly with `.from_pretrained()`
if "GPU" in Core().available_devices:
    model = OVModelForQuestionAnswering.from_pretrained("distilbert-base-uncased-distilled-squad-ov-fp16", device="GPU")
```

--------------------------------

### Apply Post-Training Static Quantization

Source: https://github.com/huggingface/optimum-intel/blob/main/README.md

Apply post-training static quantization to a model using OVQuantizationConfig. Specify the desired data type (e.g., 'int8'), a calibration dataset, and the number of samples for calibration. The quantized model is then saved to a specified directory.

```python
from optimum.intel import OVModelForSpeechSeq2Seq, OVQuantizationConfig

model_id = "openai/whisper-tiny"
q_config = OVQuantizationConfig(dtype="int8", dataset="librispeech", num_samples=50)
q_model = OVModelForSpeechSeq2Seq.from_pretrained(model_id, quantization_config=q_config)

# The directory where the quantized model will be saved
save_dir = "nncf_results"
q_model.save_pretrained(save_dir)
```

--------------------------------

### Quantize Model with Default Configuration

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/question_answering_quantization.ipynb

Performs post-training quantization on a model using default configurations. This snippet requires 'warnings', 'OVConfig', 'OVQuantizer', 'model', 'train_dataset', and 'int8_ptq_model_path' to be defined.

```python
# Hide PyTorch warnings about missing shape inference
warnings.simplefilter("ignore")

# Quantize the model
quantizer = OVQuantizer.from_pretrained(model)
ov_config = OVConfig(quantization_config=OVQuantizationConfig())
quantizer.quantize(calibration_dataset=train_dataset, ov_config=ov_config, save_directory=int8_ptq_model_path)
```

--------------------------------

### Quantize and Export Model to OpenVINO IR

Source: https://github.com/huggingface/optimum-intel/blob/main/docs/source/openvino/optimization.mdx

Applies full quantization and exports the model to OpenVINO IR format. Requires an OVConfig and a calibration dataset. The tokenizer is saved separately.

```python
from optimum.intel import OVConfig, OVQuantizationConfig

ov_config = OVConfig(quantization_config=OVQuantizationConfig())
quantizer.quantize(ov_config=ov_config, calibration_dataset=calibration_dataset, save_directory=save_dir)
# Save the tokenizer
tokenizer.save_pretrained(save_dir)
```

--------------------------------

### Weight-Only Quantization (4-bit, Data-Aware) CLI

Source: https://github.com/huggingface/optimum-intel/blob/main/docs/source/openvino/optimization.mdx

Execute this CLI command for data-aware 4-bit weight-only quantization. Specify the dataset for calibration.

```bash
optimum-cli export openvino -m TinyLlama/TinyLlama-1.1B-Chat-v1.0 --weight-format int4 --dataset wikitext2 ./save_dir
```

--------------------------------

### Apply Mixed Quantization using CLI

Source: https://github.com/huggingface/optimum-intel/blob/main/docs/source/openvino/optimization.mdx

Apply mixed quantization through the CLI by specifying the quantization mode and dataset. The `--quant-mode` argument combines the precisions for weight and full quantization, and a dataset is required for calibration.

```bash
optimum-cli export openvino -m TinyLlama/TinyLlama-1.1B-Chat-v1.0 --quant-mode cb4_f8e4m3 --dataset wikitext2 ./save_dir
```

--------------------------------

### Create Calibration Dataset for Quantization

Source: https://github.com/huggingface/optimum-intel/blob/main/docs/source/openvino/optimization.mdx

Generates a calibration dataset using OVQuantizer. Requires a preprocessing function and dataset details. The dataset is used for full quantization.

```python
from functools import partial

def preprocess_function(examples, tokenizer):
    return tokenizer(examples["sentence"], padding="max_length", max_length=128, truncation=True)

# Create the calibration dataset used to perform full quantization
calibration_dataset = quantizer.get_calibration_dataset(
    "glue",
    dataset_config_name="sst2",
    preprocess_function=partial(preprocess_function, tokenizer=tokenizer),
    num_samples=300,
    dataset_split="train",
)
```

--------------------------------

### Load and Export Model with Optimum

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/optimum_openvino_inference.ipynb

Load a PyTorch model from the Hugging Face Hub and export it to the OpenVINO format. The converted model is then saved locally for faster subsequent loading.

```python
from optimum.intel import OVModelForQuestionAnswering


# Load PyTorch model from the Hub and export to OpenVINO in the background
model = OVModelForQuestionAnswering.from_pretrained("distilbert-base-uncased-distilled-squad", export=True)

# Save the converted model to a local directory
model.save_pretrained("distilbert-base-uncased-distilled-squad-ov-fp32")

# Load the OpenVINO model directly from the directory
model = OVModelForQuestionAnswering.from_pretrained("distilbert-base-uncased-distilled-squad-ov-fp32")
```

--------------------------------

### Configure 4-bit Weight Quantization Parameters

Source: https://github.com/huggingface/optimum-intel/blob/main/docs/source/openvino/optimization.mdx

Manually configure 4-bit weight quantization using `OVWeightQuantizationConfig`. Specify parameters like `bits`, `sym`, `ratio`, `quant_method`, and `dataset` for fine-tuning.

```python
from optimum.intel import OVWeightQuantizationConfig

quantization_config = OVWeightQuantizationConfig(
    bits=4,
    sym=False,
    ratio=0.8,
    quant_method="awq",
    dataset="wikitext2"
)
```

--------------------------------

### Load Processor and Prepare Inputs

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/vision_language_quantization.ipynb

Loads the processor for a specified model and prepares input data, including text prompts and images, for the VLM model. Ensure transformers logging is set to error to avoid verbosity.

```python
import transformers
from transformers import AutoProcessor
from transformers.image_utils import load_image
transformers.logging.set_verbosity_error()

model_id = "echarlaix/SmolVLM2-256M-Video-Instruct-openvino"
processor = AutoProcessor.from_pretrained(model_id)
prompt, img_url = "What is on the flower?", "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": prompt}
        ]
    }
]

# Prepare inputs
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=[load_image(img_url)], return_tensors="pt")
```

--------------------------------

### Benchmark INT8 Model

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/sentence_transformer_quantization.ipynb

Use this command to benchmark an INT8 optimized OpenVINO model. Specify the model path, input shapes, and desired iteration count.

```bash
!benchmark_app -m all-MiniLM-L6-v2_int8/openvino_model.xml -shape "input_ids[1,384],attention_mask[1,384],token_type_ids[1,384]" -api sync -niter 200
```

--------------------------------

### Load and Run OpenVINO Stable Diffusion Model

Source: https://github.com/huggingface/optimum-intel/blob/main/docs/source/openvino/tutorials/diffusers.mdx

Load a pre-exported OpenVINO Stable Diffusion model and generate an image from a text prompt. Ensure the model ID points to an OpenVINO-compatible model.

```python
from optimum.intel import OVStableDiffusionPipeline

model_id = "echarlaix/stable-diffusion-v1-5-openvino"
pipeline = OVStableDiffusionPipeline.from_pretrained(model_id)
prompt = "sailing ship in storm by Rembrandt"
images = pipeline(prompt).images
```

--------------------------------

### Benchmark App Output

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/sentence_transformer_quantization.ipynb

This output shows the detailed steps and performance statistics of the benchmark_app, including model loading, input configuration, and inference results.

```text
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)


[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2024.4.1-16618-643f23d1318-releases/2024/4
[ INFO ] 
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2024.4.1-16618-643f23d1318-releases/2024/4
[ INFO ] 
[ INFO ] 
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.LATENCY.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 10.17 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ]     input_ids (node: input_ids) : i64 / [...] / [?,?]
[ INFO ]     attention_mask (node: attention_mask) : i64 / [...] / [?,?]
[ INFO ]     token_type_ids (node: token_type_ids) : i64 / [...] / [?,?]
[ INFO ] Model outputs:
[ INFO ]     last_hidden_state (node: __module.encoder.layer.5.output.LayerNorm/aten::layer_norm/Add) : f32 / [...] / [?,?,384]
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[ INFO ] Reshaping model: 'input_ids': [1,384], 'attention_mask': [1,384], 'token_type_ids': [1,384]
[ INFO ] Reshape model took 2.23 ms
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ]     input_ids (node: input_ids) : i64 / [...] / [1,384]
[ INFO ]     attention_mask (node: attention_mask) : i64 / [...] / [1,384]
[ INFO ]     token_type_ids (node: token_type_ids) : i64 / [...] / [1,384]
[ INFO ] Model outputs:
[ INFO ]     last_hidden_state (node: __module.encoder.layer.5.output.LayerNorm/aten::layer_norm/Add) : f32 / [...] / [1,384,384]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 134.63 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ]   NETWORK_NAME: Model0
[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
[ INFO ]   NUM_STREAMS: 1
[ INFO ]   INFERENCE_NUM_THREADS: 18
[ INFO ]   PERF_COUNT: NO
[ INFO ]   INFERENCE_PRECISION_HINT: <Type: 'float32'>
[ INFO ]   PERFORMANCE_HINT: LATENCY
[ INFO ]   EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
[ INFO ]   PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ]   ENABLE_CPU_PINNING: True
[ INFO ]   SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ]   MODEL_DISTRIBUTION_POLICY: set()
[ INFO ]   ENABLE_HYPER_THREADING: False
[ INFO ]   EXECUTION_DEVICES: ['CPU']
[ INFO ]   CPU_DENORMALS_OPTIMIZATION: False
[ INFO ]   LOG_LEVEL: Level.NO
[ INFO ]   CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0
[ INFO ]   DYNAMIC_QUANTIZATION_GROUP_SIZE: 32
[ INFO ]   KV_CACHE_PRECISION: <Type: 'float16'>
[ INFO ]   AFFINITY: Affinity.CORE
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'input_ids'!. This input will be filled with random values!
[ WARNING ] No input files were given for input 'attention_mask'!. This input will be filled with random values!
[ WARNING ] No input files were given for input 'token_type_ids'!. This input will be filled with random values!
[ INFO ] Fill input 'input_ids' with random values 
[ INFO ] Fill input 'attention_mask' with random values 
[ INFO ] Fill input 'token_type_ids' with random values 
[Step 10/11] Measuring performance (Start inference synchronously, limits: 200 iterations)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 12.27 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count:            200 iterations
[ INFO ] Duration:         1988.84 ms
[ INFO ] Latency:
[ INFO ]    Median:        9.74 ms
[ INFO ]    Average:       9.77 ms
[ INFO ]    Min:           9.59 ms
[ INFO ]    Max:           11.12 ms
[ INFO ] Throughput:   100.56 FPS
```

--------------------------------

### Import necessary libraries

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/demos/quantized_generation_demo.ipynb

Imports required modules from the transformers and optimum.intel libraries for model loading and quantization.

```python
import os

from transformers import AutoTokenizer

from optimum.intel import OVModelForCausalLM, OVWeightQuantizationConfig
```

--------------------------------

### Load, export, and quantize the model

Source: https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/demos/quantized_generation_demo.ipynb

Loads the specified model, exports it to OpenVINO format if not already saved, applies weight-only quantization, and saves the processed model and tokenizer locally. It also prints the resulting model size.

```python
# Load kwargs
load_kwargs = {
    "device": device,
    "ov_config": {
        "PERFORMANCE_HINT": "LATENCY",
        "INFERENCE_PRECISION_HINT": precision,
        "CACHE_DIR": os.path.join(save_name, "model_cache"),  # OpenVINO will use this directory as cache
    },
    "compile": False,
    "quantization_config": quantization_config
}

# Check whether the model was already exported
saved = os.path.exists(save_name)

model = OVModelForCausalLM.from_pretrained(
    model_name if not saved else save_name,
    export=not saved,
    **load_kwargs,
)

# Load tokenizer to be used with the model
tokenizer = AutoTokenizer.from_pretrained(model_name if not saved else save_name)

# Save the exported model locally
if not saved:
    model.save_pretrained(save_name)
    tokenizer.save_pretrained(save_name)

# TODO Optional: export to huggingface/hub

model_size = os.stat(os.path.join(save_name, "openvino_model.bin")).st_size / 1024 ** 3
print(f'Model size in FP32: ~5.4GB, current model size in 4bit: {model_size:.2f}GB')
```