### Basic Instruction Model Setup Example

Source: https://github.com/iinemo/lm-polygraph/blob/main/docs/normalization/3_instruction_tuned_model.md

A basic setup for an instruction model, combining default model settings with specific output processing and top-1 processing configurations. Use this for a straightforward instruction-following setup.

```yaml
defaults:
  - model: default_causal
  - _self_ 

process_output_fn:
  path: instruct/output_processing_scripts/default.py
  fn_name: normalize_em

top1_processing:
  enabled: true
  normalize_confidence: true
```

--------------------------------

### Multi-Task Model Setup Example

Source: https://github.com/iinemo/lm-polygraph/blob/main/docs/normalization/3_instruction_tuned_model.md

Sets up a multi-task model with Top-K processing enabled. This configuration is suitable for models handling diverse tasks and requiring aggregation of multiple outputs.

```yaml
defaults:
  - model: stablelm-chat
  - _self_ 

process_output_fn:
  path: instruct/output_processing_scripts/multi_task.py
  fn_name: normalize_mt

topk_processing:
  enabled: true
  k: 4
  aggregate_method: "max"
```

--------------------------------

### Basic MinMax Normalization Usage Example

Source: https://github.com/iinemo/lm-polygraph/blob/main/docs/normalization/1_core_normalization_configuration.md

An example demonstrating basic MinMax normalization with dataset-specific calibration.

```yaml
normalization:
  type: "minmax"
  clip: true
  calibration:
    strategy: "dataset_specific"
```

--------------------------------

### Run Benchmark using Docker Container

Source: https://context7.com/iinemo/lm-polygraph/llms.txt

Execute the benchmark using a pre-built Docker container, eliminating the need for a local environment setup. Ensure Docker is installed and GPU access is configured.

```bash
docker run --gpus '"device=0"' --rm \
  -w /app inemo/lm_polygraph \
  bash -c "polygraph_eval \
    --config-dir=./examples/configs/ \
    --config-name=polygraph_eval_coqa.yaml \
    model.path=meta-llama/Llama-3.1-8B \
    subsample_eval_dataset=100"
```

--------------------------------

### Basic QA Task Configuration Example

Source: https://github.com/iinemo/lm-polygraph/blob/main/docs/normalization/2_dataset_specific_configuration.md

This example demonstrates a basic configuration for a Question-Answering task (TriviaQA), setting the dataset, subsample size, and specifying a custom normalization function.

```yaml
hydra:
  run:
    dir: ${cache_path}/${task}/${model.path}/${dataset}/${now:%Y-%m-%d}

defaults:
  - model: default
  - _self_

dataset: triviaqa
subsample_train_dataset: 1000
normalize: true
process_output_fn:
  path: output_processing_scripts/triviaqa.py
  fn_name: normalize_qa
```

--------------------------------

### Install LM-Polygraph from GitHub

Source: https://github.com/iinemo/lm-polygraph/blob/main/docs/usage.md

Clone the repository and install using pip. Recommended for exploring notebooks and default configurations. Ensure you are in a virtual environment.

```console
$ git clone https://github.com/IINemo/lm-polygraph.git
$ python3 -m venv env
$ source env/bin/activate
(env) $ cd lm-polygraph
(env) $ pip install .
```

--------------------------------

### Install OpenAI Package

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/other/claim_level_example_ru.ipynb

Installs the OpenAI package version 0.28. This is a prerequisite for certain functionalities.

```python
# !pip install openai==0.28
```

--------------------------------

### Binned PCC with Custom Settings Usage Example

Source: https://github.com/iinemo/lm-polygraph/blob/main/docs/normalization/1_core_normalization_configuration.md

An example showcasing Binned PCC with custom parameters for the number of bins, data processing, and caching enabled.

```yaml
normalization:
  type: "binned_pcc"
  params:
    num_bins: 20
  processing:
    ignore_nans: false
    normalize_metrics: true
  cache:
    enabled: true
```

--------------------------------

### Example Method Bounds Calculation

Source: https://github.com/iinemo/lm-polygraph/blob/main/notebooks/method_bounds.ipynb

This snippet demonstrates how to calculate method bounds. Ensure necessary libraries are imported before use.

```python
import numpy as np

def get_method_bounds(method_name: str) -> tuple[float, float]:
    """Get the bounds for a given method name."""
    if method_name == "linear":
        return 0.0, 1.0
    elif method_name == "quadratic":
        return -1.0, 1.0
    else:
        raise ValueError(f"Unknown method: {method_name}")


# Example usage:
method_name = "linear"
lower_bound, upper_bound = get_method_bounds(method_name)
print(f"Bounds for {method_name}: [{lower_bound}, {upper_bound}]")

method_name = "quadratic"
lower_bound, upper_bound = get_method_bounds(method_name)
print(f"Bounds for {method_name}: [{lower_bound}, {upper_bound}]")

# Example with unknown method
try:
    get_method_bounds("cubic")
except ValueError as e:
    print(e)
```

--------------------------------

### Install and Download LM-Polygraph

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/basic_example.ipynb

Installs the lm-polygraph library from GitHub and downloads the necessary spaCy model. Ensure you have pip and git installed.

```python
# Assume that you have installed lm-polygraph: 
# pip install git+https://github.com/artemshelmanov/lm-polygraph.git

!python -m spacy download en_core_web_sm
```

--------------------------------

### Install LM-Polygraph from GitHub (Stable Release)

Source: https://github.com/iinemo/lm-polygraph/blob/main/docs/usage.md

Clone the repository, checkout to a specific tag for a stable release, and then install using pip. This ensures stability over the latest main branch code.

```console
$ git clone https://github.com/IINemo/lm-polygraph.git
$ git checkout tags/v0.5.0
$ python3 -m venv env
$ source env/bin/activate
(env) $ cd lm-polygraph
(env) $ pip install .
```

--------------------------------

### Initialize Model and Estimators

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/other/minimal_changes.ipynb

Initializes the model adapter, statistical calculators, and the claim uncertainty estimator. Ensure you have the necessary libraries installed and API keys configured.

```python
import os

from lm_polygraph.model_adapters import WhiteboxModelBasic
from lm_polygraph.estimators import ClaimConditionedProbabilityClaim
from lm_polygraph.stat_calculators import *
from lm_polygraph.utils.openai_chat import OpenAIChat
from lm_polygraph.utils.deberta import Deberta


max_new_tokens = 50
generation_config.temperature = 0.9
generation_config.do_sample = True

model_adapter = WhiteboxModelBasic(model, tokenizer, tokenizer_args={}, generation_parameters=generation_config)

calc_infer_llm = InferCausalLMCalculator(tokenize=False)

os.environ["OPENAI_API_KEY"] = "<Your OpenAI API key>"
calc_claim_extractor = ClaimsExtractor(OpenAIChat("gpt-4o"))

calc_claim_nli = GreedyAlternativesNLICalculator(Deberta(device=device))

estimator = ClaimConditionedProbabilityClaim()
```

--------------------------------

### Install LM-Polygraph from GitHub using a tag

Source: https://github.com/iinemo/lm-polygraph/blob/main/README.md

Installs a specific tagged version of LM-Polygraph from GitHub using pip.

```shell
pip install git+https://github.com/IINemo/lm-polygraph.git@v0.5.0
```

--------------------------------

### Multilingual Configuration Example

Source: https://github.com/iinemo/lm-polygraph/blob/main/docs/normalization/2_dataset_specific_configuration.md

This example shows a multilingual configuration for a 'person_bio' dataset in Chinese. It enables multilingual normalization with language-specific bins and sets the subsample and background dataset parameters.

```yaml
dataset: person_bio
language: zh
multilingual_normalization:
  enabled: true
  use_language_specific_bins: true
subsample_train_dataset: 1000
background_train_dataset: allenai/c4
```

--------------------------------

### Install LM-Polygraph

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/basic_example_visual.ipynb

Assume that you have installed lm-polygraph.

```python
# Assume that you have installed lm-polygraph:

```

--------------------------------

### Install lm-polygraph

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/basic_example_visual.ipynb

Install the lm-polygraph package from GitHub. This command ensures you have the latest version for use.

```bash
# pip install git+https://github.com/artemshelmanov/lm-polygraph.git
```

--------------------------------

### Global Isotonic PCC Usage Example

Source: https://github.com/iinemo/lm-polygraph/blob/main/docs/normalization/1_core_normalization_configuration.md

An example of using global Isotonic PCC with a specified background dataset.

```yaml
normalization:
  type: "isotonic_pcc"
  params:
    y_min: 0.0
    y_max: 1.0
    increasing: false
  calibration:
    strategy: "global"
    background_dataset: "allenai/c4"
```

--------------------------------

### Translation Task Configuration Example

Source: https://github.com/iinemo/lm-polygraph/blob/main/docs/normalization/2_dataset_specific_configuration.md

Example configuration for a translation task (WMT). It sets the dataset, subsample size, source text cleaning regex, and enables translation normalization, while disabling the background dataset.

```yaml
dataset: wmt14_deen
subsample_train_dataset: 2000
source_ignore_regex: "^Translation: "
normalize_translations: true
background_train_dataset: null
```

--------------------------------

### Install LM-Polygraph from PyPI

Source: https://github.com/iinemo/lm-polygraph/blob/main/README.md

Installs the latest tagged version of LM-Polygraph from the Python Package Index (PyPI).

```shell
pip install lm-polygraph
```

--------------------------------

### Clone LM-Polygraph and Install Dependencies

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/other/claim_level_example_ru.ipynb

Clones the LM-Polygraph repository, navigates into its directory, and installs the project and its dependencies, including transformers, rouge-score, datasets, and openai. It also installs Node.js and npm packages required for the application. Uncomment to run in Google Colab.

```python
##uncomment this to use code in Google Colab
# from IPython.display import clear_output
# !git clone https://github.com/alfekka/lm-polygraph.git
# %cd lm-polygraph
# %pip install .
# %cd src
# %pip install transformers rouge-score datasets openai
# !curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
# !apt-get install -y nodejs
# %cd lm_polygraph/app
# !npm install
# %cd ../..
# %cd /content/lm-polygraph/src
# %load_ext autoreload
# %autoreload 2
```

--------------------------------

### Install LM-Polygraph from GitHub

Source: https://github.com/iinemo/lm-polygraph/blob/main/README.md

Installs the latest stable version of LM-Polygraph from the main branch using pip. It's recommended to use a virtual environment.

```shell
python -m venv env # Substitute this with your virtual environment creation command
source env/bin/activate
pip install git+https://github.com/IINemo/lm-polygraph.git
```

--------------------------------

### Install Core LM-Polygraph Dependencies

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/other/claim_level_example_ru.ipynb

Installs essential libraries for LM-Polygraph, including fastchat, wget, evaluate, sentence_transformers, bert_score, datasets, and sacrebleu. Uncomment to run in Google Colab.

```python
##uncomment this to use code in Google Colab
# !pip install fastchat
# !pip install wget
# !pip install evaluate
# !pip install sentence_transformers
# !pip install bert_score
# !pip install datasets
# !pip install sacrebleu
```

--------------------------------

### Visualize Benchmark Results

Source: https://github.com/iinemo/lm-polygraph/blob/main/notebooks/vizualization_tables.ipynb

Example of how to call the `pretty_plot` function to visualize benchmark results for a specific dataset and output files from multiple seeds.

```python
# visualize results in a table
pretty_plot(
    'TriviaQA, Dolly3b',
    # outputs generated by scripts/polygraph_eval benchmark
    # provide several seeds to calculate variance
    ['./workdir/output_seed' + str(x)
     for x in range(1, 10)])
```

--------------------------------

### Chain-of-Thought (CoT) Model Configuration Example

Source: https://github.com/iinemo/lm-polygraph/blob/main/docs/normalization/3_instruction_tuned_model.md

Configures a model for Chain-of-Thought processing, enabling CoT and final answer extraction. Use this for models that generate reasoning steps.

```yaml
defaults:
  - model: mistral-instruct
  - _self_ 

cot_processing:
  enabled: true
  extract_final_answer: true

process_output_fn:
  path: instruct/output_processing_scripts/cot.py
  fn_name: normalize_cot
```

--------------------------------

### Get Method Bounds with Specific Parameters

Source: https://github.com/iinemo/lm-polygraph/blob/main/notebooks/method_bounds.ipynb

Retrieves method bounds for a method with specified parameters. Ensure the parameters match the method signature.

```python
bounds = get_method_bounds(method_name, param1, param2)
```

--------------------------------

### Get Bounds for a Specific Method

Source: https://github.com/iinemo/lm-polygraph/blob/main/notebooks/method_bounds.ipynb

Retrieves the specific bounds set for a particular method. This can be used for inspection or further processing.

```python
lower, upper = method_bounds.get_bounds('my_method')
```

--------------------------------

### Get Default Bounds

Source: https://github.com/iinemo/lm-polygraph/blob/main/notebooks/method_bounds.ipynb

Retrieves the default bounds configured for the MethodBounds object. This is useful when a specific method's bounds are not explicitly set.

```python
default_lower, default_upper = method_bounds.get_default_bounds()
```

--------------------------------

### Initialization of Paths, Models, Datasets, and Metrics

Source: https://github.com/iinemo/lm-polygraph/blob/main/notebooks/result_tables.ipynb

Sets up the necessary configurations including file paths, model names, datasets, generation metrics, and user-defined metrics for the experiment.

```python
paths = ["../workdir/camera_ready_exps/v1", "../workdir/camera_ready_exps/bertscore"]
models = ["vicuna", "llama"]
datasets = ["aeslc", "xsum", "coqa", "babiqa", "wmt14_deen", "wmt14_fren"]
gen_metrics = ["Rouge_rougeL", "Bert"]
ue_metrics = {"prr": PredictionRejectionArea(), 
              "kendalltau": KendallTauCorrelation(),
              "spearmanr": SpearmanRankCorrelation()
             }
dfs, quality_dfs = get_tables(paths, models, datasets, gen_metrics, ue_metrics)
```

--------------------------------

### Launch Dataset Builder

Source: https://github.com/iinemo/lm-polygraph/blob/main/dataset_builders/README.md

The main entry point for the dataset builder. Use this command to initiate dataset building processes.

```bash
python3 manager.py
```

--------------------------------

### Install LM-Polygraph

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/basic_example_visual.ipynb

Install the LM-Polygraph library using pip. This command installs the library directly from its GitHub repository.

```bash
pip install git+https://github.com/artemshelmanov/lm-polygraph.git
```

--------------------------------

### Model and Device Configuration

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/low_level_example.ipynb

Sets up the model name, target device, and batch size for LLM inference.

```python
model_name_or_path = "mistralai/Mistral-7B-Instruct-v0.2"
device = "cuda:0"
batch_size = 2
```

--------------------------------

### Initialize and run the UE Manager

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/other/ats_example.ipynb

Creates a UEManager instance with the configured data, model, estimators, metrics, and processors, then executes it to compute results.

```python
man = UEManager(
    data=dataset,
    model=model,
    estimators=ue_methods,
    builder_env_stat_calc=builder_env_stat_calc,
    available_stat_calculators=available_stat_calculators,
    generation_metrics=metrics,
    ue_metrics=ue_metrics,
    processors=loggers,
    ignore_exceptions=False,
    max_new_tokens=100
)

results = man()
```

--------------------------------

### Define model and dataset configuration parameters

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/other/ats_example.ipynb

Sets up key parameters for the model and dataset, including model path, device, dataset name, model type, batch size, and random seed.

```python
model_path = "facebook/bart-large-cnn"
device = "cpu"
dataset_name = "xsum"
model_type = "Whitebox"
batch_size = 4
seed = 42
```

--------------------------------

### Install LM-Polygraph with COMET metric

Source: https://github.com/iinemo/lm-polygraph/blob/main/README.md

Installs LM-Polygraph with optional dependencies for the COMET metric. Note that 'unbabel-comet' may conflict with packages requiring numpy 2.x. If numpy 2.x is needed, install LM-Polygraph without extras and add 'unbabel-comet' manually without its dependencies.

```shell
pip install lm-polygraph[comet]
```

```shell
pip install lm-polygraph
pip install unbabel-comet --no-deps
```

--------------------------------

### Initialize Whitebox Model

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/basic_example_visual.ipynb

Load a pre-trained vision-to-sequence model and processor, then wrap them in a VisualWhiteboxModel for UQ tasks. Requires the `transformers` library.

```python
from lm_polygraph.models import VisualWhiteboxModel
from transformers import AutoModelForVision2Seq, AutoProcessor

base_model = AutoModelForVision2Seq.from_pretrained("microsoft/kosmos-2-patch14-224")
processor = AutoProcessor.from_pretrained("microsoft/kosmos-2-patch14-224")

# Create whitebox model
model = VisualWhiteboxModel(base_model, processor)

# Test with input text and image
input_text = ["<grounding>An image of"]
url = "https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.png"
```

--------------------------------

### Initialize and run the UEManager

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/other/qa_example.ipynb

Initializes the UEManager with the prepared dataset, model, estimators, metrics, and stat calculators. The manager is then called to execute the uncertainty estimation process.

```python
man = UEManager(
    data=dataset,
    model=model,
    estimators=ue_methods,
    builder_env_stat_calc=builder_env_stat_calc,
    available_stat_calculators=available_stat_calculators,
    generation_metrics=metrics,
    ue_metrics=ue_metrics,
    processors=loggers,
    ignore_exceptions=False,
    max_new_tokens=10
)

results = man()
```

--------------------------------

### Install Specific LM-Polygraph Version from PyPI

Source: https://github.com/iinemo/lm-polygraph/blob/main/docs/usage.md

Install a particular version of LM-Polygraph from PyPI by specifying the version number. This is useful for reproducibility or when a specific version is required.

```console
$ python3 -m venv env
$ source env/bin/activate
$ pip install lm-polygraph==0.5.0
```

--------------------------------

### Initialize BlackboxModel with OpenAI API Key

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/basic_example.ipynb

Set up the BlackboxModel for interacting with OpenAI's API. Ensure you replace '<Your OpenAI key>' with your actual key.

```python
OPENAI_KEY = '<Your OpenAI key>'
model = BlackboxModel(
    OPENAI_KEY,
    'gpt-4o-mini'
)
```

--------------------------------

### Build and Save Dataset to Disk

Source: https://github.com/iinemo/lm-polygraph/blob/main/dataset_builders/README.md

Builds a specified dataset and saves it to the local disk. The dataset can later be loaded using `datasets.load_from_disk`.

```bash
python3 manager.py --dataset {DATASET} --save-to-disk
```

--------------------------------

### Install LM-Polygraph from PyPI

Source: https://github.com/iinemo/lm-polygraph/blob/main/docs/usage.md

Install the latest stable version of LM-Polygraph directly from the Python Package Index using pip. Ensure you are in an activated virtual environment.

```console
$ python3 -m venv env
$ source env/bin/activate
$ pip install lm-polygraph
```

--------------------------------

### Initialize Whitebox Model

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/basic_example.ipynb

Sets up a whitebox language model using Hugging Face's transformers library. This involves loading a pre-trained model and tokenizer, and configuring generation parameters. The model is loaded onto the CPU.

```python
model_name = 'Qwen/Qwen2.5-0.5B-Instruct'
base_model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map='cpu',
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

gen_params = GenerationParameters(
        do_sample=False,
        max_new_tokens=80,
        )

model = WhiteboxModel(base_model, tokenizer, generation_parameters=gen_params)
```

--------------------------------

### vLLM Model and Device Configuration

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/low_level_vllm_example.ipynb

Sets up the model name, device, and batch size for vLLM inference. Ensures CUDA is visible on the specified device.

```python
model_name_or_path = "meta-llama/Llama-3.1-8B-Instruct"
device = "cuda:0"
batch_size = 2

import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
```

--------------------------------

### Register stat calculators and build environment

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/other/ats_example.ipynb

Registers default stat calculators and a custom TrainingStatisticExtractionCalculator, then initializes the BuilderEnvironmentStatCalculator.

```python
# register default stat calculators
result_stat_calculators = dict()
scs = register_default_stat_calculators(model_type)
for sc in scs:
    result_stat_calculators[sc.name] = sc

# register TrainingStatisticExtractionCalculator for the Mahalanobis Distance method
result_stat_calculators.update(
    {
        "TrainingStatisticExtractionCalculator": StatCalculatorContainer(
            name="TrainingStatisticExtractionCalculator",
            cfg=OmegaConf.create(TrainingStatistic_config),
            stats=["train_embeddings", "background_train_embeddings", "train_greedy_log_likelihoods"],
            dependencies=[],
            builder="lm_polygraph.defaults.stat_calculator_builders.default_TrainingStatisticExtractionCalculator",
        )
    }
)
    
builder_env_stat_calc = BuilderEnvironmentStatCalculator(model=model)
available_stat_calculators = list(result_stat_calculators.values())
```

--------------------------------

### Define Uncertainty Estimation methods, metrics, and processors

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/other/ats_example.ipynb

Initializes a list of UE methods, UE metrics, generation metrics, and loggers to be used by the UEManager.

```python
ue_methods = [MaximumSequenceProbability(), 
              SemanticEntropy(),
              MahalanobisDistanceSeq("encoder"),]

ue_metrics = [PredictionRejectionArea()]

metrics = [RougeMetric('rougeL')]

loggers = [Logger()]
```

--------------------------------

### Initialize LM-Polygraph Components

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/low_level_example.ipynb

Sets up LM-Polygraph calculators, NLI model, generation parameters, and the model adapter for uncertainty estimation.

```python
from lm_polygraph.stat_calculators import GreedyProbsCalculator, GreedyAlternativesNLICalculator
from lm_polygraph.estimators.claim_conditioned_probability import ClaimConditionedProbability
from lm_polygraph.utils.deberta import Deberta
from lm_polygraph.utils.generation_parameters import GenerationParameters
from lm_polygraph.model_adapters import WhiteboxModel


max_new_tokens = 100
generation_params = GenerationParameters()
generation_params.temperature = 0.9
generation_params.do_sample = True

model_adapter = WhiteboxModel(model, tokenizer, generation_parameters=generation_params)

calc_greedy_probs = GreedyProbsCalculator()
nli_model = Deberta(device=device)
nli_model.setup()
calc_nli = GreedyAlternativesNLICalculator(nli_model=nli_model)

estimator = ClaimConditionedProbability()
```

--------------------------------

### Calculate Claim Conditioned Probability

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/other/claim_level_example_ru.ipynb

Calculates the conditional probability of a claim, potentially using NLI calculators. This example uses GreedyAlternativesNLICalculator.

```python
# Claim Conditional Probability
for calculator in [
    GreedyAlternativesNLICalculator(MultilingualDeberta())
]:
    stat.update(calculator(stat, texts, model))

ccp = ClaimConditionedProbabilityClaim()
ccp (stat)
```

--------------------------------

### Subsample and Split Datasets

Source: https://context7.com/iinemo/lm-polygraph/llms.txt

Shows how to subsample a dataset to a specific size and split it into training and testing sets with a specified test size and random seed.

```python
ds.subsample(size=50, seed=42)
train_x, test_x, train_y, test_y = ds.train_test_split(test_size=0.2, seed=42, split="train")
```

--------------------------------

### Initialize vLLM LLM and Sampling Parameters

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/low_level_vllm_example.ipynb

Initializes the vLLM LLM with a specified model and GPU memory utilization. Defines sampling parameters including max tokens and log probabilities.

```python
from vllm import LLM, SamplingParams

llm = LLM(model=model_name_or_path, gpu_memory_utilization=0.70)
sampling_params = SamplingParams(max_tokens=30, logprobs=20)
```

--------------------------------

### Method Bounds with Default Values

Source: https://github.com/iinemo/lm-polygraph/blob/main/notebooks/method_bounds.ipynb

Demonstrates setting method bounds with default values. This is useful when a method can operate within a standard range.

```python
method_bounds = MethodBounds(default_lower_bound=0.0, default_upper_bound=1.0)
```

--------------------------------

### Load Datasets from Various Sources

Source: https://context7.com/iinemo/lm-polygraph/llms.txt

Demonstrates loading datasets from Python lists, local CSV files, and HuggingFace datasets. Supports automatic dispatch between local CSV and HuggingFace if the path exists.

```python
ds = Dataset(
    x=["What is 2+2?", "Name a planet.", "Who wrote Faust?"],
    y=["4",            "Mars",           "Goethe"],
    batch_size=2,
)

# From a local CSV file
ds_csv = Dataset.from_csv(
    csv_path="./qa_data.csv",
    x_column="question",
    y_column="answer",
    batch_size=4,
    prompt="Answer the following question: {text}",
)

# From HuggingFace datasets
ds_hf = Dataset.from_datasets(
    dataset_path="trivia_qa",
    x_column="question",
    y_column="answer",
    batch_size=4,
    split="validation",
    size=200,   # subsample to 200 examples
)

# Automatic dispatch: CSV if path exists locally, else HF
ds_auto = Dataset.load("coqa", "question", "answers", batch_size=4, split="validation")
```

--------------------------------

### Riddle Uncertainty Estimation

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/basic_example.ipynb

Estimates uncertainty for a riddle prompt using the same model and estimator setup. This demonstrates uncertainty quantification for different types of inputs.

```python
estimator = EigValLaplacian(verbose=True)
estimate_uncertainty(model, estimator, input_text='What has a head and a tail but no body?')
```

--------------------------------

### Initialize WhiteboxModel with HuggingFace Model

Source: https://github.com/iinemo/lm-polygraph/blob/main/docs/usage.md

Initialize the base model and tokenizer from HuggingFace, then use them to create a WhiteboxModel instance for evaluation. Requires `transformers` library.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from lm_polygraph.utils.model import WhiteboxModel

model_path = "bigscience/bloomz-560m"
base_model = AutoModelForCausalLM.from_pretrained(model_path, device_map="cuda:0")
tokenizer = AutoTokenizer.from_pretrained(model_path)

model = WhiteboxModel(base_model, tokenizer, model_path=model_path)
```

--------------------------------

### Define model and dataset parameters

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/other/qa_example.ipynb

Sets up key parameters for the language model and dataset, including the model path, device, model type, dataset name, batch size, and random seed for reproducibility.

```python
model_path = "bigscience/bloomz-560m"
device = "cuda"
model_type = "Whitebox"
dataset_name = ("trivia_qa", "rc.nocontext")
batch_size = 4
seed = 42
```

--------------------------------

### Load Autoreload and Import Libraries

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/basic_example.ipynb

Enables automatic reloading of modules and imports necessary classes and functions from transformers and lm-polygraph. This setup is required before using the library's functionalities.

```python
%load_ext autoreload
%autoreload 2

from transformers import AutoModelForCausalLM, AutoTokenizer
from lm_polygraph.utils.model import WhiteboxModel, BlackboxModel
from lm_polygraph import estimate_uncertainty
from lm_polygraph.estimators import MaximumTokenProbability, MaximumSequenceProbability, SemanticEntropy, EigValLaplacian
from lm_polygraph.utils.generation_parameters import GenerationParameters
```

--------------------------------

### Initialize WhiteboxModel using from_pretrained (Deprecated)

Source: https://github.com/iinemo/lm-polygraph/blob/main/docs/usage.md

Initialize WhiteboxModel using the `from_pretrained` method. This approach is deprecated and will be removed in a future release. LM-Polygraph will handle model and tokenizer downloads.

```python
from lm_polygraph.utils.model import WhiteboxModel

model = WhiteboxModel.from_pretrained(
    "bigscience/bloomz-3b",
    device_map="cuda:0",
)
```

--------------------------------

### Generate and Save MSE Plot

Source: https://github.com/iinemo/lm-polygraph/blob/main/notebooks/normalization.ipynb

Sets up a matplotlib figure and axes to plot the MSE results using the `plot_mses` function. The plot is then saved to a PDF file. Ensure matplotlib is installed.

```python
import matplotlib.pyplot as plt

x = np.array(list(range(len(UE_METHOD_NAMES))))

f, ax = plt.subplots(1, 1, figsize=(9, 7))

plot_mses(ax, mses, 'MSE between AlignScore and confidence')

# handles, labels = ax.get_legend_handles_labels()
# f.legend(handles, labels, bbox_to_anchor=(1.15, 0.96), fontsize=12)

plt.tight_layout()
# Change this to plt.show() to display inline
plt.savefig(f'normalization_mse_total.pdf', bbox_inches='tight')
# plt.show()
plt.clf()
```

--------------------------------

### Build and Publish Dataset to Hugging Face

Source: https://github.com/iinemo/lm-polygraph/blob/main/dataset_builders/README.md

Builds a specified dataset and publishes it to Hugging Face. Requires the `HF_TOKEN` environment variable to be set. A new repository is created if the dataset has not been published before.

```bash
export HF_TOKEN={your hf token}
python3 manager.py --dataset {DATASET} --publish
```

--------------------------------

### Use LM-Polygraph with OpenAI-Compatible Services

Source: https://github.com/iinemo/lm-polygraph/blob/main/README.md

Instantiate a BlackboxModel from an OpenAI-compatible API. Ensure 'supports_logprobs' is set to True for deployments that provide log probabilities. This setup is suitable for models like GPT-4o.

```python
from lm_polygraph import BlackboxModel
from lm_polygraph.estimators import Perplexity, MaximumSequenceProbability

model = BlackboxModel.from_openai(
    openai_api_key='YOUR_API_KEY',
    model_path='gpt-4o',
    supports_logprobs=True  # Enable for deployments
)

ue_method = Perplexity()  # or MeanTokenEntropy(), EigValLaplacian(), etc.
estimate_uncertainty(model, ue_method, input_text='What has a head and a tail but no body?')

```

--------------------------------

### Import necessary LM-Polygraph and Transformers libraries

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/other/qa_example.ipynb

Imports core components from transformers and lm-polygraph for model loading, uncertainty estimation, dataset handling, and utility functions. This setup is crucial for all subsequent operations.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from lm_polygraph.estimators import *
from lm_polygraph.utils.model import WhiteboxModel
from lm_polygraph.utils.dataset import Dataset
from lm_polygraph.utils.processor import Logger
from lm_polygraph.utils.manager import UEManager
from lm_polygraph.ue_metrics import PredictionRejectionArea
from lm_polygraph.generation_metrics import RougeMetric, BartScoreSeqMetric, ModelScoreSeqMetric, ModelScoreTokenwiseMetric, AggregatedMetric
from lm_polygraph.utils.builder_enviroment_stat_calculator import \
    BuilderEnvironmentStatCalculator
from lm_polygraph.defaults.register_default_stat_calculators import \
    register_default_stat_calculators
from lm_polygraph.utils.factory_stat_calculator import StatCalculatorContainer
from omegaconf import OmegaConf
```

--------------------------------

### Estimate Uncertainty for Single Input

Source: https://github.com/iinemo/lm-polygraph/blob/main/docs/usage.md

Use the `estimate_uncertainty` function to get predictions and uncertainty scores for a given input text, model, and UE method. The output includes uncertainty, input text, generation text, and model path.

```python
from lm_polygraph.utils.manager import estimate_uncertainty

input_text = "Who is George Bush?"
ue = estimate_uncertainty(model, ue_method, input_text=input_text)
print(ue)
# UncertaintyOutput(uncertainty=-6.504108926902215, input_text='Who is George Bush?', generation_text=' President of the United States', model_path='bigscience/bloomz-560m')
```

--------------------------------

### Initialize LM-Polygraph Components for Uncertainty Estimation

Source: https://github.com/iinemo/lm-polygraph/blob/main/examples/low_level_vllm_example.ipynb

Sets up various LM-Polygraph components including model adapters, statistical calculators, and estimators for uncertainty quantification. Requires Deberta for NLI models.

```python
from lm_polygraph.model_adapters import WhiteboxModelvLLM
from lm_polygraph.stat_calculators.greedy_alternatives_nli import GreedyAlternativesNLICalculator
from lm_polygraph.stat_calculators.cross_encoder_similarity import CrossEncoderSimilarityMatrixCalculator
from lm_polygraph.stat_calculators.semantic_matrix import SemanticMatrixCalculator
from lm_polygraph.stat_calculators.semantic_classes import SemanticClassesCalculator
from lm_polygraph.stat_calculators.greedy_probs import GreedyProbsCalculator
from lm_polygraph.stat_calculators.sample import SamplingGenerationCalculator
from lm_polygraph.estimators import MaximumSequenceProbability, ClaimConditionedProbability, DegMat, SemanticEntropy, SAR
from lm_polygraph.utils.deberta import Deberta

from torch.utils.data import DataLoader

model_adapter = WhiteboxModelvLLM(llm, sampling_params=sampling_params, device=device)

calc_infer_llm = GreedyProbsCalculator()
nli_model = Deberta(device=device)
nli_model.setup()
calc_nli = GreedyAlternativesNLICalculator(nli_model=nli_model)

calc_samples = SamplingGenerationCalculator()
calc_cross_encoder = CrossEncoderSimilarityMatrixCalculator()
calc_semantic_matrix = SemanticMatrixCalculator(nli_model=nli_model)
calc_semantic_classes = SemanticClassesCalculator()

# You can use one of the estimators from the library, here, just for example, we use multiple estimators
estimators = [MaximumSequenceProbability(), 
              ClaimConditionedProbability(),
              DegMat(), 
              SemanticEntropy(), 
              SAR()]
```