### Load Example Dataset

Source: https://github.com/cvs-health/uqlm/blob/main/examples/black_box_demo.ipynb

Loads the 'popqa' dataset for use in the demonstration. Displays the head of the dataset after loading.

```python
popqa = load_example_dataset("popqa", n=200)
popqa.head()
```

--------------------------------

### Install UQLM Library

Source: https://github.com/cvs-health/uqlm/blob/main/assets/README_PYPI.md

Install the latest version of the UQLM library from PyPI using pip.

```bash
pip install uqlm
```

--------------------------------

### Load example dataset (FactScore)

Source: https://github.com/cvs-health/uqlm/blob/main/examples/long_text_uq_demo.ipynb

Loads the FactScore example dataset and selects relevant columns. This snippet is useful for quickly setting up a dataset for testing.

```python
# Load example dataset (FactScore)
factscore = load_example_dataset("factscore", n=5)[["hundredw_prompt", "wikipedia_text"]].rename(columns={"hundredw_prompt": "prompt"})
factscore.head()
```

--------------------------------

### Install Development Dependencies

Source: https://github.com/cvs-health/uqlm/blob/main/CONTRIBUTING.md

Install the project's development dependencies using uv. This ensures you have all necessary tools for development.

```bash
uv sync --group dev
```

--------------------------------

### Load Example Dataset and Define Prompts

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/score_calibration_demo.ipynb

Loads the NQ-Open dataset and defines training and testing prompts for calibration. Ensure the dataset is available or replace with your own data.

```python
from uqlm import WhiteBoxUQ
from uqlm.calibration import ScoreCalibrator, evaluate_calibration
from uqlm.utils import load_example_dataset, LLMGrader

n_train, n_test = 1000, 500
n_prompts = n_train + n_test

# Load example dataset for prompts/answers (optional, for context)
nq_open = load_example_dataset("nq_open", n=n_prompts)

# Define prompts
QA_INSTRUCTION = "You will be given a question. Return only the answer as concisely as possible without providing an explanation.\n"
prompts = [QA_INSTRUCTION + prompt for prompt in nq_open.question]
train_prompts = prompts[:n_train]
test_prompts = prompts[-n_test:]
```

--------------------------------

### Load Example Dataset

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/black_box_demo.ipynb

Loads the PopQA dataset for use in the demo. This function can be used to load other example datasets as well.

```python
import numpy as np
from sklearn.metrics import precision_score, recall_score, f1_score

from uqlm import BlackBoxUQ
from uqlm.utils import load_example_dataset, plot_model_accuracies, LLMGrader, Tuner

# Load example dataset (popqa)
popqa = load_example_dataset("popqa", n=200)
popqa.head()
```

--------------------------------

### Load Example Dataset

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/ensemble_tuning_demo.ipynb

Loads the SVAMP math question dataset for demonstration purposes. Replace with your own dataset for custom use cases.

```python
import numpy as np
from sklearn.metrics import precision_score, recall_score, f1_score

from uqlm import UQEnsemble
from uqlm.utils import load_example_dataset, math_postprocessor, plot_model_accuracies, plot_ranked_auc

# Load example dataset (svamp)
svamp = load_example_dataset("svamp", n=200)
svamp.head()
```

--------------------------------

### Load Example Dataset

Source: https://github.com/cvs-health/uqlm/blob/main/examples/langgraph_demo.ipynb

Loads the 'popqa' dataset for use in the demonstration. This function is used to fetch sample data for question-answering tasks.

```python
popqa = load_example_dataset("popqa", n=100)
popqa.head()
```

--------------------------------

### Load Example Dataset

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/ensemble_off_the_shelf_demo.ipynb

Loads the HotpotQA dataset for demonstration purposes. Replace with your own data for custom use cases.

```python
import numpy as np
from sklearn.metrics import precision_score, recall_score, f1_score

from uqlm import UQEnsemble
from uqlm.utils import load_example_dataset, plot_model_accuracies, LLMGrader, Tuner

# Load example dataset (hotpotqa)
hotpotqa = load_example_dataset("hotpotqa", n=200)
hotpotqa.head()
```

--------------------------------

### Load Example Dataset

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/semantic_density_demo.ipynb

Loads the 'simpleqa' dataset for demonstration purposes. This function is used to fetch pre-defined question-answer pairs for testing.

```python
import numpy as np
from sklearn.metrics import precision_score, recall_score, f1_score

from uqlm.utils import load_example_dataset, plot_model_accuracies, LLMGrader, Tuner
from uqlm.scorers import SemanticDensity

# Load example dataset (simpleqa)
simpleqa = load_example_dataset("simpleqa", n=200)
simpleqa.head()
```

--------------------------------

### Install uv Package Manager

Source: https://github.com/cvs-health/uqlm/blob/main/CONTRIBUTING.md

Install the uv package manager using pip or a shell script. This is a prerequisite for installing project dependencies.

```bash
pip install uv
```

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

--------------------------------

### Initialize ChatVertexAI Models (Alternative)

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/judges_demo.ipynb

Example of initializing ChatVertexAI models. This requires the 'langchain-google-vertexai' package and appropriate Google Cloud authentication. Replace with your preferred Vertex AI models.

```python
# import sys
# !{sys.executable} -m pip install langchain-google-vertexai
# from langchain_google_vertexai import ChatVertexAI

# gemini_pro = ChatVertexAI(model_name="gemini-2.5-pro")
# gemini_flash = ChatVertexAI(model_name="gemini-2.5-flash")
```

--------------------------------

### Load Example Dataset

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/white_box_single_generation_demo.ipynb

Loads the gsm8k dataset for use in the demo. Replace with your own data for custom use cases.

```python
import numpy as np
from sklearn.metrics import precision_score, recall_score, f1_score

from uqlm import WhiteBoxUQ
from uqlm.utils import load_example_dataset, math_postprocessor, plot_model_accuracies, Tuner
```

```python
# Load example dataset (gsm8k)
gsm8k = load_example_dataset("gsm8k", n=200)
gsm8k.head()
```

--------------------------------

### Install Pre-commit Hooks

Source: https://github.com/cvs-health/uqlm/blob/main/CONTRIBUTING.md

Install pre-commit hooks to automatically check and format code before each commit, ensuring code style compliance.

```bash
pre-commit install
```

--------------------------------

### Load Example Dataset

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/semantic_entropy_demo.ipynb

Loads the CommonSense QA dataset for demonstration purposes. Replace 'csqa' with your dataset name and adjust 'n' for the number of samples.

```python
import numpy as np
from sklearn.metrics import precision_score, recall_score, f1_score

from uqlm.utils import load_example_dataset, plot_model_accuracies, Tuner
from uqlm import SemanticEntropy

# Load example dataset (csqa)
csqa = load_example_dataset("csqa", n=200)
csqa.head()
```

--------------------------------

### Load example dataset

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/long_text_qa_demo.ipynb

Loads the 'factscore' dataset and selects relevant columns for the QA task. This prepares the data for use with the LongTextQA model.

```python
# Load example dataset (FactScore)
factscore = load_example_dataset("factscore", n=15)[["hundredw_prompt", "wikipedia_text"]].rename(columns={"hundredw_prompt": "prompt"})
factscore.head()
```

--------------------------------

### Load Example Dataset

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/judges_demo.ipynb

Loads the 'ai2_arc' dataset for use in the demo. This dataset contains multiple-choice questions. Replace with your own data as needed.

```python
from uqlm import LLMPanel
from uqlm.utils import load_example_dataset

# Load example dataset (ai2_arc)
ai2_arc = load_example_dataset("ai2_arc", n=75)
ai2_arc.head()
```

--------------------------------

### Load Example Dataset

Source: https://github.com/cvs-health/uqlm/blob/main/examples/white_box_single_generation_demo.ipynb

Loads the GSM8K math question dataset for use in the demonstration. The `n` parameter controls the number of samples loaded.

```python
import numpy as np
from sklearn.metrics import precision_score, recall_score, f1_score

from uqlm import WhiteBoxUQ
from uqlm.utils import load_example_dataset, math_postprocessor, plot_model_accuracies, Tuner

# Load example dataset (gsm8k)
gsm8k = load_example_dataset("gsm8k", n=200)
gsm8k.head()
```

--------------------------------

### Install UQLM in editable mode

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/contribute.rst

Install the UQLM package in editable mode. This allows you to make changes to the source code and have them reflected immediately without reinstalling.

```bash
pip install -e .
```

--------------------------------

### Load example dataset (FactScore)

Source: https://github.com/cvs-health/uqlm/blob/main/examples/long_text_graph_demo.ipynb

Loads the 'factscore' dataset and renames the 'hundredw_prompt' column to 'prompt'. This snippet is useful for quickly setting up a dataset for demonstration purposes.

```python
# Load example dataset (FactScore)
factscore = load_example_dataset("factscore", n=5)[["hundredw_prompt", "wikipedia_text"]].rename(columns={"hundredw_prompt": "prompt"})
factscore.head()

# # Alternative dataset (FactScore-STEM-Geo)
# factscore_stem_geo = load_example_dataset("factscore-stem-geo", n=5)
# factscore_stem_geo.head()
```

--------------------------------

### Load Example Dataset

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/white_box_multi_generation_demo.ipynb

Loads the gsm8k dataset for use in the demonstration. This function is used to fetch predefined datasets for testing.

```python
import numpy as np
from sklearn.metrics import precision_score, recall_score, f1_score

from uqlm import WhiteBoxUQ
from uqlm.utils import load_example_dataset, math_postprocessor, plot_model_accuracies, Tuner
```

```python
# Load example dataset (gsm8k)
gsm8k = load_example_dataset("gsm8k", n=100)
gsm8k.head()
```

--------------------------------

### Initialize WhiteBoxUQ with P(True) Scorer

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/scorer_definitions/white_box/p_true.rst

Demonstrates how to initialize the WhiteBoxUQ class with the 'p_true' scorer. This setup is necessary before generating responses and computing scores.

```python
from uqlm import WhiteBoxUQ

# Initialize with p_true scorer
wbuq = WhiteBoxUQ(
    llm=llm,
    scorers=["p_true"]
)

# Generate responses and compute scores
# Note: p_true generates one additional call per prompt
results = await wbuq.generate_and_score(prompts=prompts)

# Access the p_true scores
print(results.to_df()["p_true"])
```

--------------------------------

### Instantiate ChatVertexAI LLM

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/ensemble_off_the_shelf_demo.ipynb

Initializes the ChatVertexAI model. This can be replaced with any LangChain Chat Model. Ensure the necessary libraries are installed.

```python
# API example:
# !{sys.executable} -m pip install langchain-google-vertexai
from langchain_google_vertexai import ChatVertexAI

gemini_flash = ChatVertexAI(model="gemini-2.5-flash")
```

--------------------------------

### Initialize LLMPanel with Judges and Explanations

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/judges_demo.ipynb

Initializes the LLMPanel with a primary LLM, a list of judge LLMs, additional context, and enables explanations. This setup is used for generating responses and confidence scores.

```python
from uqlm import LLMPanel
from langchain_ollama import ChatOllama

# Assuming ollama_mistral, ollama_llama, ollama_qwen are already initialized
judges = [ollama_mistral, ollama_llama, ollama_qwen]
additional_context = "You are an expert in general-knowledge reasoning questions. Your task is to evaluate the correctness of the proposed answers to the provided questions."

# Option 1: With explanations
panel = LLMPanel(llm=ollama_mistral, judges=judges, additional_context=additional_context, explanations=True)
```

--------------------------------

### Import necessary libraries

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/long_text_graph_demo.ipynb

Imports core libraries for data manipulation, graph creation, and LLM integration. Ensure 'networkx' is installed for graph-based scoring.

```python
# import sys
# !{sys.executable} -m pip install networkx

import numpy as np

from uqlm import LongTextGraph
from uqlm.utils import load_example_dataset, display_response_refinement, claims_dicts_to_lists, plot_model_accuracies
from uqlm.longform import FactScoreGrader
```

--------------------------------

### Initialize WhiteBoxUQ with Probability Margin

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/scorer_definitions/white_box/probability_margin.rst

Initialize the WhiteBoxUQ class specifying 'probability_margin' as a scorer and setting top_k_logprobs to at least 2. This setup is required to enable the scorer.

```python
from uqlm import WhiteBoxUQ

# Initialize with probability_margin scorer
wbuq = WhiteBoxUQ(
    llm=llm,
    scorers=["probability_margin"],
    top_k_logprobs=5  # At least 2 required
)

# Generate responses and compute scores
results = await wbuq.generate_and_score(prompts=prompts)

# Access the probability_margin scores
print(results.to_df()["probability_margin"])
```

--------------------------------

### Initialize BlackBoxUQ with Semantic Sets Confidence

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/scorer_definitions/black_box/semantic_sets_confidence.rst

Demonstrates how to initialize the BlackBoxUQ class, specifying the 'semantic_sets_confidence' scorer and an NLI model. This setup is required before generating responses and computing scores.

```python
from uqlm import BlackBoxUQ

# Initialize with semantic_sets_confidence scorer
bbuq = BlackBoxUQ(
    llm=llm,
    scorers=["semantic_sets_confidence"],
    nli_model_name="microsoft/deberta-large-mnli"
)

# Generate responses and compute scores
results = await bbuq.generate_and_score(prompts=prompts, num_responses=5)

# Access the semantic_sets_confidence scores
print(results.to_df()["semantic_sets_confidence"])
```

--------------------------------

### Initialize CodeGenUQ with Token-Probability Scorers

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/scorer_definitions/code_generation/token_probability.rst

Instantiate CodeGenUQ with a list of desired token-probability scorer names and the programming language. This setup is used for generating code and scoring it based on token probabilities.

```python
from uqlm import CodeGenUQ

code_uq = CodeGenUQ(
    llm=llm,
    scorers=["sequence_probability", "min_probability"],
    language="python",
)

results = await code_uq.generate_and_score(prompts=prompts)
```

--------------------------------

### Initialize WhiteBoxUQ with Min Token Negentropy

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/scorer_definitions/white_box/min_token_negentropy.rst

Initialize the WhiteBoxUQ class with the 'min_token_negentropy' scorer and specify the number of top logprobs to use. This setup is required for generating responses and computing MinTN scores.

```python
from uqlm import WhiteBoxUQ

# Initialize with min_token_negentropy scorer
wbuq = WhiteBoxUQ(
    llm=llm,
    scorers=["min_token_negentropy"],
    top_k_logprobs=15  # Number of top logprobs to use
)

# Generate responses and compute scores
results = await wbuq.generate_and_score(prompts=prompts)

# Access the min_token_negentropy scores
print(results.to_df()["min_token_negentropy"])
```

--------------------------------

### LongTextUQ: Claim-Level Scoring and Refinement

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/getstarted.rst

Shows how to initialize LongTextUQ with specified scorers and enable response refinement. The example includes generating scores for multiple responses and accessing claim-specific data.

```python
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(model="gpt-4o")

from uqlm import LongTextUQ
luq = LongTextUQ(llm=llm, scorers=["entailment"], response_refinement=True)

results = await luq.generate_and_score(prompts=prompts, num_responses=5)
results_df = results.to_df()
results_df

# Preview the data for a specific claim in the first response
# results_df["claims_data"][0][0]
# Output:
# {
#   'claim': 'Suthida Bajrasudhabimalalakshana was born on June 3, 1978.',
#   'removed': False,
#   'entailment': 0.9548099517822266
# }
```

--------------------------------

### Generate and Score Responses with SemanticDensity

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/semantic_density_demo.ipynb

Use `generate_and_score` to get LLM responses, sampled responses, and compute the density score from prompts. This is best for end-to-end uncertainty quantification starting with prompts.

```python
results = await sd.generate_and_score(prompts=prompts, num_responses=5)
```

--------------------------------

### Initialize AzureChatOpenAI Model (Alternative)

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/judges_demo.ipynb

Example of initializing AzureChatOpenAI model. This requires the 'langchain-openai' package and a populated '.env' file with Azure OpenAI API credentials. Adjust deployment name and API version as needed.

```python
# import sys
# !{sys.executable} -m pip install langchain-openai

# # User to populate .env file with API credentials
# from dotenv import load_dotenv, find_dotenv
# from langchain_openai import AzureChatOpenAI

# load_dotenv(find_dotenv())
# original_llm = AzureChatOpenAI(
#     deployment_name="gpt-4o",
#     openai_api_type="azure",
#     openai_api_version="2024-02-15-preview",
#     temperature=1,  # User to set temperature
# )
```

--------------------------------

### Install Sphinx dependencies

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/contribute.rst

Install the Python packages required for Sphinx, the documentation generator used by the project.

```bash
make install
```

--------------------------------

### Navigate to Project Directory

Source: https://github.com/cvs-health/uqlm/blob/main/CONTRIBUTING.md

Change your current directory to the cloned UQLM project folder.

```bash
cd uqlm
```

--------------------------------

### UQEnsemble: Off-the-Shelf and Tuned Configurations

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/getstarted.rst

Demonstrates initializing UQEnsemble with default scorers or a custom list of scorers. The tuned ensemble requires tuning on provided prompts and ground truth answers before generating scores.

```python
from langchain_openai import AzureChatOpenAI
llm = AzureChatOpenAI(deployment_name="gpt-4o", openai_api_type="azure", openai_api_version="2024-12-01-preview")

from uqlm import UQEnsemble
## ---Option 1: Off-the-Shelf Ensemble---
# uqe = UQEnsemble(llm=llm)
# results = await uqe.generate_and_score(prompts=prompts, num_responses=5)

## ---Option 2: Tuned Ensemble---
scorers = [ # specify which scorers to include
   "exact_match", "noncontradiction", # black-box scorers
   "min_probability", # white-box scorer
   llm # use same LLM as a judge
]
uqe = UQEnsemble(llm=llm, scorers=scorers)

# Tune on tuning prompts with provided ground truth answers
tune_results = await uqe.tune(
   prompts=tuning_prompts, ground_truth_answers=ground_truth_answers
)
# ensemble is now tuned - generate responses on new prompts
results = await uqe.generate_and_score(prompts=prompts)
results.to_df()
```

--------------------------------

### Initialize UQEnsemble

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/ensemble_off_the_shelf_demo.ipynb

Initializes the UQEnsemble with the chosen LLM and device.

```python
bsd = UQEnsemble(llm=gemini_flash, device=device)
```

--------------------------------

### Initialize and Use Off-the-Shelf UQEnsemble

Source: https://github.com/cvs-health/uqlm/blob/main/README.md

Shows how to initialize UQEnsemble with default settings for immediate use without prior tuning. This is suitable for quick application of ensemble scoring.

```python
from langchain_openai import AzureChatOpenAI
llm = AzureChatOpenAI(deployment_name="gpt-4o", openai_api_type="azure", openai_api_version="2024-12-01-preview")

from uqlm import UQEnsemble
uqe = UQEnsemble(llm=llm)
results = await uqe.generate_and_score(prompts=prompts, num_responses=5)
```

--------------------------------

### Load Example Dataset

Source: https://github.com/cvs-health/uqlm/blob/main/examples/semantic_entropy_demo.ipynb

Loads the CommonSense QA (csqa) dataset. This function is used to fetch pre-defined example datasets for demonstration purposes.

```python
# Load example dataset (csqa)
csqa = load_example_dataset("csqa", n=200)
csqa.head()
```

--------------------------------

### Initialize LLMPanel without Explanations (Commented Out)

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/judges_demo.ipynb

This is a commented-out alternative for initializing the LLMPanel without generating explanations. Use this if confidence scores are sufficient and detailed explanations are not required.

```python
# Option 2: Without explanations
# panel = LLMPanel(llm=ollama_mistral, judges=judges, additional_context=additional_context))
```

--------------------------------

### Initialize and Use UQEnsemble with Tuned Scorers

Source: https://github.com/cvs-health/uqlm/blob/main/assets/README_PYPI.md

Demonstrates initializing UQEnsemble with specific scorers and tuning it on provided prompts and ground truth answers before generating and scoring new prompts. Requires LangChain Chat Model.

```python
from langchain_openai import AzureChatOpenAI
llm = AzureChatOpenAI(deployment_name="gpt-4o", openai_api_type="azure", openai_api_version="2024-12-01-preview")

from uqlm import UQEnsemble
scorers = [
    "exact_match", "noncontradiction",
    "min_probability",
    llm
]
uqe = UQEnsemble(llm=llm, scorers=scorers)

tune_results = await uqe.tune(
    prompts=tuning_prompts, ground_truth_answers=ground_truth_answers
)
results = await uqe.generate_and_score(prompts=prompts)
results.to_df()
```

--------------------------------

### Initialize Vertex AI LLM

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/semantic_density_demo.ipynb

Initializes the ChatVertexAI model for generating responses. Ensure you have the necessary libraries installed (`langchain-google-vertexai`).

```python
# import sys
# !{sys.executable} -m pip install langchain-google-vertexai
from langchain_google_vertexai import ChatVertexAI

llm = ChatVertexAI(model_name="gemini-2.5-pro")
```

--------------------------------

### Locally test the documentation site

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/contribute.rst

Build and serve the documentation locally to preview how it will appear on GitHub Pages. This allows for testing before deployment.

```bash
make local
```

--------------------------------

### Build documentation for GitHub Pages

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/contribute.rst

Build the project's documentation in a format suitable for deployment to GitHub Pages.

```bash
make github
```

--------------------------------

### Initialize WhiteBoxUQ with Semantic Density

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/scorer_definitions/white_box/semantic_density.rst

Initialize the WhiteBoxUQ class with the 'semantic_density' scorer. This setup is used for generating responses and computing scores.

```python
from uqlm import WhiteBoxUQ

# Initialize with semantic_density scorer
wbuq = WhiteBoxUQ(
    llm=llm,
    scorers=["semantic_density"],
    sampling_temperature=1.0,
    length_normalize=True
)

# Generate responses and compute scores
results = await wbuq.generate_and_score(prompts=prompts, num_responses=5)

# Access the semantic_density scores
print(results.to_df()["semantic_density"])
```

--------------------------------

### Initialize FactScoreGrader

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/long_text_qa_demo.ipynb

Set up the FactScoreGrader with a specified LLM. This is the first step before grading claims.

```python
grader = FactScoreGrader(llm=gemini_flash)
```

--------------------------------

### Instantiate LLM with Azure OpenAI

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/long_text_uq_demo.ipynb

Sets up an AzureChatOpenAI LLM instance. Ensure your API credentials are set in a .env file. Replace with your preferred LLM and configuration.

```python
# import sys
# !{sys.executable} -m pip install langchain-openai

## User to populate .env file with API credentials
from dotenv import load_dotenv, find_dotenv
from langchain_openai import AzureChatOpenAI

load_dotenv(find_dotenv())
llm = AzureChatOpenAI(
    deployment_name="gpt-4o",
    openai_api_type="azure",
    openai_api_version="2024-12-01-preview",
    temperature=1,  # User to set temperature
)
```

--------------------------------

### Initialize WhiteBoxUQ with Monte Carlo Probability Scorer

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/scorer_definitions/white_box/monte_carlo_probability.rst

Initialize the WhiteBoxUQ class with the 'monte_carlo_probability' scorer. This setup requires multiple samples to be generated for scoring.

```python
from uqlm import WhiteBoxUQ

# Initialize with monte_carlo_probability scorer
wbuq = WhiteBoxUQ(
    llm=llm,
    scorers=["monte_carlo_probability"],
    sampling_temperature=1.0
)

# Generate responses and compute scores (requires multiple samples)
results = await wbuq.generate_and_score(prompts=prompts, num_responses=5)

# Access the monte_carlo_probability scores
print(results.to_df()["monte_carlo_probability"])
```

--------------------------------

### Initialize AzureChatOpenAI LLM

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/semantic_entropy_demo.ipynb

Initializes the AzureChatOpenAI model. Ensure your environment variables (.env file) are set with the necessary API credentials. Replace 'gpt-4.1-mini' and '2024-02-15-preview' with your specific deployment details and API version. The temperature should be adjusted based on desired creativity/determinism.

```python
## AzureChatOpenAI example
# import sys
# !{sys.executable} -m pip install langchain-openai

# # User to populate .env file with API credentials
from dotenv import load_dotenv, find_dotenv
from langchain_openai import AzureChatOpenAI

load_dotenv(find_dotenv())
llm = AzureChatOpenAI(
    deployment_name="gpt-4.1-mini",
    openai_api_type="azure",
    openai_api_version="2024-02-15-preview",
    temperature=1,  # User to set temperature
)
```

--------------------------------

### Initialize and Tune UQEnsemble for Hallucination Detection

Source: https://github.com/cvs-health/uqlm/blob/main/README.md

Demonstrates initializing UQEnsemble with specified scorers and tuning it using provided prompts and ground truth answers for supervised learning. This is useful for training a custom ensemble model.

```python
from langchain_openai import AzureChatOpenAI
llm = AzureChatOpenAI(deployment_name="gpt-4o", openai_api_type="azure", openai_api_version="2024-12-01-preview")

from uqlm import UQEnsemble

scorers = [
    "exact_match", "noncontradiction",
    "min_probability",
    llm
]
uqe = UQEnsemble(llm=llm, scorers=scorers)

tune_results = await uqe.tune(
    prompts=tuning_prompts, ground_truth_answers=ground_truth_answers
)

results = await uqe.generate_and_score(prompts=prompts)
```

--------------------------------

### Initialize WhiteBoxUQ with Min Probability Scorer

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/scorer_definitions/white_box/min_probability.rst

Initialize the WhiteBoxUQ class with the 'min_probability' scorer. This setup is required to compute MTP scores for generated responses.

```python
from uqlm import WhiteBoxUQ

# Initialize with min_probability scorer
wbuq = WhiteBoxUQ(
    llm=llm,
    scorers=["min_probability"]
)

# Generate responses and compute scores
results = await wbuq.generate_and_score(prompts=prompts)

# Access the min_probability scores
print(results.to_df()["min_probability"])
```

--------------------------------

### Initialize and Use CodeGenUQ

Source: https://github.com/cvs-health/uqlm/blob/main/README.md

Demonstrates how to initialize CodeGenUQ with a specific LangChain chat model and scorer, and then use it to generate and score multiple responses for given prompts.

```python
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(model="gpt-4o-mini")

from uqlm import CodeGenUQ
cguq = CodeGenUQ(
    llm=llm,
    scorers=["functional_equivalence_rate"]
)

results = await cguq.generate_and_score(prompts=prompts, num_responses=5)
results.to_df()
```

--------------------------------

### Initialize LLMJudge with Continuous Template

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/scorer_definitions/llm_judges/continuous.rst

Initialize an LLMJudge instance using the 'continuous' scoring template. This setup is for scoring individual question-response pairs directly.

```python
from uqlm.judges import LLMJudge

# Initialize with continuous template
judge = LLMJudge(
    llm=judge_llm,
    scoring_template="continuous"
)

# Score responses
result = await judge.judge_responses(
    prompts=prompts,
    responses=responses
)

# Scores will be continuous values between 0 and 1
print(result["scores"])
```

--------------------------------

### Configure UQEnsemble with Scorers

Source: https://github.com/cvs-health/uqlm/blob/main/examples/ensemble_tuning_demo.ipynb

Sets up the UQEnsemble with a specified LLM and a list of scoring metrics. The `use_n_param` argument should be set to True when using OpenAI or AzureChatOpenAI models for potentially faster generation.

```python
scorers = [
    "exact_match",  # Measures proportion of candidate responses that match original response (black-box)
    "noncontradiction",  # mean non-contradiction probability between candidate responses and original response (black-box)
    # "cosine_sim", # Cosine similarity between candidate responses and original response (black-box)
    "sequence_probability",  # length-normalized joint token probability (white-box)
    gpt4o_mini,  # LLM-as-a-judge (self)
    # gemini_flash, # LLM-as-a-judge (separate LLM)
]

uqe = UQEnsemble(
    llm=gpt4o_mini,
    max_calls_per_min=1000,
    use_n_param=True,  # Set True if using AzureChatOpenAI or ChatOpenAI for faster generation
    scorers=scorers,
)
```

--------------------------------

### Initialize WhiteBoxUQ with Consistency and Confidence Scorer

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/scorer_definitions/white_box/consistency_and_confidence.rst

Initializes the WhiteBoxUQ class, specifying 'consistency_and_confidence' as a scorer. This setup is used for generating responses and computing their associated scores.

```python
from uqlm import WhiteBoxUQ

# Initialize with consistency_and_confidence scorer
wbuq = WhiteBoxUQ(
    llm=llm,
    scorers=["consistency_and_confidence"],
    sampling_temperature=1.0
)

# Generate responses and compute scores
results = await wbuq.generate_and_score(prompts=prompts, num_responses=5)

# Access the consistency_and_confidence scores
print(results.to_df()["consistency_and_confidence"])
```

--------------------------------

### Initialize BlackBoxUQ with Non-Contradiction Scorer

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/scorer_definitions/black_box/noncontradiction.rst

Initialize the BlackBoxUQ class with the 'noncontradiction' scorer and specify an NLI model. This setup is required before generating responses and computing scores.

```python
from uqlm import BlackBoxUQ

# Initialize with noncontradiction scorer
bbuq = BlackBoxUQ(
    llm=llm,
    scorers=["noncontradiction"],
    nli_model_name="microsoft/deberta-large-mnli"
)

# Generate responses and compute scores
results = await bbuq.generate_and_score(prompts=prompts, num_responses=5)

# Access the noncontradiction scores
print(results.to_df()["noncontradiction"])
```

--------------------------------

### WhiteBoxUQ.generate_and_score

Source: https://github.com/cvs-health/uqlm/blob/main/examples/white_box_multi_generation_demo.ipynb

Generates LLM responses and computes confidence scores for given prompts. This method is best used for complete end-to-end uncertainty quantification when starting with prompts.

```APIDOC
## WhiteBoxUQ.generate_and_score

### Description
Generate LLM responses and compute confidence scores for the provided prompts.

### Method
This is a class method.

### Parameters
#### Parameters
- **prompts** (List[str] or List[List[BaseMessage]]) - Required - A list of input prompts for the model.
- **num_responses** (int, default=5) - Optional - The number of sampled responses to generate for sampling-based white-box UQ methods. Only applies to "monte_carlo_negentropy", "consistency_and_confidence", "semantic_negentropy", "semantic_density".
- **show_progress_bars** (bool, default=True) - Optional - If True, displays a progress bar while generating and scoring responses.

### Returns
- <code>UQResult</code> - Containing data (prompts, responses, log probabilities, and confidence scores) and metadata

### Request Example
```python
results = await wbuq.generate_and_score(prompts=prompts, num_responses=5)
```

### Response Example
```json
Output()
```
```

--------------------------------

### Initialize AzureChatOpenAI LLM

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/ensemble_tuning_demo.ipynb

Initializes an AzureChatOpenAI model. Ensure your .env file contains the necessary API credentials. Replace with your LLM of choice if needed.

```python
# import os
# import sys
# !{sys.executable} -m pip install python-dotenv
# !{sys.executable} -m pip install langchain-openai

# # User to populate .env file with API credentials. In this step, replace with your LLM of choice.
from dotenv import load_dotenv, find_dotenv
from langchain_openai import AzureChatOpenAI

load_dotenv(find_dotenv())
gpt4o_mini = AzureChatOpenAI(deployment_name="gpt-4o-mini", openai_api_type="azure", openai_api_version="2024-02-15-preview")
```

--------------------------------

### Generate and Score LLM Responses

Source: https://github.com/cvs-health/uqlm/blob/main/docs/source/_notebooks/examples/judges_demo.ipynb

Use `generate_and_score` to generate responses to prompts and have them scored by judges. This is best for complete end-to-end uncertainty quantification when starting with prompts.

```python
result = await panel.generate_and_score(prompts=prompts)
```

--------------------------------

### Load and sample dataset

Source: https://github.com/cvs-health/uqlm/blob/main/examples/codegen_demo.ipynb

Loads the 'livecodebench' dataset and samples 50 entries, excluding 'easy' difficulty problems. Resets the index for further processing.

```python
df = load_example_dataset("livecodebench")[["question_content", "platform", "starter_code", "public_test_cases", "metadata", "difficulty"]]
df = df.loc[df.difficulty != "easy"].sample(50).reset_index(drop=True)
```

--------------------------------

### Initialize WhiteBoxUQ Scorer

Source: https://github.com/cvs-health/uqlm/blob/main/examples/langgraph_demo.ipynb

Initializes the WhiteBoxUQ class with the LLM and specifies 'consistency_and_confidence' as a scorer. This setup is for generating multiple sampled responses per prompt to assess confidence.

```python
wbuq = WhiteBoxUQ(
    llm=llm,
    scorers=[
        "consistency_and_confidence"  # requires multiple sampled responses per prompt
    ],
    max_calls_per_min=125,
)
```