### Installation and Setup

Source: https://context7.com/shmsw25/factscore/llms.txt

Instructions for installing FActScore and setting up the necessary data and models.

```APIDOC
## Installation and Setup

Download and install FActScore with required dependencies, including the knowledge source database.

```bash
# Install FActScore from PyPI
pip install --upgrade factscore
python -m spacy download en_core_web_sm

# Download knowledge source and example data
# Requires LLAMA-7B weights path for local model (optional - skip for ChatGPT-only usage)
python -m factscore.download_data --llama_7B_HF_path "llama-7B"

# ChatGPT-only setup (no local model required)
python -m factscore.download_data

# Optional: specify custom directories
python -m factscore.download_data \
    --data_dir "/path/to/data" \
    --model_dir "/path/to/models" \
    --llama_7B_HF_path "llama-7B"
```
```

--------------------------------

### Install FActScore and Dependencies

Source: https://context7.com/shmsw25/factscore/llms.txt

Installs the FActScore library from PyPI, downloads necessary spaCy models, and optionally downloads knowledge source data and LLAMA weights for local model usage.

```bash
pip install --upgrade factscore
python -m spacy download en_core_web_sm
python -m factscore.download_data
# Optional: specify custom directories
# python -m factscore.download_data \
#     --data_dir "/path/to/data" \
#     --model_dir "/path/to/models" \
#     --llama_7B_HF_path "llama-7B"
```

--------------------------------

### Run FActScore Evaluation from Command Line

Source: https://context7.com/shmsw25/factscore/llms.txt

Provides examples of how to run FActScore evaluations directly from the command line using JSONL input files. It covers basic usage with ChatGPT, full options with LLAMA models, and using pre-computed atomic facts for reproducibility.

```bash
# Basic usage with ChatGPT
python -m factscore.factscorer \
    --input_path data/unlabeled/InstructGPT.jsonl \
    --model_name retrieval+ChatGPT \
    --openai_key api.key

# Full options with LLAMA model
python -m factscore.factscorer \
    --input_path data/generations.jsonl \
    --model_name retrieval+llama+npm \
    --openai_key api.key \
    --data_dir .cache/factscore \
    --model_dir .cache/factscore \
    --cache_dir .cache/factscore \
    --gamma 10 \
    --n_samples 100 \
    --verbose \
    --cost_estimate consider_cache \
    --abstain_detection_type generic

# Use pre-computed atomic facts (for reproducibility)
python -m factscore.factscorer \
    --input_path data/labeled/InstructGPT.jsonl \
    --model_name retrieval+ChatGPT \
    --openai_key api.key \
    --use_atomic_facts

# Input JSONL format (one JSON object per line):
# {"topic": "Albert Einstein", "output": "Albert Einstein was a German physicist..."}
# {"topic": "Marie Curie", "output": "Marie Curie was a Polish scientist..."}
```

--------------------------------

### Install FActScore Package and Dependencies

Source: https://github.com/shmsw25/factscore/blob/main/README.md

Installs the FActScore Python package and downloads the necessary spaCy English language model. This is a prerequisite for using FActScore.

```bash
pip install --upgrade factscore
python -m spacy download en_core_web_sm
```

--------------------------------

### Download FActScore Data

Source: https://github.com/shmsw25/factscore/blob/main/README.md

Downloads the knowledge source and example data for FActScore. It can also reconstruct the Inst-LLAMA model if the path to LLaMA 7B HuggingFace weights is provided. If the path is omitted, it defaults to using the ChatGPT version.

```python
python -m factscore.download_data --llama_7B_HF_path "llama-7B"
```

--------------------------------

### Register and Use Custom Knowledge Source with FActScore

Source: https://github.com/shmsw25/factscore/blob/main/README.md

Demonstrates how to register a custom knowledge source (e.g., from a .jsonl file) and then use it to compute FActScore. This involves creating a database from the custom data, which can be time-consuming for large datasets.

```python
from factscore.factscorer import FactScorer

fs = FactScorer()

# Register a custom knowledge source
# Replace 'name_of_your_knowledge_source', 'path_to_jsonl_file', and 'path_to_output_db_file' with actual paths and names
fs.register_knowledge_source(name_of_your_knowledge_source,
                             data_path=path_to_jsonl_file,
                             db_path=path_to_output_db_file)

# Compute score using the custom knowledge source
# Assuming topics and generations are defined lists of strings
out = fs.get_score(topics, generations, knowledge_source=name_of_your_knowledge_source)
print (out["score"])
print (out["respond_ratio"])
print (out["num_facts_per_response"])
```

--------------------------------

### Retrieve Passages from Knowledge Database

Source: https://context7.com/shmsw25/factscore/llms.txt

Explains how to initialize a document database and use the Retrieval system to fetch relevant passages for fact verification based on topics and queries.

```python
from factscore.retrieval import DocDB, Retrieval

db = DocDB(db_path=".cache/factscore/enwiki-20230401.db", data_path=None)

retrieval = Retrieval(
    db=db,
    cache_path=".cache/factscore/retrieval-enwiki.json",
    embed_cache_path=".cache/factscore/retrieval-enwiki.pkl",
    retrieval_type="gtr-t5-large",
    batch_size=256
)

passages = retrieval.get_passages(topic="Albert Einstein", question="Einstein developed the theory of relativity.", k=5)

for i, passage in enumerate(passages):
    print(f"Passage {i+1}: {passage['title']}")

retrieval.save_cache()
```

--------------------------------

### Register and Use Custom Knowledge Source

Source: https://context7.com/shmsw25/factscore/llms.txt

Demonstrates how to register a custom knowledge source from a JSONL file and use it within the FactScorer to evaluate model generations against specific data.

```python
from factscore.factscorer import FactScorer

fs = FactScorer(openai_key="api.key")

fs.register_knowledge_source(
    name="my_knowledge_base",
    data_path="/path/to/knowledge.jsonl",
    db_path="/path/to/knowledge.db"
)

topics = ["Custom Entity"]
generations = ["Custom Entity is known for various achievements."]

result = fs.get_score(
    topics=topics,
    generations=generations,
    knowledge_source="my_knowledge_base"
)

print(f"Score: {result['score']:.2%}")
```

--------------------------------

### Initialize FactScorer Class for Evaluation

Source: https://context7.com/shmsw25/factscore/llms.txt

Demonstrates how to initialize the FactScorer class for evaluating generated text. It shows configuration options for using ChatGPT or local LLAMA models, specifying data and cache directories, and handling API keys.

```python
from factscore.factscorer import FactScorer

# Initialize with ChatGPT (recommended - requires OpenAI API key)
fs = FactScorer(
    model_name="retrieval+ChatGPT",  # or "retrieval+llama+npm"
    openai_key="api.key",            # path to file containing OpenAI API key
    data_dir=".cache/factscore",     # knowledge source directory
    cache_dir=".cache/factscore",    # cache for API responses
    cost_estimate="consider_cache",  # estimate API costs
    abstain_detection_type=None      # optional: "generic" or "perplexity_ai"
)

# Define topics (Wikipedia article titles) and model generations
topics = [
    "Albert Einstein",
    "Marie Curie"
]

generations = [
    "Albert Einstein was a German-born theoretical physicist who developed the theory of relativity. He received the Nobel Prize in Physics in 1921 for his explanation of the photoelectric effect.",
    "Marie Curie was a Polish-French physicist and chemist. She was the first woman to win a Nobel Priz and remains the only person to win Nobel Prizes in two different sciences."
]

# Compute FActScore
result = fs.get_score(
    topics=topics,
    generations=generations,
    gamma=10,              # length penalty hyperparameter (0 to disable)
    knowledge_source=None, # uses default Wikipedia (enwiki-20230401)
    verbose=True           # show progress bar
)

# Access results
print(f"FActScore: {result['score']:.2%}")              # Main score with length penalty
print(f"FActScore (no penalty): {result['init_score']:.2%}")  # Score without length penalty
print(f"Response ratio: {result['respond_ratio']:.2%}")  # % of non-abstained responses
print(f"Facts per response: {result['num_facts_per_response']:.1f}")

# Access per-fact decisions
for i, decisions in enumerate(result['decisions']):
    if decisions:
        print(f"\n{topics[i]}:")
        for d in decisions:
            status = "supported" if d['is_supported'] else "NOT supported"
            print(f"  - {d['atom']}: {status}")
```

--------------------------------

### Command Line Interface

Source: https://context7.com/shmsw25/factscore/llms.txt

How to run FActScore evaluations using the command-line interface.

```APIDOC
## Command Line Interface

Run FActScore evaluation from the command line on JSONL input files.

```bash
# Basic usage with ChatGPT
python -m factscore.factscorer \
    --input_path data/unlabeled/InstructGPT.jsonl \
    --model_name retrieval+ChatGPT \
    --openai_key api.key

# Full options with LLAMA model
python -m factscore.factscorer \
    --input_path data/generations.jsonl \
    --model_name retrieval+llama+npm \
    --openai_key api.key \
    --data_dir .cache/factscore \
    --model_dir .cache/factscore \
    --cache_dir .cache/factscore \
    --gamma 10 \
    --n_samples 100 \
    --verbose \
    --cost_estimate consider_cache \
    --abstain_detection_type generic

# Use pre-computed atomic facts (for reproducibility)
python -m factscore.factscorer \
    --input_path data/labeled/InstructGPT.jsonl \
    --model_name retrieval+ChatGPT \
    --openai_key api.key \
    --use_atomic_facts

# Input JSONL format (one JSON object per line):
# {"topic": "Albert Einstein", "output": "Albert Einstein was a German physicist..."}
# {"topic": "Marie Curie", "output": "Marie Curie was a Polish scientist..."}
```
```

--------------------------------

### Initialize FactScorer for LM Evaluation (Python)

Source: https://github.com/shmsw25/factscore/blob/main/README.md

Instantiate the FactScorer class in Python to evaluate language models. This requires an OpenAI API key and a list of topics (human entities) for which generations will be evaluated.

```python
from factscore.factscorer import FactScorer

fs = FactScorer(openai_key="...")

# topics: list of strings (human entities used to generate bios)
```

--------------------------------

### Verifying Facts with NPM

Source: https://context7.com/shmsw25/factscore/llms.txt

Utilizes the Nonparametric Masked Language Model (NPM) for token-level probability estimation to verify facts. This includes setting up BM25 retrieval and integrating NPM into the FactScorer pipeline.

```python
from factscore.npm import NPM
from factscore.retrieval import DocDB, Retrieval

# Set up BM25 retrieval
db = DocDB(db_path=".cache/factscore/enwiki-20230401.db")
bm25_retrieval = Retrieval(db=db, retrieval_type="bm25")

# Initialize NPM model
npm = NPM(bm25=bm25_retrieval, model_name="npm-single")

# Get probability score
probability = npm.get_probabilty("Albert Einstein", "Einstein was born in Germany.")
print(f"NPM Probability: {probability:.4f}")
```

--------------------------------

### Run FActScore via Command Line

Source: https://github.com/shmsw25/factscore/blob/main/README.md

Execute FActScore to evaluate generated text using specified input paths, models, and OpenAI API keys. This command-line interface allows for detailed configuration through various flags to control data sources, model directories, caching, and evaluation parameters.

```bash
python -m factscore.factscorer --input_path {input_path} --model_name {estimator_name} --openai_key {openai_key}
```

--------------------------------

### Generate Atomic Facts from Text

Source: https://context7.com/shmsw25/factscore/llms.txt

Shows how to use the AtomicFactGenerator to break down long-form text into independent, verifiable statements using OpenAI models.

```python
from factscore.atomic_facts import AtomicFactGenerator

generator = AtomicFactGenerator(
    key_path="api.key",
    demon_dir=".cache/factscore/demos",
    gpt3_cache_file=".cache/factscore/InstructGPT.pkl"
)

text = "Thierry Henry (born 17 August 1977) is a French professional football coach..."

atomic_facts, para_breaks = generator.run(text)

for sentence, facts in atomic_facts:
    print(f"Sentence: {sentence}")
    for fact in facts:
        print(f"  - {fact}")

generator.save_cache()
```

--------------------------------

### Detecting Response Abstentions with FactScore

Source: https://context7.com/shmsw25/factscore/llms.txt

Demonstrates how to detect if a model has abstained from answering a prompt. It supports both generic detection and model-specific detection for Perplexity AI, and shows integration with the FactScorer class.

```python
from factscore.factscorer import FactScorer

# Direct function calls
text = "I could not find any information about John Doe."
print(f"Generic: {generic_abstain_detect(text)}")
print(f"Perplexity: {perplexity_ai_abstain_detect(text)}")

# Using in FactScorer
fs = FactScorer(
    openai_key="api.key",
    abstain_detection_type="generic"
)
```

--------------------------------

### Calculate FActScore with Default Knowledge Source

Source: https://github.com/shmsw25/factscore/blob/main/README.md

Calculates the FActScore for given topics and model generations using the default Wikipedia knowledge source. It outputs the main FActScore, initial score without length penalty, response ratio, and average number of facts per response.

```python
from factscore.factscorer import FactScorer

fs = FactScorer()
# Assuming topics and generations are defined lists of strings
out = fs.get_score(topics, generations, gamma=10)
print (out["score"])
print (out["init_score"])
print (out["respond_ratio"])
print (out["num_facts_per_response"])
```

--------------------------------

### Detect Model Abstention

Source: https://context7.com/shmsw25/factscore/llms.txt

Provides an overview of using abstain detection utilities to identify when a model fails to provide a valid answer.

```python
from factscore.abstain_detection import (
    is_response_abstained,
    perplexity_ai_abstain_detect,
    generic_abstain_detect
)

responses = [
    "Albert Einstein was a theoretical physicist who developed relativity.",
    "I'm sorry, I cannot provide information about that topic."
]
```

--------------------------------

### Calculate FActScore Statistics from Unlabeled Data

Source: https://github.com/shmsw25/factscore/blob/main/README.md

This script iterates through prediction files, parses JSON lines to extract atomic facts and labels, and computes the response ratio, average facts per response, and FactScore metrics using ChatGPT and LLAMA labels.

```python
import os
import json
import numpy as np

dirname = "factscore-unlabeled-predictions"
for fn in os.listdir(dirname):
    chatgpt_fs = []
    llama_fs = []
    n_facts = []
    with open(os.path.join(dirname, fn)) as f:
        for line in f:
            dp = json.loads(line)
            n_facts.append(len(dp["facts"]))
            if "ChatGPT_Labels" in dp:
                chatgpt_fs.append(np.mean([l=="S" for l in dp["ChatGPT_Labels"]]))
            llama_fs.append(np.mean([l=="S" for l in dp["LLAMA+NP_Labels"]]))
    print ("Model=%s\t(%.1f%% responding, %.1f facts/response)\tFactScore=%.1f (ChatGPT)\t%.1f (LLAMA)" % (
        fn.split(".")[0], len(n_facts)*100/500, np.mean(n_facts), np.mean(chatgpt_fs)*100, np.mean(llama_fs)*100
    ))
```

--------------------------------

### FactScorer Class Usage

Source: https://context7.com/shmsw25/factscore/llms.txt

How to use the FactScorer class in Python to compute factual precision scores.

```APIDOC
## FactScorer Class

The main class for computing factual precision scores on generated text using retrieval-augmented LLM verification.

```python
from factscore.factscorer import FactScorer

# Initialize with ChatGPT (recommended - requires OpenAI API key)
fs = FactScorer(
    model_name="retrieval+ChatGPT",  # or "retrieval+llama+npm"
    openai_key="api.key",            # path to file containing OpenAI API key
    data_dir=".cache/factscore",     # knowledge source directory
    cache_dir=".cache/factscore",    # cache for API responses
    cost_estimate="consider_cache",  # estimate API costs
    abstain_detection_type=None      # optional: "generic" or "perplexity_ai"
)

# Define topics (Wikipedia article titles) and model generations
topics = [
    "Albert Einstein",
    "Marie Curie"
]

generations = [
    "Albert Einstein was a German-born theoretical physicist who developed the theory of relativity. He received the Nobel Prize in Physics in 1921 for his explanation of the photoelectric effect.",
    "Marie Curie was a Polish-French physicist and chemist. She was the first woman to win a Nobel Prize and remains the only person to win Nobel Prizes in two different sciences."
]

# Compute FActScore
result = fs.get_score(
    topics=topics,
    generations=generations,
    gamma=10,              # length penalty hyperparameter (0 to disable)
    knowledge_source=None, # uses default Wikipedia (enwiki-20230401)
    verbose=True           # show progress bar
)

# Access results
print(f"FActScore: {result['score']:.2%}")              # Main score with length penalty
print(f"FActScore (no penalty): {result['init_score']:.2%}")  # Score without length penalty
print(f"Response ratio: {result['respond_ratio']:.2%}")  # % of non-abstained responses
print(f"Facts per response: {result['num_facts_per_response']:.1f}")

# Access per-fact decisions
for i, decisions in enumerate(result['decisions']):
    if decisions:
        print(f"\n{topics[i]}:")
        for d in decisions:
            status = "supported" if d['is_supported'] else "NOT supported"
            print(f"  - {d['atom']}: {status}")
```
```

--------------------------------

### Processing and Analyzing Evaluation Results

Source: https://context7.com/shmsw25/factscore/llms.txt

Parses JSON prediction files to compute aggregate statistics such as response rates, average facts per response, and FActScore accuracy for different models.

```python
import json
import numpy as np
import os

# Process prediction files
for filename in os.listdir("factscore-unlabeled-predictions"):
    with open(os.path.join("factscore-unlabeled-predictions", filename)) as f:
        for line in f:
            dp = json.loads(line)
            # Calculate per-response accuracy
            if "ChatGPT_Labels" in dp:
                score = np.mean([label == "S" for label in dp["ChatGPT_Labels"]])
                print(f"Score: {score}")
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.