### Install Python Requirements

Source: https://github.com/allenai/scifact/blob/master/README.md

Installs all necessary Python packages listed in the requirements.txt file.

```bash
pip install -r requirements.txt
```

--------------------------------

### Create Anaconda Environment

Source: https://github.com/allenai/scifact/blob/master/README.md

Recommended setup for creating an isolated Python environment for the project. Ensure you have conda installed.

```bash
conda create --name scifact python=3.7 conda-build
```

--------------------------------

### Example Claim Data

Source: https://github.com/allenai/scifact/blob/master/doc/data.md

An example of a claim entry in JSONL format, illustrating the structure for ID, claim text, evidence, and cited documents.

```json
{
  "id": 263,
  "claim": "Citrullinated proteins externalized in neutrophil extracellular traps act indirectly to disrupt the inflammatory cycle.",
  "evidence": {
    "11328820": [
      { "sentences": [7, 9],
        "label": "CONTRADICT" }
    ],
    "30041340": [
      { "sentences": [0, 1],
        "label": "CONTRADICT" },
      { "sentences": [11],
        "label": "CONTRADICT" }
    ]
  },
  "cited_doc_ids": [
    11328820,
    30041340,
    14853989
  ]
}
```

--------------------------------

### SciFact Submission Format Example

Source: https://github.com/allenai/scifact/blob/master/doc/evaluation.md

This is an example of the JSON structure expected for a single claim in a leaderboard submission. It includes the claim ID and a dictionary of evidence documents, each with a predicted label and a list of supporting sentence indices.

```python
{
    "id": number,                   # An integer claim ID.
    "evidence": {                   # The evidence for the claim.
        [doc_id]: {                 # The sentences and label for a single document, keyed by S2ORC ID.
            "label": enum("SUPPORT" | "CONTRADICT"),
            "sentences": number[]
        }
    },
}
```

```python
{
    'id': 84,
    'evidence': {
        '22406695': {'sentences': [1], 'label': 'SUPPORT'},
        '7521113': {'sentences': [4], 'label': 'SUPPORT'}
    }
}
```

--------------------------------

### Example Corpus Data

Source: https://github.com/allenai/scifact/blob/master/doc/data.md

An example of a corpus entry, showing document ID, title, abstract, and structured flag.

```json
{
  "doc_id": 4983,
  "title": "Microstructural development of human newborn cerebral white matter assessed in vivo by diffusion tensor magnetic resonance imaging.",
  "abstract": [
    "Alterations of the architecture of cerebral white matter in the developing human brain can affect cortical development and result in functional disabilities.",
    ...
  ],
  "structured": false
}
```

--------------------------------

### Run Abstract Retrieval Oracle

Source: https://github.com/allenai/scifact/blob/master/doc/model.md

Executes the oracle for abstract retrieval. Use this to get perfect abstract retrieval predictions.

```sh
python verisci/inference/abstract_retrieval/oracle.py \
    --dataset data/claims_dev.jsonl \
    --output abstract_retrieval.jsonl
```

--------------------------------

### Verify COVID-19 Claims with Options

Source: https://github.com/allenai/scifact/blob/master/README.md

An example of using the COVID-19 claim verification script with specific optional arguments for document count, rationale selection, and verbosity.

```shell
python script/verify_covid.py \
  "Coronavirus droplets can remain airborne for hours" \
  results/covid-report \
  --n_documents=100 \
  --rationale_selection_method=threshold \
  --rationale_threshold=0.2 \
  --verbose \
  --full_abstract
```

--------------------------------

### Run Full Pipeline: Oracle Retrieval + Oracle Rationale

Source: https://context7.com/allenai/scifact/llms.txt

Executes the full pipeline using oracle retrieval and oracle rationale for an upper bound evaluation. Specify 'dev' or 'test' for the dataset split.

```bash
./script/pipeline.sh oracle oracle-rationale dev
```

--------------------------------

### Run Rationale Selection First Sentence

Source: https://github.com/allenai/scifact/blob/master/doc/model.md

Selects the first sentence as the rationale. A simple baseline for rationale selection.

```sh
python verisci/inference/rationale_selection/first.py \
    --abstract-retrieval abstract_retrieval.jsonl \
    --output rationale_selection.jsonl
```

--------------------------------

### Run Rationale Selection Metrics

Source: https://github.com/allenai/scifact/blob/master/README.md

Executes scripts to reproduce rationale selection metrics from the paper. Requires specifying BERT variant, training dataset, and target dataset.

```bash
./script/rationale-selection.sh [bert-variant] [training-dataset] [dataset]
```

--------------------------------

### Download Pre-trained Model Components

Source: https://github.com/allenai/scifact/blob/master/README.md

Download pre-trained model components using the provided script. Specify the model component, BERT variant, and training dataset.

```bash
./script/download-model.sh [model-component] [bert-variant] [training-dataset]
```

--------------------------------

### Run Oracle Abstract Retrieval

Source: https://context7.com/allenai/scifact/llms.txt

Executes the oracle retriever to obtain gold evidence annotations. Use the `--include-nei` flag to also include documents for NOT_ENOUGH_INFO claims.

```bash
python verisci/inference/abstract_retrieval/oracle.py \
    --dataset data/claims_dev.jsonl \
    --output prediction/abstract_retrieval.jsonl
```

```bash
python verisci/inference/abstract_retrieval/oracle.py \
    --dataset data/claims_dev.jsonl \
    --include-nei \
    --output prediction/abstract_retrieval.jsonl
```

--------------------------------

### Run Full Pipeline: Open Retrieval + Zero-Shot Model

Source: https://context7.com/allenai/scifact/llms.txt

Executes the full pipeline using open retrieval and a zero-shot model trained only on FEVER, without SciFact fine-tuning. Specify 'dev' or 'test'.

```bash
./script/pipeline.sh open zero-shot dev
```

--------------------------------

### Train Rationale Selection on SciFact

Source: https://github.com/allenai/scifact/blob/master/doc/training.md

Use this script to train the rationale selection module on the SciFact dataset. Ensure the corpus and claim files are correctly specified.

```shell
python verisci/training/rationale_selection/transformer_scifact.py \
    --corpus "PATH TO corpus.jsonl" \
    --claim-train "PATH TO claims_train.jsonl" \
    --claim-dev "PATH TO claims_dev.jsonl" \
    --model "roberta-large" \
    --dest "PATH TO WEIGHT SAVING FOLDER"
```

```shell
python verisci/training/rationale_selection/transformer_scifact.py \
    --corpus "data/corpus.jsonl" \
    --claim-train "data/claims_train.jsonl" \
    --claim-dev "data/claims_dev.jsonl" \
    --model "roberta-large" \
    --dest "model/rationale_roberta_large_scifact"
```

--------------------------------

### Run Pipeline Evaluation

Source: https://context7.com/allenai/scifact/llms.txt

Computes precision, recall, and F1 scores at both abstract and sentence levels. This script evaluates sentence selection, sentence labeling, abstract labeling (only), and abstract rationalization.

```bash
python verisci/evaluate/pipeline.py \
    --gold data/claims_dev.jsonl \
    --corpus data/corpus.jsonl \
    --prediction prediction/merged_predictions.jsonl \
    --output prediction/metrics.json
```

--------------------------------

### Run Full Pipeline: Open TF-IDF Retrieval + VeriSci Model

Source: https://context7.com/allenai/scifact/llms.txt

Executes the full pipeline with open TF-IDF retrieval and the VeriSci transformer model, replicating paper results. Specify 'dev' or 'test'.

```bash
./script/pipeline.sh open verisci dev
```

--------------------------------

### Run Rationale Selection Oracle + TF-IDF

Source: https://github.com/allenai/scifact/blob/master/doc/model.md

Combines oracle and TF-IDF for rationale selection, using the oracle when gold rationales are present.

```sh
python verisci/inference/rationale_selection/oracle_tfidf.py \
    --corpus data/corpus.jsonl \
    --dataset data/claims_dev.jsonl \
    --abstract-retrieval abstract_retrieval.jsonl \
    --output rationale_selection.jsonl
```

--------------------------------

### Download and Preprocess FEVER Dataset

Source: https://github.com/allenai/scifact/blob/master/doc/training.md

Commands to download the FEVER dataset and its Wikipedia dump, followed by preprocessing to convert it to the required format.

```shell
# Download Fever dataset:
wget https://fever.ai/download/fever/train.jsonl
wget https://fever.ai/download/fever/paper_dev.jsonl
wget https://fever.ai/download/fever/paper_test.jsonl

# Download Wikipedia Dump and unzip it manually.
wget https://fever.ai/download/fever/wiki-pages.zip
unzip wiki-pages.zip
```

```shell
python verisci/training/preprocess_fever.py \
    --wiki-folder "PATH TO wiki-pages folder" \
    --input "PATH TO train.jsonl or paper_dev.jsonl" \
    --output "PATH TO CONVERTED FILE fever_train.jsonl or fever_dev.jsonl"
```

--------------------------------

### Run TF-IDF Baseline Rationale Selection

Source: https://context7.com/allenai/scifact/llms.txt

Implements a TF-IDF baseline for rationale selection, identifying top-k similar sentences to the claim without a neural model. Configure n-gram range with `--min-gram` and `--max-gram`.

```bash
python verisci/inference/rationale_selection/tfidf.py \
    --corpus data/corpus.jsonl \
    --dataset data/claims_dev.jsonl \
    --abstract-retrieval prediction/abstract_retrieval.jsonl \
    --min-gram 1 \
    --max-gram 1 \
    --k 2 \
    --output prediction/rationale_selection.jsonl
```

--------------------------------

### Download SciFact Dataset

Source: https://github.com/allenai/scifact/blob/master/README.md

Use this script to download the SciFact dataset and store it in the 'data' directory. Alternatively, download the tarball directly.

```bash
./script/download-data.sh
```

--------------------------------

### Preprocess FEVER Data for Training

Source: https://context7.com/allenai/scifact/llms.txt

Preprocesses the FEVER dataset into the SciFact format for subsequent training steps. Requires the wiki folder and input FEVER JSONL file.

```bash
python verisci/training/preprocess_fever.py \
    --wiki-folder wiki-pages/ \
    --input train.jsonl \
    --output fever_train.jsonl
```

--------------------------------

### Run Label Prediction Metrics

Source: https://github.com/allenai/scifact/blob/master/README.md

Executes scripts to reproduce label prediction metrics from the paper. Requires specifying BERT variant, training dataset, and target dataset.

```bash
./script/label-prediction.sh [bert-variant] [training-dataset] [dataset]
```

--------------------------------

### Download Claim Generation Data

Source: https://github.com/allenai/scifact/blob/master/README.md

Download the collection of claims along with their source documents and citation contexts using wget. The data will be saved in the 'data' directory.

```bash
wget https://scifact.s3-us-west-2.amazonaws.com/release/latest/claims_with_citances.jsonl -P data
```

--------------------------------

### Run Transformer Rationale Selection

Source: https://context7.com/allenai/scifact/llms.txt

Performs rationale selection using a transformer model. The `--threshold` argument controls sentence selection for flex output. Additional outputs for top-k sentences can be generated.

```bash
# Flex output: keep sentences with score >= 0.5
python verisci/inference/rationale_selection/transformer.py \
    --corpus data/corpus.jsonl \
    --dataset data/claims_dev.jsonl \
    --abstract-retrieval prediction/abstract_retrieval.jsonl \
    --model model/rationale_roberta_large_scifact/ \
    --threshold 0.5 \
    --output-flex prediction/rationale_selection.jsonl \
    --output-k2 prediction/rationale_selection_k2.jsonl \
    --output-k3 prediction/rationale_selection_k3.jsonl
```

```bash
# Pass only abstract sentences (no claim) to the model
python verisci/inference/rationale_selection/transformer.py \
    --corpus data/corpus.jsonl \
    --dataset data/claims_dev.jsonl \
    --abstract-retrieval prediction/abstract_retrieval.jsonl \
    --model model/rationale_roberta_large_scifact/ \
    --only-rationale \
    --output-flex prediction/rationale_selection.jsonl
```

--------------------------------

### Evaluate Label Prediction with Gold Rationale Inputs

Source: https://context7.com/allenai/scifact/llms.txt

Evaluates label prediction accuracy when using gold rationales as input. Requires corpus, dataset, and label prediction files.

```bash
python verisci/evaluate/label_accuracy_gold_inputs.py \
    --corpus data/corpus.jsonl \
    --dataset data/claims_dev.jsonl \
    --label-prediction prediction/label_prediction.jsonl
```

--------------------------------

### Train Rationale Selection on FEVER

Source: https://github.com/allenai/scifact/blob/master/doc/training.md

Use this script to train the rationale selection module on the FEVER dataset. Ensure the preprocessed train and dev files are correctly specified.

```shell
python verisci/training/rationale_selection/transformer_fever.py \
    --train "PATH TO fever_train.jsonl" \
    --dev "PATH TO fever_dev.jsonl" \
    --model "roberta-large" \
    --dest "PATH TO WEIGHT SAVING FOLDER"
```

--------------------------------

### Evaluate Rationale Selection

Source: https://context7.com/allenai/scifact/llms.txt

Evaluates the rationale selection component. Requires corpus, dataset, and rationale selection prediction files.

```bash
python verisci/evaluate/rationale_selection.py \
    --corpus data/corpus.jsonl \
    --dataset data/claims_dev.jsonl \
    --rationale-selection prediction/rationale_selection.jsonl
```

--------------------------------

### Train Rationale Selection on Snopes

Source: https://github.com/allenai/scifact/blob/master/doc/training.md

Use this script to train the rationale selection module on the Snopes dataset. Ensure the corpus and evidence files are correctly specified.

```shell
python verisci/training/rationale_selection/transformer_snopes.py \
    --corpus "PATH TO snopes.pages.json" \
    --evidence-train "PATH TO snopes.evidence.train.jsonl" \
    --evidence-dev "PATH TO snopes.evidence.dev.jsonl" \
    --model "roberta-large" \
    --dest "PATH TO WEIGHT SAVING FOLDER"
```

--------------------------------

### Load and Pretty-Print Claim Data

Source: https://github.com/allenai/scifact/blob/master/doc/data.md

Loads claim and corpus data using GoldDataset and pretty-prints a specific claim with its evidence. Requires 'verisci' library.

```python
from verisci.evaluate.lib.data import GoldDataset

data = GoldDataset("data/corpus.jsonl", "data/claims_train.jsonl")
claim = data.get_claim(123)
claim.pretty_print()
```

--------------------------------

### Run Rationale Selection Oracle

Source: https://github.com/allenai/scifact/blob/master/doc/model.md

Executes the oracle for rationale selection. Provides perfect rationale predictions.

```sh
python verisci/inference/rationale_selection/oracle.py \
    --dataset data/claims_dev.jsonl \
    --abstract-retrieval abstract_retrieval.jsonl \
    --output rationale_selection.jsonl
```

--------------------------------

### Run Full Pipeline Predictions

Source: https://github.com/allenai/scifact/blob/master/README.md

Executes the complete prediction pipeline, including retrieval and model inference. Specify retrieval method, model, and dataset.

```bash
./script/pipeline.sh [retrieval] [model] [dataset]
```

--------------------------------

### Generate Leaderboard Submission File

Source: https://context7.com/allenai/scifact/llms.txt

Python script to generate a JSONL file in the merged prediction format for leaderboard submission. Each JSON object represents one test claim.

```python
import json

from verisci.data import SciFactDataset
from verisci.predict import predict_merged


if __name__ == "__main__":
    # Load SciFact dataset
    dataset = SciFactDataset("data/claims_test.jsonl")

    # Predict merged results
    predictions = predict_merged(
        dataset,
        "model/rationale_roberta_large_scifact/",
        "model/label_roberta_large_fever_scifact/",
        "data/corpus.jsonl",
    )

    # Save predictions to JSONL file
    with open("prediction/merged_predictions.jsonl", "w", encoding="utf-8") as f:
        for prediction in predictions:
            f.write(json.dumps(prediction) + "\n")

```

--------------------------------

### Run Rationale Selection Last Sentence

Source: https://github.com/allenai/scifact/blob/master/doc/model.md

Selects the last sentence as the rationale. Another simple baseline for rationale selection.

```sh
python verisci/inference/rationale_selection/last.py \
    --corpus data/corpus.jsonl \
    --abstract-retrieval abstract_retrieval.jsonl \
    --output rationale_selection.jsonl
```

--------------------------------

### Load and Access Claims with GoldDataset

Source: https://context7.com/allenai/scifact/llms.txt

Use GoldDataset to load the corpus and claims. Retrieve individual claims by ID or iterate over all claims to access their evidence.

```python
from verisci.evaluate.lib.data import GoldDataset

# Load corpus and labelled claims (train or dev split)
data = GoldDataset("data/corpus.jsonl", "data/claims_dev.jsonl")
print(data)
# -> Corpus of 5183 documents. 300 claims.

# Retrieve a single claim by integer ID
claim = data.get_claim(263)
print(claim)
# -> Example 263: Citrullinated proteins externalized in neutrophil extracellular traps act indirectly...

# Pretty-print the claim together with all annotated evidence
claim.pretty_print()
# -> Example 263: ...
#    Evidence sets:
#    ####################
#    11328820: REFUTES
#    Set 0:
#        - <sentence 7 text>
#        - <sentence 9 text>
#    30041340: REFUTES
#    Set 0:
#        - <sentence 0 text>
#        - <sentence 1 text>
#    Set 1:
#        - <sentence 11 text>

# Iterate over all claims
for claim in data.claims:
    print(claim.id, len(claim.evidence), "evidence docs")
```

--------------------------------

### Train Rationale Selection Transformer on SciFact

Source: https://context7.com/allenai/scifact/llms.txt

Trains or fine-tunes the transformer rationale selection module on the SciFact dataset. Requires corpus, training, and development claim files, and a base model. Checkpoints are saved to the specified destination.

```bash
python verisci/training/rationale_selection/transformer_scifact.py \
    --corpus data/corpus.jsonl \
    --claim-train data/claims_train.jsonl \
    --claim-dev data/claims_dev.jsonl \
    --model roberta-large \
    --dest model/rationale_roberta_large_scifact \
    --batch-size-gpu 1 \
    --batch-size-accumulated 16 \
    --lr-base 1e-5 \
    --lr-linear 1e-3
```

--------------------------------

### Train Label Prediction Transformer on FEVER

Source: https://context7.com/allenai/scifact/llms.txt

Trains a transformer model for label prediction on the FEVER dataset. Requires preprocessed FEVER training and development files.

```bash
python verisci/training/label_prediction/transformer_fever.py \
    --train fever_train.jsonl \
    --dev fever_dev.jsonl \
    --model roberta-large \
    --dest model/label_roberta_large_fever
```

--------------------------------

### Run TF-IDF Abstract Retrieval

Source: https://context7.com/allenai/scifact/llms.txt

Execute the TF-IDF script to retrieve top-k relevant document IDs for claims. Configure corpus, dataset, k value, and n-gram range.

```bash
# Retrieve top 3 abstracts per claim using bigrams
python verisci/inference/abstract_retrieval/tfidf.py \
    --corpus data/corpus.jsonl \
    --dataset data/claims_dev.jsonl \
    --k 3 \
    --min-gram 1 \
    --max-gram 2 \
    --output prediction/abstract_retrieval.jsonl

# Output format (one JSON object per line):
# {"claim_id": 1, "doc_ids": [2500193, 4983, 11328820]}
# When run on non-test data, also prints:
# Mid reciprocal rank: 3
# Avg reciprocal rank: 4.2
```

--------------------------------

### Run Label Prediction Oracle

Source: https://github.com/allenai/scifact/blob/master/doc/model.md

Executes the oracle for label prediction. Provides perfect label predictions.

```sh
python verisci/inference/label_prediction/oracle.py \
    --dataset data/claims_dev.jsonl \
    --rationale-selection rationale_selection.jsonl \
    --output label_prediction.jsonl
```

--------------------------------

### Train Label Prediction on SciFact

Source: https://github.com/allenai/scifact/blob/master/doc/training.md

Use this script to train the label prediction module on the SciFact dataset. Ensure the corpus and claim files are correctly specified.

```shell
python verisci/training/label_prediction/transformer_scifact.py \
    --corpus "PATH TO corpus.jsonl" \
    --claim-train "PATH TO claims_train.jsonl" \
    --claim-dev "PATH TO claims_dev.jsonl" \
    --model "roberta-large" \
    --dest "PATH TO WEIGHT SAVING FOLDER"
```

--------------------------------

### Develop SciFact Package

Source: https://github.com/allenai/scifact/blob/master/README.md

Adds the scifact code to your PYTHONPATH after creating the conda environment. Run this from the project root.

```bash
conda develop .
```

--------------------------------

### Train Label Prediction on Snopes

Source: https://github.com/allenai/scifact/blob/master/doc/training.md

Use this script to train the label prediction module on the Snopes dataset. Ensure the corpus and stance files are correctly specified.

```shell
python verisci/training/label_prediction/transformer_snopes.py \
    --corpus "PATH TO snopes.pages.json" \
    --evidence-train "PATH TO snopes.stance.train.jsonl" \
    --evidence-dev "PATH TO snopes.stance.dev.jsonl" \
    --model "roberta-large" \
    --dest "PATH TO WEIGHT SAVING FOLDER"
```

--------------------------------

### Claim Data Schema

Source: https://github.com/allenai/scifact/blob/master/doc/data.md

Schema for claim data in JSONL format. Includes claim text, evidence, and cited document IDs.

```python
{
    "id": number,                   # An integer claim ID.
    "claim": string,                # The text of the claim.
    "evidence": {                   # The evidence for the claim.
        [doc_id]: [                 # The rationales for a single document, keyed by S2ORC ID.
            {
                "label": enum("SUPPORT" | "CONTRADICT"),
                "sentences": number[]
            }
        ]
    },
    "cited_doc_ids": number[]       # The documents cited by this claim's source citation sentence.
}
```

--------------------------------

### Train Label Prediction on FEVER

Source: https://github.com/allenai/scifact/blob/master/doc/training.md

Use this script to train the label prediction module on the FEVER dataset. Ensure the preprocessed train and dev files are correctly specified.

```shell
python verisci/training/label_prediction/transformer_fever.py \
    --train "PATH TO fever_train.jsonl" \
    --dev "PATH TO fever_dev.jsonl" \
    --model "roberta-large" \
    --dest "PATH TO WEIGHT SAVING FOLDER"
```

--------------------------------

### Merge Label and Rationale Predictions

Source: https://github.com/allenai/scifact/blob/master/doc/model.md

Combines the outputs from rationale selection and label prediction into a single prediction file.

```sh
python verisci/inference/merge_predictions.py \
    --rationale-file prediction/rationale_selection.jsonl \
    --label-file prediction/label_prediction.jsonl \
    --result-file prediction/merged_predictions.jsonl
```

--------------------------------

### Evaluate Rationale Selection

Source: https://github.com/allenai/scifact/blob/master/doc/model.md

Evaluates the performance of the rationale selection component.

```sh
python verisci/evaluate/rationale_selection.py \
    --corpus data/corpus.jsonl \
    --dataset data/claims_dev.jsonl \
    --rationale-selection rationale_selection.jsonl
```

--------------------------------

### SciFact Prediction File Format

Source: https://context7.com/allenai/scifact/llms.txt

Each line in the submission file represents a claim with its predicted evidence and label. The 'evidence' field may be empty for claims where no evidence is found.

```json
{
    "id": 84,                     # integer claim ID
    "evidence": {                 # may be empty {} for NEI claims
        "22406695": {
            "sentences": [1],     # predicted rationale sentence indices
            "label": "SUPPORT"    # "SUPPORT" or "CONTRADICT" only (no NEI)
        },
        "7521113": {
            "sentences": [4],
            "label": "SUPPORT"
        }
    }
}
```

--------------------------------

### Fine-tune Label Prediction Transformer on SciFact

Source: https://context7.com/allenai/scifact/llms.txt

Fine-tunes a label prediction model, previously trained on FEVER, on the SciFact dataset. Requires corpus, SciFact claims, and the FEVER-trained model checkpoint.

```bash
python verisci/training/label_prediction/transformer_scifact.py \
    --corpus data/corpus.jsonl \
    --claim-train data/claims_train.jsonl \
    --claim-dev data/claims_dev.jsonl \
    --model model/label_roberta_large_fever/ \
    --dest model/label_roberta_large_fever_scifact
```

--------------------------------

### Run Rationale Selection TF-IDF

Source: https://github.com/allenai/scifact/blob/master/doc/model.md

Performs rationale selection using TF-IDF. This method ranks sentences based on TF-IDF scores.

```sh
python verisci/inference/rationale_selection/tfidf.py \
    --corpus data/corpus.jsonl \
    --dataset data/claims_dev.jsonl \
    --abstract-retrieval abstract_retrieval.jsonl \
    --min-gram 1 \
    --max-gram 1 \
    --k 2 \
    --output rationale_selection.jsonl
```

--------------------------------

### Evaluate Abstract Retrieval

Source: https://context7.com/allenai/scifact/llms.txt

Evaluates the abstract retrieval component of the pipeline. Requires dataset and abstract retrieval prediction files.

```bash
python verisci/evaluate/abstract_retrieval.py \
    --dataset data/claims_dev.jsonl \
    --abstract-retrieval prediction/abstract_retrieval.jsonl
```

--------------------------------

### Data Loading - GoldDataset

Source: https://context7.com/allenai/scifact/llms.txt

The GoldDataset class is used to load the corpus of evidence abstracts and the annotated claim file. It allows for iterating and retrieving individual claims with their evidence sets.

```APIDOC
## GoldDataset

### Description
Loads the corpus of evidence abstracts and the annotated claim file, resolves cross-references, and exposes methods for iterating and retrieving individual claims with their evidence sets.

### Methods
- `__init__(corpus_path: str, claims_path: str)`: Initializes the dataset by loading corpus and claims.
- `get_claim(claim_id: int)`: Retrieves a single claim by its integer ID.
- `claims`: Property to iterate over all claims.

### Usage Example
```python
from verisci.evaluate.lib.data import GoldDataset

data = GoldDataset("data/corpus.jsonl", "data/claims_dev.jsonl")
claim = data.get_claim(263)
claim.pretty_print()
for claim in data.claims:
    print(claim.id, len(claim.evidence), "evidence docs")
```
```

--------------------------------

### Access Corpus Documents with Corpus Class

Source: https://context7.com/allenai/scifact/llms.txt

Load a corpus from a JSONL file and access individual documents by their S2ORC ID. Documents contain titles and sentences.

```python
from verisci.evaluate.lib.data import Corpus

corpus = Corpus.from_jsonl("data/corpus.jsonl")
print(corpus)
# -> Corpus of 5183 documents.

# Retrieve a document by its S2ORC ID
doc = corpus.get_document(4983)
print(doc.title)
# -> Microstructural development of human newborn cerebral white matter...
print(doc.sentences[0])
# -> Alterations of the architecture of cerebral white matter...

# Serialize a document back to JSON
print(doc.dump())
# -> {"doc_id": 4983, "title": "...", "abstract": [...], "structured": false}
```

--------------------------------

### Inspect and Pretty-Print Claim Predictions

Source: https://context7.com/allenai/scifact/llms.txt

Inspect individual claim predictions by accessing the PredictedDataset by index. Use pretty_print() for a human-readable output of predictions, including evidence sets and rationale sentences.

```python
# Inspect a single claim's predictions
claim_pred = preds[0]
print(claim_pred)
# -> Predictions for 1: <claim text>

claim_pred.pretty_print()
# -> Predictions for 1: ...
#    Evidence sets:
#    ####################
#    4983: SUPPORTS
#        - <predicted rationale sentence>
```

--------------------------------

### Label Utilities - Label and make_label

Source: https://context7.com/allenai/scifact/llms.txt

Label is an enum for claim verification labels (SUPPORTED, CONTRADICTED, NOT_ENOUGH_INFO). make_label converts string representations to Label enum values.

```APIDOC
## Label Utilities

### Description
`Label` is an enum with three values: `SUPPORTS`, `REFUTES`, and `NEI`. `make_label` converts string representations found in JSONL files to `Label` enum values.

### Functions
- `make_label(label_str: str, allow_NEI: bool = True)`: Converts string labels to `Label` enum. Raises `ValueError` if `allow_NEI` is `False` and the input is 'NOT_ENOUGH_INFO'.

### Enum Values
- `Label.SUPPORTS`
- `Label.REFUTES`
- `Label.NEI`

### Usage Example
```python
from verisci.evaluate.lib.data import Label, make_label

print(make_label("SUPPORT"))
print(make_label("CONTRADICT"))
print(make_label("NOT_ENOUGH_INFO"))
try:
    make_label("NOT_ENOUGH_INFO", allow_NEI=False)
except ValueError as e:
    print(e)

gold_label = make_label("SUPPORT")
pred_label = Label.SUPPORTS
print(gold_label == pred_label)
```
```

--------------------------------

### Abstract Retrieval - TF-IDF

Source: https://context7.com/allenai/scifact/llms.txt

A command-line script to retrieve the top-k most relevant document IDs for each claim using TF-IDF.

```APIDOC
## Abstract Retrieval - TF-IDF

### Description
Ranks corpus documents by cosine similarity to the claim text using TF-IDF, returning the top-k most relevant document IDs for each claim.

### Command
```bash
python verisci/inference/abstract_retrieval/tfidf.py \
    --corpus data/corpus.jsonl \
    --dataset data/claims_dev.jsonl \
    --k 3 \
    --min-gram 1 \
    --max-gram 2 \
    --output prediction/abstract_retrieval.jsonl
```

### Parameters
- `--corpus` (str): Path to the corpus JSONL file.
- `--dataset` (str): Path to the claims JSONL file.
- `--k` (int): Number of top abstracts to retrieve per claim.
- `--min-gram` (int): Minimum n-gram size for TF-IDF calculation.
- `--max-gram` (int): Maximum n-gram size for TF-IDF calculation.
- `--output` (str): Path to save the prediction results.

### Output Format
Each line is a JSON object:
```json
{"claim_id": 1, "doc_ids": [2500193, 4983, 11328820]}
```
```

--------------------------------

### Run Rationale Selection Transformer

Source: https://github.com/allenai/scifact/blob/master/doc/model.md

Uses a transformer model for rationale selection. This is a more advanced method that requires a pre-trained model.

```sh
python verisci/inference/rationale_selection/transformer.py \
    --corpus data/corpus.jsonl \
    --dataset data/claims_dev.jsonl \
    --abstract-retrieval abstract_retrieval.jsonl \
    --model PATH_TO_MODEL \
    --output-flex rationale_selection_flex.jsonl \
    --output-k2 rationale_selection_k2.jsonl \
    --output-k3 rationale_selection_k3.jsonl \
    --output-k4 rationale_selection_k4.jsonl \
    --output-k5 rationale_selection_k5.jsonl
```

--------------------------------

### Utilize Label Enum and make_label Function

Source: https://context7.com/allenai/scifact/llms.txt

Convert string labels from datasets to the Label enum (SUPPORTS, REFUTES, NEI). The make_label function can optionally disallow NEI labels.

```python
from verisci.evaluate.lib.data import Label, make_label

# Convert dataset string labels to enum
print(make_label("SUPPORT"))       # -> Label.SUPPORTS
print(make_label("CONTRADICT"))    # -> Label.REFUTES
print(make_label("NOT_ENOUGH_INFO"))  # -> Label.NEI

# Disallow NEI (used in prediction contexts)
try:
    make_label("NOT_ENOUGH_INFO", allow_NEI=False)
except ValueError as e:
    print(e)  # -> An NEI was given.

# Compare labels in evaluation logic
gold_label = make_label("SUPPORT")
pred_label = Label.SUPPORTS
print(gold_label == pred_label)  # -> True
```

--------------------------------

### Run Abstract Retrieval TF-IDF

Source: https://github.com/allenai/scifact/blob/master/doc/model.md

Performs abstract retrieval using TF-IDF. This is a baseline method for retrieving relevant abstracts.

```sh
python verisci/inference/abstract_retrieval/tfidf.py \
    --corpus data/corpus.jsonl \
    --dataset data/claims_dev.jsonl \
    --k 3 \
    --min-gram 1 \
    --max-gram 2 \
    --output abstract_retrieval.jsonl
```

--------------------------------

### Pretty-Print Specific Evidence Document Predictions

Source: https://context7.com/allenai/scifact/llms.txt

Filter the pretty-print output to show predictions for a specific evidence document ID. This is useful for detailed analysis of a particular piece of evidence.

```python
# Check a specific predicted document
claim_pred.pretty_print(evidence_doc_id=4983)
```

--------------------------------

### Load Gold Dataset and Predictions with PredictedDataset

Source: https://context7.com/allenai/scifact/llms.txt

Use PredictedDataset to load a merged prediction file alongside the gold dataset for side-by-side comparison. Ensure the GoldDataset is initialized with the correct corpus and claims file paths.

```python
from verisci.evaluate.lib.data import GoldDataset, PredictedDataset

gold = GoldDataset("data/corpus.jsonl", "data/claims_dev.jsonl")
preds = PredictedDataset(gold, "prediction/merged_predictions.jsonl")
```

--------------------------------

### Generate Predictions for Test Set

Source: https://context7.com/allenai/scifact/llms.txt

Runs the pipeline to generate predictions for the test set without evaluation, as labels are not public. Predictions are saved to prediction/merged_predictions.jsonl.

```bash
./script/pipeline.sh open verisci test
```

--------------------------------

### Evaluate Label Prediction

Source: https://github.com/allenai/scifact/blob/master/doc/model.md

Evaluates the performance of the label prediction component.

```sh
python verisci/evaluate/label_prediction.py \
    --corpus data/corpus.jsonl \
    --dataset data/claims_dev.jsonl \
    --label-prediction label_prediction.jsonl
```

--------------------------------

### Evaluate Merged Pipeline Predictions

Source: https://github.com/allenai/scifact/blob/master/doc/model.md

Evaluates the performance of the entire pipeline using the merged predictions.

```sh
python verisci/evaluate/pipeline.py \
    --gold data/claims_dev.jsonl \
    --corpus data/corpus.jsonl \
    --prediction prediction/merged_predictions.jsonl \
    --output predictions/metrics.json   # If no `output`, provided, print to console.
```

--------------------------------

### Gold Data Structure for SciFact Evaluation

Source: https://github.com/allenai/scifact/blob/master/doc/evaluation.md

This JSON represents the gold standard data for a single claim, detailing which abstracts and sentences support or contradict the claim. It includes the claim text and cited document IDs.

```javascript
{
  "id": 52,
  "claim": "ALDH1 expression is associated with poorer prognosis for breast cancer primary tumors.",
  "evidence": {
      "11": [                           # 2 evidence sets in document 11 support the claim.
          { "sentences": [ 0, 1 ],      # Sentences 0 and 1, taken together, support the claim.
             "label": "SUPPORT" },
          { "sentences": [ 11 ],        # Sentence 11, on its own, supports the claim.
             "label": "SUPPORT" }
      ],
      "15": [                           # A single evidence set in document 15 supports the claim.
          { "sentences": [ 4 ],
             "label": "SUPPORT" }
      ]
  },
  "cited_doc_ids": [11, 15]
}
```

--------------------------------

### Evaluate Abstract Retrieval

Source: https://github.com/allenai/scifact/blob/master/doc/model.md

Evaluates the performance of the abstract retrieval component.

```sh
python verisci/evalaute/abstract_retrieval.py \
    --dataset data/claims_test.jsonl \
    --abstract-retrieval abstract_retrieval.jsonl
```

--------------------------------

### Predicted Data Structure for SciFact Evaluation

Source: https://github.com/allenai/scifact/blob/master/doc/evaluation.md

This JSON shows a model's prediction for a single claim. It includes the predicted relevant abstracts, their predicted labels, and the predicted rationale sentences.

```javascript
{
  "id": 52,
  "evidence": {
      "11": {
          "sentences": [ 1, 11, 13 ],     # Predicted rationale sentences.
          "label": "SUPPORT"              # Predicted label.
      },
      "16": {
        "sentences": [18, 20 ],
        "label": "REFUTES"
      }
  }
}
```

--------------------------------

### Run Label Prediction Transformer

Source: https://github.com/allenai/scifact/blob/master/doc/model.md

Uses a transformer model for label prediction. This method requires a pre-trained model and allows different input modes.

```sh
python verisci/inference/label_prediction/transformer.py \
    --corpus data/corpus.jsonl \
    --dataset data/claims_dev.jsonl \
    --rationale-selection rationale_selection.jsonl \
    --model PATH_TO_MODEL \
    --output label_prediction.jsonl
```

--------------------------------

### Verify COVID-19 Claims with Markdown Output

Source: https://context7.com/allenai/scifact/llms.txt

End-to-end script to verify COVID-19 claims, outputting a Markdown report. Useful when Pandoc is not available. Configurable with document count and option to keep 'NEI' (not enough information) results.

```bash
python script/verify_covid.py \
    "Remdesivir reduces COVID-19 mortality" \
    results/remdesivir-report \
    --output_format markdown \
    --n_documents 20 \
    --keep_nei
```

--------------------------------

### Verify COVID-19 Claims with PDF Output

Source: https://context7.com/allenai/scifact/llms.txt

End-to-end script to verify COVID-19 claims using the CORD-19 dataset. Generates a PDF report with ranked evidence, decisions, and highlighted sentences. Configurable with document count, rationale/label thresholds, and output format.

```bash
python script/verify_covid.py \
    "Coronavirus droplets can remain airborne for hours" \
    results/covid-report \
    --n_documents 100 \
    --rationale_selection_method threshold \
    --rationale_threshold 0.2 \
    --label_threshold 0.5 \
    --output_format pdf \
    --full_abstract \
    --verbose
```

--------------------------------

### Corpus Data Schema

Source: https://github.com/allenai/scifact/blob/master/doc/data.md

Schema for corpus data, including document ID, title, abstract sentences, and a structured abstract indicator.

```python
{
    "doc_id": number,               # The document's S2ORC ID.
    "title": string,                # The title.
    "abstract": string[],           # The abstract, written as a list of sentences.
    "structured": boolean           # Indicator for whether this is a structured abstract.
}
```

--------------------------------

### Corpus Access - Corpus and Document

Source: https://context7.com/allenai/scifact/llms.txt

The Corpus class wraps a list of Document objects. Each Document contains a document ID, title, and abstract split into sentences.

```APIDOC
## Corpus and Document

### Description
`Corpus` wraps a list of `Document` objects. Each `Document` holds a document ID (S2ORC), title, and abstract split into individual sentences.

### Methods
- `from_jsonl(jsonl_path: str)`: Class method to create a Corpus from a JSONL file.
- `get_document(doc_id: int)`: Retrieves a document by its S2ORC ID.

### Document Object Attributes
- `doc_id` (str): The S2ORC document ID.
- `title` (str): The title of the document.
- `sentences` (list[str]): A list of sentences in the abstract.

### Usage Example
```python
from verisci.evaluate.lib.data import Corpus

corpus = Corpus.from_jsonl("data/corpus.jsonl")
doc = corpus.get_document(4983)
print(doc.title)
print(doc.sentences[0])
print(doc.dump())
```
```

--------------------------------

### Verify COVID-19 Claims

Source: https://github.com/allenai/scifact/blob/master/README.md

Use this Python script to fact-check COVID-19 claims. It requires the claim text and a report file name. Optional arguments can be provided for customization.

```shell
python script/verify_covid.py [claim-text] [report-file] [optional-arguments].
```

--------------------------------

### Run Transformer Label Prediction

Source: https://context7.com/allenai/scifact/llms.txt

Classifies the relationship between a claim and concatenated rationale sentences. The `--mode only_claim` flag performs an ablation study by encoding only the claim.

```bash
# Default mode: encode claim concatenated with rationale
python verisci/inference/label_prediction/transformer.py \
    --corpus data/corpus.jsonl \
    --dataset data/claims_dev.jsonl \
    --rationale-selection prediction/rationale_selection.jsonl \
    --model model/label_roberta_large_fever_scifact/ \
    --output prediction/label_prediction.jsonl
```

```bash
# Encode only the claim (ablation)
python verisci/inference/label_prediction/transformer.py \
    --corpus data/corpus.jsonl \
    --dataset data/claims_dev.jsonl \
    --rationale-selection prediction/rationale_selection.jsonl \
    --model model/label_roberta_large_fever_scifact/ \
    --mode only_claim \
    --output prediction/label_prediction.jsonl
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.