### Install open_clip

Source: https://github.com/raivnlab/sugar-crepe/blob/main/README.md

Install the open_clip library, which is used for loading pretrained models. This is a prerequisite for running evaluations.

```bash
pip install open_clip_torch
```

--------------------------------

### Running GPT-4V Evaluation (Positive First)

Source: https://context7.com/raivnlab/sugar-crepe/llms.txt

Example of calling the evaluate_gpt4v function with the positive caption as the first option. This helps in evaluating position bias.

```python
# positive-first: correct caption is caption1
result = evaluate_gpt4v(
    image_path='./data/coco/images/val2017/000000219578.jpg',
    caption1='A cat and dog napping together on the couch.',      # positive
    caption2='A cat and a dog napping together under a blanket.', # negative
    client=openai.OpenAI()
)
print(result)  # '(1)' if correct
```

--------------------------------

### Text Retrieval with GrammarModel

Source: https://context7.com/raivnlab/sugar-crepe/llms.txt

This snippet demonstrates how to use the GrammarModel for text retrieval. It takes positive and negative text examples to determine a score.

```python
grammar = GrammarModel()
result = text_retrieval(
    pos_text='An assortment of rags hang on a metal rack.',
    neg_text='An assortment of rags and tools hang on a metal rack.',
    model=grammar
)
print(result)  # 1 or 0
```

--------------------------------

### Load OpenCLIP Model and Transforms

Source: https://context7.com/raivnlab/sugar-crepe/llms.txt

Instantiates a pretrained OpenCLIP model, its image preprocessing transform, and its text tokenizer. Sets the model to evaluation mode on the target device.

```python
import argparse
import open_clip
import torch

device = 'cuda' if torch.cuda.is_available() else 'cpu'

def load_model(args, pretrained, device):
    model, _, transform = open_clip.create_model_and_transforms(
        model_name=args.model,
        pretrained=pretrained,
        cache_dir=args.model_cache_dir,
        device=device
    )
    model = model.to(device)
    tokenizer = open_clip.get_tokenizer(args.model)
    model.eval()
    return model, tokenizer, transform

# Example usage
args = argparse.Namespace(model='ViT-B-32', pretrained='openai', model_cache_dir=None)
model, tokenizer, transform = load_model(args, 'openai', device)
print(type(model))      # <class 'open_clip.model.CLIP'>
print(type(tokenizer))  # <class 'open_clip.tokenizer.HFTokenizer'>
```

--------------------------------

### load_model

Source: https://context7.com/raivnlab/sugar-crepe/llms.txt

Instantiates a pretrained OpenCLIP model, its image preprocessing transform, and its text tokenizer, then sets the model to evaluation mode on the target device.

```APIDOC
## load_model — Load an OpenCLIP Model

Instantiates a pretrained OpenCLIP model, its image preprocessing transform, and its text tokenizer, then sets the model to evaluation mode on the target device.

```python
import argparse
import open_clip
import torch

device = 'cuda' if torch.cuda.is_available() else 'cpu'

def load_model(args, pretrained, device):
    model, _, transform = open_clip.create_model_and_transforms(
        model_name=args.model,
        pretrained=pretrained,
        cache_dir=args.model_cache_dir,
        device=device
    )
    model = model.to(device)
    tokenizer = open_clip.get_tokenizer(args.model)
    model.eval()
    return model, tokenizer, transform

# Example usage
args = argparse.Namespace(model='ViT-B-32', pretrained='openai', model_cache_dir=None)
model, tokenizer, transform = load_model(args, 'openai', device)
print(type(model))      # <class 'open_clip.model.CLIP'>
print(type(tokenizer))  # <class 'open_clip.tokenizer.HFTokenizer'>
```
```

--------------------------------

### Evaluate Single or All Pretrained CLIP Models via CLI

Source: https://context7.com/raivnlab/sugar-crepe/llms.txt

Command-line interface for evaluating CLIP models. Supports evaluating a single model with specified parameters or all models included in the paper. Can utilize a local model cache directory.

```bash
# Evaluate a single model (ViT-B-32 with OpenAI weights)
python main_eval.py \
    --model ViT-B-32 \
    --pretrained openai \
    --output ./output \
    --coco_image_root ./data/coco/images/val2017/ \
    --data_root ./data/
```

```bash
# Evaluate all 13 pretrained models from the paper
python main_eval.py \
    --all \
    --output ./output \
    --coco_image_root ./data/coco/images/val2017/ \
    --data_root ./data/
```

```bash
# Use a local model cache directory to avoid re-downloading
python main_eval.py \
    --model ViT-H-14 \
    --pretrained laion2b_s32b_b79k \
    --model_cache_dir ./model_cache \
    --output ./output \
    --coco_image_root ./data/coco/images/val2017/ \
    --data_root ./data/
```

--------------------------------

### `main_eval.py` CLI — Evaluate a Single or All Pretrained CLIP Models

Source: https://context7.com/raivnlab/sugar-crepe/llms.txt

Command-line entry point for running SugarCrepe evaluation on one OpenCLIP model or all 13 models included in the paper.

```APIDOC
## `main_eval.py` CLI — Evaluate a Single or All Pretrained CLIP Models

### Description
Command-line entry point for running SugarCrepe evaluation on one OpenCLIP model or all 13 models included in the paper.

### Usage

#### Evaluate a single model (ViT-B-32 with OpenAI weights)
```bash
python main_eval.py \
    --model ViT-B-32 \
    --pretrained openai \
    --output ./output \
    --coco_image_root ./data/coco/images/val2017/ \
    --data_root ./data/
```

#### Evaluate all 13 pretrained models from the paper
```bash
python main_eval.py \
    --all \
    --output ./output \
    --coco_image_root ./data/coco/images/val2017/ \
    --data_root ./data/
```

#### Use a local model cache directory to avoid re-downloading
```bash
python main_eval.py \
    --model ViT-H-14 \
    --pretrained laion2b_s32b_b79k \
    --model_cache_dir ./model_cache \
    --output ./output \
    --coco_image_root ./data/coco/images/val2017/ \
    --data_root ./data/
```

### Parameters
- **--model** (string) - The CLIP model architecture (e.g., ViT-B-32).
- **--pretrained** (string) - The pretrained weights to use (e.g., openai, laion2b_s32b_b79k).
- **--output** (string) - Directory to save the evaluation results.
- **--coco_image_root** (string) - Path to the COCO dataset image root directory.
- **--data_root** (string) - Path to the dataset root directory.
- **--all** (flag) - Evaluate all 13 pretrained models.
- **--model_cache_dir** (string) - Directory for caching downloaded models.
```

--------------------------------

### Run Full Benchmark Evaluation with CLIP

Source: https://context7.com/raivnlab/sugar-crepe/llms.txt

Iterates over dataset samples, performs text retrieval for image-caption pairs, and returns per-split accuracy scores. Requires dataset, model, tokenizer, transform, and device as input.

```python
import os
import json
from tqdm import tqdm
from PIL import Image

def evaluate(image_root, dataset, model, tokenizer, transform, device):
    metrics = {}
    for c, data_dict in dataset.items():
        correct_cnt = 0
        for i, data in tqdm(data_dict.items(), desc=f'evaluating {c}'):
            image_path = os.path.join(image_root, data['filename'])
            image = Image.open(image_path)
            correct = text_retrieval(
                data['caption'], data['negative_caption'],
                image, model, tokenizer, transform, device
            )
            correct_cnt += correct
        metrics[c] = correct_cnt / len(data_dict)
    return metrics

# Run evaluation and save results
metrics = evaluate('./data/coco/images/val2017/', full_dataset, model, tokenizer, transform, device)
print(metrics)
# {
#   'add_obj':     0.823,
#   'add_att':     0.791,
#   'replace_obj': 0.856,
#   'replace_att': 0.742,
#   'replace_rel': 0.698,
#   'swap_obj':    0.601,
#   'swap_att':    0.634
# }

os.makedirs('./output', exist_ok=True)
json.dump(metrics, open('./output/ViT-B-32-openai.json', 'w'), indent=4)
```

--------------------------------

### Evaluate text models

Source: https://github.com/raivnlab/sugar-crepe/blob/main/README.md

Run this command to evaluate text-only models, such as Vera and the Grammar model, on the SugarCrepe benchmark. Specify the output directory and data root.

```python
python text_model_eval.py
    --output ./output \
    --data_root ./data/
```

--------------------------------

### Load SugarCrepe Dataset Splits

Source: https://context7.com/raivnlab/sugar-crepe/llms.txt

Demonstrates how to load individual or all seven splits of the SugarCrepe dataset using Python's json library. Ensure the 'data/' directory contains the JSON files.

```python
import json

# Load a single SugarCrepe split
with open('./data/add_obj.json', 'r', encoding='utf-8') as f:
    dataset = json.load(f)

# Inspect a sample
sample = dataset['0']
print(sample['filename'])          # '000000219578.jpg'
print(sample['caption'])           # 'A cat and dog napping together on the couch.'
print(sample['negative_caption'])  # 'A cat and a dog napping together under a blanket on the couch.'

# Load all seven splits at once
data_root = './data'
splits = ['add_obj', 'add_att', 'replace_obj', 'replace_att', 'replace_rel', 'swap_obj', 'swap_att']
full_dataset = {}
for split in splits:
    with open(f'{data_root}/{split}.json', 'r', encoding='utf-8') as f:
        full_dataset[split] = json.load(f)

for split, data in full_dataset.items():
    print(f"{split}: {len(data)} samples")
# add_obj:     2062 samples
# add_att:      692 samples
# replace_obj: 1652 samples
# replace_att:  788 samples
# replace_rel: 1406 samples
# swap_obj:     246 samples
# swap_att:     666 samples
```

--------------------------------

### Evaluate all pretrained CLIP models

Source: https://github.com/raivnlab/sugar-crepe/blob/main/README.md

This command evaluates all 17 pretrained CLIP models included in the paper. It requires the output directory, COCO image root, and data root to be specified.

```python
python main_eval.py --all
    --output ./output \
    --coco_image_root ./data/coco/images/val2017/ \
    --data_root ./data/
```

--------------------------------

### text_model_eval.py CLI for Text-Only Evaluation

Source: https://context7.com/raivnlab/sugar-crepe/llms.txt

Command-line interface to evaluate Vera and GrammarModel on SugarCrepe splits using text-only retrieval. Supports specifying output and data directories, and an optional model cache directory.

```bash
python text_model_eval.py \
    --output ./output \
    --data_root ./data/

# With local model cache
python text_model_eval.py \
    --output ./output \
    --data_root ./data/ \
    --model_cache_dir ./model_cache

# Output files:
#   ./output/vera.json
#   ./output/grammar.json
#
# Example vera.json:
# {
#     "add_obj":     0.512,
#     "add_att":     0.508,
#     "replace_obj": 0.503,
#     "replace_att": 0.497,
#     "replace_rel": 0.495,
#     "swap_obj":    0.501,
#     "swap_att":    0.499
# }
# Near-chance performance (~0.50) confirms SugarCrepe's hard negatives are artifact-free.
```

--------------------------------

### evaluate (CLIP) — Run Full Benchmark Evaluation

Source: https://context7.com/raivnlab/sugar-crepe/llms.txt

Iterates over all samples in each dataset split, calls `text_retrieval` for every image-caption pair, and returns per-split accuracy scores.

```APIDOC
## `evaluate` (CLIP) — Run Full Benchmark Evaluation

### Description
Iterates over all samples in each dataset split, calls `text_retrieval` for every image-caption pair, and returns per-split accuracy scores.

### Parameters
- **image_root** (string) - Path to the root directory of images.
- **dataset** (dict) - A dictionary containing dataset splits and their corresponding data.
- **model** - The CLIP model to use for evaluation.
- **tokenizer** - The tokenizer for the CLIP model.
- **transform** - The image transformation to apply.
- **device** - The device to run the evaluation on (e.g., 'cuda' or 'cpu').

### Returns
- **metrics** (dict) - A dictionary containing accuracy scores for each dataset split.
```

--------------------------------

### GPT-4V Prompt for Image Captioning Evaluation

Source: https://github.com/raivnlab/sugar-crepe/blob/main/gpt-4v-results/readme.md

This Python snippet defines the prompt structure used to evaluate GPT-4V's ability to select the best caption for an image. It incorporates two provided captions and asks for a binary output.

```python
prompt = f"Which caption best describes the image?\n" \
                 f"(1) {caption1}\n (2) {caption2}\n" \
                 f"Output (1) or (2)."
```

--------------------------------

### Evaluate a pretrained model

Source: https://github.com/raivnlab/sugar-crepe/blob/main/README.md

Use this command to evaluate a specific pretrained model (e.g., RN50 with openai pretrained weights) on the SugarCrepe benchmark. Ensure you have downloaded the COCO-2017 validation set and placed it in the specified directory.

```python
python main_eval.py --model RN50 \
    --pretrained openai \
    --output ./output \
    --coco_image_root ./data/coco/images/val2017/ \
    --data_root ./data/
```

--------------------------------

### `Vera` Class — Plausibility Scoring with a T5 Text Model

Source: https://context7.com/raivnlab/sugar-crepe/llms.txt

Wraps the `liujch1998/vera` T5EncoderModel to score the factual plausibility of a text statement as a calibrated probability. Supports single-statement (`run`) and batched (`runs`) inference.

```APIDOC
## `Vera` Class — Plausibility Scoring with a T5 Text Model

### Description
Wraps the `liujch1998/vera` T5EncoderModel to score the factual plausibility of a text statement as a calibrated probability. Supports single-statement (`run`) and batched (`runs`) inference.

### Initialization
```python
Vera(model, model_cache_dir=None)
```

#### Parameters
- **model** (string) - The name or path of the T5 model to load (e.g., 'liujch1998/vera').
- **model_cache_dir** (string, optional) - Directory for caching downloaded models. Defaults to None.

### Methods

#### `run(statement)`
Score a single statement.

##### Parameters
- **statement** (string) - The text statement to score.

##### Returns
- **score** (float) - The plausibility score of the statement (a calibrated probability).

#### `runs(statements)`
Score a batch of statements.

##### Parameters
- **statements** (list of strings) - A list of text statements to score.

##### Returns
- **scores** (numpy.ndarray) - An array of plausibility scores for each statement in the batch.
```

--------------------------------

### GrammarModel for Grammaticality Scoring

Source: https://context7.com/raivnlab/sugar-crepe/llms.txt

Wraps a DistilBERT model fine-tuned on the CoLA corpus to score the grammatical acceptability of text. Scores are returned as probabilities in the range [0, 1].

```python
from transformers import pipeline
import torch
import numpy as np

class GrammarModel:
    def __init__(self, model_cache_dir=None):
        self.model = pipeline(
            "text-classification",
            model="textattack/distilbert-base-uncased-CoLA"
        )

    def run(self, statement):
        """Score grammaticality of a single statement."""
        with torch.no_grad():
            output = self.model(statement)[0]
            score = output['score'] if output['label'] == 'LABEL_1' else 1 - output['score']
        return score

    def runs(self, statements):
        """Score grammaticality for a list of statements."""
        with torch.no_grad():
            scores = []
            for output in self.model(statements):
                score = output['score'] if output['label'] == 'LABEL_1' else 1 - output['score']
                scores.append(score)
        return np.array(scores)

# Usage
grammar = GrammarModel()
print(grammar.run('A cat and dog napping together on the couch.'))   # ~0.97
print(grammar.run('Dog cat napping the a on together couch.'))       # ~0.03

scores = grammar.runs([
    'Three girls walking down the sidewalk.',
    'Girls three the sidewalk down walking.'
])
print(scores)  # array([0.96, 0.04])
```

--------------------------------

### Vera Class for Text Plausibility Scoring

Source: https://context7.com/raivnlab/sugar-crepe/llms.txt

Wraps a T5EncoderModel to score factual plausibility of text statements as calibrated probabilities. Supports scoring single statements or batches. Requires transformers and torch libraries.

```python
import transformers
import torch
import numpy as np

device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

class Vera:
    def __init__(self, model, model_cache_dir=None):
        self.tokenizer = transformers.AutoTokenizer.from_pretrained(model, cache_dir=model_cache_dir)
        self.model = transformers.T5EncoderModel.from_pretrained(model, torch_dtype='auto', cache_dir=model_cache_dir)
        self.model = self.model.to(device)
        self.model.D = self.model.shared.embedding_dim
        self.linear = torch.nn.Linear(self.model.D, 1, dtype=self.model.dtype).to(device)
        self.linear.weight = torch.nn.Parameter(self.model.shared.weight[32099, :].unsqueeze(0))
        self.linear.bias = torch.nn.Parameter(self.model.shared.weight[32098, 0].unsqueeze(0))
        self.model.eval()
        self.t = self.model.shared.weight[32097, 0].item()

    def run(self, statement):
        """Score a single statement."""
        input_ids = self.tokenizer.batch_encode_plus(
            [statement], return_tensors='pt', padding='longest',
            truncation='longest_first', max_length=128
        ).input_ids.to(device)
        with torch.no_grad():
            output = self.model(input_ids)
            hidden = output.last_hidden_state[0, -1, :]
            logit = self.linear(hidden).squeeze(-1)
            score = (logit / self.t).sigmoid()
        return score.item()

    def runs(self, statements):
        """Score a batch of statements."""
        tok = self.tokenizer.batch_encode_plus(statements, return_tensors='pt', padding='longest')
        with torch.no_grad():
            output = self.model(tok.input_ids.to(device), attention_mask=tok.attention_mask.to(device))
            last_indices = (tok.attention_mask.sum(dim=1, keepdim=True) - 1).unsqueeze(-1).expand(-1, -1, self.model.D)
            hidden = output.last_hidden_state.to(device).gather(dim=1, index=last_indices).squeeze(1)
            scores = (self.linear(hidden).squeeze(-1) / self.t).sigmoid()
        return np.array([s.item() for s in scores.detach().cpu()])
```

--------------------------------

### GPT-4V Evaluation Function

Source: https://context7.com/raivnlab/sugar-crepe/llms.txt

This function evaluates GPT-4V by comparing two captions for a given image. It requires the openai library and base64 encoding for image data. Ensure you are using openai version 1.0 or higher.

```python
import openai  # requires openai>=1.0
import base64

def encode_image(image_path):
    with open(image_path, 'rb') as f:
        return base64.b64encode(f.read()).decode('utf-8')

def evaluate_gpt4v(image_path, caption1, caption2, client):
    prompt = (
        f"Which caption best describes the image?\n"
        f"(1) {caption1}\n (2) {caption2}\n"
        f"Output (1) or (2)."
    )
    response = client.chat.completions.create(
        model='gpt-4-vision-preview',
        messages=[{
            'role': 'user',
            'content': [
                {'type': 'image_url', 'image_url': {'url': f"data:image/jpeg;base64,{encode_image(image_path)}"}},
                {'type': 'text', 'text': prompt}
            ]
        }],
        max_tokens=10
    )
    return response.choices[0].message.content.strip()
```

--------------------------------

### Vera Plausibility Scoring

Source: https://context7.com/raivnlab/sugar-crepe/llms.txt

Uses the Vera model to score the plausibility of a single statement or a batch of statements. Ensure the Vera model is initialized with the correct model path.

```python
vera = Vera('liujch1998/vera')
print(vera.run('A cat is sitting on a mat.'))   # ~0.92 (plausible)
print(vera.run('A cat is flying over the sun.')) # ~0.21 (implausible)

batch_scores = vera.runs([
    'A cat and dog napping together on the couch.',
    'A cat and a dog napping together under a blanket on the couch.'
])
print(batch_scores)  # array([0.89, 0.86])
```

--------------------------------

### text_retrieval Function for Text-Only Comparison

Source: https://context7.com/raivnlab/sugar-crepe/llms.txt

Compares a positive and negative caption using a provided model with a `.run()` method. Returns 1 if the positive caption scores higher, otherwise 0. This function is used in `text_model_eval.py`.

```python
import torch

@torch.no_grad()
def text_retrieval(pos_text, neg_text, model):
    pos_score = model.run(pos_text)
    neg_score = model.run(neg_text)
    return 1 if pos_score > neg_score else 0

# Use with Vera
vera = Vera('liujch1998/vera')
result = text_retrieval(
    pos_text='A plate is filled with broccoli and noodles.',
    neg_text='A plate is filled with broccoli, noodles, and carrots.',
    model=vera
)
print(result)  # 1 or 0
```

--------------------------------

### Adversarial Dataset Refinement

Source: https://context7.com/raivnlab/sugar-crepe/llms.txt

Balances a dataset of scored caption pairs by equalizing counts between mirror-image score-gap buckets. This function is useful for preventing models from exploiting systematic difficulty biases.

```python
import numpy as np
from adversarial_refine import adversarial_refine

# Simulate score gaps for two reference models over N samples
# gap > 0 means model chose correctly, gap < 0 means incorrectly
np.random.seed(42)
N = 1000
model1_score_gap = 2 * (np.random.random(N) - 0.5)  # values in (-1, 1)
model2_score_gap = 2 * (np.random.random(N) - 0.5)

# Returns list of indices to KEEP after adversarial balancing
keep_indices = adversarial_refine(model1_score_gap, model2_score_gap)

print(f"Original samples:  {N}")
print(f"Retained samples:  {len(keep_indices)}")
# Original samples:  1000
# Retained samples:  ~480  (balanced subset)

# Apply to actual dataset
import json
with open('./data/add_obj.json') as f:
    data = json.load(f)

keys = list(data.keys())
# Keep only the adversarially-refined subset
refined = {keys[i]: data[keys[i]] for i in keep_indices if i < len(keys)}
print(f"Refined add_obj split: {len(refined)} samples")
```

--------------------------------

### CLIP Text Retrieval: Score Caption Pair Against Image

Source: https://context7.com/raivnlab/sugar-crepe/llms.txt

Encodes a positive caption, a negative caption, and a PIL image using a CLIP model. Returns 1 if the positive caption has a higher cosine similarity to the image embedding than the negative caption, otherwise 0. Requires the model, tokenizer, transform, and device to be pre-initialized.

```python
import torch
from PIL import Image

@torch.no_grad()
def text_retrieval(pos_text, neg_text, image, model, tokenizer, transform, device):
    pos_text_tok = tokenizer(pos_text).to(device)
    pos_embedding = model.encode_text(pos_text_tok, normalize=True)

    neg_text_tok = tokenizer(neg_text).to(device)
    neg_embedding = model.encode_text(neg_text_tok, normalize=True)

    image_embedding = model.encode_image(
        transform(image).unsqueeze(dim=0).to(device), normalize=True
    )
    pos_score = (pos_embedding @ image_embedding.t()).item()
    neg_score = (neg_embedding @ image_embedding.t()).item()
    return 1 if pos_score > neg_score else 0

# Example
image = Image.open('./data/coco/images/val2017/000000219578.jpg')
result = text_retrieval(
    pos_text='A cat and dog napping together on the couch.',
    neg_text='A cat and a dog napping together under a blanket on the couch.',
    image=image,
    model=model,
    tokenizer=tokenizer,
    transform=transform,
    device=device
)
print(result)  # 1 (correct) or 0 (incorrect)
```

--------------------------------

### BibTeX Citation for SugarCrepe Paper

Source: https://github.com/raivnlab/sugar-crepe/blob/main/gpt-4v-results/readme.md

This is the BibTeX entry for citing the SugarCrepe paper. It includes title, authors, and publication venue details.

```bibtex
@inproceedings{hsieh2023sugarcrepe,
  title={SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality},
  author={Hsieh, Cheng-Yu and Zhang, Jieyu and Ma, Zixian and Kembhavi, Aniruddha and Krishna, Ranjay},
  booktitle={Thirty-Seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
  year={2023}
}
```

--------------------------------

### text_retrieval

Source: https://context7.com/raivnlab/sugar-crepe/llms.txt

Scores a caption pair against an image using a CLIP model. Returns 1 if the positive caption has a higher cosine similarity to the image embedding than the negative caption, otherwise 0.

```APIDOC
## text_retrieval (CLIP) — Score a Caption Pair Against an Image

Given a positive caption, a negative caption, and a PIL image, encodes all three with the CLIP model and returns 1 if the positive caption has a higher cosine similarity to the image embedding than the negative caption, otherwise 0.

```python
import torch
from PIL import Image

@torch.no_grad()
def text_retrieval(pos_text, neg_text, image, model, tokenizer, transform, device):
    pos_text_tok = tokenizer(pos_text).to(device)
    pos_embedding = model.encode_text(pos_text_tok, normalize=True)

    neg_text_tok = tokenizer(neg_text).to(device)
    neg_embedding = model.encode_text(neg_text_tok, normalize=True)

    image_embedding = model.encode_image(
        transform(image).unsqueeze(dim=0).to(device), normalize=True
    )
    pos_score = (pos_embedding @ image_embedding.t()).item()
    neg_score = (neg_embedding @ image_embedding.t()).item()
    return 1 if pos_score > neg_score else 0

# Example
image = Image.open('./data/coco/images/val2017/000000219578.jpg')
result = text_retrieval(
    pos_text='A cat and dog napping together on the couch.',
    neg_text='A cat and a dog napping together under a blanket on the couch.',
    image=image,
    model=model,
    tokenizer=tokenizer,
    transform=transform,
    device=device
)
print(result)  # 1 (correct) or 0 (incorrect)
```
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.