### Build Examples

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Builds a large set of examples. Use `build_examples(28000)` to generate 28000 examples.

```python
build_examples(28000)
```

--------------------------------

### Launch Gradio Interface

Source: https://context7.com/fpgaminer/joycaption/llms.txt

Commands to install dependencies and start the interactive Gradio application.

```bash
# Install dependencies
pip install -r gradio-app/requirements.txt

# Optional: Install Liger Kernel for faster inference
pip install liger-kernel

# Optional: Install bitsandbytes for quantization
pip install bitsandbytes

# Run the app
python gradio-app/app.py
# Opens at http://127.0.0.1:7860
```

--------------------------------

### Prepare Examples for Manual Evaluation

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Filters and prepares examples from the database for manual evaluation. It selects examples that are not yet done and not in progress. Requires `list_db_preferences`, `list_db_rankings`, and `prep_evaluation`.

```python
examples = list_db_preferences() + list_db_rankings()
examples = [e for e in examples if not e.is_done]
examples = [e for e in examples if e.in_progress is None]
print(f"Found {len(examples)} examples to run")
```

--------------------------------

### Build Ranked Examples

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Builds ranked examples with a specified count. Use `build_ranked_examples(1000)` to generate 1000 ranked examples.

```python
build_ranked_examples(1000)
```

--------------------------------

### Insert examples into database

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Uses a database connection context manager to batch-process and commit example entries.

```python
with db_conn() as (conn, cur):
	examples = [example.add_to_database(cur) for example in tqdm(examples)]
	conn.commit()
```

--------------------------------

### Add New Examples to Database

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Commits the newly generated examples to the database using a provided database connection. Each example is added individually, followed by a commit.

```python
with db_conn() as (conn, cur):
	for example in new_examples:
		example.add_to_database(cur)
		conn.commit()
```

--------------------------------

### Install Dependencies

Source: https://github.com/fpgaminer/joycaption/blob/main/gradio-app/README.md

Install all required Python packages listed in the requirements.txt file. Ensure your virtual environment is activated.

```bash
pip install -r requirements.txt
```

--------------------------------

### Run and Evaluate Example with OpenAI Model

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Evaluates a prepared example using an OpenAI model with specified parameters. Includes options for `use_openrouter`, `temperature`, and `flex`. Requires `run_openai_model`.

```python
#evaluated_example = run_openai_model(example, model="gpt-4.1", reasoning_effort=None, temperature=0.5, use_openrouter=False)
#winner, reasoning = run_openrouter_model(system, user, image_dataurl, model="anthropic/claude-3.7-sonnet:beta", reasoning_effort=None)
#evaluated_example = run_openai_model(example, model="google/gemini-2.5-pro-preview-03-25", reasoning_effort=None, temperature=0.4, use_openrouter=True)
#winner, reasoning = run_openai_model(system, user, image_dataurl, temperature=None, model='o3', reasoning_effort='medium')
evaluated_example = run_openai_model(example, model="o4-mini", reasoning_effort="medium", temperature=None, use_openrouter=False, flex=False)
#evaluated_example = run_openai_model(example, model="o3", reasoning_effort="medium", temperature=None, use_openrouter=False, flex=True)
#evaluated_example = run_openai_model(example, model="qwen/qwen2.5-vl-32b-instruct:free", reasoning_effort=None, temperature=0.6, use_openrouter=True)

print(f"Reasoning: {evaluated_example.judge_reasoning}")
if isinstance(evaluated_example, EvalSample):
	print(f"Response A Score: {evaluated_example.judge_response_a_score}")
	print(f"Response B Score: {evaluated_example.judge_response_b_score}")
	print(f"Winner: {evaluated_example.judge_winner}")
else:
	print(f"Rankings: {evaluated_example.judge_rankings}")
```

--------------------------------

### Display and Prepare Single Example for Evaluation

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Selects a random example, decodes and resizes its image data, and displays it for manual review. Requires `random`, `Image`, `io`, `base64`, and `display`.

```python
example = random.choice(examples)
example = prep_evaluation(example)
image = Image.open(io.BytesIO(base64.b64decode(example.image_dataurl.split(",")[1])))
scale = 512 / max(image.size)
image = image.resize((int(image.size[0] * scale), int(image.size[1] * scale)), Image.LANCZOS)
print(f"Filehash: {example.filehash.hex()}")
display(image)
print(f"User: {example.judge_user}")
```

--------------------------------

### Build Hammered Examples (v3)

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Builds hammered examples using the `build_hammered_examples3` function. This is part of extracting head-to-head preferences.

```python
_ = build_hammered_examples3()
```

--------------------------------

### Build Hammered Examples (v1)

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Builds hammered examples using the `build_hammered_examples` function. This is part of extracting head-to-head preferences.

```python
_ = build_hammered_examples()
```

--------------------------------

### Install bitsandbytes for Quantization

Source: https://github.com/fpgaminer/joycaption/blob/main/gradio-app/README.md

Install the bitsandbytes library to enable 4-bit and 8-bit model quantization for reduced VRAM usage.

```bash
pip install bitsandbytes
```

--------------------------------

### Build Ranked Examples from Existing Data

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Generates new ranked examples for the database based on existing preferences. It avoids duplicates by checking against existing rankings and stops when a maximum number of examples is reached or all preferences are processed.

```python
def build_ranked_examples(max_n: int | None):
	"""
	Build MultiRanked examples based on the existing examples
	"""
	existing_preferences = list_db_preferences()
	existing_rankings = set()
	for example in list_db_rankings():
		existing_rankings.add((example.filehash, example.system, example.question))
	
	random.shuffle(existing_preferences)
	
	new_examples = []
	for example in tqdm(existing_preferences, desc="Building ranked examples"):
		if (example.filehash, example.system, example.question) in existing_rankings:
			continue
		
		# Generate a new example
		new_examples.append(EvalSampleMulti(
			filehash=example.filehash,
			system=example.system,
			question=example.question,
			question_type=question_type,
			ground_truth_knowledge=build_ground_truth_knowledge(example.filehash, example.question),
		))
		existing_rankings.add((example.filehash, example.system, example.question))

		if max_n is not None and len(new_examples) >= max_n:
			break
	
	print(f"Generated {len(new_examples)} rankings to add to the database")

	# Now insert all the new examples into the database
	with db_conn() as (conn, cur):
		for example in tqdm(new_examples):
			example.add_to_database(cur)
		conn.commit()
```

--------------------------------

### Build Hammered Examples (v2)

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Builds hammered examples using the `build_hammered_examples2` function. This is part of extracting head-to-head preferences.

```python
_ = build_hammered_examples2()
```

--------------------------------

### Select Best Example Per Filehash

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Filters the processed examples to select the best one per filehash. It prioritizes examples with the highest winner score, then considers those with repper losers, and finally selects the one with the largest score margin.

```python
# Now filter down to the best example per filehash
all_examples: list[DatasetEntry] = []

for examples in tqdm(examples_by_filehash.values(), desc="Filtering examples"):
	# Sort by winner score, descending
	examples.sort(key=lambda ex: ex.response_w_score, reverse=True)

	# Keep only the examples with the highest score
	best_score = examples[0].response_w_score
	examples = [ex for ex in examples if ex.response_w_score >= best_score]

	# If we have repper losers, keep only those examples
	if any(is_repper(ex.response_l) for ex in examples):
		examples = [ex for ex in examples if is_repper(ex.response_l)]
	
	# Now pick the example with the largest margin
	examples.sort(key=lambda ex: abs(ex.response_w_score - ex.response_l_score), reverse=True)

	all_examples.append(examples[0])
```

--------------------------------

### Add Unique Examples to Database

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Processes a list of database preferences to identify and add new, unique examples to the database. Requires `tqdm`, `list_db_preferences`, `db_conn`, and `build_ground_truth_knowledge`.

```python
unique_examples = set()
existing_examples = set()

for example in tqdm(list_db_preferences()):
	unique_examples.add((example.filehash, example.system, example.question, example.question_type))
	#if example.response_a_model is None or (example.response_a_model == CURRENT_RESPONSE_MODEL and example.response_b_model == CURRENT_RESPONSE_MODEL):
	#	existing_examples.add((example.filehash, example.system, example.question, example.question_type))

with db_conn() as (conn, cur):
	n_added = 0
	for filehash, system, question, question_type in tqdm(list(unique_examples - existing_examples)):

		new_example = EvalSampleMulti(
			filehash=filehash,
			system=system,
			question=question,
			question_type=question_type,
			ground_truth_knowledge=build_ground_truth_knowledge(filehash, question)
		)
		new_example.add_to_database(cur)
		n_added += 1
	
	conn.commit()

print(f"Added {n_added} new examples to database.")
```

--------------------------------

### Update Example in Database

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Updates the status of a prepared example in the database after manual evaluation. Requires `db_conn`.

```python
print(example.id)
with db_conn() as (conn, cur):
	example.update_in_database(cur)
	conn.commit()
```

--------------------------------

### Save Evaluated Example Preference

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Saves the preference of an evaluated example back to the database. Requires `db_conn`.

```python
# Save the preference
with db_conn() as (conn, cur):
	evaluated_example.update_in_database(cur)
	conn.commit()
```

--------------------------------

### Generate Evaluation Examples

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Generates evaluation examples based on remaining VQA data and available API images. It shuffles and selects examples based on target counts for different question types and inserts them into the database.

```python
def build_examples(n: int):
	# List all existing preferences
	existing_types = defaultdict(int)
	existing_filehashes = set()
	for example in list_db_preferences():
		existing_types[example.question_type] += 1
		existing_filehashes.add(example.filehash)

	# Figure out what VQA examples we have left and what API images we can use
	remaining_vqa = [e for e in vqa_examples if e.filehash not in existing_filehashes]
	remaining_api_images = list(set(API_IMAGES_TO_USE) - existing_filehashes)

	print(f"# remaining VQA examples: {len(remaining_vqa)}")
	print(f"# remaining API images: {len(remaining_api_images)}")

	random.shuffle(remaining_vqa)
	random.shuffle(remaining_api_images)

	print({k: (int(v * n), existing_types[k]) for k, v in question_types.items()})

	target_counts = {k: max(0, int(v * n) - existing_types[k]) for k, v in question_types.items()}
	print(target_counts)
	examples = []

	for question_type, target_count in target_counts.items():
		for _ in range(target_count):
			if question_type == 'questionAnswer':
				if len(remaining_vqa) == 0:
					continue

				examples.append(remaining_vqa.pop())
				continue

			if len(remaining_api_images) == 0:
				continue

			filehash = remaining_api_images.pop()
			system = random.choice(system_prompts).strip()
			question = get_random_prompt(question_type, filehash)
			examples.append(EvalSample(
				filehash=filehash,
				system=system,
				question=question,
				question_type=question_type,
				task_type="overall",
				ground_truth_knowledge=build_ground_truth_knowledge(filehash, question),
			))
	
	print(f"Generated {len(examples)} examples to add to the database")

	# Now insert all the new examples into the database
	with db_conn() as (conn, cur):
		examples = [example.add_to_database(cur) for example in tqdm(examples)]
		conn.commit()
```

--------------------------------

### Inference Example with Finetuned Model

Source: https://github.com/fpgaminer/joycaption/blob/main/finetuning/README.md

This Python script demonstrates how to load and use a finetuned JoyCaption model for inference. It's crucial to use the same base model and prompts during inference as were used during training to maintain performance.

```python
import torch
from PIL import Image
from transformers import AutoProcessor, LlavaForConditionalGeneration
from peft import PeftModel


IMAGE_PATH = "image.jpg"
PROMPT = "Please write a question or prompt for this image. The questions or prompts you write are just like what a user might write. The prompt/question should usually be related to the image, but may occasionally not, so as not to bias things. The prompts/questions you write cover the entire range of things users might write, including the entire range of ways users might write, english level, typos, grammar mistakes, etc."
MODEL_NAME = "fancyfeast/llama-joycaption-beta-one-hf-llava"
LORA_PATH = "../joy-caption-finetune/checkpoints/cuu2y0sx/samples_1984/model"


# Load JoyCaption
# bfloat16 is the native dtype of the LLM used in JoyCaption (Llama 3.1)

```

--------------------------------

### Use a JSON file for prompts

Source: https://github.com/fpgaminer/joycaption/blob/main/scripts/README.md

This example demonstrates using a file list for images and a JSON file for prompts. This is useful for managing multiple prompts efficiently.

```python
python batch-caption.py --filelist "image_paths.txt" --prompt-file "prompts.json"
```

--------------------------------

### Run JoyCaption Application

Source: https://github.com/fpgaminer/joycaption/blob/main/gradio-app/README.md

Start the Gradio application by running the main Python script. Access the GUI via the provided URL.

```bash
python app.py
```

--------------------------------

### Deploy vLLM Inference Server

Source: https://context7.com/fpgaminer/joycaption/llms.txt

Start an OpenAI-compatible API server for JoyCaption using vLLM to handle high-throughput requests.

```bash
vllm serve fancyfeast/llama-joycaption-beta-one-hf-llava \
    --max-model-len 4096 \
    --enable-prefix-caching
```

```bash
docker run --gpus all --ipc=host -p 8000:8000 \
    -v "%USERPROFILE%\.cache\huggingface:/root/.cache/huggingface" \
    vllm/vllm-openai:latest \
    --model fancyfeast/llama-joycaption-beta-one-hf-llava \
    --max-model-len 4096 \
    --enable-prefix-caching
```

--------------------------------

### Filter and Count Examples

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Retrieves all database preferences and filters them based on specific criteria such as having a judge winner, scores for both responses, and matching response models. Prints the count of filtered examples.

```python
all_examples = list_db_preferences()
all_examples = [e for e in all_examples if e.judge_winner is not None and e.judge_response_a_score is not None and e.judge_response_b_score is not None]
all_examples = [e for e in all_examples if e.response_a_model == CURRENT_RESPONSE_MODEL and e.response_b_model == CURRENT_RESPONSE_MODEL]
#all_examples = [e for e in all_examples if e.response_a_model == "8gjfxjdm" and e.response_b_model == "8gjfxjdm"]
print(f"{len(all_examples)} examples with scores")
```

--------------------------------

### Process and Filter Database Examples

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Iterates through database preferences, filters them based on response models, presence of responses and scores, score differences, and winner criteria. It then structures the valid examples into DatasetEntry objects, grouped by filehash.

```python
for example in tqdm(list_db_preferences(), desc="Loading examples from database"):
	# Must be from the current response model
	if example.response_a_model != CURRENT_RESPONSE_MODEL or example.response_b_model != CURRENT_RESPONSE_MODEL:
		continue

	# Must have responses
	if example.response_a is None or example.response_b is None or example.judge_winner is None:
		continue

	# Must have scores
	if example.judge_response_a_score is None or example.judge_response_b_score is None:
		continue

	# Gap between scores must be significant
	if abs(example.judge_response_a_score - example.judge_response_b_score) < MIN_SCORE_DIFF:
		continue

	# Winner must have a reasonable score and cannot be a repper
	if example.judge_winner == 'a':
		if example.judge_response_a_score < MIN_WINNER_SCORE or is_repper(example.response_a):
			continue
	elif example.judge_winner == 'b':
		if example.judge_response_b_score < MIN_WINNER_SCORE or is_repper(example.response_b):
			continue
	
	examples_by_filehash[example.filehash].append(DatasetEntry(
		filehash=example.filehash,
		system=example.system,
		question=example.question,
		question_type=example.question_type,
		response_w=example.response_a if example.judge_winner == 'a' else example.response_b,
		response_l=example.response_b if example.judge_winner == 'a' else example.response_a,
		response_w_score=example.judge_response_a_score if example.judge_winner == 'a' else example.judge_response_b_score,
		response_l_score=example.judge_response_b_score if example.judge_winner == 'a' else example.judge_response_a_score,
	))

```

--------------------------------

### Build and Split Dataset

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Constructs a Hugging Face Dataset from the processed examples and then splits it into training and testing sets.

```python
# Build dataset
dataset = datasets.Dataset.from_list([example.to_dataset() for example in all_examples], features=dataset_features)

# Split
dataset = dataset.train_test_split(test_size=TEST_SIZE, shuffle=True, seed=69)

print("Pushing to hub")
dataset.push_to_hub(DATASET_PATH, private=True)
```

--------------------------------

### Add Evaluation Samples to Database

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

This snippet demonstrates how to commit the collected evaluation samples to a database. It uses a context manager for database connection and iterates through all collected examples, adding each to the database before committing the transaction. Ensure `db_conn` is properly defined and `tqdm` is imported for progress visualization.

```python
with db_conn() as (conn, cur):
	for example in tqdm(all_examples):
		example.add_to_database(cur)
	conn.commit()

```

--------------------------------

### Build Hammered Preference Examples

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Helper function and wrappers to generate preference pairs from ranked model responses, ensuring no duplicate comparisons are added to the database.

```python
def _build_hammered_examples_helper(ranking_to_response: Callable[[list[str]], tuple[str, str] | None]):
	# List all existing preferences and rankings
	existing_preferences = defaultdict(list)
	for example in list_db_preferences():
		existing_preferences[(example.filehash, example.system, example.question, example.response_a, example.response_b)].append(example)

	# Go through the rankings and find ones that haven't been head-to-head compared
	new_examples: list[EvalSample] = []
	for ranking_example in tqdm(list_db_rankings(), desc="Building hammered examples"):
		rankings = ranking_example.judge_rankings
		if rankings is None:
			continue

		if ranking_example.responses_model != CURRENT_RESPONSE_MODEL:
			continue
		
		assert len(rankings) >= 2, f"Rankings are not valid for {ranking_example.id}: {rankings}"
		assert ranking_example.responses is not None and len(ranking_example.responses) == len(rankings), f"Responses are not valid for {ranking_example.id}"

		ranked_responses = [ranking_example.responses[i] for i in rankings]
		responses = ranking_to_response(ranked_responses)
		if responses is None:
			continue

		assert len(responses) == 2, f"Expected 2 responses, got {len(responses)}: {responses}"

		# Check if we've already compared these two responses
		existing = existing_preferences.get((ranking_example.filehash, ranking_example.system, ranking_example.question, responses[0], responses[1]), [])
		existing = existing + existing_preferences.get((ranking_example.filehash, ranking_example.system, ranking_example.question, responses[1], responses[0]), [])
		existing = [e for e in existing if e.response_a_model == ranking_example.responses_model and e.response_b_model == ranking_example.responses_model]
		if len(existing) > 0:
			# We already have a preference for these two responses, so skip
			continue

		# Randomize the order of the responses
		responses = list(responses)
		random.shuffle(responses)

		# Create a new example
		new_examples.append(EvalSample(
			filehash=ranking_example.filehash,
			system=ranking_example.system,
			question=ranking_example.question,
			question_type=ranking_example.question_type,
			task_type="overall",
			response_a=responses[0],
			response_b=responses[1],
			response_a_model=ranking_example.responses_model,
			response_b_model=ranking_example.responses_model,
			ground_truth_knowledge=build_ground_truth_knowledge(ranking_example.filehash, ranking_example.question),
		))
		existing_preferences[(ranking_example.filehash, ranking_example.system, ranking_example.question, responses[0], responses[1])].append(new_examples[-1])
	
	# Now insert all the new examples into the database
	with db_conn() as (conn, cur):
		new_examples = [example.add_to_database(cur) for example in tqdm(new_examples)]
		conn.commit()
	
	print(f"Added {len(new_examples)} new examples to the database")
```

```python
def build_hammered_examples():
	"""
	Build preference pairs from the MultiRanked dataset
	Selects the best and worst responses from the rankings
	"""
	_build_hammered_examples_helper(lambda ranked_responses: (ranked_responses[0], ranked_responses[-1]))
```

```python
def build_hammered_examples2():
	"""
	Build preference pairs from the MultiRanked dataset
```

--------------------------------

### Define Training Data Structure

Source: https://github.com/fpgaminer/joycaption/blob/main/finetuning/README.md

The training data must be a JSON array where each object contains a 'messages' list and an 'images' list. Each example requires exactly one image and one assistant response for training.

```json
[
  {
    "messages": [
      {
        "role": "user",
        "content": "Respond in JSON, describing the image. Break the image down into different properties like camera angle and framing, subjects, characteristics of each subject, objects, characteristics of each object, background, etc."
      },
      {
        "role": "assistant",
        "content": "```json\n{\n  \"cameraAngle\": \"Straight on\",\n  \"framing\": \"Cowboy shot, capturing the subject from the middle thighs up\",\n  \"subjects\": [\n    {\n      \"type\": \"Man\",\n      \"appearance\": {\n        \"build\": \"Muscular\",\n        \"hair\": \"Short, light brown, styled back\",\n        \"facialHair\": \"Full beard, light brown\",\n        \"chest\": \"Hairy\",\n        \"attire\": {\n          \"top\": \"Shirtless\",\n          \"bottom\": \"Dark blue jeans, unbuttoned and pulled down to reveal black underwear\",\n          \"accessories\": \"Silver chain necklace with pendant\"\n        },\n\t\t\"expression\": \"Smiling, looking at the camera\"\n      }\n    }\n  ],\n  \"objects\": [\n    {\n      \"type\": \"Couch\",\n      \"characteristics\": {\n        \"color\": \"Dark gray\",\n        \"material\": \"Fabric\",\n\t\t\"style\": \"Modern\"\n      }\n    },\n    {\n      \"type\": \"Pillow\",\n      \"characteristics\": {\n        \"shape\": \"Rectangular\",\n        \"color\": \"Cream with gray stripes\",\n        \"material\": \"Fabric with a fuzzy trim\"\n      }\n    },\n    {\n      \"type\": \"Curtains\",\n      \"characteristics\": {\n        \"color\": \"Gray and white\",\n        \"material\": \"Fabric\",\n\t\t\"style\": \"Floor-length\"\n      }\n    },\n\t{\n\t  \"type\": \"Ottoman\",\n\t  \"characteristics\": {\n\t\t\"shape\": \"Rectangular\",\n\t\t\"color\": \"Beige\",\n\t\t\"material\": \"Fabric\",\n\t\t\"style\": \"Tufted\"\n\t  }\n\t}\n  ],\n  \"background\": {\n    \"type\": \"Wall\",\n    \"color\": \"Dark blue\",\n\t\"style\": \"Paneled\"\n  }\n}\n```"
      }
    ],
    "images": [
      "training-images/14780432.jpg"
    ]
  },
  {
    "messages": [
      {
        "role": "user",
        "content": "What's going on in the top left?"
      },
      {
        "role": "assistant",
        "content": "The top left of the image is an inset panel showing an \"X-Ray\" view.  This X-Ray view is typical of pornographic artwork.  In this case it shows the male's penis ejaculating inside of the female character's vaginal canal."
      }
    ],
    "images": [
      "training-images/581402.png"
    ]
  }
]
```

--------------------------------

### Parallel Sample Preparation

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Prepares evaluation samples in parallel using a ThreadPoolExecutor. It iterates through all examples, submits them for preparation, and updates their status in the database upon successful completion. Error handling is included for individual sample processing.

```python
def handle_prep_job(example: EvalSample | EvalSampleMulti) -> EvalSample | EvalSampleMulti:
	return prep_evaluation(example)


with ThreadPoolExecutor(max_workers=8) as executor, tqdm(total=len(all_examples)) as pbar, db_conn() as (conn, cur):
	futures = [executor.submit(handle_prep_job, example) for example in all_examples]
	for future in as_completed(futures):
		pbar.update(1)
		try:
			result = future.result()
		except Exception as e:
			print(f"Error: {e}")
			continue

		assert isinstance(result, EvalSample) or isinstance(result, EvalSampleMulti), f"Expected EvalSample or EvalSampleMulti, got {type(result)}"
 
		result.update_in_database(cur)
		conn.commit()
```

--------------------------------

### Comment placeholder

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

A comment indicating the start of a VQA example list.

```python
# Build our list of possible VQA examples
```

--------------------------------

### Set Up Python Virtual Environment

Source: https://github.com/fpgaminer/joycaption/blob/main/gradio-app/README.md

Create and activate a Python virtual environment to manage project dependencies. This is optional but recommended.

```bash
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
```

--------------------------------

### Process and Filter Examples

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Iterates through all examples, populating data structures with response pairs, scores, and validity flags. It filters out examples with abnormal responses, close score differences, or low winning scores.

```python
by_pref_key = defaultdict(list)
by_pref_key_is_valid = set()
existing_pairs = set()
existing_responses = defaultdict(set)

for example in tqdm(all_examples):
	assert example.response_a is not None and example.response_b is not None, "Missing response_a or response_b"
	assert example.judge_response_a_score is not None and example.judge_response_b_score is not None, "Missing judge response scores"

	existing_pairs.add((example.filehash, example.system, example.question, example.response_a, example.response_b))

	pref_key = (example.filehash, example.system, example.question, example.question_type)

	by_pref_key[pref_key].append((example.response_a, example.judge_response_a_score))
	by_pref_key[pref_key].append((example.response_b, example.judge_response_b_score))
	existing_responses[pref_key].add(example.response_a)
	existing_responses[pref_key].add(example.response_b)

abnormal_response_a = is_repper(example.response_a)
abnormal_response_b = is_repper(example.response_b)

if example.judge_winner == "a" and abnormal_response_a:
		continue
elif example.judge_winner == "b" and abnormal_response_b:
		continue

	if abs(example.judge_response_a_score - example.judge_response_b_score) < 0.2:
		continue

	if example.judge_winner == "a" and example.judge_response_a_score < 0.5:
		continue
elif example.judge_winner == "b" and example.judge_response_b_score < 0.5:
		continue

	by_pref_key_is_valid.add(pref_key)


needs_reroll = []
new_examples = []

for pref_key, responses in tqdm(by_pref_key.items()):
	if pref_key in by_pref_key_is_valid:
		continue

	sorted_responses = sorted(responses, key=lambda x: x[1], reverse=True)
	#assert sorted_responses[0][1] > sorted_responses[-1][1]

	best_responses = [(r, s) for r, s in sorted_responses if not is_repper(r) and s > 0.5]
	if len(best_responses) == 0:
		needs_reroll.append(pref_key)
		continue

	best_response, best_score = best_responses[0]
	worst_response, worst_score = sorted_responses[-1]

	if abs(best_score - worst_score) < 0.2 or best_score < worst_score:
		needs_reroll.append(pref_key)
		continue

	# Looks like we have a valid pair to try
	if (pref_key[0], pref_key[1], pref_key[2], best_response, worst_response) in existing_pairs or (pref_key[0], pref_key[1], pref_key[2], worst_response, best_response) in existing_pairs:
		# Already tried it, so spin again
		needs_reroll.append(pref_key)
		continue

	response_pair = [best_response, worst_response]
	random.shuffle(response_pair)

	new_examples.append(EvalSample(
		filehash=pref_key[0],
		system=pref_key[1],
		question=pref_key[2],
		question_type=pref_key[3],
		task_type='overall',
		response_a=response_pair[0],
		response_b=response_pair[1],
		ground_truth_knowledge=build_ground_truth_knowledge(pref_key[0], pref_key[2]),
	))

print(f"Found {len(new_examples)} new examples to try")
print(f"Found {len(needs_reroll)} examples to reroll")
print(f"Found {len(by_pref_key_is_valid)} valid examples")
print(f"Out of {len(by_pref_key)} keys")
```

--------------------------------

### Initialize and Run Inference with JoyCaption

Source: https://github.com/fpgaminer/joycaption/blob/main/finetuning/README.md

Loads a model with LoRA adapters and performs image captioning using the Hugging Face processor. Ensure the input conversation template is carefully formatted to avoid tokenization issues.

```python
# device_map=0 loads the model into the first GPU
processor = AutoProcessor.from_pretrained(MODEL_NAME)
llava_model = LlavaForConditionalGeneration.from_pretrained(MODEL_NAME, torch_dtype="bfloat16", device_map=0)
llava_model = PeftModel.from_pretrained(llava_model, LORA_PATH)
llava_model.eval()

with torch.no_grad():
	# Load image
	image = Image.open(IMAGE_PATH)

	# Build the conversation
	convo = [
		{
			"role": "system",
			"content": "You are a helpful image captioner.",
		},
		{
			"role": "user",
			"content": PROMPT,
		},
	]

	# Format the conversation
	# WARNING: HF's handling of chat's on Llava models is very fragile.  This specific combination of processor.apply_chat_template(), and processor() works
	# but if using other combinations always inspect the final input_ids to ensure they are correct.  Often times you will end up with multiple <bos> tokens
	# if not careful, which can make the model perform poorly.
	convo_string = processor.apply_chat_template(convo, tokenize = False, add_generation_prompt = True)
	assert isinstance(convo_string, str)

	# Process the inputs
	inputs = processor(text=[convo_string], images=[image], return_tensors="pt").to('cuda')
	inputs['pixel_values'] = inputs['pixel_values'].to(torch.bfloat16)

	# Generate the captions
	generate_ids = llava_model.generate(
		**inputs,
		max_new_tokens=300,
		do_sample=True,
		suppress_tokens=None,
		use_cache=True,
		temperature=0.6,
		top_k=None,
		top_p=0.9,
	)[0]

	# Trim off the prompt
	generate_ids = generate_ids[inputs['input_ids'].shape[1]:]

	# Decode the caption
	caption = processor.tokenizer.decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)
	caption = caption.strip()
	print(caption)
```

--------------------------------

### Filter Pending Evaluation Examples

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Retrieves and filters database records to identify examples that are not yet completed or in progress.

```python
all_examples = list_db_preferences() + list_db_rankings()
#all_examples = list_db_rankings()
#all_examples = list_db_preferences()
all_examples = [e for e in all_examples if not e.is_done]
all_examples = [e for e in all_examples if e.in_progress is None]
print(f"Loaded {len(all_examples)} examples to process")
```

--------------------------------

### Serve JoyCaption with vLLM

Source: https://github.com/fpgaminer/joycaption/blob/main/README.md

Command to launch the JoyCaption model using the vLLM inference engine.

```bash
vllm serve fancyfeast/llama-joycaption-beta-one-hf-llava --max-model-len 4096 --enable-prefix-caching
```

--------------------------------

### Initialize JoyCaption with Transformers

Source: https://github.com/fpgaminer/joycaption/blob/main/README.md

Load the JoyCaption model and prepare it for inference using the Hugging Face transformers library.

```python
import torch
from PIL import Image
from transformers import AutoProcessor, LlavaForConditionalGeneration


IMAGE_PATH = "image.jpg"
PROMPT = "Write a long descriptive caption for this image in a formal tone."
MODEL_NAME = "fancyfeast/llama-joycaption-beta-one-hf-llava"


# Load JoyCaption
# bfloat16 is the native dtype of the LLM used in JoyCaption (Llama 3.1)
```

--------------------------------

### Analyze and Categorize Examples

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Iterates through a list of examples, categorizing them based on various criteria including response abnormalities, score differences, and winner scores. It calculates and prints detailed statistics for each question type, including valid examples, rejection reasons, and score distributions.

```python
by_question_type = defaultdict(list)
valids = defaultdict(list)
rejects = defaultdict(list)
invalids = defaultdict(list)
valids_by_key = set()

for example in tqdm(all_examples):
	assert example.response_a is not None and example.response_b is not None, "Missing response_a or response_b"
	assert example.judge_response_a_score is not None and example.judge_response_b_score is not None, "Missing judge response scores"
	by_question_type[example.question_type].append((example.judge_response_a_score, example.judge_response_b_score))

abnormal_response_a = is_repper(example.response_a)
abnormal_response_b = is_repper(example.response_b)

	if example.judge_winner == "a" and abnormal_response_a:
		rejects[example.question_type].append("abnormal_winner")
		invalids[example.question_type].append(example)
		continue
	elif example.judge_winner == "b" and abnormal_response_b:
		rejects[example.question_type].append("abnormal_winner")
		invalids[example.question_type].append(example)
		continue

	if abs(example.judge_response_a_score - example.judge_response_b_score) < 0.2:
		rejects[example.question_type].append("close_scores")
		invalids[example.question_type].append(example)
		continue

	if example.judge_winner == "a" and example.judge_response_a_score < 0.5:
		rejects[example.question_type].append("winner_low_score")
		invalids[example.question_type].append(example)
		continue
	elif example.judge_winner == "b" and example.judge_response_b_score < 0.5:
		rejects[example.question_type].append("winner_low_score")
		invalids[example.question_type].append(example)
		continue

	valids[example.question_type].append(example.judge_response_a_score if example.judge_winner == "a" else example.judge_response_b_score)
	k = (example.filehash, example.question)
	valids_by_key.add(k)

total_valid = sum(len(v) for v in valids.values())

print("Question type statistics:")
for question_type in by_question_type.keys():
	print(f"{question_type}")
	scores = [score for scores in by_question_type[question_type] for score in scores]
	n_scores = len(scores)
	n_examples = n_scores // 2
	mean_score = sum(scores) / n_scores
	min_score = min(scores)
	max_score = max(scores)
	n_valid = len(valids[question_type])
	min_valid = min(valids[question_type])
	max_valid = max(valids[question_type])
	mean_valid = sum(valids[question_type]) / n_valid

	print(f"  {n_examples} examples ({n_valid / total_valid:.2%} vs {question_types[question_type]:.2%} target)")
	print(f"  {n_valid} valid examples ({n_valid / n_examples:.2%})")
	print(f"  Mean score: {mean_score:.4f}")
	print(f"  Min score: {min_score:.4f}")
	print(f"  Max score: {max_score:.4f}")
	print(f"  Valid scores: {mean_valid:.4f} ({min_valid:.4f} - {max_valid:.4f})")

	for reject_reason, count in Counter(rejects[question_type]).items():
		print(f"  {reject_reason}: {count} ({count / n_examples:.2%})")

print(f"Total Valid: {sum(len(v) for v in valids.values())}")
```

--------------------------------

### Execute Fine-tuning

Source: https://context7.com/fpgaminer/joycaption/llms.txt

Run the training process using torchrun with specified hyperparameters.

```bash
# Run fine-tuning with torchrun
torchrun --standalone --nproc_per_node=1 finetuning/train.py \
    --wandb-project my-joycaption-finetune \
    --dataset training_data.json \
    --images-path ./training-images \
    --device-batch-size 4 \
    --batch-size 32 \
    --max-samples 600 \
    --test-size 20 \
    --test-every 200 \
    --learning-rate 5e-5 \
    --lora-r 64 \
    --lora-alpha 64
```

--------------------------------

### Ingest Human VQA Examples

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Searches for VQA examples and adds them to the database, applying filtering logic for specific categories and system prompts.

```python
# Add all of our human VQA examples
existing_filehashes = set()
existing_filehashes.update(test_filehashes)

for example in tqdm(list_db_rankings()):
	if example.responses_model == CURRENT_RESPONSE_MODEL:
		existing_filehashes.add(example.filehash)

results = api.search("EXISTS(vqa_category) AND EXISTS(questionAnswer)", ["hash", "attributes"])
assert isinstance(results, list)

n_added = 0
with db_conn() as (conn, cur):
	for result in tqdm(results):
		assert isinstance(result, tag_machine_api.SearchResultImage) and result.hash is not None and result.attributes is not None
		if result.hash in existing_filehashes:
			continue

		sources = result.attributes.get('source', [])
		vqa_category, = result.attributes['vqa_category'].keys()
		questionAnswer, = result.attributes['questionAnswer'].keys()
		questionAnswer = json.loads(questionAnswer)
		vqa_category = vqa_category.strip()
		question = questionAnswer['question'].strip()
		answer = questionAnswer['answer'].strip()

		if vqa_category == '' or question == '' or answer == '' or 'benchmark' in vqa_category.lower():
			continue

		if 'bra_sizing' in sources and result.hash.hex()[0] != '0':
			# Filter the bra related questions, so they don't overload the dataset.
			# using the hash knocks it down by 1/16th, deterministically
			continue

		if "<system>" in question:
			i = question.index("<system>")
			j = question.index("</system>")
			system_message = question[i + len("<system>"):j]
			question = question[:i] + question[j + len("</system>"):]
		else:
			system_message = random.choice(system_prompts)
		
		question = question.strip()

		example = EvalSampleMulti(
			filehash=result.hash,
			system=system_message,
			question=question,
			question_type='questionAnswer',
			ground_truth_knowledge=build_ground_truth_knowledge(result.hash, question),
			responses_model=CURRENT_RESPONSE_MODEL,  # So we can track it
		)
		example.add_to_database(cur)
		existing_filehashes.add(result.hash)
		n_added += 1
	
	conn.commit()

print(f"Added {n_added} examples")
```

--------------------------------

### Filter and Validate Evaluation Examples

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Iterates through examples to categorize them by question type and filter out invalid responses based on score thresholds and repetition checks.

```python
by_question_type   = defaultdict(list)   # (a_score, b_score)
valids             = defaultdict(list)   # winner score only
rejects            = defaultdict(list)   # list of textual reasons
invalid_examples   = defaultdict(list)   # the examples themselves (if needed)

for ex in tqdm(all_examples, desc="Scanning examples"):
    assert ex.response_a and ex.response_b,          "Missing response text"
    assert ex.judge_response_a_score is not None and ex.judge_response_b_score is not None, "Missing scores"

    by_question_type[ex.question_type].append(
        (ex.judge_response_a_score, ex.judge_response_b_score)
    )

    abnormal_a = is_repper(ex.response_a)
    abnormal_b = is_repper(ex.response_b)

    # ---- rejection checks ----------------------------------------------------
    reason = None
    if ex.judge_winner == "a" and abnormal_a:
        reason = "abnormal_winner"
    elif ex.judge_winner == "b" and abnormal_b:
        reason = "abnormal_winner"
    elif abs(ex.judge_response_a_score - ex.judge_response_b_score) < 0.20:
        reason = "close_scores"
    elif ex.judge_winner == "a" and ex.judge_response_a_score < 0.50:
        reason = "winner_low_score"
    elif ex.judge_winner == "b" and ex.judge_response_b_score < 0.50:
        reason = "winner_low_score"

    if reason:
        rejects[ex.question_type].append(reason)
        invalid_examples[ex.question_type].append(ex)
        continue

    # ---- valid example -------------------------------------------------------
    winner_score = (
        ex.judge_response_a_score if ex.judge_winner == "a" else ex.judge_response_b_score
    )
    valids[ex.question_type].append(winner_score)
```

--------------------------------

### Serve Merged Model with vLLM

Source: https://github.com/fpgaminer/joycaption/blob/main/finetuning/README.md

Launches the vLLM server using the previously merged model directory.

```bash
vllm serve ./questions-cuu2y0sx --max-model-len 4096 --enable-prefix-caching
```

--------------------------------

### Run JoyCaption in Docker on Windows

Source: https://github.com/fpgaminer/joycaption/blob/main/README.md

Command to deploy JoyCaption using a Docker container with GPU support on Windows, persisting the Hugging Face cache.

```bash
docker run --gpus all --ipc=host -p 8000:8000 -v "%USERPROFILE%\.cache\huggingface:/root/.cache/huggingface" vllm/vllm-openai:latest --model fancyfeast/llama-joycaption-beta-one-hf-llava --max-model-len 4096 --enable-prefix-caching
```

--------------------------------

### Handle New Example Responses with Thread Pool

Source: https://github.com/fpgaminer/joycaption/blob/main/training/BuildAlignmentDataset.ipynb

Generates new responses for examples that need rerolling, ensuring that generated responses have not been previously tried. It uses a ThreadPoolExecutor for parallel processing and handles potential exceptions during response generation.

```python
def handle_new_example_responses(example: EvalSample) -> EvalSample | None:
	example = example.get_image()

	responses = []

	while True:
		response = generate_response(example)
		if response in existing_responses[(example.filehash, example.system, example.question, example.question_type)]:
			print("Already tried response, spinning again")
			continue

		responses.append(response)
		if len(responses) == 2:
			break
	
	return example.model_copy(update={"response_a": responses[0], "response_b": responses[1], "response_a_model": CURRENT_RESPONSE_MODEL, "response_b_model": CURRENT_RESPONSE_MODEL})

work = []
for pref_key in tqdm(needs_reroll):
	new_example = EvalSample(
		filehash=pref_key[0],
		system=pref_key[1],
		question=pref_key[2],
		question_type=pref_key[3],
		task_type='overall',
		ground_truth_knowledge=build_ground_truth_knowledge(pref_key[0], pref_key[2]),
	)
	work.append(new_example)

with ThreadPoolExecutor(max_workers=8) as executor, tqdm(total=len(work)) as pbar, db_conn() as (conn, cur):
	futures = [executor.submit(handle_new_example_responses, example) for example in work]
	for future in as_completed(futures):
		pbar.update(1)
		try:
			result = future.result()
		except Exception as e:
			print(f"Error: {e}")
			continue

		if result is None:
			continue

		assert isinstance(result, EvalSample), f"Expected EvalSample, got {type(result)}"
```

--------------------------------

### Training Command for JoyCaption

Source: https://github.com/fpgaminer/joycaption/blob/main/finetuning/README.md

Use this command to initiate the training process for JoyCaption. Ensure to check the train.py script for a full list of arguments and consider adjusting parameters like learning rate, batch size, and LORA settings for optimal results.

```bash
torchrun --standalone --nproc_per_node=1 train.py --wandb-project finetune-2 --device-batch-size 4 --dataset ../instruction-dataset/answers-train.json --max-samples 1800 --images-path ../instruction-dataset --test-every 2000 --test-size 128
```