Prodigy

https://prodi.gy/docs
Admin
Prodigy is a modern, downloadable annotation tool and Python library for creating training and...
Tokens:155,376
Snippets:985
Trust Score:10
Update:1 year ago
Show doc for...
Tokens:
Raw
### Prodigy CLI Command for Manual Image Annotation Quickstart

Source: https://prodi.gy/docs/docs/computer-vision

This command-line example demonstrates how to launch the `image.manual` recipe in Prodigy, specifying an image dataset and labels for annotation. It's a quick way to start drawing bounding boxes and polygons.

```CLI
prodigy image.manual images_dataset ./images --label PERSON,SKATEBOARD
```

--------------------------------

### SpaCy LLM NER v3 Configuration Example

Source: https://prodi.gy/docs/docs/large-language-models

An example `config.cfg` for a `spacy-llm` pipeline, demonstrating how to configure the `spacy.NER.v3` task with custom labels, descriptions, and a reference to a few-shot examples file for improved NER performance.

```Config
### config.cfg

[nlp]

lang = "en"

pipeline = ["llm"]

[components]

[components.llm]

factory = "llm"

[components.llm.task]

@llm_tasks = "spacy.NER.v3"

labels = ["DISH", "INGREDIENT", "EQUIPMENT"]

description = "Entities are the names food dishes, ingredients, and any kind of cooking equipment. Adjectives, verbs, adverbs are not entities. Pronouns are not entities."

[components.llm.task.label_definitions]

DISH = "Known food dishes, e.g. Lobster Ravioli, garlic bread"

INGREDIENT = "Individual parts of a food dish, including herbs and spices."

EQUIPMENT = "Any kind of cooking equipment. e.g. oven, cooking pot, grill"

[components.llm.task.examples]

@misc = "spacy.FewShotReader.v1"

path = "few-shot-examples.json"

[components.llm.model]

@llm_models = "spacy.GPT-3-5.v1"
```

--------------------------------

### Prodigy Command with Binary Examples Path

Source: https://prodi.gy/docs/docs/large-language-models

Command to run Prodigy's `textcat.openai.correct` recipe, including the `--examples-path` flag to provide a YAML file with binary classification examples to the OpenAI prompt.

```Shell
prodigytextcat.openai.correctrecipe-comments-textcatcomments.jsonl--label recipe--examples-path ./textcat-binary.yaml
```

--------------------------------

### Prodigy Command with Multi-Label Examples Path

Source: https://prodi.gy/docs/docs/large-language-models

Command to run Prodigy's `textcat.openai.correct` recipe, including the `--examples-path` flag to provide a YAML file with multi-label examples to the OpenAI prompt.

```Shell
prodigytextcat.openai.correctrecipe-comments-textcatcomments.jsonl--label recipe,feedback,question--examples-path ./textcat-multilabel.yaml
```

--------------------------------

### YAML Examples for Binary OpenAI Prompt

Source: https://prodi.gy/docs/docs/large-language-models

Configuration in YAML format providing examples for a binary text classification task, used to guide the OpenAI model's responses with specific text, 'accept'/'reject' answers, and reasons.

```YAML
- text: 'This is a recipe for scrambled egg: 2 eggs, 1 butter, batter them, and then fry in a hot pan for 2 minutes'

answer: 'accept'

reason: 'This is a recipe for making a scrambled egg'

- text: 'This is a recipe for fried rice: 1 cup of day old rice, 1 butter, 2 cloves of garlic: put them all in a wok and stir them together.'

answer: 'accept'

reason: 'This is a recipe for making a fried rice'

- text: "I tried it and it's not good"

answer: 'reject'

reason: "It doesn't talk about a recipe."
```

--------------------------------

### Prodigy CLI Example: Generating Terms with Query

Source: https://prodi.gy/docs/docs/large-language-models

This command-line example shows how to use Prodigy to generate terms in a specific language by providing a descriptive query directly, without needing a separate custom template. It also specifies the number of examples to generate.

```Shell
prodigyterms.openai.fetch"Dutch insults"dutch-insults.jsonl--n 100
```

--------------------------------

### YAML Examples for Multi-Label OpenAI Prompt

Source: https://prodi.gy/docs/docs/large-language-models

Configuration in YAML format providing examples for a multi-label text classification task, used to guide the OpenAI model's responses with specific text, answers, and reasons.

```YAML
- text: 'Can someone try this recipe?'

answer: 'question'

reason: 'It is a question about trying a recipe.'

- text:

'1 cup of rice then egg and then mix them well. Should I add garlic last?'

answer: 'question,recipe'

reason: 'It is a question about the steps in making a fried rice.'
```

--------------------------------

### OpenAI Prompt with Few-Shot Examples for NER

Source: https://prodi.gy/docs/docs/large-language-models

This snippet displays the updated prompt sent to OpenAI after adding few-shot examples from `ner-examples.yaml`. The examples are included in the prompt to guide the model and improve the accuracy of Named Entity Recognition.

```Plaintext
From the text below, extract the following entities in the following format:

dish: <comma delimited list of strings>

ingredient: <comma delimited list of strings>

equipment: <comma delimited list of strings>

For example:

Text:

"""

You can't get a great chocolate flavor with carob.

"""

dish:

ingredient: carob

equipment:

Text:

"""

You can probably sand-blast it if it's an anodized aluminum pan.

"""

dish:

ingredient:

equipment: anodized aluminum pan

Here is the text that needs predictions.

Text:

"""

Sriracha sauce goes really well with hoisin stir fry, but you should add it after you use the wok.

"""
```

--------------------------------

### Example Prodigy Configuration File

Source: https://prodi.gy/docs/docs/install

An example `prodigy.json` file demonstrating the default structure and common settings for configuring the Prodigy web application.

```JSON
{
"theme": "basic",
"custom_theme": {},
"buttons": ["accept", "reject", "ignore", "undo"],
"batch_size": 10,
"history_size": 10,
"port": 8080,
"host": "localhost",
"cors": true,
"db": "sqlite",
"db_settings": {},
"validate": true,
"auto_exclude_current": true,
"instant_submit": false,
"feed_overlap": false,
"annotations_per_task": null,
"allow_work_stealing": true,
"total_examples_target": 0,
"ui_lang": "en",
"ui_lang_locale": {},
"project_info": ["dataset", "session", "lang", "recipe_name", "view_id", "label"],
"show_stats": false,
"hide_meta": false,
"show_flag": false,
"instructions": false,
"swipe": false,
"swipe_gestures": { "left": "accept", "right": "reject" },
"split_sents_threshold": false,
"html_template": false,
"global_css": null,
"global_css_urls": null,
"global_css_dir": null,
"javascript": null,
"javascript_urls": null,
"javascript_dir": null,
"writing_dir": "ltr",
"show_whitespace": false,
"exclude_by": "task"
}
```

--------------------------------

### Running `prodigy stats` command and viewing output

Source: https://prodi.gy/docs/docs

Demonstrates the `prodigy stats` command to verify installation and view details about the Prodigy setup, including version, database, and session information.

```Shell
prodigy stats
============================== Prodigy Stats ==============================

Version 1.11.0
Database Name SQLite
Database Id sqlite
Total Datasets 0
Total Sessions 0
```

--------------------------------

### Prodigy Recipe: Fetching LLM-Generated Span Examples

Source: https://prodi.gy/docs/docs/large-language-models

Command-line example for running the `spans.llm.fetch` Prodigy recipe. This recipe is used to pre-generate and fetch examples for span categorization, similar to `spans.llm.correct` but focused on data preparation rather than live annotation.

```Shell
dotenvrun--prodigyspans.llm.fetchconfig.cfgexamples.jsonlspancat-annotated.jsonl
```

--------------------------------

### Prodigy Command Line Example with Pages

Source: https://prodi.gy/docs/docs/named-entity-recognition

An example command-line invocation for Prodigy's `manualner_wikipediablank` recipe, demonstrating how to load data from pages and specify labels for annotation.

```Shell
prodigyner.manualner_wikipediablank:en./animals.txt--loader pages--label ANIMAL,LOCATION
```

--------------------------------

### Few-Shot Examples for spacy-llm Span Categorization (YAML)

Source: https://prodi.gy/docs/docs/large-language-models

An example YAML file demonstrating the format for few-shot examples used by `spacy.FewShotReader.v1`. It includes text and corresponding entity annotations for DISH and INGREDIENT labels, showing how to define examples for the LLM.

```YAML
- text: 'Mac and Cheese is a popular American pasta variant.'
entities:
INGREDIENT: ['Cheese']
DISH: ['Mac and Cheese']
```

--------------------------------

### Advanced spaCy LLM Configuration for NER

Source: https://prodi.gy/docs/docs/large-language-models

This configuration file expands the basic `spacy-llm` setup for Named Entity Recognition. It includes label definitions for 'DISH', 'INGREDIENT', and 'EQUIPMENT', configures few-shot examples loaded from `ner_examples.yml`, and sets up a batch cache to store responses, reducing costs on reruns and improving efficiency.

```ini
[nlp]
lang = "en"
pipeline = ["llm"]

[components]
[components.llm]
factory = "llm"
save_io = true

[components.llm.task]
@llm_tasks = "spacy.NER.v2"
labels = ["DISH", "INGREDIENT", "EQUIPMENT"]

[components.llm.task.label_definitions]
DISH = "Extract the name of a known dish."
INGREDIENT = "Extract the name of a cooking ingredient, including herbs and spices."
EQUIPMENT = "Extract any mention of cooking equipment. e.g. oven, cooking pot, grill"

[components.llm.model]
@llm_models = "spacy.GPT-3-5.v1"
config = {"temperature": 0.3}

[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "ner_examples.yml"

[components.llm.cache]
@llm_misc = "spacy.BatchCache.v1"
path = "local-cached"
batch_size = 3
max_batches_in_mem = 10
```

--------------------------------

### Prodigy CLI: Generate Terminology with Seeds (terms.openai.fetch)

Source: https://prodi.gy/docs/docs/large-language-models

This command extends the `terms.openai.fetch` recipe by incorporating initial seed terms via the `--seeds` option. These examples guide OpenAI's generation process, helping to produce more relevant and focused terminology lists.

```Shell
prodigyterms.openai.fetch"skateboard tricks"skateboard-tricks.jsonl--n 100--seeds ollie,kickflip
```

--------------------------------

### OpenAI Prompt with Multi-Label Examples

Source: https://prodi.gy/docs/docs/large-language-models

The internal OpenAI prompt generated by Prodigy, augmented with user-provided multi-label examples to improve classification accuracy and consistency.

```Natural Language
Classify the text below to any of the following labels: recipe, feedback, question

The task is non-exclusive, so you can provide more than one label as long as they are comma-delimited.

For example: Label1, Label2, Label3.

Your answer should only be in the following format:

answer:

reason:

Below are some examples (only use these as a guide):

Text:

"""

Can someone try this recipe?

"""

answer: question

reason: It is a question about trying a recipe.

Text:

"""

1 cup of rice then egg and then mix them well. Should I add garlic last?

"""

answer: question,recipe

reason: It is a question about the steps in making a fried rice.

Here is the text that needs classification

Text:

"""

Cream cheese is really good in mashed potatoes.

"""
```

--------------------------------

### Example Input JSONL File

Source: https://prodi.gy/docs/docs/large-language-models

An example of a JSONL file containing text data that will be used as input for text classification tasks with Prodigy.

```JSON
{"text": "Cream cheese is really good in mashed potatoes."}
```

--------------------------------

### Example Output: Prodigy Generated Patterns

Source: https://prodi.gy/docs/docs/large-language-models

This JSONL snippet shows examples of patterns generated by the `prodigyterms.to-patterns` command. Each line represents a pattern with a specified label and a list of token components. These patterns can then be used in other Prodigy recipes like `ner.manual`.

```JSONL
{"label":"skateboard-trick","pattern":[{"lower":"pop"},{"lower":"shove"},{"lower":"it"}]}
{"label":"skateboard-trick","pattern":[{"lower":"switch"},{"lower":"flip"}]}
{"label":"skateboard-trick","pattern":[{"lower":"nose"},{"lower":"slides"}]}
{"label":"skateboard-trick","pattern":[{"lower":"lazerflip"}]}
{"label":"skateboard-trick","pattern":[{"lower":"lipslide"}]}
```

--------------------------------

### Prodigy CLI Example: Custom Term Generation Prompt

Source: https://prodi.gy/docs/docs/large-language-models

This command-line example illustrates how to run a Prodigy term generation recipe using a custom Jinja2 prompt template. It specifies the query, the number of terms to generate, and the path to the custom template file.

```Shell
prodigyterms.openai.fetch"Dutch insults"dutch-insults.jsonl--n 100--prompt-path ./custom-terms.jinja2
```

--------------------------------

### YAML Few-Shot Examples for NER

Source: https://prodi.gy/docs/docs/large-language-models

This YAML file (`ner-examples.yaml`) provides few-shot examples to improve the performance of OpenAI's NER model. Each entry contains a text and its corresponding correct entity annotations, which are then added to the prompt.

```YAML
- text: "You can't get a great savory flavor with carob."
  entities:
    dish: []
    ingredient: ['carob']
    equipment: []
- text: "You can probably sand-blast it if it's an anodized aluminum pan."
  entities:
    dish: []
    ingredient: []
    equipment: ['anodized aluminum pan']
```

--------------------------------

### Run Prodigy LLM Tournament Recipe

Source: https://prodi.gy/docs/docs/large-language-models

Command-line invocation to start the Prodigy LLM tournament recipe, specifying input data, prompt and configuration directories, a display template, and the --resume flag to continue an existing session.

```Shell
prodigyab.llm.tournamenthaiku-tournamentinputs.jsonl./prompts./configsdisplay-template.jinja2--resume
```

--------------------------------

### Prodigy CLI Example: Custom Text Classification Prompt

Source: https://prodi.gy/docs/docs/large-language-models

This command-line example demonstrates how to run a Prodigy text classification recipe using a custom Jinja2 prompt template. It specifies the input dataset, labels for classification, and the path to the custom template file.

```Shell
prodigytextcat.openai.correctrecipe-comments-textcatexamples.jsonl--label recipe,feedback,question--prompt-path ./custom-textcat.jinja2
```

--------------------------------

### Install Prodigy Python Wheel

Source: https://prodi.gy/docs/docs/install

Commands to install Prodigy from a local wheel file or a directory of wheels using pip. Includes examples for specific wheel files and using the -f option to automatically pick the best match.

```Python
python -mpip installprodigy-1.16.0-py3-none-any.whl
```

```Python
python -mpip installprodigy-f/path/to/wheels
```

```Python
python -m pipinstallprodigy-1.11.0-cp38-cp38m-macosx_10_14_x86_64.whl
```

--------------------------------

### Prodigy Command Line Example with Custom NER Prompt

Source: https://prodi.gy/docs/docs/large-language-models

This command-line example demonstrates how to use a custom Jinja2 template (`custom-ner.jinja2`) with Prodigy's `ner.openai.correct` recipe. It specifies the labels to be extracted and the path to the custom prompt file.

```Shell
prodigyner.openai.correctrecipe-nerexamples.jsonl--label dish,ingredient,equipment--prompt-path ./custom-ner.jinja2
```

--------------------------------

### Fetch OpenAI NER Annotations to Disk

Source: https://prodi.gy/docs/docs/large-language-models

Command to use the `ner.openai.fetch` recipe to download pre-annotated named entities from OpenAI to a local JSONL file, allowing for later review with `ner.manual`.

```Shell
prodigyner.openai.fetchexamples.jsonlner-annotated.jsonl”dish,ingredient,equipment”
```

--------------------------------

### Prodigy CLI: Classify Text with textcat.openai.correct

Source: https://prodi.gy/docs/docs/large-language-models

This command initiates a Prodigy session using the `textcat.openai.correct` recipe. It processes a JSONL file with text examples, generating prompts for OpenAI to classify them into specified labels, enabling interactive review within Prodigy.

```Shell
prodigytextcat.openai.correctrecipe-comments-textcatexamples.jsonl--label recipe,feedback,question
```

--------------------------------

### Prodigy CLI: Resume Terminology Generation (terms.openai.fetch)

Source: https://prodi.gy/docs/docs/large-language-models

This command allows resuming an existing `terms.openai.fetch` process using the `--resume` flag. It reuses previously generated terms as seeds for OpenAI, preventing duplicates and efficiently continuing the generation of additional examples.

```Shell
prodigyterms.openai.fetch"skateboard tricks"skateboard-tricks.jsonl--n 50--resume
```

--------------------------------

### Example spacy-llm Configuration for Prodigy `ab.llm.tournament` (GPT-4)

Source: https://prodi.gy/docs/docs/recipes

This configuration file provides another example for `spacy-llm` setup for the `prodigy ab.llm.tournament` recipe, specifically using the GPT-4 model. Similar to the GPT-3.5 config, it defines the pipeline, `llm` component, and `prodigy.TextPrompter.v1` task, but specifies `spacy.GPT-4.v1` as the model. This allows for direct comparison of different LLM models within the tournament.

```Configuration
### Example spacy-llm config for the tournament recipe

[nlp]

lang = "en"

pipeline = ["llm"]

[components]

[components.llm]

factory = "llm"

[components.llm.task]

@llm_tasks = "prodigy.TextPrompter.v1"

[components.llm.model]

@llm_models = "spacy.GPT-4.v1"

config = {"temperature": 0.3}
```

--------------------------------

### Shell Script for Prodigy Container Startup

Source: https://prodi.gy/docs/docs/deployment

This `run.sh` script orchestrates the startup process within the Docker container. It first executes `mkconfig.py` to configure `prodigy.json` with environment variables, then downloads example data, displays Prodigy statistics for debugging, and finally launches the Prodigy `ner.manual` recipe.

```Bash
# Make sure prodigy.json has appropriate environment variables

python scripts/mkconfig.py

# Small bit of code that downloads data, can be replaced with proper script

# that fetches data from s3 or another storage source.

wget https://raw.githubusercontent.com/explosion/prodigy-recipes/master/example-datasets/news_headlines.jsonl

# Show the stats, just for the logs

python -m prodigy stats

# Start Prodigy

python -m prodigy ner.manual nyt blank:en news_headlines.jsonl --label "COMPANY,PERSON"
```

--------------------------------

### OpenAI Prompt for Binary Classification (Recipe Detection)

Source: https://prodi.gy/docs/docs/large-language-models

This snippet demonstrates a structured OpenAI prompt for binary classification, specifically for determining if a given text contains a recipe. It includes examples of 'accept' and 'reject' responses with reasons, guiding the model on the expected output format.

```APIDOC
From the text below, determine whether or not it contains a recipe. If it is
a recipe, answer "accept." If it is not a recipe, answer "reject."
Your answer should only be in the following format:
answer: <string>
reason: <string>
Below are some examples (only use these as a guide):
Text:
"""
This is a recipe for scrambled egg: 2 eggs, 1 butter, batter them, and then fry in a hot pan for 2 minutes
"""
answer: accept
reason: This is a recipe for making a scrambled egg
Text:
"""
This is a recipe for fried rice: 1 cup of day old rice, 1 butter, 2 cloves of garlic: put them all in a wok and stir them together.
"""
answer: accept
reason: This is a recipe for making a fried rice
Text:
"""
I tried it and it's not good
"""
answer: reject
reason: It doesn't talk about a recipe.
Here is the text that needs classification
Text:
"""
Cream cheese is really good in mashed potatoes.
"""
```

--------------------------------

### Generated Prompt for spacy-llm Span Categorization

Source: https://prodi.gy/docs/docs/large-language-models

This text illustrates the prompt structure generated by spacy-llm for span categorization. It instructs the LLM to act as an NER system, defines labels and their descriptions, and provides a few-shot example to guide the entity extraction process, emphasizing overlapping spans.

```Text
You are an expert Named Entity Recognition (NER) system. Your task is to accept
Text as input and extract named entities for the set of predefined entity
labels.
The entities you extract for each label can overlap with each other.
From the Text input provided, extract named entities for each label in
the following format:
DISH: <comma delimited list of strings>
EQUIPMENT: <comma delimited list of strings>
INGREDIENT: <comma delimited list of strings>
Below are definitions of each label to help aid you in what kinds of named entities
to extract for each label. Assume these definitions are written by an expert and
follow them closely.
DISH: Extract the name of a known dish.
INGREDIENT: Extract the name of a cooking ingredient, including herbs and spices.
EQUIPMENT: Extract any mention of cooking equipment. e.g. oven, cooking pot, grill.
Below are some examples (only use these as a guide):
Text:
'''
Mac and Cheese is a popular American pasta
variant.
'''
INGREDIENT: Cheese
DISH: Mac and Cheese
Here is the text that needs labeling:
Text:
'''
Spaghetti Bolognaise is a dish.
'''
```

--------------------------------

### Prodigy Configuration Settings Reference

Source: https://prodi.gy/docs/docs/install

Detailed reference for available settings in the `prodigy.json` configuration file, including their descriptions, default values, and usage notes.

```APIDOC
Setting: theme
  Description: Name of UI theme to use.
  Default: "basic"
Setting: custom_theme
  Description: Custom UI theme overrides, keyed by name.
  Default: {}
Setting: buttons
  Description: New: 1.10 Buttons to show at the bottom of the screen in order. If an answer button is disabled, the user will be unable to submit this answer and the [keyboard shortcut](/docs/api-web-app#actions) will be disabled as well. Only the "undo" action will stay available via keyboard shortcut or click on the sidebar history.
  Default: ["accept", "reject", "ignore", "undo"]
Setting: batch_size
  Description: Number of tasks to return to and receive back from the web app at once. A low batch size means more frequent updates.
  Default: 10
Setting: history_size
  Description: New: 1.10 Maximum number of examples to show in the sidebar history. Defaults to the value of `batch_size`. Note that the history size can’t be larger than the batch size.
  Default: 10
Setting: port
  Description: Port to use for serving the web application. Can be overwritten by the `PRODIGY_PORT` environment variable.
  Default: 8080
Setting: host
  Description: Host to use for serving the web application. Can be overwritten by the `PRODIGY_HOST` environment variable.
  Default: "localhost"
Setting: cors
  Description: Enable or disable [cross-origin resource sharing](https://en.wikipedia.org/wiki/Cross-origin_resource_sharing) (CORS) to allow the REST API to receive requests from other domains.
  Default: true
Setting: db
  Description: Name of database to use.
  Default: "sqlite"
Setting: db_settings
  Description: Additional settings for the respective databases.
  Default: {}
Setting: validate
  Description: Validate incoming tasks and raise an error with more details if the format is incorrect.
  Default: true
Setting: auto_exclude_current
  Description: Automatically exclude examples already present in current dataset.
  Default: true
Setting: instant_submit
  Description: Instantly submit a task after it’s answered in the app, skipping the history and immediately triggering the update callback if available.
  Default: false
Setting: feed_overlap
  Description: Whether to send out each example once so it’s annotated by *someone* (`false`) or whether to send out each example to every session (`true`). Should be used with [custom user sessions](/docs/api-web-app#multi-user-sessions) set via the app (via `/?session=user_name`). Setting `feed_overlap` to `true` and `annotations_per_task` will result in a config validation error as they are contradictory.
  Default: false
Setting: annotations_per_task
  Description: New: 1.12 Configure [partial overlap](/docs/task-routing#task-routing) by specifying the average number of expected annotations per task. Setting `feed_overlap` to `true` and `annotations_per_task` will result in a config validation error as they are contradictory. Should be used with [custom user sessions](/docs/api-web-app#multi-user-sessions) set via the app (via `/?session=user_name`).
  Default: null
Setting: allow_work_stealing
  Description: New: 1.12 Whether it should be possible for a session that has run out of tasks to queue up tasks from other (possibly idle) sessions. This setting makes sure that all tasks are annotated as soon as possible at the cost of potentially introducing some duplicates in the final dataset.
  Default: true
Setting: total_examples_target
  Description: New: 1.11 Total number of examples that should be annotated to reach a progress of 100%. Useful for infinite streams or if completion doesn’t map to stream size. If `0`, the target will be read from stream if possible.
  Default: 0
Setting: ui_lang
  Description: New: 1.10 Language of descriptions, messages and tooltips in the UI. [See here](/docs/api-web-app#ui-language) for available translations.
  Default: "en"
Setting: ui_lang_locale
  Description: New: 1.18 Overrides for individual descriptions, messages and tooltips in the UI. [See here](/docs/api-web-app#ui-language-custom) for available options.
  Default: {}
Setting: project_info
  Description: New: 1.10 Project info shown [in sidebar](/docs/api-web-app#statistics), if available. The list of string IDs can be modified to hide or re-order the items.
  Default: ["dataset", "session", "lang", "recipe_name", "view_id", "label"]
```

--------------------------------

### OpenAI NER Response Example

Source: https://prodi.gy/docs/docs/large-language-models

An example of the non-deterministic response received from OpenAI after processing the NER prompt, showing extracted entities in the specified format.

```APIDOC
dish: hoisin stir fry

ingredient: Sriracha sauce

equipment: wok
```

--------------------------------

### Example Prodigy metric.iaa.span Output

Source: https://prodi.gy/docs/docs/metrics

This snippet displays the console output of the Prodigy `metric.iaa.span` recipe. It includes annotation statistics (e.g., total examples, categories, annotators), agreement statistics (pairwise F1 per category, micro-average F1), and a normalized global confusion matrix, providing insights into annotator consistency and label confusion.

```Text
ℹ Annotation Statistics

Attribute                     Value

--------------------------    -----

Examples                      22    # total examples in the dataset

Categories                    2     # the number of labels

Coincident Examples*          5    # overlapping examples (annotated by at least 2 annotators)

Single Annotation Examples    0     # examples annotated by a single annotator

Annotators                    5     # the number of annotators

Avg. Annotations per Example  4.40  # average number of annotations per example

* (>1 annotation)

ℹ Agreement Statistics

Category                     Pairwise F1   Support

--------------------------   -----------   -------

PERSON                        0.50         150

ORGANISATION                  0.40         200

Micro-average F1:  0.45

ℹ Confusion matrix

dish  ingredient  NONE

----------  ----  ----------  ----

dish        0.50  -           0.50

ingredient     -  0.37        0.63

NONE        0.31  0.28        0.0
```

--------------------------------

### Example Invocation of Prodigy ab.openai.prompts Recipe

Source: https://prodi.gy/docs/docs/recipes

An example command demonstrating how to run the `ab.openai.prompts` recipe. It uses specific input and template files (`haiku-inputs.jsonl`, `title.jinja2`, `prompt1.jinja2`, `prompt2.jinja2`) and sets the `--repeat` parameter to 5 for multiple evaluations.

```Shell
prodigy ab.openai.prompts haiku haiku-inputs.jsonl title.jinja2 prompt1.jinja2 prompt2.jinja2 --repeat 5
```

--------------------------------

### Prodigy textcat.llm.fetch Command Example

Source: https://prodi.gy/docs/docs/recipes

An example command demonstrating how to run the `prodigy textcat.llm.fetch` recipe, specifying the spacy-llm configuration, source data, and the output dataset.

```Shell
prodigy textcat.llm.fetch config.cfg examples.jsonl dataset:textcat-annotated
```

--------------------------------

### Get Total Examples by Session in Controller

Source: https://prodi.gy/docs/docs/api-components

Retrieves the total number of examples available in the dataset for a specific named session. This helps in tracking progress for multi-user annotation setups.

```APIDOC
Controller.get_total_by_session(session_id: str)
  session_id: ID of the named session.
RETURNS: int: The number of annotations.
```

--------------------------------

### Serve Prodigy Web Application from Python

Source: https://prodi.gy/docs/docs/api-components

This API documentation describes the `prodigy.serve` function, which allows users to programmatically start the Prodigy web application and serve a recipe directly from a Python script. It mirrors the functionality of the `prodigy` command-line tool, simplifying the process for custom recipe development by eliminating the need for the `-F` flag.

```APIDOC
Function: prodigy.serve()
Description: Serve a Prodigy recipe and start the web app from Python. This function provides the same core functionality as the 'prodigy' command-line interface. When used within a Python script, it eliminates the need for the '-F' flag to specify custom recipe files. Users can directly import the recipe function or define it within the same file as the 'prodigy.serve' call. The '@prodigy.recipe' decorator automatically registers the recipe for availability.
```

--------------------------------

### Jinja2 Template for Term Generation (terms)

Source: https://prodi.gy/docs/docs/large-language-models

This Jinja2 template is used for generating a specified number of examples based on a given query. It includes placeholders for the number of examples (`n`) and the query itself, and can incorporate seed examples if provided.

```Jinja2
### terms.jinja2

Generate me {{n}} examples of {{query}}.

Here are the examples:

{%- if examples -%}

{%- for example in examples -%}

{id="" whitespace #}

- {{example}}

{%- endfor -%}

{%- endif -%}

{id="" whitespace #}

-
```

--------------------------------

### Example Output of Prodigy Basic Logging

Source: https://prodi.gy/docs/docs/install

Shows a detailed console output when Prodigy is run with `PRODIGY_LOGGING=basic` using the `ner.teach` recipe. It illustrates the types of events logged, such as database connections, recipe calls, model initialization, and annotation progress, providing insight into the application's flow.

```shell
$ PRODIGY_LOGGING=basic prodigy ner.teach test en_core_web_sm "rats" --api guardian

13:11:39 - DB: Connecting to database 'sqlite'

13:11:40 - RECIPE: Calling recipe 'ner.teach'

13:11:40 - RECIPE: Starting recipe ner.teach

13:11:40 - LOADER: Loading stream from API 'guardian'

13:11:40 - LOADER: Using API 'guardian' with query 'rats'

13:11:42 - MODEL: Added sentence boundary detector to model pipeline

13:11:43 - RECIPE: Initialised EntityRecognizer with model en_core_web_sm

13:11:43 - SORTER: Resort stream to prefer uncertain examples (bias 0.8)

13:11:43 - PREPROCESS: Splitting sentences

13:11:44 - MODEL: Predicting spans for batch (batch size 64)

13:11:44 - Model: Sorting batch by entity type (batch size 32)

13:11:44 - CONTROLLER: Initialising from recipe

13:11:44 - DB: Loading dataset 'test' (168 examples)

13:11:44 - DB: Creating dataset '2017-11-20_13-07-44'

✨  Starting the web server at http://localhost:8080 ...

Open the app in your browser and start annotating!

13:11:59 - GET: /project

13:11:59 - GET: /get_questions

13:11:59 - CONTROLLER: Iterating over stream

13:11:59 - Model: Sorting batch by entity type (batch size 32)

13:11:59 - CONTROLLER: Returning a batch of tasks from the queue

13:11:59 - RESPONSE: /get_questions (10 examples)

13:12:07 - GET: /get_questions

13:12:07 - CONTROLLER: Returning a batch of tasks from the queue

13:12:07 - RESPONSE: /get_questions (10 examples)

13:12:07 - POST: /give_answers (received 7)

13:12:07 - CONTROLLER: Receiving 7 answers

13:12:07 - MODEL: Updating with 7 examples

13:12:08 - MODEL: Updated model (loss 0.0172)

13:12:08 - PROGRESS: Estimating progress of 0.0909

13:12:08 - DB: Getting dataset 'test'

13:12:08 - DB: Getting dataset '2017-11-20_13-07-44'

13:12:08 - DB: Added 7 examples to 2 datasets

13:12:08 - CONTROLLER: Added 7 answers to dataset 'test' in database SQLite

13:12:08 - RESPONSE: /give_answers

13:12:11 - DB: Saving database

Saved 7 annotations to database SQLite

Dataset: test

Session ID: 2017-11-20_13-07-44
```

--------------------------------

### Prodigy textcat.llm.correct Command Example

Source: https://prodi.gy/docs/docs/recipes

An example command demonstrating how to run the `prodigy textcat.llm.correct` recipe with a specified dataset, spacy-llm configuration file, and source data.

```Shell
prodigy textcat.llm.correct annotated-recipes config.cfg examples.jsonl
```

--------------------------------

### Prodigy Command for NER Correction with Few-Shot Examples

Source: https://prodi.gy/docs/docs/large-language-models

This command extends the `ner.openai.correct` recipe by incorporating few-shot examples from `ner-examples.yaml`. It helps improve the accuracy of Named Entity Recognition by providing the OpenAI model with pre-annotated examples.

```Bash
prodigyner.openai.correctrecipe-nerexamples.jsonl--label dish,ingredient,equipment--examples-path ./ner-examples.yaml
```

--------------------------------

### Example Command for `ner.llm.fetch`

Source: https://prodi.gy/docs/docs/recipes

This command demonstrates how to run the `ner.llm.fetch` recipe. It specifies `config.cfg` as the spacy-llm configuration file, `examples.jsonl` as the source data, and `ner-annotated.jsonl` as the output file where the fetched annotations will be saved.

```bash
prodigy ner.llm.fetch config.cfg examples.jsonl ner-annotated.jsonl
```

--------------------------------

### Run Prodigy with Basic Logging

Source: https://prodi.gy/docs/docs/install

Demonstrates how to run a Prodigy recipe (`ner.manual`) with the `PRODIGY_LOGGING` environment variable set to `basic` for minimal logging output, showing the command-line syntax.

```shell
PRODIGY_LOGGING=basic prodigy ner.manual my_dataset en_core_web_sm ./my_data.jsonl --label PERSON,ORG
```

--------------------------------

### Example of LLM Generated Terms

Source: https://prodi.gy/docs/docs/large-language-models

This JSONL snippet shows examples of terms generated by an LLM, each with a 'text' field and 'meta' data indicating the topic. These terms can be reviewed and processed further in Prodigy.

```JSON
{"text":"pop shove it","meta":{"topic":"skateboard tricks"}}
{"text":"switch flip","meta":{"topic":"skateboard tricks"}}
{"text":"nose slides","meta":{"topic":"skateboard tricks"}}
{"text":"lazerflip","meta":{"topic":"skateboard tricks"}}
{"text":"lipslide","meta":{"topic":"skateboard tricks"}}
```

--------------------------------

### Example Usage of Prodigy `ab.openai.tournament` Recipe

Source: https://prodi.gy/docs/docs/recipes

Provides a concrete example of how to invoke the `prodigy ab.openai.tournament` recipe with specific arguments, demonstrating a typical command for running a prompt tournament.

```Shell
prodigyab.openai.tournamenthaiku-tournamentinput.jsonltitle.jinja2prompt_folder--resume
```

--------------------------------

### Run Prodigy A/B Test for OpenAI Prompts

Source: https://prodi.gy/docs/docs/large-language-models

This command initiates an A/B test using the `ab.openai.prompts` recipe. It takes the input `topics.jsonl`, a display template, and two prompt templates (`prompt1.jinja2`, `prompt2.jinja2`). The `--repeat 4` option ensures each topic generates prompts at least four times for comparison.

```Shell
prodigyab.openai.promptshaikutopics.jsonldisplay-template.jinja2prompt1.jinja2prompt2.jinja2--repeat 4
```

--------------------------------

### Prodigy `review` Command Example (Binary Classification)

Source: https://prodi.gy/docs/docs/recipes

An example demonstrating the `prodigy review` command for binary classification tasks. It shows how to specify the output dataset and multiple input datasets for review.

```shell
prodigyreviewfood_reviews_finalfood_reviews2019,food_reviews2018
```

--------------------------------

### Few-Shot Examples for spacy.SpanCat.v3 Task

Source: https://prodi.gy/docs/docs/large-language-models

This JSON array provides structured few-shot examples for training the `spacy.SpanCat.v3` model. Each example includes text and a list of spans, detailing the extracted text, whether it's an entity, its assigned label, and a chain-of-thought reason for the classification.

```JSON
[

{

"text": "Spaghetti Bolognaise is a great dish.",

"spans": [

{

"text": "Spaghetti",

"is_entity": true,

"label": "INGREDIENT",

"reason": "It is part of the dish name, but it indicates a key ingredient."

},

{

"text": "Spaghetti Bolognaise",

"is_entity": true,

"label": "DISH",

"reason": "It is the name of a popular pasta dish."

}

]

}

]
```

--------------------------------

### Execute Prodigy `terms.llm.fetch` with Example Arguments

Source: https://prodi.gy/docs/docs/recipes

This command-line example shows how to run the `prodigy terms.llm.fetch` recipe. It specifies `skateboard-trick-terms` as the dataset, `config.cfg` as the spacy-llm configuration file, and `skateboard tricks` as the topic. The output demonstrates the progress bar for the LLM requests.

```Shell
prodigyterms.llm.fetchskateboard-trick-termsconfig.cfg”skateboard tricks”

100%|████████████████████████████| 50/50 [00:12<00:00, 3.88it/s]
```

--------------------------------

### Build Dockerfile for Prodigy Deployment

Source: https://prodi.gy/docs/docs/deployment

This Dockerfile defines the steps to create a Docker image for Prodigy, including setting the base Python image, copying dependencies and environment variables, installing Prodigy, and configuring environment variables for logging and user sessions. It highlights considerations like platform specification and secure handling of the Prodigy key.

```Dockerfile
### Dockerfile
# You may need to specify the --platform depending on cloud provider
FROM --platform=linux/amd64 python:3.9

# Copy requirements first
COPY requirements.txt .

# Next, copy in the .env file, which contains the PRODIGY_KEY variable
COPY .env .

# Install everything
RUN python -m pip install --upgrade pip && \
python -m pip install -r requirements.txt && \
export $(cat .env) && \
python -m pip install prodigy -f https://${PRODIGY_KEY}@download.prodi.gy

# Copy the rest in, keeping .dockerignore in mind
COPY . .

# Set some environment variables
ENV PRODIGY_LOGGING "basic"
ENV PRODIGY_ALLOWED_SESSIONS "user1,user2,user3,user4,user5,user6"
ENV PRODIGY_BASIC_AUTH_USER "prodigy-user"
ENV PRODIGY_BASIC_AUTH_PASS "simple-but-not-super-easy-to-guess"
```

--------------------------------

### Prodigy `review` Command Example (Manual Annotation)

Source: https://prodi.gy/docs/docs/recipes

An example demonstrating the `prodigy review` command for manual annotation tasks. It illustrates how to specify input datasets and optional labels for display during the review process.

```shell
prodigyreviewner_finalner_news,ner_misc--label ORG,PRODUCT
```

--------------------------------

### JSONL Example for NER Annotation

Source: https://prodi.gy/docs/docs/large-language-models

This JSON Lines snippet provides an example text used for Named Entity Recognition (NER) annotation. It demonstrates the expected input format for the `ner.openai.correct` recipe in Prodigy.

```JSON
{"text": "Sriracha sauce goes really well with hoisin stir fry, but you should add it after you use the wok."}
```

--------------------------------

### Example Response from spacy-llm Span Categorization

Source: https://prodi.gy/docs/docs/large-language-models

An example of the expected output format from the LLM for span categorization, showing extracted entities for INGREDIENT and DISH labels based on the provided text and prompt structure.

```Text
INGREDIENT: Spaghetti
DISH: Spaghetti Bolognaise
```

--------------------------------

### Example LLM Response for NER

Source: https://prodi.gy/docs/docs/large-language-models

This snippet shows an example of the output generated by a Large Language Model in response to the NER prompt. It demonstrates the extracted 'dish' and 'ingredient' entities based on the provided text.

```Text Response
dish: pizza

ingredient: anchovis

equipment:
```

--------------------------------

### Start Prodigy Text Classification Annotation Session

Source: https://prodi.gy/docs/docs

Shows the command to start a manual text classification annotation session with Prodigy, specifying input data, labels, and the web server startup message. It explains how to interact with the UI for multi-label classification.

```bash
prodigytextcat.manualnews_topics./news_headlines.jsonl--label Technology,Politics,Economy,Entertainment
```

```log
✨ Starting the web server at http://localhost:8080 ...
Open the app in your browser and start annotating!
```

--------------------------------

### Run Prodigy NER Manual Annotation Recipe

Source: https://prodi.gy/docs/docs/sso

This command initiates a Prodigy manual named entity recognition (NER) annotation session. It uses the `manualner_news_headlinesblank` recipe on the `news_headlines.jsonl` dataset, allowing users to label entities such as PERSON, ORG, PRODUCT, and LOCATION. This command is executed from the command line to start the Prodigy web server.

```shell
prodigyner.manualner_news_headlinesblank:en./news_headlines.jsonl--label PERSON,ORG,PRODUCT,LOCATION
```

--------------------------------

### Few-Shot Examples JSON Structure for NER

Source: https://prodi.gy/docs/docs/large-language-models

This JSON snippet illustrates the structure of `examples.json` used for few-shot learning in Named Entity Recognition (NER). It defines text examples with annotated spans, indicating whether a span is an entity, its label, and a reason for the classification, including negative examples.

```JSON
[

{

"text": "You can't get a great chocolate flavor with carob.",

"spans": [

{

"text": "chocolate",

"is_entity": false,

"label": "==NONE==",

"reason": "is a flavor in this context, not an ingredient"

},

{

"text": "carob",

"is_entity": true,

"label": "INGREDIENT",

"reason": "is an ingredient to add chocolate flavor"

}

]

},

{

"text": "You can probably sand-blast it if it's an anodized aluminum pan",

"spans": [

{

"text": "sand-blast",

"is_entity": false,

"label": "==NONE==",

"reason": "is a cleaning technique, not some kind of equipment"

},

{

"text": "anodized aluminum pan",

"is_entity": true,

"label": "EQUIPMENT",

"reason": "is a piece of cooking equipment, anodized is included since it describes the type of pan"

}

]

}

]
```

--------------------------------

### Train Named Entity Recognition (NER) Model with Prodigy

Source: https://prodi.gy/docs/docs

Demonstrates how to initiate training for a Named Entity Recognition (NER) model using Prodigy's `train` command. It shows the command line invocation and the subsequent training log, including component setup, data split, and performance metrics over epochs.

```bash
prodigytrain--ner ner_news_headlines
```

```log
======================== Generating Prodigy config ========================
ℹ Auto-generating config with spaCy
✔ Generated training config
============================ Training pipeline ============================

Components: ner

Merging training and evaluation data for 1 components

- [ner] Training: 298 | Evaluation: 74 (20% split)

Training: 298 | Evaluation: 74

Labels: ner (4)


ℹ Pipeline: [‘tok2vec’, ‘ner’]ℹ Initial learn rate: 0.001
============================ Training pipeline ============================

E    #       LOSS TOK2VEC  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE
---  ------  ------------  --------  ------  ------  ------  ------
0       0          0.00     55.22    0.00    0.00    0.00    0.00
7     200        100.81   1831.94   70.91   81.25   62.90    0.71
17     400         83.33    117.20   71.70   86.36   61.29    0.72
28     600         13.83     17.33   75.93   89.13   66.13    0.76
43     800          0.44      0.41   72.38   88.37   61.29    0.72
60    1000          0.01      0.01   71.15   88.10   59.68    0.71
81    1200         68.97     16.09   72.07   81.63   64.52    0.72
107    1400        138.88     19.77   75.47   90.91   64.52    0.75
139    1600        165.04     18.88   72.38   88.37   61.29    0.72
177    1800        135.35     15.34   75.73   95.12   62.90    0.76
223    2000        165.23     20.49   73.08   90.48   61.29    0.73
278    2200        297.13     26.26   71.03   84.44   61.29    0.71
```

--------------------------------

### Prodigy `data-to-spacy` Command Line Example

Source: https://prodi.gy/docs/docs/recipes

Example of the `prodigy data-to-spacy` command demonstrating its general structure and available flags for specifying output directory and various dataset types for different spaCy pipeline components.

```terminal
prodigydata-to-spacyoutput_dir--ner--textcat--textcat-multilabel--tagger--parser--senter--spancat--coref--eval-split--config--base-model--lang--verbose-F
```

--------------------------------

### Run Prodigy Manual NER Recipe for News Headlines

Source: https://prodi.gy/docs/docs

This command initiates the Prodigy web server and starts a manual Named Entity Recognition annotation session. It uses the `manualner` recipe with a blank English model, loads data from `news_headlines.jsonl`, and allows annotation for PERSON, ORG, PRODUCT, and LOCATION labels. The server will be accessible at `http://localhost:8080`.

```Shell
prodigyner.manualner_news_headlinesblank:en./news_headlines.jsonl--label PERSON,ORG,PRODUCT,LOCATION
```

--------------------------------

### Start Prodigy `ner.teach` for Binary ORG Annotation

Source: https://prodi.gy/docs/docs/named-entity-recognition

This command initializes the Prodigy server to perform binary named entity recognition annotation using the `ner.teach` recipe. It uses the `en_core_web_sm` model and `news_headlines.jsonl` dataset, focusing on the `ORG` entity type. The progress indicator in the sidebar provides an estimate of the remaining annotation work until the model has learned sufficiently.

```bash
prodigy ner.teach news_headlines_binary en_core_web_sm ./news_headlines.jsonl --label ORG
```

--------------------------------

### Prodigy Tournament Command Line Example

Source: https://prodi.gy/docs/docs/recipes

This command demonstrates how to run a Prodigy A/B tournament for LLM prompts. It specifies the recipe, input data, prompt and config folders, and the `--resume` flag.

```Shell
prodigyab.llm.tournamenthaiku-tournamentinputs.jsonl./prompt_folder./config_folder--resume
```
Tokens:
Raw