### Install Palimpzest (PZ) via pip

Source: https://github.com/mitdbg/palimpzest/blob/main/README.md

This command installs the stable version of the Palimpzest package from PyPI. It is the recommended way to get started with PZ for general use.

```bash
pip install palimpzest
```

--------------------------------

### Install Palimpzest from GitHub source

Source: https://github.com/mitdbg/palimpzest/blob/main/docs/docs/getting-started/installation.md

Clones the Palimpzest GitHub repository to your local machine. After navigating into the cloned directory, it installs the library from the local source using pip, useful for development or specific versions.

```bash
git clone git@github.com:mitdbg/palimpzest.git
cd palimpzest
pip install .
```

--------------------------------

### Launch Jupyter Notebook for Palimpzest quickstart

Source: https://github.com/mitdbg/palimpzest/blob/main/README.md

This command starts the Jupyter Notebook server, allowing users to access and run the `quickstart.ipynb` notebook in their web browser. The notebook demonstrates the full Palimpzest workflow.

```bash
jupyter notebook
```

--------------------------------

### Install Palimpzest and Dependencies

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

This Python snippet installs the Palimpzest library (`palimpzest==0.7.6`) and its required dependencies (`pyarrow`, `chromadb`) using pip. It also imports the `palimpzest` module as `pz` for subsequent use. PIP dependency error messages are expected and can be safely ignored.

```python
!pip install palimpzest==0.7.6
!pip install --upgrade pyarrow
!pip install chromadb==0.6.3
import palimpzest as pz
```

--------------------------------

### Install Palimpzest (PZ) from GitHub repository

Source: https://github.com/mitdbg/palimpzest/blob/main/README.md

These commands clone the Palimpzest repository from GitHub and install the package in editable mode. Use this method for development or to access the very latest unreleased features.

```bash
git clone git@github.com:mitdbg/palimpzest.git
cd palimpzest
pip install .
```

--------------------------------

### Install Palimpzest with pip

Source: https://github.com/mitdbg/palimpzest/blob/main/docs/docs/getting-started/installation.md

Installs the latest stable version of the Palimpzest library using the pip package manager. This is the recommended method for most users and fetches the package from PyPI.

```bash
pip install palimpzest
```

--------------------------------

### Instantiate BioDEX Training and Test DataReaders

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

Demonstrates how to create instances of the `BiodexReader` for both training and testing datasets, specifying parameters like `split`, `num_samples`, and `seed` to control data loading and shuffling.

```python
SEED = 123

# create train dataset
train_datareader = BiodexReader(split="train", seed=SEED)
test_datareader = BiodexReader(split="test", num_samples=20, seed=SEED)
```

--------------------------------

### Setup Palimpzest and Download Sample Data

Source: https://github.com/mitdbg/palimpzest/blob/main/docs/docs/index.md

This snippet provides terminal commands for setting up the Palimpzest library, including installation via pip, configuring the OpenAI API key, and downloading/unzipping a sample dataset of emails for demonstration purposes.

```bash
# setup in the terminal
$ pip install palimpzest
$ export OPENAI_API_KEY="<your-api-key>"
$ wget https://palimpzest-workloads.s3.us-east-1.amazonaws.com/emails.zip
$ unzip emails.zip
```

--------------------------------

### Load ChromaDB Index with OpenAI Embeddings

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

Initializes a persistent ChromaDB client, configures an OpenAI embedding function using a specified model and API key, and retrieves a specific collection named 'biodex-reaction-terms'. This setup is crucial for enabling semantic search capabilities by associating text with vector embeddings.

```python
# load index [text-embedding-3-small]
chroma_client = chromadb.PersistentClient(".chroma-biodex")
openai_ef = OpenAIEmbeddingFunction(
  api_key=os.environ["OPENAI_API_KEY"],
  model_name="text-embedding-3-small",
)
index = chroma_client.get_collection("biodex-reaction-terms", embedding_function=openai_ef)
```

--------------------------------

### Download and Extract Palimpzest Test Datasets

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

This Python snippet downloads three compressed tar files (`enron-tiny.tar.gz`, `real-estate-eval-5.tar.gz`, `chroma-biodex.tar.gz`) containing test data for Palimpzest demonstrations. It then extracts their contents into the current directory using `tar -xzf`, preparing the local environment with sample data.

```python
# download tar files with testdata
!wget -nc https://people.csail.mit.edu/gerarvit/PalimpzestData/enron-tiny.tar.gz
!wget -nc wget -nc https://people.csail.mit.edu/gerarvit/PalimpzestData/real-estate-eval-5.tar.gz
!wget -nc https://palimpzest-workloads.s3.us-east-1.amazonaws.com/chroma-biodex.tar.gz

# open tar files
!tar -xzf enron-tiny.tar.gz
!tar -xzf real-estate-eval-5.tar.gz
!tar -xzf chroma-biodex.tar.gz
```

--------------------------------

### Execute Palimpzest Program with MaxQuality Policy

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

This snippet demonstrates how to execute a configured Palimpzest data processing program. It sets up a `QueryProcessorConfig` to use the `MaxQuality` policy, enabling parallel execution and progress tracking, and then runs the defined dataset pipeline.

```python
config = pz.QueryProcessorConfig(policy=pz.MaxQuality(), execution_strategy="parallel", progress=True)
output = ds.run(config)
```

--------------------------------

### Executing Palimpzest Plan with Validation Data for Optimized Quality

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

This snippet demonstrates an advanced execution of the Palimpzest plan, incorporating a `train_datareader` as a validation dataset. The `QueryProcessorConfig` uses a `sentinel` processing strategy and `pareto` optimizer strategy to achieve optimized quality, showcasing how validation data can guide the plan's execution for better results. The output is then scored.

```python
import logging
logger = logging.getLogger()
logger.disabled = True

# execute pz plan
config = pz.QueryProcessorConfig(
    policy=pz.MaxQuality(),
    val_datasource=train_datareader,
    processing_strategy="sentinel",
    optimizer_strategy="pareto",
    sentinel_execution_strategy="mab",
    execution_strategy="parallel",
    use_final_op_quality=True,
    max_workers=64,
    progress=True,
)

output = plan.run(config=config, k=6, j=4, sample_budget=72, seed=SEED)
score_output(output, seed=SEED)
```

--------------------------------

### Constructing a Palimpzest Data Processing Plan

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

This snippet demonstrates how to build a data processing plan using the `palimpzest` library. It initializes a dataset from a datareader, adds semantic columns, retrieves data using an index and search function, and enriches the plan with additional semantically derived columns based on existing data.

```python
plan = pz.Dataset(test_datareader)
plan = plan.sem_add_columns(biodex_reactions_cols)
plan = plan.retrieve(
    index=index,
    search_func=search_func,
    search_attr="reactions",
    output_attrs=biodex_reaction_labels_cols,
)
plan = plan.sem_add_columns(biodex_ranked_reactions_labels_cols, depends_on=["title", "abstract", "fulltext", "reaction_labels"])
```

--------------------------------

### Configure API Keys for Palimpzest in Colab

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

This Python snippet retrieves API keys (OPENAI_API_KEY, TOGETHER_API_KEY) from Google Colab's `userdata` and sets them as environment variables. It ensures secure access to external APIs for Palimpzest operations. This setup requires a Google Colab environment and pre-configured secrets.

```python
from google.colab import userdata
import os

# set environment variables
def set_api_key_from_secret(key_name):
  try:
    os.environ[key_name] = userdata.get(key_name)
  except:
    pass

set_api_key_from_secret('OPENAI_API_KEY')
set_api_key_from_secret('TOGETHER_API_KEY')
```

--------------------------------

### Process email data with Palimpzest (PZ) quick start

Source: https://github.com/mitdbg/palimpzest/blob/main/README.md

This Python snippet demonstrates a basic Palimpzest workflow: defining columns, applying semantic filters to a dataset, configuring a query processor, and executing the pipeline to get results. It processes email data from a local directory, showcasing data transformation and filtering with LLMs.

```python
import palimpzest as pz

# define the fields we wish to compute
email_cols = [
    {"name": "sender", "type": str, "desc": "The email address of the sender"},
    {"name": "subject", "type": str, "desc": "The subject of the email"},
    {"name": "date", "type": str, "desc": "The date the email was sent"},
]

# lazily construct the computation to get emails about holidays sent in July
dataset = pz.Dataset("testdata/enron-tiny/")
dataset = dataset.sem_add_columns(email_cols)
dataset = dataset.sem_filter("The email was sent in July")
dataset = dataset.sem_filter("The email is about holidays")

# execute the computation w/the MinCost policy
config = pz.QueryProcessorConfig(policy=pz.MinCost(), verbose=True)
output = dataset.run(config)

# display output (if using Jupyter, otherwise use print(output_df))
output_df = output.to_df(cols=["date", "sender", "subject"])
display(output_df)
```

--------------------------------

### Launch Jupyter Notebook for Palimpzest Quick Start

Source: https://github.com/mitdbg/palimpzest/blob/main/docs/docs/index.md

This command launches the Jupyter Notebook server, allowing users to access and run the `quickstart.ipynb` notebook. The notebook demonstrates the full workflow of Palimpzest, including dataset registration, pipeline composition, and result access.

```bash
$ jupyter notebook
```

--------------------------------

### Executing Palimpzest Plan with MaxQuality Policy (No Training Data)

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

This code block executes the previously defined Palimpzest plan using a `QueryProcessorConfig` set for `MaxQuality` policy and parallel execution. It runs the plan without leveraging training data for optimization and then calls the `score_output` function to evaluate the results and display performance metrics.

```python
import logging
logger = logging.getLogger()
logger.disabled = True

# execute pz plan
config = pz.QueryProcessorConfig(
    policy=pz.MaxQuality(),
    execution_strategy="parallel",
    max_workers=64,
    progress=True,
)

output = plan.run(config=config, seed=SEED)
score_output(output, seed=SEED)
```

--------------------------------

### Visualizing Multi-Modal Real Estate Data with Gradio

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

This Python code loads image and text data for real estate listings from local files, converts images to NumPy arrays, and then uses the Gradio library to create an interactive web interface for visualizing these multi-modal datasets. It sets up rows and columns to display multiple images and associated text descriptions for each listing, launching a local demo.

```python
from PIL import Image
import numpy as np
import gradio as gr

# Boilerplate code to build our visualization
fst_imgs, snd_imgs, thrd_imgs, texts = [], [], [], []
for idx in range(1, 6):
    listing = f"listing{idx}"
    with open(os.path.join("real-estate-eval-5", listing, "listing-text.txt")) as f:
        texts.append(f.read())
    for idx, img_name in enumerate(["img1.png", "img2.png", "img3.png"]):
        path = os.path.join("real-estate-eval-5", listing, img_name)
        img = Image.open(path)
        img_arr = np.asarray(img)
        if idx == 0:
            fst_imgs.append(img_arr)
        elif idx == 1:
            snd_imgs.append(img_arr)
        elif idx == 2:
            thrd_imgs.append(img_arr)

with gr.Blocks() as demo:
    fst_img_blocks, snd_img_blocks, thrd_img_blocks, text_blocks = [], [], [], []
    for fst_img, snd_img, thrd_img, text in zip(fst_imgs, snd_imgs, thrd_imgs, texts):
        with gr.Row(equal_height=True):
            with gr.Column():
                fst_img_blocks.append(gr.Image(value=fst_img))
            with gr.Column():
                snd_img_blocks.append(gr.Image(value=snd_img))
            with gr.Column():
                thrd_img_blocks.append(gr.Image(value=thrd_img))
        with gr.Row():
            with gr.Column():
                text_blocks.append(gr.Textbox(value=text, info="Text Description"))

demo.launch()
```

--------------------------------

### Accessing Palimpzest Execution Statistics and Final Plan

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

This code illustrates how to retrieve and print detailed execution statistics from a Palimpzest 'output' object, including optimization time and cost, as well as plan execution time and cost. It also shows how to access and display the string representation of the final optimized plan that Palimpzest executed.

```python
print(f"Optimization Time: {output.execution_stats.optimization_time:.2f}s")
print(f"Optimization Cost: ${output.execution_stats.optimization_cost:.3f}")
print("---")
print(f"Plan Execution Time: {output.execution_stats.plan_execution_time:.2f}s")
print(f"Plan Execution Cost: ${output.execution_stats.plan_execution_cost:.3f}")

print("Final plan executed:")
print("---")
final_plan_id = list(output.execution_stats.plan_strs.keys())[-1]
print(output.execution_stats.plan_strs[final_plan_id])
```

--------------------------------

### Visualize Palimpzest Real Estate Output with Gradio

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

This Python code processes the output from a Palimpzest real estate program and visualizes it using the Gradio library. It extracts image paths, addresses, and prices from the processed records, then displays them in an interactive web interface, including the query execution plan.

```python
from PIL import Image
import numpy as np
import gradio as gr

demo.close()

# Boilerplate code to build our visualization
fst_imgs, snd_imgs, thrd_imgs, addrs, prices = [], [], [], [], []
for record in output:
    addrs.append(record.address)
    prices.append(record.price)
    for idx, img_name in enumerate(["img1.png", "img2.png", "img3.png"]):
        path = os.path.join("real-estate-eval-5", record.listing, img_name)
        img = Image.open(path)
        img_arr = np.asarray(img)
        if idx == 0:
            fst_imgs.append(img_arr)
        elif idx == 1:
            snd_imgs.append(img_arr)
        elif idx == 2:
            thrd_imgs.append(img_arr)

with gr.Blocks() as demo:
    fst_img_blocks, snd_img_blocks, thrd_img_blocks, addr_blocks, price_blocks = [], [], [], [], []
    for fst_img, snd_img, thrd_img, addr, price in zip(fst_imgs, snd_imgs, thrd_imgs, addrs, prices):
        with gr.Row(equal_height=True):
            with gr.Column():
                fst_img_blocks.append(gr.Image(value=fst_img))
            with gr.Column():
                snd_img_blocks.append(gr.Image(value=snd_img))
            with gr.Column():
                thrd_img_blocks.append(gr.Image(value=thrd_img))
        with gr.Row():
            with gr.Column():
                addr_blocks.append(gr.Textbox(value=addr, info="Address"))
            with gr.Column():
                price_blocks.append(gr.Textbox(value=price, info="Price"))

    plan_str = list(output.execution_stats.plan_strs.values())[0]
    gr.Textbox(value=plan_str, info="Query Plan")

demo.launch()
```

--------------------------------

### Execute Palimpzest Pipeline with MaxQuality Policy

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

This Python snippet executes a previously defined Palimpzest dataset computation. It configures the query processor to use the `MaxQuality` policy, enabling parallel execution and progress display. The `run()` method triggers the actual data processing based on the defined semantic graph, optimizing for output quality.

```python
# execute the computation w/the MaxQuality policy
config = pz.QueryProcessorConfig(policy=pz.MaxQuality(), execution_strategy="parallel", progress=True)
output = dataset.run(config)
```

--------------------------------

### Construct Palimpzest Program for Real Estate Data Filtering

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

This code constructs a Palimpzest data processing pipeline for real estate listings. It initializes a dataset, semantically adds 'address' and 'price' columns, applies a semantic filter for property descriptions, and then uses a UDF to filter records based on a price range.

```python
ds = pz.Dataset(RealEstateListingReader("real-estate-eval-5"))
ds = ds.sem_add_columns(real_estate_text_cols, depends_on="text_content")
ds = ds.sem_filter(
    "The interior is modern and attractive, and has lots of natural sunlight",
    depends_on="image_filepaths",
)
ds = ds.filter(in_price_range, depends_on="price")
```

--------------------------------

### Download test data for Palimpzest (PZ) demos

Source: https://github.com/mitdbg/palimpzest/blob/main/README.md

These commands make the test data download script executable and then run it. This step is crucial to obtain the large Enron email dataset, which is used in Palimpzest's demo programs and is not included in the repository due to its size.

```bash
chmod +x testdata/download-testdata.sh
./testdata/download-testdata.sh
```

--------------------------------

### Palimpzest DataReader API Specification

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

This entry outlines the essential methods and properties required for any custom 'pz.DataReader' implementation in Palimpzest. It details the 'schema' attribute for defining data structure, and the '__len__' and '__getitem__' methods for dataset iteration and data retrieval.

```APIDOC
pz.DataReader:
  - schema: list of dict
    description: Defines the fields present in each output record. Each dict should contain 'name', 'type', and 'desc'.
  - __len__(): int
    description: Returns the total number of items in the dataset.
  - __getitem__(idx: int): dict
    description: Returns the item at the specified index 'idx' as a dictionary, conforming to the defined schema.
```

--------------------------------

### Close Gradio Application

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

This simple line of code is used to close an active Gradio application instance, releasing any associated resources.

```python
demo.close()
```

--------------------------------

### Closing a Gradio Demo Instance

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

This simple line of code is used to gracefully shut down and close an active Gradio demo instance. It releases the resources held by the running Gradio application.

```python
demo.close()
```

--------------------------------

### Install Palimpzest by Cloning Repository

Source: https://github.com/mitdbg/palimpzest/blob/main/docs/docs/index.md

These commands clone the Palimpzest GitHub repository, navigate into the project directory, and then install the package in editable mode. This method is useful for development or accessing the latest unreleased features directly from the source.

```bash
git clone git@github.com:mitdbg/palimpzest.git
cd palimpzest
pip install .
```

--------------------------------

### Palimpzest DataReader `__getitem__` Return Value Specification

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

Details the required dictionary structure returned by the `__getitem__` method of a `pz.DataReader` instance, especially for data used in optimization processes within the Palimpzest framework.

```APIDOC
__getitem__ return dictionary structure:
{
  "fields": dict, // Contains the raw data emitted by the pz.DataReader
  "labels": dict, // (For 'train' data only) Expected results for each output field
  "score_fn": dict // (For 'train' data only) Scoring functions for each output field
}
```

--------------------------------

### Implementing a Custom Palimpzest DataReader for Real Estate Listings

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

This Python code defines a custom 'pz.DataReader' named 'RealEstateListingReader' to enable Palimpzest to load multi-modal real estate listing data. It specifies a schema for the data, including listing name, text content, and image file paths, and implements '__len__' and '__getitem__' methods to read and structure data from a directory of listings.

```python
from palimpzest.core.lib.fields import ImageFilepathField, ListField

# we first define the schema for each record output by the DataReader
real_estate_listing_cols = [
    {"name": "listing", "type": str, "desc": "The name of the listing"},
    {"name": "text_content", "type": str, "desc": "The content of the listing's text description"},
    {"name": "image_filepaths", "type": ListField(ImageFilepathField), "desc": "A list of the filepaths for each image of the listing"}
]

# we then implement the DataReader
class RealEstateListingReader(pz.DataReader):
    def __init__(self, listings_dir):
        super().__init__(schema=real_estate_listing_cols)
        self.listings_dir = listings_dir
        self.listings = sorted(os.listdir(self.listings_dir))

    def __len__(self):
        return len(self.listings)

    def __getitem__(self, idx: int):
        # get listing
        listing = self.listings[idx]

        # get fields
        image_filepaths, text_content = [], None
        listing_dir = os.path.join(self.listings_dir, listing)
        for file in os.listdir(listing_dir):
            if file.endswith(".txt"):
                with open(os.path.join(listing_dir, file), "rb") as f:
                    text_content = f.read().decode("utf-8")
            elif file.endswith(".png"):
                image_filepaths.append(os.path.join(listing_dir, file))

        # construct and return dictionary with fields
        return {"listing": listing, "text_content": text_content, "image_filepaths": image_filepaths}
```

--------------------------------

### Example Palimpzest Execution Statistics Output

Source: https://github.com/mitdbg/palimpzest/blob/main/docs/docs/getting-started/quickstart.md

An example of the console output when printing total execution time and cost from a Palimpzest program's execution statistics, demonstrating the typical format of these performance metrics.

```text
Total time: 41.7
Total cost: 0.081
```

--------------------------------

### Implement BioDEX DataReader for Palimpzest

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

Defines the `BiodexReader` class, a custom `pz.DataReader` for loading and processing the BioDEX-Reactions dataset. It handles data loading, shuffling, and provides methods for computing labels and scoring predictions like rank precision at K and term recall, essential for evaluating model performance.

```python
class BiodexReader(pz.DataReader):
    def __init__(
        self,
        rp_at_k: int = 5,
        num_samples: int = 10,
        split: str = "test",
        shuffle: bool = True,
        seed: int = 42,
    ):
        super().__init__(biodex_entry_cols)

        self.dataset = datasets.load_dataset("BioDEX/BioDEX-Reactions", split=split).to_pandas()
        if shuffle:
            self.dataset = self.dataset.sample(n=num_samples, random_state=seed).to_dict(orient="records")
        else:
            self.dataset = self.dataset.to_dict(orient="records")[:num_samples]

        self.rp_at_k = rp_at_k
        self.num_samples = num_samples
        self.shuffle = shuffle
        self.seed = seed
        self.split = split

    def compute_label(self, entry: dict) -> dict:
        """Compute the label for a BioDEX report given its entry in the dataset."""
        reactions_lst = [
            reaction.strip().lower().replace("'", "").replace("^", "")
            for reaction in entry["reactions"].split(",")
        ]
        label_dict = {
            "reactions": reactions_lst,
            "reaction_labels": reactions_lst,
            "ranked_reaction_labels": reactions_lst,
        }
        return label_dict

    @staticmethod
    def rank_precision_at_k(preds, targets, k: int):
        if preds is None:
            return 0.0

        try:
            # lower-case each list
            preds = [pred.strip().lower().replace("'", "").replace("^", "") for pred in preds]
            targets = set([target.strip().lower().replace("'", "").replace("^", "") for target in targets])

            # compute rank-precision at k
            rn = len(targets)
            denom = min(k, rn)
            total = 0.0
            for i in range(k):
                total += preds[i] in targets if i < len(preds) else 0.0

            return total / denom

        except Exception:
            return 0.0

    @staticmethod
    def term_recall(preds, targets):
        if preds is None:
            return 0.0

        try:
            # normalize terms in each list
            pred_terms = set([
                term.strip()
                for pred in preds
                for term in pred.lower().replace("'", "").replace("^", "").split(" ")
            ])
            target_terms = ([
                term.strip()
                for target in targets
                for term in target.lower().replace("'", "").replace("^", "").split(" ")
            ])

            # compute term recall and return
            intersect = pred_terms.intersection(target_terms)
            term_recall = len(intersect) / len(target_terms)

            return term_recall

        except Exception:
            return 0.0

    def __len__(self):
        return len(self.dataset)

    def __getitem__(self, idx: int):
        # get entry
        entry = self.dataset[idx]

        # get input fields
        pmid = entry["pmid"]
        title = entry["title"]
        abstract = entry["abstract"]
        fulltext = entry["fulltext"]

        # create item with fields
        item = {"fields": {}, "labels": {}, "score_fn": {}}
        item["fields"]["pmid"] = pmid
        item["fields"]["title"] = title
        item["fields"]["abstract"] = abstract
        item["fields"]["fulltext"] = fulltext

        if self.split == "train":
            # add label info
            item["labels"] = self.compute_label(entry)

            # add scoring functions for list fields
            rank_precision_at_k = partial(BiodexReader.rank_precision_at_k, k=self.rp_at_k)
            item["score_fn"]["reactions"] = BiodexReader.term_recall
            item["score_fn"]["reaction_labels"] = BiodexReader.term_recall
            item["score_fn"]["ranked_reaction_labels"] = rank_precision_at_k

        return item
```

--------------------------------

### Install Palimpzest via PyPI

Source: https://github.com/mitdbg/palimpzest/blob/main/docs/docs/index.md

This command installs the Palimpzest library from the Python Package Index (PyPI). It provides a stable version of the package for immediate use, ensuring all necessary dependencies are met.

```bash
$ pip install palimpzest
```

--------------------------------

### Displaying Palimpzest Output DataFrame

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

This snippet demonstrates how to convert a Palimpzest 'output' object into a Pandas DataFrame for structured data display. It specifically selects 'date', 'sender', and 'subject' columns and uses the 'display' function, which is common in Jupyter environments, to render the DataFrame.

```python
output_df = output.to_df(cols=["date", "sender", "subject"])
display(output_df)
```

--------------------------------

### Example Directory Structure for Palimpzest Dataset Input

Source: https://github.com/mitdbg/palimpzest/blob/main/docs/docs/getting-started/quickstart.md

Illustrates the expected flat directory structure for input data when initializing a `pz.Dataset` from a local path. Each file within this directory is treated as a separate data record by Palimpzest, with its contents and filename extracted by default.

```bash
emails
├── email1.txt
├── email2.txt
...
└── email9.txt
```

--------------------------------

### Configure API keys for Palimpzest (PZ) LLM access

Source: https://github.com/mitdbg/palimpzest/blob/main/README.md

These commands set environment variables for OpenAI or Together.ai API keys. At least one of these keys is required for Palimpzest to interact with large language models, enabling its data processing capabilities.

```bash
export OPENAI_API_KEY=<your-api-key>
export TOGETHER_API_KEY=<your-api-key>
```

--------------------------------

### Execute simple Palimpzest (PZ) demo with Enron dataset

Source: https://github.com/mitdbg/palimpzest/blob/main/README.md

This command runs a basic Palimpzest demo script, processing the Enron email test dataset. It showcases how to execute a pre-defined task with specified data and verbose output, demonstrating a complete end-to-end execution.

```bash
python demos/simple-demo.py --task enron --dataset testdata/enron-eval-tiny --verbose
```

--------------------------------

### Define Schema for BioDEX Medical Report DataReader

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

This snippet defines the schema for records loaded by a Palimpzest DataReader for the BioDEX dataset. It specifies fields such as PubMed ID, title, abstract, and full text for medical papers, including their types and descriptions, to facilitate structured data extraction.

```python
import datasets
from functools import partial

# define the schema for records returned by the DataReader
biodex_entry_cols = [
    {"name": "pmid", "type": str, "desc": "The PubMed ID of the medical paper"},
    {"name": "title", "type": str, "desc": "The title of the medical paper"},
    {"name": "abstract", "type": str, "desc": "The abstract of the medical paper"},
    {"name": "fulltext", "type": str, "desc": "The full text of the medical paper, which contains information relevant for creating a drug safety report."},
]
```

--------------------------------

### Defining a Performance Scoring Function for Palimpzest Output

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

This Python function, `score_output`, evaluates the quality of a Palimpzest plan's execution. It loads a test dataset, computes target labels, and calculates 'Rank Precision at K' (RP@K) to measure prediction accuracy. The function also prints detailed execution statistics, including optimization time, cost, and plan execution metrics.

```python
def score_output(output, seed):
    # score output
    test_dataset = datasets.load_dataset("BioDEX/BioDEX-Reactions", split="test").to_pandas()
    test_dataset = test_dataset.sample(n=20, random_state=seed).to_dict(orient="records")

    # construct mapping from pmid --> label (field, value) pairs
    def compute_target_record(entry):
        reactions_lst = [
            reaction.strip().lower().replace("'", "").replace("^", "")
            for reaction in entry["reactions"].split(",")
        ]
        label_dict = {"ranked_reaction_labels": reactions_lst}
        return label_dict

    label_fields_to_values = {
        entry["pmid"]: compute_target_record(entry) for entry in test_dataset
    }

    def rank_precision_at_k(preds: list, targets: list, k: int):
        if preds is None:
            return 0.0

        # lower-case each list
        preds = [pred.lower().replace("'", "").replace("^", "") for pred in preds]
        targets = set([target.lower().replace("'", "").replace("^", "") for target in targets])

        # compute rank-precision at k
        rn = len(targets)
        denom = min(k, rn)
        total = 0.0
        for i in range(k):
            total += preds[i] in targets if i < len(preds) else 0.0

        return total / denom

    def compute_avg_rp_at_k(records, k=5):
        total_rp_at_k = 0
        bad = 0
        for record in records:
            pmid = record['pmid']
            preds = record['ranked_reaction_labels']
            targets = label_fields_to_values[pmid]['ranked_reaction_labels']
            try:
                total_rp_at_k += rank_precision_at_k(preds, targets, k)
            except Exception:
                bad += 1

        return total_rp_at_k / len(records), bad

    rp_at_k, bad = compute_avg_rp_at_k([record.to_dict() for record in output], k=5)
    final_plan_id = list(output.execution_stats.plan_stats.keys())[0]
    final_plan_str = output.execution_stats.plan_strs[final_plan_id]
    print("---")
    print("#########################")
    print(f"##### RP@5: {rp_at_k:.5f} #####")
    print("#########################")
    print("---")
    print(f"Optimization time: {output.execution_stats.optimization_time:.2f}s")
    print(f"Optimization cost: ${output.execution_stats.optimization_cost:.3f}")
    print("---")
    print(f"Plan exec. time: {output.execution_stats.plan_execution_time:.2f}s")
    print(f"Plan exec. cost: ${output.execution_stats.plan_execution_cost:.3f}")
    print("---")
    print(f"Total time: {output.execution_stats.total_execution_time:.2f}s")
    print(f"Total Cost: ${output.execution_stats.total_execution_cost:.3f}")
    print("---")
    print("Final Plan:")
    print(final_plan_str)
```

--------------------------------

### BibTeX Citation for Palimpzest Project

Source: https://github.com/mitdbg/palimpzest/blob/main/README.md

This BibTeX entry provides the necessary information to cite the Palimpzest project in academic publications. It includes authors, title, publication venue, and date for the CIDR 2025 paper.

```BibTeX
@inproceedings{palimpzestCIDR,
    title={Palimpzest: Optimizing AI-Powered Analytics with Declarative Query Processing},
    author={Liu, Chunwei and Russo, Matthew and Cafarella, Michael and Cao, Lei and Chen, Peter Baile and Chen, Zui and Franklin, Michael and Kraska, Tim and Madden, Samuel and Shahout, Rana and Vitagliano, Gerardo},
    booktitle = {Proceedings of the {{Conference}} on {{Innovative Database Research}} ({{CIDR}})},
    date = 2025,
}
```

--------------------------------

### Perform Semantic Search on ChromaDB Index

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

Defines a Python function `search_func` that queries a ChromaDB collection with a list of query embeddings. It processes the raw results to calculate cosine similarity, sorts the results by similarity, removes duplicates, and returns the top-k most relevant reaction labels. This function encapsulates the core logic for retrieving and ranking medical terms based on semantic similarity.

```python
def search_func(index: chromadb.Collection, query: list[list[float]], k: int) -> list[str]:
    # execute query with embeddings
    results = index.query(query, n_results=5)

    # get list of result terms with their cosine similarity scores
    final_results = []
    for query_docs, query_distances in zip(results["documents"], results["distances"]):
        for doc, dist in zip(query_docs, query_distances):
            cosine_similarity = 1 - dist
            final_results.append({"content": doc, "similarity": cosine_similarity})

    # sort the results by similarity score
    sorted_results = sorted(final_results, key=lambda result: result["similarity"], reverse=True)

    # remove duplicates
    sorted_results_set = set()
    final_sorted_results = []
    for result in sorted_results:
        if result["content"] not in sorted_results_set:
            sorted_results_set.add(result["content"])
            final_sorted_results.append(result["content"])

    # return the top-k similar results and generation stats
    return {"reaction_labels": final_sorted_results[:k]}
```

--------------------------------

### Define Palimpzest Program Schemas for Medical Conditions

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

Defines three Python lists, each representing a schema for columns within a Palimpzest (PZ) program. These schemas specify the name, data type, and a descriptive explanation for different stages of medical condition processing: raw reactions, official reaction labels, and ranked reaction labels. These definitions are crucial for structuring data flow and computation steps within a PZ pipeline.

```python
# define the schema for each computation in our program
biodex_reactions_cols = [
    {"name": "reactions", "type": list[str], "desc": "The list of all medical conditions experienced by the patient as discussed in the report. Try to provide as many relevant medical conditions as possible."},
]
biodex_reaction_labels_cols = [
    {"name": "reaction_labels", "type": list[str], "desc": "Official terms for medical conditions listed in `reactions`"},
]
biodex_ranked_reactions_labels_cols = [
    {"name": "ranked_reaction_labels", "type": list[str], "desc": "The ranked list of medical conditions experienced by the patient. The most relevant label occurs first in the list. Be sure to rank ALL of the inputs."},
]
```

--------------------------------

### Define Real Estate Schema and Price Range UDF in Palimpzest

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

This snippet defines a schema for extracting 'address' and 'price' from real estate text content. It also includes a Python User-Defined Function (UDF) named `in_price_range` to filter records based on a specified price range, handling string-to-numeric conversion and cleaning of price data.

```python
real_estate_text_cols = [
    {"name": "address", "type": str, "desc": "The address of the property"},
    {"name": "price", "type": int | float, "desc": "The listed price of the property"},
]

def in_price_range(record: dict):
    try:
        price = record["price"]
        if isinstance(price, str):
            price = price.strip()
            price = int(price.replace("$", "").replace(",", ""))
        return 6e5 < price <= 2e6
    except Exception:
        return False
```

--------------------------------

### Palimpzest End-to-End Data Processing Example

Source: https://github.com/mitdbg/palimpzest/blob/main/docs/docs/index.md

This Python snippet demonstrates a complete Palimpzest workflow, from dataset registration and loading to schema definition, query construction, and execution. It showcases how to define a custom schema, apply filters, and run the optimized computation using a specified policy like `MinCost`.

```python
import pandas as pd
import palimpzest.datamanager.datamanager as pzdm
from palimpzest.sets import Dataset
from palimpzest.core.lib.fields import Field
from palimpzest.core.lib.schemas import Schema, TextFile
from palimpzest.policy import MinCost, MaxQuality
from palimpzest.query.processor.config import QueryProcessorConfig

# Dataset registration
dataset_path = "testdata/enron-tiny"
dataset_name = "enron-tiny"
pzdm.DataDirectory().register_local_directory(dataset_path, dataset_name)

# Dataset loading
dataset = Dataset(dataset_name, schema=TextFile)

# Schema definition for the fields we wish to compute
class Email(Schema):
    """Represents an email, which in practice is usually from a text file"""
    sender = Field(desc="The email address of the sender")
    subject = Field(desc="The subject of the email")
    date = Field(desc="The date the email was sent")

# Lazy construction of computation to filter for emails about holidays sent in July
dataset = dataset.convert(Email, desc="An email from the Enron dataset")
dataset = dataset.filter("The email was sent in July")
dataset = dataset.filter("The email is about holidays")

# Executing the compuation
policy = MinCost()
config = QueryProcessorConfig(
    policy=policy,
    verbose=True,
    processing_strategy="no_sentinel",
    execution_strategy="sequential",
    optimizer_strategy="pareto",
)
results, execution_stats = dataset.run(config)

# Writing output to disk
```

--------------------------------

### Define Palimpzest Semantic Email Filtering Pipeline

Source: https://github.com/mitdbg/palimpzest/blob/main/quickstart.ipynb

This Python snippet defines a lazy Palimpzest computation graph to filter Enron emails. It specifies columns to extract (sender, subject, date) using `sem_add_columns` and applies two natural language filters ('The email was sent in July', 'The email is about holidays') using `sem_filter`. This sets up the declarative pipeline without immediate execution.

```python
# define the fields we wish to compute
email_cols = [
    {"name": "sender", "type": str, "desc": "The email address of the sender"},
    {"name": "subject", "type": str, "desc": "The subject of the email"},
    {"name": "date", "type": str, "desc": "The date the email was sent"},
]

# lazily construct the computation to get emails about holidays sent in July
dataset = pz.Dataset("enron-tiny/")
dataset = dataset.sem_add_columns(email_cols)
dataset = dataset.sem_filter("The email was sent in July")
dataset = dataset.sem_filter("The email is about holidays")
```

--------------------------------

### Process and Print Palimpzest Dataset Tables

Source: https://github.com/mitdbg/palimpzest/blob/main/demos/biofabric-demo-matching.ipynb

This example demonstrates how to load a dataset using Palimpzest, semantically add columns based on predefined schemas, configure a query processor with a cost policy, and execute the query. It includes a helper function `print_tables` to display the processed data records, showing table names, headers, and a subset of rows.

```python
def print_tables(output):
    for table in output:
        header = table.header
        subset_rows = table.rows[:3]

        print("Table name:", table.name)
        print(" | ".join(header)[:100], "...")
        for row in subset_rows:
            print(" | ".join(row)[:100], "...")
        print()


xls = pz.Dataset("testdata/biofabric-tiny")
patient_tables = xls.sem_add_columns(table_cols, cardinality=pz.Cardinality.ONE_TO_MANY)
output = patient_tables

policy = pz.MinCost()
config = pz.QueryProcessorConfig(
    policy=policy,
    cache=False,
    processing_strategy="no_sentinel",
)
data_record_collection = output.run(config)

print_tables(data_record_collection.data_records)
```

--------------------------------

### Process Data with Palimpzest using MaxQuality Policy (GPT-4)

Source: https://github.com/mitdbg/palimpzest/blob/main/demos/biofabric-demo-matching.ipynb

This snippet illustrates a data processing pipeline similar to the minimum cost example, but configured for maximum quality. It uses the `palimpzest` library with a `MaxQuality` policy, likely utilizing a more capable model like GPT-4, to process data. The processed output is then converted into a Pandas DataFrame for display.

```python
xls = pz.Dataset("testdata/biofabric-tiny")
patient_tables = xls.sem_add_columns(table_cols, cardinality=pz.Cardinality.ONE_TO_MANY)
patient_tables = patient_tables.sem_filter("The table contains biometric information about the patient")
case_data = patient_tables.sem_add_columns(case_data_cols, cardinality=pz.Cardinality.ONE_TO_MANY)

policy = pz.MaxQuality()
config = pz.QueryProcessorConfig(
    policy=policy,
    cache=False,
    allow_code_synth=False,
    processing_strategy="streaming",
    execution_strategy="sequential",
)
iterable = case_data.run(config)

output_rows = []
for data_record_collection in iterable:  # noqa: B007
    for output_table in data_record_collection:
        print(output_table.to_dict().keys())
        output_rows.append(output_table.to_dict()) 

output_df = pd.DataFrame(output_rows)
display(output_df)
```

--------------------------------

### Execute Palimpzest Program with Optimization

Source: https://github.com/mitdbg/palimpzest/blob/main/docs/docs/getting-started/quickstart.md

Shows how to trigger the execution of a Palimpzest program using the `run()` method on a dataset. This method initiates the processing defined by semantic operators and supports various optimization objectives and optional constraints to configure the execution strategy.

```python
output = emails.run(max_quality=True)
```

--------------------------------

### Palimpzest Dataset Run Method Optimization Parameters

Source: https://github.com/mitdbg/palimpzest/blob/main/docs/docs/getting-started/quickstart.md

Documents the various optimization objectives and constraints that can be passed as keyword arguments to the `pz.Dataset.run()` method. These parameters allow users to configure the program's execution to prioritize factors like output quality, cost, or runtime, and to set specific budget or quality thresholds for the optimization process.

```APIDOC
pz.Dataset.run() method keyword arguments:

Optimization Objectives:
- max_quality=True: Boolean. When set to True, the optimizer prioritizes maximizing the output quality of the program.
- min_cost=True: Boolean. When set to True, the optimizer aims to minimize the financial cost of the program's execution.
- min_time=True: Boolean. When set to True, the optimizer seeks to minimize the total execution time of the program.

Constraints:
- quality_threshold=<float>: Float, range [0, 1]. An optional constraint specifying the minimum acceptable output quality. The optimizer will try to find a plan that meets or exceeds this threshold.
- cost_budget=<float>: Float. An optional constraint specifying the maximum allowable cost for the program's execution, in US Dollars.
- time_budget=<float>: Float. An optional constraint specifying the maximum allowable runtime for the program's execution, in seconds.
```