### Install Promptolution
Source: https://finitearth.github.io/promptolution/examples/getting_started
Installs the Promptolution library with API support using pip. This is the initial step to begin using the library's functionalities.
```bash
! pip install promptolution[api]
```
--------------------------------
### Set Promptolution API Key
Source: https://finitearth.github.io/promptolution/examples/getting_started
This snippet shows how to define and assign your Promptolution API key, which is necessary for authentication when interacting with the service.
```Python
api_key = "YOUR_API_KEY" # Replace with your Promptolution API key
```
--------------------------------
### Import Promptolution Modules
Source: https://finitearth.github.io/promptolution/examples/getting_started
Imports necessary modules from the Promptolution library, including ExperimentConfig and run_experiment, along with nest_asyncio for notebook compatibility. These are essential for setting up and running experiments.
```python
import pandas as pd
from promptolution.utils import ExperimentConfig
from promptolution.helpers import run_experiment
import nest_asyncio
nest_asyncio.apply() # Required for notebook environments
```
--------------------------------
### Prepare Data for Experiment
Source: https://finitearth.github.io/promptolution/examples/getting_started
Loads and preprocesses a dataset for use in a Promptolution experiment. It renames columns to 'x' and 'y', replaces labels, and defines a task description for the meta-LLM.
```python
df = pd.read_csv("hf://datasets/tasksource/subjectivity/train.csv").sample(500)
df = df.rename(columns={"Sentence": "x", "Label": "y"})
df = df.replace({"OBJ": "objective", "SUBJ": "subjective"})
task_description = (
"The dataset contains sentences labeled as either subjective or objective. "
"The task is to classify each sentence as either subjective or objective. "
"The class mentioned in between the answer tags ", self._format_instructions()).replace(
"", self._sample_examples()
)
```
--------------------------------
### Configure Experiment with ExperimentConfig
Source: https://finitearth.github.io/promptolution/examples/getting_started
This Python code demonstrates how to instantiate the `ExperimentConfig` class with specific parameters for prompt optimization. It includes settings for the optimizer, task description, initial prompts, number of steps, API URL, model ID, and API key.
```Python
config = ExperimentConfig(
optimizer="capo",
task_description=task_description,
prompts=init_prompts,
n_steps=10,
api_url="https://api.deepinfra.com/v1/openai",
model_id="meta-llama/Meta-Llama-3-8B-Instruct",
api_key=api_key,
n_subsamples=30,
)
```
--------------------------------
### Display Optimized Prompts and Scores
Source: https://finitearth.github.io/promptolution/examples/getting_started
Displays the results of the prompt optimization, showing the optimized prompt text and its corresponding performance score. This table illustrates the variations in performance among semantically similar prompts.
```python
prompts
```
--------------------------------
### Run Prompt Optimization Experiment
Source: https://finitearth.github.io/promptolution/examples/getting_started
Executes the prompt optimization process using the `run_experiment` function. This function takes a DataFrame and a configuration object as input to run the optimization and evaluate the results on a holdout set. The process may take a few minutes to complete.
```python
prompts = run_experiment(df, config)
```
--------------------------------
### Install Promptolution using pip
Source: https://finitearth.github.io/promptolution/index
This snippet shows how to install the Promptolution library using the pip package manager. Ensure you have Python and pip installed.
```Shell
pip install promptolution
```
--------------------------------
### CAPOPrompt Initialization
Source: https://finitearth.github.io/promptolution/api/optimizers
Initializes the CAPOPrompt with instruction text and a list of few-shot examples. It stores the instruction and examples, cleaning up whitespace.
```Python
class CAPOPrompt:
"""Represents a prompt consisting of an instruction and few-shot examples."""
def __init__(self, instruction_text: str, few_shots: List[str]):
"""Initializes the Prompt with an instruction and associated examples.
Args:
instruction_text (str): The instruction or prompt text.
few_shots (List[str]): List of examples as string.
"""
self.instruction_text = instruction_text.strip()
self.few_shots = few_shots
```
--------------------------------
### Label Text: Personal vs. Factual
Source: https://finitearth.github.io/promptolution/examples/getting_started
Scrutinizes provided text to classify them as either vastly personal ('subjective') or dispassionately factual ('objective') based on the presence of opinions, biases, or verifiable information. Outputs the classification within tags.
```English
A labeling exercise necessitates scrutinizing provided text to classify them as either vastly personal ('subjective') or dispassionately factual ('objective') based on the presence of opinions, biases, or verifiable information. Your mission is to accurately determine whether the supplied sentence leans more towards subjective expression of personal thought or objective presentation of facts, then output the corresponding classification within the format ", objective "). Recognize that subjective sentences usually embody the writer's own views or emotions, whereas objective sentences present data without personal investment or allegiance. The predicted outcome will be the one first mentioned in the response, and the extracted class label will be positioned between the markers and , which can only be one of the two categories: subjective or objective.
Input:
The last may go very deep.
Output:
objective
```
--------------------------------
### Python: Define Initial Prompts for Text Classification
Source: https://finitearth.github.io/promptolution/examples/getting_started
This Python code defines a list of strings, where each string is a prompt designed to classify a given text as either objective or subjective. These prompts vary in their instructions regarding tone, language analysis, and output format.
```python
init_prompts = [
'Classify the given text as either an objective or subjective statement based on the tone and language used: e.g. the tone and language used should indicate whether the statement is a neutral, factual summary (objective) or an expression of opinion or emotional tone (subjective). Include the output classes "objective" or "subjective" in the prompt.',
"What kind of statement is the following text: [Insert text here]? Is it or ?",
'Identify whether a sentence is objective or subjective by analyzing the tone, language, and underlying perspective. Consider the emotion, opinion, and bias present in the sentence. Are the authors presenting objective facts or expressing a personal point of view? The output will be either "objective" (output class: objective) or "subjective" (output class: subjective).',
"Classify the following sentences as either objective or subjective, indicating the name of the output classes: [input sentence]. Output classes: objective, subjective",
'_query a text about legal or corporate-related issues, and predict whether the tone is objective or subjective, outputting the corresponding class "objective" for non-subjective language or "subjective" for subjective language_',
'Classify a statement as either "subjective" or "objective" based on whether it reflects a personal opinion or a verifiable fact. The output classes to include are "objective" and "subjective".',
"Classify the text as objective or subjective based on its tone and language.",
"Classify the text as objective or subjective based on the presence of opinions or facts. Output classes: objective, subjective.",
"Classify the given text as objective or subjective based on its tone, focusing on its intention, purpose, and level of personal opinion or emotional appeal, with outputs including classes such as objective or subjective.",
"Categorize the text as either objective or subjective, considering whether it presents neutral information or expresses a personal opinion/bias.\n\nObjective: The text has a neutral tone and presents factual information about the actions of Democrats in Congress and the union's negotiations.\n\nSubjective: The text has a evaluative tone and expresses a positive/negative opinion/evaluation about the past performance of the country.",
'Given a sentence, classify it as either "objective" or "subjective" based on its tone and language, considering the presence of third-person pronouns, neutral language, and opinions. Classify the output as "objective" if the tone is neutral and detached, focusing on facts and data, or as "subjective" if the tone is evaluative, emotive, or biased.',
'Identify whether the given sentence is subjective or objective, then correspondingly output "objective" or "subjective" in the form of ", (e.g. "objective"), without quotes. Please note that the subjective orientation typically describes a sentence where the writer expresses their own opinion or attitude, whereas an objective sentence presents facts or information without personal involvement or bias. '
]
```
--------------------------------
### Abstract Method for Pre-Optimization Setup
Source: https://finitearth.github.io/promptolution/api/optimizers
Defines the abstract method `_pre_optimization_loop` which concrete optimizer classes must implement to perform any necessary setup before the optimization loop begins.
```Python
@abstractmethod
def_pre_optimization_loop(self):
"""Prepare for the optimization loop.
This method should be implemented by concrete optimizer classes to define
any setup required before the optimization loop starts.
"""
pass
```
--------------------------------
### Create Few-Shot Examples (Python)
Source: https://finitearth.github.io/promptolution/api/optimizers
Generates few-shot examples for a given instruction by sampling from a DataFrame. It creates example pairs and optionally uses a predictor model to generate reasoning for correct predictions.
```Python
def _create_few_shot_examples(self, instruction: str, num_examples: int) -> List[Tuple[str, str]]:
if num_examples == 0:
return []
few_shot_samples = self.df_few_shots.sample(num_examples, replace=False)
sample_inputs = few_shot_samples[self.task.x_column].values
sample_targets = few_shot_samples[self.task.y_column].values
few_shots = [
CAPO_FEWSHOT_TEMPLATE.replace("", f"{self.target_begin_marker}{t}{self.target_end_marker}"
)
for i, t in zip(sample_inputs, sample_targets)
]
# Select partition of the examples to generate reasoning from downstream model
preds, seqs = self.predictor.predict(
[instruction] * num_examples,
sample_inputs,
return_seq=True,
)
# Check which predictions are correct and get a single one per example
for j in range(num_examples):
# Process and clean up the generated sequences
seqs[j] = seqs[j].replace(sample_inputs[j], "").strip()
# Check if the prediction is correct and add reasoning if so
if preds[j] == sample_targets[j]:
few_shots[j] = CAPO_FEWSHOT_TEMPLATE.replace("", seqs[j])
return few_shots
```
--------------------------------
### Initialize OPRO Optimizer
Source: https://finitearth.github.io/promptolution/api/optimizers
Initializes the OPRO optimizer with a predictor, task, meta-LLM, and optional parameters like prompt templates, number of instructions, few-shot examples, and callbacks. It sets up the meta-LLM and prompt template, and calls the parent class initializer.
```Python
class OPRO(BaseOptimizer):
"""OPRO: Optimization by PROmpting.
Implementation of the technique proposed in "Large Language Models as Optimizers"
(Yang et al., 2023: https://arxiv.org/abs/2309.03409).
OPRO works by providing a meta-LLM with task descriptions and previous
prompt-score pairs to generate improved prompts for a downstream LLM.
"""
def __init__(
self,
predictor: "BasePredictor",
task: "BaseTask",
prompt_template: Optional[str],
meta_llm: "BaseLLM",
initial_prompts: List[str] = None,
max_num_instructions: int = 20,
num_instructions_per_step: int = 8,
num_few_shots: int = 3,
callbacks: List["BaseCallback"] = None,
config: "ExperimentConfig" = None,
) -> None:
"""Initialize the OPRO optimizer.
Args:
predictor: Predictor for prompt evaluation
task: Task object for prompt evaluation
meta_llm: LLM that generates improved prompts
initial_prompts: Initial set of prompts to start optimization with
prompt_template: Custom meta prompt template (uses OPRO_TEMPLATE if None)
max_num_instructions: Maximum previous instructions to include in meta prompt
num_instructions_per_step: Number of prompts to generate in each step
num_few_shots: Number of few-shot examples to include (0 for none)
callbacks: List of callback functions
config: "ExperimentConfig" overwriting default parameters
"""
self.meta_llm = meta_llm
self.meta_prompt_template = prompt_template if prompt_template else OPRO_TEMPLATE
self.max_num_instructions = max_num_instructions
self.num_instructions_per_step = num_instructions_per_step
self.num_few_shots = num_few_shots
super().__init__(
predictor=predictor, task=task, initial_prompts=initial_prompts, callbacks=callbacks, config=config
)
```
--------------------------------
### Initialize Prompt Population (Python)
Source: https://finitearth.github.io/promptolution/api/optimizers
Initializes a population of CAPOPrompt objects from a list of initial instructions. It generates few-shot examples for each prompt, incorporating random numbers of examples and using a predefined template.
```Python
def _initialize_population(self, initial_prompts: List[str]) -> List[CAPOPrompt]:
"""Initializes the population of Prompt objects from initial instructions.
Args:
initial_prompts (List[str]): List of initial prompt instructions.
Returns:
List[Prompt]: Initialized population of prompts with few-shot examples.
"""
population = []
for instruction_text in initial_prompts:
num_examples = random.randint(0, self.upper_shots)
few_shots = self._create_few_shot_examples(instruction_text, num_examples)
population.append(CAPOPrompt(instruction_text, few_shots))
return population
```
--------------------------------
### Initialize CAPO Optimizer
Source: https://finitearth.github.io/promptolution/api/optimizers
Initializes the CAPO optimizer with parameters for prompt evolution, including predictor, task, meta LLM, and optimization settings like crossover rate, few-shot examples, and statistical tests.
```Python
class CAPO(BaseOptimizer):
"""CAPO: Cost-Aware Prompt Optimization.
This class implements an evolutionary algorithm for optimizing prompts in large language models
by incorporating racing techniques and multi-objective optimization. It uses crossover, mutation,
and racing based on evaluation scores and statistical tests to improve efficiency while balancing
performance with prompt length. It is adapted from the paper "CAPO: Cost-Aware Prompt Optimization" by Zehle et al., 2025.
"""
def __init__(
self,
predictor: "BasePredictor",
task: "BaseTask",
meta_llm: "BaseLLM",
initial_prompts: List[str] = None,
crossovers_per_iter: int = 4,
upper_shots: int = 5,
max_n_blocks_eval: int = 10,
test_statistic: "TestStatistics" = "paired_t_test",
alpha: float = 0.2,
length_penalty: float = 0.05,
df_few_shots: pd.DataFrame = None,
crossover_template: str = None,
mutation_template: str = None,
callbacks: List[Callable] = [],
config: "ExperimentConfig" = None,
):
"""Initializes the CAPOptimizer with various parameters for prompt evolution.
Args:
predictor (BasePredictor): The predictor for evaluating prompt performance.
task (BaseTask): The task instance containing dataset and description.
meta_llm (BaseLLM): The meta language model for crossover/mutation.
initial_prompts (List[str]): Initial prompt instructions.
crossovers_per_iter (int): Number of crossover operations per iteration.
upper_shots (int): Maximum number of few-shot examples per prompt.
p_few_shot_reasoning (float): Probability of generating llm-reasoning for few-shot examples, instead of simply using input-output pairs.
max_n_blocks_eval (int): Maximum number of evaluation blocks.
test_statistic (TestStatistics): Statistical test to compare prompt performance. Default is "paired_t_test".
alpha (float): Significance level for the statistical test.
length_penalty (float): Penalty factor for prompt length.
df_few_shots (pd.DataFrame): DataFrame containing few-shot examples. If None, will pop 10% of datapoints from task.
crossover_template (str, optional): Template for crossover instructions.
mutation_template (str, optional): Template for mutation instructions.
callbacks (List[Callable], optional): Callbacks for optimizer events.
config (ExperimentConfig, optional): Configuration for the optimizer.
"""
self.meta_llm = meta_llm
self.downstream_llm = predictor.llm
self.crossover_template = crossover_template or CAPO_CROSSOVER_TEMPLATE
self.mutation_template = mutation_template or CAPO_MUTATION_TEMPLATE
self.crossovers_per_iter = crossovers_per_iter
self.upper_shots = upper_shots
self.max_n_blocks_eval = max_n_blocks_eval
self.test_statistic = get_test_statistic_func(test_statistic)
self.alpha = alpha
self.length_penalty = length_penalty
self.token_counter = get_token_counter(self.downstream_llm)
self.scores = np.empty(0)
super().__init__(predictor, task, initial_prompts, callbacks, config)
```
--------------------------------
### Sample Few-Shot Examples
Source: https://finitearth.github.io/promptolution/api/optimizers
Samples a specified number of few-shot examples from the task's dataset. It randomly selects inputs and their corresponding outputs to be formatted into a string for inclusion in the meta-prompt.
```Python
def _sample_examples(self) -> str:
"""Sample few-shot examples from the dataset.
Returns:
Formatted string of few-shot examples with inputs and expected outputs
"""
idx = np.random.choice(len(self.task.xs), self.num_few_shots)
sample_x = self.task.xs[idx]
sample_y = self.task.ys[idx]
return "\n".join([f"Input: {x}\nOutput: {y}" for x, y in zip(sample_x, sample_y)])
```
--------------------------------
### Classify Sentence Tone (Objective/Subjective)
Source: https://finitearth.github.io/promptolution/examples/getting_started
This prompt classifies sentences as either objective or subjective by analyzing their linguistic tone, intent, and purpose. It considers whether the text presents a neutral, factual account or expresses personal opinion or emotional bias. The output includes the classification label extracted from the text between and markers.
```text
Classify each sentence as either objective or subjective by examining its linguistic tone, underlying intent, and purpose. Consider whether the text presents a neutral, factual account or expresses a personal opinion or emotional bias. Evaluate whether the text is neutral and provides mere reportage, such as a factual report on congressional Democrats' actions and labor union negotiations, or if it reveals an evaluative tone, offering a positive or negative appraisal of a nation's past performance. Outputs will include classifications like objective or subjective. The class mentioned first in the response will serve as the prediction, with the class label extracted from the text between the markers and .
Input:
Over several decades, Prime Central London – or PCL – had become a repository for cash from wealthy foreigners, whether they actually wanted to live there or not.
Output:
objective
Input:
Faced with a tighter labor market, many districts are raising base salaries and offering signing and relocation bonuses — up to a whopping $25,000 in one New Mexico school district.
Output:
objective
Input:
That when liquidation of commodities and securities has gone too far it becomes the business of government to stop it, using public credit by such means as it may think fit.
Output:
subjective
```
--------------------------------
### Initialize OPRO Optimizer
Source: https://finitearth.github.io/promptolution/api/optimizers
Initializes the OPRO optimizer with essential components like a predictor, task, and meta LLM. It also configures parameters for prompt generation, including the number of instructions and few-shot examples, and allows for custom prompt templates and callbacks.
```Python
def __init__(
self,
predictor: "BasePredictor",
task: "BaseTask",
prompt_template: Optional[str],
meta_llm: "BaseLLM",
initial_prompts: List[str] = None,
max_num_instructions: int = 20,
num_instructions_per_step: int = 8,
num_few_shots: int = 3,
callbacks: List["BaseCallback"] = None,
config: "ExperimentConfig" = None,
) -> None:
"""Initialize the OPRO optimizer.
Args:
predictor: Predictor for prompt evaluation
task: Task object for prompt evaluation
meta_llm: LLM that generates improved prompts
initial_prompts: Initial set of prompts to start optimization with
prompt_template: Custom meta prompt template (uses OPRO_TEMPLATE if None)
max_num_instructions: Maximum previous instructions to include in meta prompt
num_instructions_per_step: Number of prompts to generate in each step
num_few_shots: Number of few-shot examples to include (0 for none)
callbacks: List of callback functions
config: "ExperimentConfig" overwriting default parameters
"""
self.meta_llm = meta_llm
self.meta_prompt_template = prompt_template if prompt_template else OPRO_TEMPLATE
self.max_num_instructions = max_num_instructions
self.num_instructions_per_step = num_instructions_per_step
self.num_few_shots = num_few_shots
super().__init__(
predictor=predictor, task=task, initial_prompts=initial_prompts, callbacks=callbacks, config=config
)
```
--------------------------------
### CAPOPrompt Prompt Construction
Source: https://finitearth.github.io/promptolution/api/optimizers
Constructs the full prompt string by replacing placeholders in a template with the instruction text and formatted few-shot examples. It also handles extra newlines if no few-shot examples are provided.
```Python
def construct_prompt(self) -> str:
"""Constructs the full prompt string by replacing placeholders in the template with the instruction and formatted examples.
Returns:
str: The constructed prompt string.
"""
few_shot_str = "\n\n".join(self.few_shots).strip()
prompt = (
CAPO_DOWNSTREAM_TEMPLATE.replace("", self.instruction_text)
.replace("", few_shot_str)
.replace("\n\n\n\n", "\n\n") # replace extra newlines if no few shots are provided
.strip()
)
return prompt
```
--------------------------------
### Optimize Prompts with BaseOptimizer
Source: https://finitearth.github.io/promptolution/api/optimizers
Executes the prompt optimization process for a specified number of steps. It includes pre-optimization setup, step-by-step optimization, and post-optimization callbacks, with error handling for individual steps.
```Python
defoptimize(self, n_steps: int) -> List[str]:
"""Perform the optimization process.
This method should be implemented by concrete optimizer classes to define
the specific optimization algorithm.
Args:
n_steps (int): Number of optimization steps to perform.
Returns:
The optimized list of prompts after all steps.
"""
# validate config
if self.config is not None:
self.config.validate()
self._pre_optimization_loop()
for _ in range(n_steps):
try:
self.prompts = self._step()
except Exception as e:
# exit training loop and gracefully fail
logger.error(f"⛔ Error during optimization step: {e}")
logger.error("⚠️ Exiting optimization loop.")
continue_optimization = False
# Callbacks at the end of each step
continue_optimization = self._on_step_end()
if not continue_optimization:
break
self._on_train_end()
return self.prompts
```
--------------------------------
### Promptolution Local LLM Initialization
Source: https://finitearth.github.io/promptolution/api/llms
Illustrates the initialization of the LocalLLM class, designed for interacting with language models that run locally. This class likely handles the setup and management of local model instances.
```Python
from promptolution.llms.local_llm import LocalLLM
# Example initialization (assuming a model path or name is provided)
# local_llm_instance = LocalLLM(model_path="/path/to/local/model")
```
--------------------------------
### Setup Logging for Promptolution
Source: https://finitearth.github.io/promptolution/api/utils
Configures the root logger for the promptolution library with a specified logging level and a standard message format. It uses Python's built-in `logging` module. The function takes an optional `level` parameter, defaulting to `logging.INFO`.
```Python
import logging
def setup_logging(level: int = logging.INFO) -> None:
"""Set up logging for the promptolution library.
This function configures the root logger for the library with appropriate
formatting and level.
Args:
level (int, optional): Logging level. Defaults to logging.INFO.
"""
# Configure the root logger
logging.basicConfig(
level=level,
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
)
```
--------------------------------
### Implement select_exemplars Method (Python)
Source: https://finitearth.github.io/promptolution/api/exemplar_selectors
Defines the abstract method `select_exemplars` for selecting prompt exemplars. Subclasses must implement this to provide specific selection logic. It takes a prompt and the desired number of examples, returning a new prompt with selected exemplars.
```python
@abstractmethod
defselect_exemplars(self, prompt: str, n_examples: int = 5) -> str:
"""Select exemplars based on the given prompt.
Args:
prompt (str): The input prompt to base the exemplar selection on.
n_examples (int, optional): The number of exemplars to select. Defaults to 5.
Returns:
str: A new prompt that includes the original prompt and the selected exemplars.
Raises:
NotImplementedError: This method should be implemented by subclasses.
"""
raise NotImplementedError("This method should be implemented by subclasses.")
```
--------------------------------
### Generate Prompts from Samples - Python
Source: https://finitearth.github.io/promptolution/api/utils
Generates a set of prompts from dataset examples sampled from a given task. It supports uniform label sampling for classification tasks and random sampling for other tasks. The function takes a task, a language model, and optional parameters for meta-prompting and sampling.
```python
def create_prompts_from_samples(
task: "BaseTask",
llm: "BaseLLM",
meta_prompt: str = None,
n_samples: int = 3,
task_description: str = None,
n_prompts: int = 1,
get_uniform_labels: bool = False,
) -> List[str]:
"""Generate a set of prompts from dataset examples sampled from a given task.
Idea taken from the paper Zhou et al. (2021) https://arxiv.org/pdf/2211.01910
Samples are selected, such that
(1) all possible classes are represented
(2) the samples are as representative as possible
Args:
task (BaseTask): The task to generate prompts for.
Xs and Ys from this object are used to generate the prompts.
llm (BaseLLM): The language model to use for generating the prompts.
meta_prompt (str): The meta prompt to use for generating the prompts.
If None, a default meta prompt is used.
n_samples (int): The number of samples to use for generating prompts.
task_description (str): The description of the task to include in the prompt.
n_prompts (int): The number of prompts to generate.
get_uniform_labels (bool): If True, samples are selected such that all classes are represented.
Returns:
List[str]: A list of generated prompts.
"""
if meta_prompt is None and task_description is None:
meta_prompt_template = PROMPT_CREATION_TEMPLATE
elif meta_prompt is None and task_description is not None:
meta_prompt_template = PROMPT_CREATION_TEMPLATE_TD.replace("", task_description)
elif meta_prompt is not None and task_description is None:
meta_prompt_template = meta_prompt
elif meta_prompt is not None and task_description is not None:
meta_prompt_template = meta_prompt.replace("", task_description)
meta_prompts = []
for _ in range(n_prompts):
if isinstance(task, ClassificationTask) and get_uniform_labels:
# if classification task sample such that all classes are represented
unique_labels, counts = np.unique(task.ys, return_counts=True)
proportions = counts / len(task.ys)
samples_per_class = np.round(proportions * n_samples).astype(int)
samples_per_class = np.maximum(samples_per_class, 1)
# sample
xs = []
ys = []
for label, n_samples in zip(unique_labels, samples_per_class):
indices = np.where(task.ys == label)[0]
indices = np.random.choice(indices, n_samples, replace=False)
xs.extend(task.xs[indices])
ys.extend(task.ys[indices])
else:
# if not classification task, sample randomly
indices = np.random.choice(len(task.xs), n_samples, replace=False)
xs = task.xs[indices].tolist()
ys = task.ys[indices].tolist()
examples = "\n\n".join([f"Input: {x}\nOutput: {y}" for x, y in zip(xs, ys)])
meta_prompt = meta_prompt_template.replace("", examples)
meta_prompts.append(meta_prompt)
prompts = llm.get_response(meta_prompts)
prompts = [prompt.split("")[0].split("")[-1].strip() for prompt in prompts]
return prompts
```
--------------------------------
### Initialize APILLM with Configuration
Source: https://finitearth.github.io/promptolution/api/llms
Initializes the APILLM with specific model and API configurations. It requires parameters like API URL, model ID, API key, and settings for concurrent calls and token limits. The initialization may raise an ImportError if necessary libraries are not installed.
```Python
def __init__(
self,
api_url: str = None,
model_id: str = None,
api_key: str = None,
max_concurrent_calls=50,
max_tokens=512,
config: "ExperimentConfig" = None,
):
"""Initialize the APILLM with a specific model and API configuration.
Args:
api_url (str): The base URL for the API endpoint.
model_id (str): Identifier for the model to use.
api_key (str, optional): API key for authentication. Defaults to None.
max_concurrent_calls (int, optional): Maximum number of concurrent API calls. Defaults to 50.
max_tokens (int, optional): Maximum number of tokens in model responses. Defaults to 512.
config (ExperimentConfig, optional): Configuration for the LLM, overriding defaults.
Raises:
ImportError: If required libraries are not installed.
"""
if not import_successful:
raise ImportError(
"Could not import at least one of the required libraries: openai, asyncio. "
"Please ensure they are installed in your environment."
)
self.api_url = api_url
self.model_id = model_id
self.api_key = api_key
self.max_concurrent_calls = max_concurrent_calls
self.max_tokens = max_tokens
super().__init__(config=config)
self.client = AsyncOpenAI(base_url=self.api_url, api_key=self.api_key)
self.semaphore = asyncio.Semaphore(self.max_concurrent_calls)
```
--------------------------------
### Classify Statement: Subjective or Objective
Source: https://finitearth.github.io/promptolution/examples/getting_started
Classifies a statement as either 'subjective' or 'objective' based on whether it reflects a personal opinion or a verifiable fact. The output classes to include are 'objective' and 'subjective'.
```English
Classify a statement as either "subjective" or "objective" based on whether it reflects a personal opinion or a verifiable fact. The output classes to include are "objective" and "subjective".
Input:
The promotion of it for many is an avocation, for increasing numbers it is a profession, and for a very great number of more or less trained men and women it is employment and livelihood.
Output:
objective
```
--------------------------------
### Classify Sentence: Objective vs. Subjective
Source: https://finitearth.github.io/promptolution/examples/getting_started
Classifies each sentence as objective or subjective based on language characteristics, identifying whether it presents neutral information or expresses a personal opinion. The output is the predicted class enclosed in tags.
```English
Classify each sentence as objective or subjective by recognizing its language characteristics. Identify whether each sentence presents neutral information or expresses a personal opinion. If the sentence provides factual information without taking a bias, classify it as objective. Conversely, if the sentence conveys the author's perspective, emotions, or beliefs, label it as subjective. As our language model expert, carefully analyze each sentence, extracting its tone, and determine whether it presents verifiable data or the author's biased thoughts. For instance, compare a factual news report on politics to a blog post about a historical event and highlight the differences between objective and subjective writing. Our output will be the predicted class enclosed within the markers and , with the first-mentioned class being the predicted label.
Input:
“This latest rule will open our borders even more, and the Court seems to relish making arbitrary decisions without thinking about consequences.
Output:
objective
```
--------------------------------
### Python: Initialize Config with keyword arguments
Source: https://finitearth.github.io/promptolution/api/utils
Initializes the configuration object with provided keyword arguments. It sets up a set to track used attributes and assigns each keyword argument as an attribute of the configuration object.
```Python
def__init__(self, **kwargs):
"""Initialize the configuration with the provided keyword arguments."""
self._used_attributes: Set[str] = set()
for key, value in kwargs.items():
setattr(self, key, value)
```
--------------------------------
### Categorize Text: Objective vs. Subjective
Source: https://finitearth.github.io/promptolution/examples/getting_started
Categorizes text as objective or subjective, considering whether it presents neutral information or expresses a personal opinion/bias. Objective text has a neutral tone and factual information, while subjective text has an evaluative tone and expresses opinions.
```English
Categorize the text as either objective or subjective, considering whether it presents neutral information or expresses a personal opinion/bias.
Objective: The text has a neutral tone and presents factual information about the actions of Democrats in Congress and the union's negotiations.
Subjective: The text has a evaluative tone and expresses a positive/negative opinion/evaluation about the past performance of the country.
Input:
Over several decades, Prime Central London – or PCL – had become a repository for cash from wealthy foreigners, whether they actually wanted to live there or not.
Output:
objective
```
--------------------------------
### Classify Sentences: Fact vs. Opinion
Source: https://finitearth.github.io/promptolution/examples/getting_started
Classifies a collection of labeled sentences as either based on fact or reflecting personal opinion, using linguistic features to distinguish between objective statements and subjective expressions. The objective class is denoted by objective and the subjective class by subjective .
```English
Classify a collection of labeled sentences as either based on fact or reflecting personal opinion, using linguistic features to distinguish between objective statements presenting verifiable information and subjective expressions of opinion or attitude, with the objective class being denoted by objective and the subjective class by subjective , where the first-mentioned class in the response will serve as the predicted outcome.
Input:
The principal reason, from the point of view of government, is that a universal income tax would be a powerful restraint upon the expansion of government.
Output:
objective
```
--------------------------------
### Initialize BaseOptimizer
Source: https://finitearth.github.io/promptolution/api/optimizers
Initializes the BaseOptimizer with a predictor, task, initial prompts, optional callbacks, and configuration. It sets up the optimizer's state and applies configuration settings if provided.
```Python
class BaseOptimizer(ABC):
"""Abstract base class for prompt optimizers.
This class defines the basic structure and interface for prompt optimization algorithms.
Attributes:
config (ExperimentConfig, optional): Configuration for the optimizer, overriding defaults.
prompts (List[str]): List of current prompts being optimized.
task (BaseTask): The task object used for evaluating prompts.
callbacks (List[Callable]): List of callback functions to be called during optimization.
predictor: The predictor used for prompt evaluation (if applicable).
"""
def__init__(
self,
predictor,
task: "BaseTask",
initial_prompts: List[str],
callbacks: List[Callable] = None,
config: "ExperimentConfig" = None,
):
"""Initialize the optimizer with a configuration and/or direct parameters.
Args:
initial_prompts: Initial set of prompts to start optimization with.
task: Task object for prompt evaluation.
callbacks: List of callback functions.
predictor: Predictor for prompt evaluation.
config (ExperimentConfig, optional): Configuration for the optimizer, overriding defaults.
"""
# Set up optimizer state
self.prompts = initial_prompts
self.task = task
self.callbacks = callbacks or []
self.predictor = predictor
if config is not None:
config.apply_to(self)
self.config = config
```
--------------------------------
### Mutate Prompts with Few-Shot Examples (Python)
Source: https://finitearth.github.io/promptolution/api/optimizers
Mutates a list of prompts by either adding a random few-shot example, removing a random few-shot example, or shuffling existing few-shot examples. This function is part of an evolutionary optimization process for prompts.
```Python
new_instructions = self.meta_llm.get_response(mutation_prompts)
mutated = []
for new_instruction, prompt in zip(new_instructions, offsprings):
new_instruction = new_instruction.split("")[-1].split(" ")[0].strip()
p = random.random()
if p < 1 / 3 and len(prompt.few_shots) < self.upper_shots: # add a random few shot
new_few_shot = self._create_few_shot_examples(new_instruction, 1)
new_few_shots = prompt.few_shots + new_few_shot
if 1 / 3 <= p < 2 / 3 and len(prompt.few_shots) > 0: # remove a random few shot
new_few_shots = random.sample(prompt.few_shots, len(prompt.few_shots) - 1)
else: # do not change few shots, but shuffle
new_few_shots = prompt.few_shots
random.shuffle(new_few_shots)
mutated.append(CAPOPrompt(new_instruction, new_few_shots))
return mutated
```
--------------------------------
### Initialize EvoPromptGA Optimizer
Source: https://finitearth.github.io/promptolution/api/optimizers
Initializes the EvoPromptGA optimizer with essential components like a predictor, task, prompt template, and a meta-LLM. It supports different selection modes for parent prompts and allows for optional callbacks and configuration.
```Python
def __init__(
self,
predictor: "BasePredictor",
task: "BaseTask",
prompt_template: str,
meta_llm: "BaseLLM",
initial_prompts: List[str] = None,
selection_mode: str = "wheel",
callbacks: List["BaseCallback"] = None,
config: "ExperimentConfig" = None,
):
"""Initialize the EvoPromptGA optimizer."""
self.prompt_template = prompt_template
self.meta_llm = meta_llm
self.selection_mode = selection_mode
super().__init__(
predictor=predictor, initial_prompts=initial_prompts, task=task, callbacks=callbacks, config=config
)
assert self.selection_mode in ["random", "wheel", "tour"], "Invalid selection mode."
```
--------------------------------
### Classify Sentence Tone (Objective/Subjective) - Nuanced
Source: https://finitearth.github.io/promptolution/examples/getting_started
Given a sentence, this prompt classifies it as either "objective" or "subjective" based on its tone and language, considering factors like third-person pronouns, neutral language, and opinions. It assigns "objective" if the tone is neutral and detached, focusing on facts, or "subjective" if the tone is evaluative, emotive, or biased.
```text
Given a sentence, classify it as either "objective" or "subjective" based on its tone and language, considering the presence of third-person pronouns, neutral language, and opinions. Classify the output as "objective" if the tone is neutral and detached, focusing on facts and data, or as "subjective" if the tone is evaluative, emotive, or biased.
Input:
Transportation Secretary Pete Buttigieg confirmed to The Associated Press on Thursday that $104.6 million in federal funds coming from last year’s bipartisan infrastructure bill will go toward a plan to dismantle Interstate 375, a highway built to bisect Detroit’s Black Bottom neighborhood and its epicenter of Black business, Paradise Valley.
Output:
objective
Input:
“This latest rule will open our borders even more, and the Court seems to relish making arbitrary decisions without thinking about consequences.
Output:
objective
Input:
He is fairly secure.
Output:
objective
Input:
In a recent report on the “new poor,” made by the Welfare Council of New York City, there is a reference to “the mental infection of dependency.” This was upon the investigation of unemployment relief.
Output:
objective
```
--------------------------------
### Initialize EvoPromptGA Optimizer
Source: https://finitearth.github.io/promptolution/api/optimizers
Initializes the EvoPromptGA optimizer with a predictor, task, prompt template, meta LLM, and optional parameters like initial prompts, selection mode, callbacks, and configuration. It validates the selection mode upon initialization.
```Python
class EvoPromptGA(BaseOptimizer):
"""EvoPromptGA: Genetic Algorithm-based Prompt Optimizer.
This class implements a genetic algorithm for optimizing prompts in large language models.
It is adapted from the paper "Connecting Large Language Models with Evolutionary Algorithms
Yields Powerful Prompt Optimizers" by Guo et al., 2023.
The optimizer uses crossover operations to generate new prompts from existing ones,
with different selection methods available for choosing parent prompts.
Attributes:
prompt_template (str): Template for generating meta-prompts during crossover.
meta_llm: Language model used for generating child prompts from meta-prompts.
selection_mode (str): Method for selecting parent prompts ('random', 'wheel', or 'tour').
Args:
prompt_template (str): Template for meta-prompts.
meta_llm: Language model for child prompt generation.
selection_mode (str, optional): Parent selection method. Defaults to "wheel".
Raises:
AssertionError: If an invalid selection mode is provided.
"""
def __init__(
self,
predictor: "BasePredictor",
task: "BaseTask",
prompt_template: str,
meta_llm: "BaseLLM",
initial_prompts: List[str] = None,
selection_mode: str = "wheel",
callbacks: List["BaseCallback"] = None,
config: "ExperimentConfig" = None,
):
"""Initialize the EvoPromptGA optimizer."""
self.prompt_template = prompt_template
self.meta_llm = meta_llm
self.selection_mode = selection_mode
super().__init__(
predictor=predictor, initial_prompts=initial_prompts, task=task, callbacks=callbacks, config=config
)
assert self.selection_mode in ["random", "wheel", "tour"], "Invalid selection mode."
```
--------------------------------
### Sentence Classification: Objective or Subjective
Source: https://finitearth.github.io/promptolution/examples/getting_started
This prompt classifies a given sentence as either "objective" or "subjective" based on its linguistic characteristics. It determines whether the sentence presents neutral information or expresses a personal opinion/bias. The classification is "objective" if the tone is neutral and detached, focusing on verifiable facts, and "subjective" if the tone is evaluative, emotive, or reveals bias.
```text
Classify a given sentence as either "objective" or "subjective" based on its linguistic characteristics. Determine whether the sentence presents neutral information or expresses a personal opinion/bias. If the text maintains a detached tone, focusing on verifiable facts and data, assign the label "objective". Conversely, if the tone is evaluative, emotive, or reveals a bias, categorize it as "subjective". Compare the tone of a factual text discussing political events to a text expressing a clear opinion about a historical event to grasp the distinction between the two genres. The predicted class will be the first class mentioned in the language model's response, enclosed within the marks and .
Input:
“This latest rule will open our borders even more, and the Court seems to relish making arbitrary decisions without thinking about consequences.
Output:
objective
Input:
Transportation Secretary Pete Buttigieg confirmed to The Associated Press on Thursday that $104.6 million in federal funds coming from last year’s bipartisan infrastructure bill will go toward a plan to dismantle Interstate 375, a highway built to bisect Detroit’s Black Bottom neighborhood and its epicenter of Black business, Paradise Valley.
Output:
objective
Input:
The last may go very deep.
Output:
objective
```