FrugalGPT

https://github.com/stanford-futuredata/frugalgpt
Admin
FrugalGPT is a framework offering techniques for building LLM applications with budget constraints,...
Tokens:55,106
Snippets:342
Trust Score:9.4
License:Apache-2.0
Update:5 months ago
Show doc for...
Tokens:
Raw
### Install FrugalGPT Locally

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/README.md

Commands to clone the repository, install the package, and download necessary data files for local FrugalGPT setup.

```bash
git clone https://github.com/stanford-futuredata/FrugalGPT
cd FrugalGPT
pip install git+https://github.com/stanford-futuredata/FrugalGPT
wget  https://github.com/lchen001/DataHolder/releases/download/v0.0.1/HEADLINES.zip
unzip HEADLINES.zip -d strategy/
rm HEADLINES.zip
wget -P db/ https://github.com/lchen001/DataHolder/releases/download/v0.0.1/HEADLINES.sqlite
wget -P db/ https://github.com/lchen001/DataHolder/releases/download/v0.0.1/qa_cache.sqlite
```

--------------------------------

### Install FrugalGPT and Download Data

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_SCIQ2024.ipynb

Installs FrugalGPT from GitHub, downloads datasets and models, and sets up the environment. Ensure you have git and wget installed.

```python
# set up the environment
%%capture
! git clone https://github.com/stanford-futuredata/FrugalGPT
! pip install git+https://github.com/stanford-futuredata/FrugalGPT
!mkdir -p FrugalGPT/strategy
! wget  https://github.com/lchen001/DataHolder/releases/download/v0.0.3/SCIQ_Model20241128.zip
! unzip SCIQ_Model20241128.zip -d FrugalGPT/strategy/SCIQ_Model20241128
! rm SCIQ_Model20241128.zip
! wget https://github.com/lchen001/DataHolder/releases/download/v0.0.3/SCIQ.sqlite.zip
! unzip SCIQ.sqlite.zip -d FrugalGPT/db
!mkdir -p FrugalGPT/data/SCIQ
!wget -P FrugalGPT/data/SCIQ https://github.com/stanford-futuredata/FrugalGPT/releases/download/0.0.1/SCIQ_train.csv
!wget -P FrugalGPT/data/SCIQ https://github.com/stanford-futuredata/FrugalGPT/releases/download/0.0.1/SCIQ_test.csv
```

--------------------------------

### Install FrugalGPT and Dependencies

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_OVERRULING2024.ipynb

Installs the FrugalGPT library, downloads necessary model data, and sets up the project directory structure. Ensure you have git and wget installed.

```python
# set up the environment
%%capture
! git clone https://github.com/stanford-futuredata/FrugalGPT
! pip install git+https://github.com/stanford-futuredata/FrugalGPT
!mkdir -p FrugalGPT/strategy
! wget  https://github.com/lchen001/DataHolder/releases/download/v0.0.2/OVERRULING_Model20241128.zip
! unzip OVERRULING_Model20241128.zip -d FrugalGPT/strategy/OVERRULING_Model20241128
! rm OVERRULING_Model20241128.zip
! wget -P FrugalGPT/db/ https://github.com/lchen001/DataHolder/releases/download/v0.0.3/OVERRULING.sqlite
!mkdir -p FrugalGPT/data/OVERRULING
!wget -P FrugalGPT/data/OVERRULING https://github.com/stanford-futuredata/FrugalGPT/releases/download/0.0.1/OVERRULING_train.csv
!wget -P FrugalGPT/data/OVERRULING https://github.com/stanford-futuredata/FrugalGPT/releases/download/0.0.1/OVERRULING_test.csv

```

--------------------------------

### Install FrugalGPT and Dependencies

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/analysis/FrugalGPT_GenerationAnalysis.ipynb

Sets up the environment by cloning the FrugalGPT repository, installing the package, downloading and unzipping datasets, and removing temporary files. This is a prerequisite for running the analysis.

```python
# set up the environment
%%capture
! git clone https://github.com/stanford-futuredata/FrugalGPT
! pip install git+https://github.com/stanford-futuredata/FrugalGPT
!mkdir -p FrugalGPT/strategy/
! wget  https://github.com/lchen001/DataHolder/releases/download/v0.0.3/HEADLINES_Model2024_New.zip
! unzip HEADLINES_Model2024_New.zip -d FrugalGPT/strategy/HEADLINES_Model2024_New
! rm HEADLINES_Model2024_New.zip
! wget -P FrugalGPT/db/ https://github.com/lchen001/DataHolder/releases/download/v0.0.3/HEADLINES.sqlite

```

--------------------------------

### Install FrugalGPT and Dependencies

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_COQA2024.ipynb

Sets up the environment by cloning the FrugalGPT repository, installing the package, downloading and unzipping model data, and fetching necessary databases and datasets.

```python
# set up the environment
%%capture
! git clone https://github.com/stanford-futuredata/FrugalGPT
! pip install git+https://github.com/stanford-futuredata/FrugalGPT
!mkdir -p FrugalGPT/strategy
! wget  https://github.com/lchen001/DataHolder/releases/download/v0.0.3/COQA_Model20241128.zip
! unzip COQA_Model20241128.zip -d FrugalGPT/strategy/COQA_Model202411
! rm COQA_Model20241128.zip
! wget -P FrugalGPT/db/ https://github.com/lchen001/DataHolder/releases/download/v0.0.3/COQA.sqlite
!mkdir -p FrugalGPT/data/COQA
!wget -P FrugalGPT/data/COQA https://github.com/stanford-futuredata/FrugalGPT/releases/download/0.0.1/COQA_train.csv
!wget -P FrugalGPT/data/COQA https://github.com/stanford-futuredata/FrugalGPT/releases/download/0.0.1/COQA_test.csv

```

--------------------------------

### Install FrugalGPT and Download Data

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_HEADLINES.ipynb

Installs the FrugalGPT library, clones the repository, and downloads necessary data files for the HEADLINES dataset. Ensure you have git and wget installed.

```python
# set up the environment
%%capture
! git clone https://github.com/stanford-futuredata/FrugalGPT
%cd FrugalGPT
! pip install git+https://github.com/stanford-futuredata/FrugalGPT
!mkdir -p strategy
! wget  https://github.com/lchen001/DataHolder/releases/download/v0.0.2/HEADLINES_Model2024.zip
! unzip HEADLINES_Model2024.zip -d strategy/
! rm HEADLINES_Model2024.zip
! wget -P db/ https://github.com/lchen001/DataHolder/releases/download/v0.0.2/HEADLINES.sqlite
```

--------------------------------

### Install FrugalGPT and Download Datasets

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_SCIQ.ipynb

Installs the FrugalGPT library, clones the repository, and downloads necessary datasets and models for the SCIQ benchmark. Ensure you have sufficient disk space and an internet connection.

```python
# set up the environment
%%capture
! git clone https://github.com/stanford-futuredata/FrugalGPT
%cd FrugalGPT
! pip install git+https://github.com/stanford-futuredata/FrugalGPT
!mkdir -p strategy
! wget  https://github.com/lchen001/DataHolder/releases/download/v0.0.2/SCIQ_Model2024.zip
! unzip SCIQ_Model2024.zip -d strategy/SCIQ_Model2024
! rm SCIQ_Model2024.zip
! wget -P db/ https://github.com/lchen001/DataHolder/releases/download/v0.0.2/SCIQ.sqlite
!mkdir -p data/SCIQ
!wget -P data/SCIQ https://github.com/stanford-futuredata/FrugalGPT/releases/download/0.0.1/SCIQ_train.csv
!wget -P data/SCIQ https://github.com/stanford-futuredata/FrugalGPT/releases/download/0.0.1/SCIQ_test.csv

```

--------------------------------

### Install and Load FrugalGPT

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/intro.ipynb

Installs FrugalGPT and sets up the environment for use. Requires autoreload for development.

```python
%load_ext autoreload
%autoreload 2
import sys, json, copy 
import logging
logging.disable(logging.CRITICAL)
sys.path.append("src/")
```

--------------------------------

### Install FrugalGPT and Download Data

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_AGNEWS2024.ipynb

Installs the FrugalGPT library, clones the repository, and downloads necessary datasets and models for the AGNEWS benchmark. Ensure you have sufficient disk space and an active internet connection.

```python
# set up the environment
%%capture
! git clone https://github.com/stanford-futuredata/FrugalGPT
! pip install git+https://github.com/stanford-futuredata/FrugalGPT
!mkdir -p FrugalGPT/strategy
! wget  https://github.com/lchen001/DataHolder/releases/download/v0.0.2/AGNEWS_Model202411.zip
! unzip AGNEWS_Model202411.zip -d FrugalGPT/strategy/AGNEWS_Model202411
! rm AGNEWS_Model202411.zip
!mkdir -p FrugalGPT/data/AGNEWS
!wget -P FrugalGPT/data/AGNEWS https://github.com/stanford-futuredata/FrugalGPT/releases/download/0.0.1/AGNEWS_train.csv
!wget -P FrugalGPT/data/AGNEWS https://github.com/stanford-futuredata/FrugalGPT/releases/download/0.0.1/AGNEWS_test.csv
! wget -P FrugalGPT/db/ https://github.com/lchen001/DataHolder/releases/download/v0.0.3/AGNEWS.sqlite
```

--------------------------------

### Define Budget List for Training

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_SCIQ2024.ipynb

Generates a list of budgets for training FrugalGPT strategies using `numpy.linspace`. It defines the start and end budgets, the number of evaluation points, and sets an initial minimum budget. This setup is crucial for exploring the performance-cost curve.

```python
start_budget = 0.000085
end_budget = 0.018
num_eval = 20

name = f'{dataname}_Model20241128'
budget_list = numpy.linspace(start_budget, end_budget, num_eval)
budget_list[0] = 0.00025

```

--------------------------------

### Setup Environment and API Keys

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_SCIQ2024.ipynb

Configures the Python environment for FrugalGPT by enabling autoreload and setting up necessary system paths. It also demonstrates how to set environment variables for various LLM API keys, though they are optional for running the notebook's core functionality.

```python
%load_ext autoreload
%autoreload 2
import sys, json, copy
import logging
logging.disable(logging.CRITICAL)
sys.path.append("src/")
```

```python
import os
os.environ['OPENAI_API_KEY'] = 'YOUR_OPENAI_KEY'
os.environ['AI21_STUDIO_API_KEY'] = 'YOUR_AI21_KEY'
os.environ['COHERE_STUDIO_API_KEY'] = 'YOUR_COHERE_KEY'
os.environ['TEXTSYNTH_API_SECRET_KEY'] = 'YOUR_TEXTSYNTH_KEY'
os.environ['ANTHROPIC_API_KEY'] = 'YOUR_ANTHROPIC_KEY'
os.environ['TOGETHER_API_KEY'] = 'YOUR_TOGETHER_KEY'
os.environ['GEMINI_API_KEY'] = 'YOUR_GEMINI_KEY'

from IPython.display import display
import FrugalGPT
supported_LLM = FrugalGPT.getservicename()
print("supported LLMs:",supported_LLM)
```

--------------------------------

### Setup LLM API Keys for FrugalGPT

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/intro.ipynb

Sets up environment variables for various LLM API keys. These are necessary for FrugalGPT to interact with services like OpenAI, AI21, Cohere, TextSynth, and Anthropic.

```python
import os
os.environ['OPENAI_API_KEY'] = 'OPENAI_API_KEY'
os.environ['AI21_STUDIO_API_KEY'] = 'AI21_STUDIO_API_KEY'
os.environ['COHERE_STUDIO_API_KEY'] = 'COHERE_STUDIO_API_KEY'
os.environ['TEXTSYNTH_API_SECRET_KEY'] = 'TEXTSYNTH_API_SECRET_KEY'
os.environ['ANTHROPIC_API_KEY'] = 'ANTHROPIC_API_KEY'
from IPython.display import display
import FrugalGPT
supported_LLM = FrugalGPT.getservicename()
print("supported LLMs:",supported_LLM)
```

--------------------------------

### Setup Environment and API Keys

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_COQA2024.ipynb

Configures the Python environment by enabling autoreload and setting up necessary system paths. It also demonstrates how to set environment variables for various LLM API keys, although they are not strictly required for notebook execution.

```python
%load_ext autoreload
%autoreload 2
import sys, json, copy
import logging
logging.disable(logging.CRITICAL)
sys.path.append("src/")

```

```python
import os
os.environ['OPENAI_API_KEY'] = 'YOUR_OPENAI_KEY'
os.environ['AI21_STUDIO_API_KEY'] = 'YOUR_AI21_KEY'
os.environ['COHERE_STUDIO_API_KEY'] = 'YOUR_COHERE_KEY'
os.environ['TEXTSYNTH_API_SECRET_KEY'] = 'YOUR_TEXTSYNTH_KEY'
os.environ['ANTHROPIC_API_KEY'] = 'YOUR_ANTHROPIC_KEY'
os.environ['TOGETHER_API_KEY'] = 'YOUR_TOGETHER_KEY'
os.environ['GEMINI_API_KEY'] = 'YOUR_GEMINI_KEY'

from IPython.display import display
import FrugalGPT
supported_LLM = FrugalGPT.getservicename()
print("supported LLMs:",supported_LLM)

```

--------------------------------

### Unified LLM Interface with LLMforAll

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/intro.ipynb

Demonstrates how to use LLMforAll to get a completion from a specific LLM service and retrieve its cost. Ensure supported_LLM is defined and accessible.

```python
MyLLMforAll = FrugalGPT.LLMforAll()
query = "Question: Who is Matei Zaharia in 2023?\nAnswer:"
service_name = supported_LLM[-1]
genparams = FrugalGPT.GenerationParameter(max_tokens=50, temperature=0.1, stop=['\n\n\n\n'])
answer = MyLLMforAll.get_completion(query,service_name,genparams=genparams)
cost = MyLLMforAll.get_cost()
print("API:",service_name,"answer:",answer,"cost:",cost)
```

--------------------------------

### FrugalGPT COQA Dataset Evaluation Setup

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_COQA2024.ipynb

This script sets up the evaluation for FrugalGPT on the COQA dataset. It defines the list of service names (models) to test, configures generation parameters, loads and formats the training and testing data, and then calls `generate_dataframe` to produce a summary DataFrame. Ensure that the data files and prompt configuration files exist at the specified paths.

```python
dataname = "COQA"
service_names = [
    'openaichat/gpt-4o-2024-05-13',
    #'openaichat/gpt-4o-2024-08-06',
    'openaichat/gpt-4o-mini',
    #'openaichat/gpt-4o',
    'openaichat/gpt-4-turbo',

    'google/gemini-1.5-flash',
    'google/gemini-1.5-pro',
    'google/gemini-1.5-flash-8b',

    #'togetherai/mistralai/Mistral-7B-Instruct-v0.1',
    #'togetherai/Qwen/Qwen2-72B-Instruct',
    #'togetherai/meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo',
    'togetherai/meta-llama/Meta-Llama-3-70B-Instruct-Turbo',
    'togetherai/google/gemma-2-9b-it',

    'anthropic/claude-3-5-sonnet-20240620',
    #'anthropic/claude-3-haiku-20240307',

    #'ai21/jamba-1.5-mini',
    'ai21/jamba-1.5-large',
                 ]
genparams=FrugalGPT.GenerationParameter(max_tokens=50, temperature=0.1, stop=['\n'])

test_data = FrugalGPT.loadcsvdata(f"data/{dataname}/test.csv")
prefix = open(f'config/prompt/{dataname}/prefix_e2.txt').read()
test_data = FrugalGPT.formatdata(test_data,prefix)

train_data = FrugalGPT.loadcsvdata(f"data/{dataname}/train.csv")
prefix = open(f'config/prompt/{dataname}/prefix_e2.txt').read()
train_data = FrugalGPT.formatdata(train_data,prefix)

sample_size = 10000
individualmodel_df = generate_dataframe(service_names,
                                        train_data[0:sample_size], test_data[0:sample_size],
                                        genparams,
                                        db_path=f"db/{dataname}.sqlite",
                                        max_workers=4)
display(individualmodel_df)
individualmodel_df.to_csv(f"summary_{dataname}_e2_2024.csv")
```

--------------------------------

### Setup Environment and API Keys

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/analysis/FrugalGPT_GenerationAnalysis.ipynb

Configures the Python environment by enabling autoreload and setting up necessary system paths. It also configures API keys for various LLM providers by setting environment variables. Note that API keys are optional if only using pre-configured models.

```python
%load_ext autoreload
%autoreload 2
import sys, json, copy
import logging
logging.disable(logging.CRITICAL)
sys.path.append("src/")

```

```python
import os
os.environ['OPENAI_API_KEY'] = 'OPENAI_API_KEY'
os.environ['AI21_STUDIO_API_KEY'] = 'AI21_STUDIO_API_KEY'
os.environ['COHERE_STUDIO_API_KEY'] = 'COHERE_STUDIO_API_KEY'
os.environ['TEXTSYNTH_API_SECRET_KEY'] = 'TEXTSYNTH_API_SECRET_KEY'
os.environ['ANTHROPIC_API_KEY'] = 'ANTHROPIC_API_KEY'
os.environ['TOGETHER_API_KEY'] = 'TOGETHER_API_KEY'
os.environ['GEMINI_API_KEY'] = 'GEMINI_API_KEY'

from IPython.display import display
import FrugalGPT
supported_LLM = FrugalGPT.getservicename()
print("supported LLMs:",supported_LLM)

```

--------------------------------

### Setup Environment and Load FrugalGPT

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_AGNEWS2024.ipynb

Configures the Python environment for automatic reloading of modules and imports necessary libraries. It also disables critical logging messages to keep the output clean.

```python
%load_ext autoreload
%autoreload 2
import sys, copy
import logging
logging.disable(logging.CRITICAL)
sys.path.append("src/")
```

--------------------------------

### Define Budget Range for Training

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_COQA2024.ipynb

Sets up a list of budgets for training FrugalGPT, defining the start, end, and number of evaluation points. Includes a specific adjustment for the initial budget value.

```python
start_budget = 0.000074
end_budget = 0.016
num_eval = 20

name = f'{dataname}_Model202411'
budget_list = numpy.linspace(start_budget, end_budget, num_eval)
budget_list[0] = 0.00007
#budget_list = budget_list[::-1]
```

--------------------------------

### New Training Start Message

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_COQA2024.ipynb

Indicates that training for a new budget is starting after previous ones were skipped.

```text
cannot find, start new training
first train
```

--------------------------------

### Define Budget Range for FrugalGPT Training

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_HEADLINES.ipynb

Sets up the start and end budgets, and the number of evaluation points, to create a linear space of budgets for training FrugalGPT strategies.

```python
start_budget = 3.5e-05
end_budget = 0.0036
num_eval = 20

name = f'{dataname}_Model2024'
budget_list = numpy.linspace(start_budget, end_budget, num_eval)
```

--------------------------------

### AGNEWS Dataset Evaluation Setup

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_AGNEWS2024.ipynb

This code block sets up the dataset, service names, and generation parameters for evaluating FrugalGPT on the AGNEWS dataset. It loads and formats training and testing data, and defines the models to be tested.

```python
dataname = "AGNEWS"
service_names = [
    'openaichat/gpt-4o-2024-05-13',
    #'openaichat/gpt-4o-2024-08-06',
    'openaichat/gpt-4o-mini',
    #'openaichat/gpt-4o',
    'openaichat/gpt-4-turbo',

    #'togetherai/mistralai/Mistral-7B-Instruct-v0.1',
    #'togetherai/Qwen/Qwen2-72B-Instruct',
    #'togetherai/meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo',
    'togetherai/meta-llama/Meta-Llama-3-70B-Instruct-Turbo',
    'togetherai/google/gemma-2-9b-it',

    'anthropic/claude-3-5-sonnet-20240620',
    #'anthropic/claude-3-haiku-20240307',

    #'ai21/jamba-1.5-mini',
    'ai21/jamba-1.5-large',

    'google/gemini-1.5-flash',
    'google/gemini-1.5-pro',
    'google/gemini-1.5-flash-8b',
                 ]
genparams=FrugalGPT.GenerationParameter(max_tokens=50, temperature=0.1, stop=['\n'])

test_data = FrugalGPT.loadcsvdata(f"data/{dataname}/test.csv")
prefix = open(f'config/prompt/{dataname}/prefix_e8.txt').read()
test_data = FrugalGPT.formatdata(test_data,prefix)

train_data = FrugalGPT.loadcsvdata(f"data/{dataname}/train.csv")
prefix = open(f'config/prompt/{dataname}/prefix_e8.txt').read()
train_data = FrugalGPT.formatdata(train_data,prefix)

sample_size = 20000
individualmodel_df = generate_dataframe(service_names,
                                        train_data[0:sample_size], test_data[0:sample_size],
                                        genparams,
                                        db_path=f"db/{dataname}.sqlite",
                                        max_workers=2)
display(individualmodel_df)
individualmodel_df.to_csv(f"summary_{dataname}_e8_2024.csv")

```

--------------------------------

### Define Budget List for Training

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_AGNEWS2024.ipynb

Generates a list of budgets for FrugalGPT training using linear spacing between a start and end budget. Allows for custom adjustment of the initial budget value.

```python
start_budget = 0.000025
end_budget = 0.0065
num_eval = 20

name = f'{dataname}_Model20241128'
budget_list = numpy.linspace(start_budget, end_budget, num_eval)
budget_list[0] = 0.00009
#budget_list = budget_list[::-1]
```

--------------------------------

### Compute Trade-offs for Different Budgets

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_HEADLINES.ipynb

Initializes the FrugalGPT cascade model and iterates through a list of budgets to train the strategy. This function is a starting point for exploring cost-performance trade-offs.

```python
import numpy
from tqdm import tqdm
def compute_tradeoffs(
    train_data,
                      budget_list,
                      name = "test",

                      service_names = ['openaichat/gpt-4o-mini',
                                       'openaichat/gpt-4o',
                                      'openaichat/gpt-4-turbo',
                 'togetherai/meta-llama/Meta-Llama-3-70B-Instruct-Turbo',
                                      'togetherai/google/gemma-2-9b-it',
                 ],
                      prefix="",
                      skip=0,
    MyCascade = FrugalGPT.LLMCascade(
          score_noise_injection=False,
  db_path="db/SCIQ.sqlite",
  ),

    cascade_depth=3,

                      ):

  for idx,budget in tqdm(enumerate(budget_list)):
    # train the model

```

--------------------------------

### Import FrugalGPT and Get Supported LLMs

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_SCIQ.ipynb

Import the FrugalGPT library and retrieve a list of all supported Large Language Models (LLMs). This function helps in understanding the available services for use with FrugalGPT.

```python
from IPython.display import display
import FrugalGPT
supported_LLM = FrugalGPT.getservicename()
print("supported LLMs:",supported_LLM)
```

--------------------------------

### Generate Budget List for Training

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_HEADLINES2024.ipynb

Defines the start and end budgets, the number of evaluation points, and generates a list of budgets using `numpy.linspace`. The first budget value is adjusted to a specific lower bound.

```python
start_budget = 0.0000171
end_budget = 0.004
num_eval = 20

name = f'{dataname}_Model20241128'
budget_list = numpy.linspace(start_budget, end_budget, num_eval)
budget_list[0] = 0.0000105
```

--------------------------------

### Debug Output for Trace

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/src/visualize/visualize_tradeoffs_newmodels.ipynb

This snippet shows the debug output for a trace, indicating the start of a Scatter plot visualization with its configuration.

```python
DEBUG:root:trace is Scatter({
```

--------------------------------

### Load and Prepare Data with LLM Cascade Initialization

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_SCIQ.ipynb

Loads training data from a CSV file and initializes an LLM Cascade object. Configure the cascade with parameters like score noise injection, database path, and batch building. The load path for pre-trained strategies is commented out.

```python
dev = FrugalGPT.loadcsvdata(f"data/{dataname}/train.csv")
train_data = FrugalGPT.formatdata(dev,prefix)
MyCascade= FrugalGPT.LLMCascade(
          score_noise_injection=False,
  db_path=f"db/{dataname}.sqlite",
  batch_build=True,
  )
#MyCascade.load(loadpath=f"strategy/{name}/",budget=0.00017)
```

--------------------------------

### Get Completions from All Services with LLMforAll

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/intro.ipynb

Retrieves completions from all supported LLM services for a given query and generation parameters. The results are displayed in a pandas DataFrame.

```python
responses = MyLLMforAll.get_completion_allservice(query, supported_LLM, genparams=genparams)
print("full responses")
display(responses)
```

--------------------------------

### Prepare Financial News Data for FrugalGPT

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/intro.ipynb

Prepares a dataset of financial news headlines for FrugalGPT by formatting raw data with a provided prefix. This involves creating a list of query-answer pairs and loading a prefix from a text file.

```python
dev = [['Q: april gold down 20 cents to settle at $1,116.10/oz
A:', 'down','0'],
       ['Q: gold suffers third straight daily decline
A:', 'down','1'],
       ['Q: Gold futures edge up after two-session decline
A:', 'up','2'],
       ['Q: Dec. gold climbs $9.40, or 0.7%, to settle at $1,356.90/oz
A:','up','3'],
       ['Q: Gold struggles; silver slides, base metals falter
A:','up','4'],
       ['Q: feb. gold ends up $9.60, or 1.1%, at $901.60 an ounce
A:','up','5'],
        ['Q: dent research : is gold\'s day in the sun coming soon?
A:','none','6']
      ]
prefix = open('config/prompt/HEADLINES/prefix_e8.txt').read()
raw_data = copy.deepcopy(dev)
data = FrugalGPT.formatdata(dev,prefix)
```

--------------------------------

### Load Data and Initialize FrugalGPT for Evaluation

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/intro.ipynb

Loads evaluation data from a CSV file, formats it with a specified prefix, and initializes the FrugalGPT LLMCascade model with a specific budget for performance evaluation. Ensure the data path and strategy load path are correct.

```python
test = FrugalGPT.loadcsvdata("data/HEADLINES/test.csv")
prefix = open('config/prompt/HEADLINES/prefix_e8.txt').read()
data_eval = FrugalGPT.formatdata(test,prefix)
print("test data size:",len(data_eval))
MyCascade = FrugalGPT.LLMCascade()
MyCascade.load(loadpath="strategy/HEADLINES/",budget=0.000665)
```

--------------------------------

### Initialize and Compute Tradeoffs with FrugalGPT

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_HEADLINES2024.ipynb

Initializes an LLMCascade object and computes tradeoffs using specified training data, budget list, and service names. Use this to configure and run FrugalGPT's core analysis.

```python
MyCascade2 = FrugalGPT.LLMCascade(
          score_noise_injection=False,
  db_path=f"db/{dataname}.sqlite",
  batch_build=True,
  )

#print(name)
compute_tradeoffs(train_data=train_data,
                  budget_list=budget_list,
                  #name=name,
                  #service_names=service_names,
                  name = 'testtest',
                  service_names=['openaichat/gpt-4o-mini',
                                'openaichat/gpt-4-turbo',
                                'togetherai/google/gemma-2-9b-it',
                 ],
                  prefix=prefix,
                  skip=0, # you can manually skip the first few budgets if they have already been trained.
                  MyCascade=MyCascade2,
                  cascade_depth=2,
                  )
```

--------------------------------

### FrugalGPT Model Training and Data Preparation

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_SCIQ2024.ipynb

Sets up the dataset, generation parameters, and service names for evaluating individual LLM models. Loads and formats data, then calls the `generate_dataframe` function to collect performance metrics. Saves the results to a CSV file.

```python
dataname = "SCIQ"
service_names = [
    'ai21/jamba-1.5-large',

    'togetherai/meta-llama/Meta-Llama-3-70B-Instruct-Turbo',
    'togetherai/google/gemma-2-9b-it',

    'google/gemini-1.5-flash',
    'google/gemini-1.5-pro',
    'google/gemini-1.5-flash-8b',

    'openaichat/gpt-4o-2024-05-13',
    'openaichat/gpt-4o-mini',
    'openaichat/gpt-4-turbo',

    'anthropic/claude-3-5-sonnet-20240620',
                 ]
genparams=FrugalGPT.GenerationParameter(max_tokens=50, temperature=0.1, stop=['\n'])

test_data = FrugalGPT.loadcsvdata(f"data/{dataname}/test.csv")
prefix = open(f'config/prompt/{dataname}/prefix_e8.txt').read()
test_data = FrugalGPT.formatdata(test_data,prefix)

train_data = FrugalGPT.loadcsvdata(f"data/{dataname}/train.csv")
prefix = open(f'config/prompt/{dataname}/prefix_e8.txt').read()
train_data = FrugalGPT.formatdata(train_data,prefix)

sample_size = 10000
individualmodel_df = generate_dataframe(service_names,
                                        train_data[0:sample_size], test_data[0:sample_size],
                                        genparams,
                                        db_path=f"db/{dataname}.sqlite",
                                        max_workers=2)
display(individualmodel_df)
individualmodel_df.to_csv(f"summary_{dataname}_e8_2024.csv")
```

--------------------------------

### Configure and Run FrugalGPT for Model Evaluation

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_OVERRULING2024.ipynb

This script sets up the necessary data, service names, and generation parameters for evaluating FrugalGPT. It loads and formats training and testing data, then calls the `generate_dataframe` function to assess various language models. The results are displayed and saved to a CSV file. Ensure that the data files and configuration prompts exist at the specified paths.

```python
dataname = "OVERRULING"
service_names = [
    'openaichat/gpt-4o-2024-05-13',
    #'openaichat/gpt-4o-2024-08-06',
    'openaichat/gpt-4o-mini',
    #'openaichat/gpt-4o',
    'openaichat/gpt-4-turbo',

    'google/gemini-1.5-flash',
    'google/gemini-1.5-pro',
    'google/gemini-1.5-flash-8b',

    #'ai21/jamba-1.5-mini',
    'ai21/jamba-1.5-large',

    #'togetherai/mistralai/Mistral-7B-Instruct-v0.1',
    #'togetherai/Qwen/Qwen2-72B-Instruct',
    #'togetherai/meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo',
    'togetherai/meta-llama/Meta-Llama-3-70B-Instruct-Turbo',
    'togetherai/google/gemma-2-9b-it',

    'anthropic/claude-3-5-sonnet-20240620',
    #'anthropic/claude-3-haiku-20240307',

                 ]
genparams=FrugalGPT.GenerationParameter(max_tokens=50, temperature=0.1, stop=['\n'])

test_data = FrugalGPT.loadcsvdata(f"data/{dataname}/test.csv")
prefix = open(f'config/prompt/{dataname}/prefix_e5.txt').read()
test_data = FrugalGPT.formatdata(test_data,prefix)

train_data = FrugalGPT.loadcsvdata(f"data/{dataname}/train.csv")
prefix = open(f'config/prompt/{dataname}/prefix_e5.txt').read()
train_data = FrugalGPT.formatdata(train_data,prefix)

sample_size = 5000
individualmodel_df = generate_dataframe(service_names,
                                        train_data[0:sample_size], test_data[0:sample_size],
                                        genparams,
                                        db_path=f"db/{dataname}.sqlite",
                                        max_workers=10)
display(individualmodel_df)
individualmodel_df.to_csv(f"summary_{dataname}_e5_2024.csv")

```

--------------------------------

### Initial Performance Metrics

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_AGNEWS2024.ipynb

Displays the initial performance metrics recorded for the FrugalGPT method on the AGNEWS dataset, including training and testing accuracy, cost, and data size.

```python
 {'Test_cost': 0.0022732568092105263,
 'Test_size': 3800,
 'Train_acc': 0.9105263157894737,
 'Train_cost': 0.002314328309210526,
 'Train_size': 3800,
 'Budget': 0.002751315789473684,
 'Method': 'FrugalGPT',
 'Provider': 'FrugalGPT',
 'Marker': 1}
```

--------------------------------

### Load and Format Data with FrugalGPT

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_COQA2024.ipynb

Loads CSV data and formats it for FrugalGPT. Initializes an LLM cascade with specified database path and batch build settings.

```python
dev = FrugalGPT.loadcsvdata(f"data/{dataname}/train.csv")
train_data = FrugalGPT.formatdata(dev,prefix)
MyCascade= FrugalGPT.LLMCascade(
          score_noise_injection=False,
  db_path=f"db/{dataname}.sqlite",
  batch_build=True,
  )
```

--------------------------------

### Load and Format Data with LLM Cascade Initialization

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_SCIQ2024.ipynb

Loads CSV data, formats it, and initializes an LLM Cascade. Configure parameters like score noise injection, database path, and batch building.

```python
dev = FrugalGPT.loadcsvdata(f"data/{dataname}/train.csv")
train_data = FrugalGPT.formatdata(dev,prefix)
MyCascade= FrugalGPT.LLMCascade(
          score_noise_injection=False,
  db_path=f"db/{dataname}.sqlite",
  batch_build=True,
  )
budget_list
#MyCascade.load(loadpath=f"strategy/{name}/",budget=0.00017)
```

--------------------------------

### Evaluate FrugalGPT and Save Results

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_SCIQ2024.ipynb

Initializes the FrugalGPT LLM Cascade, generates a performance DataFrame, displays it, and saves it to a CSV file. This snippet is used after defining the cascade and data parameters.

```python
MyCascade_eval = FrugalGPT.LLMCascade()
MyCascade_eval.prefix = prefix
frugalgpt_df = generate_dataframe_from_cascade(MyCascade_eval,
                                               budget_list, train_data, test_data, genparams,
                                               name)
display(frugalgpt_df)
frugalgpt_df.to_csv(f"summary_{dataname}_e8_frugalgpt_2024.csv")
```

--------------------------------

### Instantiate LLM Model Providers with make_model

Source: https://context7.com/stanford-futuredata/frugalgpt/llms.txt

The make_model factory function simplifies creating model service instances for various LLM providers by accepting provider names and model identifiers. Ensure API keys are set as environment variables before instantiation. The returned model object can then be used to get completions, along with generation parameters, and will provide cost and latency information.

```python
import os
from service.modelservice import make_model, GenerationParameter

os.environ["OPENAI_API_KEY"] = "sk-..."
os.environ["ANTHROPIC_API_KEY"] = "sk-ant-..."

# Instantiate a GPT-4o-mini chat model
gpt_mini = make_model("openaichat", "gpt-4o-mini")

# Instantiate Claude 3 Haiku
claude_haiku = make_model("anthropic", "claude-3-haiku-20240307")

genparams = GenerationParameter(max_tokens=50, temperature=0.1, stop=["\n"])
result = gpt_mini.getcompletion("What is the capital of France?", genparams=genparams)
print(result["completion"])   # " Paris"
print(result["cost"])
# e.g. 0.000003
print(result["latency"])
# e.g. 0.45
```

--------------------------------

### Load Autoreload and Initialize Environment

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_HEADLINES.ipynb

Configures the Python environment for development by enabling autoreload and setting up the system path to include the FrugalGPT source directory. Logging is also disabled for cleaner output.

```python
%load_ext autoreload
%autoreload 2
import sys, json, copy
import logging
logging.disable(logging.CRITICAL)
sys.path.append("src/")
```

--------------------------------

### File Download Progress: config.json

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_SCIQ.ipynb

Shows the download progress for the configuration JSON file. The progress is indicated as 0% complete.

```text
config.json:   0%|          | 0.00/483 [00:00<?, ?B/s]
```

--------------------------------

### Download Strategy Zip File

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_COQA2024.ipynb

Initiates the download of the created zip file containing the strategy.

```python
files.download(output_zip_file)
```

--------------------------------

### Batch Generation Comparison: FrugalGPT vs. GPT-4

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/intro.ipynb

Performs batch generation using both FrugalGPT LLMCascade and vanilla GPT-4 for a set of queries. It then displays the results and computes performance scores (e.g., accuracy and cost) for comparison.

```python
# batch generation
result = MyCascade.get_completion_batch(queries=data,genparams=genparams)
result_GPT4 = MyLLMforAll.get_completion_batch(queries=data,genparams=genparams,service_name='openaichat/gpt-4')
print("FrugalGPT LLMCascade generations")
display(result)
display(FrugalGPT.compute_score(result))
print("GPT-4 generations")
display(result_GPT4)
display(FrugalGPT.compute_score(result_GPT4))
```

--------------------------------

### Load and Configure FrugalGPT LLMCascade

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/intro.ipynb

Initializes and loads an LLMCascade instance from FrugalGPT. Specify the path to load the cascade strategy and set a budget for the operations.

```python
MyCascade = FrugalGPT.LLMCascade()
MyCascade.load(loadpath="strategy/HEADLINES/",budget=0.000665)
```

--------------------------------

### Load Custom Training Data for FrugalGPT

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/intro.ipynb

Loads a custom training dataset from a CSV file for FrugalGPT and formats it using a specified prefix. This snippet demonstrates how to prepare your own data for training.

```python
# load data
dev = FrugalGPT.loadcsvdata("data/HEADLINES/train.csv")
dev = dev[0:10]
prefix = open('config/prompt/HEADLINES/prefix_e8.txt').read()
data = FrugalGPT.formatdata(dev,prefix)
```

--------------------------------

### Change Directory to FrugalGPT

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_SCIQ2024.ipynb

Navigates the file system to the root directory of the FrugalGPT project. This is typically done after cloning the repository.

```python
%cd FrugalGPT
```

--------------------------------

### Prepare and Save FrugalGPT Datasets

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_COQA2024.ipynb

Loads reshuffled COCOA data, converts it into the FrugalGPT format using helper functions, and saves the processed train and test files without headers. Requires FrugalGPT library and pandas.

```python
train_raw = FrugalGPT.loadcsvdata("data/COQA_new/COQA_train.csv")
test_raw = FrugalGPT.loadcsvdata("data/COQA_new/COQA_test.csv")
train_df = list_to_dataframe(train_raw)
test_df = list_to_dataframe(test_raw)
converted_train, converted_test = convert_and_merge_dataframes(train_df, test_df)
columns_to_save = ['query', 'ref_answer', 'query_id']
converted_train[columns_to_save].to_csv("data/COQA_new/train.csv", index=False, header=False)
converted_test[columns_to_save].to_csv("data/COQA_new/test.csv",index=False, header=False)
```

--------------------------------

### File Download Progress: tokenizer_config.json

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_SCIQ.ipynb

Shows the download progress for the tokenizer configuration file. The progress is indicated as 0% complete.

```text
tokenizer_config.json:   0%|          | 0.00/48.0 [00:00<?, ?B/s]
```

--------------------------------

### Load and Format Dataset for FrugalGPT

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/analysis/FrugalGPT_GenerationAnalysis.ipynb

Loads CSV data and formats it using a specified prefix. Ensure the data path and prefix file path are correct.

```python
dataname = "HEADLINES"

test_data = FrugalGPT.loadcsvdata(f"data/{dataname}/test.csv")
prefix = open(f'config/prompt/{dataname}/prefix_e8.txt').read()
test_data = FrugalGPT.formatdata(test_data,prefix)

train_data = FrugalGPT.loadcsvdata(f"data/{dataname}/train.csv")
prefix = open(f'config/prompt/{dataname}/prefix_e8.txt').read()
train_data = FrugalGPT.formatdata(train_data,prefix)
```

--------------------------------

### Zip and Download Strategy Folder

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_OVERRULING2024.ipynb

Compresses the saved strategy folder into a zip file and then initiates a download of the created zip archive. This is useful for backing up or sharing the trained strategies.

```python
import shutil

# Specify the folder to zip
folder_to_zip = f'strategy/{name}'
output_zip_file = f'{name}.zip'

# Create the zip file
shutil.make_archive(output_zip_file.replace('.zip', ''), 'zip', folder_to_zip)
print(f"Folder '{folder_to_zip}' zipped as '{output_zip_file}'.")
```

```python
files.download(f"OVERRULING_Model20241128.zip")
```

--------------------------------

### Execute FrugalGPT Strategy Training and Save Results

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_HEADLINES2024.ipynb

This snippet defines a list of service names and generation parameters, then calls the `generate_dataframe` function to evaluate individual models. The resulting DataFrame is displayed and saved to a CSV file.

```python
service_names = [
    'ai21/jamba-1.5-large',

    'togetherai/meta-llama/Meta-Llama-3-70B-Instruct-Turbo',
    'togetherai/google/gemma-2-9b-it',

    'google/gemini-1.5-flash',
    'google/gemini-1.5-pro',
    'google/gemini-1.5-flash-8b',

    'openaichat/gpt-4o-2024-05-13',
    'openaichat/gpt-4o-mini',
    'openaichat/gpt-4-turbo',

    'anthropic/claude-3-5-sonnet-20240620',
                 ]

genparams=FrugalGPT.GenerationParameter(max_tokens=50, temperature=0.1, stop=['\n'])

sample_size = 10000
individualmodel_df = generate_dataframe(service_names,
                                        train_data[0:sample_size], test_data[0:sample_size],
                                        genparams,
                                        db_path=f"db/{dataname}.sqlite",
                                        max_workers=2)
display(individualmodel_df)
individualmodel_df.to_csv(f"summary_{dataname}_e8_2024.csv")
```

--------------------------------

### Train FrugalGPT LLMCascade Model

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/intro.ipynb

Initializes and trains an LLMCascade model with a specified budget and a list of available services. Ensure 'data' is pre-loaded and formatted.

```python
MyCascade = FrugalGPT.LLMCascade()
service_names = ['openaichat/gpt-3.5-turbo','openaichat/gpt-4','ai21/j1-large','textsynth/gptj_6B']
result = MyCascade.train(data,budget=100,service_names=service_names)
```

--------------------------------

### Generate Single Completion with FrugalGPT LLMCascade

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/intro.ipynb

Generates a single text completion using the FrugalGPT LLMCascade. This involves selecting a query, defining generation parameters like max tokens and temperature, and then retrieving the completion and cost.

```python
index = 2
query = data[index][0]
query_raw = raw_data[index][0]
genparams=FrugalGPT.GenerationParameter(max_tokens=50, temperature=0.1, stop=['\n'])
answer = MyCascade.get_completion(query=query,genparams=genparams)
cost = MyCascade.get_cost()
print("query:",query_raw)
print("FrugalGPT LLMCascade answer:",answer)
```

--------------------------------

### Zip and Download Strategy Files

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_AGNEWS2024.ipynb

Compresses the generated strategy files into a zip archive and initiates a download. Requires the 'shutil' and 'files' modules.

```python
import shutil

# Specify the folder to zip
folder_to_zip = f'strategy/{name}'
output_zip_file = f'{name}.zip'

# Create the zip file
shutil.make_archive(output_zip_file.replace('.zip', ''), 'zip', folder_to_zip)
print(f"Folder '{folder_to_zip}' zipped as '{output_zip_file}'.")
files.download(output_zip_file)
```

--------------------------------

### File Download Progress: model.safetensors

Source: https://github.com/stanford-futuredata/frugalgpt/blob/main/examples/FrugalGPT_gen_tradeoff_SCIQ.ipynb

Shows the download progress for the model's safetensors file. The progress is indicated as 0% complete.

```text
model.safetensors:   0%|          | 0.00/268M [00:00<?, ?B/s]
```
Tokens:
Raw