### Web UI Setup

Source: https://github.com/openbmb/toolbench/blob/master/README.md

Clone the repository, install dependencies, and run the development server for the ToolBench Web UI.

```bash
git clone https://github.com/lilbillybiscuit/chatbot-ui-toolllama
cd chatbot-ui-toolllama
npm install
npm run dev
```

--------------------------------

### Start ToolBench Backend Server

Source: https://context7.com/openbmb/toolbench/llms.txt

Starts the Flask backend server for the ToolBench web interface, enabling streaming API responses. Requires setting PYTHONPATH and providing various model and data paths.

```bash
# Start backend server
export PYTHONPATH=./
python toolbench/inference/toolbench_server.py \
    --tool_root_dir data/toolenv/tools/ \
    --corpus_tsv_path data/retrieval/G1/corpus.tsv \
    --retrieval_model_path /path/to/retrieval_model \
    --retrieved_api_nums 5 \
    --backbone_model toolllama \
    --model_path /path/to/model \
    --method DFS_woFilter_w2 \
    --rapidapi_key $RAPIDAPI_KEY
```

--------------------------------

### Install ToolBench Package

Source: https://github.com/openbmb/toolbench/blob/master/toolbench/tooleval/README.md

Install the ToolBench package using pip. Ensure you have Python 3.9 or higher.

```bash
pip install -r requirements.txt
```

--------------------------------

### Clone ToolBench Repository

Source: https://github.com/openbmb/toolbench/blob/master/README.md

Clone the ToolBench repository and navigate into the directory. This is the first step before installing dependencies.

```bash
git clone git@github.com:OpenBMB/ToolBench.git
cd ToolBench
```

--------------------------------

### Clone ToolBench Repository and Install Dependencies

Source: https://context7.com/openbmb/toolbench/llms.txt

Clone the ToolBench repository from GitHub and install the required Python packages using pip.

```bash
git clone git@github.com:OpenBMB/ToolBench.git
cd ToolBench
pip install -r requirements.txt
```

--------------------------------

### Start ToolBench Backend Server

Source: https://github.com/openbmb/toolbench/blob/master/README.md

Export PYTHONPATH and run the ToolBench server script with specified directories, model paths, and configuration parameters. Ensure all paths and API keys are correctly set.

```bash
export PYTHONPATH=./
python toolbench/inference/toolbench_server.py \
    --tool_root_dir data/toolenv/tools/ \
    --corpus_tsv_path data/retrieval/G1/corpus.tsv \
    --retrieval_model_path /path/to/your/retrival_model \
    --retrieved_api_nums 5 \
    --backbone_model toolllama \
    --model_path huggyllama/llama-7b \
    --lora \
    --lora_path /path/to/your/toolllama_lora \
    --max_observation_length 1024 \
    --method DFS_woFilter_w2 \
    --input_query_file data/test_instruction/G1_instruction.json \
    --output_answer_file toolllama_lora_dfs_open_domain_result \
    --rapidapi_key $RAPIDAPIKEY
```

--------------------------------

### Command Line Usage for Data Preprocessing

Source: https://context7.com/openbmb/toolbench/llms.txt

Example command to run the data preprocessing script. Ensure PYTHONPATH is set correctly and specify the input data directory, method, and output file.

```bash
# Command line usage
export PYTHONPATH=./
python preprocess/preprocess_toolllama_data.py \
    --tool_data_dir data/answer/G1_answer \
    --method DFS_woFilter_w2 \
    --output_file data/answer/toolllama_G1_dfs.json
```

--------------------------------

### QA Pipeline Inference Commands

Source: https://context7.com/openbmb/toolbench/llms.txt

Command-line interface examples for running inference with different backbone models, including ToolLLaMA, LoRA-tuned models, and ChatGPT.

```bash
# Inference with ToolLLaMA
export PYTHONPATH=./
export TOOLBENCH_KEY="your_toolbench_key"
python toolbench/inference/qa_pipeline.py \
    --tool_root_dir data/toolenv/tools/ \
    --backbone_model toolllama \
    --model_path ToolBench/ToolLLaMA-7b \
    --max_observation_length 1024 \
    --observ_compress_method truncate \
    --method DFS_woFilter_w2 \
    --input_query_file data/test_instruction/G1_instruction.json \
    --output_answer_file toolllama_dfs_inference_result \
    --toolbench_key $TOOLBENCH_KEY

# Inference with ToolLLaMA-LoRA
python toolbench/inference/qa_pipeline.py \
    --tool_root_dir data/toolenv/tools/ \
    --backbone_model toolllama \
    --model_path huggyllama/llama-7b \
    --lora \
    --lora_path /path/to/ToolLLaMA-7b-LoRA \
    --method DFS_woFilter_w2 \
    --input_query_file data/test_instruction/G1_instruction.json \
    --output_answer_file toolllama_lora_dfs_result \
    --toolbench_key $TOOLBENCH_KEY

# Inference with ChatGPT
export OPENAI_KEY="your_openai_key"
python toolbench/inference/qa_pipeline.py \
    --tool_root_dir data/toolenv/tools/ \
    --backbone_model chatgpt_function \
    --openai_key $OPENAI_KEY \
    --method DFS_woFilter_w2 \
    --input_query_file data/test_instruction/G1_instruction.json \
    --output_answer_file chatgpt_dfs_result \
    --toolbench_key $TOOLBENCH_KEY
```

--------------------------------

### Open-Domain Inference with Retriever

Source: https://context7.com/openbmb/toolbench/llms.txt

Command-line example for running inference on queries without pre-specified API lists, utilizing a retrieval model to identify relevant tools.

```bash
# Open-domain inference with retriever
export PYTHONPATH=./
python toolbench/inference/qa_pipeline_open_domain.py \
    --tool_root_dir data/toolenv/tools/ \
    --corpus_tsv_path data/retrieval/G1/corpus.tsv \
    --retrieval_model_path /path/to/retrieval_model \
    --retrieved_api_nums 5 \
    --backbone_model toolllama \
    --model_path huggyllama/llama-7b \
    --lora \
    --lora_path /path/to/toolllama_lora \
    --max_observation_length 1024 \
    --method DFS_woFilter_w2 \
    --input_query_file data/test_instruction/G1_instruction.json \
    --output_answer_file open_domain_result \
    --toolbench_key $TOOLBENCH_KEY
```

--------------------------------

### Install ToolBench Dependencies

Source: https://github.com/openbmb/toolbench/blob/master/README.md

Install the required Python packages for ToolBench. Ensure you have Python 3.9 or higher. For ToolEval only, use the specific requirements file.

```bash
pip install -r requirements.txt
```

```bash
pip install -r toolbench/tooleval/requirements.txt
```

--------------------------------

### ToolBench Server Request Format

Source: https://github.com/openbmb/toolbench/blob/master/README.md

Send a GET or POST request to http://localhost:5000/stream with a JSON object containing the query text, top_k, and evaluation method.

```json
{
    "text": "What is the weather in New York today?",
    "top_k": 5,
    "method": "DFS_woFilter_w2"
}
```

--------------------------------

### Send Client Request to ToolBench Backend

Source: https://context7.com/openbmb/toolbench/llms.txt

Example cURL command to send a POST request to the ToolBench backend's streaming endpoint. Specifies user input, top_k, and method.

```bash
# Client request example
curl -X POST http://localhost:5000/stream \
    -H "Content-Type: application/json" \
    -d '{"text": "What is the weather in New York today?", "top_k": 5, "method": "DFS_woFilter_w2"}'
```

--------------------------------

### Custom API Code Example

Source: https://github.com/openbmb/toolbench/blob/master/README.md

Example Python code for a custom API function. This code should be placed in `api.py` within the custom tool's directory.

```python
def get_hello_world():
    """
    To get hello world 
    """
    observation = "hello world"
    return observation
```

--------------------------------

### Download and Unzip Dataset

Source: https://github.com/openbmb/toolbench/blob/master/README.md

Download the ToolBench dataset using wget and then unzip it. Ensure the data directory is placed under the ToolBench folder.

```bash
wget --no-check-certificate 'https://drive.google.com/uc?export=download&id=1XFjDxVZdUY7TXYF2yvzx3pJlS2fy78jk&confirm=yes' -O data.zip
unzip data.zip
```

--------------------------------

### Full Fine-tuning ToolLLaMA with FSDP

Source: https://context7.com/openbmb/toolbench/llms.txt

Trains ToolLLaMA models using full fine-tuning on ToolBench data with FSDP (Fully Sharded Data Parallel) for distributed training. Requires specifying model paths, data paths, and training configurations.

```bash
# Training with 2x A100 GPUs (80GB)
export PYTHONPATH=./
torchrun --nproc_per_node=2 --master_port=20001 toolbench/train/train_mem.py \
    --model_name_or_path huggyllama/llama-7b \
    --data_path data/toolllama_G123_dfs_train.json \
    --eval_data_path data/toolllama_G123_dfs_eval.json \
    --conv_template tool-llama-single-round \
    --bf16 True \
    --output_dir toolllama \
    --num_train_epochs 2 \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 2 \
    --gradient_accumulation_steps 8 \
    --evaluation_strategy "epoch" \
    --save_strategy "epoch" \
    --save_total_limit 8 \
    --learning_rate 5e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.04 \
    --lr_scheduler_type "cosine" \
    --fsdp "full_shard auto_wrap" \
    --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
    --source_model_max_length 2048 \
    --model_max_length 8192 \
    --gradient_checkpointing True \
    --lazy_preprocess True
```

--------------------------------

### Integrate custom APIs

Source: https://context7.com/openbmb/toolbench/llms.txt

Define API documentation, implement the function, and create a query file to enable custom tool usage.

```python
# Step 1: Create API documentation (data/toolenv/tools/Customized/hello_world.json)
api_doc = {
    "tool_description": "Return hello world message.",
    "tool_name": "hello world",
    "title": "hello world",
    "api_list": [
        {
            "name": "get_hello_world",
            "url": "",
            "description": "Returns a hello world greeting.",
            "method": "GET",
            "required_parameters": [],
            "optional_parameters": [
                {"name": "name", "type": "STRING", "description": "Name to greet", "default": "World"}
            ]
        }
    ],
    "standardized_name": "hello_world"
}

# Step 2: Create API implementation (data/toolenv/tools/Customized/hello_world/api.py)
def get_hello_world(name="World"):
    """
    Returns a personalized hello world greeting.

    Args:
        name: Name to include in greeting (default: "World")

    Returns:
        Greeting string
    """
    return f"Hello, {name}!"

# Step 3: Create query file for inference
query = [
    {
        "query": "I want to get a personalized greeting for Alice.",
        "query_id": 200001,
        "api_list": [
            {
                "category_name": "Customized",
                "tool_name": "hello world",
                "api_name": "get_hello_world"
            }
        ]
    }
]
```

--------------------------------

### Run inference with custom API

Source: https://context7.com/openbmb/toolbench/llms.txt

Execute the QA pipeline using a custom API configuration and input query file.

```bash
export PYTHONPATH=./
python toolbench/inference/qa_pipeline.py \
    --tool_root_dir data/toolenv/tools/ \
    --backbone_model toolllama \
    --model_path ToolBench/ToolLLaMA-7b \
    --method DFS_woFilter_w2 \
    --input_query_file /path/to/custom_query.json \
    --output_answer_file custom_api_result \
    --api_customization
```

--------------------------------

### Configure Chart Options

Source: https://github.com/openbmb/toolbench/blob/master/docs/index.html

Sets up configuration options for a chart, including colors, borders, and axis scales. Useful for customizing data visualizations.

```javascript
ckgroundColors, borderColor: borderColors, borderWidth: 2, }, ], }, options: { indexAxis:'y', scales: { x: { //max:1.0, beginAtZero: true, }, y:{ ticks:{ font:{ size:16, weight:'bold' }, } } }, plugins: { //title:{ // display:true, // text:'Win Rate Leaderboard', // font:{ // size:20 // } //}, legend: { display: false, }, tooltip: { //callbacks:{ // footer:(tooltipItems)=>{ // return 
`SEM: ${tooltipItems[0].dataset.sems[tooltipItems[0].dataIndex].substring(0,5)}`
 // } //}
 }, }, }, }
) };
```

--------------------------------

### Inference with Custom API

Source: https://github.com/openbmb/toolbench/blob/master/README.md

Run inference with custom APIs enabled. Ensure `api_customization` flag is set and the `tool_root_dir` points to your custom tools.

```bash
export PYTHONPATH=./
python toolbench/inference/qa_pipeline.py \
    --tool_root_dir data/toolenv/tools/ \
    --backbone_model toolllama \
    --model_path ToolBench/ToolLLaMA-7b \
    --max_observation_length 1024 \
    --observ_compress_method truncate \
    --method DFS_woFilter_w2 \
    --input_query_file /path/to/your/query/file \
    --output_answer_file /path/to/your/output/file \
    --api_customization
```

--------------------------------

### Initialize and Use ChatGPT Function Calling Model

Source: https://context7.com/openbmb/toolbench/llms.txt

Initializes the ChatGPT function calling model and demonstrates its usage for parsing messages with available functions. Requires an OpenAI API key.

```python
import openai
from tenacity import retry, wait_random_exponential, stop_after_attempt

class ChatGPTFunction:
    def __init__(self, model="gpt-3.5-turbo-16k-0613", openai_key=""):
        """
        Initialize ChatGPT function calling model.

        Args:
            model: OpenAI model identifier
            openai_key: OpenAI API key
        """
        self.model = model
        self.openai_key = openai_key
        self.conversation_history = []

    def change_messages(self, messages):
        """Update conversation history."""
        self.conversation_history = messages

    def parse(self, functions, process_id=0):
        """
        Generate next action using ChatGPT function calling.

        Args:
            functions: List of available function schemas
            process_id: Process identifier

        Returns:
            tuple: (message_dict, error_code, total_tokens)
        """
        openai.api_key = self.openai_key

        response = openai.ChatCompletion.create(
            model=self.model,
            messages=self.conversation_history,
            functions=functions,
            max_tokens=1024
        )

        message = response["choices"][0]["message"]
        total_tokens = response["usage"]["total_tokens"]

        # Handle function calls
        if "function_call" in message:
            return message, 0, total_tokens

        return message, 0, total_tokens

# Usage
llm = ChatGPTFunction(openai_key="sk-...")
llm.change_messages([
    {"role": "system", "content": "You are a helpful assistant with tool access."},
    {"role": "user", "content": "What's the weather in New York?"}
])
response, error, tokens = llm.parse(functions=[{
    "name": "get_weather",
    "description": "Get weather for a location",
    "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}
}])
```

--------------------------------

### API Documentation Format

Source: https://github.com/openbmb/toolbench/blob/master/README.md

This is the expected JSON format for documenting custom APIs. Ensure all fields are correctly populated.

```json
{
    "tool_description": "Return hello world.",
    "tool_name": "hello world",
    "title": "hello world",
    "api_list": [
        {
            "name": "get_hello_world",
            "url": "",
            "description": "To get 'hello world'.",
            "method": "GET",
            "required_parameters": [],
            "optional_parameters": []
        }
    ],
    "standardized_name": "hello_world"
}
```

--------------------------------

### Inference with ChatGPT

Source: https://github.com/openbmb/toolbench/blob/master/README.md

Use this command to run inference with the ChatGPT model. Ensure API keys and necessary paths are exported.

```bash
export TOOLBENCH_KEY=""
export OPENAI_KEY=""
export PYTHONPATH=./
python toolbench/inference/qa_pipeline.py \
    --tool_root_dir data/toolenv/tools/ \
    --backbone_model chatgpt_function \
    --openai_key $OPENAI_KEY \
    --max_observation_length 1024 \
    --method DFS_woFilter_w2 \
    --input_query_file data/test_instruction/G1_instruction.json \
    --output_answer_file chatgpt_dfs_inference_result \
    --toolbench_key $TOOLBENCH_KEY
```

--------------------------------

### Preprocess Data for ToolLLaMA Training

Source: https://context7.com/openbmb/toolbench/llms.txt

Converts raw solution path annotations into a training format suitable for ToolLLaMA. This script extracts conversation turns with reasoning traces, actions, and observations from JSON answer files.

```python
# preprocess/preprocess_toolllama_data.py
from toolbench.utils import process_system_message
import json
import os

def preprocess_rapidapi(tool_data_dir, method, output_file):
    """
    Process answer files into training data format.

    Args:
        tool_data_dir: Directory containing answer JSON files
        method: Data annotation method (e.g., "DFS_woFilter_w2")
        output_file: Output path for preprocessed data
    """
    out_list = []
    for data_file in os.listdir(tool_data_dir):
        if method not in data_file:
            continue
        data_dict = json.load(open(os.path.join(tool_data_dir, data_file), "r"))
        answer_generation = data_dict["answer_generation"]

        if not answer_generation["valid_data"]:
            continue

        train_messages = answer_generation["train_messages"]
        functions = answer_generation["function"]

        for train_message in train_messages:
            conversations = []
            for message_dict in train_message:
                role = message_dict["role"]
                if role == "system":
                    content = process_system_message(message_dict["content"], functions)
                else:
                    content = message_dict["content"]
                conversations.append({"from": role, "value": content})
            out_list.append({"conversations": conversations})

    json.dump(out_list, open(output_file, "w"), indent=4, ensure_ascii=False)
```

--------------------------------

### Initialize Leaderboard and Radar Chart Selectors

Source: https://github.com/openbmb/toolbench/blob/master/docs/index.html

Initializes JavaScript variables for DOM elements and a URL mapping for leaderboard data. Sets up event listeners for changes in leaderboard and radar chart selections to update the displayed data.

```javascript
const leaderboardSelect = document.getElementById('leaderboardBaseline');
const radarSelect = document.getElementById('radarBaseline')
const records_urls={
  'ChatGPT-ReACT':'https://raw.githubusercontent.com/OpenBMB/ToolBench/master/toolbench/tooleval/results/leaderboard%23%23%23default_evalset%23%23%23tooleval_gpt-3.5-turbo_normalized%23%23%23ChatGPT-DFSDT.csv',
}
var myRadarChart = null;
var myBarChart=null;
var backgroundColors=null;
var borderColors=null;
leaderboardSelect.addEventListener('change', function() {
  radarSelect.value = leaderboardSelect.value;
  updateLeaderboard(records_urls[leaderboardSelect.value]);
});
radarSelect.addEventListener('change',function(){
  leaderboardSelect.value = radarSelect.value;
  updateLeaderboard(records_urls[leaderboardSelect.value]);
});
updateLeaderboard(records_urls[leaderboardSelect.value]);
```

--------------------------------

### Inference with Text-Davinci-003

Source: https://github.com/openbmb/toolbench/blob/master/README.md

Execute this command for inference using the Text-Davinci-003 model. Set the appropriate environment variables.

```bash
export TOOLBENCH_KEY=""
export OPENAI_KEY=""
export PYTHONPATH=./
python toolbench/inference/qa_pipeline.py \
    --tool_root_dir data/toolenv/tools/ \
    --backbone_model davinci \
    --openai_key $OPENAI_KEY \
    --max_observation_length 1024 \
    --method DFS_woFilter_w2 \
    --input_query_file data/test_instruction/G1_instruction.json \
    --output_answer_file davinci_dfs_inference_result \
    --toolbench_key $TOOLBENCH_KEY
```

--------------------------------

### Inference with RapidAPI Key

Source: https://github.com/openbmb/toolbench/blob/master/README.md

Run inference using your own RapidAPI account by providing your key and specifying the `use_rapidapi_key` argument.

```bash
export RAPIDAPI_KEY=""
export OPENAI_KEY=""
export PYTHONPATH=./
python toolbench/inference/qa_pipeline.py \
    --tool_root_dir data/toolenv/tools/ \
    --backbone_model chatgpt_function \
    --openai_key $OPENAI_KEY \
    --max_observation_length 1024 \
    --method DFS_woFilter_w2 \
    --input_query_file data/test_instruction/G1_instruction.json \
    --output_answer_file chatgpt_dfs_inference_result \
    --rapidapi_key $RAPIDAPI_KEY \
    --use_rapidapi_key
```

--------------------------------

### Inference with ToolLLaMA

Source: https://github.com/openbmb/toolbench/blob/master/README.md

Performs inference using the ToolLLaMA model. Requires specifying tool directories, model paths, and input/output files. Ensure PYTHONPATH is set and TOOLBENCH_KEY is exported.

```bash
export PYTHONPATH=./
python toolbench/inference/qa_pipeline.py \
    --tool_root_dir data/toolenv/tools/ \
    --backbone_model toolllama \
    --model_path ToolBench/ToolLLaMA-7b \
    --max_observation_length 1024 \
    --observ_compress_method truncate \
    --method DFS_woFilter_w2 \
    --input_query_file data/test_instruction/G1_instruction.json \
    --output_answer_file toolllama_dfs_inference_result \
    --toolbench_key $TOOLBENCH_KEY
```

--------------------------------

### Preprocess ToolLLaMA Data

Source: https://github.com/openbmb/toolbench/blob/master/README.md

Preprocesses data for ToolLLaMA training using a specified method and outputs to a JSON file. Ensure the PYTHONPATH is set correctly.

```bash
export PYTHONPATH=./
python preprocess/preprocess_toolllama_data.py \
    --tool_data_dir data/answer/G1_answer \
    --method DFS_woFilter_w2 \
    --output_file data/answer/toolllama_G1_dfs.json
```

--------------------------------

### Train ToolLLaMA-7b Model

Source: https://github.com/openbmb/toolbench/blob/master/README.md

Trains the ToolLLaMA-7b model using FastChat-based code with preprocessed data. Requires specific hardware (e.g., 2 x A100) and configuration. Ensure PYTHONPATH is set.

```bash
export PYTHONPATH=./
torchrun --nproc_per_node=2 --master_port=20001 toolbench/train/train_mem.py \
    --model_name_or_path huggyllama/llama-7b  \
    --data_path  data/toolllama_G123_dfs_train.json \
    --eval_data_path  data/toolllama_G123_dfs_eval.json \
    --conv_template tool-llama-single-round \
    --bf16 True \
    --output_dir toolllama \
    --num_train_epochs 2 \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 2 \
    --gradient_accumulation_steps 8 \
    --evaluation_strategy "epoch" \
    --prediction_loss_only \
    --save_strategy "epoch" \
    --save_total_limit 8 \
    --learning_rate 5e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.04 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --fsdp "full_shard auto_wrap" \
    --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
    --tf32 True \
    --source_model_max_length 2048 \
    --model_max_length 8192 \
    --gradient_checkpointing True \
    --lazy_preprocess True \
    --report_to none
```

--------------------------------

### Preprocess Retriever Data

Source: https://github.com/openbmb/toolbench/blob/master/README.md

Preprocess data for training the tool retriever. This script requires specifying query and index files, dataset name, and an output directory.

```bash
export PYTHONPATH=./
python preprocess/preprocess_retriever_data.py \
    --query_file data/instruction/G1_query.json \
    --index_file data/test_query_ids/G1_instruction_test_query_ids.json \
    --dataset_name G1 \
    --output_dir data/retrieval/G1
```

--------------------------------

### Inference with ToolLLaMA-LoRA (Open-Domain)

Source: https://github.com/openbmb/toolbench/blob/master/README.md

Performs open-domain inference with a LoRA-enabled ToolLLaMA model. Requires additional parameters for retrieval and corpus paths. Ensure PYTHONPATH is set and TOOLBENCH_KEY is exported.

```bash
export PYTHONPATH=./
python toolbench/inference/qa_pipeline_open_domain.py \
    --tool_root_dir data/toolenv/tools/ \
    --corpus_tsv_path data/retrieval/G1/corpus.tsv \
    --retrieval_model_path /path/to/your/retrival_model \
    --retrieved_api_nums 5 \
    --backbone_model toolllama \
    --model_path huggyllama/llama-7b \
    --lora \
    --lora_path /path/to/your/toolllama_lora \
    --max_observation_length 1024 \
    --observ_compress_method truncate \
    --method DFS_woFilter_w2 \
    --input_query_file data/test_instruction/G1_instruction.json \
    --output_answer_file toolllama_lora_dfs_open_domain_inference_result \
    --toolbench_key $TOOLBENCH_KEY
```

--------------------------------

### RapidAPI Wrapper Implementation

Source: https://context7.com/openbmb/toolbench/llms.txt

A wrapper class for interacting with RapidAPI services within the ToolBench environment. It handles API action execution and converts API specifications into OpenAI function calling formats.

```python
import json
import requests
from toolbench.inference.Downstream_tasks.base_env import base_env

class rapidapi_wrapper(base_env):
    def __init__(self, query_json, tool_descriptions, retriever, args, process_id=0):
        """
        Initialize RapidAPI execution environment.

        Args:
            query_json: Dict with 'query' and 'api_list'
            tool_descriptions: List of [tool_name, description] pairs
            retriever: Optional ToolRetriever for open-domain
            args: Configuration arguments
        """
        self.service_url = "http://8.130.32.149:8080/rapidapi"
        self.toolbench_key = args.toolbench_key
        self.input_description = query_json["query"]
        self.functions = self._build_openai_functions(query_json, tool_descriptions)

    def step(self, action_name="", action_input=""):
        """
        Execute an API action.

        Args:
            action_name: Name of the function to call
            action_input: JSON string of function arguments

        Returns:
            tuple: (observation_string, status_code)
            Status codes: 0=success, 1=invalid_function, 2=invalid_input,
                         3=final_answer, 4=give_up, 5=timeout, 6=api_error
        """
        if action_name == "Finish":
            return self._handle_finish(action_input)

        payload = {
            "category": self.cate_names[k],
            "tool_name": self.tool_names[k],
            "api_name": action_name,
            "tool_input": action_input,
            "toolbench_key": self.toolbench_key
        }

        response = requests.post(self.service_url, json=payload, timeout=15)
        return json.dumps(response.json()), 0

    def api_json_to_openai_json(self, api_json, standard_tool_name):
        """Convert API specification to OpenAI function calling format."""
        return {
            "name": f"{api_json['api_name']}_for_{standard_tool_name}",
            "description": f"Subfunction for {standard_tool_name}: {api_json['api_description'][:256]}",
            "parameters": {
                "type": "object",
                "properties": {
                    param["name"]: {
                        "type": self._map_type(param["type"]),
                        "description": param["description"][:256]
                    }
                    for param in api_json["required_parameters"]
                },
                "required": [p["name"] for p in api_json["required_parameters"]]
            }
        }
```

--------------------------------

### Preprocess Model Predictions

Source: https://github.com/openbmb/toolbench/blob/master/toolbench/tooleval/README.md

Commands to convert raw model predictions into the format required for ToolEval.

```bash
export RAW_ANSWER_PATH=../../data/reproduction_data/model_predictions/
export CONVERTED_ANSWER_PATH=../../data/reproduction_data/model_predictions_converted/
export MODEL_NAME=chatgpt_cot
export METHOD=CoT
mkdir ${CONVERTED_ANSWER_PATH}/${MODEL_NAME}
for test_set in G1_instruction G1_category G1_tool G2_category G2_instruction G3_instruction
do
    answer_dir=${RAW_ANSWER_PATH}/${MODEL_NAME}/${test_set}
    output_file=${CONVERTED_ANSWER_PATH}/${MODEL_NAME}/${test_set}.json
    python convert_to_answer_format.py\
        --answer_dir ${answer_dir} \
        --method ${METHOD} \
        --output ${output_file}
done
```

--------------------------------

### Evaluate candidate model preference

Source: https://context7.com/openbmb/toolbench/llms.txt

Run the preference evaluation script to compare a candidate model against a reference model using specified test IDs.

```bash
export REFERENCE_MODEL=chatgpt_cot
export CANDIDATE_MODEL=gpt-4-0613_cot
export API_POOL_FILE=path/to/openai_keys.json

python toolbench/tooleval/eval_preference.py \
    --converted_answer_path ${CONVERTED_ANSWER_PATH} \
    --reference_model ${REFERENCE_MODEL} \
    --output_model ${CANDIDATE_MODEL} \
    --test_ids data/test_ids/ \
    --save_path preference_results \
    --pass_rate_result_path pass_rate_results \
    --max_eval_threads 20 \
    --use_pass_rate true \
    --evaluate_times 7
```

--------------------------------

### Directory Structure for Model Predictions

Source: https://github.com/openbmb/toolbench/blob/master/toolbench/tooleval/README.md

The expected file hierarchy for model predictions before running the conversion script.

```text
├── /chatgpt_cot/
│  ├── /G1_instruction/
│  │  ├── /10160_CoT@1.json
│  │  └── ...
│  ├── /G1_tool/
│  │  ├── /10221_CoT@1.json
│  │  └── ...
│  ├── ...
│  ├── /G3_instruction/
│  │  ├── /10221_CoT@1.json
│  │  └── ...
```

--------------------------------

### Train LoRA with DeepSpeed

Source: https://context7.com/openbmb/toolbench/llms.txt

Executes LoRA training for Llama models using DeepSpeed distributed training configurations.

```bash
export PYTHONPATH=./
deepspeed --master_port=20001 toolbench/train/train_lora.py \
    --model_name_or_path huggyllama/llama-7b \
    --data_path data/toolllama_G123_dfs_train.json \
    --eval_data_path data/toolllama_G123_dfs_eval.json \
    --conv_template tool-llama-single-round \
    --bf16 True \
    --output_dir toolllama_lora \
    --num_train_epochs 5 \
    --per_device_train_batch_size 4 \
    --per_device_eval_batch_size 2 \
    --gradient_accumulation_steps 2 \
    --evaluation_strategy "epoch" \
    --save_strategy "epoch" \
    --learning_rate 5e-5 \
    --warmup_ratio 0.04 \
    --lr_scheduler_type "cosine" \
    --source_model_max_length 2048 \
    --model_max_length 8192 \
    --gradient_checkpointing True \
    --deepspeed ds_configs/stage2.json
```

--------------------------------

### Inference with ToolLLaMA-LoRA

Source: https://github.com/openbmb/toolbench/blob/master/README.md

Performs inference using a LoRA-enabled ToolLLaMA model. Requires specifying the LoRA path and other inference parameters. Ensure PYTHONPATH is set and TOOLBENCH_KEY is exported.

```bash
export PYTHONPATH=./
python toolbench/inference/qa_pipeline.py \
    --tool_root_dir data/toolenv/tools/ \
    --backbone_model toolllama \
    --model_path huggyllama/llama-7b \
    --lora \
    --lora_path /path/to/your/downloaded/ToolLLaMA-7b-LoRA \
    --max_observation_length 1024 \
    --observ_compress_method truncate \
    --method DFS_woFilter_w2 \
    --input_query_file data/test_instruction/G1_instruction.json \
    --output_answer_file toolllama_lora_dfs_inference_result \
    --toolbench_key $TOOLBENCH_KEY
```

--------------------------------

### Define input query structure

Source: https://context7.com/openbmb/toolbench/llms.txt

Format input queries as a JSON array specifying the query text, ID, and required APIs.

```json
[
    {
        "query": "I'm working on a logistics project. Can you check the SQUAKE API health and retrieve available projects?",
        "query_id": 1,
        "api_list": [
            {
                "category_name": "Logistics",
                "tool_name": "SQUAKE",
                "api_name": "Checkhealth"
            },
            {
                "category_name": "Logistics",
                "tool_name": "SQUAKE",
                "api_name": "Projects"
            }
        ],
        "relevant APIs": [
            ["SQUAKE", "Checkhealth"],
            ["SQUAKE", "Projects"]
        ]
    },
    {
        "query": "Find weather information for Tokyo and convert 100 USD to JPY.",
        "query_id": 2,
        "api_list": [
            {
                "category_name": "Weather",
                "tool_name": "weather_api",
                "api_name": "get_current_weather"
            },
            {
                "category_name": "Finance",
                "tool_name": "currency_converter",
                "api_name": "convert"
            }
        ]
    }
]
```

--------------------------------

### Set ToolBench Key

Source: https://github.com/openbmb/toolbench/blob/master/README.md

Sets the TOOLBENCH_KEY environment variable, which is required for inference. Obtain your key by filling out the provided form.

```bash
export TOOLBENCH_KEY="your_toolbench_key"
```

--------------------------------

### Train Tool Retriever

Source: https://github.com/openbmb/toolbench/blob/master/README.md

Train the tool retriever model using preprocessed data. This command specifies data path, model name, output path, training epochs, batch size, learning rate, warmup steps, and max sequence length.

```bash
export PYTHONPATH=./
python toolbench/retrieval/train.py \
    --data_path data/retrieval/G1/ \
    --model_name bert-base-uncased \
    --output_path retrieval_model \
    --num_epochs 5 \
    --train_batch_size 32 \
    --learning_rate 2e-5 \
    --warmup_steps 500 \
    --max_seq_length 256
```

--------------------------------

### Evaluate Model Pass Rate with ToolEval

Source: https://context7.com/openbmb/toolbench/llms.txt

Evaluates the pass rate of models using ToolEval. Requires converted answer paths, reference model name, test IDs, and specifies evaluation threads and times.

```bash
# Evaluate pass rate
python toolbench/tooleval/eval_pass_rate.py \
    --converted_answer_path ${CONVERTED_ANSWER_PATH} \
    --save_path pass_rate_results \
    --reference_model ${MODEL_NAME} \
    --test_ids data/test_ids/ \
    --max_eval_threads 20 \
    --evaluate_times 7
```

--------------------------------

### Implement Tool Retriever for Inference

Source: https://context7.com/openbmb/toolbench/llms.txt

Provides a class for semantic API retrieval using a pre-trained sentence transformer model.

```python
# toolbench/inference/LLM/retriever.py
from sentence_transformers import SentenceTransformer, util
import pandas as pd

class ToolRetriever:
    def __init__(self, corpus_tsv_path, model_path):
        """
        Initialize tool retriever with corpus and trained model.

        Args:
            corpus_tsv_path: Path to API corpus TSV file
            model_path: Path to trained SentenceTransformer model
        """
        self.embedder = SentenceTransformer(model_path)
        documents_df = pd.read_csv(corpus_tsv_path, sep='\t')
        self.corpus, self.corpus2tool = self._process_corpus(documents_df)
        self.corpus_embeddings = self.embedder.encode(self.corpus, convert_to_tensor=True)

    def retrieving(self, query, top_k=5, excluded_tools={}):
        """
        Retrieve top-k relevant APIs for a given query.

        Args:
            query: Natural language user query
            top_k: Number of APIs to retrieve
            excluded_tools: Dict of tools to exclude {category: [tool_names]}

        Returns:
            List of dicts with category, tool_name, api_name
        """
        query_embedding = self.embedder.encode(query, convert_to_tensor=True)
        hits = util.semantic_search(
            query_embedding,
            self.corpus_embeddings,
            top_k=10*top_k,
            score_function=util.cos_sim
        )

        retrieved_tools = []
        for hit in hits[0]:
            category, tool_name, api_name = self.corpus2tool[self.corpus[hit['corpus_id']]].split('\t')
            if category not in excluded_tools or tool_name not in excluded_tools.get(category, []):
                retrieved_tools.append({
                    "category": category,
                    "tool_name": tool_name,
                    "api_name": api_name
                })
                if len(retrieved_tools) >= top_k:
                    break
        return retrieved_tools

# Usage example
retriever = ToolRetriever(
    corpus_tsv_path="data/retrieval/G1/corpus.tsv",
    model_path="ToolBench/ToolBench_IR_bert_based_uncased"
)
apis = retriever.retrieving("Find weather information for New York", top_k=5)
```

--------------------------------

### Preprocess Model Predictions for ToolEval

Source: https://context7.com/openbmb/toolbench/llms.txt

Preprocesses raw model predictions into the required format for ToolEval's pass rate evaluation. This involves converting answers for different test sets and methods.

```bash
# Preprocess model predictions
export RAW_ANSWER_PATH=data/reproduction_data/model_predictions/
export CONVERTED_ANSWER_PATH=data/reproduction_data/model_predictions_converted/
export MODEL_NAME=chatgpt_cot
export METHOD=CoT

mkdir -p ${CONVERTED_ANSWER_PATH}/${MODEL_NAME}
for test_set in G1_instruction G1_category G1_tool G2_category G2_instruction G3_instruction
do
    python toolbench/tooleval/convert_to_answer_format.py \
        --answer_dir ${RAW_ANSWER_PATH}/${MODEL_NAME}/${test_set} \
        --method ${METHOD} \
        --output ${CONVERTED_ANSWER_PATH}/${MODEL_NAME}/${test_set}.json
done
```

--------------------------------

### Check task solvability

Source: https://github.com/openbmb/toolbench/blob/master/toolbench/tooleval/evaluators/tooleval_gpt-3.5-turbo_default/template.txt

Determines if a task is 'Solvable' or 'Unsolvable' based on input validity and tool availability.

```text
<function>
<name>check_task_solvable</name>
<description>
Please check whether the given task solvable with following rules:
1. If the `query` provide invalid information (e.g. invalid email address or phone number), return "Unsolvable"
2. If the `query` needs more information to solve (e.g. the target restaurant name in a navigation task), return "Unsolvable"
3. If you are unable to draw a conclusion, return "Unsure"
4. If the currently `available_tools` are enough to solve the query, return "Solvable"

Task:
{task}

Now give your reason in "content" and `task_status` of JSON to `check_task_solvable`.
</description>
</function>
```

--------------------------------

### Update Leaderboard Data and Plot Charts

Source: https://github.com/openbmb/toolbench/blob/master/docs/index.html

Fetches data from a given URL using PapaParse, assigns colors based on the data, and then plots a bar chart and a radar chart. Handles success and potential errors during data fetching.

```javascript
function updateLeaderboard(url){
  Papa.parse(url,{
    download: true,
    header: true,
    skipEmptyLines: true,
    complete: function(results) {
      if (results.data.length>0){
        console.log("Success to downlad file.")
        const data = results.data
        assignColors(data);
        plotBarChart(document.getElementById('leaderboard'),data);
        plotRadarChart(document.getElementById('radarchart'), data);
      }
    }
  })
}
```

--------------------------------

### Plot Radar Chart for Win Rate Subsets

Source: https://github.com/openbmb/toolbench/blob/master/docs/index.html

Creates or updates a radar chart using the Chart.js library. It displays win rates across different subsets (G1 Tool, G1 Category, etc.) for various methods. Datasets are initially hidden except for the first few.

```javascript
function plotRadarChart(canvas, data) {
  if (myRadarChart){
    myRadarChart.destroy()
  }
  myRadarChart = new Chart(canvas, {
    type: 'radar',
    data: {
      labels: ['G1 Tool','G1 Category','G1 Instruction', 'G2 Category','G2 Instruction','G3 Instruction'],
      datasets: data.map((item,i)=>{
        var hidden = true;
        if (i<=5){
          hidden = false;
        }
        return {
          label: item.Method,
          data: [item.G1_tool_WinRate, item.G1_category_WinRate, item.G1_instruction_WinRate,item.G2_category_WinRate, item.G2_instruction_WinRate, item.G3_instruction_WinRate],
          backgroundColor:backgroundColors[i],
          borderColor:borderColors[i],
          fill: false,
          hidden:hidden
        }
      })
    },
    options: {
      plugins:{ 
        //title:{
        // display:true,
        // text:'Win Rate on Subsets',
        // font:{
        // size:20
        // }
        //},
        legend:{ 
        //position:'left', labels:{ font:{ size:16, weight:'bold' } }
        }
      },
      scales: {
        r: {
          //max:1.0,
          pointLabels:{ font:{ size:16, weight:'bold' }, },
        },
      },
    },
  })
}
```

--------------------------------

### OpenAI API Key Configuration

Source: https://github.com/openbmb/toolbench/blob/master/toolbench/tooleval/README.md

The JSON format for storing OpenAI credentials used by the evaluator.

```json
[
    {
        "username": "your_user_name",
        "passwd": "your_password",
        "api_key": "your_openai_key",
        "organization": "your_organization"
    },
    ...
]
```

--------------------------------

### Query File Format for Custom APIs

Source: https://github.com/openbmb/toolbench/blob/master/README.md

This JSON structure defines how to format your query file when using custom APIs. It includes the query, ID, and the specific API to be called.

```json
[
    {
        "query": "I want to get a 'hello world' string.",
        "query_id": 200001,
        "api_list": [
            {
                "category_name": "Customized",
                "tool_name": "hello world",
                "api_name": "get_hello_world"
            }
        ]
    }
]
```

--------------------------------

### Implement DFS Tree Search Algorithm

Source: https://context7.com/openbmb/toolbench/llms.txt

Defines the DFS_tree_search class for managing multi-step reasoning chains and tool execution. Requires base_search_method and tree structures from the project's internal modules.

```python
# toolbench/inference/Algorithms/DFS.py
from Algorithms.base_search import base_search_method
from Tree.Tree import my_tree, tree_node
from copy import deepcopy

class DFS_tree_search(base_search_method):
    def __init__(self, llm, io_func, process_id=0, callbacks=None):
        """
        Initialize DFS tree search.

        Args:
            llm: Language model for generating actions
            io_func: Environment wrapper for API execution
            process_id: Process identifier for parallel execution
            callbacks: List of callback handlers
        """
        super().__init__(llm, io_func, process_id, callbacks)
        self.terminal_node = []
        self.give_up_node = []
        self.query_count = 0

    def start(self, single_chain_max_step=12, tree_beam_size=2, max_query_count=60,
              answer=1, with_filter=True):
        """
        Start DFS tree search.

        Args:
            single_chain_max_step: Maximum depth of reasoning chain
            tree_beam_size: Number of children per node
            max_query_count: Maximum LLM queries before termination
            answer: Number of valid answers to find
            with_filter: True for DFS with ranking, False for DFSDT

        Returns:
            Search result with terminal nodes
        """
        self.tree = my_tree()
        self.tree.root.io_state = deepcopy(self.io_func)

        # Initialize with system and user prompts
        self.tree.root.messages.append({
            "role": "system",
            "content": self._format_system_prompt()
        })
        self.tree.root.messages.append({
            "role": "user",
            "content": self.io_func.input_description
        })

        return self.DFS(
            self.tree.root,
            single_chain_max_step,
            tree_beam_size,
            max_query_count,
            answer,
            with_filter
        )

    def to_json(self, answer=True, process=True):
        """Export search results to JSON format for evaluation."""
        return {
            "win": len(self.terminal_node) > 0,
            "tree": self.tree.to_json_recursive(),
            "answer_generation": {
                "valid_data": any(not n.pruned for n in self.terminal_node),
                "query_count": self.query_count,
                "final_answer": self.terminal_node[0].description if self.terminal_node else ""
            }
        }
```

--------------------------------

### Evaluate Win Rate

Source: https://github.com/openbmb/toolbench/blob/master/toolbench/tooleval/README.md

Command to evaluate the preference result of a candidate model against a reference model.

```bash
export CONVERTED_ANSWER_PATH=../../data/reproduction_data/model_predictions_converted/
export SAVE_PATH=preference_results
export PASS_TARE_PATH=pass_rate_results
export REFERENCE_MODEL=chatgpt_cot
export CANDIDATE_MODEL=gpt-4-0613_cot
export API_POOL_FILE=path/to/your/openai_key_json_file.json

python eval_preference.py \
    --converted_answer_path ${CONVERTED_ANSWER_PATH} \
    --reference_model ${REFERENCE_MODEL} \
    --output_model ${CANDIDATE_MODEL} \
    --test_ids ../../data/test_ids/ \
    --save_path ${SAVE_PATH} \
    --pass_rate_result_path ${PASS_TARE_PATH} \
    --max_eval_threads 20 \
    --use_pass_rate true \
    --evaluate_times 4
```

--------------------------------

### Plot Bar Chart for Overall Win Rate

Source: https://github.com/openbmb/toolbench/blob/master/docs/index.html

Creates or updates a bar chart using the Chart.js library to display the overall win rate for different methods. It takes a canvas element and the data as input.

```javascript
function plotBarChart(canvas,data) {
  if (myBarChart){
    myBarChart.destroy();
  }
  myBarChart = new Chart(canvas, {
    type: 'bar',
    data: {
      labels: data.map((item,i)=>item.Method),
      datasets: [
        {
          label: 'Win Rate',
          data: data.map((item,i)=>item.WinRate),
          //sems: sems,
          backgroundColor: ba
```