### EasyTool Setup and ToolBench Evaluation

Source: https://context7.com/microsoft/jarvis/llms.txt

Set up EasyTool by installing requirements and unzipping tool embeddings. Then, run ToolBench evaluation for G2 and G3 data types.

```bash
cd easytool
pip install -r requirements.txt
export OPENAI_API_KEY="your-openai-key"
export RAPIDAPI_KEY="your-rapidapi-key"

unzip data_toolbench/tool_instruction/API_description_embeddings.zip \
    -d data_toolbench/tool_instruction/

python main.py \
    --model_name gpt-3.5-turbo \
    --task toolbench \
    --data_type G2 \
    --tool_root_dir ./toolenv/tools

python main.py \
    --model_name gpt-3.5-turbo \
    --task toolbench \
    --data_type G3 \
    --tool_root_dir ./toolenv/tools
```

--------------------------------

### Start Jarvis Web Client

Source: https://github.com/microsoft/jarvis/blob/main/hugginggpt/README.md

Start the development server for the Jarvis web client. Ensure Node.js and npm are installed.

```bash
npm run dev
```

--------------------------------

### Start HuggingGPT Server

Source: https://context7.com/microsoft/jarvis/llms.txt

Starts the HuggingGPT server using a default configuration. Navigate to the server directory and run the Python script.

```bash
cd hugginggpt/server
python awesome_chat.py --config configs/config.default.yaml --mode server
```

--------------------------------

### Server Setup and Model Download

Source: https://github.com/microsoft/jarvis/blob/main/README.md

Commands to set up the environment for the JARVIS server, including creating a conda environment, installing dependencies, and downloading necessary models for local or hybrid inference modes.

```bash
cd server
conda create -n jarvis python=3.8
conda activate jarvis
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
pip install -r requirements.txt

cd models
bash download.sh
```

--------------------------------

### Start Jarvis Gradio Demo

Source: https://github.com/microsoft/jarvis/blob/main/hugginggpt/README.md

Run the Gradio demo locally by starting the models server with Gradio configuration and then launching the demo script.

```bash
python models_server.py --config configs/config.gradio.yaml
python run_gradio_demo.py --config configs/config.gradio.yaml
```

--------------------------------

### Install Web Client Dependencies

Source: https://github.com/microsoft/jarvis/blob/main/hugginggpt/README.md

Install the necessary Node.js packages for the Jarvis web client. This should be run after navigating to the 'web' directory.

```bash
cd web
npm install
```

--------------------------------

### Install FastChat and Dependencies

Source: https://github.com/microsoft/jarvis/blob/main/taskbench/README.md

Install necessary libraries for deploying LLMs locally using an OpenAI-compatible API. This includes fastchat, vllm, and fastapi.

```bash
pip install fastchat
pip install vllm
pip install "fastapi[all]"
```

--------------------------------

### Optional: Install ffmpeg for Video Generation

Source: https://github.com/microsoft/jarvis/blob/main/README.md

Command to install ffmpeg, which may be required for video generation features. Ensure this command executes without errors.

```bash
# Optional: Install ffmpeg
# This command need be executed without errors.
LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/ffmpeg -i input.mp4 -vcodec libx264 output.mp4
```

--------------------------------

### Install and Run Jarvis Web Client

Source: https://github.com/microsoft/jarvis/blob/main/README.md

Steps to set up and run the Jarvis web client. Requires Node.js and npm. Ensure the `HUGGINGGPT_BASE_URL` is correctly configured if the client is on a different machine.

```bash
cd web
npm install
npm run dev
```

--------------------------------

### TaskBench Inference Setup and Execution

Source: https://context7.com/microsoft/jarvis/llms.txt

This bash script demonstrates setting up the TaskBench environment and running inference on the TaskBench dataset. It configures LLM, data paths, and inference parameters.

```bash
# Setup environment
cd taskbench
conda create -n taskbench python=3.8
conda activate taskbench
pip install -r requirements.txt

# Run inference on TaskBench dataset
export YOUR_API_KEY=your-api-key
python inference.py \
    --llm gpt-4 \
    --data_dir data_multimedia \
    --temperature 0.2 \
    --top_p 0.1 \
    --api_addr localhost \
    --api_port 4000 \
    --api_key $YOUR_API_KEY \
    --multiworker 5 \
    --use_demos 2 \
    --reformat true \
    --reformat_by self \
    --dependency_type resource \
    --tag true
```

--------------------------------

### Start Jarvis Models Server

Source: https://github.com/microsoft/jarvis/blob/main/hugginggpt/README.md

Run the models server, required for local or hybrid inference modes. Use the default configuration.

```bash
cd ..
python models_server.py --config configs/config.default.yaml
```

--------------------------------

### Run Gradio Demo Locally

Source: https://github.com/microsoft/jarvis/blob/main/README.md

Commands to start the Gradio demo locally. This involves running two Python scripts sequentially or using a Docker image.

```bash
python models_server.py --config configs/config.gradio.yaml
python run_gradio_demo.py --config configs/config.gradio.yaml
```

```bash
# or run the HF Space as a Docker image (Build with `inference_mode=hibrid` and `local_deployment=standard`)
docker run -it -p 7860:7860 --platform=linux/amd64 registry.hf.space/microsoft-hugginggpt:latest python app.py
```

--------------------------------

### Start Awesome Chat on Jetson

Source: https://github.com/microsoft/jarvis/blob/main/README.md

Executes the awesome_chat.py script within the running JARVIS container to start the chat server. This command assumes the model server has completed initialization.

```bash
# (wait for model server to complete initialization)

# start awesome_chat.py 
docker exec jarvis python3 awesome_chat.py --config configs/config.default.yaml --mode server

```

--------------------------------

### Start JARVIS Web Application on Jetson

Source: https://github.com/microsoft/jarvis/blob/main/README.md

Launches the JARVIS web application inside the container. The application will be accessible at http://localhost:9999.

```bash
#start the web application (application will be acessible at http://localhost:9999)
docker exec jarvis npm run dev --prefix=/app/web

```

--------------------------------

### Start Jarvis Chat Server

Source: https://github.com/microsoft/jarvis/blob/main/hugginggpt/README.md

Launch the chat interface in server mode, specifically for text-davinci-003. Use the default configuration.

```bash
python awesome_chat.py --config configs/config.default.yaml --mode server
```

--------------------------------

### TaskBench Dataset Generation Command

Source: https://context7.com/microsoft/jarvis/llms.txt

This bash command initiates the generation of new TaskBench datasets using the Back-Instruct method, starting from tool descriptions.

```bash
# Generate tool graph from tool descriptions
python generate_graph.py \
    --tool_desc tool_desc.json \
    --dependency_type resource \
    --data_dir data_multimedia
```

--------------------------------

### Start Jarvis Server

Source: https://github.com/microsoft/jarvis/blob/main/README.md

Run the Python scripts to start the Jarvis server. The `models_server.py` script is required for local or hybrid inference modes. `awesome_chat.py` is used for text-davinci-003 in server mode.

```bash
cd ..
python models_server.py --config configs/config.default.yaml
```

```bash
python awesome_chat.py --config configs/config.default.yaml --mode server
```

--------------------------------

### Start FastChat Services

Source: https://github.com/microsoft/jarvis/blob/main/taskbench/README.md

Launch the FastChat controller, a vLLM worker for model serving, and an OpenAI-compatible API server. These commands are essential for local LLM deployment.

```bash
python3 -m fastchat.serve.controller
python3 -m fastchat.serve.vllm_worker --model-path lmsys/vicuna-7b-v1.3
python3 -m fastchat.serve.openai_api_server --host localhost --port 4000
```

--------------------------------

### Jarvis CLI Interaction Examples

Source: https://github.com/microsoft/jarvis/blob/main/README.md

Illustrates typical user requests and Jarvis's responses in CLI mode, showcasing its capabilities in image analysis, named entity recognition, and image generation.

```text
Welcome to Jarvis! A collaborative system that consists of an LLM as the controller and numerous expert models as collaborative executors. Jarvis can plan tasks, schedule Hugging Face models, generate friendly responses based on your requests, and help you with many things. Please enter your request (`exit` to exit).
[ User ]: Given a collection of image A: /examples/a.jpg, B: /examples/b.jpg, C: /examples/c.jpg, please tell me how many zebras in these picture?
[ Jarvis ]: Based on your request, I have used several models to process the images you provided. For the image A: /examples/a.jpg, I used the image-to-text model nlpconnect/vit-gpt2-image-captioning to generate the text "a cat sitting on a window sill looking out". Then I used the object-detection model facebook/detr-resnet-50 to generate the image with predicted box '/images/a3dc.jpg', which contains two objects with labels 'potted plant' and 'cat'. For the image B: /examples/b.jpg, I used the image-to-text model nlpconnect/vit-gpt2-image-captioning to generate the text "two zebras standing in a field of dry grass". Then I used the object-detection model facebook/detr-resnet-50 to generate the image with predicted box '/images/f5df.jpg', which contains three objects with labels 'zebra'. For the image C: /examples/c.jpg, I used the image-to-text model nlpconnect/vit-gpt2-image-captioning to generate the text "a zebra standing in a field of tall grass". Then I used the object-detection model facebook/detr-resnet-50 to generate the image with predicted box '/images/69c7.jpg', which contains one object with label 'zebra'. Finally, I used the canny-control model canny-control to generate the image '/images/7e18.png' based on the image /examples/f.jpg. Therefore, there are four zebras in these pictures. Is there anything else I can help you with?
[ User ]: Please answer all the named entities in the sentence: Iron Man is a superhero appearing in American comic books published by Marvel Comics. The character was co-created by writer and editor Stan Lee, developed by scripter Larry Lieber, and designed by artists Don Heck and Jack Kirby.
[ Jarvis ]: Sure, I understand your request. Based on the inference results of the models, I will answer your request step by step. Firstly, I used the token-classification model dslim/bert-base-NER to recognize the named entities in the sentence. The inference result is that the named entities are Iron Man, Stan Lee, Larry Lieber, Don Heck and Jack Kirby. Then, I used the question-answering model deepset/roberta-base-squad2 to answer your request. The inference result is that there is no output since the context cannot be empty. Therefore, I cannot make it. I hope this answer your request. Is there anything else I can help you with?
[ User ]: Please generate a canny image based on /examples/f.jpg
[ Jarvis ]: Sure. I have generated a canny image based on /examples/f.jpg. To do this, I first used the image-to-text model nlpconnect/vit-gpt2-image-captioning to generate the text description of the image, which is "a herd of giraffes and zebras grazing in a field". Then I used the canny-control model to generate the canny image of the original image, which is stored in the path "/images/1ea2.png". Finally, I used the canny-text-to-image model lllyasviel/sd-controlnet-canny to generate the canny image based on the text description and the canny image, which is stored in the path "/images/ba66.png". Do you need anything else?
```

--------------------------------

### Run Gradio Demo as Docker Image

Source: https://github.com/microsoft/jarvis/blob/main/hugginggpt/README.md

Deploy the Hugging Face Space version of the Jarvis Gradio demo as a Docker container. Ensure Docker is installed and the image is built with the correct inference mode and deployment settings.

```bash
docker run -it -p 7860:7860 --platform=linux/amd64 registry.hf.space/microsoft-hugginggpt:latest python app.py
```

--------------------------------

### Start HuggingGPT CLI Mode

Source: https://context7.com/microsoft/jarvis/llms.txt

Launches the HuggingGPT interactive terminal interface. This mode maintains conversation history and supports multi-turn dialogues.

```bash
cd hugginggpt/server
python awesome_chat.py --config configs/config.default.yaml --mode cli
```

--------------------------------

### Define Tool Descriptions for Dataset Construction

Source: https://github.com/microsoft/jarvis/blob/main/taskbench/README.md

JSON examples for defining tool descriptions. Supports type-specific parameters for resource dependency graphs and API request parameters for temporal dependency graphs.

```json
// Tool with type-specific parameters
{
  "id": "Image-to-Image",
  "desc": "Image-to-image is the task of transforming a source image to match the characteristics of a target image or a target image domain. Any image manipulation and enhancement is possible with image to image models.",
  "input-type": [
    "image"
  ],
  "output-type": [
    "image"
  ]
}
// API with request parameters
{
  "id": "send_sms",
  "desc": "Send an sms to a specific phone number",
  "parameters": [
    {
      "name": "phone_number",
      "type": "string",
      "desc": "The phone number to send the sms to"
    },
    {
      "name": "content",
      "type": "string",
      "desc": "The content of the sms"
    }
  ]
}
```

--------------------------------

### Run JARVIS Container on Jetson Orin AGX

Source: https://github.com/microsoft/jarvis/blob/main/README.md

Starts the JARVIS container, automatically launching the model server. This command requires specific configurations for Jetson Orin AGX devices and mounts local model and config directories into the container. The container will be accessible via the host network.

```bash
# run the container which will automatically start the model server
docker run --name jarvis --net=host --gpus all -v ~/jarvis/configs:/app/server/configs -v ~/src/JARVIS/server/models:/app/server/models toolboc/nv-jarvis:r35.2.1

```

--------------------------------

### TMDB Tool Instruction Format

Source: https://context7.com/microsoft/jarvis/llms.txt

This JSON array defines tools for the TMDB API, including their IDs, descriptions, parameters, and usage examples.

```json
[
    {
        "ID": "search_movie",
        "desc": "Search for movies by title",
        "parameters": [
            {"name": "query", "type": "string", "required": true},
            {"name": "year", "type": "integer", "required": false}
        ],
        "example": "search_movie(query='Inception', year=2010)"
    }
]
```

--------------------------------

### Interact with Jarvis API using curl

Source: https://github.com/microsoft/jarvis/blob/main/README.md

Example of how to make a POST request to the `/tasks` endpoint of the Jarvis API using curl. This demonstrates sending a user query with image and text content.

```bash
# request
curl --location 'http://localhost:8004/tasks' \
--header 'Content-Type: application/json' \
--data '{ 
    "messages": [ 
        { 
            "role": "user", 
            "content": "based on pose of /examples/d.jpg and content of /examples/e.jpg, please show me a new image" 
        } 
    ]
}'
```

```json
# response
[{"args":{"image":"/examples/d.jpg"},"dep":[-1],"id":0,"task":"openpose-control"},{"args":{"image":"/examples/e.jpg"},"dep":[-1],"id":1,"task":"image-to-text"},{"args":{"image":"<GENERATED>-0","text":"<GENERATED>-1"},"dep":[1,0],"id":2,"task":"openpose-text-to-image"}]
```

--------------------------------

### Interact with Jarvis API using Curl

Source: https://github.com/microsoft/jarvis/blob/main/hugginggpt/README.md

Example of making a POST request to the /tasks endpoint of the Jarvis Web API to process an image-based request. Ensure the server is running.

```bash
# request
curl --location 'http://localhost:8004/tasks' \
--header 'Content-Type: application/json' \
--data '{ 
    "messages": [ 
        { 
            "role": "user", 
            "content": "based on pose of /examples/d.jpg and content of /examples/e.jpg, please show me a new image"
        } 
    ]
}'

# response
[{"args":{"image":"/examples/d.jpg"},"dep":[-1],"id":0,"task":"openpose-control"},{"args":{"image":"/examples/e.jpg"},"dep":[-1],"id":1,"task":"image-to-text"},{"args":{"image":"<GENERATED>-0","text":"<GENERATED>-1"},"dep":[1,0],"id":2,"task":"openpose-text-to-image"}]
```

--------------------------------

### TaskBench Evaluation Metrics Output

Source: https://context7.com/microsoft/jarvis/llms.txt

This is an example of the evaluation metrics output from TaskBench, showing scores for step support, node F1, link F1, argument prediction, and text similarity metrics.

```text
# Evaluation metrics output
# Tools Number: overall, Task Split: overall
# Step Supports: 5584 / 5584
# Node Micro F1 [ No Matching ]: 0.9090
# Link Binary F1: 0.6927
# Argument Task-ArgName-Value Binary F1: 0.7231
# Step rouge1: 0.6084
# Step BERTScore f1: 0.9119

# Results saved to data_multimedia/metrics/gpt-4.json
```

--------------------------------

### Clone and Navigate Repository

Source: https://github.com/microsoft/jarvis/blob/main/CONTRIBUTING.md

Use these commands to clone a forked repository and navigate into its directory.

```bash
git clone https://github.com/YOUR-GITHUB-USERNAME/JARVIS
cd JARVIS
```

--------------------------------

### Generate Dataset with Back-Instruct

Source: https://context7.com/microsoft/jarvis/llms.txt

Use this command to generate a dataset using the Back-Instruct method. Ensure you have the necessary graph and tool description JSON files.

```bash
python data_engine.py \
    --graph_desc data_multimedia/graph_desc.json \
    --tool_desc data_multimedia/tool_desc.json \
    --llm gpt-4 \
    --temperature 1.0 \
    --top_p 1.0 \
    --dependency_type resource \
    --api_addr localhost \
    --api_port 4000 \
    --api_key $YOUR_API_KEY \
    --check true \
    --use_async true \
    --multiworker 5 \
    --number_of_samples 5000
```

--------------------------------

### Set API Keys for Tool Usage

Source: https://github.com/microsoft/jarvis/blob/main/easytool/README.md

Before running any commands, set your API keys. Replace the placeholder values with your actual keys.

```bash
export OPENAI_API_KEY="your_openai_api_key_here"
export RAPIDAPI_KEY="your_rapidapi_key_here"
```

--------------------------------

### Recommended Personal Laptop Configuration

Source: https://github.com/microsoft/jarvis/blob/main/hugginggpt/README.md

For personal laptops, a hybrid inference mode with minimal local deployment is recommended. Be aware that remote Hugging Face Inference Endpoints can be unstable, potentially limiting available models.

```text
On a personal laptop, we recommend the configuration of inference_mode: hybrid  and local_deployment: minimal. But the available models under this setting may be limited due to the instability of remote Hugging Face Inference Endpoints.
```

--------------------------------

### Run Lite Jarvis with Configuration

Source: https://github.com/microsoft/jarvis/blob/main/README.md

Execute Jarvis in a lightweight mode without local model deployment by using a lite configuration file. This mode relies on models running on HuggingFace Inference Endpoints.

```bash
python awesome_chat.py --config configs/config.lite.yaml
```

--------------------------------

### Jarvis Server Configuration Parameters

Source: https://github.com/microsoft/jarvis/blob/main/hugginggpt/README.md

This section outlines key parameters within the `server/configs/config.default.yaml` file, including model selection, inference modes (local, Hugging Face, hybrid), and local deployment scales based on available RAM.

```yaml
model: LLM, currently supports text-davinci-003. We are working on integrating more open-source LLMs.
inference_mode:
  local: only use the local inference endpoints
  huggingface: only use the Hugging Face Inference Endpoints **(free of local inference endpoints)**
  hybrid: both of local and huggingface
local_deployment:
  minimal (RAM>12GB, ControlNet only)
  standard (RAM>16GB, ControlNet + Standard Pipelines)
  full (RAM>42GB, All registered models)
```

--------------------------------

### EasyTool ToolBench Evaluation with Retrieval

Source: https://context7.com/microsoft/jarvis/llms.txt

Run ToolBench evaluation with retrieval enabled. Specify the number of retrieved tools to use.

```bash
python main.py \
    --model_name gpt-3.5-turbo \
    --task toolbench_retrieve \
    --data_type G2 \
    --retrieval_num 5 \
    --tool_root_dir ./toolenv/tools
```

--------------------------------

### ToolBench Inference Commands

Source: https://github.com/microsoft/jarvis/blob/main/easytool/README.md

Run these commands for ToolBench inference. Ensure you have downloaded the tool execution code and unzipped the API descriptions.

```bash
unzip data_toolbench/tool_instruction/API_description_embeddings.zip -d data_toolbench/tool_instruction/
```

```bash
export OPENAI_API_KEY=""
export RAPIDAPI_KEY=""
```

```bash
python3 main.py \
    --model_name gpt-3.5-turbo \
    --task toolbench \
    --data_type G2 \
    --tool_root_dir ./toolenv/tools
```

```bash
python3 main.py \
    --model_name gpt-3.5-turbo \
    --task toolbench \
    --data_type G3 \
    --tool_root_dir ./toolenv/tools
```

```bash
python3 main.py \
    --model_name gpt-3.5-turbo \
    --task toolbench_retrieve \
    --data_type G2 \
    --tool_root_dir ./toolenv/tools
```

```bash
python3 main.py \
    --model_name gpt-3.5-turbo \
    --task toolbench_retrieve \
    --data_type G3 \
    --tool_root_dir ./toolenv/tools
```

--------------------------------

### Generate Dataset with Data Engine

Source: https://github.com/microsoft/jarvis/blob/main/taskbench/README.md

Script to generate the dataset using the data engine. Requires specifying graph and tool description files, LLM parameters, and API details. Supports asynchronous operations and multi-worker processing.

```bash
# specify the graph and tool description file
python data_engine.py \
    --graph_desc data_multimedia/graph_desc.json \
    --tool_desc data_multimedia/tool_desc.json \
    --llm gpt-4 \
    --temperature 1.0 \
    --top_p 1.0 \
    --dependency_type resource \
    --save_figure false \
    --api_addr localhost \
    --api_port 4000 \
    --api_key $YOUR_API_KEY \
    --check true \
    --use_async true \
    --multiworker 5

# specify the data_dir to resume data generation
python data_engine.py \
    --data_dir result_20240317170826_gpt-4-32k_t1_0_p1_0_check \
    --llm gpt-4-32k \
    --temperature 1.0 \
    --top_p 1.0 \
    --dependency_type temporal \
    --save_figure false \
    --api_addr localhost \
    --api_port 4000 \
    --api_key $YOUR_API_KEY \
    --check true \
    --use_async true \
    --multiworker 5
```

--------------------------------

### HuggingGPT Default Configuration

Source: https://context7.com/microsoft/jarvis/llms.txt

This YAML configuration file sets up HuggingGPT, controlling LLM selection, inference modes, and model deployment. Ensure sensitive tokens are kept secure.

```yaml
# configs/config.default.yaml
openai:
  api_key: sk-your-openai-api-key-here

huggingface:
  token: hf_your-huggingface-token-here

# Model configuration
model: text-davinci-003  # or gpt-4, gpt-3.5-turbo
use_completion: true

# Inference mode: local, huggingface, or hybrid
inference_mode: hybrid

# Local deployment scale: minimal (12GB), standard (16GB), full (42GB+)
local_deployment: full

# Device configuration
device: cuda:0

# Server settings
http_listen:
  host: 0.0.0.0
  port: 8004

local_inference_endpoint:
  host: localhost
  port: 8005

# Model selection parameters
num_candidate_models: 5
max_description_length: 100
```

--------------------------------

### Build JARVIS Docker Image for Jetson

Source: https://github.com/microsoft/jarvis/blob/main/README.md

Use this command to build the Docker image with experimental support for NVIDIA Jetson embedded devices. Ensure the default Docker runtime is set to 'nvidia'.

```bash
#Build the docker image
docker build --pull --rm -f "Dockerfile.jetson" -t toolboc/nv-jarvis:r35.2.1 

```

--------------------------------

### FuncQA Inference Commands

Source: https://github.com/microsoft/jarvis/blob/main/easytool/README.md

Run these commands for FuncQA inference. Ensure your OpenAI API key is set.

```bash
export OPENAI_API_KEY=""
```

```bash
python3 main.py \
    --model_name gpt-3.5-turbo \
    --task funcqa \
    --data_type funcqa_mh
```

```bash
python3 main.py \
    --model_name gpt-3.5-turbo \
    --task funcqa \
    --data_type funcqa_oh
```

--------------------------------

### EasyTool FuncQA One-hop Evaluation

Source: https://context7.com/microsoft/jarvis/llms.txt

Execute the FuncQA module for one-hop mathematical function question-answering. Ensure the OpenAI API key is set.

```bash
python main.py \
    --model_name gpt-3.5-turbo \
    --task funcqa \
    --data_type funcqa_oh
```

--------------------------------

### Generate Tool Graph

Source: https://github.com/microsoft/jarvis/blob/main/taskbench/README.md

Use this script to generate a tool graph from tool descriptions. Specify the dependency type (resource or temporal) and the data directory.

```bash
python generate_graph.py \
    --tool_desc tool_desc.json \
    --dependency_type resource \
    --data_dir data_multimedia
```

--------------------------------

### EasyTool FuncQA Multi-hop Evaluation

Source: https://context7.com/microsoft/jarvis/llms.txt

Execute the FuncQA module for multi-hop mathematical function question-answering. Ensure the OpenAI API key is set.

```bash
export OPENAI_API_KEY="your-openai-key"

python main.py \
    --model_name gpt-3.5-turbo \
    --task funcqa \
    --data_type funcqa_mh
```

--------------------------------

### Generated Output Format

Source: https://context7.com/microsoft/jarvis/llms.txt

This JSON represents the output generated by the Back-Instruct process, including task steps and user requests.

```json
{
    "id": "12345678",
    "seed": 42,
    "method": "chain",
    "number_of_tools": 3,
    "sampled_nodes": [...],
    "sampled_links": [...],
    "result": {
        "task_steps": ["Step 1: ...", "Step 2: ..."],
        "user_request": "Please convert my photo to a sketch and then colorize it",
        "invoking_graph": {
            "nodes": [...],
            "links": [...] 
        }
    }
}
```

--------------------------------

### Tool Description Format

Source: https://context7.com/microsoft/jarvis/llms.txt

This JSON structure defines tools, their descriptions, input/output types, and parameters for use with Back-Instruct.

```json
{
    "nodes": [
        {
            "id": "Image-to-Image",
            "desc": "Transform a source image to match target characteristics",
            "input-type": ["image"],
            "output-type": ["image"]
        },
        {
            "id": "send_sms",
            "desc": "Send an SMS to a specific phone number",
            "parameters": [
                {"name": "phone_number", "type": "string", "desc": "Target phone number"},
                {"name": "content", "type": "string", "desc": "SMS content"}
            ]
        }
    ]
}
```

--------------------------------

### Stage Changes

Source: https://github.com/microsoft/jarvis/blob/main/CONTRIBUTING.md

Add all modified files to the staging area before committing.

```bash
git add .
```

--------------------------------

### Run Jarvis in CLI Mode

Source: https://github.com/microsoft/jarvis/blob/main/README.md

Navigate to the server directory and execute the Python script with the desired configuration and mode. This enables interactive command-line usage.

```bash
cd server
python awesome_chat.py --config configs/config.default.yaml --mode cli
```

--------------------------------

### Format Dataset

Source: https://github.com/microsoft/jarvis/blob/main/taskbench/README.md

Format the generated dataset based on the specified data directory and dependency type.

```bash
python format_data.py \
    --data_dir data_multimedia \
    --dependency_type resource
```

--------------------------------

### EasyTool RestBench Evaluation

Source: https://context7.com/microsoft/jarvis/llms.txt

Run the RestBench module to evaluate LLM agents on RESTful API tasks using the TMDB API. Ensure the OpenAI API key is set.

```bash
export OPENAI_API_KEY="your-openai-key"

python main.py \
    --model_name gpt-3.5-turbo \
    --task restbench
```

--------------------------------

### HuggingGPT Full Service API Request

Source: https://context7.com/microsoft/jarvis/llms.txt

Sends a request to the full service endpoint to process an entire workflow, including task planning, model selection, execution, and response generation. Requires an OpenAI API key and endpoint.

```bash
curl --location 'http://localhost:8004/hugginggpt' \
--header 'Content-Type: application/json' \
--data '{ \
    "messages": [ \
        { \
            "role": "user", \
            "content": "Given an image /examples/a.jpg, please detect all objects and describe what you see" \
        } \
    ], \
    "api_key": "your-openai-key", \
    "api_type": "openai", \
    "api_endpoint": "https://api.openai.com/v1/chat/completions" \
}'
```

--------------------------------

### ToolBench Output Format

Source: https://context7.com/microsoft/jarvis/llms.txt

This JSONL format shows the output of ToolBench evaluation, including question, final answer, subtasks, and execution logs.

```json
{
    "ID": 1,
    "question": "What are the top movies released this year?",
    "final_answer": "Based on the API response, the top movies...",
    "subtask": [
        {"task": "Get current year movies", "id": 1, "dep": [-1]},
        {"task": "Sort by rating", "id": 2, "dep": [1]}
    ],
    "answer_subtask": [...],
    "check_index": 1,
    "execute_log": {
        "api_result_ls": [...],
        "call_result_ls": [...] 
    }
}
```

--------------------------------

### RestBench Inference Command

Source: https://github.com/microsoft/jarvis/blob/main/easytool/README.md

Run this command for RestBench inference. Ensure your OpenAI API key is set.

```bash
export OPENAI_API_KEY=""
```

```bash
python3 main.py \
    --model_name gpt-3.5-turbo \
    --task restbench 
```

--------------------------------

### Generate API Parameters with EasyTool

Source: https://context7.com/microsoft/jarvis/llms.txt

Employ `choose_parameter` to generate the correct parameters for a selected API call based on the user's question and the API's documentation. This function ensures that API calls are made with the right arguments.

```python
# API detail documentation
api_doc = {
    "name": "get_forecast",
    "description": "Get weather forecast for a location",
    "required_params": ["location", "days"],
    "optional_params": ["units"],
    "example": "get_forecast(location='Seattle', days=1)"
}

# Generate parameters for the API call
params = choose_parameter(API_instruction, "get_forecast", api_doc, question, "gpt-3.5-turbo")
# Returns: {"location": "Seattle", "days": 1}
```

--------------------------------

### Tool Dictionary Format for FuncQA

Source: https://context7.com/microsoft/jarvis/llms.txt

This JSONL format defines mathematical functions, their parameters, and return values for the FuncQA module.

```json
{"add": "Adds two numbers: add(a, b) returns a + b"}
{"multiply": "Multiplies two numbers: multiply(a, b) returns a * b"}
{"power": "Raises base to exponent: power(base, exp) returns base^exp"}
```

--------------------------------

### POST /hugginggpt

Source: https://github.com/microsoft/jarvis/blob/main/README.md

Access the full Jarvis services via the HuggingGPT API.

```APIDOC
## POST /hugginggpt

### Description
Access the full Jarvis services through this endpoint.

### Method
POST

### Endpoint
/hugginggpt

### Request Body
(Schema not provided in source text)

### Response
(Schema not provided in source text)
```

--------------------------------

### Create and Activate Conda Environment

Source: https://github.com/microsoft/jarvis/blob/main/taskbench/README.md

Use this command to create a new conda environment named 'taskbench' with Python 3.8 and activate it. This is the first step in setting up the evaluation environment.

```bash
conda create -n taskbench python=3.8
conda activate taskbench
pip install -r requirements.txt
```

--------------------------------

### Generate Context-Aware API Parameters with EasyTool

Source: https://context7.com/microsoft/jarvis/llms.txt

Use `choose_parameter_depend` to generate API parameters for dependent tasks, incorporating information from previous execution logs. This allows for dynamic parameter generation based on prior task outcomes.

```python
# For dependent tasks, use previous context
previous_log = [{"task": "Get Seattle coordinates", "answer": "47.6062° N, 122.3321° W"}]
params_with_context = choose_parameter_depend(
    API_instruction, "get_forecast", api_doc,
    "Get forecast for the location found earlier",
    previous_log, "gpt-3.5-turbo"
)
```

--------------------------------

### POST /tasks - Task Planning Endpoint

Source: https://context7.com/microsoft/jarvis/llms.txt

Returns intermediate results from the task planning stage of the pipeline, showing how user requests are decomposed into structured tasks.

```APIDOC
## POST /tasks

### Description
This endpoint returns intermediate results from Stage 1 of the pipeline: task planning. It parses user requests and decomposes them into structured tasks with dependencies, allowing developers to inspect how complex requests are broken down.

### Method
POST

### Endpoint
/tasks

### Parameters
#### Request Body
- **messages** (array) - Required - An array of message objects, typically containing a user's message.

### Request Example
```json
{
    "messages": [
        {
            "role": "user",
            "content": "based on pose of /examples/d.jpg and content of /examples/e.jpg, please show me a new image"
        }
    ]
}
```

### Response
#### Success Response (200)
- Returns an array of task objects, where each object details a subtask, its arguments, dependencies, and the model to be used.
```

--------------------------------

### POST /hugginggpt - Full Service Endpoint

Source: https://context7.com/microsoft/jarvis/llms.txt

Processes the entire HuggingGPT workflow, from user message to AI-generated response, by orchestrating multiple expert models.

```APIDOC
## POST /hugginggpt

### Description
This endpoint provides access to the complete HuggingGPT service pipeline. It accepts user messages, performs task planning, model selection, task execution, and returns comprehensive AI-generated responses by orchestrating multiple expert models from HuggingFace.

### Method
POST

### Endpoint
/hugginggpt

### Parameters
#### Request Body
- **messages** (array) - Required - An array of message objects, typically containing a user's message.
- **api_key** (string) - Required - Your OpenAI API key.
- **api_type** (string) - Required - The type of API to use, e.g., "openai".
- **api_endpoint** (string) - Required - The endpoint for the API, e.g., "https://api.openai.com/v1/chat/completions".

### Request Example
```json
{
    "messages": [
        {
            "role": "user",
            "content": "Given an image /examples/a.jpg, please detect all objects and describe what you see"
        }
    ],
    "api_key": "your-openai-key",
    "api_type": "openai",
    "api_endpoint": "https://api.openai.com/v1/chat/completions"
}
```

### Response
#### Success Response (200)
- **message** (string) - The AI-generated response integrating predictions from various models.
```

--------------------------------

### Select API with EasyTool

Source: https://context7.com/microsoft/jarvis/llms.txt

Use `choose_API` to automatically select the most relevant API from a provided list based on a user's question and API documentation. This function helps in identifying the correct tool for a given task.

```python
from easytool.toolbench import choose_API, choose_parameter

# API documentation
API_instruction = "Weather API provides current conditions and forecasts"
API_list = ["get_current_weather", "get_forecast", "get_historical"]
question = "What will the weather be like tomorrow in Seattle?"

# Select appropriate APIs
selected_apis = choose_API(API_instruction, API_list, question, "gpt-3.5-turbo")
# Returns: ["get_forecast"]
```

--------------------------------

### HuggingGPT Task Planning API Request

Source: https://context7.com/microsoft/jarvis/llms.txt

Requests intermediate results from Stage 1 (task planning) of the HuggingGPT pipeline. This endpoint parses user requests and returns structured tasks with dependencies.

```bash
curl --location 'http://localhost:8004/tasks' \
--header 'Content-Type: application/json' \
--data '{ \
    "messages": [ \
        { \
            "role": "user", \
            "content": "based on pose of /examples/d.jpg and content of /examples/e.jpg, please show me a new image" \
        } \
    ] \
}'
```

--------------------------------

### Run Inference Script

Source: https://github.com/microsoft/jarvis/blob/main/taskbench/README.md

Execute the inference script to generate predictions on the TaskBench dataset. This command requires setting an API key and specifies various parameters for the LLM and data processing.

```bash
export YOUR_API_KEY=API_KEY
python inference.py \
    --llm gpt-4 \
    --data_dir data_multimedia \
    --temperature 0.2 \
    --top_p 0.1 \
    --api_addr localhost \
    --api_port 4000 \
    --api_key $YOUR_API_KEY \
    --multiworker 5 \
    --use_demos 0 \
    --reformat true \
    --reformat_by self \
    --log_first_detail true \
    --use_demos 2 \
    --dependency_type resource \
    --tag true
```

--------------------------------

### Establish Task Dependencies with EasyTool

Source: https://context7.com/microsoft/jarvis/llms.txt

Utilize `task_topology` to define the logical dependencies between decomposed tasks. This function takes the original question and a list of tasks with their IDs, returning a structured list indicating task dependencies.

```python
# Establish task dependencies
task_ls = [
    {"task": "Get weather forecast for New York", "id": 1},
    {"task": "Recommend activities based on weather", "id": 2}
]
topology = task_topology(question, task_ls, model_name="gpt-3.5-turbo")
# Returns:
# [
#     {"task": "Get weather forecast for New York", "id": 1, "dep": [-1]},
#     {"task": "Recommend activities based on weather", "id": 2, "dep": [1]}
# ]
```

--------------------------------

### POST /tasks

Source: https://github.com/microsoft/jarvis/blob/main/README.md

Retrieve intermediate results for Stage #1 of the Jarvis services.

```APIDOC
## POST /tasks

### Description
Access intermediate results for Stage #1 of the Jarvis services.

### Method
POST

### Endpoint
/tasks

### Parameters
#### Request Body
- **messages** (array) - Required - An array of message objects, typically containing user content.
  - **role** (string) - Required - The role of the message sender (e.g., "user").
  - **content** (string) - Required - The content of the message.

### Request Example
```json
{
    "messages": [
        {
            "role": "user",
            "content": "based on pose of /examples/d.jpg and content of /examples/e.jpg, please show me a new image"
        }
    ]
}
```

### Response
#### Success Response (200)
- **(array of objects)** - Description: An array of objects representing intermediate tasks and their arguments.
  - **args** (object) - Arguments for the task.
    - **image** (string) - Path to an image file.
    - **text** (string) - Text content.
  - **dep** (array) - Dependencies for the task.
  - **id** (integer) - Unique identifier for the task.
  - **task** (string) - The name of the task.

### Response Example
```json
[
    {
        "args": {"image": "/examples/d.jpg"},
        "dep": [-1],
        "id": 0,
        "task": "openpose-control"
    },
    {
        "args": {"image": "/examples/e.jpg"},
        "dep": [-1],
        "id": 1,
        "task": "image-to-text"
    },
    {
        "args": {"image": "<GENERATED>-0", "text": "<GENERATED>-1"},
        "dep": [1, 0],
        "id": 2,
        "task": "openpose-text-to-image"
    }
]
```
```

--------------------------------

### TaskBench Evaluation Command

Source: https://context7.com/microsoft/jarvis/llms.txt

This bash command initiates the evaluation process for TaskBench predictions, calculating various metrics to assess the performance of LLMs on task automation.

```bash
# Run evaluation on predictions
python evaluate.py \
    --data_dir data_multimedia \
    --prediction_dir predictions \
    --llm gpt-4 \
    --splits all \
    --n_tools all \
    --mode add \
    --dependency_type resource \
    -m all
```

--------------------------------

### Evaluate LLM Predictions

Source: https://github.com/microsoft/jarvis/blob/main/taskbench/README.md

Execute this command to calculate evaluation metrics after obtaining LLM predictions. Ensure the prediction directory and data directory are correctly specified.

```bash
python evaluate.py \
    --data_dir data_multimedia \
    --prediction_dir $prediction_dir \
    --llm gpt-4 \
    --splits all \
    --n_tools all \
    --mode add \
    --dependency_type resource \
    -m all
```

--------------------------------

### POST /hugginggpt

Source: https://github.com/microsoft/jarvis/blob/main/hugginggpt/README.md

Access the full Jarvis service by sending a POST request to this endpoint. This is the primary endpoint for interacting with the complete functionality of Jarvis.

```APIDOC
## POST /hugginggpt

### Description
Access the full Jarvis service.

### Method
POST

### Endpoint
/hugginggpt

### Request Body
- **messages** (array) - Required - An array of message objects, where each object has a 'role' (e.g., 'user') and 'content' (the user's query).

### Request Example
```json
{
    "messages": [
        {
            "role": "user",
            "content": "based on pose of /examples/d.jpg and content of /examples/e.jpg, please show me a new image"
        }
    ]
}
```

### Response
#### Success Response (200)
- **tasks** (array) - A list of tasks generated by Jarvis, including arguments, dependencies, and task type.

#### Response Example
```json
[
    {
        "args": {"image": "/examples/d.jpg"},
        "dep": [-1],
        "id": 0,
        "task": "openpose-control"
    },
    {
        "args": {"image": "/examples/e.jpg"},
        "dep": [-1],
        "id": 1,
        "task": "image-to-text"
    },
    {
        "args": {"image": "<GENERATED>-0", "text": "<GENERATED>-1"},
        "dep": [1, 0],
        "id": 2,
        "task": "openpose-text-to-image"
    }
]
```
```

--------------------------------

### Push Changes to Fork

Source: https://github.com/microsoft/jarvis/blob/main/CONTRIBUTING.md

Push your local branch with committed changes to your forked repository on GitHub.

```bash
git push origin branch-name
```

--------------------------------

### Create a New Branch

Source: https://github.com/microsoft/jarvis/blob/main/CONTRIBUTING.md

Create a new branch for your feature or bug fix to keep changes isolated.

```bash
git checkout -b "branch-name"
```

--------------------------------

### HuggingGPT Results API Request

Source: https://context7.com/microsoft/jarvis/llms.txt

Requests intermediate results from Stages 1-3 (task planning, model selection, and task execution) of the HuggingGPT pipeline. This provides detailed visibility before final response generation.

```bash
curl --location 'http://localhost:8004/results' \
--header 'Content-Type: application/json' \
--data '{ \
    "messages": [ \
        { \
            "role": "user", \
            "content": "Please classify the image /examples/b.jpg" \
        } \
    ] \
}'
```

--------------------------------

### Decompose Question into Subtasks with EasyTool

Source: https://context7.com/microsoft/jarvis/llms.txt

Use `task_decompose` to break down a user's complex question into a list of smaller, manageable subtasks. Specify the language model to be used for decomposition.

```python
from easytool.toolbench import task_decompose, task_topology

# Decompose a complex question into subtasks
question = "Find the weather forecast for New York and recommend activities based on the conditions"
decomposed = task_decompose(question, model_name="gpt-3.5-turbo")
# Returns: {"Tasks": ["Get weather forecast for New York", "Recommend activities based on weather"]}
```

--------------------------------

### POST /results - Results Endpoint

Source: https://context7.com/microsoft/jarvis/llms.txt

Returns intermediate results from Stages 1-3, providing detailed visibility into task planning, model selection, and execution results.

```APIDOC
## POST /results

### Description
This endpoint returns intermediate results from Stages 1-3: task planning, model selection, and task execution results. This provides detailed visibility into the entire processing pipeline before final response generation.

### Method
POST

### Endpoint
/results

### Parameters
#### Request Body
- **messages** (array) - Required - An array of message objects, typically containing a user's message.

### Request Example
```json
{
    "messages": [
        {
            "role": "user",
            "content": "Please classify the image /examples/b.jpg"
        }
    ]
}
```

### Response
#### Success Response (200)
- Returns an object where keys are task IDs. Each value contains the original task details, the chosen model with its reasoning, and the inference result.
```

--------------------------------

### Commit Changes

Source: https://github.com/microsoft/jarvis/blob/main/CONTRIBUTING.md

Commit staged changes with a descriptive message.

```bash
git commit -m "your commit message"
```

--------------------------------

### POST /tasks

Source: https://github.com/microsoft/jarvis/blob/main/hugginggpt/README.md

Access intermediate results for Stage #1 of the Jarvis service. This endpoint is useful for monitoring or retrieving early-stage processing information.

```APIDOC
## POST /tasks

### Description
Access intermediate results for Stage #1.

### Method
POST

### Endpoint
/tasks

### Request Body
- **messages** (array) - Required - An array of message objects, where each object has a 'role' (e.g., 'user') and 'content' (the user's query).

### Request Example
```bash
curl --location 'http://localhost:8004/tasks' \
--header 'Content-Type: application/json' \
--data '{
    "messages": [
        {
            "role": "user",
            "content": "based on pose of /examples/d.jpg and content of /examples/e.jpg, please show me a new image"
        }
    ]
}'
```

### Response
#### Success Response (200)
- **tasks** (array) - A list of tasks generated for Stage #1, including arguments, dependencies, and task type.

#### Response Example
```json
[
    {
        "args": {"image": "/examples/d.jpg"},
        "dep": [-1],
        "id": 0,
        "task": "openpose-control"
    },
    {
        "args": {"image": "/examples/e.jpg"},
        "dep": [-1],
        "id": 1,
        "task": "image-to-text"
    }
]
```
```

--------------------------------

### Update Forked Repository

Source: https://github.com/microsoft/jarvis/blob/main/CONTRIBUTING.md

Fetch upstream changes and merge them into your local main branch to keep your fork up-to-date.

```bash
git fetch upstream
git checkout main
git merge upstream/main
```