WorFBench

https://github.com/zjunlp/worfbench
Admin
WorFBench is a unified benchmark and evaluation protocol designed to assess and quantify the...
Tokens:905
Snippets:12
Trust Score:10
License:MIT
Update:2 months ago
Show doc for...
Tokens:
Raw
### Install WorFBench dependencies

Source: https://github.com/zjunlp/worfbench/blob/main/README.md

Clone the repository and install the required Python packages to begin using the benchmark.

```bash
git clone https://github.com/zjunlp/WorFBench
cd WorFBench
pip install -r requirements.txt
```

--------------------------------

### Deploy Local Model with LLaMA-Factory

Source: https://github.com/zjunlp/worfbench/blob/main/README.md

Clones the LLaMA-Factory repository and starts an OpenAI-style API server on port 8000.

```bash
git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e ".[torch,metrics]"
API_PORT=8000 llamafactory-cli api examples/inference/llama3_vllm.yaml
```

--------------------------------

### Generate Workflows with Local LLM API

Source: https://github.com/zjunlp/worfbench/blob/main/README.md

Iterates through specified tasks to generate workflows using the node_eval.py script.

```bash
tasks=(wikihow toolbench toolalpaca lumos alfworld webshop os)
model_name=your_model_name
for task in ${tasks[@]}; do
    python node_eval.py \
        --task gen_workflow \
        --model_name ${model_name} \
        --gold_path ./gold_traj/${task}/graph_eval.json \
        --pred_path ./pred_traj/${task}/${model_name}/graph_eval_two_shot.json\
        --task_type ${task} \
        --few_shot \

done
```

--------------------------------

### Citation BibTeX

Source: https://github.com/zjunlp/worfbench/blob/main/README.md

BibTeX entry for citing the WorFBench research paper.

```bibtex
@article{qiao2024benchmarking,
  title={Benchmarking Agentic Workflow Generation},
  author={Qiao, Shuofei and Fang, Runnan and Qiu, Zhisong and Wang, Xiaobin and Zhang, Ningyu and Jiang, Yong and Xie, Pengjun and Huang, Fei and Chen, Huajun},
  journal={arXiv preprint arXiv:2410.07869},
  year={2024}
}
```

--------------------------------

### Evaluate Workflow Performance

Source: https://github.com/zjunlp/worfbench/blob/main/README.md

Evaluates generated workflows in node or graph mode using the node_eval.py script.

```bash
tasks=(wikihow toolbench toolalpaca lumos alfworld webshop os)
model_name=your_model_name
for task in ${tasks[@]}; do
    python node_eval.py \
        --task eval_workflow \
        --model_name ${model_name} \
        --gold_path ./gold_traj/${task}/graph_eval.json \
        --pred_path ./pred_traj/${task}/${model_name}/graph_eval_two_shot.json\
        --eval_model all-mpnet-base-v2 \
        --eval_output ./eval_result/${model_name}_${task}_graph_eval_two_shot.json \
        --eval_type node \
        --task_type ${task} \

done
```
Tokens:
Raw