jev-decision-benchmarks

# JEV Decision Benchmarks

We use these three benchmarks to evaluate whether JEV selects the right tools, knows when to call or abstain, and avoids choosing tool calls when no tools are available.

English | [简体中文](README.zh-CN.md)

## MetaTool

MetaTool tests whether a model selects the right tool or combination of tools for a user's request and abstains when no suitable tool is available.

| Model / mode | Similar 0-shot ↑ | Similar 5-shot ↑ | Abstain 0-shot ↑ | Abstain 5-shot ↑ | At most two ↑ | Exactly two ↑ |
|---|---:|---:|---:|---:|---:|---:|
| **JEV 1.13 (Choice)** | **77.79%** | **77.79%** | **87.04%** | **88.54%** | 81.29% | 88.33% |
| **ChatGPT** | 69.05% | 72.94% | 50.35% | 78.49% | **88.28%** | 88.53% |
| ChatGLM2 | 54.17% | 57.44% | 6.63% | 15.68% | 20.20% | 23.34% |
| Llama2-7b | 45.95% | 51.12% | 0.90% | 2.51% | 35.69% | 57.34% |
| Llama2-13b | 44.06% | 49.85% | 2.31% | 5.93% | 81.49% | 77.87% |
| Vicuna-7b | 73.46% | 63.67% | 1.50% | 1.81% | 44.06% | 64.34% |
| Vicuna-13b | 58.23% | 63.15% | 2.51% | 3.42% | 83.70% | 78.47% |
| **Vicuna-33b** | 53.96% | 60.54% | 2.81% | 3.11% | 48.69% | **91.15%** |
| Koala-13b | 56.34% | 60.85% | 1.70% | 5.83% | 39.03% | 25.10% |

CSR measures exact tool-set selection. JEV uses a Choice adapter; paper baselines generate tool names followed by answer extraction. The public similar-tool and abstention sets each contain 995 examples, versus 975 each in the paper; each multi-tool condition has 497 examples. [Paper v6, Tables 3–4](https://arxiv.org/html/2310.03128v6) · [Full results and protocol (中文)](reports/metatool.zh-CN.md) · [CSV](data/metatool/comparison.csv)

## When2Call

When2Call tests whether a model chooses the right next step: answer directly, call a tool, ask for missing information, or say it cannot complete t