zorazrw/trove

[ICML'24] TroVE: Inducing Verifiable and Efficient Toolboxes for Solving Programmatic Tasks

TROVE: Inducing Verifiable and Efficient Toolboxes for Solving Programmatic Tasks 🛠️

Setup

Install the required packages:

pip install -r requirements.txt

Tasks and datasets are organized as follows:

├── MATH
│   ├── algebra
│   ├── counting_and_probability
│   ├── geometry
│   ├── intermediate_algebra
│   ├── number_theory
│   ├── prealgebra
│   └── precalculus
├── TableQA
│   ├── TabMWP
│   ├── WTQ
│   └── HiTab
├── VQA
└── └── GQA

Running Experiments

Our Method: TroVE

python run_trove.py --task_name "math/algebra"

• For MATH tasks, specify the task name as math/${dataset_name}, e.g., math/algebra.
• For TableQA and VQA tasks, directly used the dataset name: [tabmwp, wtq, hitab, gqa].

Note that the specified --task_name argument should be lowercased.

Baseline Methods: Primitive & Instance

python baseline.py --task_name "math/algebra" --suffix "primitive"  # or "instance"

Note that for GQA dataset, we implement the locate_objects and visual_qa functions as fast apis. So you need to launch the server first (as below), then run the trove/baseline experiments.

uvicorn server.gqa:app

Evaluation

python -m utils.eval --results_path ${RESULTS_PATH}

trove's Issues

locate_objects and visual_qa

Hello, thanks for your great work! I wonder how you implemented locate_objects and visual_qa?

Plans to release model outputs?

Hi,

Interesting work! Are there any plans to release the model outputs (especially the final set of functions and corresponding solutions) for the models and datasets studied in the paper?

System vs. user prompt in api.py?

Hi, I noticed that you included the code used to call ChatGPT, but as far as I can, didn't include any code that calls it.

trove/utils/api.py

Line 29 in 7aadd05

def chat_api_wait(

Could you quickly explain how you split the prompt between the system & user prompts in your experiments?
Thanks!