Is it possible to provide complete experimental results?

[cavalab]

Is it possible to provide complete experimental results?

[lacava]

see https://github.com/EpistasisLab/srbench/tree/master/results

[lacava]

yep, they will be available shortly. had to change the project from a public fork to its own repo in order to use Git LFS for storage. Stay tuned!

[lacava]

also, @zhenlingcn, if you contribute a method in the near future, we can benchmark it for you. we may not have the resources to do that forever, so now is a good time to take advantage of that :)

[hengzhe-zhang]

@lacava
Thanks for providing the experimental data. I see that there are 25346 items in that file, however, there are 21 algorithms. After a brief investigation, I found that some data might be missing, such as those related to "AIFeynamn", "Feat" and "MRGP". So, how should we deal with this problem?

[lacava]

Yes, but we encountered some training failures. Our protocol for failed runs was as follows. For the black-box problems, if a job was killed due to the time limit, we re-ran the experiment without hyperparameter tuning, thereby only requiring a single training iteration to complete within 48 hours. If the job still failed, or failed for other reasons, the results for that method-dataset-seed combination were mostly treated as missing.
Our statistical comparisons of methods were based on comparisons of summary performance per dataset, and so a failed trial only affected the fidelity of our model-dataset performance estimate, unless the method failed for all trials of a given dataset.
In the latter case (which only occurred for AIFeynman), the models were given last-place rankings in our non-parametric statistical comparisons, but other summary statistics (i.e. $R^2$, solution rates, etc) were reported without those trials.

We received at least one successful experiment for all methods on every dataset and noise level except for AIFeynman, which completed training for 107/122 of the black-box regression problems and 128/130 of the ground-truth problems.
We encountered the difficulty completing the benchmark with AIFeynman due to the need to implement a sklearn API, time limits, and handle path over-write issues in the published code (our fork is available here). In addition, the time limit for AIFeynman fails if a specific step (e.g. the brute force search) takes the entire training time, so there may not be a straightforward remedy other than to treat it as a failure.

We're continuing to debug the runs and will update the results as we do so.