Is there any future plan for supporting classification benchmarks?

[cavalab]

Is there any future plan for supporting classification benchmarks?

[athril]

I don't think we currently plan to work on a large scale benchmarking of classifiers with specific focus on GP-based ones.
Please notice that the datasets included in srbench are regression problems, PMLB covers over 165 classification problems as well.
Nonetheless, you might be interested in taking a look at the following papers that cover more recent methods:
https://biodatamining.biomedcentral.com/articles/10.1186/s13040-017-0154-4
https://www.worldscientific.com/doi/pdf/10.1142/9789813235533_0018
https://arxiv.org/abs/2107.06475

[lacava]

We have certainly thought about incorporating symbolic classification algorithms into our benchmarking. They are a bit less common in SR literature, but nonetheless I agree such a benchmark would be very useful. I could see it being an addition to this repo.

[hengzhe-zhang]

Are there any specific plans with respect to this matter? In my opinion, it seems all analysis scripts can be reused, and we only need to change experimental datasets to those classification datasets in PMLB database, and change those machine learning estimators to its classification counterpart.

[hengzhe-zhang]

By the way, I'm not sure about whether we should reuse the results reported in the previous large-scale benchmark paper. The classifiers used in that paper are rather old, and it doesn't include SOTA classifiers such as XGBoost and LightGBM. Consequently, it is questionable if it is necessary to use the existing results of that article. And even worse, some papers pointed out that results are obtained under a flawed experimental protocol [1], e.g., that paper uses test data to tune the hyper-parameter. Consequently, the results obtained by that article are not reliable.

[1]. Wainberg M, Alipanahi B, Frey B J. Are random forests truly the best classifiers?[J]. The Journal of Machine Learning Research, 2016, 17(1): 3837-3841.

[hengzhe-zhang]

@athril Thank you for providing the DIGEN package. This is exactly what I am looking for, excellent work!