gurol/benchmetricsprob

BenchMetrics Prob: Benchmarking of probabilistic error performance evaluation instruments for binary-classification problems

A reproducible research compedium of

BenchMetrics Prob: benchmarking of probabilistic error/loss performance evaluation instruments for binary classification problems

Gürol Canbek (2023). BenchMetrics Prob: Benchmarking of probabilistic error / loss performance evaluation instruments for binary-classi cation problems. International Journal of Machine Learning and Cybernetics. doi: 10.1007/s13042-023-01826-5

🔗 Access the free-access full text here.

This repository provides BenchMetrics Prob probabilistic error/loss instrument calculator and simulation tool for benchmarking the robustness of 31 probabilistic error/loss measures/metrics via five criteria and seven simulation cases proposed in my article above.

The proposed method was tested on 30 probabilistic error instruments given in the table below and LogLoss as a probabilistic loss instrument.

Note: Please, cite my article if you would like to use and/or adapt the tool, simulation cases, methodology, and other materials provided and let me know. Thank you for your interest.

Subtype	Instrument Name	Abbreviation
(Raw)	Mean Error	ME
Squared	Mean Squared Error	MSE
	Root Mean Square Error	RMSE
	Median Squared Error	MdSE
	Sum Squared Error	SSE
	Normalized Mean Squared Error (v1-5)	nMSE
Absolute	Mean Absolute Error	MAE
	Median Absolute Error	MdAE
	Maximum Absolute Error	MxAE
	Geometric Mean Absolute Error	GMAE
Relative	Mean Relative Absolute Error	MRAE
	Median Relative Absolute Error	MdRAE
Relative	Geometric Mean Relative Absolute Error	GMRAE
	Relative Absolute Error	RAE
	Relative Squared Error	RSE
Percentage	Mean Percentage Error	MPE
	Mean Absolute Percentage Error	MAPE
	Median Absolute Percentage Error	MdAPE
	Root Mean Square Percentage Error	RMSPE
	Root Median Square Percentage Error	RMdSPE
Percentage (Symmetric)	Symmetric Mean Absolute Percentage Error	sMAPE
	Normalized Symmetric Mean Absolute Percentage Error	nsMAPE
	Normalized Symmetric Median Absolute Percentage Error	nsMdAPE
Scaled	Mean Absolute Scaled Error	MASE
	Median Absolute Scaled Error	MdASE
	Root Mean Squared Scaled Error	RMSSE