Add user guide: Defining and parametrizing benchmarks by applying decorators

[Maciej818]

Add user guide: Defining and parametrizing benchmarks by applying decorators

[Maciej818]

To keep information complete I copy-paste here comments from #33 (which is an issue that duplicates this one):

msavinash commented 15 hours ago
I don't understand how parametrize is supposed to work. From my understanding its something like this:

parametrize_parameters = [
{"y_test": [1, 2, 3], "y_pred": [4, 5, 6]},
{"y_test": [1, 2, 3], "y_pred": [1, 2, 3]},
]

@nnbench.parametrize(parameters=parametrize_parameters)
def accuracy(model: base.BaseEstimator, y_test: np.ndarray, y_pred: np.ndarray) -> float:
accuracy = metrics.accuracy_score(y_test, y_pred)
return accuracy

Is this accurate?

nicholasjng commented 31 minutes ago
Is this accurate?

Yes.

(The arguments in the decorator dictionaries should match the parametrized benchmark function's typing, so the arguments should be np.array([1, 2, 3]), np.array([4, 5, 6]), ..., but you got it right conceptually.)

[msavinash]

I tried running this:

import nnbench
from sklearn import base
import numpy as np


parametrize_parameters = [
    {"y_test": np.array([1, 2, 3]), "y_pred": np.array([4, 5, 6])},
    {"y_test": np.array([1, 2, 3]), "y_pred": np.array([1, 2, 3])},
]

@nnbench.parametrize(parameters=parametrize_parameters)
def accuracy(model: base.BaseEstimator, y_test: np.ndarray, y_pred: np.ndarray) -> float:
    accuracy = metrics.accuracy_score(y_test, y_pred)
    return accuracy


from nnbench import runner
r = runner.BenchmarkRunner()
y_pred = model.predict(X_test)
result = r.run("__main__", params={"model": model})

r.report(to='console', result=result)

Got this error:

  File "C:\Users\Public\Documents\Avinash\Projects\OpenSource\nnbench\test.py", line 65, in <module>
    result = r.run("__main__", params={"model": model})#, "y_pred": y_pred, "y_test": y_test})
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Public\Documents\Avinash\Projects\OpenSource\nnbench\src\nnbench\runner.py", line 218, in run
    _check(dparams, self.benchmarks)
  File "C:\Users\Public\Documents\Avinash\Projects\OpenSource\nnbench\src\nnbench\runner.py", line 84, in _check
    raise ValueError(f"missing value for required parameter {name!r}")
ValueError: missing value for required parameter 'y_test'

I believe the code expects me to add y_pred and y_test as well, but they are defined for parameterize decorator. What is the expected behavior here?
result = r.run("__main__", params={"model": model, "y_pred": y_pred, "y_test": y_test})