kNN classification accuracy task (P2P)

[dkobak]

kNN classification accuracy task (P2P)

[loicmagne]

I think you can already use knn classifier to evaluate classification tasks, with method="kNN":

mteb/mteb/abstasks/AbsTaskClassification.py

Line 75 in fb90c02

if self.method == "kNN":

However this is just a different way to train a classifier, there is no particular "kNN graph quality" metrics. You also cannot use knn classification evaluation on clustering datasets like BiorxivP2P, for this we would need to create train sets for clustering data. I don't know if the samples within clustering datasets would provide meaningful insights when used for classification, and it would require a clear methodology to ensure that the kNN graph quality metrics correctly reflects unsupervised learning performance

Is method="kNN" satisfactory, or do you think it would be useful to add dedicated evaluations for kNN graphs ?

[dkobak]

Thanks @loicmagne for your reply!

I think you can already use knn classifier to evaluate classification tasks, with method="kNN"

Oh, that's cool, I did not realize that.

Is method="kNN" satisfactory, or do you think it would be useful to add dedicated evaluations for kNN graphs ?

I am not sure. In principle one can compute kNN accuracy on the entire dataset without any train/test split by explicitly constructing the full kNN graph (it's implicitly a leave-one-out procedure):

from sklearn.neighbors import NearestNeighbors
from scipy.stats import mode

def knn_accuracy_loocv(X, y, n_neighbors=10):
    neigh = NearestNeighbors(n_neighbors=n_neighbors).fit(X)
    knn = neigh.kneighbors(return_distance=False) # returns kNN graph of X
    yhat = mode(y[knn], axis=1).mode.flatten()    # kNN classifier predictions
    return np.mean(yhat == y)                     # kNN accuracy

This is nice because it evaluates kNN accuracy over the entire dataset. But in practice running KNeighborsClassifier on a train/test split would yield close results. So it's not a huge difference.

You also cannot use knn classification evaluation on clustering datasets like BiorxivP2P, for this we would need to create train sets for clustering data.

That's perhaps the biggest problem right now. Datasets like BiorxivP2P seem to me to be very good candidates for this metric. Would it make sense to create train/test splits for all of them? Then any classification metric could be run on them, including kNN. Can't one simply create train/test split at runtime and fix the random seed so that it's deterministic?

Alternative option would be to implement kNN graph evaluation like I suggested above without any train/test split.

[loicmagne]

Thanks for your reply, those are good options. I don't know how it should be integrated with the MTEB lib, new task? new evaluator? @Muennighoff what do you think?

[Muennighoff]

Really cool discussion. I think it'd be interesting to have it as an option (while the default remains as is)

[dkobak]

I think it'd be interesting to have it as an option (while the default remains as is)

Hi @Muennighoff, I'm not quite sure what you mean here. To have it as an option where exactly?

[Muennighoff]

If you think it's better as a standalone evaluator (not an option for one of the existing ones) that's fine too I think