Cannot reproduce accuracy?

[bittremieux]

Cannot reproduce accuracy?

[bittremieux]

Hi Jared, I'd be very surprised if the multithreading would cause information leakage, because the different threads are completely independent of each other. Using joblib or not only helps to speed up the computations.

Unfortunately we didn't set a global random seed, so it might be impossible to obtain completely identical results. We did fix the random seed in some key functions that use random operations though.

I can see a few small differences between our notebook and your code. We use a StratifiedShuffleSplit strategy to split the data into training and test sets, whereas you just shuffle the data with NumPy and hence don't necessarily preserve the proportion of samples for each class.

If you want to reproduce the results but don't want to use multithreading, rather than running 100 different jobs you can just set the n_jobs parameter for the joblib.Parallel call to only use a single thread. Or you can remove the joblib wrapper and run the model like this:

train_index, test_index in sss.split(X, y):
    result.append(run_predictor(predictor, X, y, train_index, test_index))

This requires minimal code changes, and I would expect you to get similar performance as we did.

[jostmey]

I am getting the accuracy you report in your paper. I think I see what I did wrong! I had the training and test data flipped in my code. So basically I fit the model with 20% of the data and tested the model with 80% of the data. Now that I am using the data correctly my accuracy is approximately 75%

[bittremieux]

Aha, that would indeed influence the results significantly. 🙂 Thank you for tracking that down.