Cleaning takes too long time on multi-cores cpu

[axelderomblay]

Cleaning takes too long time on multi-cores cpu

[a1a2y3]

from http://pythonhosted.org/joblib/parallel.html#common-usage
I found this "Under Windows, it is important to protect the main loop of code to avoid recursive spawning of subprocesses when using joblib.Parallel."..."No code should run outside of the “if name == ‘main’” blocks, only imports and definitions."

Problem solved.

[a1a2y3]

Thank you for reply. I think joblib or multiprocessing cause this problem, and trying to solve it.
I use windows10 + anaconda + python3.6 + vs2015, may have conflict with joblib?

set n_jobs=1, seems OK
reading csv : train.csv ...
cleaning data ...
CPU time: 0.22528505325317383 seconds
reading csv : test.csv ...
cleaning data ...
CPU time: 0.1932668685913086 seconds

set n_jobs=2, it dies.

[a1a2y3]

Drift_thresholder() has same problem.
It takes 1.38s on kaggle kernel, and176s on my PC with E5-2683v3 cpu.

[AxeldeRomblay]

Hum... sounds very weird ! Because it takes only 2 sec on my computer (7 cores). Have you tried to set n_jobs = 1 and run again ?

[AxeldeRomblay]

Yes this is what I was wondering. At the moment, MLBox does not support Windows but soon :)
Thank you very much for reporting this issue !!

[DarquesM]

I've got same issue, where should I set n_jobs=1 ?
mlbox.preprocessing.Reader does not have "n_jobs" parameter

[AxeldeRomblay]

Hello @DarquesM !
The problem is due to windows... At the moment what you can do is to set n_jobs=1 in the source code :

df = pd.concat(Parallel(n_jobs=-1)(delayed(convert_list)(df[col]) for col in df.columns), axis=1)

df = pd.concat(Parallel(n_jobs=-1)(delayed(convert_float_and_dates)(df[col]) for col in df.columns), axis=1)

Otherwise, I will release soon a new version with reading and cleaning separate classes...

[AxeldeRomblay]

Hello, thanks for reporting this issue. I will close it since this will be fixed in a next release (MLBox 0.7.1 probably)