uses 3gb+ memory running predictions

[bigmlcom]

uses 3gb+ memory running predictions

[mmerce]

Yes, certainly! That was the original idea. That's why they get loaded in slots and we have a --max-batch-models flag to control the amount of models in each slot. I'll review it ASAP. Just one question. How many predictions did you test with them? Each prediction encloses a considerable amount of information too, the distribution of training data in the node, for instance. I was wondering how much of that memory is occupied by prediction information.

[rossmeissl]

@seamusabshere ping

[mmerce]

I had a look at the code and my impression was that the thing causing the memory raise was that all test predictions are accumulated before being combined and for big test files it certainly uses a lot of memory as we need fit the size of:

(prediction + distribution in the node + confidence + labels) * number of test cases in the file * number of models in the slot determined by --max-batch-models

all of this is added at each step of the loop, until all the the models in the ensemble contribute to it and all the predictions are combined. It could be possible to store partial results to disk, but that would slow down the computation, so maybe we could include that as an option. My impression was that models were not further referenced once they had been used, and so were garbage collectable. However, this was a while ago. I'll check it back as soon as I can finish bringing anomaly detection to bigmler. Hopefully early next week.

[mmerce]

I've rechecked the predictions code, and found that, although no references where pointing to the slots of models, the garbage collector was not removing them immediately after their use, so I forced that and now it looks like memory drops at every step of the loop. I've also tried to improve performance by getting rid of the files where predictions for each of the models were stored (you can get them back by using --no-fast). It would be great if you could check our lastly released bigmler 1.14.2 and see if it works for you to close this issue.

[mmerce]

Closing the issue. If you experience this issue again, please reopen.