collect results of jobs as they finish

[alphatwirl]

collect results of jobs as they finish

[TaiSakuma]

should measure what is actually slow. it might not be collecting results. it might be loading pickle files that is slow

alphatwirl/alphatwirl/concurrently/WorkingArea.py

Line 71 in 1e97635

result = pickle.load(f)

[TaiSakuma]

it is gzip (unzip) that is slow rather than loading pickle files itself

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1   10.684   10.684 8287.002 8287.002 framework_delphes.py:89(__call__)
        1    0.000    0.000 8213.789 8213.789 EventsInDatasetReader.py:57(end)
        1    0.000    0.000 7779.011 7779.011 MPEventLoopRunner.py:84(end)
        1    0.000    0.000 7779.011 7779.011 CommunicationChannel.py:120(receive)
        1    0.079    0.079 7779.011 7779.011 TaskPackageDropbox.py:38(receive)
     1042 5225.470    5.015 5225.470    5.015 {time.sleep}
      271    0.005    0.000 2211.092    8.159 WorkingArea.py:61(collect_result)
      271  426.554    1.574 2211.065    8.159 {cPickle.load}
533073087 1349.945    0.000 1784.455    0.000 gzip.py:248(read)
        1    0.004    0.004  434.778  434.778 CollectorComposite.py:40(collect)
       11    0.217    0.020  434.765   39.524 Collector.py:35(collect)

[TaiSakuma]

This is implemented bb0c612.

big performance improvement!

results of the event loops are merged as they finish. in merging, the orders of results are preserved.