Heap and memory issues @GACRC

[milad181]

Heap and memory issues @GACRC

[magsol]

This is for all of @quinngroup/bigneuron --

@milad181 posted this out of memory (OOM) issue a few days ago. According to the logs (specifically GACRC.Spark_4tasks10new_124g_48c.txt), it fails on the very first distributed action: zipWithIndex. Offhand, it looks like Spark is attempting to load too many partitions into memory at once. Without more details it's hard to know for sure.

More broadly, there are a bunch of tickets that still need fixing and may contribute to issues down the road. In particular, issues #52, #53, and #58 are easy fixes that will make things much easier. Please have a look at them and see if you can submit fixes; 53 in particular is a very nasty bug that may crash any analysis we do down the road. 58 will help control the number of partitions by making it a command-line parameter, potentially fixing memory issues.

I'm still looking into things, but these issues need to be fixed ASAP.

[milad181]

Thank you @magsol .

We are currently working on MICCAI write-ups and afterward we will work to fix the current issues.
@LindberghLi, would you look at the bugs specially #53 to have more sense of issues?

[magsol]

@quinngroup/bigneuron I have a local cluster that is correctly configured to run this job, if you would prefer to do your testing there instead.

(I was only yesterday given instructions for how to create custom images, and now have an image of Spark 1.6 with Python 3.5)

[magsol]

Internal testing suggests the memory errors were due to incorrectly configured executors. Setting

spark.executor.memory=14g

within the configuration did the trick.