java.lang.OutOfMemoryError: GC overhead limit exceeded

[crockpotveggies]

java.lang.OutOfMemoryError: GC overhead limit exceeded

[crockpotveggies]

Looks like SBT ran out of memory. The quick fix is adding a file named ".sbt_config" to your home directory (for example ~/.sbt_config) with these contents:

SBT_OPTS="-XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=2G -Xmx2048M"

[crockpotveggies]

There's one more way. Can you add the following line to ~/.bash_profile and restart terminal?

export _JAVA_OPTIONS="-Xms1024m -Xmx2G -Xss256m -XX:MaxPermSize=4G"

[damianpoole]

I found running this command to start Tinderbox worked for me.

sbt -mem 2048 run

[tibbon]

I encountered this as well

[dylancaponi]

Thanks, nice find! After modifying .bash_profile, Boot memory usage is at 1.40GB after 10 minutes of running. No heap space errors yet. Confirmation of options and warning occurs on 'sbt run':

Picked up _JAVA_OPTIONS: -Xms1024m -Xmx2G -Xss256m -XX:MaxPermSize=4G
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=4G; support was removed in 8.0

[troubleshouting]

Tried it, same crash, seems the config file is deprecated, I'm trying this
https://medium.com/@jan______/sbtconfig-is-deprecated-650d6ff10236
..and we'll see if 2Gb is enough

[crockpotveggies]

Thanks everyone for sharing your tips. Going to add these to the wiki.

[geometry9]

Even though it fixed it the first time around, it keeps crashing even after running with -mem 2048

[crockpotveggies]

@geometry9 how good are your Java skills to figure out which garbage collector it is using?

[geometry9]

Not very good, but a little googling came up with:
-XX:InitialHeapSize=134217728 -XX:MaxHeapSize=2147483648 -XX:+PrintCommandLineFlags -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseParallelGC
java version "1.8.0_31"
Java(TM) SE Runtime Environment (build 1.8.0_31-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode)

[crockpotveggies]

OK one more suggestion, can you try modifying your .sbt_config file with the following flag -XX:+UseConcMarkSweepGC so it looks like:

SBT_OPTS="-XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:MaxPermSize=2G -Xmx2048M"

Also, how much memory do you have on your particular machine?

[geometry9]

8 Gigs, trying this out for now.

[dylancaponi]

Added .sbt_config file with all options you suggested. Now getting Java heap space error. Should I open a new issue or probably same culprit? Haven't run long enough to see if I also still get GC overhead limit error.

java.lang.OutOfMemoryError: Java heap space
at scala.reflect.ManifestFactory$$anon$12.newArray(Manifest.scala:138)
at scala.reflect.ManifestFactory$$anon$12.newArray(Manifest.scala:136)
at scala.Array$.ofDim(Array.scala:222)
at utils.face.EigenFaces$.computeEigenFaces(EigenFaces.scala:39)
at utils.face.EigenFaces$.computeEigenFaces(EigenFaces.scala:27)
at services.FacialAnalysisService$.computeAverageFace(FacialAnalysisService.scala:199)
at services.FacialAnalysisService$FaceAnalysisTask$$anonfun$receive$1$$anonfun$applyOrElse$5.apply(FacialAnalysisService.scala:213)
at services.FacialAnalysisService$FaceAnalysisTask$$anonfun$receive$1$$anonfun$applyOrElse$5.apply(FacialAnalysisService.scala:208)
at scala.collection.immutable.List.foreach(List.scala:318)
at services.FacialAnalysisService$FaceAnalysisTask$$anonfun$receive$1.applyOrElse(FacialAnalysisService.scala:208)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[crockpotveggies]

@dylancaponi looks like a similar error. Were you able to try with the .sbt_config file?

In the future, I'm really going to add a dist to this, SBT is getting a little unreliable at scale.

[dylancaponi]

That is with the .sbt_config file added in home directory and Tinderbox directory. Not sure how to make sure it's being loaded.

[crockpotveggies]

Since it looks like the ~/.bash_profile addition is solving these problems, going to close this issue. I have also added this to the Common Errors wiki: https://github.com/crockpotveggies/tinderbox/wiki/Common-Errors

[dylancaponi]

Sadly, I am still getting heap space out of memory errors after ~2 hours of use.

Uncaught error from thread [application-akka.actor.default-dispatcher-7] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[application]
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236)
at org.mapdb.DataOutput2.ensureAvail(DataOutput2.java:56)
at org.mapdb.DataOutput2.write(DataOutput2.java:74)
at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1189)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at org.mapdb.SerializerPojo.serializeUnknownObject(SerializerPojo.java:401)
at org.mapdb.SerializerBase.serialize2(SerializerBase.java:531)
at org.mapdb.SerializerBase.serialize(SerializerBase.java:150)
at org.mapdb.SerializerBase.serialize(SerializerBase.java:94)
at org.mapdb.SerializerPojo.serialize(SerializerPojo.java:696)
at org.mapdb.BTreeMap$NodeSerializer.serialize(BTreeMap.java:397)
at org.mapdb.BTreeMap$NodeSerializer.serialize(BTreeMap.java:288)
at org.mapdb.Store.serialize(Store.java:154)
at org.mapdb.StoreWAL.update(StoreWAL.java:403)
at org.mapdb.Caches$HashTable.update(Caches.java:269)
at org.mapdb.EngineWrapper.update(EngineWrapper.java:63)
at org.mapdb.BTreeMap.put2(BTreeMap.java:707)
at org.mapdb.BTreeMap.put(BTreeMap.java:643)
at services.FacialAnalysisService$$anonfun$resetModels$1.applyOrElse(FacialAnalysisService.scala:156)
at services.FacialAnalysisService$$anonfun$resetModels$1.applyOrElse(FacialAnalysisService.scala:156)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
at scala.Option.collect(Option.scala:250)
at services.FacialAnalysisService$.resetModels(FacialAnalysisService.scala:156)
at models.bot.tasks.FacialAnalysisTask$$anonfun$receive$1.applyOrElse(FacialAnalysisTask.scala:33)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)

[Zott820]

I am on a Windows 8 machine with 16GB of RAM, and have been struggling with all these issues. I tried '.sbt_config' and 'sbt -mem 2048 run' but maybe I didn't do them properly since they led both led to the same errors.

Based on some testing, if the RAM usage of Java in the task manager goes over 700 megs, then the heap or GC errors are thrown. After a lot of trial and error, I found that if I keep the number of data sets for the faces under 80-100, then I can leave the system running all night without error. If I "like" or "dislike" too many faces, then the system will throw the heap error, and every subsequent attempt gives the same problem.

[AlecZadikian9001]

Remember to either source ~/.bash_profile or create a new shell after you edit your Bash profile! Otherwise, it won't take effect.

Anyway, I set the max heap size to 8G, and it's only crashing rarely now. But will it actually work? I'm still waiting to see it start automatically swiping. It's showing a face average for likes but not for dislikes even though the log is showing that it's storing pixels for dislikes. I'll edit with updates.

Edit: No, it didn't ever auto-swipe anyone. The dislike face average was still default.