Comments (5)
Here's what is printed when it crashes:
Exception in thread "main" java.lang.OutOfMemoryError
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677)
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:735)
at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at edu.uw.cs.lil.amr.data.LabeledAmrSentenceCollection.(LabeledAmrSentenceCollection.java:83)
at edu.uw.cs.lil.amr.data.LabeledAmrSentenceCollection$Creator.create(LabeledAmrSentenceCollection.java:169)
at edu.uw.cs.lil.amr.data.LabeledAmrSentenceCollection$Creator.create(LabeledAmrSentenceCollection.java:97)
at edu.cornell.cs.nlp.spf.explat.ParameterizedExperiment.readResrouces(ParameterizedExperiment.java:204)
at edu.cornell.cs.nlp.spf.explat.DistributedExperiment.readResrouces(DistributedExperiment.java:206)
at edu.uw.cs.lil.amr.exp.AmrExp.(AmrExp.java:105)
at edu.uw.cs.lil.amr.exp.AmrExp.(AmrExp.java:117)
at edu.uw.cs.lil.amr.exp.AmrGenericExperiment.main(AmrGenericExperiment.java:28)
at edu.uw.cs.lil.amr.Main.main(Main.java:61)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at com.google.common.collect.Tables.immutableCell(Tables.java:67)
at com.google.common.collect.StandardTable$CellIterator.next(StandardTable.java:323)
at com.google.common.collect.StandardTable$CellIterator.next(StandardTable.java:306)
at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
at uk.ac.ed.easyccg.syntax.ParserAStar.parseAstar(ParserAStar.java:339)
at uk.ac.ed.easyccg.syntax.ParserAStar.doParsing(ParserAStar.java:226)
at uk.ac.ed.easyccg.syntax.ParserAStar.parseTokens(ParserAStar.java:120)
at edu.uw.cs.lil.amr.ccgbank.easyccg.EasyCCGWrapper.getSpans(EasyCCGWrapper.java:54)
at edu.uw.cs.lil.amr.data.LabeledAmrSentence.(LabeledAmrSentence.java:45)
at edu.uw.cs.lil.amr.data.LabeledAmrSentenceCollection.lambda$new$0(LabeledAmrSentenceCollection.java:81)
at edu.uw.cs.lil.amr.data.LabeledAmrSentenceCollection$$Lambda$39/1560160481.apply(Unknown Source)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747)
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721)
at java.util.stream.AbstractTask.compute(AbstractTask.java:316)
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
from amr.
It looks like a memory problem. I don't remember how much memory we used for training, but our machines had quite a bit of memory, maybe 64GB or 128GB. The exception seems to be happening in one of the threads. We don't have a graceful way to bring down the entire system when a thread throws an exception. Whatever happens after this kind of exception is not to be relied on, and you should just kill the process.
from amr.
One thing you can do is limit to very short sentences. Memory consumption is tightly coupled with sentence length due to the use of a CKY chart. The model that you will get might not be good, but it will be a good way to test your setup.
from amr.
I fixed my memory problems and was able to train a model on all the ldc data. However, an amr.pre.sp model was created and not all the files were present in the logs folder that were present when I trained a much smaller model. Do you know why this is? Is it cause of the use of split up data? Would putting all the data into one file fix this? My exp and inc files seems exactly the same across experiments, excpet when I trained the smaller model and got an amr.sp file, the data wasn't split up.
from amr.
If you use separate files, they have to be merged. I think the exp files we released do that. Once you merge the data resource, it should be used a single resource, so I am not sure why it will behave differently.
from amr.
Related Issues (5)
- Train HOT 4
- binary files in resources/propbank HOT 1
- AMR v.s SPF HOT 3
- Pretrained Model HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from amr.