error of preprocess train data

[tech-srl]

error of preprocess train data

[urialon]

It seems that the training set fails to parse (ParseProblemException). Does it work if you preprocess our raw datasets (java-small for example)? If so, there is something wrong with your training data.

…

On Tue, Jun 2, 2020 at 6:22 PM lizhuo-1994 ***@***.***> wrote: code2seq$ head -c 2500 nohup.out Extracting paths from validation set... Finished extracting paths from validation set Extracting paths from test set... Finished extracting paths from test set Extracting paths from training set... b'java.util.concurrent.ExecutionException: com.github.javaparser.ParseProblemException: Encountered unexpected token: "-" "-"\n at line 90, column 21.\n\nWas expecting:\n\n "("\n\n\n\tat java.util.concurrent.FutureTask.report(FutureTask.java:122)\n\tat java.util.concurrent.FutureTask.get(FutureTask.java:192)\n\tat JavaExtractor.App.lambda$extractDir$3(App.java:59)\n\tat java.util.ArrayList.forEach(ArrayList.java:1257)\n\tat JavaExtractor.App.extractDir(App.java:57)\n\tat JavaExtractor.App.main(App.java:32)\nCaused by: com.github.javaparser.ParseProblemException: Encountered unexpected token: "-" "-"\n at line 90, column 21.\n\nWas expecting:\n\n "("\n\n\n\tat com.github.javaparser.JavaParser.simplifiedParse(JavaParser.java:242)\n\tat com.github.javaparser.JavaParser.parse(JavaParser.java:210)\n\tat JavaExtractor.FeatureExtractor.parseFileWithRetries(FeatureExtractor.java:73)\n\tat JavaExtractor.FeatureExtractor.extractFeatures(FeatureExtractor.java:45)\n\tat JavaExtractor.ExtractFeaturesTask.extractSingleFile(ExtractFeaturesTask.java:64)\n\tat JavaExtractor.ExtractFeaturesTask.processFile(ExtractFeaturesTask.java:34)\n\tat JavaExtractor.ExtractFeaturesTask.call(ExtractFeaturesTask.java:27)\n\tat JavaExtractor.ExtractFeaturesTask.call(ExtractFeaturesTask.java:16)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\njava.util.concurrent.ExecutionException: com.github.javaparser.ParseProblemException: Lexical error at line 13, column 24. Encountered: "l" (108), after : "\'p"\n\n\tat java.util.concurrent.FutureTask.report(FutureTask.java:122)\n\tat java.util.concurrent.FutureTask.get(FutureTask.java:192)\n\tat JavaExtractor.App.lambda$extractDir$3(App.java:59)\n\tat java.util.ArrayList.forEach(ArrayList.java:1257)\n\tat JavaExtractor.App.extractDir(App.java:57)\n\tat JavaExtractor.App.main(App.java:32)\nCaused by: com.github.javaparser.ParseProblemException: Lexical error at line 13, column 24. Encountered: "l" (108), after : "\'p"\n\n\tat com.github.javaparser.JavaParser.simplifiedParse(JavaParser.java:242)\n\tat com.github.... and a very long text fulfill with this kind of information. code2seq$ tail -22 nohup.out Finished extracting paths from training set Creating histograms from the training data subtoken vocab size: 0 node vocab size: 0 target vocab size: 0 File: 1.test.raw.txt Average total contexts: 874.889587565501 Average final (after sampling) contexts: 97.56637393088936 Total examples: 479955 Max number of contexts per word: 6200535 File: 1.val.raw.txt Average total contexts: 627.8482423720924 Average final (after sampling) contexts: 92.95679282278428 Total examples: 283564 Max number of contexts per word: 834114 File: 1.train.raw.txt Traceback (most recent call last): File "preprocess.py", line 115, in max_contexts=int(args.max_contexts), max_data_contexts=int(args.max_data_contexts)) File "preprocess.py", line 53, in process_file print('Average total contexts: ' + str(float(sum_total) / total)) ZeroDivisionError: float division by zero code2seq$ ll data/1/ total 4604380 drwxrwxr-x 2 malei malei 4096 Jun 2 23:08 ./ drwxrwxrwx 6 malei malei 4096 May 29 23:55 ../ -rw-rw-r-- 1 malei malei 3047633598 Jun 2 23:06 1.test.c2s -rw-rw-r-- 1 malei malei 0 Jun 2 23:08 1.train.c2s -rw-rw-r-- 1 malei malei 1667230637 Jun 2 23:08 1.val.c2s here is my final result of preprocessing data, it works well in processing test and validation data, but for train data, it failed all the time.[sad] actually the extract might be okey, but for histograms, it failed. thanks for help! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#52>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADSOXMDIQ5M4RSW3FWVBJR3RUUKMRANCNFSM4NQ2PN3A> .

[lizhuo-1994]

thanks for helping me, I will try to figure it out, maybe the reason is that I modified java-large, thanks again!

[urialon]

No problem, let me know if you have any more questions.
Make sure that if you modify the data - it still parses. Maybe you have changed the method names in the raw *.java files to invalid names?

[lizhuo-1994]

is the preprocessing part of code2seq the same as code2vec?
if so, I will save the time for preprocess and start training directly since I can use data from code2vec.

thanks a lot!

[urialon]

No, sorry. The code and pipeline are very similar, but preprocessing is different.

…

On Tue, Jun 9, 2020 at 4:07 PM lizhuo-1994 ***@***.***> wrote: is the preprocessing part of code2seq the same as code2vec? if so, I will save the time for preprocess and start training directly since I can use data from code2vec. thanks a lot! — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#52 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADSOXMEERYRKMMSNFFVGHJ3RVYXXPANCNFSM4NQ2PN3A> .

[lizhuo-1994]

No, sorry. The code and pipeline are very similar, but preprocessing is different.
…

ok , thanks again,

could you please recommend me the java version? Now I use :

openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-8u252-b09-1~16.04-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)

[urialon]

Yes, version 1.8 of JDK is the minimal requirement.

…

On Wed, Jun 10, 2020 at 2:46 PM lizhuo-1994 ***@***.***> wrote: No, sorry. The code and pipeline are very similar, but preprocessing is different. … <#m_1653675539481649712_> ok , thanks again, could you please recommend me the java version? Now I use : openjdk version "1.8.0_252" OpenJDK Runtime Environment (build 1.8.0_252-8u252-b09-1~16.04-b09) OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode) — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#52 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADSOXMBIE6ZFXOMMBGT43MDRV5XCPANCNFSM4NQ2PN3A> .

[lizhuo-1994]

Yes, version 1.8 of JDK is the minimal requirement.
…

thank you very much!!