Spurious temp buffers called *eclim-async...*

[senny]

Spurious temp buffers called *eclim-async...*

[fred-o]

These buffers are indeed used when emacs-eclim calls eclipse asynchronously. I tried to ensure that the buffers are reused as much as possible, but maybe that's not working. How many buffers do you typically have?

[root42]

I have to test this a bit more, but sometimes up to eight. Is it possible to delay the completion, while typing? I rather want the completion to start when idle, or when I hit C-return only. Otherwise Emacs gets very laggy, because it is constantly trying to complete using eclim, which seems to be a bit costly.

[root42]

I did some further research on the performance issue: Whenever I save my buffer in Emacs, Eclipse rebuilds the current project. Is that an Eclim feature, or rather an Eclipse thing? Because that's where the performance goes. I guess if this were not the case, I wouldn't get so many async buffers either. Compilation takes a while, since the project is large and we use a custom build tool...

[root42]

And one more thing: I found it, my Eclipse was set to "Build automatically". :)

I will try to keep an eye on the async buffers, but performance went massively up. :)

[fred-o]

Every time you save (or remove) a java file, emacs-eclim tells Eclipse to refresh that file, which will usually trigger a compilation. It shouldn't rebuild the whole project though, just the changed file.

I've been thinking it might be best just to discard the async buffers when we're done with them. I decided to keep them around as I was unsure how costly creating new buffers are.

[root42]

Question: can't you use temp-buffers? Or is that not possible because of the async calls?

[root42]

I just checked: right now I had 22 old eclim-async buffers open. Our project is huge, and generates quite a few warnings (deprecated function calls etc). So each buffer was 750KB in size (mainly the warnings output). This slows down auto-completion significantly, so I disabled auto-build in Eclipse. After that, I get only a few async buffers.

[fred-o]

We can't use with-temp-buffers due to the asynchronous communication (as you noted).

I've made an experimental branch in my own repo where we don't reuse the async buffers, instead killing them when we're done. Could you have a look at b7b04aa and see if it's any help to you?

[root42]

Testing your commit right now. Will report back when I have used it for some hours. Over night the async buffers grew to 160 (750kB each). :) So I guess it will definitely help. Maybe we can make the re-use behaviour optional, by introducing a defcustom boolean.

[senny]

I would advise against customize options, which change internal behavior. It's very hard for users to decide if they want to use some internal stuff for optimizations, when they don't know the problems it could cause. I hope we find a solution to keep the tmp buffers down, which works in all circumstances.

[root42]

Seems fine. So far I haven't experienced any problems. Performance is good. Emacs/Eclim is only getting slow when a syntax check is performed. As mentioned above, our project generates thousands of warnings, and this stalls Emacs for about 10 seconds. Is it possible to disable syntax checks and/or do them on demand?

[root42]

I stand corrected: even with the change, I now have a problem of leftover buffers. I started typing something, completion went up, and the I left my computer. When I came back, Emacs was busy at 100% CPU, and I had 25 async buffers. I guess there is some idle stuff going on, even during completion? This was while using the ac-complete-emacs-eclim-* functions.

[fred-o]

It could be that while eclipse is compiling, emacs makes additional calls which are then queued. How large is your project? The one I'm working on is ~3500 classes spread over 12 maven modules. It's a bit slow, but definitely workable.

You could try (setq eclim-print-debug-messages t), which makes eclim print out every call it makes to eclipse in the messages buffer. That way we might be able to see what's going on.

[root42]

9965 source files (i.e. roughly the same number of classes). I will try to gather some debug output, as soon as I have some spare time.

[root42]

I anonymized the debug output, it looks like this:

Completion started at 4046, ac-point is 4046
Saving file MyClass.java...
Wrote MyClass.java
Executing: ~/bin/eclipse/eclim -command ping 
Executing: ~/bin/eclipse/eclim -command java_src_update -p my.project -f MyClass.java 
Executing: ~/bin/eclipse/eclim -command java_complete -p my.project -f MyClass.java -e iso-8859-1 -l standard -o 4045 
Completion started at 4051, ac-point is 4046
Saving file MyClass.java...
Wrote MyClass.java
Executing: ~/bin/eclipse/eclim -command ping 
Executing: ~/bin/eclipse/eclim -command java_src_update -p my.project -f MyClass.java 
Executing: ~/bin/eclipse/eclim -command java_complete -p my.project -f MyClass.java -e iso-8859-1 -l standard -o 4050 
Completion started at 4050, ac-point is 4046
Saving file MyClass.java...
Wrote MyClass.java
Executing: ~/bin/eclipse/eclim -command ping 
Executing: ~/bin/eclipse/eclim -command java_src_update -p my.project -f MyClass.java 
Executing: ~/bin/eclipse/eclim -command java_complete -p my.project -f MyClass.java -e iso-8859-1 -l standard -o 4048 
Completion started at 4048, ac-point is 4046
Saving file MyClass.java...
Wrote MyClass.java
Executing: ~/bin/eclipse/eclim -command ping 
Executing: ~/bin/eclipse/eclim -command java_src_update -p my.project -f MyClass.java 
Executing: ~/bin/eclipse/eclim -command java_complete -p my.project -f MyClass.java -e iso-8859-1 -l standard -o 4047 
Completion started at 4045, ac-point is 4033
Saving file MyClass.java...
Wrote MyClass.java
Executing: ~/bin/eclipse/eclim -command ping 
Executing: ~/bin/eclipse/eclim -command java_src_update -p my.project -f MyClass.java 
Executing: ~/bin/eclipse/eclim -command java_complete -p my.project -f MyClass.java -e iso-8859-1 -l standard -o 4044

What I did was that I started typing somewhere in the code. After the first character, Emacs hangs for several seconds, and then returns with the autocompletion. I am not sure which step is the costly one, but what I noticed is (as I said before) that my eclim-async buffers are about 750kb each. And especially when I go away and come back, there will be dozens of buffers running, which are eventually killed, but which make Emacs unresponsive.

[senny]

This looks like a loop problem within emacs-eclim. It seems that the 4 commands result in an endless loop.

[root42]

Not sure about that. I just typed a couple of characters before. When I type only one character, I get this:

Completion started at 3349, ac-point is 3349
Saving file MyClass.java...
Wrote MyClass.java
Executing: ~/bin/eclipse/eclim -command ping 
Executing: ~/bin/eclipse/eclim -command java_src_update -p my.project -f MyClass.java 
Executing: ~/bin/eclipse/eclim -command java_complete -p my.project -f MyClass.java -e iso-8859-1 -l standard -o 3348 

And it still takes 5-10 seconds for Emacs to respond.

[root42]

Now I have the debug output for when I go away and/or switch to some C++ buffers in between:

Executing: ~/bin/eclipse/eclim -command project_list 
refreshing... MyClass.java 
Using async buffer *eclim-async*<2>
Executing: ~/bin/eclipse/eclim -command problems -p my.project 
refreshing... MyClass.java 
Using async buffer *eclim-async*<3>
Executing: ~/bin/eclipse/eclim -command problems -p my.project 
refreshing... MyClass.java 
Using async buffer *eclim-async*<4>
Executing: ~/bin/eclipse/eclim -command problems -p my.project 
refreshing... MyClass.java 
Using async buffer *eclim-async*<5>
Executing: ~/bin/eclipse/eclim -command problems -p my.project 
refreshing... MyClass.java 
Using async buffer *eclim-async*<6>
Executing: ~/bin/eclipse/eclim -command problems -p my.project 
refreshing... MyClass.java 
Using async buffer *eclim-async*<7>
Executing: ~/bin/eclipse/eclim -command problems -p my.project 
refreshing... MyClass.java 
Using async buffer *eclim-async*<8>
Executing: ~/bin/eclipse/eclim -command problems -p my.project 
refreshing... MyClass.java 
Using async buffer *eclim-async*<9>
Executing: ~/bin/eclipse/eclim -command problems -p my.project 
refreshing... MyClass.java 
Using async buffer *eclim-async*<10>
Executing: ~/bin/eclipse/eclim -command problems -p my.project 
refreshing... MyClass.java 
Using async buffer *eclim-async*<11>
Executing: ~/bin/eclipse/eclim -command problems -p my.project 
refreshing... MyClass.java 
Using async buffer *eclim-async*<12>
Executing: ~/bin/eclipse/eclim -command problems -p my.project 
refreshing... MyClass.java 
Using async buffer *eclim-async*<13>
Executing: ~/bin/eclipse/eclim -command problems -p my.project 
refreshing... MyClass.java 
Using async buffer *eclim-async*<14>
Executing: ~/bin/eclipse/eclim -command problems -p my.project 
refreshing... MyClass.java 
Using async buffer *eclim-async*<15>
Executing: ~/bin/eclipse/eclim -command problems -p my.project 
refreshing... MyClass.java 
Using async buffer *eclim-async*<16>
Executing: ~/bin/eclipse/eclim -command problems -p my.project 
refreshing... MyClass.java 
Using async buffer *eclim-async*<17>
Executing: ~/bin/eclipse/eclim -command problems -p my.project 
refreshing... MyClass.java 
Using async buffer *eclim-async*<18>
Executing: ~/bin/eclipse/eclim -command problems -p my.project 
refreshing... MyClass.java 
Using async buffer *eclim-async*<19>
Executing: ~/bin/eclipse/eclim -command problems -p my.project 
refreshing... MyClass.java 
Using async buffer *eclim-async*<20>
Executing: ~/bin/eclipse/eclim -command problems -p my.project 
Executing: ~/bin/eclipse/eclim -command project_list  [18 times]
Eclim reports 8 errors, 1355 warnings.
Executing: ~/bin/eclipse/eclim -command project_list 
Eclim reports 8 errors, 1355 warnings.
Executing: ~/bin/eclipse/eclim -command project_list 
Eclim reports 8 errors, 1355 warnings.
Executing: ~/bin/eclipse/eclim -command project_list 
Eclim reports 8 errors, 1355 warnings.
Executing: ~/bin/eclipse/eclim -command project_list 
Eclim reports 8 errors, 1355 warnings.
Executing: ~/bin/eclipse/eclim -command project_list 
Eclim reports 8 errors, 1355 warnings.
Executing: ~/bin/eclipse/eclim -command project_list 
Eclim reports 8 errors, 1355 warnings.
Executing: ~/bin/eclipse/eclim -command project_list 
Eclim reports 8 errors, 1355 warnings.
Executing: ~/bin/eclipse/eclim -command project_list 
Eclim reports 8 errors, 1355 warnings.
Executing: ~/bin/eclipse/eclim -command project_list 
Eclim reports 8 errors, 1355 warnings.
Executing: ~/bin/eclipse/eclim -command project_list 
Eclim reports 8 errors, 1355 warnings.
Executing: ~/bin/eclipse/eclim -command project_list 
Eclim reports 8 errors, 1355 warnings.
Executing: ~/bin/eclipse/eclim -command project_list 
Eclim reports 8 errors, 1355 warnings.
Executing: ~/bin/eclipse/eclim -command project_list 
Eclim reports 8 errors, 1355 warnings.
Executing: ~/bin/eclipse/eclim -command project_list 
Eclim reports 8 errors, 1355 warnings.
Executing: ~/bin/eclipse/eclim -command project_list 
Eclim reports 8 errors, 1355 warnings.
Executing: ~/bin/eclipse/eclim -command project_list 
Eclim reports 8 errors, 1355 warnings.
Executing: ~/bin/eclipse/eclim -command project_list 
Eclim reports 8 errors, 1355 warnings.
Executing: ~/bin/eclipse/eclim -command project_list 
Eclim reports 8 errors, 1355 warnings.

This really blocks Emacs for quite a long time...

[fred-o]

I guess emacs-eclim keeps calling eclim over and over while eclipse is busy building the project. I've added some experimental performance tweaks to try and keep track of the currently running calls in order to avoid making the same one again. Please have a look at https://github.com/fred-o/emacs-eclim/commits/no-pooling-of-async-buffers and see if it helps.

I've also changed the behaviour of the problems list; if your are filtering on 'errors' emacs-eclim will now only fetch the errors, which cuts down on the amount of data we need to transfer and parse quite a bit.

[root42]

I just checked it out and will test it for a couple of hours. I'll probably report back tomorrow. As for a quick test, it feels quite snappy. But let me code for a while and see how it fares.

[root42]

Performance seems vastly improved. Sometimes I have to wait a bit still, but that's when Eclipse has to rebuild lots of subprojects. I guess that is ok.

[root42]

One problem that I still see: After a morning of work, I now had 140 leftover buffers... :-/ However, they were all empty. Performance ist definitely better, though.

[fred-o]

There was a bug that caused buffers to be created even if no call was made. Should be fixed now.

[root42]

Looks better, so far. Now I only had two async buffers when returning from lunch, both contained: Missing required option: p.

[fred-o]

Yeah, those are from when an error is raised during the result parsing, I think. I'll need to go through the error handling routines before this is merged into the master branch.

[fred-o]

Fixed that last issue and merged everything into master. I think I'll also have a closer look at the calls we're making; for example it doesn't really seem that necessary to trigger a problem buffer refresh while we're autocompleting.

[root42]

I'll check out your master and test it for a while, to see if it works. Thanks.

[fred-o]

It should be pushed to the regular master repo, but it seems github is acting up and won't display it.

[root42]

Quick result: Now auto-complete (no output in Messages) does not work anymore, and eclim-project-build gives this:

error in process sentinel: cond: Missing required option: p

error in process sentinel: Missing required option: p

Any ideas?

[fred-o]

Hm... works fine on my end. The 'Missing required option: p' message usually means that emacs-eclim couldn't figure out which project the current file belonged to. What buffer are you executing eclim-project-build from?

Also, have you restarted emacs since the update?

[root42]

Yes, I have restarted Emacs. I am executing eclim-project-build from one of my Java-source-file buffers. Completion doesn't work, neither does eclim-java-find-declaration. That gives an exception in eclim:

cond: org.eclipse.core.runtime.AssertionFailedException: null argument:
    at org.eclipse.core.runtime.Assert.isNotNull(Assert.java:85)
    at org.eclipse.core.runtime.Assert.isNotNull(Assert.java:73)
    at org.eclipse.core.runtime.Path.initialize(Path.java:577)
    at org.eclipse.core.runtime.Path.<init>(Path.java:185)
    at org.eclipse.core.internal.resources.WorkspaceRoot.getProject(WorkspaceRoot.java:182)
    at org.eclim.plugin.core.util.ProjectUtils.getProject(ProjectUtils.java:110)
    at org.eclim.plugin.core.util.ProjectUtils.getProject(ProjectUtils.java:96)
    at org.eclim.plugin.core.util.ProjectUtils.getFilePath(ProjectUtils.java:130)
    at org.eclim.plugin.core.command.AbstractCommand.getOffset(AbstractCommand.java:79)
    at org.eclim.plugin.jdt.command.search.SearchCommand.executeSearch(SearchCommand.java:155)
    at org.eclim.plugin.jdt.command.search.SearchCommand.execute(SearchCommand.java:112)
    at org.eclim.command.Main$1.run(Main.java:98)
    at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:35)
    at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:135)
    at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:3563)
    at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3212)
    at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.java:2701)
    at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2665)
    at org.eclipse.ui.internal.Workbench.access$4(Workbench.java:2499)
    at org.eclipse.ui.internal.Workbench$7.run(Workbench.java:679)
    at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
    at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:668)
    at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149)
    at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:123)
    at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
    at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
    at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:344)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:622)
    at org.eclipse.equinox.launcher.Main.basicRun(Main.java:577)
    at org.eclipse.equinox.launcher.Main.run(Main.java:1410)
    at org.eclipse.equinox.launcher.Main.main(Main.java:1386)

[fred-o]

I have a hunch that the variable eclim--project-name has been set to something other for some reason. Could you check the value of that variable when you are in the java buffer?

Also, could you eval the function with the same name?

[root42]

C-h v eclim--project-name:

eclim--project-name is a variable defined in `eclim.el'.
Its value is nil
Local in buffer Foo.java; global value is nil

M-x eval-expression (eclim--project-name) also returns nil.

[root42]

Hm... I rolled back a couple of revisions, and still get the same behaviour. Could it be that something is wrong with my eclim? Are there any settings that can break / caches that can deteriorate? If so, how can I undo this?

[root42]

I reinstalled Eclips & Eclim, deleted ~/.eclim and I deleted my Workspace. Nothing did help. I am currently clueless what is going wrong here. I have now enabled debug information again, and get this upon eclim-project-build:

Executing: ~/bin/eclipse/eclim -command project_list  [3 times]
Executing: ~/bin/eclipse/eclim -command project_build 
Using async buffer *eclim-async*
error in process sentinel: cond: Missing required option: p

error in process sentinel: Missing required option: p

When I run eclim -command project_list on the command line, I get a long JSON answer with all my projects.

[fred-o]

It looks like the java file isn't recognized as belonging to an eclipse project, for some reason. Is the .project file missing from the project root? Emacs-eclim uses this when trying to determine the name of the current project.

Also, what does (eclim/project-list) and (eclim--project-dir buffer-file-name) eval to?

[fred-o]

FWIW I just experienced the same thing that you have, when I tried to interact with files belonging to a project that I'd mistakenly deleted from the eclipse workspace.

[root42]

First, M-x eval-expression (eclim/project-list) yields (anonymized):

[((path . "/path_to/my/project") (name . "my_project_name") (open . t)) ((path . "/path_to/my/project") (name . "my_project_name") (open . t)) ((path . "/path_to/my/project") (name . "my_project_name") (open . t)) ((path . "/path_to/my/project") (name . "my_project_name") (open . t)) ((path . "/path_to/my/project") (name . "my_project_name") (open . t)) ((path . "/path_to/my/project") (name . "my_project_name") (open . t)) ((path . "/path_to/my/project") (name . "my_project_name") (open . t)) ((path . "/path_to/my/project") (name . "my_project_name") (open . t)) ((path . "/path_to/my/project") (name . "my_project_name") (open . t)) ((path . "/path_to/my/project") (name . "my_project_name") (open . t)) ((path . "/path_to/my/project") (name . "my_project_name") (open . t)) ((path . "/path_to/my/project") (name . "my_project_name") (open . t)) ...]

And M-x eval-expression (eclim--project-dir buffer-file-name) yields (anonymized):

"/valid/path/to/one/project/in/workspace"

The path is valid, and I examined it even via ls.

[root42]

PS: each entry in the anonymized list is distinct... I just cannot hand out our project names. ;)

[root42]

Allright, I found the error! We have a symlink from the project base to another directory. Eclipse uses that symlink, instead of the "real" path, that I use in Emacs. So your exlim--project-name function fails, since it does text matching. I recommend that file-truename should be applied to the paths.

[root42]

Kept the new version running for some days, and it seems fine. I'll close the ticket for now.

[fred-o]

Great! I never would have thought about symlinks... :)