Doesn't work with latest Emacs git master?

[razzmatazz]

Doesn't work with latest Emacs git master?

[razzmatazz]

ok, status update:

• fix for emacs@emacs-28 (on macos) will not be provided, we need to actually apply workarounds there;
  • https://lists.gnu.org/archive/html/emacs-devel/2022-02/msg00461.html
• fix for emacs@master may be coming later, but we need workarounds in the mean time;
  • https://lists.gnu.org/archive/html/emacs-devel/2022-02/msg00350.html
• I have provided workarounds in lsp-mode for macos (emacs-28) and linux (emacs-29)
  • see emacs-lsp/lsp-mode#3349

[razzmatazz]

OK, so it appears the was a bug in the current implementation and patch from Robert has actually fixed the issue:

• https://lists.gnu.org/archive/html/emacs-devel/2022-02/msg00009.html

Great! So things should be back to normal once the patch is applied on emacs@master.

[razzmatazz]

closing as the fix has been merged with the latest emacs (both master and emacs-28) and we have a workaround applied in lsp-csharp.el in lsp-mode

ATM we only need this workaround on `emacs-28.1/macos" -- emacs-28.2/macos will include the fix too when released later; emacs-28.1/linux still used the old vfork mechanism for subprocesses, so no workaround is needed there

[razzmatazz]

Ok, I can’t let you get off :)) will have to debug :)

[razzmatazz]

hmm.. I will release a new version with rewritten LSP backend (from ionide/LanguageServerProtocol#10) -- so maybe that will help as the underlying IPC lib would now be StreamJsonRpc from https://github.com/microsoft/vs-streamjsonrpc which seems to be more mature

that will get us $/cancelRequest support for free as well, for better performance

Maybe that will help, as I suspect there is low level stuff that doesn't play right with the current IPC code that reads/write to stdout/stdin..

[razzmatazz]

ok, I found the bad/causing commit:

a60053f8368e058229721f1bf1567c2b1676b239 is the first bad commit
commit a60053f8368e058229721f1bf1567c2b1676b239
Author: Philipp Stephani <[email protected]>
Date:   Wed Dec 30 14:42:01 2020 +0100

    Use posix_spawn if possible.
    
    posix_spawn is less error-prone than vfork + execve, and can make
    better use of system-specific enhancements like 'clone' on Linux.  Use
    it if we don't need to configure a pseudoterminal.
    
    * configure.ac (HAVE_SPAWN_H, HAVE_POSIX_SPAWN)
    (HAVE_POSIX_SPAWN_FILE_ACTIONS_ADDCHDIR)
    (HAVE_POSIX_SPAWN_FILE_ACTIONS_ADDCHDIR_NP)
    (HAVE_POSIX_SPAWNATTR_SETFLAGS, HAVE_DECL_POSIX_SPAWN_SETSID): New
    configuration variables.
    * src/callproc.c (USABLE_POSIX_SPAWN): New configuration macro.
    (emacs_posix_spawn_init_actions)
    (emacs_posix_spawn_init_attributes, emacs_posix_spawn_init): New
    helper functions.
    (emacs_spawn): Use posix_spawn if possible.

 configure.ac   |  17 ++++++
 src/callproc.c | 189 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 205 insertions(+), 1 deletion(-)

there is a related emacs-devel thread:

• https://lists.gnu.org/archive/html/emacs-devel/2021-11/msg00174.html

there is related code in lsp-mode that makes a process to communicate over stdio that launches the process:

• https://github.com/emacs-lsp/lsp-mode/blob/master/lsp-mode.el#L6925

Tagging @phst just in case he reads github notifications and can provide guidance how we can avoid this issue in lsp-mode. Otherwise, if @phst does not reply here we will have to post to emacs-devel

This is apparently an interplay between dotnet runtime and posix_spawn and not sure how to fix this easily, this is over my head.. We could force a pseudo tty connection somehow maybe, that would would avoid posix_spawn but that incurs performance problems, I think..

[razzmatazz]

Another option would be following up that thread and reporting that this new optimisation causes breakage for LSP-mode, at least on Linux. I don’t think this change is “crucial” anywhere except MacOS?

Right now there doesn’t seem to be much awareness that this is causing issues.

Also: good digging!

ok, I did post on emacs-devel. I don't have much hope but lets see

• https://lists.gnu.org/archive/html/emacs-devel/2022-01/msg01561.html

[razzmatazz]

oh, it looks like that migration to posix_spawn broke emacs-28/stable on macos too :(

reopening until both patches (to master and -28 are applied)..

[razzmatazz]

mhm, can you run csharp-ls from M-x shell? do you have any i/o visible when M-x lsp-toggle-trace-io is enabled?

i don't think there is much magic.. i'm running 28.0.60 (emacs-28 prerelease branch) with --with-native-compilation -- but didn't try 29.0 yet (master)

this is on linux, right?

[razzmatazz]

I'm wondering if there could be changes with how $PATH is exposed or $DOTNET (env variables in general) on 29.0

[josteink]

Yeah running Ubuntu Linux (hirsute).

Just for kicks, I tried invoking csharp-ls from the command line, but I didn't get any output at all, even when pointing to a valid solution file using -s or even when enabling full logging using -l log or -l info. Same result when I tried building csharp-ls from latest source in Git master.

But I honestly didn't know what to expect, so I therefore didn't include it when filing this issue.

I'm honestly not sure what fails or why 😄

[razzmatazz]

try csharp-ls --version or csharp-ls --help -- it should emit things

» dotnet tool install -g csharp-ls
dotnet tool install -g csharp-ls
You can invoke the tool using the following command: csharp-ls
Tool 'csharp-ls' (version '0.1.8') was successfully installed.

» csharp-ls --version
csharp-ls --version
csharp-ls, 0.1.8.0

» csharp-ls --help
csharp-ls --help
USAGE: csharp-ls [--help] [--version] [--loglevel <level>] [--solution <solution>]

OPTIONS:

    --version, -v         display versioning information
    --loglevel, -l <level>
                          log level, <log|info|warning|error>; default is `log`
    --solution, -s <solution>
                          .sln file to load (relative to CWD)
    --help                display this list of options.

[josteink]

That does seem to work without any issues (which is why I knew about -s and -l in the first place 😄 ).

jostein@thinkpad-t480s:~$ csharp-ls -v
csharp-ls, 0.1.8.0
jostein@thinkpad-t480s:~$ csharp-ls -h
ERROR: unrecognized argument: '-h'.
USAGE: csharp-ls [--help] [--version] [--loglevel <level>] [--solution <solution>]

OPTIONS:

    --version, -v         display versioning information
    --loglevel, -l <level>
                          log level, <log|info|warning|error>; default is `log`
    --solution, -s <solution>
                          .sln file to load (relative to CWD)
    --help                display this list of options.

[razzmatazz]

sorry to repeat, but do you have any LSP i/o visible when M-x lsp-toggle-trace-io is enabled? it should show what requests get sent and what is the response? is the server process alive, or does lsp-mode kill it or even does not start?

[josteink]

Enabling lsp-toggle-trace-io and then doing lsp-restart-workspace, I get the following outout in the `lsp-log buffer:

Command "csharp-ls" is present on the path.
Command "csharp-ls" is present on the path.
Found the following clients for /home/jostein/build/ZeroFootprintDocuments/src/SuperOffice.Documents.Web/Program.cs: (server-id csharp-ls, priority -2)
The following clients were selected based on priority: (server-id csharp-ls, priority -2)
Command "csharp-ls" is present on the path.
Command "csharp-ls" is present on the path.
Found the following clients for /home/jostein/build/ZeroFootprintDocuments/src/SuperOffice.Documents.Web/Controllers/DeploymentController.cs: (server-id csharp-ls, priority -2)
The following clients were selected based on priority: (server-id csharp-ls, priority -2)
...

And so it goes on. Nothing else shows up.

I see I now also have some other buffers like lsp-log: csharp-ls:271234. It contains something this:

[Trace - 08:34:01 ] Sending request 'initialize - (159)'.
Params: {
  "processId": null,
  "rootPath": "/home/jostein/build/ZeroFootprintDocuments",
  "clientInfo": {
    "name": "emacs",
    "version": "GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.25, cairo version 1.16.0)\n of 2021-12-15"
  },
  "rootUri": "file:///home/jostein/build/ZeroFootprintDocuments",
  "capabilities": {
    "workspace": {
      "workspaceEdit": {
        "documentChanges": true,
        "resourceOperations": [
          "create",
          "rename",
          "delete"
        ]
      },
      "applyEdit": true,
      "symbol": {
        "symbolKind": {
          "valueSet": [
            1,
            2,
            3,
            4,
            5,
            6,
            7,
            8,
            9,
            10,
            11,
            12,
            13,
            14,
            15,
            16,
            17,
            18,
            19,
            20,
            21,
            22,
            23,
            24,
            25,
            26
          ]
        }
      },
      "executeCommand": {
        "dynamicRegistration": false
      },
      "didChangeWatchedFiles": {
        "dynamicRegistration": true
      },
      "workspaceFolders": true,
      "configuration": true,
      "codeLens": {
        "refreshSupport": true
      },
      "fileOperations": {
        "didCreate": false,
        "willCreate": false,
        "didRename": false,
        "willRename": false,
        "didDelete": false,
        "willDelete": false
      }
    },
    "textDocument": {
      "declaration": {
        "linkSupport": true
      },
      "definition": {
        "linkSupport": true
      },
      "implementation": {
        "linkSupport": true
      },
      "typeDefinition": {
        "linkSupport": true
      },
      "synchronization": {
        "willSave": true,
        "didSave": true,
        "willSaveWaitUntil": true
      },
      "documentSymbol": {
        "symbolKind": {
          "valueSet": [
            1,
            2,
            3,
            4,
            5,
            6,
            7,
            8,
            9,
            10,
            11,
            12,
            13,
            14,
            15,
            16,
            17,
            18,
            19,
            20,
            21,
            22,
            23,
            24,
            25,
            26
          ]
        },
        "hierarchicalDocumentSymbolSupport": true
      },
      "formatting": {
        "dynamicRegistration": true
      },
      "rangeFormatting": {
        "dynamicRegistration": true
      },
      "rename": {
        "dynamicRegistration": true,
        "prepareSupport": true
      },
      "codeAction": {
        "dynamicRegistration": true,
        "isPreferredSupport": true,
        "codeActionLiteralSupport": {
          "codeActionKind": {
            "valueSet": [
              "",
              "quickfix",
              "refactor",
              "refactor.extract",
              "refactor.inline",
              "refactor.rewrite",
              "source",
              "source.organizeImports"
            ]
          }
        },
        "resolveSupport": {
          "properties": [
            "edit",
            "command"
          ]
        },
        "dataSupport": true
      },
      "completion": {
        "completionItem": {
          "snippetSupport": true,
          "documentationFormat": [
            "markdown",
            "plaintext"
          ],
          "resolveAdditionalTextEditsSupport": true,
          "insertReplaceSupport": true,
          "deprecatedSupport": true,
          "resolveSupport": {
            "properties": [
              "documentation",
              "details",
              "additionalTextEdits",
              "command"
            ]
          },
          "insertTextModeSupport": {
            "valueSet": [
              1,
              2
            ]
          }
        },
        "contextSupport": true
      },
      "signatureHelp": {
        "signatureInformation": {
          "parameterInformation": {
            "labelOffsetSupport": true
          }
        }
      },
      "documentLink": {
        "dynamicRegistration": true,
        "tooltipSupport": true
      },
      "hover": {
        "contentFormat": [
          "markdown",
          "plaintext"
        ]
      },
      "foldingRange": {
        "dynamicRegistration": true
      },
      "callHierarchy": {
        "dynamicRegistration": false
      },
      "publishDiagnostics": {
        "relatedInformation": true,
        "tagSupport": {
          "valueSet": [
            1,
            2
          ]
        },
        "versionSupport": true
      },
      "linkedEditingRange": {
        "dynamicRegistration": true
      }
    },
    "window": {
      "workDoneProgress": true,
      "showMessage": null,
      "showDocument": {
        "support": true
      }
    }
  },
  "initializationOptions": null,
  "workDoneToken": "1"
}

Note how request seems to be sent, but no response seems to be received.

[razzmatazz]

this most probably something to do with inferior process/IPC support vs LSP lib we use..

could be something related to pipes.. https://raw.githubusercontent.com/emacs-mirror/emacs/master/ChangeLog.3 has some references to pipe changes..

have you checked FSAC (F# language server) -- does it work on master ? I am using the same LSP lib as FSAC does so you could check if it is the LSP lib or something to do with csharp-ls itself..

[josteink]

Sorry about the slow reply!

Trying to update to latest FASC and testing in a F# buffer, I'm getting a different error. Basically the LSP process crashes and LSP asks if I want to restart it.

LSP :: Connected to [fsac:1400258/starting].
LSP :: fsac has exited (exited abnormally with code 3)
Server fsac:1400258/starting exited with status exit(check corresponding stderr buffer for details). Do you want to restart it? (y or n) n

So it's failing, but in a completely different way than csharp-language-server does.

Not sure what that means honestly :)

[razzmatazz]

uh oh.. I am not on emacs-29 so you're on your own for now :E

I could bother to take a look if you really need it, but since this looks to be something generic (related to multiple LSP servers..) I'm afraid I would not be effective in finding the cause

[josteink]

No worries. I’ll just use Omnisharp in the meantime. 😄

[hungyiloo]

I'm on emacs 29 too, with the same issue on both csharp-ls and fsac. Not super familiar with inner workings of LSP, but happy to help answer questions and debug when I have time.

From the quick test I just did, fsac also fails with eglot-fsharp, so the issue might even be outside of lsp-mode (?!)

I can confirm omnisharp works as a fallback. My other LSP backends also seem to be working.

[razzmatazz]

I have just published 0.2.0, which now uses StreamJsonRpc lib for LSP pipes, can you check if things are better now?

fsac uses the same lib and once I publish the changes to Ionide.LanguageServerProtocol they will use it so hopefully we will hit two rabbits with one stone

[hungyiloo]

Thanks so much for the release! Unfortunately it's still behaving the same for me. I've also tried updating to the latest emacs .deb from konstare.

Is there anything else I can help you check?

[josteink]

Yeah. Same here. Tested and it still behaves the same. Very weird!

[josteink]

Might be worth emailing [email protected] about?

Sounds like a breaking change somewhere, and everyone may not be aware of it, even internally among the core Emacs-developers.

[wizardlink]

Would just like to add that I am having the same issues in my machine, this from a clean install of Fedora 35 too.

[razzmatazz]

I have a suspicion that 50944dc might fix the issue. I was googling a bit and this rings bells to me: https://stackoverflow.com/questions/6081946/why-does-console-readline-have-a-limit-on-the-length-of-text-it-allows/6081967#6081967

Release here:

• https://www.nuget.org/packages/csharp-ls/0.2.1 (might take a couple of minutes to be visible)

Otherwise I will have to compile emacs-29 and check it myself :)

[wizardlink]

I have a suspicion that 50944dc might fix the issue. I was googling a bit and this rings bells to me: https://stackoverflow.com/questions/6081946/why-does-console-readline-have-a-limit-on-the-length-of-text-it-allows/6081967#6081967

Release here:

* https://www.nuget.org/packages/csharp-ls/0.2.1 (might take a couple of minutes to be visible)

Otherwise I will have to compile emacs-29 and check it myself :)

Still having the same issue:

Here's the evidence of my versionings:

I wonder if this is just another cursed problem with my environment though, so maybe it's not the same issue as the other reporters.

[razzmatazz]

ok,I can replicate this with emacs@master too -- bisecting the causal commit

[josteink]

Another option would be following up that thread and reporting that this new optimisation causes breakage for LSP-mode, at least on Linux. I don’t think this change is “crucial” anywhere except MacOS?

Right now there doesn’t seem to be much awareness that this is causing issues.

Also: good digging!

[razzmatazz]

thing is, it mostly related to dotnet runtime -- this not an emacs-lsp-wide issue, as something like clojure-lsp seems to work ok on the same emacs@master.. so dotnet/runtime does some sniffing here and turns a switch or two that breaks IPC over stdio when spawned with posix_spawn instead of vfork..

[razzmatazz]

Mhm, ms-Python-ls runs on dotnet, wonder if that breaks too..

[josteink]

Right.

IME it’s by far muuuch easier to run a git snapshot version of Emacs than dotnet, and chances we get to push Emacs-specific bug fixes into the .NET HAL/platform layer is also much smaller than getting proven regressions in Emacs reverted 😉

[josteink]

I've added my 2 cents to that thread.

Always nice to have more than one guy voicing his concerns when you want to someone to revert merged commits 😄

[phst]

I'm happy to take a look, but I fear that I'd need a minimal example as well, since this appears to be a rather subtle bug.

[razzmatazz]

dotnet stack report shows this for csharp-ls, the process is waiting for another process to exit

[bob@fedora emacs]$ dotnet stack report -p 53193
Thread (0xCFC9):
  [Native Frames]
  System.Private.CoreLib!System.Threading.WaitHandle.WaitOneNoCheck(int32)
  System.Private.CoreLib!System.Threading.WaitHandle.WaitOne(int32)
  System.Diagnostics.Process!System.Diagnostics.ProcessWaitState.WaitForExit(int32)
  System.Diagnostics.Process!System.Diagnostics.Process.WaitForExitCore(int32)
  Microsoft.Build.Locator!Microsoft.Build.Locator.DotNetSdkLocationHelper+<GetDotNetBasePaths>d__5.MoveNext()
  Microsoft.Build.Locator!Microsoft.Build.Locator.DotNetSdkLocationHelper+<GetInstances>d__4.MoveNext()
  Microsoft.Build.Locator!Microsoft.Build.Locator.MSBuildLocator+<GetInstances>d__20.MoveNext()
  System.Linq!System.Linq.Enumerable.TryGetFirst(class System.Collections.Generic.IEnumerable`1<!!0>,bool&)
  System.Linq!System.Linq.Enumerable.FirstOrDefault(class System.Collections.Generic.IEnumerable`1<!!0>)
  Microsoft.Build.Locator!Microsoft.Build.Locator.MSBuildLocator.RegisterDefaults()
  CSharpLanguageServer!CSharpLanguageServer.Program.entry(class System.String[])

ps axl shows there is a zombie process that dotnet is waiting on but somehow it could not detect it has ended and did not call wait(3) yet; -- is some signal is blocked with posix_spawn?

0  1000   53193   53129  20   0 3437284 59416 -     Ssl  ?          0:00 /home/bob/src/csharp-language-server/src/CSharpLanguageServer/bin/Debug/net6.0/CSharpLanguageServer
0  1000   53203   53193  20   0      0     0 -      Z    ?          0:00 [dotnet] <defunct>

[razzmatazz]

FSAC seems to be blocked with a very similar trace, i.e. waiting for a process to exit (SIGCHLD) but not getting it(?)

[bob@fedora emacs]$ dotnet stack report -p55454
Thread (0xD89E):
  [Native Frames]
  System.Private.CoreLib!System.Threading.WaitHandle.WaitOneNoCheck(int32)
  System.Private.CoreLib!System.Threading.WaitHandle.WaitOne(int32)
  System.Diagnostics.Process!System.Diagnostics.ProcessWaitState.WaitForExit(int32)
  System.Diagnostics.Process!System.Diagnostics.Process.WaitForExitCore(int32)
  Ionide.ProjInfo!Ionide.ProjInfo.SdkDiscovery.execDotnet(class System.IO.DirectoryInfo,class System.IO.FileInfo,class System.Collections.Generic.IEnumerable`1<class System.String>)
  Ionide.ProjInfo!Ionide.ProjInfo.SdkDiscovery.versionAt(class System.IO.DirectoryInfo,class System.IO.FileInfo)
  Ionide.ProjInfo!Ionide.ProjInfo.Init.init(class System.IO.DirectoryInfo,class Microsoft.FSharp.Core.FSharpOption`1<class System.IO.FileInfo>)
  fsautocomplete!FsAutoComplete.Program.entry(class System.String[])

[razzmatazz]

ok, attempting to patch src/callproc.cs with

diff --git a/src/callproc.c b/src/callproc.c
index 4d3b0bb8e0..8ea290f108 100644
--- a/src/callproc.c
+++ b/src/callproc.c
@@ -1350,14 +1350,7 @@ emacs_posix_spawn_init_attributes (posix_spawnattr_t *attributes)
   sigset_t sigdefault;
   sigemptyset (&sigdefault);
 
-#ifdef DARWIN_OS
-  /* Work around a macOS bug, where SIGCHLD is apparently
-     delivered to a vforked child instead of to its parent.  See:
-     https://lists.gnu.org/r/emacs-devel/2017-05/msg00342.html
-  */
   sigaddset (&sigdefault, SIGCHLD);
-#endif
-
   sigaddset (&sigdefault, SIGINT);
   sigaddset (&sigdefault, SIGQUIT);
 #ifdef SIGPROF

I really don't know what I'm doing but maybe that will lead us somewhere.. :)

[razzmatazz]

nah, that didn't work ^^^

There is also some code in src/callproc.c that says

  /* Stop blocking SIGCHLD in the child.  */
  sigset_t oldset;
  error = pthread_sigmask (SIG_SETMASK, NULL, &oldset);
  if (error != 0)
    goto out;
  error = posix_spawnattr_setsigmask (attributes, &oldset);
  if (error != 0)
    goto out;

but I'm out of my depth to muck with it @phst

[josteink]

Awesome. So I actually reported a core Emacs bug, and didn't know it.

Case closed, I guess? 😄