Server crash when hedgedoc container infrastructure is missing avx cpu flag with "Illegal instruction"

[Guilamb]

Server crash when hedgedoc container infrastructure is missing avx cpu flag with "Illegal instruction"

[gberche-orange]

The issue we were observing seems related to hedgedoc container image provided at https://docs.hedgedoc.org/setup/docker/, likely requiring use of AVX extension in the CPU instruction set, see full details into orange-cloudfoundry/paas-templates#2195

The problem does not reproduce after modifying the container infrastructure with a linux kernel properly exposing the CPU avx capabilities.

If other hedgedoc users are running old cpus without the avx capability, then it would be helpful to confirm which nodejs library is requiring avx to consider alternatives.

For our part, the lack of avx capability is rather an infrastructure-specific issue which can be best addressed at the infrastructure level.

[Guilamb]

I've tried to get a dump core of the app using the flag : "--abort-on-uncaught-exception", but without success.
I have also tried a bunch of other options in my search for useful traces to debug the application without getting anything that seemed relevant to me.

NODE_DEBUG=cluster,net,http,fs,tls,module,timers nodejs --report-uncaught-exception --report-on-signal --report-on-fatalerror --report-compact --abort-on-uncaught-exception --report-filename="/hedgedoc/public/uploads/report.json" app.js >public/uploads/logs.txt 2>&1

Here a tail of the 50 last lines before crashing with the previous flags :

NET 7: _read - n 16384 isConnecting? false hasHandle? true
NET 7: onconnection
NET 7: _read - n 16384 isConnecting? false hasHandle? true
NET 7: Socket._handle.readStart
HTTP 7: SERVER new http connection
NET 7: onconnection
NET 7: _read - n 16384 isConnecting? false hasHandle? true
NET 7: Socket._handle.readStart
HTTP 7: SERVER new http connection
HTTP 7: SERVER socketOnParserExecute 111
HTTP 7: write ret = true
HTTP 7: SERVER socketOnParserExecute 110
2023-11-08T15:30:03.765Z info:  Executing (default): SELECT `sid`, `expires`, `data`, `createdAt`, `updatedAt` FROM `Sessions` AS `Session` WHERE `Session`.`sid` = 'G9gb8WsrBXmlRZAoj-p99ktuaV5ozV04';
2023-11-08T15:30:03.766Z info:  Executing (default): SELECT count(*) AS `count` FROM `Notes` AS `Note`;
NET 7: _read - n 16384 isConnecting? false hasHandle? true
NET 7: _read - n 16384 isConnecting? false hasHandle? true
2023-11-08T15:30:03.767Z info:  Executing (default): SELECT count(*) AS `count` FROM `Users` AS `User`;
2023-11-08T15:30:03.768Z info:  Executing (default): INSERT INTO `Sessions` (`sid`,`expires`,`data`,`createdAt`,`updatedAt`) VALUES (?,?,?,?,?);
NET 7: _read - n 16384 isConnecting? false hasHandle? true
HTTP 7: write ret = true
HTTP 7: outgoing message end.
NET 7: _final: not ended, call shutdown()
2023-11-08T15:30:03.769Z info:  10.16.18.69 - - [08/Nov/2023:15:30:03 +0000] "GET /status HTTP/1.1" 200 266 "-" "kube-probe/1.24"

NET 7: afterShutdown destroyed=false
NET 7: destroy
NET 7: close
NET 7: close handle
NET 7: has server
NET 7: SERVER _emitCloseIfDrained
NET 7: SERVER handle? true   connections? 2
NET 7: _read - n 16384 isConnecting? false hasHandle? true
NET 7: emit close
HTTP 7: server socket close
HTTP 7: write ret = true
HTTP 7: outgoing message end.
NET 7: _final: not ended, call shutdown()
2023-11-08T15:30:03.770Z info:  10.16.18.69 - - [08/Nov/2023:15:30:03 +0000] "GET /_health HTTP/1.1" 200 14 "-" "kube-probe/1.24"

NET 7: afterShutdown destroyed=false
NET 7: destroy
NET 7: close
NET 7: close handle
NET 7: has server
NET 7: SERVER _emitCloseIfDrained
NET 7: SERVER handle? true   connections? 1
NET 7: emit close
HTTP 7: server socket close
NET 7: _read - n 16384 isConnecting? false hasHandle? true
Illegal instruction

[timmwille]

Wasn't that fixed some time back? I remember we had that issue about 2019 or so? I can research deeper if info needed

[Guilamb]

If you found the old issue maybe it could help, also this post from the forum seems to have similar symptoms : https://community.hedgedoc.org/t/cannot-cut-copy-paste/996/4

[Guilamb]

Wasn't that fixed some time back? I remember we had that issue about 2019 or so? I can research deeper if info needed

So, I looked up similar old issues to try to find out the issue you were talking about :

• #67
• TFNS/CTFNote#18
• #130
• #697

However, they seem to be related to configuration issues with unset CMD_DOMAIN, CMD_PROTOCOL_USESSL, and CMD_URL_ADDPORT. In my case, they are set up, as you can see in the configuration specified above. Only the CMD_URL_ADDPORT hasn't been specified because the default value is already false.

If you have any ideas, it would be helpful. On my side, I'll try to obtain more logs or traces from Hedgedoc.