Fulcrum crashes with sigsegv every few minutes

[cculianu]

Fulcrum crashes with sigsegv every few minutes

[jhoenicke]

dmesg shows address and code:

[750055.820563] PeerMgr[1290]: segfault at 20 ip 000055da03629b02 sp 00007fdbd27fbb60 error 4 in Fulcrum[55da034c2000+ae6000]
[750055.820573] Code: 48 8b 75 10 49 8d 7d 20 e8 a5 a4 65 00 4c 89 ef e8 73 e0 07 00 e9 ae fd ff ff 66 0f 1f 44 00 00 4c 8b b5 98 00 00 00 4d 8b 3e <45> 8b 6f 20 45 85 ed 75 65 bf 10 00 00 00 4c 8d ac 24 d0 00 00 00
[774142.623418] PeerMgr[2742997]: segfault at 20 ip 000055d3b24e9b02 sp 00007f461f3fbb60 error 4 in Fulcrum[55d3b2382000+ae6000]
[774142.623428] Code: 48 8b 75 10 49 8d 7d 20 e8 a5 a4 65 00 4c 89 ef e8 73 e0 07 00 e9 ae fd ff ff 66 0f 1f 44 00 00 4c 8b b5 98 00 00 00 4d 8b 3e <45> 8b 6f 20 45 85 ed 75 65 bf 10 00 00 00 4c 8d ac 24 d0 00 00 00
[774879.016268] PeerMgr[2829805]: segfault at 20 ip 00005555944acb02 sp 00007f9262bfbb60 error 4 in Fulcrum[555594345000+ae6000]
[774879.016277] Code: 48 8b 75 10 49 8d 7d 20 e8 a5 a4 65 00 4c 89 ef e8 73 e0 07 00 e9 ae fd ff ff 66 0f 1f 44 00 00 4c 8b b5 98 00 00 00 4d 8b 3e <45> 8b 6f 20 45 85 ed 75 65 bf 10 00 00 00 4c 8d ac 24 d0 00 00 00
[778555.002453] PeerMgr[2832287]: segfault at 20 ip 000055788f80bb02 sp 00007fc6cb3fbb60 error 4 in Fulcrum[55788f6a4000+ae6000]
[778555.002463] Code: 48 8b 75 10 49 8d 7d 20 e8 a5 a4 65 00 4c 89 ef e8 73 e0 07 00 e9 ae fd ff ff 66 0f 1f 44 00 00 4c 8b b5 98 00 00 00 4d 8b 3e <45> 8b 6f 20 45 85 ed 75 65 bf 10 00 00 00 4c 8d ac 24 d0 00 00 00

Binary is Fulcrum-1.5.0-x86_64-linux (unmodified). The position seems to be in the function:
_ZN3RPC14ConnectionBase11processJsonEO10QByteArray, address 167B02:

  167af8:       4c 8b b5 98 00 00 00    mov    0x98(%rbp),%r14
  167aff:       4d 8b 3e                mov    (%r14),%r15
  167b02:       45 8b 6f 20             mov    0x20(%r15),%r13d
  167b06:       45 85 ed                test   %r13d,%r13d
  167b09:       75 65                   jne    167b70 <_ZN3RPC14ConnectionBase11processJsonEO10QByteArray+0x310>
  167b0b:       bf 10 00 00 00          mov    $0x10,%edi
  167b10:       4c 8d ac 24 d0 00 00    lea    0xd0(%rsp),%r13
  167b17:       00 
  167b18:       e8 33 90 f0 ff          callq  70b50 <__cxa_allocate_exception@plt>
  167b1d:       48 8d 35 7a bd 6b 00    lea    0x6bbd7a(%rip),%rsi        # 82389e <_ZTSN10RecordFile13FileOpenErrorE+0x77e>
  167b24:       4c 89 ef                mov    %r13,%rdi
  167b27:       49 89 c6                mov    %rax,%r14
  167b2a:       e8 c1 6f f3 ff          callq  9eaf0 <_ZN7QStringC1EPKc>
  167b2f:       4c 89 ee                mov    %r13,%rsi
  167b32:       4c 89 f7                mov    %r14,%rdi
  167b35:       e8 66 7e f3 ff          callq  9f9a0 <_ZN9ExceptionC1ERK7QString>
  167b3a:       48 8d 05 a7 21 b9 00    lea    0xb921a7(%rip),%rax        # cf9ce8 <_ZTVN3RPC14ConnectionBase13UnknownMethodE>

Should be roughly: https://github.com/cculianu/Fulcrum/blob/v1.5.0/src/RPC.cpp#L430

[cculianu]

Yeah, I know. There was a dangling reference bug in the codebase. It has been fixed in 1.5.2. Please upgrade to latest.

[cculianu]

Also thanks for the detailed report and sorry about the crash and need to db resynch. I .. have no explanation other than this was a bug lurking in the codebase since forever and somehow some misbehaving BTC servers were able to trigger it (either intentionally or accidentally). The good thing is the bug was fixed. The bad thing is you have to resynch. But please do use 1.5.2.

If it makes you feel better my server also got hit hard by this.. and I suffered as well until I fixed it. Sorry. I promise now Fulcrum is bug-free! :)