segfault on gdb 7.9

[snare]

segfault on gdb 7.9

[richo]

Argh, yeah I think this is my fault. I'll give you a patch to test today.

I wouldn't expect that to actually segfault though. Does it still fault
without voltron?

On Tue, May 19, 2015 at 8:46 AM, ocean [email protected] wrote:

os: Ubuntu 15.04 gdb: GNU gdb (Ubuntu 7.9-1ubuntu1) 7.9
elf file is
https://github.com/ctfs/write-ups-2015/tree/master/defcon-qualifier-ctf-2015/reverse/pr0dk3y

i do load the file with gdb: gdb ./pr0dk3y_40687b492c80205cccb34db1eabf6456
bp *0 to break as soon as it executes then info file, bpd 1 and bp
*entry_point (this is to break on entry point since the elf has PIE)

it will break and seems to work but as soon as i single step one or at
most two instructions (either si/ni) gdb will segfault:

0x0000555555554a65 in ?? ()
/build/buildd/gdb-7.9/gdb/common/cleanups.c:265: internal-warning: restore_my_cleanups has found a stale cleanup
A problem internal to GDB has been detected,
further debugging may prove unreliable.

This is a bug, please report it. For instructions, see:
http://www.gnu.org/software/gdb/bugs/.

[1] 5903 abort (core dumped) gdb pr0dk3y_40687b492c80205cccb34db1eabf6456or:Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/voltron-0.1-py3.4.egg/voltron/core.py", line 215, in run
rfds, _, _ = select.select([serv, self.exit_pipe] + self.clients, [], [])
InterruptedError: [Errno 4] Chiamata di sistema interrotta

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.4/threading.py", line 920, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.4/dist-packages/voltron-0.1-py3.4.egg/voltron/core.py", line 218, in run
if ex[0] == errno.EINTR: # interrupted system call
TypeError: 'InterruptedError' object is not subscriptable

0x0000555555554a84 in ?? ()```

here is my gdb-init:

voltron init

#source ~/.fg_gdbinit```
i did both import and remove https://github.com/gdbinit/Gdbinit but still keeps on crashing.

—
Reply to this email directly or view it on GitHub
https://github.com/snare/voltron/issues/81.

[richo]

I pushed the ocean-debug branch to richo/voltron. This patch is definitely
not something that should go upstream, but can you let me know if it fixes
the bug, and I can do something less ghetto.

On Tue, May 19, 2015 at 8:53 AM, Richo Healey [email protected] wrote:

Argh, yeah I think this is my fault. I'll give you a patch to test today.

I wouldn't expect that to actually segfault though. Does it still fault
without voltron?

On Tue, May 19, 2015 at 8:46 AM, ocean [email protected] wrote:

os: Ubuntu 15.04 gdb: GNU gdb (Ubuntu 7.9-1ubuntu1) 7.9
elf file is
https://github.com/ctfs/write-ups-2015/tree/master/defcon-qualifier-ctf-2015/reverse/pr0dk3y

i do load the file with gdb: gdb
./pr0dk3y_40687b492c80205cccb34db1eabf6456
bp *0 to break as soon as it executes then info file, bpd 1 and bp
*entry_point (this is to break on entry point since the elf has PIE)

it will break and seems to work but as soon as i single step one or at
most two instructions (either si/ni) gdb will segfault:

0x0000555555554a65 in ?? ()
/build/buildd/gdb-7.9/gdb/common/cleanups.c:265: internal-warning: restore_my_cleanups has found a stale cleanup
A problem internal to GDB has been detected,
further debugging may prove unreliable.

This is a bug, please report it. For instructions, see:
http://www.gnu.org/software/gdb/bugs/.

[1] 5903 abort (core dumped) gdb pr0dk3y_40687b492c80205cccb34db1eabf6456or:Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/voltron-0.1-py3.4.egg/voltron/core.py", line 215, in run
rfds, _, _ = select.select([serv, self.exit_pipe] + self.clients, [], [])
InterruptedError: [Errno 4] Chiamata di sistema interrotta

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.4/threading.py", line 920, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.4/dist-packages/voltron-0.1-py3.4.egg/voltron/core.py", line 218, in run
if ex[0] == errno.EINTR: # interrupted system call
TypeError: 'InterruptedError' object is not subscriptable

0x0000555555554a84 in ?? ()```

here is my gdb-init:

voltron init

#source ~/.fg_gdbinit```
i did both import and remove https://github.com/gdbinit/Gdbinit but still keeps on crashing.

—
Reply to this email directly or view it on GitHub
https://github.com/snare/voltron/issues/81.

[ocean1]

it does still crash, tough it lasts a few ni more (obviously without voltron it doesn't crash), this is the backtrace from the core dump:

#0  0x00007f796ab47267 in __GI_raise (sig=sig@entry=0x6) at ../sysdeps/unix/sysv/linux/raise.c:55
#1  0x00007f796ab48eca in __GI_abort () at abort.c:89
#2  0x0000000000680f96 in dump_core ()
#3  0x000000000068353d in ?? ()
#4  0x00000000006835f9 in internal_verror ()
#5  0x00000000006b41ff in internal_error ()
#6  0x00000000005ad7dc in ?? ()
#7  0x000000000059664a in ?? ()
#8  0x000000000067ec7c in execute_command ()
#9  0x000000000067ed6b in execute_command_to_string ()
#10 0x00000000004f5fec in ?? ()
#11 0x00007f796bb3500f in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#12 0x00007f796bb34ebb in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#13 0x00007f796bb36cf3 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#14 0x00007f796bb34d66 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#15 0x00007f796bb36cf3 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#16 0x00007f796babb496 in ?? () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#17 0x00007f796bb3b138 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#18 0x00007f796bb31b8b in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#19 0x00007f796bb36cf3 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#20 0x00007f796bb34d66 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#21 0x00007f796bb36cf3 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#22 0x00007f796babb496 in ?? () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#23 0x00007f796bb3b138 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#24 0x00007f796bb31b8b in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#25 0x00007f796bb36cf3 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#26 0x00007f796bb34d66 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#27 0x00007f796bb36cf3 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#28 0x00007f796bb34d66 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#29 0x00007f796bb36cf3 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#30 0x00007f796bb34d66 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#31 0x00007f796bb34ebb in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#32 0x00007f796bb34ebb in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#33 0x00007f796bb36cf3 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#34 0x00007f796babb3a9 in ?? () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#35 0x00007f796bb3b138 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#36 0x00007f796bb8e1ad in ?? () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#37 0x00007f796bb3b138 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#38 0x00007f796bb2dab7 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#39 0x00007f796bb70df2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#40 0x00007f796b7756aa in start_thread (arg=0x7f7965b0a700) at pthread_create.c:333
#41 0x00007f796ac18eed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

[richo]

could you install the gdb-dbg package so that those other frames get
annotated?

On Tue, May 19, 2015 at 9:39 AM, ocean [email protected] wrote:

it does still crash, tough it lasts a few ni more (obviously without
voltron it doesn't crash):

#0 0x00007f796ab47267 in __GI_raise (sig=sig@entry=0x6) at ../sysdeps/unix/sysv/linux/raise.c:55
#1 0x00007f796ab48eca in __GI_abort () at abort.c:89
#2 0x0000000000680f96 in dump_core ()
#3 0x000000000068353d in ?? ()
#4 0x00000000006835f9 in internal_verror ()
#5 0x00000000006b41ff in internal_error ()
#6 0x00000000005ad7dc in ?? ()
#7 0x000000000059664a in ?? ()
#8 0x000000000067ec7c in execute_command ()
#9 0x000000000067ed6b in execute_command_to_string ()
#10 0x00000000004f5fec in ?? ()
#11 0x00007f796bb3500f in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#12 0x00007f796bb34ebb in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#13 0x00007f796bb36cf3 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#14 0x00007f796bb34d66 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#15 0x00007f796bb36cf3 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#16 0x00007f796babb496 in ?? () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#17 0x00007f796bb3b138 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#18 0x00007f796bb31b8b in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#19 0x00007f796bb36cf3 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#20 0x00007f796bb34d66 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#21 0x00007f796bb36cf3 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#22 0x00007f796babb496 in ?? () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#23 0x00007f796bb3b138 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#24 0x00007f796bb31b8b in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#25 0x00007f796bb36cf3 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#26 0x00007f796bb34d66 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#27 0x00007f796bb36cf3 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#28 0x00007f796bb34d66 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#29 0x00007f796bb36cf3 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#30 0x00007f796bb34d66 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#31 0x00007f796bb34ebb in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#32 0x00007f796bb34ebb in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#33 0x00007f796bb36cf3 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#34 0x00007f796babb3a9 in ?? () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#35 0x00007f796bb3b138 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#36 0x00007f796bb8e1ad in ?? () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#37 0x00007f796bb3b138 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#38 0x00007f796bb2dab7 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#39 0x00007f796bb70df2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#40 0x00007f796b7756aa in start_thread (arg=0x7f7965b0a700) at pthread_create.c:333
#41 0x00007f796ac18eed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

—
Reply to this email directly or view it on GitHub
#81 (comment).

[ocean1]

this is the backtrace: http://pastebin.com/mCYygFKq

sometimes the error changes:

Using the running image of child process 5003.
Program stopped at 0x555555554a6d.
It stopped after being stepped.
[1]    4373 segmentation fault (core dumped)  gdb pr0dk3y_40687b492c80205cccb34db1eabf6456

0x00005555555549d0 in __libc_start_main@plt ()
gdb$ Fatal Python error: This thread state must be current when releasing

Thread 0x00007f861e338700 (most recent call first):
  File "/usr/lib/python3.4/threading.py", line 290 in wait
  File "/usr/lib/python3.4/threading.py", line 553 in wait
  File "/usr/local/lib/python3.4/dist-packages/voltron-0.1-py3.4.egg/voltron/plugins/api/wait.py", line 49 in dispatch
  File "/usr/local/lib/python3.4/dist-packages/voltron-0.1-py3.4.egg/voltron/api.py", line 100 in inner
  File "/usr/local/lib/python3.4/dist-packages/voltron-0.1-py3.4.egg/voltron/core.py", line 163 in dispatch_request
  File "/usr/lib/python3.4/threading.py", line 868 in run
  File "/usr/lib/python3.4/threading.py", line 920 in _bootstrap_inner
  File "/usr/lib/python3.4/threading.py", line 888 in _bootstrap

Current thread 0x00007f861f33a700 (most recent call first):
  File "/usr/local/lib/python3.4/dist-packages/voltron-0.1-py3.4.egg/voltron/core.py", line 486 in send
  File "/usr/local/lib/python3.4/dist-packages/voltron-0.1-py3.4.egg/voltron/core.py", line 532 in send_response
  File "/usr/local/lib/python3.4/dist-packages/voltron-0.1-py3.4.egg/voltron/core.py", line 175 in dispatch_request
  File "/usr/local/lib/python3.4/dist-packages/voltron-0.1-py3.4.egg/voltron/core.py", line 136 in handle_request
  File "/usr/local/lib/python3.4/dist-packages/voltron-0.1-py3.4.egg/voltron/core.py", line 242 in run
  File "/usr/lib/python3.4/threading.py", line 920 in _bootstrap_inner
  File "/usr/lib/python3.4/threading.py", line 888 in _bootstrap

Thread 0x00007f862679d780 (most recent call first):
[1]    6779 abort (core dumped)  gdb pr0dk3y_40687b492c80205cccb34db1eabf6456

0x0000555555554a65 in ?? ()
/build/buildd/gdb-7.9/gdb/common/cleanups.c:265: internal-warning: restore_my_cleanups has found a stale cleanup
A problem internal to GDB has been detected,
further debugging may prove unreliable.
*** Error in `gdb': free(): invalid pointer: 0x00000000007c18c0 ***
[1]    2509 segmentation fault (core dumped)  gdb pr0dk3y_40687b492c80205cccb34db1eabf6456

[ocean1]

yet two other backtraces which look different:
http://pastebin.com/SF0bDjFg this one trying to execute reg read from fg! gdbinit
http://pastebin.com/FbGADJXm this one just stepping

I did notice also that sometimes gdb hangs on quit command and I've got to kill it, and only a few rare times it quits correctly but showing an error statement: Python Exception <class 'AttributeError'> 'NoneType' object has no attribute 'stop': `Python Exception <class 'AttributeError'> 'NoneType' object has no attribute 'stop':``.
Could it be some problem with gdb cleanups (maybe even python garbage collector)?

[ocean1]

@richo : I asked on freenode #gdb referring to this bug report and they told me this: "from the backtrace at that link I see you're doing gdb operations from a separate thread." I don't know if this is actually true because I'm still digging inside voltron source but it's worth investigating this.

[snare]

That is true, Voltron does its queries from a background thread in the debugger. This was never a problem before but has started causing problems in GDB in recent versions.

On 21 May 2015, at 6:46 am, ocean [email protected] wrote:

@richo https://github.com/richo : I asked on freenode #gdb referring to this bug report and they told me this: "from the backtrace at that link I see you're doing gdb operations from a separate thread." I don't know if this is actually true because I'm still digging inside voltron source but it's worth investigating this.

—
Reply to this email directly or view it on GitHub #81 (comment).

[snare]

@ocean1 just to clarify, did they also follow that up with "don't do that"?

[snare]

Reworking the core comms stuff at the moment and will attempt to solve this issue.

[ocean1]

yes @snare, sorry but I didn't notice the last message, the GDB docs are clear about this, it is needed to call gdb commands only on the main thread! :) glad to hear you are trying to fix it.

[snare]

Awesome thanks

[snare]

Pushed out a version that should work on Linux w/GDB (tested on some random Ubuntu VM I had - I think from memory it's 14.04). There's a few bugs that I need to iron out still (e.g. it hangs for ~10s when first running the inferior) but it should be working if you wanna give it a run.

[snare]

"Fixed" by reworking the API so it can optionally block on the back end, in which case the request is added to a queue that is processed on the main thread each time the debugger stops. If the debugger is deemed to support async actions then they are performed immediately on a background thread, so we don't lose any functionality on LLDB (like updating views immediately upon launch etc).