wenzel/pyvmidbg

LibVMI-based debug server, implemented in Python. Building a guest aware, stealth and agentless full-system debugger

pyvmidbg

LibVMI-based GDB server, implemented in Python

Table of Contents

• Overview
• Features
• Requirements
• Install
• Usage
• Demo
• References
• Maintainers
• Contributing
• License

Overview

This GDB stub allows you to debug a remote process running in a VM with your favorite GDB frontend.

By leveraging virtual machine introspection, the stub remains stealth and requires no modification of the guest.

Why debugging from the hypervisor ?

Operating systems debug API's are problematic:

1. they have never been designed to deal with malwares, and lack the stealth and robustness required when analyzing malicious code
2. they have an observer effect, by implicitly modifying the process environment being debugged
3. this observer effect might be intentional to protect OS features (Windows PatchGuard/Protected Media Path are disabled)
4. modern OS have a high degree of kernel security mechanisms that narrows the debugger's view of the system (Windows 10 Virtual Secure Mode)
5. debugging low-level processes and kernel functions interacting directly with the transport protocol used by the debug agent can turn into a infinite recursion hell (eg. debugging TCP connections and having a kernel debug stub communicating via TCP)
6. in special cases the "Operating System" lacks debugging capabilities (unikernels)

Existing solutions like GDB stubs included in QEMU, VMware or VirtualBox can only pause the VM and debug the kernel, but lack the guest knowledge to track and follow the rest of the processes.

Project presentation at Insomni'Hack 2019:

• video
• slides

Vision

Current support:

• Stubs:
  • GDB
• Hypervisors:
  • Xen
  • KVM

State of hypervisor's VMI support

• Xen
  • 2011: Xen 4.1: first hypervisor to support VMI upstream
  • 2015: Xen 4.6: best hypervisor for VMI
  • libvmi: fully supported
  • pyvmidbg: supported
• KVM
  • 2017: BitDefender published a set VMI patches on the mailing list
  • libvmi: support is ongoing, see kvm-vmi/libvmi (branch kvmi)
  • pyvmidbg: supported
• VirtualBox
  • unofficial VMI patches thanks to Winbagility project
• VMware/Hyper-V: no sign of interest as of today

Features

• attach to existing process
  • Windows: find EPROCESS and ETHREADS state
  • Linux: pause at CR3 load
• attach new process (entrypoint):
  • Windows: follow first thread creation and break at entrypoint
  • Linux: not implemented
• singlestep/continue: wait for the process to be scheduled
  • process must have a single thread
• breakin (CTRL-C)
• software breakpoints

Requirements

• Python >= 3.4
• python3-docopt
• python3-lxml
• python3-libvmi
• Xen

Install

virtualenv -p python3 venv
source venv/bin/activate
pip install .

Note: If you don't want to install Xen, vagrant-xen-pyvmidbg provides a Vagrant environment based on KVM, with ready to use Windows and Linux VMs.

Usage

vmidbg <port> <vm> [<process>]

Demo

Debugging cmd.exe in Windows XP

Demo link

1. starts cmd.exe in Windows XP nested VM in Xen
2. starts pyvmidbg and target a process named cmd
3. connects to stub with radare2
4. set breakpoints on ntdll!NtOpenFile and ntkrnlpa!NtOpenFile
5. avoid breakpoints from the rest of the system, only hit if cmd.exe is executing

Debugging mspaint.exe in Windows 10

Debugging mspaint.exe

Limitations

• the VM must have 1 VCPU
• no steath breakpoints implemented yet (int3 into memory)

References

• vmidbg: original idea and C implementation
• plutonium-dbg: GDB server protocol parsing
• ollydbg2-python: GDB server protocol parsing
• GDB RSP protocol specifications

Maintainers

@Wenzel

Contributing

PRs accepted.

Small note: If editing the Readme, please conform to the standard-readme specification.

License

GNU General Public License v3.0

pyvmidbg's Issues

Truncated register 26 in remote 'g' packet

Trying to connect to an Ubuntu 20.04 VM results in this error on the gdb side:

(gdb) target remote 192.168.1.10:4567
Remote debugging using 192.168.1.10:4567
Truncated register 26 in remote 'g' packet

The vmidbg side looks like this:

# vmidbg -a 0.0.0.0 4567 ubuntu-20.04
INFO:server:listening on 0.0.0.0:4567
INFO:server:new client ('192.168.1.201', 50501)
INFO:RawDebugContext:attaching on ubuntu-20.04
INFO:GDBStub:connected
INFO:GDBStub:new packet: b'qSupported:multiprocess+;swbreak+;hwbreak+;qRelocInsn+;fork-events+;vfork-events+;exec-events+;vContSupported+;QThreadEvents+;no-resumed+;xmlRegisters=i386'
INFO:GDBStub:command q: GEN_QUERY_GET
INFO:GDBStub:command q: DONE
INFO:GDBStub:new packet: b'vMustReplyEmpty'
INFO:GDBStub:command v: V_FEATURES
INFO:GDBStub:command v: DONE
INFO:GDBStub:new packet: b'QStartNoAckMode'
INFO:GDBStub:command Q: GEN_QUERY_SET
INFO:GDBStub:command Q: DONE
INFO:GDBStub:new packet: b'Hg0'
INFO:GDBStub:command H: SET_THREAD_ID
INFO:GDBStub:command H: DONE
INFO:GDBStub:new packet: b'qTStatus'
INFO:GDBStub:command q: GEN_QUERY_GET
INFO:GDBStub:command q: DONE
INFO:GDBStub:new packet: b'qTfV'
INFO:GDBStub:command q: GEN_QUERY_GET
INFO:GDBStub:command q: FAIL
INFO:GDBStub:new packet: b'?'
INFO:GDBStub:command ?: TARGET_STATUS
INFO:GDBStub:command ?: DONE
INFO:GDBStub:new packet: b'qfThreadInfo'
INFO:GDBStub:command q: GEN_QUERY_GET
INFO:GDBStub:command q: DONE
INFO:GDBStub:new packet: b'qsThreadInfo'
INFO:GDBStub:command q: GEN_QUERY_GET
INFO:GDBStub:command q: DONE
INFO:GDBStub:new packet: b'qAttached'
INFO:GDBStub:command q: GEN_QUERY_GET
INFO:GDBStub:command q: DONE
INFO:GDBStub:new packet: b'Hc-1'
INFO:GDBStub:command H: SET_THREAD_ID
INFO:GDBStub:command H: DONE
INFO:GDBStub:new packet: b'qC'
INFO:GDBStub:command q: GEN_QUERY_GET
INFO:GDBStub:command q: DONE
INFO:GDBStub:new packet: b'qOffsets'
INFO:GDBStub:command q: GEN_QUERY_GET
INFO:GDBStub:command q: FAIL
INFO:GDBStub:new packet: b'g'
INFO:GDBStub:command g: READ_REGISTERS
INFO:GDBStub:command g: DONE

With gdbsx it works:

(gdb) target remote 192.168.1.10:4567
Remote debugging using 192.168.1.10:4567
0xffffffff81b906be in native_safe_halt () at ./arch/x86/include/asm/irqflags.h:60
60              asm volatile("sti; hlt": : :"memory");

Is it possible to use this project to debug userland processes running inside the Linux kernel guest from the QEMU GDB stub, and without Xen?

Salut Mathieu,

I've been looking for such capability for a while, and this project seems really close.

I summarized my findings so far: https://stackoverflow.com/questions/9561546/thread-aware-gdb-for-the-linux-kernel/51515470#51515470

My main question is: is that possible with this project? The main thing which might not work is that I'm not using Xen, just the QEMU GDB stub directly.

But you seem to be parsing Linux kernel process data structures, which is the hard part.

Or do you know any other project which has achieved this?

Fix pagefault injection

The current pagefault injection method uses a custom shellcode, 32 bits only.

We should use the vmi_request_page_fault API:
#35

VMI_EVENTS_VERSION

Hello dear @Wenzel,
When I run vmidbg on Arch Linux I got this:

Traceback (most recent call last):
  File "/usr/bin/vmidbg", line 33, in <module>
    sys.exit(load_entry_point('vmidbg==0.1', 'console_scripts', 'vmidbg')())
  File "/usr/bin/vmidbg", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.8/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/lib/python3.8/site-packages/vmidbg/__main__.py", line 21, in <module>
    from vmidbg.libvmistub import LibVMIStub
  File "/usr/lib/python3.8/site-packages/vmidbg/libvmistub.py", line 11, in <module>
    from .rawdebugcontext import RawDebugContext
  File "/usr/lib/python3.8/site-packages/vmidbg/rawdebugcontext.py", line 5, in <module>
    from vmidbg.abstractdebugcontext import AbstractDebugContext
  File "/usr/lib/python3.8/site-packages/vmidbg/abstractdebugcontext.py", line 3, in <module>
    from vmidbg.breakpoint import BreakpointManager
  File "/usr/lib/python3.8/site-packages/vmidbg/breakpoint.py", line 6, in <module>
    from libvmi.event import EventResponse, IntEvent, SingleStepEvent, DebugEvent, RegEvent, RegAccess
  File "/usr/lib/python3.8/site-packages/libvmi/event.py", line 8, in <module>
    EVENTS_VERSION = lib.VMI_EVENTS_VERSION
AttributeError: cffi library '_libvmi' has no function, constant or global variable named 'VMI_EVENTS_VERSION'

libvmi and python bindings are from https://github.com/libvmi/python (git clone && build)
Thanks in advance!

Handle unknown DTB's in continue_until

For unclear reasons, it is possible to get a DTB that doesn't match an process descriptor in the handle_breakpoint
Add an exception handler and deal with it

Missing WinDBG support

Hi everyone. I'm trying to add WinDBG support on pyvmidbg and opening this issue so we can track progress. I'm new to VM introspection so any help is welcome.

debloat readme: remove 30 MB demo video

on my metered connection, that video costs 4 cents to download ..
please stop robbing my cents, and only show the video on demand

maybe use mp4 format, since gif is not seekable

GDB breakin: target is not responding to interrupt-requests

the breakin \x03 is working with radare2 debugger, but not with GDB.

(gdb) cont
Continuing.
^C




^CThe target is not responding to interrupt requests.
Stop debugging it? (y or n) Quit
(gdb) 

Inaccurate readings randomize_layout [Linux Kernel]

The current implementation for the Linux kernel debugging lacks support for kernel 4.13+ because of the randomize_layout security feature which randomizes the location of struct members during the kernel compilation process, thus the offset of each element may vary resulting in inaccurate readings.

More information here about the feature
https://lwn.net/Articles/722293/

note the randomized_struct_fields_start used in

Kernel 4.13+

struct task_struct 
{
#ifdef CONFIG_THREAD_INFO_IN_TASK
	struct thread_info		thread_info;
#endif
	volatile long			state;
	randomized_struct_fields_start
	void				*stack;
...

which was not yet implemented in previous kernel versions

Kernel 4.12-

struct task_struct {
#ifdef CONFIG_THREAD_INFO_IN_TASK
	struct thread_info		thread_info;
#endif
	volatile long			state;
	void				*stack;

gdb.io.open: Cannot connect to host

Hi @Wenzel
I tried to connect to the gdbserver but it doesn't work.
even my firewalls are off and still it's not connecting.

aliadmin@alixen:~$ sudo xl list
Name                                        ID   Mem VCPUs	State	Time(s)
Domain-0                                     0  4095     4     r-----    3250.4
windows10                                  220  3000     1     -b----     296.4

I don't know if it makes any difference but I'm connecting to my quest vm with vncviewer:

aliadmin@alixen:~/iso$ gvncviewer 192.168.1.9::5900
Connected to server
Remote desktop size changed to 1024x768
Connection initialized

note: instead of gvncviewer 192.168.1.9::5900 I can use gvncviewer localhost too
At last this is my vm config, I changed the number of vcpu and max cpus to 1 as it was in the limitation section:

arch = 'x86_64'
name = "windows10"
maxmem = 3000
memory = 3000
vcpus = 1
maxcpus = 1
builder = "hvm"
boot = "cd"
hap = 1
acpi = 1
on_poweroff = "destroy"
on_reboot = "destroy"
on_crash = "destroy"
vnc=1
vnclisten="0.0.0.0"
usb = 1
usbdevice = "tablet"
shadow_memory = 16
audio=1
soundhw='hda'
vif = [ 'type=ioemu,model=e1000,bridge=xenbr0,mac=00:06:5B:BA:7C:02' ]
disk = [ 'phy:/dev/vg/windows10,hda,w', 'file:/home/aliadmin/iso/Windows10.iso,hdc:cdrom,r' ]

And this is the Error:

Slack registration link

Hey,

I saw you added a link to Slack. It probably only makes sense if other people can join it, so can you add some auto-invite link for it?

I found this on google: https://publicslack.com/ but I have no idea if it is legit.

Thanks!

ffi.error: the C compiler says 'VMI_EVENTS_VERSION' is equal to 6 (0x6), but the cdef disagrees

hi @Wenzel
when i run vmidbg i get this error:
ffi.error: the C compiler says 'VMI_EVENTS_VERSION' is equal to 6 (0x6), but the cdef disagrees
what should i do??

Windows: attach_new_process catches first process

attach_new_process should be filtering on the targeted name instead of catching the first process that hits the Thread startup routine

KVM support

Start vmidbg as:
"python3 -m vmidbg 5000 win10 --address 0.0.0.0 cmd -d"
then
Start radare2 as:
"r2 -d gdb://127.0.0.1:5000 -b 64"

vmidbg outputs:
"INFO:server:listening on 0.0.0.0:5000
DEBUG:server:ready for next client
INFO:server:new client ('127.0.0.1', 54186)
VMI_ERROR: The selected hypervisor has no events support!
..."

The reason is here: https://github.com/libvmi/libvmi/blob/master/libvmi/events.c#L96

So, is it possible to run vmidbg without Xen ?
My progress here: https://github.com/SamRSA/Tools/blob/master/Virtualization/kvm-qemu.sh#L278