tandasat/hyperplatform

Intel VT-x based hypervisor aiming to provide a thin VM-exit filtering platform on Windows.

HyperPlatform

Introduction

HyperPlatform is an Intel VT-x based hypervisor (a.k.a. virtual machine monitor) aiming to provide a thin platform for research on Windows. HyperPlatform is capable of monitoring a wide range of events, including but not limited to, access to virtual/physical memory and system registers, occurrences of interrupts and execution of certain instructions.

Researchers are free to selectively enable and/or disable any of those event monitoring and implement their own logic on the top of HyperPlatform. Some potential applications are:

• Analyzing kernel mode rootkit
• Implementing virtual-machine-based intrusion prevention system (VIPS)
• Reverse-engineering the Windows kernel

A simplified implementation of those ideas are available:

• MemoryMon detecting execution of kernel memory for rootkit analysis
• https://github.com/tandasat/MemoryMon
• EopMon spotting a successful elevation of privilege (EoP) exploit
• https://github.com/tandasat/EopMon
• DdiMon monitoring and controlling kernel API calls with stealth hook using EPT
• https://github.com/tandasat/DdiMon
• GuardMon observing some of PatchGuard activities
• https://github.com/tandasat/GuardMon

Advantages

HyperPlatform is designed to be easy to read and extend by researchers, especially those who are familiar with Windows. For instance:

• HyperPlatform runs on Windows 7, 8.1 and 10 in both 32 and 64 bit architectures without any special configuration (except for enabling Intel-VT technology).
• HyperPlatform compiles in Visual Studio and can be debugged though Windbg just like a regular software driver.
• Source code of HyperPlatform is written and formatted in existing styles (Google C++ Style Guide and clang-format), and well commented.
• HyperPlatform has no dependencies, supports use of STL and is released under a relaxed license.

For more details, see the HyperPlatform User Document and Programmer's Reference.

• https://tandasat.github.io/HyperPlatform/userdocument/
• https://tandasat.github.io/HyperPlatform/doxygen/

Build

To build HyperPlatform for x64 Windows 10 and later, the following are required.

• Visual Studio Community 2022
• Windows Software Development Kit (SDK) for Windows 10 (10.0.22621 or later)
• Windows Driver Kit (WDK) 10 (10.0.22621 or later)

To build HyperPlatform for x86 and Windows 7 and 8.1, the following are required.

• Visual Studio Community 2019
• Windows Software Development Kit (SDK) for Windows 10 (10.0.22000)
• Windows Driver Kit (WDK) 10 (10.0.22000)

Installation and Uninstallation

Clone full source code from Github with a below command and compile it on Visual Studio.

$ git clone --recursive https://github.com/tandasat/HyperPlatform.git

On the x64 platform, you have to enable test signing to install the driver. To do that, open the command prompt with the administrator privilege and type the following command, and then restart the system to activate the change:

>bcdedit /set testsigning on

To install and uninstall the driver, use the 'sc' command. For installation:

>sc create HyperPlatform type= kernel binPath= C:\Users\user\Desktop\HyperPlatform.sys
>sc start HyperPlatform

Note that the system must support the Intel VT-x and EPT technology to successfully install the driver. On Windows 10 RS4+ systems, this technology can automatically be disabled by the Windows kernel which results in the following error.

>sc start HyperPlatform
[SC] StartService FAILED 3224698910:

A hypervisor feature is not available to the user.

This is due to Windows Defender Credential Guard being enabled by default. To disable Windows Defender Credential Guard and enable the virtualization technology for HyperPlatform, follow this instruction.

• https://docs.microsoft.com/en-us/windows/security/identity-protection/credential-guard/credential-guard-manage

For uninstallation:

>sc stop HyperPlatform
>sc delete HyperPlatform
>bcdedit /deletevalue testsigning

To install the driver on a virtual machine on VMware Workstation, see an "Using VMware Workstation" section in the HyperPlatform User Document.

• https://tandasat.github.io/HyperPlatform/userdocument/

Output

All logs are printed out to DbgView and saved in C:\Windows\HyperPlatform.log.

Supported Platforms

• x86 and x64 Windows 7, 8.1 and 10
• The system must support the Intel VT-x and EPT technology

Related Project(s)

• SimpleVisor
• http://ionescu007.github.io/SimpleVisor/

SimpleVisor is a very (very) simple and readable Windows-specific hypervisor. I recommend taking a look at the project to learn VT-x if you are new to hypervisor development. It should give you a clearer view of how a hypervisor is initialized and executed.

• hvpp

• https://github.com/wbenny/hvpp hvpp is a lightweight Intel x64/VT-x hypervisor written in C++. This is about the same size as HyperPlatform in LOC yet written in a more polished matter with focus on x64, making the entire code base more readable. This project also addresses some issues remain unresolved in HyperPlatform and comes with educational comments and demonstration code to learn VT-x in more depth. Unless you are allergic to C++ or looking for x86 support, I strongly encourage you to study this project too.

• ksm

• https://github.com/asamy/ksm

ksm is lightweight-ish x64 hypervisor written in C for Windows for Intel processors. It demonstrates some advanced VT-x features like #VE and VMFUNC where HyperPlatform does not include.

• Bareflank Hypervisor
• http://bareflank.github.io/hypervisor/

Bareflank Hypervisor is an actively developed open source hypervisor. It comes with rich documents, tests, and comments, supports multiple platforms. The size of code is larger than that of HyperPlatform, but you will find it interesting if you are looking for more comprehensive yet still lightweight-ish hypervisors.

License

This software is released under the MIT License, see LICENSE.

hyperplatform's Issues

Tips: Using STL in the HyperPlatform User Document is not accurate

Description

The Tips: Using STL section explains "Define _HAS_EXCEPTIONS as 0 and include any STL headers after that." but at the current master df00162, it is expected to include kernel_stl.h instead of it since master is not updated to take off the header yet.

Update master or the document to reflect the latest status.

Expected behavior

The document reflect the latest status of master and explains accurate instructions to use STL.

Actual behavior

The instructions in the document is inconsistent with the current code base (master)

Steps to reproduce the problem

N/A

Specifications

• Commit (eg, 2099af7): df00162

• OS version (ie, outout of the "ver" command): N/A

• Architecture: N/A

• Hardware (eg, Physical, VMware 11, Bochs): N/A

• Details:
  Link: http://tandasat.github.io/HyperPlatform/userdocument/#__RefHeading___Toc906_1354011707

Usage of MmAllocateContiguousMemory results in RWX Stack

Hi,

The function MmAllocateContiguousMemory always allocates RWX memory. In hyperplatform, this is used to allocate the stack of the VMM. This makes the VMM stack executable, which means that bugs in the VMM are hyper-exploitable.

In order to fix this, you can use MmAllocateContiguousNodeMemory and pass in PAGE_READWRITE to avoid executable rights. Note that this function only exists on Windows 8 and later.

On Windows 7, there is no good solution that I am aware of, because even NP memory is executable.

Hide elapsed time of VM-exit from a guest

The VMM does not virtualize the time stamp counter (TSC) or any other timers. This can lead timer interrupt (IDT d1: hal!HalpTimerClockInterrupt, on my test system) immediately after VM-enter when the VMM runs an VM-exit handler overly long time. While this is not an issue by itself, it could cause an infinite between a VM-exit and a timer interrupt handler under certain scenarios. A situation I have seen is as followings:

1. VM-exit occurs at address X
2. VM-exit runs overly long time to address an cause of the VM-exit
3. VM-entry
4. Timer interrupt occurs immediately (before an instruction at X is executed) changing state of the system such that VM-exit will occur at X
5. The timer interrupt ends and X is fetched for execution
6. Return to 1

A quick fix would be streamlining the long run VM-exit handler, but the VMM should not limit what a developer can do on VM-exit in that manner. A more correct way to address this issue is hiding an elapsed time of VM-exit handler from a guest and protect guest context from triggering timer interrupt.

vhu.enable is probably unnecessary

Summary

It was pointed out that addition of

vhu.enable = "TRUE"

to the VMX file for VMware did not seem to be necessary to run HyperPlatform proper on VMware based on some manual tests and the fact that a string "vhu.enable" is not found in any VMware binary files while others are.

Requirement of vhu.enable was from this article written by a Microsoft partner, but neither Microsoft nor VMware explained such option was required formally. For example, search with "vhu.enable" on VMware's Knowledge Base page does give us no result.

Run some tests to verify if vhu.enable is necessary or not, and if not, update the HyperPlatform User Document accordingly.
http://tandasat.github.io/HyperPlatform/userdocument/

No vhu.enable found

There may be confusion with vhv.enable, which corresponds to the "Virtualize Intel VT-x/EPT or AMD-V/RVI" option.

> strings -n 5 -s *.exe | findstr /i /c:"hypervisor.cpuid.v0" 2>NUL                                     
C:\Program Files (x86)\VMware\VMware Workstation\vmware-hostd.exe: hypervisor.cpuid.v0                  
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx-debug.exe: hypervisor.cpuid.v0          
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx-stats.exe: hypervisor.cpuid.v0          
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx.exe: hypervisor.cpuid.v0                

> strings -n 5 -s *.exe | findstr /i /c:"mce.enable" 2>NUL
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx-debug.exe: mce.enable
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx-stats.exe: mce.enable
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx.exe: mce.enable

> strings -n 5 -s *.exe | findstr /i /c:"vhu.enable" 2>NUL

> strings -n 5 -s *.dll | findstr /i /c:"vhu.enable" 2>NUL

> strings -n 5 -s *.exe | findstr /i /c:"vhv.enable" 2>NUL
C:\Program Files (x86)\VMware\VMware Workstation\vmware-hostd.exe: vhv.enable
C:\Program Files (x86)\VMware\VMware Workstation\OVFTool\ovftool.exe: vhv.enable
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx-debug.exe: vhv.enable
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx-debug.exe: @&!*@*@(msg.cpuid.vhv.enablemismatch)Configuration mismatch: The virtual machine cannot be restored because the snapshot was taken with VHV enabled. To restore, set vhv.enable to true.
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx-debug.exe: (guestos-get-bool "vhv.enable" (get-guestosid))
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx-debug.exe:         ((string-ci=? config "vhv.enable")
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx-debug.exe:             ((and (guestos-get-bool "vhv.enable" guestosid)
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx-debug.exe:              (guestos-get-bool "vhv.enable" guestosid)
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx-debug.exe:        (guestos-get-bool "vhv.enable" guestosid)))
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx-stats.exe: vhv.enable
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx-stats.exe: @&!*@*@(msg.cpuid.vhv.enablemismatch)Configuration mismatch: The virtual machine cannot be restored because the snapshot was taken with VHV enabled. To restore, set vhv.enable to true.
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx-stats.exe:         ((string-ci=? config "vhv.enable")
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx-stats.exe:             ((and (guestos-get-bool "vhv.enable" guestosid)
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx-stats.exe:              (guestos-get-bool "vhv.enable" guestosid)
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx-stats.exe:        (guestos-get-bool "vhv.enable" guestosid)))
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx.exe: vhv.enable
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx.exe: @&!*@*@(msg.cpuid.vhv.enablemismatch)Configuration mismatch: The virtual machine cannot be restored because the snapshot was taken with VHV enabled. To restore, set vhv.enable to true.
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx.exe:         ((string-ci=? config "vhv.enable")
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx.exe:             ((and (guestos-get-bool "vhv.enable" guestosid)
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx.exe:              (guestos-get-bool "vhv.enable" guestosid)
C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx.exe:        (guestos-get-bool "vhv.enable" guestosid)))

@herosi_t: thanks for reporting the issue and providing all those information.

TRAP_CAUSE_UNKNOWN bugcheck 100% occurs on latest build in win7 sp1 32-bit

Steps to reproduce the problem

1. Compile the project.
2. Copy HyperPlatform.sys to target machine.
3. Load HyperPlatform.sys.
4. Bugcheck occurs immediately.

Specifications

• Commit 643b81c

• OS version 7601

• Architecture: Only x86

• Hardware Physical & VMware 12

• Details:

It seems that exceptions was generated by "movaps" instruction which is newly added to x86.asm. Dmp&pdb&log was sent to your gmail.

Thanks!

Process Crash when use a debugger "single step" over cpuid instruction

Description

when i use a debugger(e.g. OllyDbg) to debug a proces, if i
first:set a hardware breakpoint in this process
second:use single step to execute over instruction "cpuid"
the process debuged will crash only execute over cpuid

Steps to reproduce the problem

1.use ollydbg attach/debug a process with cpuid instruction
2.stop before cpuid instruction
3.set a hardware breakpoint in this process
4.start use single step(F8 in OllyDBG) before cpuid
5."single step" over "cpuid"

other

it's will crash every time when i try this
but it will not crash when "single step" instruction like rdtsc

Is vmxoff path really safe/correct?

Hi,

I experimented with turning VMX off by using DPC routine targeted to each processor, with synchronization barrier. If DPCs were delivered as part of SYSTEM process (such as from idle thread, or from system thread), everything worked fine. However, if a process was currently running, and the DPC interrupted the process, many times the process crashes with invalid memory access, or sometimes a kernel driver itself crashes (such as win32kfull running on the context of dwm.exe), when accessing session memory.

These problems go away if doing VMX off the way you do it, by sticking with a single thread that is already running under SYSTEM process. However, even if you have a comment that suggests you saw some strangeness.

So therefore, I have two questions:

1. Are you sure that all the register state is correctly saved and restored? You only save integer registers, but not xmm0,1, 2, 3, 4, 5 on x64, which are volatile registers (and neither the other xmm registers). Why do you only save integer registers during vmm_exit and vmxoff restore? Shouldn't FPU registers be saved too?

Must vmx_launch and vmx_off be done in the same address space? In other words, will vmx_off restore the CR3 that was saved during vmx_launch? Because if so, I can understand the danger of capturing CR3 in system process (session -1), but then doing vmx_off in dwm.exe (session 0). The totally different CR3 will certainly cause a crash. Does this mean that it's only safe to do on/off from within SYSTEM process?

Thank you.

Incorrect emulation of SGDT and SIDT for WOW64 processes

Description

On 64bit systems, SGDT and SIDT can be executed from a process running in non 64bit mode, namely, wow64 processes. In such a case, those instructions should only write 6 bytes or GDTR and IDTR, versus full 10 bytes. HyperPlatform does not emulate this behaviour correctly and always writes 10 bytes.

Expected behavior

SGDT and SIDT returns only 6 bytes when executed from a WOW64 process.

Actual behavior

SGDT and SIDT returns 10 bytes when executed from a WOW64 process.

Steps to reproduce the problem

The test program code attached. Compile the code as 32bit process and run. the last four bytes of buffers for SIDT and SGDT should remain 0xCE (uninitialized), but not when HyperPlatform is installed.
sgdt_sidt_test.zip

Specifications

• Commit (eg, 2099af7): 591822f

• OS version (ie, outout of the "ver" command, or N/A): Microsoft Windows [Version 10.0.17134.285]

• Architecture: x64

• Hardware (eg, Physical, VMware 11, Bochs, or N/A): Physical, VMware 14

• Details: Nothing

Triple Fault BSOD randomly

Description

Latest HyperPlatform crash the system when running programs randomly.

Expected behavior

the system should not crash.

Actual behavior

the system crashed.

Steps to reproduce the problem

I compiled the lastest HyperPlatform, and loaded it in vmware 12.0 Win7 x64 7600, then run some programs. then the system went BSOD.
sometimes it even crashed when I did nothing(except loading the driver).

Specifications

• Commit :
• OS version Windows 7 7600
• Architecture: x64
• Hardware VMware 12
• Anything else:

`BUGCHECK_STR: 0xE2

PROCESS_NAME: HipsDaemon.exe

CURRENT_IRQL: 2

ANALYSIS_SESSION_HOST: HZQST-PC

ANALYSIS_SESSION_TIME: 09-19-2016 20:48:22.0815

ANALYSIS_VERSION: 10.0.10586.567 amd64fre

BAD_STACK_POINTER: fffff88006ea6e98

LAST_CONTROL_TRANSFER: from fffff8800397724f to fffff80003e88f00

STACK_TEXT:
fffff88006ea6e98 fffff8800397724f : 00000000000000e2 0000000000000002 fffff88006ea6f10 000000006e001010 : nt!KeBugCheckEx
fffff88006ea6ea0 00000000000000e2 : 0000000000000002 fffff88006ea6f10 000000006e001010 0000000000000000 : HyperPlatform+0x424f
fffff88006ea6ea8 0000000000000002 : fffff88006ea6f10 000000006e001010 0000000000000000 0000000000000000 : 0xe2
fffff88006ea6eb0 fffff88006ea6f10 : 000000006e001010 0000000000000000 0000000000000000 00000000c0000100 : 0x2
fffff88006ea6eb8 000000006e001010 : 0000000000000000 0000000000000000 00000000c0000100 0000000000000282 : 0xfffff88006ea6f10 fffff88006ea6ec0 0000000000000000 : 0000000000000000 00000000c0000100 0000000000000282 00000000fffffff7 : 0x6e001010

what HyperPlatform+0x424f is in IDA
if ( v6 == 2 ) { VmmpDumpGuestSelectors(); if ( !(_BYTE)KdDebuggerNotPresent ) __debugbreak(); KeBugCheckEx(0xE2u, 2ui64, v3, *(_QWORD *)(v3 + 16), 0i64); }

what HyperPlatform+0x424f is in vmm.c

case VmxExitReason::kTripleFault: VmmpHandleTripleFault(guest_context); break;

// Triple fault VM-exit. Fatal error. _Use_decl_annotations_ static void VmmpHandleTripleFault( GuestContext *guest_context) { VmmpDumpGuestSelectors(); HYPERPLATFORM_COMMON_BUG_CHECK(HyperPlatformBugCheck::kTripleFaultVmExit, reinterpret_cast<ULONG_PTR>(guest_context), guest_context->ip, 0); }

Provide an issue template for smoother bug triage

Let us have one so that issues reports can be easily written and smoothly handled.

https://help.github.com/articles/creating-an-issue-template-for-your-repository/

WRMSR does not vm exit?

HI~

  I added some debugging information, as follows.

`

driver.cpp

void dump(void)
{
ULONG64 msr_test = __readmsr((ULONG)Msr::kIa32FeatureControl);

HYPERPLATFORM_LOG_DEBUG("msr_test  = 0x%p", msr_test );
__writemsr((ULONG)Msr::kIa32FeatureControl, msr_test );

}
VOID SystemThread(PVOID lpParam)
{
UNREFERENCED_PARAMETER(lpParam);
HYPERPLATFORM_LOG_DEBUG("enter SystemThread");

while (1)
{
	dump();

	HYPERPLATFORM_LOG_DEBUG("\n\n");
	UtilSleep(3000);
}
HYPERPLATFORM_LOG_DEBUG("SystemThread over");

PsTerminateSystemThread(STATUS_SUCCESS);

}

void Test()
{
NTSTATUS status;
HANDLE hThread = NULL;
CLIENT_ID cid;

status = PsCreateSystemThread(&hThread, 0, NULL, NULL, &cid, (PKSTART_ROUTINE)SystemThread, (PVOID)NULL);

if (NT_SUCCESS(status))
{
	ZwClose(hThread);
}

}

vmm.cpp (VmmpHandleMsrAccess)
...
const auto msr = static_cast(guest_context->gp_regs->cx);
if (read_access)
{
if (msr != (Msr)0x10 && msr != (Msr)0x19a)
{
HYPERPLATFORM_LOG_DEBUG_SAFE("------------------------------------------------");
HYPERPLATFORM_LOG_DEBUG_SAFE("read msr : 0x%08x", msr);
HYPERPLATFORM_LOG_DEBUG_SAFE("------------------------------------------------");
}
}
else
{
HYPERPLATFORM_COMMON_DBG_BREAK();

	LARGE_INTEGER tmp_value = {};

	tmp_value.LowPart = static_cast<ULONG>(guest_context->gp_regs->ax);
	tmp_value.HighPart = static_cast<ULONG>(guest_context->gp_regs->dx);

	HYPERPLATFORM_LOG_DEBUG_SAFE("====================================================");
	HYPERPLATFORM_LOG_DEBUG_SAFE("write msr : 0x%08x 0x%p", msr, tmp_value.QuadPart);
	HYPERPLATFORM_LOG_DEBUG_SAFE("====================================================");
}

bool transfer_to_vmcs = false;
...

`

The WinDbg output information is like this:

14:33:54.541 DBG #2 4 1472 System ------------------------------------------------
14:33:54.541 DBG #2 4 1472 System read msr : 0x0000003a
14:33:54.541 DBG #2 4 1472 System ------------------------------------------------
14:33:54.541 DBG #2 4 1472 System test msr = 0x0000000000000005
14:33:54.603 DBG #2 4 1472 System

14:33:57.661 DBG #2 4 1472 System ------------------------------------------------
14:33:57.661 DBG #2 4 1472 System read msr : 0x0000003a
14:33:57.661 DBG #2 4 1472 System ------------------------------------------------
14:33:57.661 DBG #2 4 1472 System test msr = 0x0000000000000005
14:33:57.739 DBG #2 4 1472 System

WinDbg has no wrmsr information, and the breakpoint is not broken.

Is there a problem?

EoPMon build failed due to commit b41f7e95065fbc69c29f4578ca849d30215f6ae3

eopmon still needs HyperPlatform/kernel_stl.h which is removed in above commit.

A link error against __stdio_common_vswprintf() occurs with RtlStringCchPrintfW()

Using RtlStringCchPrintfW() leads a link error with __stdio_common_vswprintf() when a project setting is changed to enable STL according with the User's Document. As RtlStringCchPrintfA() is already supported, -W() should also be able to used.

Contents of XMM registers should be preserved during VM-exit handling

As it is reported in #3, HyperPlatform does not preserve contents of XMM registers when VM-exit occurs. This can corrupt guest's context if a VM-exit handler modifies XMM registers and a guest uses those modified values, and result in unexpected failure in a guest (most likely system crash).

In order to avoid this issue, the VMM should probably save contents of XMM registers on VM-exit, and restore them before VM-enter so that modification of those registers by the VMM does not leak to guest's state.

Support per processor EPT tables over shared tables

HyperPlatform creates a single set of EPT tables for all processors so that a guest PA is always translated into the same host PA regardless of which processor is being used. While it allows EPT code be simple, it make nearly impossible to modify EPT entries dynamically under a high concurrent situation because more than one of processors can change the same EPT entry simultaneously. This is essentially the race condition issue, which is challenging to debug and makes code unreliable, and EPT manipulation based cannot support multiple processors due to this issue.

Consider creating a set of EPT tables for each processor.

Solving the stack "out of bounds" issue (allowing DriverVerifier to run)

Hi,

Just to let you know, the correct way to "fix" this issue is to edit the PFNs for the kernel stack, and to set them as kernel stack pages (one of the flags in e2.u1 I believe) as well as write the KTHREAD as the u1.KernelStackOwner, and then OR'ing with 1. You can see this by doing a !pte on a kernel stack page (a real one), and then !pfn, and analyzing the MMPFN structure.

This requires pretty low-level hacking of the PFN database, but in case you were curious, that's why Verifier/Kernel don't like it when you do API calls from VMM context. It's also technically possible to get 109 bug check in this situation.

You may want to investigate KeExpandKernelStackAndCalloutEx as a possible way to avoid this -- it should create a 'legitimate' stack for the VMM.

Investigate a way to retrieve device physical memory ranges

Attached code in How to hide a hook: A hypervisor for rootkits uses the below registry data to retrieve physical memory ranges.

• \Registry\machine\HARDWARE\RESOURCEMAP\System Resources\Physical Memory

It seems that it contains physical memory ranges for devices, which is not listed by MmGetPhysicalMemoryRanges(). If we can obtain device memory ranges, we may be able to delete EPT entries pre-allocation code because we could allocate all of them at initialization stage.

Investigate the contents of the registry data and update code if possible.

NonPagedPoolNx is used despite support of Windows 7

Use of NonPagedPoolNx is undefined on Windows 7. It should simply be replaced with NonPagedPool since ExInitializeDriverRuntime() is used in place.

Expose an interface to get an EPT page table entry

A function to get an EPT page table entry from a physical address, namely EptpGetEptPtEntry(), is almost always required by other modules to develop a tool manipulate EPT tables, for example, MemoryMon that uses EptGetEptPtEntry().

Make the function non-static and accessible from other modules by default.

No support for processor groups

Hi,

Hyperplatform is currently using Vista/deprecated CPU management/query APIs such as KeGetCurrentProcessorNumber, KeSetSystemAffinityEx, etc. These APIs do not function correctly when "processor groups" are enabled.

You should test hyperplatform with the information available here: https://msdn.microsoft.com/en-us/library/windows/hardware/ff542298(v=vs.85).aspx. This lets you test processor groups with as little as 2 processors (4 is better).

A few examples, you should use KeGetCurrentProcessorNumberEx(NULL) instead of the non-Ex version. You should use KeSetSystemGroupAffinityThread as you hop around processors, and pass in the correct group affinity for each group being iterated.

I realize you might say "nobody cares about this, no-one will run this on NUMA systems with > 64 processors"...

Is there any way to install shadow hook for function in win32k.sys?

I tried KeStackAttachProcess to csrss.exe before installing shadow hook to win32k!NtUserSetWindowsHookEx, and KeUnstackAttachProcess to switch the context back to System process after installing hook.
but my detour function never get called.

Windbg said that win32k!NtUserSetWindowsHookEx is in session address space that I can't set software- breakpoint at it or it may cause bugcheck. I guess this is the reason why my shadow hook won't work since the shadow hook need int3 breakpoint to work.

Msr Exit doesn't work?

I compiled the default project and tested it on VMware win7 (x64) Sp1.

VMware has been configured according to the instructions.

Hypervisor.cpuid.v0 = "FALSE"

Mce.enable = "TRUE"

The following is the output information of the WinDbg. There is no trigger information for MSR exit.

I also added print information in VmmpHandleMsrAccess, and I didn't have any output.

19:58:46.917 INF #3 4 52 System Initialized successfully.
19:58:46.949 INF #3 4 52 System The VMM has been installed.
20:04:32.226 INF #1 4 68 System Uninstalling VMM.
20:04:32.242 INF #0 4 68 System Terminating VMX for the processor 0.
20:04:32.273 DBG #0 4 68 System Context at FFFFF880047CBA20 FFFFFA8003C1F070
20:04:32.273 DBG #0 4 68 System Used pre-allocated entries = 3 / 50
20:04:32.304 INF #1 4 68 System Terminating VMX for the processor 1.
20:04:32.320 DBG #1 4 68 System Context at FFFFF880047CBA20 FFFFFA80039CD7B0
20:04:32.320 DBG #1 4 68 System Used pre-allocated entries = 2 / 50
20:04:32.366 INF #2 4 68 System Terminating VMX for the processor 2.
20:04:32.398 DBG #2 4 68 System Context at FFFFF880047CBA20 FFFFFA8003AAB330
20:04:32.398 DBG #2 4 68 System Used pre-allocated entries = 2 / 50
20:04:32.429 INF #3 4 68 System Terminating VMX for the processor 3.
20:04:32.444 DBG #3 4 68 System Context at FFFFF880047CBA20 FFFFFA800196F170
20:04:32.444 DBG #3 4 68 System Used pre-allocated entries = 2 / 50
20:04:32.507 DBG #3 4 68 System Freeing shared data...
20:04:32.522 INF #3 4 68 System The VMM has been uninstalled.
20:04:32.538 INF #1 4 68 System FunctionName(Line) ,Execution Count ,Elapsed Time
20:04:32.554 INF #1 4 68 System VmmpHandleCpuid(438) , 80, 516,
20:04:32.569 INF #1 4 68 System VmmpHandleVmExit(228) , 50554, 749860,
20:04:32.585 INF #1 4 68 System EptHandleEptViolation(736) , 34, 2168,
20:04:32.616 INF #1 4 68 System VmmpHandleEptViolation(1296) , 34, 2365,
20:04:32.647 INF #3 4 68 System VmmpHandleCrAccess(1123) , 46825, 255310,
20:04:32.663 INF #1 4 68 System VmmpHandleCrAccess(1088) , 46727, 408260,
20:04:32.678 INF #1 4 68 System VmmpHandleGdtrOrIdtrAccess(648) , 666, 50877,
20:04:32.756 INF #1 4 68 System VmmpHandleLdtrOrTrAccess(738) , 661, 12659,
20:04:32.772 INF #1 4 68 System VmmpHandleDrAccess(834) , 2309, 15240,
20:04:32.803 DBG #1 4 68 System Finalizing... (Max log usage = 445/65536 bytes)
20:04:32.819 INF #1 4 68 System Bye!

Is it wrong with me? Does it need additional configuration to trigger the MSR?

About Shadow Hook Question

An export name to hook is ok,(SSDT)
How hook noexport api? like eg: NtGdiBitBlt (shadow hook)

CONSTANT_STRING(L"NTGDIBITBLT"),
DdimonpHandleNtGdiBitblt, nullptr, not work

thank you..

UtilIsAccessibleAddress() causes a bug check on Windows 10 14316+

Calling UtilIsAccessibleAddress() can cause a bug check with code DRIVER_IRQL_NOT_LESS_OR_EQUAL or PAGE_FAULT_IN_NONPAGED_AREA due to hard-coded kUtilpP*eBase (tandasat/GuardMon#2). As it explained here, those page table base addresses are now determined at the run-time, resulting in UtilIsAccessibleAddress() to access somewhere else and a bug check.

While this function is not used by HyperPlatform and Windows 10 14316+ is still on the preview stage, this issue must be solved in order to support feature Windows builds.

A potential fix could be retrieving right addresses by disassembling MiGetPhysicalAddress() on Windows 10 (and when a build version is equal or greater than 14316).

1: kd> u nt!MiGetPhysicalAddress l50
nt!MiGetPhysicalAddress:
...
fffff802`fdd71897 48b900409f3e7dfaffff mov rcx,0FFFFFA7D3E9F4000h  ; PXE_BASE
...
fffff802`fdd718b2 48b90000803e7dfaffff mov rcx,0FFFFFA7D3E800000h  ; PPE_BASe
...
fffff802`fdd718d0 48b9000000007dfaffff mov rcx,0FFFFFA7D00000000h   ; PDE_BASE

; Note:
;    0xFFFFFA00`00000000 ; PTE_BASE 

Activate VM-Exit I/O port 0x5658 crash vmtoolsd.exe

Description

I test I/O interception for vmware backdoor instruction in 0x5658, however when I activate VM-Exit I/O port 0x5658 crash vmtoolsd.exe on VMWare workstation 12

Expected behavior

no crash and log instruction IN 0x5658

Actual behavior

always crash vmtools when load driver

Steps to reproduce the problem

modify code in VmpBuildIoBitmaps, add one line RtlSetBits(&bitmap_a_header, 0x5658, 0x1);
build and load HyperPlatform dirver
crash...

Specifications

• OS version:
  host Windows 7 x64
  guest Windows 7 x64
• Hardware:
  VMware 12
• Anything else:

some log found vmware-vmsvc.log

[Sep 03 15:23:11.720] [ message] [vmsvc] backtrace[00] frame 0x01a7f3f8 IP 0x01a7f440 params 0x4f4c4354 0x80000000 0x7fef8fdaf42 0x338 [no module data] ???
[Sep 03 15:23:11.720] [ message] [vmsvc] backtrace[01] frame 0x01a7f400 IP 0x0096c2f0 params 0x80000000 0x7fef8fdaf42 0x338 0 [no module data] ???
[Sep 03 15:23:11.720] [ message] [vmsvc] backtrace[02] frame 0x01a7f408 IP 0x4f4c4354 params 0x7fef8fdaf42 0x338 0 0x1a7f628 [no module data] ???
[Sep 03 15:23:11.720] [ message] [vmsvc] backtrace[03] frame 0x01a7f410 IP 0x80000000 params 0x338 0 0x1a7f628 0x1 [no module data] ???
[Sep 03 15:23:11.736] [ message] [vmsvc] backtrace[04] frame 0x01a7f418 IP 0x7fef8fdaf42 params 0 0x1a7f628 0x1 0x564d5868 [C:\Program Files\VMware\VMware Tools\vmtools.dll base 0x000007fef8f80000 0x0001:0x0000000000059f42] GuestApp_GetConfPath

when decompile vmtools.dll crash in address 0x7fef8fdaf42

.text:000007FEF8FDAF38                 mov     [r11-38h], r14w
.text:000007FEF8FDAF3D                 call    Backdoor
.text:000007FEF8FDAF42                 test    byte ptr [rsp+68h+var_36], 1

.text:000007FEF8FE1A00 Backdoor        proc near               ; CODE XREF: sub_7FEF8FDAEF0+4D�p
.text:000007FEF8FE1A00                                         ; sub_7FEF8FDAEF0+7D�p ...
.text:000007FEF8FE1A00                 mov     eax, 5658h
.text:000007FEF8FE1A05                 mov     dword ptr [rcx], 564D5868h
.text:000007FEF8FE1A0B                 mov     [rcx+18h], ax
.text:000007FEF8FE1A0F                 jmp     sub_7FEF8FE1A60
.text:000007FEF8FE1A0F Backdoor        endp

.text:000007FEF8FE1A60 sub_7FEF8FE1A60 proc near               ; CODE XREF: Backdoor+F�j
.text:000007FEF8FE1A60                                         ; DATA XREF: .pdata:000007FEF903F868�o
.text:000007FEF8FE1A60
.text:000007FEF8FE1A60 var_20          = qword ptr -20h
.text:000007FEF8FE1A60
.text:000007FEF8FE1A60                 push    rbx
.text:000007FEF8FE1A62                 push    rsi
.text:000007FEF8FE1A63                 push    rdi
.text:000007FEF8FE1A64                 mov     rax, rcx
.text:000007FEF8FE1A67                 push    rax
.text:000007FEF8FE1A68                 mov     rdi, [rax+28h]
.text:000007FEF8FE1A6C                 mov     rsi, [rax+20h]
.text:000007FEF8FE1A70                 mov     rdx, [rax+18h]
.text:000007FEF8FE1A74                 mov     rcx, [rax+10h]
.text:000007FEF8FE1A78                 mov     rbx, [rax+8]
.text:000007FEF8FE1A7C                 mov     rax, [rax]
.text:000007FEF8FE1A7F                 in      eax, dx
.text:000007FEF8FE1A80                 xchg    rax, [rsp+20h+var_20]
.text:000007FEF8FE1A84                 mov     [rax+28h], rdi
.text:000007FEF8FE1A88                 mov     [rax+20h], rsi
.text:000007FEF8FE1A8C                 mov     [rax+18h], rdx
.text:000007FEF8FE1A90                 mov     [rax+10h], rcx
.text:000007FEF8FE1A94                 mov     [rax+8], rbx
.text:000007FEF8FE1A98                 pop     qword ptr [rax]
.text:000007FEF8FE1A9A                 pop     rdi
.text:000007FEF8FE1A9B                 pop     rsi
.text:000007FEF8FE1A9C                 pop     rbx
.text:000007FEF8FE1A9D                 retn
.text:000007FEF8FE1A9D sub_7FEF8FE1A60 endp

PatchGuard triggers BSOD when manual mapping Hypervisor

Description

When you manual map an empty hypervisor (just setting up the VMM and pinging the VM once) PatchGuard seems to trigger CRITICAL_STRUCTURE_CORRUPTION with Kernel notification callout modification (arg4: 0x18). This is without modifying anything in the kernel, just an empty hypervisor. It seems very weird that PG would trigger just because a system thread is doing virtualization without being linked in the kernel module list.

It doesn't seem to be caused by initializing the CPU's, since if I remove the VmInitialization(); call from the DriverEntry, PG is still triggered.

I have tested the same hypervisor by signing it and loading it normally. There hasn't been a BSOD in over 4 hours. Longest I lasted with the manual mapped loading is ~60 minutes.

Note: I have removed all SEH handlers and references to the driver object.

Any ideas on why this may happen or help me point into a direction to help resolve this issue?

Expected behavior

I would expect PatchGuard not to be triggered.

Actual behavior

Manual mapping it causes a CRITICAL_STRUCTURE_CORRUPTION, Kernel notification callout modification BSOD.

Steps to reproduce the problem

Setup an empty Hypervisor based on HyperPlatform and manual map it using Blackbones implementation (https://github.com/DarthTon/Blackbone).

Specifications

• Commit: 374db11

• OS version: Microsoft Windows [Version 10.0.14393]

• Architecture: x64

• Hardware: Physical

• Details:

*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

CRITICAL_STRUCTURE_CORRUPTION (109)
This bugcheck is generated when the kernel detects that critical kernel code or
data have been corrupted. There are generally three causes for a corruption:
1) A driver has inadvertently or deliberately modified critical kernel code
 or data. See http://www.microsoft.com/whdc/driver/kernel/64bitPatching.mspx
2) A developer attempted to set a normal kernel breakpoint using a kernel
 debugger that was not attached when the system was booted. Normal breakpoints,
 "bp", can only be set if the debugger is attached at boot time. Hardware
 breakpoints, "ba", can be set at any time.
3) A hardware corruption occurred, e.g. failing RAM holding kernel code or data.
Arguments:
Arg1: a39fde59804ae530, Reserved
Arg2: b3b6eadfd2ccc1c2, Reserved
Arg3: 0000000000000006, Failure type dependent information
Arg4: 0000000000000018, Type of corrupted region, can be
	0 : A generic data region
	1 : Modification of a function or .pdata
	2 : A processor IDT
	3 : A processor GDT
	4 : Type 1 process list corruption
	5 : Type 2 process list corruption
	6 : Debug routine modification
	7 : Critical MSR modification

Debugging Details:
------------------

ADDITIONAL_DEBUG_TEXT:  

Use '!findthebuild' command to search for the target build information.

If the build information is available, run '!findthebuild -s ; .reload' to set symbol path and load symbols.

MODULE_NAME: nt

FAULTING_MODULE: fffff8015cc94000 nt

DEBUG_FLR_IMAGE_TIMESTAMP:  595f2981

BUGCHECK_STR:  0x109

CUSTOMER_CRASH_COUNT:  1

DEFAULT_BUCKET_ID:  WIN8_DRIVER_FAULT

CURRENT_IRQL:  0

LAST_CONTROL_TRANSFER:  from 0000000000000000 to fffff8015cde3960

STACK_TEXT:  
ffff9f00`ec7e4fb8 00000000`00000000 : 00000000`00000109 a39fde59`804ae530 b3b6eadf`d2ccc1c2 00000000`00000006 : nt+0x14f960


STACK_COMMAND:  kb

FOLLOWUP_IP: 
nt+14f960
fffff801`5cde3960 48894c2408      mov     qword ptr [rsp+8],rcx

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  nt+14f960

FOLLOWUP_NAME:  MachineOwner

IMAGE_NAME:  ntoskrnl.exe

BUCKET_ID:  WRONG_SYMBOLS

AMD-V Support

AMD provides hardware based virtualization called AMD-V. Although it is similar to that of Intel (Intel VT-x) in both programming-wise and functionality-wise, supporting AMD-V practically requires a physical device. Also, given lower popularity of ADM processors, support of AMD-V is not planned currently.

Support x86 Windows 10 build 14316 and newer

HyperPlatform cannot be loaded onto x86 Windows 10 build 14316 and newer because PxE_BASE resolution code is not tested on those platforms where the base addresses are randomized in case of x64 architecture.

However, I found that both PDE_BASE and PTE_BASE were located at the same fixed values: 0xc0600000 and 0xc0000000, and just using existing code worked fine on 14393.0.x86fre.rs1_release.160715-1616.

DOUBLE_FAULT exception under win7 x86 with KB4088878

Description

Under Windows 7 x86 SP1 with KB4088878 (the 2018-03 security update: http://download.windowsupdate.com/c/msdownload/update/software/secu/2018/03/windows6.1-kb4088878-x86_7512ab54d6a6df9d7e3d511d84a387aaeaeef111.msu), everytime loading and running HyperPlatform driver, a DOUBLE_FAULT exception would be raised during handling cr3 access in vmexit.

Expected behavior

Executing normally.

Actual behavior

The DOUBLE_FAULT exception was triggered by instruction mov cr3,ecx in function UtilLoadPdptes.

Steps to reproduce the problem

1. Install the 2018-03 security update for Windows 7 x86 SP1: http://download.windowsupdate.com/c/msdownload/update/software/secu/2018/03/windows6.1-kb4088878-x86_7512ab54d6a6df9d7e3d511d84a387aaeaeef111.msu

2. Load and run the latest version of HyperPlatform (without any extra modification) in the Windows 7 environment.

3. The exception would be triggered immediately.

Specifications

• Commit: 97d6ccc (the latest commit so far)

• OS version: Windows 7 SP1 7601 with KB4088878

• Architecture: x86

• Hardware: VMware Workstation (any version)

• Details:

The DOUBLE_FAULT exception was triggered by instruction mov cr3,ecx in function UtilLoadPdptes:

kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

UNEXPECTED_KERNEL_MODE_TRAP (7f)

Arguments:
Arg1: 00000008, EXCEPTION_DOUBLE_FAULT
Arg2: 80b93c00
Arg3: 00000000
Arg4: 00000000

TSS:  00000028 -- (.tss 0x28)
eax=00185000 ebx=7793fce0 ecx=5f2fd4a0 edx=87d5cff0 esi=003993d8 edi=00000000
eip=987bc8ad esp=87d5cdd0 ebp=87d5ce08 iopl=0         nv up di ng nz na po nc
cs=0008  ss=0010  ds=0020  es=0020  fs=0030  gs=0000             efl=00010082

STACK_TEXT:  
87d5ce08 987bdb3f 5f2fd4a0 86fa1000 987bc580 HyperPlatform!UtilLoadPdptes+0x1d [d:\github\hyperplatform\hyperplatform\util.cpp @ 835]
87d5cef4 987bf205 87d5cf40 00000000 003993d8 HyperPlatform!VmmpHandleCrAccess+0x14f [d:\github\hyperplatform\hyperplatform\vmm.cpp @ 966]
87d5cf34 987bd185 87d5cf40 87d5cfd8 00000006 HyperPlatform!VmmpHandleVmExit+0x145 [d:\github\hyperplatform\hyperplatform\vmm.cpp @ 268]
87d5cf64 987ba27e 87d5cfd8 00000000 00000000 HyperPlatform!VmmVmExitHandler+0xa5 [d:\github\hyperplatform\hyperplatform\vmm.cpp @ 205]
87d5cf6c 00000000 00000000 00000000 00000000 HyperPlatform!AsmVmmEntryPoint+0x37 [D:\GitHub\HyperPlatform\HyperPlatform\Arch\x86\x86.asm @ 117]

kd> ub 987bc8ad 
HyperPlatform!UtilLoadPdptes+0x6 [d:\github\hyperplatform\hyperplatform\util.cpp @ 828]:
987bc896 a108207c98      mov     eax,dword ptr [HyperPlatform!__security_cookie (987c2008)]
987bc89b 33c5            xor     eax,ebp
987bc89d 8945fc          mov     dword ptr [ebp-4],eax
987bc8a0 56              push    esi
987bc8a1 0f20d8          mov     eax,cr3
987bc8a4 8945cc          mov     dword ptr [ebp-34h],eax
987bc8a7 8b4d08          mov     ecx,dword ptr [ebp+8]
987bc8aa 0f22d9          mov     cr3,ecx

The current VMExit is handling a cr3 access from system function KiKernelSysretExit:

kd> dc 87d5cf40 
87d5cf40  87d5cfd8 00000006 83f38028 00000000  ........(.......
87d5cf50  00000100 00000000 00000001 00000000  ................
87d5cf60  00000000 0051fdc8 987ba27e 87d5cfd8  ......Q.~.{.....
87d5cf70  00000000 00000000 00000000 00000000  ................
kd> u 83f38028 
nt!KiKernelSysretExit+0x28:
83f38028 0f22d9          mov     cr3,ecx
83f3802b 59              pop     ecx
83f3802c 0fa1            pop     fs
83f3802e fb              sti
83f3802f 0f35            sysexit
83f38031 cc              int     3

Detail files: https://1drv.ms/u/s!ApQpgQkWR0QOi8ZOVXUBX83WEQIE8Q

• a Debug build version of a compiled SYS file and a PDB file
• a log file
• a system crash dump file

Set CRx ReadShadow mask causes unhandled vmexit 33(invalid guest state) in windows x86.

Description

Set some bits to 1 in CR0 / CR4 read shadow mask may cause unhandled vm-exit with exit reason 33 (VmxExitReason::kInvalidGuestState), Windows x86 only.

That's because some bits in CR0/ CR4 can not be modified casually,

VMX-FIXED BITS IN CR0

said that bit of CRx is fixed to 0 in kIa32VmxCrxFixed0, and bit of CRx is fixed to 1 in kIa32VmxCrxFixed1, and you should follow these rules when MOV to CR0/CR4. if you don't, the CRx value might be invalid and an invalid guest state vm-exit could be issued.

How to fix

Plan A : (quick fix)
Comment the UtilVmWrite(VmcsField::kGuestCr0, *register_used); in VmmpHandleCrAccess
Comment the UtilVmWrite(VmcsField::kGuestCr4, *register_used); in VmmpHandleCrAccess

Plan B (better) :

use

		  const Cr0 cr0_fixed0 = { UtilReadMsr(Msr::kIa32VmxCr0Fixed0) };
		  const Cr0 cr0_fixed1 = { UtilReadMsr(Msr::kIa32VmxCr0Fixed1) };
		  Cr0 cr0 = { *register_used };
		  cr0.all &= cr0_fixed1.all;
		  cr0.all |= cr0_fixed0.all;

          UtilVmWrite(VmcsField::kGuestCr0, cr0.all);
          UtilVmWrite(VmcsField::kCr0ReadShadow, cr0.all);

instead of simply writing *register_used to guest CR0 & CR0 read shadow in VmmpHandleCrAccess.

the CR4 follows the same way...

Expected behavior

The driver should be loaded successfully and not be supposed to crash the system.

Actual behavior

see the snapshots

// Unexpected VM-exit. Fatal error. _Use_decl_annotations_ static void VmmpHandleUnexpectedExit( GuestContext *guest_context) { VmmpDumpGuestSelectors(); HYPERPLATFORM_COMMON_BUG_CHECK(HyperPlatformBugCheck::kUnexpectedVmExit, reinterpret_cast<ULONG_PTR>(guest_context), 0, 0); }
code = 33

Steps to reproduce the problem

1.Download the latest HyperPlatform-master branch source code.
2.Uncomment the cr4_mask.fields.vmxe = true; and cr4_shadow.fields.vmxe = false; in VmpEnterVmxMode just like the comment said // For example, when we want to hide CR4.VMXE from the guest, comment in below.
3.Compile the source code in x86/release config.
4.Load the driver.

Specifications

• Commit : none

• OS version : Windows 7 sp1 (7601)

• Architecture: x86 only

• Hardware: Vmware 12

• Details:

System Crash on Insider Preview (170127-1750)

Description

System occasionally crashes when HyperPlatform is loaded on an Insider preview version of Windows due to kInvalidGuestState (0n33). This case, Guest's RIP points to the very first instruction in a guest (ie, asmResumeVm), indicating that guest setup is not done correctly and the guest did not run at all.

Some times HyperPlatform loads successfully, but subsequent execution of some applications fails doe to access violation. This might indicate that an issue is related to memory management settings (CR4, descriptor tables, EPT etc).

Confirmed on Built by: 15025.1000.amd64fre.rs_prerelease.170127-1750

Expected behavior

HyperPlatform installs and run without breaking system behaviour.

Actual behavior

System crashes, or some applications stop running properly after installation.

Steps to reproduce the problem

Compile HyperPlatform for Debug x64
Loads the driver onto the insider preview version
System crashes, or DrvLoader.exe etc starts showing access violation errors.

log.txt

Specifications

• Commit (eg, 2099af7): fd224aa

• OS version (ie, outout of the "ver" command, or N/A): Microsoft Windows [Version 10.0.15025]

• Architecture: x64

• Hardware (eg, Physical, VMware 11, Bochs, or N/A): Physical, VMware 12

• Details:

Neither disabling enable_ept nor descriptor_table_exiting in VmxProcessorBasedControls helped.

_SAFE() log entries may not be sent to a kernel debugger

_SAFE() macros store logs messages to buffer and do not send them to a kernel debugger right now. The stored log messages are, instead, sent to a kernel debugger via DbgPrintEx() and a log file when a log flush thread runs or when non-_SAFE() is called. However, in the latter case, the stored logs are not sent to a kernel debugger; therefore not printed out to the debug console. This can make debugging difficult some time as the debug console is a primary way to monitor log messages.

Should not MMIO region be Uncacheable (not WriteBack)?

Capturing a question in a thread #29.

Enhancement: Support for VMCS Shadowing

This would allow using hyperplatform while still enjoying the use of Virtual Box, VMWare, or other virtualization software (note that supporting Hyper-V would not be possible, as it enables the hypervisor at boot).

Update a document about creating a new project with regard to submodules

Using submodules instead of copying files is planned. Update the user document once we started to use submodules in order to reflect related operational changes.

(Deleted)

Duplicated

ECX, EDI and ESI are not updated on IN, INS, OUT, OUTS

As noted in VmmpHandleIoPort(), ECX should be updated when the REP prefix is used. Similarly, EDI and ESI should be updated on INS and OUTS handling.

System Crash when execute sidt instruction

System Crash when execute sidt instruction

Description

When execution some instructions like sidt with unreadable or unwriteable address,VMM crash.

Expected behavior

EVENT INJECTION

Actual behavior

VMM Crash

Steps to reproduce the problem

execute

__try
	{
		__sidt((void*)0);
	}
__except (1)
	{

	}

No CPL Checks for Privileged Instructions

Hi,

Many exit handlers for privileged instructions, such as XSETBV, MSR, IN/OUT, do not do appropriate checks for CPL and/or IOPL. As such, with HyperPlatform running, guest user-mode code is allowed to bypass ring permissions.

Support VT-x nesting?

Can HyperPlatform work(nest) with other VT-x/EPT project except VMware and VirtualBox?

I tried HyperPlatform with some VT-x projects and all of them crashed the vmware's virtual CPU while initializing.
it even crashed when I tried to load HyperPlatform twice.

Trigger PatchGuard (GDT)

Hi~

I found that the default project will trigger PatchGuard on win7 (x64) SP1, and the following is part of the blue screen. It seems that the information of GDT is detected.

The test way is to only load the drive, and then wait a moment to trigger.

CRITICAL_STRUCTURE_CORRUPTION (109)
This bugcheck is generated when the kernel detects that critical kernel code or
data have been corrupted. There are generally three causes for a corruption:

A driver has inadvertently or deliberately modified critical kernel code
or data. See http://www.microsoft.com/whdc/driver/kernel/64bitPatching.mspx
A developer attempted to set a normal kernel breakpoint using a kernel
debugger that was not attached when the system was booted. Normal breakpoints,
"bp", can only be set if the debugger is attached at boot time. Hardware
breakpoints, "ba", can be set at any time.
A hardware corruption occurred, e.g. failing RAM holding kernel code or data.
Arguments:
Arg1: a3a039d896da8849, Reserved
Arg2: b3b7465ee958c4b3, Reserved
Arg3: fffff80000b95000, Failure type dependent information
Arg4: 0000000000000003, Type of corrupted region, can be
0 : A generic data region
1 : Modification of a function or .pdata
2 : A processor IDT
3 : A processor GDT
4 : Type 1 process list corruption
5 : Type 2 process list corruption
6 : Debug routine modification
7 : Critical MSR modification
Debugging Details:
BUGCHECK_STR: 0x109

DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT

PROCESS_NAME: System

CURRENT_IRQL: 2

LAST_CONTROL_TRANSFER: from fffff80003fcdd92 to fffff80003ede490

SYMBOL_ON_RAW_STACK: 1

STACK_ADDR_RAW_STACK_SYMBOL: fffff880047bcfc0

STACK_COMMAND: dds 047BCFC0-0x20 ; kb

Virtualization is not active after returning from sleep (S1-3) or hibernate (S4) state

System is no longer virtualized after recovering from sleep or hibernate.

Steps to reproduce

1. run ping_vmm
2. installs HyperPlatform and confirm that ping_vmm shows Ping : Pong from VMM!
3. sleep or hibernate the system
4. wake up the system
5. ping_vmm shows Ping : GenuineIntel

Possible fix

How to hide a hook: A hypervisor for rootkits indicates that using a callback with PowerStateCallback let you re-virtualize processors on wake up.

KERNEL_SECURITY_CHECK_FAILURE with PcMark8

Executing benchmark with PcMark8 leads a bug check on a physical device due to unconditional execution of RDMSR in a VM-exit handler causing #GP at DISPATCH_LEVEL in case unsupported MSR by the device is given. Crash analysis logs is attached.
issue.txt

In this log, unknown MSR 0x31 was given to UtilReadMsr() resulting in #GP.

Note that this does not reproduce in a VMware VM because VMware swallows unsupported MSR I/O by returning none instead of raising #GP.

Change doxygen style from ///< to //!<

Doxygen comment ///< leads Visual Studio to an error of comment analysis. For example,

enum class HyperPlatformBugCheck : ULONG {
  kUnspecified,                    ///< An unspecified bug occured

this code shows below error:

XML comment contains invalid XML: Whitespace is not allowed at this location.

Instead, use //!<. For example,

enum class HyperPlatformBugCheck : ULONG {
  kUnspecified,                    //!< An unspecified bug occured

this code lets Visual Studio show:

!< An unspecified bug occured

While it still has unwanted !< at the beginning, it much more helpful then nothing.

Resource(s)

• https://www.stack.nl/~dimitri/doxygen/manual/docblocks.html

Support for XSAVES needed for Windows 10 Redstone / Skylake

Hi,

Intel Skylake processors add new XSAVES instruction, and Windows 10 Redstone kernel makes use of it if available. Currently, hyperplatform does not enable XSAVES support in the guest, meaning the instruction will generate #UD.

Please set enable_xsaves_xstors to true in secondary controls, in order to avoid this issue.

Hardware breakpoint does not work

Summary

Hardware breakpoint (HWBP) does not hit after HyperPlatform is installed. After VmInitialization() returned successfully, I set HWBP onto a guest kernel code running on VMware through Windbg from a host. Contents of DR registers seemed fine, but HWBP did not trigger, and also, even if I set HWBP before VmInitialization(), the HWBP did not hit after VmInitialization() was executed.

Workaround

Disable Move-to-DR-exiting by setting false to mov_dr_exiting. This lets HWBP function as it should be.

Related Resources

There are some bug reports with regard to HWBP support on VirtualBox and KVM. Those may or may not be related.

• Hardware breakpoints don't work in Virtualbox (software virtualization only)
  https://www.virtualbox.org/ticket/477
  Not much into to see.
• KVM: x86: fix root cause for missed hardware breakpoints
  https://patchwork.kernel.org/patch/8436141/
• KVM: x86: fix missed hardware breakpoints
  https://patchwork.kernel.org/patch/8359491/
• KVM: Fix and improve guest debugging and x86 debug registers
  http://thread.gmane.org/gmane.comp.emulators.kvm.devel/22838
• kvm-userspace: Fix and improve guest debugging and x86 debug registers
  http://thread.gmane.org/gmane.comp.emulators.kvm.devel/22818

Steps to Reproduce

Disassembly used for below test

$$
$$ break at the entry point of the driver
$$
MemoryMon!DriverEntry+0x9b:
fffff801`1f15409b int     3

$$ 
$$ Set software BP (SWBP) and HWBP to verify HWBP works yet
$$
kd> bp fffff801`1f1540a1
kd> ba e 1 fffff801`1f1540a6 
kd> g
Breakpoint 0 hit
MemoryMon!DriverEntry+0xa1:
fffff801`1f1540a1 call    MemoryMon!ExInitializeDriverRuntime (fffff801`1f1315e0)

$$
$$ The SWBP hit. Check contents of DR registers
$$ => DR0 seems good
$$
kd> rM 20
dr0=fffff8011f1540a6 dr1=0000000000000000 dr2=0000000000000000
dr3=0000000000000000 dr6=00000000ffff0ff0 dr7=0000000000000401 cr4=00000000001506f8
kdr0=fffff8011f1540a6 kdr1=0000000000000000 kdr2=0000000000000000
kdr3=0000000000000000 kdr6=00000000ffff0ff0 kdr7=0000000000000401
MemoryMon!DriverEntry+0xa1:
fffff801`1f1540a1 call    MemoryMon!ExInitializeDriverRuntime (fffff801`1f1315e0)

$$
$$ The HWBP works yet
$$
kd> g
Breakpoint 1 hit
MemoryMon!DriverEntry+0xa6:
fffff801`1f1540a6 mov     byte ptr [rsp+20h],0

$$
$$ Clear old BPs. Set SWBP and HWBP on code after VmInitialization()
$$
kd> bc *
kd> bp fffff801`1f154206
kd> ba e 1 fffff801`1f15420b 

$$
$$ Hit the SWBP. 
$$
kd> g
Breakpoint 0 hit
MemoryMon!DriverEntry+0x206:
fffff801`1f154206 cmp     dword ptr [rsp+24h],0

$$
$$ Contents of DR registers seem good
$$
kd> rM 20
dr0=fffff8011f15420b dr1=0000000000000000 dr2=0000000000000000
dr3=0000000000000000 dr6=00000000ffff0ff0 dr7=0000000000000401 cr4=00000000001526f8
kdr0=fffff8011f15420b kdr1=0000000000000000 kdr2=0000000000000000
kdr3=0000000000000000 kdr6=00000000ffff0ff0 kdr7=0000000000000401
MemoryMon!DriverEntry+0x206:
fffff801`1f154206 cmp     dword ptr [rsp+24h],0 ss:0018:ffffd001`321f7924=00000000

$$
$$ HWBP at fffff8011f15420b did not fire for some reasons, while DR registers still seem fine
$$
kd> g
MemoryMon!TestRwe+0x37:
fffff801`1f13bb07 int     3
kd> rM 20
dr0=fffff8011f15420b dr1=0000000000000000 dr2=0000000000000000
dr3=0000000000000000 dr6=00000000ffff0ff0 dr7=0000000000000401 cr4=00000000001526f8
kdr0=fffff8011f15420b kdr1=0000000000000000 kdr2=0000000000000000
kdr3=0000000000000000 kdr6=00000000ffff0ff0 kdr7=0000000000000401

Investigate use of 1GB or 2MB EPT mappings for MMIO

Description

HyperPlatform pre-allocates and assigned 4KB EPT entries for physical addresses (PA) used for memory mapped I/O. This design leads to two major disadvantages: complexity in code, and limited support for access to such PA ranges.
#19 identified a way to enumerate such PA ranges (NB: not idea is tested yet) so that HyperPlatform could allocate EPT entries for those PA ranges and get rid of the pre-allocation code. PA ranges reported in the way explained in #19 are too large to allocate EPT entries for them if only 4KB mapping is used, however.

This issue report is to analyze a way to utilize 1GB or 2MB EPT mappings to over come the challenge and simplify code.

Note that use of 1GB or 1MB mappings for normal PA pages is not aimed since we would like to have an ability to control PA access rights with fine (ie, 4KB) granularity for VMI. 2MB mapping can be used for reducing a number of EPT entries to manipulate, but it would introduce non negligible complexity.

-

Always bugcheck with PAGE_FAULT_IN_NONPAGED_AREA on Lenovo y700 (7601, x64)

Hi, tandasat

HyperPlatform always causes PAGE_FAULT_IN_NONPAGED_AREA bugcheck on my laptop Lenovo Y700. This issue only occurs in 7601 x64, while HyperPlatform runs normally in 14393 x64 on the same laptop. I put HyperPlatform in VMWare Workstation running a 7601 x64 and no bugcheck occurs.
I have no idea what causes the problem. :(

My laptop configuration:
i7-6700HQ, 16G DDR4, pagefile enabled, without any anti-virus software.

Build with Visual Studio 2015 Update3 & WDK 14393

Some user-mode program call vmxxxx to generate #GP but captured by vmm

Recently I found some user-mode program running ical "vmcall" with an exception handler (SEH or VEH). if it's running under HyperPlatform's monitor, the "vmcall" will be handled by the vmexit handler, and the program won't generate a #GP so that it will not be working correctly.