!monitor doesn't work when process is started using CREATE_SUSPENDED

[DarkerSquirrel]

!monitor doesn't work when process is started using CREATE_SUSPENDED

[DarkerSquirrel]

I think these are heisenbugs, cannot reproduce them anymore.

But you are right indeed , !monitor fails for large mem span , it is awfully slow.
I managed to trace smaller spans, quite reliably , but I have to read process memory from another process, so Windows doesn't swap it off.

However , I can report another bug I encountered: if you monitor a large mem address and process ends during/before "events c all" ., this happens:

Microsoft (R) Windows Debugger Version 10.0.22621.382 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [C:\Windows\MEMORY.DMP]
Kernel Bitmap Dump File: Kernel address space is available, User address space may not be available.

Symbol search path is: srv*
Executable search path is:
Windows 10 Kernel Version 19041 MP (8 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS
Edition build lab: 19041.1.amd64fre.vb_release.191206-1406
Machine Name:
Kernel base = 0xfffff80639600000 PsLoadedModuleList = 0xfffff8063a22a2d0
Debug session time: Sun Aug 27 00:13:31.702 2023 (UTC + 3:00)
System Uptime: 0 days 0:57:49.360
Loading Kernel Symbols
...............................................................
....Page 403b26 not present in the dump file. Type ".hh dbgerr004" for details
............................................................
................................................................
...................
Loading User Symbols

Loading unloaded module list
.............
For analysis of this file, run !analyze -v
3: kd> !analyze -v

---

•                                                                         *
  

•                    Bugcheck Analysis                                    *
  

•                                                                         *
  

---

CLOCK_WATCHDOG_TIMEOUT (101)
An expected clock interrupt was not received on a secondary processor in an
MP system within the allocated interval. This indicates that the specified
processor is hung and not processing interrupts.
Arguments:
Arg1: 0000000000000018, Clock interrupt time out interval in nominal clock ticks.
Arg2: 0000000000000000, 0.
Arg3: ffffc2015d736180, The PRCB address of the hung processor.
Arg4: 0000000000000002, The index of the hung processor.

Debugging Details:

KEY_VALUES_STRING: 1

Key  : Analysis.CPU.mSec
Value: 3108

Key  : Analysis.DebugAnalysisManager
Value: Create

Key  : Analysis.Elapsed.mSec
Value: 7826

Key  : Analysis.Init.CPU.mSec
Value: 2061

Key  : Analysis.Init.Elapsed.mSec
Value: 15145

Key  : Analysis.Memory.CommitPeak.Mb
Value: 96

Key  : WER.OS.Branch
Value: vb_release

Key  : WER.OS.Timestamp
Value: 2019-12-06T14:06:00Z

Key  : WER.OS.Version
Value: 10.0.19041.1

FILE_IN_CAB: MEMORY.DMP

BUGCHECK_CODE: 101

BUGCHECK_P1: 18

BUGCHECK_P2: 0

BUGCHECK_P3: ffffc2015d736180

BUGCHECK_P4: 2

FAULTING_PROCESSOR: 2

PROCESS_NAME: steamwebhelper.exe

FAULTING_THREAD: ffffd401475e5080

BLACKBOXBSD: 1 (!blackboxbsd)

BLACKBOXNTFS: 1 (!blackboxntfs)

BLACKBOXPNP: 1 (!blackboxpnp)

BLACKBOXWINLOGON: 1

BAD_STACK_POINTER: ffffd4013d050e00

STACK_TEXT:
ffffd4013d050e00 fffff80664145078 : ffffd4013d002420 00000000000001a1 0000000113305f80 ffffd4013d050ea0 : hprdbghv!EptHandlePageHookExit+0x66 [C:\games\HyperDbg\hyperdbg\hprdbghv\code\vmm\ept\Ept.c @ 693]
ffffd4013d050e70 fffff8066414a31a : ffffd4013d002420 0000000000000000 0000000000000000 0000000000000000 : hprdbghv!EptHandleEptViolation+0x58 [C:\games\HyperDbg\hyperdbg\hprdbghv\code\vmm\ept\Ept.c @ 826]
ffffd4013d050ed0 fffff80664138c57 : ffffd4013d050f60 0000000000000000 0000000000000000 0000000000000000 : hprdbghv!VmxVmexitHandler+0x18a [C:\games\HyperDbg\hyperdbg\hprdbghv\code\vmm\vmx\Vmexit.c @ 182]
ffffd4013d050f40 ffffd4013d050f60 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : hprdbghv!AsmVmexitHandler+0x27 [C:\games\HyperDbg\hyperdbg\hprdbghv\code\assembly\AsmVmexitHandler.asm @ 65]
ffffd4013d050f48 0000000000000000 : 0000000000000000 0000000000000000 0000000000000000 0000000000000028 : 0xffffd401`3d050f60

FAULTING_SOURCE_LINE: C:\games\HyperDbg\hyperdbg\hprdbghv\code\vmm\ept\Ept.c

FAULTING_SOURCE_FILE: C:\games\HyperDbg\hyperdbg\hprdbghv\code\vmm\ept\Ept.c

FAULTING_SOURCE_LINE_NUMBER: 693

FAULTING_SOURCE_CODE:
689: BOOLEAN IsExecViolation = FALSE;
690: UINT64 CurrentRip;
691: UINT32 CurrentInstructionLength;
692:

693: LIST_FOR_EACH_LINK(g_EptState->HookedPagesList, EPT_HOOKED_PAGE_DETAIL, PageHookList, HookedEntry)
694: {
695: if (HookedEntry->PhysicalBaseAddress == PAGE_ALIGN(GuestPhysicalAddr))
696: {
697: //
698: // *** We found an address that matches the details ***

SYMBOL_NAME: hprdbghv!EptHandlePageHookExit+66

MODULE_NAME: hprdbghv

IMAGE_NAME: hprdbghv.dll

STACK_COMMAND: .process /r /p 0xffffd401472b6080; .thread 0xffffd401475e5080 ; kb

BUCKET_ID_FUNC_OFFSET: 66

FAILURE_BUCKET_ID: CLOCK_WATCHDOG_TIMEOUT_INTERRUPTS_DISABLED_STACKPTR_ERROR_hprdbghv!EptHandlePageHookExit

OS_VERSION: 10.0.19041.1

BUILDLAB_STR: vb_release

OSPLATFORM_TYPE: x64

OSNAME: Windows 10

FAILURE_ID_HASH: {33a0d65a-1fe3-e344-e050-df931ac4b9e0}

Followup: MachineOwner

Sina, are you kind enough to suggest me any other possible FAST alternative to log self-modifying code of process in real time from ring-0 for a huge memory adress range? What approach should I take? Is it possible to remove W page rights of all mem area and then intercept exceptions in ring-0 and turn on/off W rights transparently from ring-3? I dont want ring-3 to handle the exceptions.

Thanks

[SinaKarvandi]

Hi,
Thanks for creating this issue.

Why HyperDbg cannot create events for suspended processes, when I look in x64dbg , KUSER_SHARED_DATA is mapped already.

This is the exact reason why we designed the '.pagein command'. Unfortunately, this command only supports Debugger Mode for now. Once you start a process, Windows doesn't initially map the addresses (or make it available through the page-tables). The reason why x64dbg successfully shows you the result is that it directly accesses the address in the user-mode (PASSIVE_LEVEL) which in turn, causes a page-fault that is thrown and handled by Windows. But, in HyperDbg we cannot access a paged-out or not yet valid page entry as the paging doesn't work in VMX-root mode.

You can use a trick that I previously showed in HyperDbg's telegram group:
https://t.me/HyperDbg/3221

Also in the moment I unload the vmm , Windows goes BSOD. Culprit: hprdbgkd.sys , reason : DRIVER_IRQL_NOT_LESS_OR_EQUAL

This is quite expected as you're running HyperDbg on a relatively old processor (4th Gen), but in order to fix it, I need more details like the result of WinDbg's '!analyze -v'.

But, is there any special reason why you are not using it in the Debugger Mode?

[DarkerSquirrel]

I am not using it in debugger mode cause I think that implies to run the debugged program inside a VM, right?
That doesn't work for this case, cause I am researching a game protection and that game has OpenGl/Directx requirements not supported by VMware , also the protection triggers inside VM.

analyze_v.txt

Please download also the .dmp file from here , in case u need it:
https://drive.google.com/file/d/1t69pK2xBWBdOKVNcKTHNH6Yy873vS5st/view?usp=sharing
(I will let it for few days, and then I will delete it).

It would be very nice to have a reliable .start in VMI mode, and even nicer to have some kind of capability to follow up
child processes. Many games/software have launchers or create new processes.

Thx for the quick response.

[DarkerSquirrel]

I think the docs state that .pagein command is just for debugger mode.

I solved it in another way , I opened game in x64dbg, bp at entry point, loaded the dll I was interesting in tracing via x64dbg functionality, after that I coded a program which did OpenProcess(game.exe) and ReadProcessMemory() of the memory interval of desired dll.

That got rid off the c0000005 err, but I hit another:

HyperDbg> !monitor w 180307000 182BB3FFF pid 12BC script { if( $ip < 0x7FFFFFFFFFF) { printf("Writing from RIP=%llx at addr=%llx\n",$ip ,$context); } }
err, the page modification is not applied, make sure that you don't put multiple EPT Hooks or Monitors on a single page (c0000026)

When !monitor command was executed, game was still bp-ed in x64dbg at entry point.

Can u advise?
Thanks

[SinaKarvandi]

Thank you for providing the result of !analyze -v, it seems to be a solvable issue. But just to confirm with you. Can you reproduce this issue? I mean do you see this BSOD, each time you try to unload HyperDbg? Or it's just one time? Is there any activated event while you unload HyperDbg?

I have to reproduce the error on my system, so I can further investigate it.

It would be very nice to have a reliable .start in VMI mode, and even nicer to have some kind of capability to follow up
child processes. Many games/software have launchers or create new processes.

Sure, I will add it to the to-do list. There are some logical flaws in VMI Mode (like finding a way of halting the process), once these logical caveats are solved, I'll implement the '.start' for the VMI Mode too.

Can u advise?
Yes, this error happens once you try to put a '!monitor' on a similar page twice. Have you applied any other '!monitor' to the target address? You can check it by using the 'events' command. But if you apply !monitor on another address and you still get this error, it means that both addresses are located at the same page-table entry. You can confirm it by using the '!pte' command, whether the PML1 entry is located on the same page or not.

If you want to use it on both addresses, then you can put only one '!monitor' on a bigger address range and filter your results from script engine.

[DarkerSquirrel]

I can't reproduce the issue reliably. In order to trigger this , try something like:
start a process with a big dll(my dll is 350 MB large) i x64dbg, bp at entry,set the !monitor on all dll address range and
after that read process memory of dll from another app with ReadProcessMemory() :
while(true){
readprocessmemory(dll_mem_range);
sleep(10sec);
}
after a while you will see weird artifacts like: you can't kill the process which does the reading, even from task manager,
hyperdbg command will become unresponsive and when it does come back eventually , and u unload vmm -> BSOD.

" Is there any activated event while you unload HyperDbg?" ->no , when I tried to !monitor dll range, hyperdbg-cli took ages to finish, and after it did and spew the err,I unloaded vmm and BSOD.

I put very big range adresses in !monitor command, even spanning all 350 MB range of dll mem space, I still get that c0000026 err.

"Have you applied any other '!monitor' to the target address?" ->Come on , I ain't that dumb! :D

"You can confirm it by using the '!pte' command, whether the PML1 entry is located on the same page or not." -> let me check this and I will get back to you.

[SinaKarvandi]

I put very big range adresses in !monitor command, even spanning all 350 MB range of dll mem space, I still get that c0000026 err.

350 MB? 😳

That's too big. The '!monitor' isn't supposed to handle that amount of memory :)

By the way, whatever address you want to put a '!monitor' on it should be valid and available in the memory. It is not surprising at all if Windows only loads a portion of your 350 MB DLL in the memory as 350 MB is too big. Furthermore, it even won't load this amount of data to the RAM. Only the portion of DLL that is used will be available in the memory. It probably occupied a lot of pages in the memory so Windows will decide to move it frequently to the disk (page-out) in order to be able to handle other processes.

[DarkerSquirrel]

I made sure it's available by reading it often from another process using ReadProcessMemory().
I tried also monitoring just some sections of dlls, much less, let's say a span of tens of MB, and still didnt work.
But not C000005 err, I encountered just C0000026 err.

Is there any hardcoded maximum limit of mem span for !monitor?

I remember I found a more or less hackish way to prevent pages being swapped in Windows in another project on github(don't remember now which one, must look for it). Would be possible tom implement such a measure in HyperDbg?

[SinaKarvandi]

Is there any hardcoded maximum limit of mem span for !monitor?
No, there is no limitation. The only place where we set this error is here:

If you set a LogInfo there, you could see what addresses were problematic.

Add this line of code like the above picture and recompile it:
LogInfo("Previous physical base address: %llx, previous physical virtual address : %llx | current physical base address: %llx, current virtual address: %llx", HookedEntry->PhysicalBaseAddress, HookedEntry->VirtualAddress, PhysicalBaseAddress, TargetAddress);

I remember I found a more or less hackish way to prevent pages being swapped in Windows in another project on github(don't remember now which one, must look for it). Would be possible to implement such a measure in HyperDbg?

I don't have any idea how to implement such a mechanism. Please send more information to me if you find any details for this.

[SinaKarvandi]

Sina, are you kind enough to suggest me any other possible FAST alternative to log self-modifying code of process in real time from ring-0 for a huge memory adress range? What approach should I take? Is it possible to remove W page rights of all mem area and then intercept exceptions in ring-0 and turn on/off W rights transparently from ring-3? I dont want ring-3 to handle the exceptions.

Sure. Generally, the best way to have a detector for memory READ/WRITE detection is hypervisors, but as you can see, monitoring a large amount of memory (like 350 MB) is problematic, the other option might be using CPU emulators like QEMU, Bochs, which I'm not sure if it works for your case or not.

Is it possible to remove W page rights of all mem area and then intercept exceptions in ring-0 and turn on/off W rights transparently from ring-3?

It's theoretically possible, but when it comes to the implementation, no it's not possible. Because by doing something like this, you also prevent the memory manager of the process itself from modifying the page-table bit. Other than that, there are plenty of processor events like exceptions that directly write content on the stack and by removing READ/WRITE bits, you also prevent the processors' capability of modifying memory and there are thousands of reasons and considerations that might BSOD your computer.

I dont want ring-3 to handle the exceptions.

You can do it by HyperDbg's exception command. Use the event short-circuiting mechanism of HyperDbg to ignore re-injecting exceptions/faults/interrupts to the guest. Other than that, you can also inject interrupts/faults/execptions in any situations by using a not yet documented function in HyperDbg, called 'event_inject()' or 'event_inject_error_code()'.

[DarkerSquirrel]

Sina , I don't have anything relevant to add to this Issue, you can close it if you want.
I hope you can fix those 2 BSOD reported, add .start and .pagein to the VMI mode, make Hyperdbg follow the newly created processes.
I think that some checks should be implemented for validity of in-mem pages.
I didn't find yet the src which showed how to keep pages in mem, when I will find it , I will let you know.

Best regards

[SinaKarvandi]

Hi,
Thanks for mentioning it, let's keep this issue open until I find a time to solve it.

So that you know, the previous bug that you mentioned at #91 is now fixed at multiple-eptp branch.

[DarkerSquirrel]

Really? You implemented multi-EPTP?

Wow , man, you're my personal hero! ;-)

[SinaKarvandi]

Yes, but please give me some time to make further tests to make sure that it's stable, once I merge it to the 'dev' branch, it would be good to use. Right now, if you really need to use this feature, you can compile and use the 'multiple-eptp' branch. But I recommend not to use it before the merge.