Comments (26)
OK, this is a bit embarrassing: a quick chat with @tklengyel revealed that I was using an outdated/unsupported Ubuntu version, where the kernel didn't have support for IOCTL_PRIVCMD_MMAP_RESOURCE. Upgrading to 20.04 (5.4 kernel) resolved the issue.
from kernel-fuzzer-for-xen-project.
@icedevml Yea, it's been on my TODO to move the submodule up closer to the latest version of the PT series. @v-p-b you can try Xen from that branch in the interim!
from kernel-fuzzer-for-xen-project.
@v-p-b I would certainly suggest trying it on another machine as well as this issue hasn't popped up on anything I tried so far.
from kernel-fuzzer-for-xen-project.
That happens if you didn't start the parent VM with the pt buffer size specified (ie processor_trace_buf_kb=65536
).
from kernel-fuzzer-for-xen-project.
I have that line in the .cfg file used when creating the domain. I installed the OS earlier without that setting though, can that be a problem? Does the position of this line matter in the config? When calling xenforeignmemory_map_resource
, pt_buf_size
also seems to be properly set, although I don't know if that's related to the VMs config - my military grade debug print shows something like:
xenforeignmemory_map_resource(0x0x563a887213c0,4,2,0,0,0x4000,0x0x563a840c7898,PROT_READ,0);
from kernel-fuzzer-for-xen-project.
I installed the OS earlier without that setting though, can that be a problem? Does the position of this line matter in the config?
No, you only need to add that when you need it, it's not exposed to the guest. It doesn't matter where in the config you have it either. The only time I ran into that was when I booted the VM without the option set. There is a tool in the Xen folder in tools/proctrace
, try to compile that and see if you can get it working on your VM. That would help narrow down where the issue might be.
from kernel-fuzzer-for-xen-project.
Same result with proctrace
:
root@zero1:/home/b/debian# xl create debian.cfg
Parsing config from debian.cfg
root@zero1:/home/b/debian# cat debian.cfg
name="debian"
builder="hvm"
vcpus=1
maxvcpus=1
memory=1024
maxmem=1024
hap=1
boot="cd"
serial="pty"
vif=['bridge=xenbr0']
vnc=1
vnclisten="0.0.0.0"
vncpasswd='1234567'
usb=1
usbdevice=['tablet']
processor_trace_buf_kb=65536
# Make sure to update the paths below!
disk=['file:/home/b/debian/vmdisk.img,xvda,w',
]
root@zero1:/home/b/debian# xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 4094 2 r----- 18.4
debian 2 1016 1 -b---- 6.8
root@zero1:/home/b/debian# ../kfx-pt/xen/tools/proctrace/proctrace 2 0
Failed to map trace buffer
Is there maybe some BIOS setting I have to activate for PT (I had to manually enable VT-x for example)? Here's how one of my cores looks (running on bare hw):
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 158
model name : Intel(R) Core(TM) i7-9700F CPU @ 3.00GHz
stepping : 13
microcode : 0xca
cpu MHz : 800.441
cache size : 12288 KB
physical id : 0
siblings : 8
core id : 1
cpu cores : 8
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 22
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities
bugs : spectre_v1 spectre_v2 spec_store_bypass swapgs taa itlb_multihit
bogomips : 6000.00
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:
from kernel-fuzzer-for-xen-project.
Does xl dmesg
show anything relevant perhaps? I didn't have to set anything in the bios on any of the systems I tried so far to get it enabled, just vt-x. I'm afraid you might need to go into Xen and add some printk's to try to figure out why that hypercall might fail. Maybe Michal who developed the Xen PT patches has some clue (/cc @icedevml)?
from kernel-fuzzer-for-xen-project.
will look tomorrow
from kernel-fuzzer-for-xen-project.
@tklengyel I think you are missing Andrew Cooper's acquire_resource patches, could you rebase xen
to:
https://github.com/icedevml/xen/commits/ipt-patch-v7s ?
It contains a whole lot of patches from Andrew that are fixing things around acquire_resource in particular.
from kernel-fuzzer-for-xen-project.
I tried the ipt-patch-v7s tag, but no luck :(
I see this in xl dmesg
:
(d2) Booting from Hard Disk...
(d2) Booting from 0000:7c00
(XEN) stdvga.c:178:d2v0 leaving stdvga mode
(XEN) stdvga.c:173:d2v0 entering stdvga mode
(XEN) Dom2 callback via changed to Direct Vector 0xf3
(XEN) irq.c:378: Dom2 PCI link 0 changed 5 -> 0
(XEN) irq.c:378: Dom2 PCI link 1 changed 10 -> 0
(XEN) irq.c:378: Dom2 PCI link 2 changed 11 -> 0
(XEN) irq.c:378: Dom2 PCI link 3 changed 5 -> 0
(XEN) vmx.c:3184:d2v0 RDMSR 0x0000064e unimplemented
(XEN) vmx.c:3184:d2v0 RDMSR 0x00000034 unimplemented
(XEN) grant_table.c:1841:d2v0 Expanding d2 grant table from 1 to 2 frames
(XEN) vmx.c:3184:d2v0 RDMSR 0x00000606 unimplemented
(XEN) vmx.c:3184:d2v0 RDMSR 0x00000611 unimplemented
(XEN) vmx.c:3184:d2v0 RDMSR 0x00000639 unimplemented
(XEN) vmx.c:3184:d2v0 RDMSR 0x00000641 unimplemented
(XEN) vmx.c:3184:d2v0 RDMSR 0x00000619 unimplemented
I'm attaching the full dmesg log:
dmesg_d3.txt
from kernel-fuzzer-for-xen-project.
OK, I just realized, that my kernel doesn't have PT support somehow (/sys/devices/intel_pt doesn't exists). I don't have a clue what can cause this, but I keep digging, and close this issue for now.
from kernel-fuzzer-for-xen-project.
After a kernel upgrade I can use PT with perf
on bare HW but not from dom0 (I recompiled Xen and booting the new kernel). Is this normal?
bare metal:
root@zero1:/home/b# perf record -e intel_pt// id
uid=0(root) gid=0(root) groups=0(root)
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.120 MB perf.data ]
root@zero1:/home/b# uname -a
Linux zero1 4.15.0-124-generic #127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
root@zero1:/home/b# xen-detect
Not running on Xen.
dom0:
root@zero1:/home/b# xen-detect
Running in PV context on Xen V4.15.
root@zero1:/home/b# uname -a
Linux zero1 4.15.0-124-generic #127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
root@zero1:/home/b# perf record -e intel_pt// id
event syntax error: 'intel_pt//'
\___ Cannot find PMU `intel_pt'. Missing kernel support?
Run 'perf list' for a list of valid events
Usage: perf record [<options>] [<command>]
or: perf record [<options>] -- <command> [<options>]
-e, --event <event> event selector. use 'perf list' to list available events
from kernel-fuzzer-for-xen-project.
I think this is normal. Xen masks some features in such way that they couldn't be normally used.
from kernel-fuzzer-for-xen-project.
I think you've ran into some bug around acquire_resource. I will look at this soon.
from kernel-fuzzer-for-xen-project.
I added some debug strings:
diff --git a/tools/libs/foreignmemory/linux.c b/tools/libs/foreignmemory/linux.c
index eec089e..4fd6ae9 100644
--- a/tools/libs/foreignmemory/linux.c
+++ b/tools/libs/foreignmemory/linux.c
@@ -317,8 +317,10 @@ int osdep_xenforeignmemory_map_resource(
fres->addr = mmap(fres->addr, fres->nr_frames << PAGE_SHIFT,
fres->prot, fres->flags | MAP_SHARED, fmem->fd, 0);
- if ( fres->addr == MAP_FAILED )
- return -1;
+ if ( fres->addr == MAP_FAILED ){
+ PERROR("mmap failed");
+ return -1;
+ }
mr.addr = (uintptr_t)fres->addr;
@@ -327,11 +329,13 @@ int osdep_xenforeignmemory_map_resource(
{
int saved_errno;
- if ( errno != fmem->unimpl_errno && errno != EOPNOTSUPP )
+ if ( errno != fmem->unimpl_errno && errno != EOPNOTSUPP ){
PERROR("ioctl failed");
- else
+ }
+ else{
errno = EOPNOTSUPP;
-
+ PERROR("EOPNOTSUPP");
+ }
saved_errno = errno;
(void)osdep_xenforeignmemory_unmap_resource(fmem, fres);
errno = saved_errno;
Got this when creating the domain:
# xl create debianpt.cfg
Parsing config from debianpt.cfg
xenforeignmemory: error: EOPNOTSUPP: Operation not supported
# ./proctrace 1 0
xenforeignmemory: error: EOPNOTSUPP: Operation not supported
Failed to map trace buffer
So it seems IOCTL_PRIVCMD_MMAP_RESOURCE
is failing, but unfortunately I'm lost at this point about where this IOCTL is supposed to be handled and I can't debug further.
from kernel-fuzzer-for-xen-project.
The ioctl is handled by the xen_privcmd
kernel module, which is effectively just a bouncer to forward the syscall as a hypercall to Xen. I find it interesting that you get an EOPNOTSUPP error though. To me that sounds like you may have a mismatch between the toolstack and the running version of Xen - perhaps the toolstack has the PT functions defined but the version of Xen you are running doesn't understand the hypercalls? Can you make sure you are running the version of Xen that has the PT patches? You could perhaps also add a gdprintk to these function in Xen (http://xenbits.xen.org/gitweb/?p=people/tklengyel/xen.git;a=blob;f=xen/arch/x86/hvm/vmx/vmx.c;h=446fbdde08a53ce4b4d136c8b3976c4c5c1846e2;hb=refs/heads/ptcov2#l431 and http://xenbits.xen.org/gitweb/?p=people/tklengyel/xen.git;a=blob;f=xen/arch/x86/hvm/vmx/vmx.c;h=446fbdde08a53ce4b4d136c8b3976c4c5c1846e2;hb=refs/heads/ptcov2#l2366) to verify that PT does get enabled for the VM.
from kernel-fuzzer-for-xen-project.
Full-spectrum printk()'s show that the return 0
path is taken in vmx_init_pt()
:
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 556kB init memory
(XEN) d0: Forcing write emulation on MFNs e0000-effff
(XEN) PCI add device 0000:00:00.0
(XEN) PCI add device 0000:00:01.0
(XEN) PCI add device 0000:00:14.0
(XEN) PCI add device 0000:00:14.2
(XEN) PCI add device 0000:00:16.0
(XEN) PCI add device 0000:00:17.0
(XEN) PCI add device 0000:00:1b.0
(XEN) PCI add device 0000:00:1c.0
(XEN) PCI add device 0000:00:1d.0
(XEN) PCI add device 0000:00:1f.0
(XEN) PCI add device 0000:00:1f.3
(XEN) PCI add device 0000:00:1f.4
(XEN) PCI add device 0000:00:1f.5
(XEN) PCI add device 0000:00:1f.6
(XEN) PCI add device 0000:01:00.0
(XEN) PCI add device 0000:01:00.1
(XEN) emul-priv-op.c:991:d0v1 RDMSR 0x0000064e unimplemented
(XEN) emul-priv-op.c:991:d0v1 RDMSR 0x00000034 unimplemented
(XEN) d0: Forcing read-only access to MFN fed00
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000606 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000606 unimplemented
(XEN) emul-priv-op.c:991:d0v1 RDMSR 0x0000060d unimplemented
(XEN) emul-priv-op.c:991:d0v1 RDMSR 0x000003f8 unimplemented
(XEN) emul-priv-op.c:991:d0v1 RDMSR 0x000003f9 unimplemented
(XEN) emul-priv-op.c:991:d0v1 RDMSR 0x000003fa unimplemented
(XEN) emul-priv-op.c:991:d0v1 RDMSR 0x00000630 unimplemented
(XEN) emul-priv-op.c:991:d0v1 RDMSR 0x00000631 unimplemented
(XEN) emul-priv-op.c:991:d0v1 RDMSR 0x00000632 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x0000060d unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x000003f8 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x000003f9 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x000003fa unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000630 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000631 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000632 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000611 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000639 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000641 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000619 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000611 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000639 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000641 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000619 unimplemented
(XEN) vmx.c:476:d0v1 XXXIPT return 0 from vmx_init_pt<G><2>HVM d1v0 save: CPU
I noticed the unimplemented MSR's before, but they looked to be related to energy consumption sensors (the current list is longer than I remember though). I see MSR_RTIT_CTL referenced by vmx_init_pt()
defined as 0x570, not reported above (or anywhere else in xl dmesg
output).
from kernel-fuzzer-for-xen-project.
So it seems your hardware is fine and PT gets setup correctly. Just have to figure out why mapping the pt buffer into dom0 fails for you. The XENMEM_resource_vmtrace_buf hypercall should eventually reach http://xenbits.xen.org/gitweb/?p=people/tklengyel/xen.git;a=blob;f=xen/common/memory.c;h=c0a22eb60f698b1a1dc45027adf7b3a5ed73a1d2;hb=refs/heads/ptcov2#l1010. Can you check if that's reached and what it returns?
from kernel-fuzzer-for-xen-project.
Interestingly I don't see EOPNOTSUPP
being set for this call anywhere O.o
from kernel-fuzzer-for-xen-project.
Never mind, found it, it's at http://xenbits.xen.org/gitweb/?p=people/tklengyel/xen.git;a=blob;f=tools/libs/foreignmemory/linux.c;h=8daa5828e3da2f1d0dee158ff91e8ba2274a4625;hb=refs/heads/ptcov2#l328. So yea, let's check what the hypervisor is doing when this request reaches it, I doubt the issue would be on the linux kernel module side that handles the ioctl.
from kernel-fuzzer-for-xen-project.
Sorry for not looking at it, backlog overflow. Maybe this is hitting some blacklist/whitelist on the Xen side? I've already seen such thing twice, there are some code pieces that are performing numerous sanity checks.
from kernel-fuzzer-for-xen-project.
I'm still debugging this, but so far I wasn't able to trigger any gdprintk's in or on the way to acquire_vmtrace_buf
.
Meanwhile can you tell me which kernel modules I should see in dom0? I can see some loading failures during boot, but I don't know what exactly is failing yet (systemd ftw!). Here's what I see:
# lsmod | fgrep xen
xen_pciback 61440 0
xen_netback 57344 1
xen_blkback 45056 1
xen_gntalloc 16384 0
xenfs 16384 1
xen_gntdev 20480 1
xen_privcmd 20480 39 xenfs
xen_evtchn 16384 5
from kernel-fuzzer-for-xen-project.
I found that do_memory_op()
(in xen/common/memory.c) is the main dispatcher that should lead execution to acquire_resource
(or am I missing something?). I placed debug prints in there, and found that the XENMEM_acquire_resource
branch is never called. I also printed all operation leading to the default branch of the switch (decimals at the end of lines are op codes):
# xl dmesg | fgrep XXX
(XEN) memory.c:1702:d0v0 XXXIPT do_memory_op default path12
(XEN) memory.c:1702:d0v0 XXXIPT do_memory_op default path10
(XEN) memory.c:1702:d0v1 XXXIPT do_memory_op default path18
(XEN) memory.c:1702:d0v1 XXXIPT do_memory_op default path19
(XEN) memory.c:1702:d0v1 XXXIPT do_memory_op default path18
(XEN) memory.c:1702:d0v1 XXXIPT do_memory_op default path19
(XEN) memory.c:1702:d0v1 XXXIPT do_memory_op default path18
(XEN) memory.c:1702:d0v1 XXXIPT do_memory_op default path19
(XEN) memory.c:1702:d0v1 XXXIPT do_memory_op default path13
XENMEM_acquire_resource
is defined as 28. So it seems that the appropriate operation is never requested.
Interestingly, the number of debug messages doesn't grow as I run the proctrace
tool against the domain (dom1 in this example).
Any tips/help on this is appreciated!
from kernel-fuzzer-for-xen-project.
Meanwhile can you tell me which kernel modules I should see in dom0?
The ones you have look fine, really just the default list, no special kernel module is needed.
XENMEM_acquire_resource is defined as 28. So it seems that the appropriate operation is never requested.
That's certainly a weird observation. That would mean the map resource from the dom0 userpsace never actually reaches Xen O.o
from kernel-fuzzer-for-xen-project.
I don't have a spare machine that supports PT unfortunately :(
Can you give me some high level description/pointers about how memory operations should reach Xen, so I can narrow down where the signal gets lost?
from kernel-fuzzer-for-xen-project.
Related Issues (20)
- Report crash when tracing fails HOT 1
- Fail to fork xen domain when running on AMD processor HOT 12
- [FAQ request] How my target runs exactly? What rules are there for harnessing? HOT 3
- Scaling HOT 7
- Paging mode not set during harness setup HOT 3
- Saving and restoring setup state: Device model spawn failed HOT 5
- Performance monitoring HOT 2
- Network setting problem HOT 2
- Unable to request new process from fork server (OOM?) HOT 3
- Error installing debian iso HOT 2
- When installing debian every time I am unable to connect to a network mirror and access wifi from within the vm HOT 22
- Too early input file read HOT 1
- Handling 0 length inputs (and/or improve IPC) HOT 2
- AFL++ communication error - Pipe read returns 0 HOT 7
- Error building Xen HOT 2
- Reopen file for new input HOT 2
- VM Network issue HOT 1
- VM network settings failed HOT 6
- Unable to request new process from fork server (OOM?) HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kernel-fuzzer-for-xen-project.