billfarrow/pcimem

Simple program to read & write to a pci device from userspace

pcimem's Issues

[Not an 'issue'] Doubt regarding PCI resource size

Hi,
Is there any restriction on setting the resource file size (I do it through Vivado).
When I change the size from 128MB to 256MB, lspci -vv stops displaying that resource corresponding to that.
Question posted in Stackoverflow

A way to control request lengths?

Hey, this tool is very nice.

Curious if it can or could be enhanced to support control of the request length.

I painfully discovered that my x86 Ubuntu system was sending MemRd requests formatted as 32 bit addressing, but Length == 2... so requesting 64 bits of data.

My endpoint design (Xilinx PCIE Example Design) only supports 32 bit requests.

So my endpoint just drops the request.

I think for now I am going to have to write my own PCIe User Logic.

Fedora 33 mmap failing

on Fedora 33
lspci
b3:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
Subsystem: Mellanox Technologies ConnectX-4 Lx Stand-up dual-port 10GbE MCX4121A-XCAT
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 37
NUMA node: 0
Region 0: Memory at 3800fe000000 (64-bit, prefetchable) [size=32M]
Expansion ROM at fbe00000 [disabled] [size=1M]
Capabilities:
Kernel driver in use: mlx5_core
Kernel modules: mlx5_core

ls /sys/bus/pci/devices/0000:b3:00.0/ -l
total 0
-r--r--r--. 1 root root 4096 Mar 1 11:41 aer_dev_correctable
-r--r--r--. 1 root root 4096 Mar 1 11:41 aer_dev_fatal
-r--r--r--. 1 root root 4096 Mar 1 11:41 aer_dev_nonfatal
-r--r--r--. 1 root root 4096 Feb 21 09:25 ari_enabled
-rw-r--r--. 1 root root 4096 Mar 1 11:41 broken_parity_status
-r--r--r--. 1 root root 4096 Feb 21 09:25 class
-rw-r--r--. 1 root root 4096 Feb 21 09:25 config
-r--r--r--. 1 root root 4096 Mar 1 11:41 consistent_dma_mask_bits
-r--r--r--. 1 root root 4096 Mar 1 11:41 current_link_speed
-r--r--r--. 1 root root 4096 Mar 1 11:41 current_link_width
-rw-r--r--. 1 root root 4096 Mar 1 11:41 d3cold_allowed
-r--r--r--. 1 root root 4096 Feb 21 09:25 device
-r--r--r--. 1 root root 4096 Mar 1 11:41 dma_mask_bits
lrwxrwxrwx. 1 root root 0 Feb 21 09:25 driver -> ../../../../bus/pci/drivers/mlx5_core
-rw-r--r--. 1 root root 4096 Mar 1 11:41 driver_override
-rw-r--r--. 1 root root 4096 Mar 1 11:41 enable
lrwxrwxrwx. 1 root root 0 Mar 1 11:41 firmware_node -> ../../../LNXSYSTM:00/LNXSYBUS:00/PNP0A08:03/device:b6/device:b9
drwxr-xr-x. 16 root root 0 Feb 21 09:25 infiniband
drwxr-xr-x. 4 root root 0 Feb 21 09:25 infiniband_mad
drwxr-xr-x. 3 root root 0 Feb 21 09:25 infiniband_verbs
-r--r--r--. 1 root root 4096 Mar 1 10:13 irq
drwxr-xr-x. 2 root root 0 Feb 24 14:45 link
-r--r--r--. 1 root root 4096 Mar 1 11:41 local_cpulist
-r--r--r--. 1 root root 4096 Mar 1 11:41 local_cpus
-r--r--r--. 1 root root 4096 Mar 1 11:41 max_link_speed
-r--r--r--. 1 root root 4096 Mar 1 11:41 max_link_width
-r--r--r--. 1 root root 4096 Mar 1 10:13 modalias
-rw-r--r--. 1 root root 4096 Mar 1 11:41 msi_bus
drwxr-xr-x. 2 root root 0 Feb 24 14:45 msi_irqs
drwxr-xr-x. 3 root root 0 Feb 21 09:25 net
-rw-r--r--. 1 root root 4096 Feb 21 09:25 numa_node
-r--r--r--. 1 root root 4096 Mar 1 11:41 pools
drwxr-xr-x. 2 root root 0 Feb 24 14:45 power
drwxr-xr-x. 3 root root 0 Feb 21 09:25 ptp
--w--w----. 1 root root 4096 Mar 1 11:41 remove
--w-------. 1 root root 4096 Mar 1 11:41 rescan
--w-------. 1 root root 4096 Mar 1 11:41 reset
-r--r--r--. 1 root root 4096 Feb 21 09:25 resource
-rw-------. 1 root root 33554432 Mar 1 11:41 resource0
-rw-------. 1 root root 33554432 Mar 1 11:41 resource0_wc
-r--r--r--. 1 root root 4096 Feb 21 09:25 revision
-rw-------. 1 root root 1048576 Feb 21 09:25 rom
-rw-r--r--. 1 root root 4096 Mar 1 11:41 sriov_drivers_autoprobe
-rw-r--r--. 1 root root 4096 Feb 21 09:25 sriov_numvfs
-r--r--r--. 1 root root 4096 Mar 1 11:41 sriov_offset
-r--r--r--. 1 root root 4096 Mar 1 11:41 sriov_stride
-r--r--r--. 1 root root 4096 Feb 21 09:25 sriov_totalvfs
-r--r--r--. 1 root root 4096 Mar 1 11:41 sriov_vf_device
lrwxrwxrwx. 1 root root 0 Feb 21 09:25 subsystem -> ../../../../bus/pci
-r--r--r--. 1 root root 4096 Feb 21 09:25 subsystem_device
-r--r--r--. 1 root root 4096 Feb 21 09:25 subsystem_vendor
-rw-r--r--. 1 root root 4096 Feb 21 09:25 uevent
-r--r--r--. 1 root root 4096 Feb 21 09:25 vendor
-rw-------. 1 root root 0 Mar 1 11:41 vpd

program output
sudo ./pcimem /sys/bus/pci/devices/0000:b3:00.0/resource0_wc 0 w
/sys/bus/pci/devices/0000:b3:00.0/resource0_wc opened.
Target offset is 0x0, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0x0)
Error at line 111, file pcimem.c (22) [Invalid argument]

Any ideas?

Invalid argument

Hi,

We are facing new error using pcimem that we didn't see before:
[BSC:root@15DZX23-A pcimem]$ ./pcimem /sys/devices/pci0000\:0a/0000\:0a\:00.0/resource0 0x0 b 0x44 /sys/devices/pci0000:0a/0000:0a:00.0/resource0 opened. Target offset is 0x0, page size is 4096 mmap(0, 4096, 0x3, 0x1, 3, 0x0) Error at line 111, file pcimem.c (22) [Invalid argument]

Any idea what can be the root cause for that?

[158333.439198] x86/PAT: pcimem:922560 conflicting memory types ca000000-ca800000 uncached-minus<->write-combining

162676.233287] x86/PAT: pcimem:978227 conflicting memory types ca000000-ca800000 uncached-minus<->write-combining
[162676.233289] x86/PAT: memtype_reserve failed [mem 0xca000000-0xca7fffff], track uncached-minus, req uncached-minus

Trying to access BAR0 of a Mellanox ConnectX-5.
Works fine on RHEL 7.7.
On RHEL it appears the BAR is mapped much higher in memory (perhaps the BIOS has the above 4G flag set)?
First time running the same logic on Ubuntu.
Running on Ubuntu produces the error.

more /sys/devices/pci0000:00/0000:00:1b.4/0000:02:00.0/resource

0x00000000ca000000 0x00000000cbffffff 0x000000000014220c
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x00000000f7100000 0x00000000f71fffff 0x0000000000046200
0x00000000ce000000 0x00000000ceffffff 0x000000000014220c
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000

02:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
Subsystem: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at ca000000 (64-bit, prefetchable) [size=32M]
Expansion ROM at f7100000 [disabled] [size=1M]
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [48] Vital Product Data
Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
Capabilities: [c0] Vendor Specific Information: Len=18 Capabilities: [40] Power Management version 3 Capabilities: [100] Advanced Error Reporting Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [180] Single Root I/O Virtualization (SR-IOV) Capabilities: [1c0] Secondary PCI Express Capabilities: [230] Access Control Services Capabilities: [320] Lane Margining at the Receiver
Capabilities: [370] Physical Layer 16.0 GT/s Capabilities: [420] Data Link Feature
Kernel driver in use: mlx5_core

System goes inaccessible after using ./pcimem

Hi,
I tried writing to a location using the following command.
sudo ./pcimem /sys/bus/pci/devices/0000\:0a\:00.0/resource0 0x100 w 0x1234abcd
The machine got stuck and I am not able to ping it now.

What could be the reason?

Extend use cases to /dev/mem

I have issues using dd to access PCI memory via /dev/mem. This tool is a great replacement:
./pcimem /dev/mem/ ...

Docs could be updated to reflect this.

Info : Maximum size of pcimem we can access

Equation : (BAR_SIZE - page_size) - 1

e.g. In my Case,
Page size = 4096
BAR_SIZE = 1 MB
so, (1048576 - 4096 ) - 1 = 1044479