tanelpoder/0xtools

0x.Tools: X-Ray vision for Linux systems

0x.Tools: X-Ray vision for Linux systems

0x.tools is a set of open-source utilities for analyzing application performance on Linux. It has a goal of deployment simplicity and minimal dependencies, to reduce friction of systematic troubleshooting. There’s no need to upgrade the OS, install kernel modules, heavy monitoring frameworks, Java agents or databases. Some of these tools also work on over-decade-old Linux kernels, like version 2.6.18 from 18 years ago.

0x.tools allow you to measure individual thread level activity, like thread sleep states, currently executing system calls and kernel wait locations. Additionally, you can drill down into CPU usage of any thread or the system as a whole. You can be systematic in your troubleshooting - no need for guessing or genius wizard tricks with traditional system utilization stats.

xcapture-bpf and xtop v2.0.2 announced! (2024-07-03)

xcapture-bpf (and xtop) are like the Linux top tool, but extended with x-ray vision and ability to view your performance data from any chosen angle (that eBPF allows to instrument). You can use it for system level overview and drill down into indivual threads' activity and soon even into individual kernel events like lock waits or memory stalls. eBPF is not only customizable, it's completely programmable and I plan to take full advantage of it. I have so far implemented less than 5% of everything this method and the new tool is capable of, stay tuned for more!

• https://0x.tools

xcapture-bpf demo

This is one of the things that you get:

xcapture-bpf screenshot

A screenshot that illustrates how xcapture-bpf output and stacktiles work with terminal search/highlighting and scroll-back ability:

xcapture-bpf install instructions and info

• Go to https://0x.tools for more info and the installation instructions of the latest eBPF-based tool

Other tools

An example of one of the tools psn (that doesn't use eBPF, just reads the usual /proc files) is here:

$ sudo psn -p "mysqld|kwork" -G syscall,wchan

Linux Process Snapper v0.14 by Tanel Poder [https://0x.tools]
Sampling /proc/syscall, stat, wchan for 5 seconds... finished.


=== Active Threads ========================================================================================

 samples | avg_threads | comm          | state                  | syscall   | wchan                        
-----------------------------------------------------------------------------------------------------------
      25 |        3.12 | (mysqld)      | Disk (Uninterruptible) | fsync     | _xfs_log_force_lsn
      16 |        2.00 | (mysqld)      | Running (ON CPU)       | [running] | 0                            
      14 |        1.75 | (mysqld)      | Disk (Uninterruptible) | pwrite64  | call_rwsem_down_write_failed
       8 |        1.00 | (mysqld)      | Disk (Uninterruptible) | fsync     | submit_bio_wait              
       4 |        0.50 | (mysqld)      | Disk (Uninterruptible) | pread64   | io_schedule                  
       4 |        0.50 | (mysqld)      | Disk (Uninterruptible) | pwrite64  | io_schedule                  
       3 |        0.38 | (mysqld)      | Disk (Uninterruptible) | pread64   | 0                            
       3 |        0.38 | (mysqld)      | Running (ON CPU)       | [running] | io_schedule                  
       3 |        0.38 | (mysqld)      | Running (ON CPU)       | pread64   | 0                            
       2 |        0.25 | (mysqld)      | Disk (Uninterruptible) | [running] | 0                            
       1 |        0.12 | (kworker/*:*) | Running (ON CPU)       | read      | worker_thread                
       1 |        0.12 | (mysqld)      | Disk (Uninterruptible) | fsync     | io_schedule                  
       1 |        0.12 | (mysqld)      | Disk (Uninterruptible) | futex     | call_rwsem_down_write_failed 
       1 |        0.12 | (mysqld)      | Disk (Uninterruptible) | poll      | 0                            
       1 |        0.12 | (mysqld)      | Disk (Uninterruptible) | pwrite64  | _xfs_log_force_lsn           
       1 |        0.12 | (mysqld)      | Running (ON CPU)       | fsync     | submit_bio_wait              
       1 |        0.12 | (mysqld)      | Running (ON CPU)       | futex     | futex_wait_queue_me

Usage info and more details here:

• https://0x.tools

Twitter:

• https://twitter.com/0xtools

Author:

• https://tanelpoder.com

0xtools's Issues

Fix missing newlines when formatting python3

Currently python3 gives this (python2 has some newlines for readability):

$ sudo psn -r -p 111525 -a -G syscall,wchan
Linux Process Snapper v1.1.0 by Tanel Poder [https://0x.tools]
Sampling /proc/stat, syscall, wchan for 5 seconds...
finished.
=== Active Threads ===========================================================================
 samples | avg_threads | comm   | state                 | syscall         | wchan             
----------------------------------------------------------------------------------------------
     100 |        1.00 | (perl) | (zombie)              | [kernel_thread] | 0                 
     100 |        1.00 | (perl) | Sleep (Interruptible) | nanosleep       | hrtimer_nanosleep 
samples: 100
(expected: 100)
total processes: 2, threads: 2
runtime: 5.00, measure time: 0.13

Python2:

$ sudo psn -r -p 111525 -a -G syscall,wchan

Linux Process Snapper v1.1.0 by Tanel Poder [https://0x.tools]
Sampling /proc/stat, syscall, wchan for 5 seconds... finished.


=== Active Threads ===========================================================================

 samples | avg_threads | comm   | state                 | syscall         | wchan             
----------------------------------------------------------------------------------------------
     100 |        1.00 | (perl) | (zombie)              | [kernel_thread] | 0                 
     100 |        1.00 | (perl) | Sleep (Interruptible) | nanosleep       | hrtimer_nanosleep 


samples: 100 (expected: 100)
total processes: 2, threads: 2
runtime: 5.00, measure time: 0.13

missing proc

hi, ive used this utility before, many times -its great -thanks

i downloaded the current version, getting this message - im sure it somthing stupid imdoing or not, but cant figure it out

/tmp/psnapper-master/bin# ./psn
Traceback (most recent call last):
File "./psn", line 31, in
import proc, psnreport
ImportError: No module named proc
root@linux4:/tmp/psnapper-master/bin#

Improve schedlat to report thread/task sched latency

Currently schedlat shows only the process (thread group leader, PID) latency.

Use /proc/PID/task/TID/schedstat instead, to allow reporting a single thread's latency

add command line option to sample processes of a user

cmdline

Hi Tanel,
Is the cmdline option supposed to include the full command line of a process? It looks like I am getting the process name (or path to it) instead of /proc/< pid >/cmdline

RHEL 7.9

Pstool snapshot

Add tid field to psn

Using pid,tid for listing PID/thread ID is more consistent naming than pid,task.

Add tid into the output of psn --list too.

psn v1.2.3 release may show wrong syscall names on x86_64 due to looking them up from a wrong/template file

unistd.h files on ARM/aarch64 work somewhat differently from x86_64... Fixed in the repo already:

0b1d25c

Will release 1.2.4 with the fix soon.

Run process by psn itself

It would be great if you are will add opportunity for psn to run some processes itself.

support wrapping/narrower output (especially when kstack is used)

for now the workaround is to run ./psn ... > output.txt and use the text editors left-right scrolling capabilities, if the output with kstack is too wide. Or use a terminal that can scroll horizontally too (the horizontal-scrolling terminal that I use is this: https://github.com/software-jessies-org/jessies/wiki/Terminator)

ubuntu 18 error while running make

make
gcc -I include -Wall -o bin/xcapture src/xcapture.c
src/xcapture.c: In function ‘main’:
src/xcapture.c:392:41: warning: ‘%s’ directive writing up to 255 bytes into a region of size 94 [-Wformat-overflow=]
sprintf(dirpath, "/proc/%s", pde->d_name);
^~
src/xcapture.c:392:17: note: ‘sprintf’ output between 7 and 262 bytes into a destination of size 100
sprintf(dirpath, "/proc/%s", pde->d_name);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/xcapture.c:401:49: warning: ‘%s’ directive writing up to 255 bytes into a region of size 94 [-Wformat-overflow=]
sprintf(dirpath, "/proc/%s/task", pde->d_name);
^~
src/xcapture.c:401:25: note: ‘sprintf’ output between 12 and 267 bytes into a destination of size 100
sprintf(dirpath, "/proc/%s/task", pde->d_name);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
root@sql3:/home/moustafa_ahmed/0xtools# lsb_release
No LSB modules are available.
root@sql3:/home/moustafa_ahmed/0xtools# cat /etc/issue
Ubuntu 18.04.5 LTS \n \l

Enhancement: Addition of checksum to go with a release

This is a really useful suite of tools - watching the development with interest.

A useful enhancement (and a requirement for some users) is to have a checksum (MD5 or SHA) to use as an indicator that the release content is uncorrupted.

Here's an example of this from the utPLSQL project:

• https://github.com/utPLSQL/utPLSQL/releases/tag/v3.1.10

Is this something that could be added to future releases please?

Thanks again,
Mat.

AlmaLinux 8.9.0 Please make sure you have the correct access rights and the repository exists.

Hello,

I am getting following error when I try git clone on Alma Linux 8.9.0

I couldn't make sure if error due to Alma Linux or 0xtools.git

Regards

`git clone [email protected]:tanelpoder/0xtools.git
Cloning into '0xtools'...
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.`

unistd_64.h

psn
Traceback (most recent call last):
File "/usr/bin/psn", line 47, in
import proc, psnreport
File "/usr/lib/0xtools/proc.py", line 331, in
syscall_id_to_name = get_system_call_names()
File "/usr/lib/0xtools/proc.py", line 328, in get_system_call_names
raise Exception('unistd_64.h not found in' + ' or '.join(unistd_64_paths) + '.\n You may need to "yum install kernel-headers" or "apt-get install libc6-dev"\n until this dependency is removed in a newer pSnapper version')
Exception: unistd_64.h not found in/usr/include/asm/unistd_64.h or /usr/include/x86_64-linux-gnu/asm/unistd_64.h or /usr/include/asm-x86_64/unistd.h or /usr/lib/0xtools/syscall_64_5.15.0.h or /usr/lib/0xtools/syscall_64.h.
You may need to "yum install kernel-headers" or "apt-get install libc6-dev"
until this dependency is removed in a newer pSnapper version

I'm on 5.15.0-1033-azure #40~20.04.1-Ubuntu SMP Tue Jan 24 16:06:12 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

after installed libc6-dev, the problem is still happening

sudo apt-get install libc6-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
libc6-dev is already the newest version (2.31-0ubuntu9.9).
0 upgraded, 0 newly installed, 0 to remove and 7 not upgraded.

Homebrew Installation / Package Managers?

Hi

This is a great set of tools but cannot figure out how to install it or use it, could it be possible as a first step for a homebrew installation? Like a homebrew tap for each or all the tools for 0xtools?

e.g.

brew install tanelponder/0xtools/0xtools
==> Fetching tanelponder/0xtools/0xtools
==> Installing 0xtools from tanelponder/0xtools
🍺  /usr/local/Cellar/0xtools: built in 4 seconds
==> Running `brew cleanup 0xtools`...
# xtop, xcapture-bpf and xcapture are available

or

brew install tanelponder/0xtools/xtop
brew install tanelponder/0xtools/xcapture-bpf
brew install tanelponder/0xtools/xcapture

Thanks!

Modify Makefile to better accommodate cross compilation

Thanks for creating this valuable tool.

Can I suggest some minor changes to the Makefile that will make it much easier to cross compile your useful tool?

These will more easily allow people to specify an alternate compiler (CC) and installation destination (PREFIX)

--- Makefile.orig       2022-09-23 12:06:14.876630117 +1000
+++ Makefile    2022-09-23 12:09:03.834063588 +1000
@@ -1,8 +1,8 @@
-CC=gcc
+CC ?= gcc
 PREFIX ?= /usr

 # build
-CFLAGS=-I include -Wall
+CFLAGS ?= -Wall

 # debuginfo included
 CFLAGS_DEBUG=-I include -ggdb -Wall
@@ -11,7 +11,7 @@
 CFLAGS_DEBUG0=-I include -ggdb -O0

 all:
-       $(CC) $(CFLAGS) -o bin/xcapture src/xcapture.c
+       $(CC) $(CFLAGS) -I include -o bin/xcapture src/xcapture.c

 debug:
        $(CC) $(CFLAGS_DEBUG) -o bin/xcapture src/xcapture.c
@@ -20,6 +20,7 @@
        $(CC) $(CFLAGS_DEBUG0) -o bin/xcapture src/xcapture.c

 install:
+       install -m 0755 -d ${PREFIX}/bin
        install -m 0755 bin/xcapture ${PREFIX}/bin/xcapture
        install -m 0755 bin/psn ${PREFIX}/bin/psn
        install -m 0755 bin/schedlat ${PREFIX}/bin/schedlat

Problem building from source on Linux with gcc 13.2.0

$ make
cc -Wall -I include -o bin/xcapture src/xcapture.c
src/xcapture.c: In function ‘main’:
src/xcapture.c:404:28: warning: ‘%03d’ directive writing between 3 and 8 bytes into a region of size 6 [-Wformat-overflow=]
404 | sprintf(usec_buf, "%03d", (int)tmnow.tv_usec/1000); // ms resolution should be ok for infrequent sampling
| ^~~~
src/xcapture.c:404:27: note: directive argument in the range [-2147483, 2147483]
404 | sprintf(usec_buf, "%03d", (int)tmnow.tv_usec/1000); // ms resolution should be ok for infrequent sampling
| ^~~~~~
src/xcapture.c:404:9: note: ‘sprintf’ output between 4 and 9 bytes into a destination of size 6
404 | sprintf(usec_buf, "%03d", (int)tmnow.tv_usec/1000); // ms resolution should be ok for infrequent sampling
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

$ cc --version
cc (Ubuntu 13.2.0-23ubuntu4) 13.2.0
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Looking forward to trying these tools out!

Bundle argparse.py with psnapper

This is needed for older python versions where argparse is not bundled by default.

Apparenlty it's tested even on python 2.3 so can be safely bundled for convenience.

https://pypi.org/project/argparse/

Not working on Fedora 35

Hi,

I'm getting the following error while trying to use any of the tools:

[root@fedora ~]# git clone https://github.com/tanelpoder/0xtools
Cloning into '0xtools'...
remote: Enumerating objects: 293, done.
remote: Counting objects: 100% (210/210), done.
remote: Compressing objects: 100% (125/125), done.
remote: Total 293 (delta 113), reused 157 (delta 73), pack-reused 83
Receiving objects: 100% (293/293), 133.78 KiB | 1.21 MiB/s, done.
Resolving deltas: 100% (154/154), done.

[root@fedora 0xtools]# make
gcc -I include -Wall -o bin/xcapture src/xcapture.c
[root@fedora 0xtools]# make install 
install -m 0755 bin/xcapture /usr/bin/xcapture
install -m 0755 bin/psn /usr/bin/psn
install -m 0755 bin/schedlat /usr/bin/schedlat
install -m 0755 -d /usr/lib/0xtools
install -m 0644 lib/0xtools/proc.py /usr/lib/0xtools/proc.py
install -m 0644 lib/0xtools/psnreport.py /usr/lib/0xtools/psnreport.py
install -m 0644 lib/0xtools/argparse.py /usr/lib/0xtools/argparse.py

[root@fedora ~]# sudo psn -p "mysqld|kwork" -G syscall,wchan
Traceback (most recent call last):
  File "/usr/bin/psn", line 113, in <module>
    args.sources = [s for s in proc.all_sources if s.name in args.sources.split(',')]
AttributeError: module 'proc' has no attribute 'all_sources'

[root@fedora ~]# uname -a
Linux fedora 5.15.12-200.fc35.x86_64 #1 SMP Wed Dec 29 15:03:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
[root@fedora ~]# which python
/usr/bin/python
[root@fedora ~]# python --version
Python 3.10.1

It's better to replace % to format

https://github.com/tanelpoder/psnapper/blob/1682a1acccf98448805b4ad08686c524d742488c/proc.py#L368

For example:
return ['{} {}'.format(depth, stack_line) for depth, stack_line in enumerate(fh.readlines()[::-1])]

bundle unistd.h with psn

This way no need to install or copy this file even on minimal Linux installs that don't have it. it's not completely trivial as different platforms (and different major kernel versions apparently) the syscall number<->name mappings are different. So might be easy to just bundle x86_64's mappings for "recent" kernels - and still use the separate unistd.h file if it's there

unistd_64.h not found

I am running this image in VirtualBox. Runs fine on another machine with VirtualBox but Ubuntu image is 4.15.0-47-generic there and not 46. Python versions are same on both but have not confirmed if same packages available on both.

Any ideas?

epost@arcshellvm:/media/sf_temp/arcshell/global/bin$ python --version
Python 2.7.15rc1

epost@arcshellvm:/media/sf_temp/arcshell/global/bin$ uname -a
Linux arcshellvm 4.15.0-46-generic #49-Ubuntu SMP Wed Feb 6 09:33:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

epost@arcshellvm:/media/sf_temp/arcshell/global/bin$ psn
Traceback (most recent call last):
  File "/media/sf_temp/arcshell/global/bin/psn", line 25, in <module>
    import proc, report
  File "/media/sf_temp/arcshell/global/bin/proc.py", line 276, in <module>
    syscall_id_to_name = get_system_call_names()
  File "/media/sf_temp/arcshell/global/bin/proc.py", line 273, in get_system_call_names
    raise 'unistd_64.h not found'
TypeError: exceptions must be old-style classes or derived from BaseException, not str

KeyError getting syscall_id

$ sudo ./bin/psn -G syscall,wchan

Linux Process Snapper v1.1.0 by Tanel Poder [https://0x.tools]
Sampling /proc/syscall, wchan, stat for 5 seconds...
Traceback (most recent call last):
  File "./bin/psn", line 375, in <module>
    main()
  File "./bin/psn", line 301, in main
    task_samples = [s.sample(event_time, pid, task) for s in sources.keys() if s.task_level == True]
  File "/home/delijn/0xtools/lib/0xtools/proc.py", line 108, in sample
    return [create_row_sample(rs) for rs in raw_samples]
  File "/home/delijn/0xtools/lib/0xtools/proc.py", line 104, in create_row_sample
    r =  [event_time, pid, task] + [convert(full_sample[idx]) for idx, convert in self.schema_extract]
  File "/home/delijn/0xtools/lib/0xtools/proc.py", line 369, in <lambda>
    ('syscall',    str,  0, lambda sn: syscall_id_to_name[sn]),  # convert syscall_id via unistd_64.h into call name
KeyError: '1879048192'

bpf_attach_raw_tracepoint (sched_switch): Invalid argument on ARM64

[root@OEL89 bin]# uname -r
5.15.0-202.135.2.el8uek.aarch64

[root@OEL89 bin]# ./xtop
Loading BPF...
bpf_attach_raw_tracepoint (sched_switch): Invalid argument
Traceback (most recent call last):
File "/tmp/0xtools/0xtools-master/bin/xcapture-bpf", line 335, in
b = BPF(text=bpf_text)
File "/usr/lib/python3.6/site-packages/bcc/init.py", line 484, in init
self._trace_autoload()
File "/usr/lib/python3.6/site-packages/bcc/init.py", line 1468, in _trace_autoload
self.attach_raw_tracepoint(tp=tp, fn_name=fn.name)
File "/usr/lib/python3.6/site-packages/bcc/init.py", line 1061, in attach_raw_tracepoint
raise Exception("Failed to attach BPF to raw tracepoint")
Exception: Failed to attach BPF to raw tracepoint

Can't build BPF program on Fedora 40 and Debian 13 (Linux 6.10)

Just a heads-up, since someone else might run into this:

=== [0x.tools] xcapture-bpf 2.0.3 BETA by Tanel Poder. Fedora Linux 40 6.10.6 x86_64
===  Loading BPF...
In file included from /virtual/main.c:31:
In file included from include/uapi/linux/ptrace.h:183:
In file included from arch/x86/include/asm/ptrace.h:175:
In file included from arch/x86/include/asm/paravirt_types.h:12:
In file included from arch/x86/include/asm/nospec-branch.h:15:
arch/x86/include/asm/current.h:47:10: warning: multiple identical address spaces specified for type [-Wduplicate-decl-specifier]
   47 |                 return this_cpu_read_const(const_pcpu_hot.current_task);
      |                        ^
arch/x86/include/asm/percpu.h:456:34: note: expanded from macro 'this_cpu_read_const'
  456 | #define this_cpu_read_const(pcp)        __raw_cpu_read(, pcp)
      |                                         ^
arch/x86/include/asm/percpu.h:426:30: note: expanded from macro '__raw_cpu_read'
  426 |         *(qual __my_cpu_type(pcp) *)__my_cpu_ptr(&(pcp));               \
      |                                     ^
arch/x86/include/asm/percpu.h:93:28: note: expanded from macro '__my_cpu_ptr'
   93 | #define __my_cpu_ptr(ptr)       (__my_cpu_type(*(ptr))*)(__force uintptr_t)(ptr)
      |                                  ^
arch/x86/include/asm/percpu.h:92:40: note: expanded from macro '__my_cpu_type'
   92 | #define __my_cpu_type(var)      typeof(var) __percpu_seg_override
      |                                             ^
arch/x86/include/asm/percpu.h:45:31: note: expanded from macro '__percpu_seg_override'
   45 | #define __percpu_seg_override   __seg_gs
      |                                 ^
<built-in>:349:33: note: expanded from macro '__seg_gs'
  349 | #define __seg_gs __attribute__((address_space(256)))
      |                                 ^
In file included from /virtual/main.c:31:
In file included from include/uapi/linux/ptrace.h:183:
In file included from arch/x86/include/asm/ptrace.h:175:
In file included from arch/x86/include/asm/paravirt_types.h:12:
In file included from arch/x86/include/asm/nospec-branch.h:15:
arch/x86/include/asm/current.h:47:10: warning: multiple identical address spaces specified for type [-Wduplicate-decl-specifier]
arch/x86/include/asm/percpu.h:456:34: note: expanded from macro 'this_cpu_read_const'
  456 | #define this_cpu_read_const(pcp)        __raw_cpu_read(, pcp)
      |                                         ^
arch/x86/include/asm/percpu.h:426:9: note: expanded from macro '__raw_cpu_read'
  426 |         *(qual __my_cpu_type(pcp) *)__my_cpu_ptr(&(pcp));               \
      |                ^
arch/x86/include/asm/percpu.h:92:40: note: expanded from macro '__my_cpu_type'
   92 | #define __my_cpu_type(var)      typeof(var) __percpu_seg_override
      |                                             ^
arch/x86/include/asm/percpu.h:45:31: note: expanded from macro '__percpu_seg_override'
   45 | #define __percpu_seg_override   __seg_gs
      |                                 ^
<built-in>:349:33: note: expanded from macro '__seg_gs'
  349 | #define __seg_gs __attribute__((address_space(256)))
      |                                 ^
In file included from /virtual/main.c:32:
In file included from include/linux/sched.h:13:
arch/x86/include/asm/processor.h:543:10: warning: multiple identical address spaces specified for type [-Wduplicate-decl-specifier]
  543 |                 return this_cpu_read_const(const_pcpu_hot.top_of_stack);
      |                        ^
arch/x86/include/asm/percpu.h:456:34: note: expanded from macro 'this_cpu_read_const'
  456 | #define this_cpu_read_const(pcp)        __raw_cpu_read(, pcp)
      |                                         ^
arch/x86/include/asm/percpu.h:426:30: note: expanded from macro '__raw_cpu_read'
  426 |         *(qual __my_cpu_type(pcp) *)__my_cpu_ptr(&(pcp));               \
      |                                     ^
arch/x86/include/asm/percpu.h:93:28: note: expanded from macro '__my_cpu_ptr'
   93 | #define __my_cpu_ptr(ptr)       (__my_cpu_type(*(ptr))*)(__force uintptr_t)(ptr)
      |                                  ^
arch/x86/include/asm/percpu.h:92:40: note: expanded from macro '__my_cpu_type'
   92 | #define __my_cpu_type(var)      typeof(var) __percpu_seg_override
      |                                             ^
arch/x86/include/asm/percpu.h:45:31: note: expanded from macro '__percpu_seg_override'
   45 | #define __percpu_seg_override   __seg_gs
      |                                 ^
<built-in>:349:33: note: expanded from macro '__seg_gs'
  349 | #define __seg_gs __attribute__((address_space(256)))
      |                                 ^
In file included from /virtual/main.c:32:
In file included from include/linux/sched.h:13:
arch/x86/include/asm/processor.h:543:10: warning: multiple identical address spaces specified for type [-Wduplicate-decl-specifier]
arch/x86/include/asm/percpu.h:456:34: note: expanded from macro 'this_cpu_read_const'
  456 | #define this_cpu_read_const(pcp)        __raw_cpu_read(, pcp)
      |                                         ^
arch/x86/include/asm/percpu.h:426:9: note: expanded from macro '__raw_cpu_read'
  426 |         *(qual __my_cpu_type(pcp) *)__my_cpu_ptr(&(pcp));               \
      |                ^
arch/x86/include/asm/percpu.h:92:40: note: expanded from macro '__my_cpu_type'
   92 | #define __my_cpu_type(var)      typeof(var) __percpu_seg_override
      |                                             ^
arch/x86/include/asm/percpu.h:45:31: note: expanded from macro '__percpu_seg_override'
   45 | #define __percpu_seg_override   __seg_gs
      |                                 ^
<built-in>:349:33: note: expanded from macro '__seg_gs'
  349 | #define __seg_gs __attribute__((address_space(256)))
      |                                 ^
In file included from /virtual/main.c:34:
In file included from include/linux/syscalls.h:93:
In file included from include/trace/syscall.h:7:
In file included from include/linux/trace_events.h:10:
In file included from include/linux/perf_event.h:62:
In file included from include/linux/security.h:35:
include/linux/bpf.h:348:10: error: invalid application of 'sizeof' to an incomplete type 'struct bpf_wq'
  348 |                 return sizeof(struct bpf_wq);
      |                        ^     ~~~~~~~~~~~~~~~
include/linux/bpf.h:348:24: note: forward declaration of 'struct bpf_wq'
  348 |                 return sizeof(struct bpf_wq);
      |                                      ^
include/linux/bpf.h:377:10: error: invalid application of '__alignof' to an incomplete type 'struct bpf_wq'
  377 |                 return __alignof__(struct bpf_wq);
      |                        ^          ~~~~~~~~~~~~~~~
include/linux/bpf.h:377:29: note: forward declaration of 'struct bpf_wq'
  377 |                 return __alignof__(struct bpf_wq);
      |                                           ^
4 warnings and 2 errors generated.
Traceback (most recent call last):
  File "/home/grayshade/Projects/0xtools/bin/xcapture-bpf", line 474, in <module>
    b = BPF(text= ifdef + bpf_text)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/bcc/__init__.py", line 479, in __init__
    raise Exception("Failed to compile BPF module %s" % (src_file or "<text>"))
Exception: Failed to compile BPF module <text>

You can ignore the warnings (biolatency shows them too), but I don't understand the problem with the missing struct. It's defined in /usr/src/kernels/6.10.6-200.fc40.x86_64/include/uapi/linux/bpf.h:

struct bpf_wq {                                                                                                                                                                                                                       
  __u64 __opaque[2];                                                                                                                                                                                                                  
} __attribute__((aligned(8)));                                                                                                                                                                                                        

Which /usr/src/kernels/6.10.6-200.fc40.x86_64/include/linux/bpf.h includes:

#include <uapi/linux/bpf.h>

// ...

static inline u32 btf_field_type_size(enum btf_field_type type)
{
        switch (type) {
        case BPF_SPIN_LOCK:
                return sizeof(struct bpf_spin_lock);
        case BPF_TIMER:
                return sizeof(struct bpf_timer);
        case BPF_WORKQUEUE:
                return sizeof(struct bpf_wq);
        case BPF_KPTR_UNREF:
        case BPF_KPTR_REF:
        case BPF_KPTR_PERCPU:
                return sizeof(u64);
        case BPF_LIST_HEAD:
                return sizeof(struct bpf_list_head);
        case BPF_LIST_NODE:
                return sizeof(struct bpf_list_node);
        case BPF_RB_ROOT:
                return sizeof(struct bpf_rb_root);
        case BPF_RB_NODE:
                return sizeof(struct bpf_rb_node);
        case BPF_REFCOUNT:
                return sizeof(struct bpf_refcount);
        default:
                WARN_ON_ONCE(1);
                return 0;
        }
}

---

TL;DR: this fails to build on 6.10:

from bcc import BPF

bpf_text = """
#include <uapi/linux/bpf.h>
#include <linux/syscalls.h>
"""
BPF(text=bpf_text)

add epoll_pwait() to the list of single-fd syscalls

so that psn would print out the filename used

make psn-embedded sqlite use a /tmp file instead of an in-memory DB

Currently sqlite uses only memory to keep track of thread samples.

This could take a noticeable amount of memory, especially when sampling all thread states, at high frequency and for a long time.

When initializing a sqlite DB/connection, configure it to use a /tmp file instead of RAM. We could make it conditional, a'la if psn is executed to sample for more than N seconds (60 seconds?), then use a file-backed sqlite DB, otherwise RAM.

KeyError on Rocky 9 when running psn

reproduce steps

yum install -y git make gcc python procps
git clone https://github.com/tanelpoder/0xtools
cd 0xtools/
make && make install

[root@tc-tikv-0 0xtools]# psn -p 1 -G syscall,wchan,filename
Linux Process Snapper v1.2.3 by Tanel Poder [https://0x.tools]
Sampling /proc/stat, syscall, wchan for 5 seconds...
Traceback (most recent call last):
  File "/usr/bin/psn", line 375, in <module>
    main()
  File "/usr/bin/psn", line 301, in main
    task_samples = [s.sample(event_time, pid, task) for s in sources.keys() if s.task_level == True]
  File "/usr/bin/psn", line 301, in <listcomp>
    task_samples = [s.sample(event_time, pid, task) for s in sources.keys() if s.task_level == True]
  File "/usr/lib/0xtools/psnproc.py", line 109, in sample
    return [create_row_sample(rs) for rs in raw_samples]
  File "/usr/lib/0xtools/psnproc.py", line 109, in <listcomp>
    return [create_row_sample(rs) for rs in raw_samples]
  File "/usr/lib/0xtools/psnproc.py", line 105, in create_row_sample
    r =  [event_time, pid, task] + [convert(full_sample[idx]) for idx, convert in self.schema_extract]
  File "/usr/lib/0xtools/psnproc.py", line 105, in <listcomp>
    r =  [event_time, pid, task] + [convert(full_sample[idx]) for idx, convert in self.schema_extract]
  File "/usr/lib/0xtools/psnproc.py", line 383, in <lambda>
    ('syscall',    str,  0, lambda sn: syscall_id_to_name[sn]),  # convert syscall_id via unistd_64.h into call name
KeyError: '45'

[root@tc-tikv-0 0xtools]# cat /etc/redhat-release 
Rocky Linux release 9.1 (Blue Onyx)