cvise does not work on OpenBSD 6.9 on PowerPC G5 Mac

[marxin]

cvise does not work on OpenBSD 6.9 on PowerPC G5 Mac

[marxin]

Hello.

Thank you for the interesting report. It seems to me that it's one of the following psutil calls:

psutil.cpu_count(logical=False)
psutil.cpu_count(logical=True)
psutil.Process().cpu_affinity()

What version of psutil do you have? Can you please run these individual calls in terminal?

[seanm]

Thanks for your reply. Here you go:

builder3$ python3 
Python 3.8.8 (default, Apr 19 2021, 14:29:36) 
[Clang 10.0.1 ] on openbsd6
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> psutil.version_info
(5, 4, 3)
>>> psutil.cpu_count(logical=False)
>>> psutil.cpu_count(logical=True)
2
>>> psutil.Process().cpu_affinity()
Segmentation fault (core dumped) 
builder3$ 

[marxin]

All right, so we know the culprit method. The method is supported by FreeBSD:
https://psutil.readthedocs.io/en/latest/index.html?highlight=cpu_affinity#psutil.Process.cpu_affinity

It's a C implementation:
https://github.com/giampaolo/psutil/blob/master/psutil/arch/freebsd/specific.c#L810

Can you please create an issue at:
https://github.com/giampaolo/psutil/issues

[seanm]

It looks like it's not supported on OpenBSD according to the comments here: https://github.com/giampaolo/psutil/blob/master/psutil/_psutil_bsd.c

So probably this would happen on Intel too, and the old PowerPC G5 is not the issue.

A seg fault seems an odd way to not support something, as opposed to just doing nothing.

Here's a ticket for psutil: giampaolo/psutil#2022

Do you think commenting out the affinity usage in cvise would work for me? Guess I'll try...

[marxin]

It looks like it's not supported on OpenBSD according to the comments here: https://github.com/giampaolo/psutil/blob/master/psutil/_psutil_bsd.c

I see!

So probably this would happen on Intel too, and the old PowerPC G5 is not the issue.

A seg fault seems an odd way to not support something, as opposed to just doing nothing.

Sure! It should not crash.

Here's a ticket for psutil: giampaolo/psutil#2022

Do you think commenting out the affinity usage in cvise would work for me? Guess I'll try...

Yeah, it's not important, it's part of the automatic detection of CPU count. Just comment it out and let me know.

[seanm]

Just comment it out and let me know

builder3$ /home/builder/external/cvise-install/bin/cvise --version 
cvise 2.4.0 (ad3e554)

Success! Now I'll see if it can do real work...

[seanm]

So I hit another issue:

builder3$ /home/builder/external/cvise-install/bin/cvise ./test.sh small.c                                                                                                                 
C-Vise cannot run because the interestingness test does not return
zero. Please ensure that it does so not only in the directory where
you are invoking C-Vise, but also in an arbitrary temporary
directory containing only the files that are being reduced. In other
words, running these commands:

  DIR=`mktemp -d`
  cp /home/builder/external/clang52438/small.c $DIR
  cd $DIR
  /home/builder/external/clang52438/test.sh
  echo $?

should result in '0' being echoed to the terminal.
Please ensure that the test script takes no arguments; it should be hard-coded to refer
to the same file that is passed as an argument to C-Vise.

See 'cvise.py --help' for more information.

Then I tried the commands it says:

builder3$ DIR=`mktemp -d`
builder3$ cp /home/builder/external/clang52438/small.c $DIR
builder3$ cd $DIR
builder3$ pwd
/tmp/tmp.Lp6WHErYWE
builder3$ /home/builder/external/clang52438/test.sh
ksh: /home/builder/external/clang52438/test.sh: No such file or directory

yet the files does exist:

builder3$ file /home/builder/external/clang52438/test.sh
/home/builder/external/clang52438/test.sh: Bourne-Again shell script text executable

But if I run it this way:

builder3$ sh /home/builder/external/clang52438/test.sh
builder3$ echo $?
0

I think this may be a bash vs other shell thing...

[marxin]

builder3$ /home/builder/external/clang52438/test.sh
ksh: /home/builder/external/clang52438/test.sh: No such file or directory

This is super weird, why does it say that the file does not exist? Can you please switch to bash shell for instance in order to run the generated .sh file. What about running ksh /home/builder/external/clang52438/test.sh?

[seanm]

I switched to bash and that gets me farther, the explanatory example steps now work. And invoking cvise no longer complains that the interestingness script doesn't return 0. But now...

builder3$ cat test.sh                                                                                                                                                                      
#!/usr/bin/env bash

grep goto small.c >/dev/null 2>&1

builder3$ cat small.c                                                                                                                                                                      
int main (void)
{
int x;
double y;
goto exit;

exit:

return 0;
}


builder3$ /home/builder/external/cvise-install/bin/cvise ./test.sh small.c
00:00:00 INFO ===< 26136 >===
00:00:00 INFO running 2 interestingness tests in parallel
00:00:00 INFO INITIAL PASSES
00:00:00 INFO ===< IncludesPass >===

I left it overnight, and it doesn't output anything more.

[marxin]

Hm, that's weird, I see:

cvise ./test.sh small.c
00:00:00 INFO ===< 341 >===
00:00:00 INFO running 8 interestingness tests in parallel
00:00:00 INFO INITIAL PASSES
00:00:00 INFO ===< IncludesPass >===
00:00:00 INFO ===< UnIfDefPass >===
00:00:00 INFO ===< CommentsPass >===
00:00:00 INFO ===< IfPass >===
00:00:00 INFO ===< LineMarkersPass >===
00:00:00 INFO ===< BlankPass >===
00:00:00 INFO (3.0%, 64 bytes, 8 lines)
00:00:00 INFO ===< ClangBinarySearchPass::replace-function-def-with-decl >===
00:00:00 INFO using C++ standard: c++98 with 1 transformation opportunities
00:00:00 INFO ===< ClangBinarySearchPass::remove-unused-function >===
00:00:01 INFO using C++ standard: c++98 with 0 transformation opportunities
00:00:01 INFO ===< LinesPass::0 >===
00:00:01 INFO ===< LinesPass::1 >===
00:00:01 INFO (51.5%, 32 bytes, 3 lines)
00:00:01 INFO (77.3%, 15 bytes, 2 lines)
00:00:01 INFO (81.8%, 12 bytes, 1 lines)
...

Please try using --debug option.

[seanm]

builder3$ /home/builder/external/cvise-install/bin/cvise --debug ./test.sh small.c
00:00:00 DEBUG perform sanity check... 
00:00:00 DEBUG sanity check tmpdir = /tmp/cvise-sanity-1mhd6wg9
00:00:00 DEBUG sanity check successful
00:00:00 INFO ===< 90522 >===
00:00:00 INFO running 2 interestingness tests in parallel
00:00:00 INFO INITIAL PASSES
00:00:00 DEBUG Creating pass root folder: /tmp/cvise-IncludesPass-_nn33m7e
00:00:00 INFO ===< IncludesPass >===

It's been "only" 30 minutes, but I think it's stuck. top says CPU is 99% idle.

[marxin]

Hmm, hard to guess. Can you attach gdb to the python process and print backtrace, please?
It may be possible that Pebble library (that I use for process dispatching) does not work on BSD.

[seanm]

To eliminate PowerPC being the issue, I've created an OpenBSD 7 VM on Intel and am trying there. I forget if I hit this on PowerPC, but I'm getting this build error:

[  5%] Building CXX object clang_delta/CMakeFiles/clang_delta.dir/TransformationManager.cpp.o
In file included from /home/sean/external/cvise/clang_delta/TransformationManager.cpp:25:
In file included from /usr/local/include/clang/Frontend/CompilerInstance.h:15:
In file included from /usr/local/include/clang/Frontend/CompilerInvocation.h:22:
/usr/local/include/clang/StaticAnalyzer/Core/AnalyzerOptions.h:437:9: error: 'PACKAGE' macro redefined [-Werror,-Wmacro-redefined]
#define PACKAGE(FULLNAME) llvm::StringLiteral(FULLNAME),
        ^
/home/sean/external/cvise-bin/config.h:35:9: note: previous definition is here
#define PACKAGE "cvise"
        ^
1 error generated.

IIRC OpenBSD sometimes defaults to using -Werror... I got past it by specifying -w in my C/C++ flags. But maybe you could fix this warning?

[marxin]

/usr/local/include/clang/StaticAnalyzer/Core/AnalyzerOptions.h:437:9: error: 'PACKAGE' macro redefined [-Werror,-Wmacro-redefined]
#define PACKAGE(FULLNAME) llvm::StringLiteral(FULLNAME),

This comes from LLVM itself, what version do you use? What compiler do you use for the compilation of C-Vise?
You remove the -Werror in CMakeLists.txt.

[seanm]

I'm using the clang that comes with OpenBSD:

$ clang --version
OpenBSD clang version 11.1.0
Target: amd64-unknown-openbsd7.0
Thread model: posix
InstalledDir: /usr/bin

and the LLVM libs that cvise needed to build are from pkg_add llvm (which gives llvm-11.1.0p7).

Yes, AnalyzerOptions.h:437 is from LLVM, but the other define is from cvise/cmake_config.h.in I think. Is it meant to be the same PACKAGE macro, or is this a name collision?

[marxin]

All right, I fixed that on master, please pull.

[seanm]

Cool, thanks!

It may be possible that Pebble library...

In fact, I think I didn't have pebble installed, or I installed it with pkg_add, not pip. I have deleted all packages on the PowerPC and will now try fresh...

[seanm]

Success! I can now reduce a toy example correctly!

For anyone else trying on OpenBSD, here's what worked for me:

doas pkg_add python3 py3-pip llvm bash cmake git
doas ln -sf /usr/local/bin/pip3.8 /usr/local/bin/pip

python3 -m pip install --upgrade psutil
python3 -m pip install --upgrade Pytest
python3 -m pip install --upgrade Pebble
python3 -m pip install --upgrade chardet
python3 -m pip install --upgrade flake8

git clone https://github.com/marxin/cvise.git
mkdir cvise-bin
mkdir cvise-install
cd cvise-bin
ccmake ../cvise
(set install folder, set to build Release because Debug fails to link)
(hack cvise to remove affinity stuff)
make -j2 install

So the only issue remaining is the affinity thing, which is not a cvise bug really.

[marxin]

Good, thank you for your help. Lemme close this.