marxin / cvise Goto Github PK
View Code? Open in Web Editor NEWSuper-parallel Python port of the C-Reduce
License: Other
Super-parallel Python port of the C-Reduce
License: Other
I was reducing a pre-processed test case with 8 jobs in parallel and 11 passes encountered a pass error. I have attached a zip archive with the created bug report for each of them.
If you need any other info, let me know.
[File deleted]
Besides installing cvise to site-packages, we need to get the modules byte-compiled. Now, that's a non-trivial task and CMake is horrible at it, so it'd be best to actually use distutils/setuptools for that. Ideally, the package could be split into two, one containing common C/C++ tooling that could shared between cvise and creduce, and the other containing the Python driver.
Why byte-compiling is hard? The exact commands used depend on Python interpreter and version used. There are different commands for Python <3.5, PyPy(2), Python 3 (and PyPy3) >=3.5, Jython… and these are the implementations we used. Even Python upstream recently gets it wrong, and we have to patch Python for it.
The 'fallback' code we have for it:
https://gitweb.gentoo.org/repo/gentoo.git/tree/eclass/python-utils-r1.eclass#n595
I don't think you want to repeat it verbatim in CMakeLists.txt
and maintain it going forward.
When your reproduction fails to reproduce, the following is printed out:
DIR=`mktemp -d`
cp c:\path\to\t.cpp $DIR
cd $DIR
C:\Path\to\checkSuccess.bat c:\path\to\t.cpp
echo $?
However, if your script relies on the fact that c:\path\to\t.cpp
being passed into it, the sanity check fails as this argument isn't being passed. (At least on windows)
Initially I noticed it as core dumps filling the disk at late stages of cvise
run. Crashes usually happen around ClexPass::rm-toks-<N>
phases.
Example crash dump of clex
:
$ file core.580492
core.580492: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from '/usr/libexec/cvise/clex rm-toks-2 10 /tmp/cvise-yohufd05/cvise-7sc5myxt/main.c', real uid: 1000, effective uid: 1000, real gid: 100, effective gid: 100, execfn: '/usr/libexec/cvise/clex', platform: 'x86_64'
$ gdb --quiet /usr/libexec/cvise/clex core.580492
Reading symbols from /usr/libexec/cvise/clex...
Reading symbols from /usr/lib/debug//usr/libexec/cvise/clex.debug...
[New LWP 580492]
Core was generated by `/usr/libexec/cvise/clex rm-toks-2 10 /tmp/cvise-yohufd05/cvise-7sc5myxt/main.c'.
Program terminated with signal SIGABRT, Aborted.
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 return ret;
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007f4461671537 in __GI_abort () at abort.c:79
#2 0x00007f446167140f in __assert_fail_base (fmt=0x7f44617d3078 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=0x559dc86042f2 "in", file=0x559dc86041cc "../cvise-9999/clex/driver.c", line=473, function=<optimized out>)
at assert.c:92
#3 0x00007f44616803c2 in __GI___assert_fail (assertion=0x559dc86042f2 "in",
file=0x559dc86041cc "../cvise-9999/clex/driver.c", line=473, function=0x559dc8604324 <__PRETTY_FUNCTION__.1> "main")
at assert.c:101
#4 0x0000559dc86003e2 in main (argc=<optimized out>, argv=0x7ffe196361c8) at ../cvise-9999/clex/driver.c:473
(gdb) frame 4
#4 0x0000559dc86003e2 in main (argc=<optimized out>, argv=0x7ffe196361c8) at ../cvise-9999/clex/driver.c:473
473 assert(in);
(gdb) list
468 int tok_index;
469 int ret = sscanf(argv[2], "%d", &tok_index);
470 assert(ret == 1);
471 // printf ("file = '%s'\n", argv[3]);
472 FILE *in = fopen(argv[3], "r");
473 assert(in);
474 yyin = in;
475
476 max_toks = initial_length;
477 tok_list = (struct tok_t *)malloc(max_toks * sizeof(struct tok_t));
No graceful file check handling. If I add a bit of debugging backtrace looks like that:
--- /home/slyfox/dev/git/cvise/cvise/passes/clex.py 2020-07-22 22:01:16.000000000 +0100
+++ /usr/share/cvise/passes/clex.py 2020-07-23 23:01:36.000000000 +0100
@@ -2,4 +2,5 @@
import shutil
import tempfile
+import traceback
from cvise.passes.abstract import AbstractPass, PassResult
@@ -19,4 +20,7 @@
def transform(self, test_case, state, process_event_notifier):
+ if not os.path.isfile(test_case):
+ traceback.print_stack()
+
tmp = os.path.dirname(test_case)
with tempfile.NamedTemporaryFile(mode="w", delete=False, dir=tmp) as tmp_file:```
00:02:05 INFO ===< ClexPass::rm-toks-8 >===
File "/usr/bin/cvise", line 265, in <module>
reducer.reduce(pass_group, skip_initial=args.skip_initial_passes)
File "/usr/share/cvise/cvise.py", line 125, in reduce
self._run_main_passes(pass_group["main"])
File "/usr/share/cvise/cvise.py", line 155, in _run_main_passes
self.test_manager.run_pass(p)
File "/usr/share/cvise/utils/testing.py", line 474, in run_pass
success_env = self.run_parallel_tests()
File "/usr/share/cvise/utils/testing.py", line 409, in run_parallel_tests
future = pool.schedule(test_env.run, timeout=self.timeout)
File "/usr/lib/python3.7/site-packages/pebble/pool/process.py", line 90, in schedule
self._check_pool_state()
File "/usr/lib/python3.7/site-packages/pebble/pool/base_pool.py", line 91, in _check_pool_state
self._update_pool_state()
File "/usr/lib/python3.7/site-packages/pebble/pool/base_pool.py", line 100, in _update_pool_state
self._start_pool()
File "/usr/lib/python3.7/site-packages/pebble/pool/process.py", line 63, in _start_pool
self._pool_manager.start()
File "/usr/lib/python3.7/site-packages/pebble/pool/process.py", line 192, in start
self.worker_manager.create_workers()
File "/usr/lib/python3.7/site-packages/pebble/pool/process.py", line 364, in create_workers
self.new_worker()
File "/usr/lib/python3.7/site-packages/pebble/pool/process.py", line 377, in new_worker
None, worker_process, True, self.worker_parameters, self.workers_channel)
File "/usr/lib/python3.7/site-packages/pebble/common.py", line 141, in launch_process
process.start()
File "/usr/lib/python3.7/multiprocessing/process.py", line 112, in start
self._popen = self._Popen(self)
File "/usr/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/usr/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
return Popen(process_obj)
File "/usr/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
self._launch(process_obj)
File "/usr/lib/python3.7/multiprocessing/popen_fork.py", line 74, in _launch
code = process_obj._bootstrap()
File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.7/site-packages/pebble/pool/process.py", line 407, in worker_process
payload.function, *payload.args, **payload.kwargs)
File "/usr/lib/python3.7/site-packages/pebble/common.py", line 174, in process_execute
return function(*args, **kwargs)
File "/usr/share/cvise/utils/testing.py", line 103, in run
ProcessEventNotifier(self.pid_queue))
File "/usr/share/cvise/passes/clex.py", line 23, in transform
traceback.print_stack()
As if occasionally temp dir was cleaned before worker processes finished execution.
I see in reductions that code like
class test : private bar
{
public:
statement1;
statement2;
};
will often be collapsed to
class test : private bar
{
public: statement1;
statement2;
};
The problem is that the statement1 line could be removed, but will trigger a build error because statement2 isn't accessible any more.
I would expect that all access specifier will be removed at a pretty early stage as they most often shouldn't matter.
That means class
should be replaced by struct
and any access specifier should be removed from a class.
/*class*/ struct test : /*private*/ public bar
{
/*public:*/ statement1;
statement2;
};
For the following file:
struct a;
template <typename> using b = a;
struct a {
template <typename c> auto d(c) {
using e = b<c>;
return e{};
}
} f;
auto g() -> a {
a h;
f.d([] {}).d(g());
return h;
}
and arguments:
clang_delta --transformation=simple-inliner --counter=1 unit.cpp
clang_delta
crashes as follows:
clang_delta: ../clang_delta/SimpleInliner.cpp:283: virtual void SimpleInliner::HandleTranslationUnit(clang::ASTContext&): Assertion `TheStmt && "NULL TheStmt!"' failed.
I just built but did not install cvise
, so tired running it from the build location, but this failed with cvise
being unable to find the tools for doing the passes.
I solved this by doing this:
export PATH=~/cvise/build/clex/:~/cvise/build/clang_delta:~/cvise/build:~/cvise/build/cvise:~/creduce/unifdef:~/creduce/delta/:$PATH
which also requires unifdef
and topflatformat
, which do not appear to be part of a normal cvise
build (but they are part of a creduce
build!).
I guess it needs to be documented that, if you want to run a non-installed build of cvise
, you need to set $PATH
, and that you need to install unifdef
and topflatformat
"from somewhere else".
unifdef
(e.g.,) is listed in the Fedora Dockerimage, but I couldn't find this documented anywhere else.
When running CVise without having all requirements, we get a message like 00:02:02 ERROR Skipping pass ClangPass::remove-namespace
however, this doesn't mention what the underlying problem is. As a result, it is impossible to resolve this.
I just added something like:
mv unit.cpp test.cpp
to my interestingness test, where unit.cpp
is the input name as given to cvise
.
However, this then led to:
00:00:03 DEBUG ***ADVANCE*** from 31 to 32 with chunk 1
Traceback (most recent call last):
File "/usr/bin/cvise", line 286, in <module>
reducer.reduce(pass_group, skip_initial=args.skip_initial_passes)
File "/usr/share/cvise/cvise.py", line 143, in reduce
self._run_additional_passes(pass_group['first'])
File "/usr/share/cvise/cvise.py", line 166, in _run_additional_passes
self.test_manager.run_pass(p)
File "/usr/share/cvise/utils/testing.py", line 521, in run_pass
self.process_result(success_env)
File "/usr/share/cvise/utils/testing.py", line 546, in process_result
shutil.copy(test_env.test_case_path, self.current_test_case)
File "/usr/lib64/python3.8/shutil.py", line 418, in copy
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/usr/lib64/python3.8/shutil.py", line 264, in copyfile
with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/baranem/cvise-g1_ch4r0/cvise-952vdbgu/unit.cpp'
and cvise
couldn't do anything (this was with cvise --version
== cvise 2.3.0
).
There's a good chance this is "play stupid games, win stupid prizes", but I'm creating this issue in case other people come across it.
so first shot at cvise
:
$ /android1/cvise/cvise.py ./repro.sh compat2.i
ERROR Prereqs not found for pass ClangBinarySearchPass::replace-function-def-with-decl
ERROR Prereqs not found for pass ClangBinarySearchPass::remove-unused-function
ERROR Prereqs not found for pass ClangBinarySearchPass::replace-function-def-with-decl
ERROR Prereqs not found for pass ClangBinarySearchPass::remove-unused-function
ERROR Prereqs not found for pass ClangPass::remove-unused-function
ERROR Prereqs not found for pass ClangPass::callexpr-to-value
ERROR Prereqs not found for pass ClangPass::replace-callexpr
ERROR Prereqs not found for pass ClangPass::simplify-callexpr
ERROR Prereqs not found for pass ClangPass::remove-unused-enum-member
ERROR Prereqs not found for pass ClangPass::remove-enum-member-value
ERROR Prereqs not found for pass ClangBinarySearchPass::remove-unused-var
ERROR Prereqs not found for pass ClangPass::remove-namespace
ERROR Prereqs not found for pass ClangPass::aggregate-to-scalar
ERROR Prereqs not found for pass ClangPass::param-to-global
ERROR Prereqs not found for pass ClangPass::param-to-local
ERROR Prereqs not found for pass ClangPass::remove-nested-function
ERROR Prereqs not found for pass ClangPass::union-to-struct
ERROR Prereqs not found for pass ClangPass::return-void
ERROR Prereqs not found for pass ClangPass::simple-inliner
ERROR Prereqs not found for pass ClangPass::reduce-pointer-level
ERROR Prereqs not found for pass ClangPass::lift-assignment-expr
ERROR Prereqs not found for pass ClangPass::copy-propagation
ERROR Prereqs not found for pass ClangPass::callexpr-to-value
ERROR Prereqs not found for pass ClangPass::simplify-callexpr
ERROR Prereqs not found for pass ClangPass::remove-unused-function
ERROR Prereqs not found for pass ClangPass::remove-unused-enum-member
ERROR Prereqs not found for pass ClangPass::remove-enum-member-value
ERROR Prereqs not found for pass ClangBinarySearchPass::remove-unused-var
ERROR Prereqs not found for pass ClangPass::simplify-if
ERROR Prereqs not found for pass ClangPass::reduce-array-dim
ERROR Prereqs not found for pass ClangPass::reduce-array-size
ERROR Prereqs not found for pass ClangPass::move-function-body
ERROR Prereqs not found for pass ClangPass::simplify-comma-expr
ERROR Prereqs not found for pass ClangPass::simplify-dependent-typedef
ERROR Prereqs not found for pass ClangPass::replace-simple-typedef
ERROR Prereqs not found for pass ClangPass::replace-dependent-typedef
ERROR Prereqs not found for pass ClangPass::replace-one-level-typedef-type
ERROR Prereqs not found for pass ClangPass::remove-unused-field
ERROR Prereqs not found for pass ClangPass::instantiate-template-type-param-to-int
ERROR Prereqs not found for pass ClangPass::instantiate-template-param
ERROR Prereqs not found for pass ClangPass::template-arg-to-int
ERROR Prereqs not found for pass ClangPass::template-non-type-arg-to-int
ERROR Prereqs not found for pass ClangPass::reduce-class-template-param
ERROR Prereqs not found for pass ClangPass::remove-trivial-base-template
ERROR Prereqs not found for pass ClangPass::class-template-to-class
ERROR Prereqs not found for pass ClangPass::remove-base-class
ERROR Prereqs not found for pass ClangPass::replace-derived-class
ERROR Prereqs not found for pass ClangPass::remove-unresolved-base
ERROR Prereqs not found for pass ClangPass::remove-ctor-initializer
ERROR Prereqs not found for pass ClangPass::replace-class-with-base-template-spec
ERROR Prereqs not found for pass ClangPass::simplify-nested-class
ERROR Prereqs not found for pass ClangPass::remove-unused-outer-class
ERROR Prereqs not found for pass ClangPass::empty-struct-to-int
ERROR Prereqs not found for pass ClangPass::remove-pointer
ERROR Prereqs not found for pass ClangPass::reduce-pointer-pairs
ERROR Prereqs not found for pass ClangPass::remove-array
ERROR Prereqs not found for pass ClangPass::rename-fun
ERROR Prereqs not found for pass ClangPass::rename-param
ERROR Prereqs not found for pass ClangPass::rename-var
ERROR Prereqs not found for pass ClangPass::rename-class
ERROR Prereqs not found for pass ClangPass::rename-cxx-method
ERROR Prereqs not found for pass ClangPass::combine-global-var
ERROR Prereqs not found for pass ClangPass::combine-local-var
ERROR Prereqs not found for pass ClangPass::simplify-struct-union-decl
ERROR Prereqs not found for pass ClangPass::move-global-var
ERROR Prereqs not found for pass ClangPass::unify-function-decl
ERROR Prereqs not found for pass ClexPass::rename-toks
ERROR Prereqs not found for pass ClexPass::delete-string
Missing prerequisites for passes ClangPass::lift-assignment-expr, ClangPass::reduce-pointer-pairs, ClangPass::remove-unresolved-base, ClexPass::rename-toks, ClexPass::rm-toks-3, ClangPass::simplify-nested-class, ClangPass::aggregate-to-scalar, ClangPass::rename-fun, ClangPass::simplify-dependent-typedef, ClangPass::rename-cxx-method, ClangPass::remove-enum-member-value, ClangPass::replace-one-level-typedef-type, ClangPass::template-non-type-arg-to-int, ClangPass::simplify-callexpr, ClangPass::rename-class, ClangPass::rename-param, ClangPass::callexpr-to-value, ClangPass::remove-ctor-initializer, ClangBinarySearchPass::replace-function-def-with-decl, ClangPass::vector-to-array, ClangPass::copy-propagation, ClangPass::param-to-local, ClexPass::rm-toks-11, ClangPass::remove-base-class, ClangPass::replace-derived-class, ClangPass::remove-namespace, ClangPass::remove-array, ClangPass::instantiate-template-param, ClexPass::rm-toks-4, ClangPass::remove-trivial-base-template, ClangPass::simplify-struct, ClangPass::replace-array-access-with-index, ClangPass::remove-unused-enum-member, ClangPass::replace-simple-typedef, ClangPass::reduce-pointer-level, ClangPass::move-function-body, ClangPass::replace-dependent-name, ClexPass::delete-string, ClexPass::rm-toks-14, ClangPass::remove-unused-outer-class, ClangPass::simplify-recursive-template-instantiation, ClexPass::rm-toks-15, ClexPass::rm-toks-10, ClangPass::remove-unused-function, ClangPass::replace-array-index-var, ClangPass::reduce-array-dim, ClangPass::combine-global-var, ClangPass::simplify-comma-expr, ClangPass::instantiate-template-type-param-to-int, ClexPass::rm-toks-12, ClangPass::remove-try-catch, ClexPass::rm-toks-8, ClangPass::move-global-var, ClangPass::replace-class-with-base-template-spec, ClangPass::simple-inliner, ClangPass::reduce-class-template-param, ClexPass::rm-toks-5, ClexPass::rm-toks-2, ClangPass::replace-callexpr, ClangPass::simplify-if, ClangPass::remove-nested-function, ClexPass::rm-toks-9, ClangPass::simplify-struct-union-decl, ClangPass::replace-undefined-function, ClangPass::remove-pointer, ClangPass::replace-dependent-typedef, ClexPass::rm-tok-pattern-4, ClexPass::define, ClangPass::unify-function-decl, ClangPass::empty-struct-to-int, ClexPass::rm-toks-6, ClangPass::rename-var, ClangPass::local-to-global, ClangBinarySearchPass::remove-unused-function, ClangPass::remove-addr-taken, ClangPass::template-arg-to-int, ClangPass::combine-local-var, ClangPass::remove-unused-field, ClangPass::class-template-to-class, ClangPass::param-to-global, ClangPass::reduce-array-size, ClangPass::return-void, ClexPass::rm-toks-16, ClexPass::rm-toks-1, ClexPass::rm-toks-13, ClangBinarySearchPass::remove-unused-var, ClangPass::union-to-struct, ClexPass::rm-toks-7!
am I missing a dependency, or invoking this incorrectly? I built cvise
via:
$ mkdir build && cd build
$ cmake .. -G Ninja -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
or should I not be doing out of tree builds?
this is a regression in 1.5, compared to 1.4:
cvise --help
Traceback (most recent call last):
File "debian/cvise/usr/bin/cvise", line 166, in
passes_group.add_argument("--pass-group", type=str, choices=get_available_pass_groups(), help="Set of passes used during the reduction")
File "debian/cvise/usr/bin/cvise", line 113, in get_available_pass_groups
CVise.parse_pass_group_dict(pass_group_dict, set(), None, None, None, None, None)
TypeError: parse_pass_group_dict() takes 7 positional arguments but 8 were given
From @mgorny:
You should add a check that SOURCE_DIR!=BINARY_DIR somewhere. Right now this permits doing cmake . which is not going to work correctly with this.
Also, this file is really begging for a for loop.
* [topformflat](http://delta.tigris.org/)
says only:
The site will be decommissioned and shut down on 1-July-2020
For the following file:
inline namespace a {
void operator"" b(unsigned long long);
}
and arguments:
clang_delta --transformation=remove-namespace --counter=1 unit.cpp
clang_delta
crashes as follows:
clang_delta: ../clang_delta/RemoveNamespace.cpp:950: void RemoveNamespace::handleOneNamedDecl(const clang::NamedDecl*, const clang::DeclContext*, const string&): Assertion `IdInfo && "Invalid IdentifierInfo!"' failed.
Sorry if this is a "dumb" question, but what's the status of making changes to clang_delta
in the cvise
repo? I am not sure how closely this repo tracks creduce
, or if cvise
is happy to have feature extensions to clang_delta
.
Why am I asking? Well, I am interested (in some circumstances) in running clang_delta
to get the instance number as well as the function name for a given transformation.
For some inputs, I know a priori a function I want to preserve, but I want to replace every other function with the declaration. If query-instances
could not only give the number of instances, but also the function name for each instance, that would be awesome.
If I had this, I could do something like:
# query all of the instances in our original file
clang_delta --query-instances=replace-function-def-with-decl test.cpp
# record the index of the function I care about; let's say it is index 3
# replace from the first to the second (leaving the third alone!)
clang_delta --transformation=replace-function-def-with-decl --counter=1 --to-counter=2 test.cpp > intermediate.cpp
# re-query to get the remaining numbers
clang_delta --query-instances=replace-function-def-with-decl intermediate.cpp
# validate that the original function should now be index 1
# replace from the second to the total number
clang_delta --transformation=replace-function-def-with-decl --counter=2 --to-counter=99 intermediate.cpp > final.cpp
# the file final.cpp now *only* has a body for the function that was at index 3 in the original file
I've taken a look at this, and it seems that it is achievable by using getQualifiedNameAsString
inside of addOneFunctionDef
.
If I was to wrap this up as a feature request, is it something that cvise
would be happy to accept?
Following up on #12. Today I pulled, and reran cmake, which failed.
$ cmake .. -G Ninja -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
...
Pytest package not available: /usr/bin/python3.6: No module named pytest
ModuleNotFoundError: No module named 'pebble'
but this worked with my previous checkout. In my shell history, I have:
$ pip3 install pebble pytest
so I thought I had these installed correctly? I tried wiping out my build dir and cmake cache files, but it made no difference.
Rerunning:
$ pip3 install pebble pytest
Requirement already satisfied: pebble in /usr/local/google/home/ndesaulniers/.local/lib/python3.7/site-packages (4.5.1)
Requirement already satisfied: pytest in /usr/local/google/home/ndesaulniers/.local/lib/python3.7/site-packages (5.4.1)
Requirement already satisfied: attrs>=17.4.0 in /usr/lib/python3/dist-packages (from pytest) (19.3.0)
Requirement already satisfied: pluggy<1.0,>=0.12 in /usr/local/google/home/ndesaulniers/.local/lib/python3.7/site-packages (from pytest) (0.13.1)
Requirement already satisfied: packaging in /usr/lib/python3/dist-packages (from pytest) (19.1)
Requirement already satisfied: wcwidth in /usr/local/google/home/ndesaulniers/.local/lib/python3.7/site-packages (from pytest) (0.1.9)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in /usr/local/google/home/ndesaulniers/.local/lib/python3.7/site-packages (from pytest) (1.6.0)
Requirement already satisfied: py>=1.5.0 in /usr/local/google/home/ndesaulniers/.local/lib/python3.7/site-packages (from pytest) (1.8.1)
Requirement already satisfied: more-itertools>=4.0.0 in /usr/local/google/home/ndesaulniers/.local/lib/python3.7/site-packages (from pytest) (8.2.0)
Requirement already satisfied: zipp>=0.5 in /usr/local/google/home/ndesaulniers/.local/lib/python3.7/site-packages (from importlib-metadata>=0.12; python_version < "3.8"->pytest) (3.1.0)
Those seem to mention python3.7, but the cmake failures for pytest mention python3.6. Is that related?
Apologies for this not being the most dug-into issue, however I thought it was worth writing up in case other people hit this issue ...
I have a build of clang-10
from the following commit: llvm/llvm-project@2e95941, which is a build that works correctly against csmith-project/creduce@d83cbda (from April 20th, 2020).
However, it seems that this build of clang
isn't compatible with cvise
.
After C=clang CXX=clang++ cmake -DENABLE_TRANS_ASSERT=OFF ..
:
-- The C compiler identification is Clang 10.0.0
-- The CXX compiler identification is Clang 10.0.0
-- Check for working C compiler: /home/avj/llvm/root/bin/clang
-- Check for working C compiler: /home/avj/llvm/root/bin/clang -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /home/avj/llvm/root/bin/clang++
-- Check for working CXX compiler: /home/avj/llvm/root/bin/clang++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found LLVM 10.0.0svn
-- Using LLVMConfig.cmake in /home/avj/llvm/root/lib/cmake/llvm
-- Using ClangConfig.cmake in /home/avj/llvm/root/lib/cmake/clang
but then when I build cvise
:
/home/avj/cvise/clang_delta/TransformationManager.cpp:119:34: error: no member named 'lang_cxx20' in 'clang::LangStandard::Kind'; did you mean 'lang_cxx2a'?
LSTD = LangStandard::Kind::lang_cxx20;
~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
lang_cxx2a
/home/avj/llvm/root/include/clang/Frontend/LangStandards.def:143:1: note: 'lang_cxx2a' declared here
LANGSTANDARD(cxx2a, "c++2a",
^
/home/avj/llvm/root/include/clang/Frontend/LangStandard.h:45:5: note: expanded from macro 'LANGSTANDARD'
lang_##id,
^
<scratch space>:107:1: note: expanded from here
lang_cxx2a
^
/home/avj/cvise/clang_delta/TransformationManager.cpp:139:27: error: use of undeclared identifier 'Language'; did you mean 'LanguageID'?
if (IK.getLanguage() == Language::C) {
^~~~~~~~
LanguageID
/home/avj/llvm/root/include/clang/Basic/Builtins.h:32:6: note: 'LanguageID' declared here
enum LanguageID {
^
/home/avj/cvise/clang_delta/TransformationManager.cpp:139:37: error: no member named 'C' in 'clang::LanguageID'
if (IK.getLanguage() == Language::C) {
~~~~~~~~~~^
/home/avj/cvise/clang_delta/TransformationManager.cpp:140:72: error: use of undeclared identifier 'Language'; did you mean 'LanguageID'?
Invocation.setLangDefaults(ClangInstance->getLangOpts(), InputKind(Language::C), T, PPOpts);
^~~~~~~~
LanguageID
/home/avj/llvm/root/include/clang/Basic/Builtins.h:32:6: note: 'LanguageID' declared here
enum LanguageID {
^
/home/avj/cvise/clang_delta/TransformationManager.cpp:140:82: error: no member named 'C' in 'clang::LanguageID'
Invocation.setLangDefaults(ClangInstance->getLangOpts(), InputKind(Language::C), T, PPOpts);
~~~~~~~~~~^
/home/avj/cvise/clang_delta/TransformationManager.cpp:142:32: error: use of undeclared identifier 'Language'; did you mean 'LanguageID'?
else if (IK.getLanguage() == Language::CXX) {
^~~~~~~~
LanguageID
/home/avj/llvm/root/include/clang/Basic/Builtins.h:32:6: note: 'LanguageID' declared here
enum LanguageID {
^
/home/avj/cvise/clang_delta/TransformationManager.cpp:142:32: error: no member named 'CXX' in 'clang::LanguageID'; did you mean 'clang::InputKind::CXX'?
else if (IK.getLanguage() == Language::CXX) {
^~~~~~~~~~~~~
clang::InputKind::CXX
/home/avj/llvm/root/include/clang/Frontend/FrontendOptions.h:165:5: note: 'clang::InputKind::CXX' declared here
CXX,
^
/home/avj/cvise/clang_delta/TransformationManager.cpp:146:72: error: use of undeclared identifier 'Language'; did you mean 'LanguageID'?
Invocation.setLangDefaults(ClangInstance->getLangOpts(), InputKind(Language::CXX), T, PPOpts, LSTD);
^~~~~~~~
LanguageID
/home/avj/llvm/root/include/clang/Basic/Builtins.h:32:6: note: 'LanguageID' declared here
enum LanguageID {
^
/home/avj/cvise/clang_delta/TransformationManager.cpp:146:72: error: no member named 'CXX' in 'clang::LanguageID'; did you mean 'clang::InputKind::CXX'?
Invocation.setLangDefaults(ClangInstance->getLangOpts(), InputKind(Language::CXX), T, PPOpts, LSTD);
^~~~~~~~~~~~~
clang::InputKind::CXX
/home/avj/llvm/root/include/clang/Frontend/FrontendOptions.h:165:5: note: 'clang::InputKind::CXX' declared here
CXX,
^
/home/avj/cvise/clang_delta/TransformationManager.cpp:148:31: error: use of undeclared identifier 'Language'; did you mean 'LanguageID'?
else if(IK.getLanguage() == Language::OpenCL) {
^~~~~~~~
LanguageID
/home/avj/llvm/root/include/clang/Basic/Builtins.h:32:6: note: 'LanguageID' declared here
enum LanguageID {
^
/home/avj/cvise/clang_delta/TransformationManager.cpp:148:31: error: no member named 'OpenCL' in 'clang::LanguageID'; did you mean 'clang::InputKind::OpenCL'?
else if(IK.getLanguage() == Language::OpenCL) {
^~~~~~~~~~~~~~~~
clang::InputKind::OpenCL
/home/avj/llvm/root/include/clang/Frontend/FrontendOptions.h:168:5: note: 'clang::InputKind::OpenCL' declared here
OpenCL,
^
/home/avj/cvise/clang_delta/TransformationManager.cpp:176:71: error: too few arguments to function call, expected 4, have 3
ClangInstance->getDiagnostics());
^
/home/avj/llvm/root/include/clang/Frontend/CompilerInvocation.h:154:3: note: 'CreateFromArgs' declared here
static bool CreateFromArgs(CompilerInvocation &Res,
^
/home/avj/cvise/clang_delta/TransformationManager.cpp:179:42: error: use of undeclared identifier 'Language'; did you mean 'LanguageID'?
InputKind(Language::OpenCL),
^~~~~~~~
LanguageID
/home/avj/llvm/root/include/clang/Basic/Builtins.h:32:6: note: 'LanguageID' declared here
enum LanguageID {
^
/home/avj/cvise/clang_delta/TransformationManager.cpp:179:42: error: no member named 'OpenCL' in 'clang::LanguageID'; did you mean 'clang::InputKind::OpenCL'?
InputKind(Language::OpenCL),
^~~~~~~~~~~~~~~~
clang::InputKind::OpenCL
/home/avj/llvm/root/include/clang/Frontend/FrontendOptions.h:168:5: note: 'clang::InputKind::OpenCL' declared here
OpenCL,
^
14 errors generated.
make[2]: *** [clang_delta/CMakeFiles/clang_delta.dir/TransformationManager.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
Is this a known issue?
For:
cat unit.c
/*
* für
*/
file unit.c
unit.c: ISO-8859 text
running cvise --start-with-pass IncludesPass
gives:
00:00:00 INFO ===< 118085 >===
00:00:00 INFO running 24 interestingness tests in parallel
00:00:00 INFO INITIAL PASSES
00:00:00 INFO ===< IncludesPass >===
00:00:00 WARNING IncludesPass has encountered a non fatal bug: pass got stuck
00:00:00 INFO ===< UnIfDefPass >===
00:00:00 INFO ===< CommentsPass >===
00:00:00 WARNING CommentsPass has encountered a non fatal bug: pass got stuck
00:00:00 INFO ===< IfPass >===
Traceback (most recent call last):
File "/usr/bin/cvise", line 286, in <module>
reducer.reduce(pass_group, skip_initial=args.skip_initial_passes)
File "/usr/share/cvise/cvise.py", line 143, in reduce
self._run_additional_passes(pass_group['first'])
File "/usr/share/cvise/cvise.py", line 166, in _run_additional_passes
self.test_manager.run_pass(p)
File "/usr/share/cvise/utils/testing.py", line 503, in run_pass
self.state = self.current_pass.new(self.current_test_case, self.check_sanity)
File "/usr/share/cvise/passes/ifs.py", line 36, in new
bs = BinaryState.create(self.__count_instances(test_case))
File "/usr/share/cvise/passes/ifs.py", line 22, in __count_instances
for line in in_file.readlines():
File "/usr/lib64/python3.8/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 6: invalid start byte
(this was after editing cvise/utils/testing.py
to set GIVEUP_CONSTANT
to 1)
and creates:
Package: cvise 2.3.0
Git version: unknown
LLVM version: 12.0.0
System: uname_result(system='Linux', node='vistrrdslin0001', release='5.12.13-1-default', version='#1 SMP Mon Jun 28 06:37:23 UTC 2021 (74bd8c0)', machine='x86_64', processor='x86_64')
***************************************************
IncludesPass has encountered a bug:
pass got stuck
state: 2
Please consider tarring up cvise_bug_0
and creating an issue at https://github.com/marxin/cvise/issues and we will try to fix the bug.
***************************************************
and
Package: cvise 2.3.0
Git version: unknown
LLVM version: 12.0.0
System: uname_result(system='Linux', node='vistrrdslin0001', release='5.12.13-1-default', version='#1 SMP Mon Jun 28 06:37:23 UTC 2021 (74bd8c0)', machine='x86_64', processor='x86_64')
***************************************************
CommentsPass has encountered a bug:
pass got stuck
state: -1
Please consider tarring up cvise_bug_1
and creating an issue at https://github.com/marxin/cvise/issues and we will try to fix the bug.
***************************************************
I'm trying to get this working on Windows. However, I'm having a lot of struggles.
Having some documentation of how this is actualy intended to work on linux could help a lot.
For example:
py -3.6 checkSuccess.sh t.cpp
+ some example content of checkSuccess.sh
Example is a https://gcc.gnu.org/PR95768 case reducer. I'll provide slightly different example here:
Input where cvise
stopped and was not able to progress further:
extern "C" void *malloc(unsigned long);
__attribute__((__noreturn__)) void a(const char *, const char *, int);
class c {
public:
enum e { d };
};
class f {
public:
int aa;
f operator=(int) {
int g, h, j, a, k, c = aa;
k = c;
h = k;
int i = h &= j;
a = i;
int d = a;
g = d;
aa = g;
return *this;
}
};
template <typename, int, int> using af = f;
template <int ae, int ag> using l = af<int, ae, ag>;
class m {
public:
l<1, 1> ai;
};
class C {
public:
m *o() { return (m *)this + 1; }
};
class n {
public:
int ak;
C *al;
n(c::e p1) {
p1 ? void() : a("", "", 0);
al = (C *)malloc(ak);
m *b = al->o();
b->ai = 0;
}
};
bool an() {
n(c::d);
return true;
}
Test for cvise
:
#!/bin/bash
args=(
/usr/bin/x86_64-pc-linux-gnu-g++
-march=sandybridge
-O2
-Wall
-c bug.cc
-o bug.o
)
#"${args[@]}"
LANG=C "${args[@]}" -fsyntax-only -Werror || exit 1
LANG=C "${args[@]}" 2>&1 | grep 'Segmentation fault'
cvise ./test_cvise.sh bug.cc
can't reduce example further.
It's not very hard to shrink it further manuall by removing unused template arguments of template <typename, int, int> using af = f;
. I did it as:
--- bug.cc__ 2020-06-19 18:56:41.000000000 +0100
+++ bug.cc___ 2020-06-20 10:27:24.000000000 +0100
@@ -19,8 +19,8 @@
return *this;
}
};
-template <typename, int, int> using af = f;
-template <int ae, int ag> using l = af<int, ae, ag>;
+using af = f;
+template <int ae, int ag> using l = af;
class m {
public:
l<1, 1> ai;
--- bug.cc___ 2020-06-20 10:27:24.000000000 +0100
+++ bug.cc____ 2020-06-20 10:39:18.000000000 +0100
@@ -20,10 +20,10 @@
}
};
using af = f;
-template <int ae, int ag> using l = af;
+using l = af;
class m {
public:
- l<1, 1> ai;
+ l ai;
};
class C {
public:
After that cvise ./test_cvise.sh bug.cc
was able to get it down to:
extern "C" void *malloc(unsigned long);
__attribute__((__noreturn__)) void a(const char *, const char *, int);
class c {
public:
enum e { d };
};
class f {
public:
int aa;
f operator=(int) {
int g, h, j, a, k, c = aa;
k = c;
h = k;
int i = h &= j;
a = i;
int d = a;
g = d;
aa = g;
return *this;
}
};
using l = f;
class m {
public:
l ai;
};
class C {
public:
m *o() { return (m *)this + 1; }
};
class n {
public:
int ak;
C *al;
n(c::e p1) {
p1 ? void() : a("", "", 0);
al = (C *)malloc(ak);
m *b = al->o();
b->ai = 0;
}
};
bool an() {
n(c::d);
return true;
}
I think creduce
had very similar limitation.
Is it something that cvise
could perform perform automatically? Or it's something up to clang's libraries to get better at?
Thank you!
The following fails:
$ cat fsize.ii
template <class V> void operator*(V);
template <class Allocator> struct cpp_int_backend {
void do_get_string() { this->size() * Allocator::limb_bits; }
};
$ clang_delta --transformation=remove-nested-function --counter=1 fsize.ii
clang_delta: /home/marxin/Programming/cvise/clang_delta/RemoveNestedFunction.cpp:362: void RemoveNestedFunction::getNewTmpVariableStr(clang::ASTContext&, std::string&): Assertion `FD && "Cannot resolve DName!"' failed.
Aborted (core dumped)
For the following file:
template <typename a> auto b() -> decltype(1(a())) {}
and arguments:
clang_delta --transformation=lift-assignment-expr --counter=1 unit.cpp
clang_delta
crashes as follows:
clang_delta: /usr/include/llvm/Support/Casting.h:104: static bool llvm::isa_impl_cl<To, const From*>::doit(const From*) [with To = clang::AttributedStmt; From = clang::Stmt]: Assertion `Val && "isa<> used on a null pointer"' failed.
The source file says to read License.txt
but the file doesn't seem to be included.
I'm giving cvise
a try on top of some examples I've been using with creduce
-- I took the inputs and interestingness test directly from something that worked for creduce
.
I let cvise
run for only a few minutes, with this on stdout
:
INFO ===< 69135 >===
INFO running 97 interestingness tests in parallel
INFO INITIAL PASSES
INFO ===< IncludesPass >===
INFO (0.1%, 22991 bytes)
INFO ===< UnIfDefPass >===
INFO ===< CommentsPass >===
INFO ===< IfPass >===
INFO ===< LineMarkersPass >===
INFO ===< BlankPass >===
INFO ===< ClangBinarySearchPass::replace-function-def-with-decl >===
INFO ===< ClangBinarySearchPass::remove-unused-function >===
INFO ===< LinesPass::0 >===
INFO ===< LinesPass::1 >===
I copied the input file (which cvise
had modified in place) to another directory and ran the "interesting test" manually ... and it failed!
Looking at the intermediate hand by eye, I agree that the file should no longer pass the interestingness test.
Is the updated file on disk supposed to always pass the interestingness test (like it does with creduce
)?
Hello!
I ran cvise
today and get a bunch of warnings:
00:00:00 ERROR Prereqs not found for pass ClangPass::vector-to-array
00:00:00 ERROR cannot find external program clang_delta
00:00:00 ERROR Prereqs not found for pass ClangPass::remove-try-catch
00:00:00 ERROR cannot find external program clang-format
00:00:00 ERROR Prereqs not found for pass IndentPass::regular
00:00:00 ERROR cannot find external program clex
cvise
still ran to completion, but it looks like some passes were skipped because these are missing.
clang-format
is in my $PATH; clang_delta
and clex
are not. (I haven't heard of those latter two) Not sure if this is an issue with relative paths based on how I ran cvise
:
$ cd <dir that's not cvise sources>
$ /<abs path to>/cvise/cvise.py --n 71 ./repro.sh slub.i
or if I've probably not enabled clang-tools-extra
for my build of llvm?
All reproduction details can be found at #26 if the mentioned pass is removed.
Running the example, using 1 thread, causes several passes to get stuck
ClangPass::callexpr-to-value has encountered a non fatal bug: pass got stuck
ClangPass::replace-callexpr has encountered a non fatal bug: pass got stuck
ClangPass::simplify-callexpr has encountered a non fatal bug: pass got stuck
ClangPass::remove-unused-enum-member has encountered a non fatal bug: pass got stuck
ClangPass::remove-enum-member-value has encountered a non fatal bug: pass got stuck
ClangPass::remove-namespace has encountered a non fatal bug: pass got stuck
ClangPass::aggregate-to-scalar has encountered a non fatal bug: pass got stuck
ClangPass::param-to-global has encountered a non fatal bug: pass got stuck
ClangPass::param-to-local has encountered a non fatal bug: pass got stuck
Currently at 18h and running.
Trying to compile this on Windows (Visual Studio 2019) doesn't work.
I use the following to configure my CMake:
set CC="C:\Program Files\VisualStudio\Vs2019\VC\Tools\MSVC\14.24.28314\bin\Hostx64\x64\cl.exe"
set CXX="C:\Program Files\VisualStudio\Vs2019\VC\Tools\MSVC\14.24.28314\bin\Hostx64\x64\cl.exe"
set LLVM_DIR=LLVM_BUILD\lib\cmake\llvm
set CLANG_DIR=LLVM_BUILD\lib\cmake\clang
C:\Program Files\VisualStudio\Vs2019\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe -G "Visual Studio 16 2019" -A x64 -Thost=x64 ..\cvise-master
cl : command line error D8021: invalid numeric argument '/Werror' [build\clang_delta\clang_delta.vcxproj]
After removing this flag from source/CMakeLists.txt (line 144-145), an actual build gets started.
Later we get other errors, like:
source\clang_delta\ExpressionDetector.cpp : fatal error C1128: number of sections exceeded object file format limit: compile with /bigobj [build\clang_delta\clang_delta.vcxproj]
cl : command line error D8021: invalid numeric argument '/Wno-unused-function' [build\clex\clex.vcxproj]
cl : command line error D8021: invalid numeric argument '/Wno-unused-function' [build\clex\strlex.vcxproj]
These last 2 warnings are a result of source/clex/CMakeLists.txt
I also encountered some issues with the .l files, however, this could be linked to older hacks I've added to our lexer. (I did have to add %option nounistd to the options).
With these changes, I got the C++ code compiling.
Can we get some native windows support instead?
Note: Compile times take long with a lot of warnings. (Over 14 minutes on 1 file)
Ran today's cvise
: and observerd the crash:
$ cvise --remove-pass IfPass ./test_cvise.sh *.c
00:00:01 INFO ===< 834890 >===
00:00:01 INFO running 4 interestingness tests in parallel
00:00:01 INFO INITIAL PASSES
00:00:01 INFO ===< IncludesPass >===
00:00:16 INFO ===< UnIfDefPass >===
00:00:41 INFO ===< CommentsPass >===
00:00:45 INFO ===< LineMarkersPass >===
00:00:45 INFO ===< BlankPass >===
00:00:55 INFO (0.0%, 683496 bytes)
00:00:59 INFO ===< ClangBinarySearchPass::replace-function-def-with-decl >===
Traceback (most recent call last):
File "/usr/share/cvise/passes/clangbinarysearch.py", line 51, in count_instances
proc = subprocess.run(cmd, universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
NameError: name 'subprocess' is not defined
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/cvise", line 265, in <module>
reducer.reduce(pass_group, skip_initial=args.skip_initial_passes)
File "/usr/share/cvise/cvise.py", line 122, in reduce
self._run_additional_passes(pass_group["first"])
File "/usr/share/cvise/cvise.py", line 145, in _run_additional_passes
self.test_manager.run_pass(p)
File "/usr/share/cvise/utils/testing.py", line 458, in run_pass
self.state = self.current_pass.new(self.current_test_case)
File "/usr/share/cvise/passes/clangbinarysearch.py", line 31, in new
self.detect_best_standard(test_case)
File "/usr/share/cvise/passes/clangbinarysearch.py", line 20, in detect_best_standard
instances = self.count_instances(test_case)
File "/usr/share/cvise/passes/clangbinarysearch.py", line 52, in count_instances
except subprocess.SubprocessError:
NameError: name 'subprocess' is not defined
Hiho,
after running cvise for an extended time
> cvise --n 40 --skip-key-off --print-diff --timeout 120 --clang-delta-std=c++20 ./creduce_probe.sh ice.cpp
> cvise --version
cvise 2.3.0 (38b42d9)
I got this error:
08:20:35 INFO (73.1%, 82693 bytes, 1644 lines)
Traceback (most recent call last):
File "/usr/local/bin/cvise", line 286, in <module>
reducer.reduce(pass_group, skip_initial=args.skip_initial_passes)
File "/usr/local/share/cvise/cvise.py", line 146, in reduce
self._run_main_passes(pass_group['main'])
File "/usr/local/share/cvise/cvise.py", line 176, in _run_main_passes
self.test_manager.run_pass(p)
File "/usr/local/share/cvise/utils/testing.py", line 517, in run_pass
success_env = self.run_parallel_tests()
File "/usr/local/share/cvise/utils/testing.py", line 443, in run_parallel_tests
test_env = TestEnvironment(self.state, order, self.test_script, folder,
File "/usr/local/share/cvise/utils/testing.py", line 55, in __init__
self.copy_files(test_case, additional_files)
File "/usr/local/share/cvise/utils/testing.py", line 61, in copy_files
shutil.copy(test_case, self.folder)
File "/usr/lib/python3.9/shutil.py", line 418, in copy
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/usr/lib/python3.9/shutil.py", line 275, in copyfile
_fastcopy_sendfile(fsrc, fdst)
File "/usr/lib/python3.9/shutil.py", line 166, in _fastcopy_sendfile
raise err from None
File "/usr/lib/python3.9/shutil.py", line 152, in _fastcopy_sendfile
sent = os.sendfile(outfd, infd, offset, blocksize)
OSError: [Errno 28] No space left on device: '/home/marehr/develope/seqan3-build/cvise/068-cvise-clang-14-gcclib-9-coordinate_matrix_simd_test/ice.cpp' -> '/tmp/cvise-zzbvxuwp/cvise-d4rw0cdm/ice.cpp'
my /tmp folder contains
> find /tmp -maxdepth 1 -type d -name "cvise-*" | wc -l
61
> du -hcs /tmp/cvise-*
492K /tmp/cvise-0q4bn__9
0 /tmp/cvise-1axqrcpq
2.1M /tmp/cvise-1nsmipmz
...
3.7M /tmp/cvise-zzbvxuwp
99M total
> find /tmp -maxdepth 1 -type f -name "ice-*" | wc -l
22278
> du -hcs /tmp/ice-*
1.9M /tmp/ice-000530.cpp
8.0K /tmp/ice-000530.sh
1.2M /tmp/ice-000769.cpp
8.0K /tmp/ice-000769.sh
...
8.0K /tmp/ice-fffc10.sh
1.2M /tmp/ice-fffd0e.cpp
8.0K /tmp/ice-fffd0e.sh
1.2M /tmp/ice-fffee2.cpp
8.0K /tmp/ice-fffee2.sh
16G total
I would expect that all those files will get regularly deleted during the reduction, as I didn't specify --save-temps
.
When I run cvise, sometimes it seems stalled on making progress. When I check htop
, it doesn't look like my cores are doing any work related to cvise. It looks like
00:00:03 INFO ===< LinesPass::0 >===
is what's running. I'm running cvise with --n 71
. Are there synchronization points where all threads need to wait on one to make progress? Bisection will eventually converge, but it seems to take quite some time to make forward progress.
Playing with ps -ef
, it looks like there are 71 instances of the python interpreter running, with 0 instances of clang, clang_delta, or my reproducer shell script.
Trying to build this on Arch I get:
CMake Warning at CMakeLists.txt:52 (message):
Pytest package not available: pytest 6.2.4
ERROR: usage: __main__.py [options] [file_or_dir] [file_or_dir] [...]
__main__.py: error: unrecognized arguments: --flake8
inifile: /home/vlovich/cloudflare/cvise/setup.cfg
rootdir: /home/vlovich/cloudflare/cvise
I tried pip install flake8
but that didn't seem to help. Removing the --flake8
from setup.cfg was a workaround but I'm not sure what the right fix is.
trying to build cvise from source again, I see:
In file included from ../clang_delta/ClassTemplateToClass.cpp:17:
/android0/llvm-project/clang/include/clang/AST/RecursiveASTVisitor.h:2154:8: error: no declaration matches ‘bool clang::RecursiveASTVisitor<Derived>::TraverseCXXAddrspaceCastExpr(clang::CXXAddrspaceCastExpr*, clang::RecursiveASTVisitor<Derived>::DataRecursionQueue*)’
maybe out of date against ToT LLVM?
This isn't a big deal to me, but I just ran cvise like this:
cvise --debug --clang-delta-std c++11 --n 97 cvise.sh ac.cpp |& cat > log.txt
but I then noticed the following in the log file:
grep "/usr/libexec/cvise/clang_delta" log.txt | egrep -oh "std=[^ ]*" | sort | uniq -c | sort -n
1002 std=c++98
4870 std=c++17
So we've done a 1002 instances of c++98 and 4870 of c++17 -- but none of c++11.
Have I missed something, or is this a bug?
This is with:
cvise --version
cvise 2.1.0
from 2.1.0+git.20210111.972f4ba-1.2
on Tumbleweed.
When creating executables with CMake, the executables are created in a directory like <builddir>/delta/<Flavor>
in my case is Debug.
These executables are not picked up by the CVise python script.
At the moment, I've hacked around line 70 of cvise.py the following code:
local_folder = os.path.join(script_path, local_folder)
path = shutil.which(prog, path=local_folder)
to become:
local_folder = os.path.join(script_path, local_folder)
path = shutil.which(prog, path=local_folder)
local_folder = os.path.join(local_folder,"Debug")
path = shutil.which(prog, path=local_folder)
that way the executables are being found
sending these for some runs from last week:
https://people.debian.org/~doko/tmp/cvise-2.1.0-bugs.tar.xz
I'm currently running cvise
like this:
cvise --start-with-pass LinesPass::0 --timeout 600 --debug --n $(nproc) cvise.sh unit.cpp
and I'm seeing this in the log:
00:00:00 DEBUG perform sanity check...
00:00:00 DEBUG sanity check tmpdir = /mnt/baranem/cvise-ja21eq2_
00:00:46 DEBUG sanity check successful
00:00:46 INFO ===< 68660 >===
00:00:46 INFO running 24 interestingness tests in parallel
00:00:46 INFO INITIAL PASSES
00:00:46 DEBUG Creating pass root folder: /mnt/baranem/cvise-m811bv_g
00:00:46 INFO ===< LinesPass::0 >===
00:00:46 DEBUG perform sanity check...
00:00:46 DEBUG sanity check tmpdir = /mnt/baranem/cvise-6403wq4c
00:01:33 DEBUG sanity check successful
what I'm surprised at here is that cvise
runs the sanity check at the start (which is really great!), but also as part of the of the start of LinesPass
.
Given cvise
has literally just run the sanity check and made no changes to the input file, I'm surprised it is doing the sanity check again.
This is such a minor nit, but as I noticed it, I thought it was worth sharing. It is "more obvious" to me currently because the interestingness test takes ~45 seconds, so I'm really aware that cvise
spends 90 seconds "not doing work".
For the following file:
template <typename...> struct a;
template <typename> struct b { b(...); };
namespace {
auto bx() -> b<a<>> {
using by = b<int>;
by bz;
return bz;
}
} // namespace
(yes, it seems oops, didn't have cvise
itself can't make that smaller!)-Wno-unused
!
The following holds:
#!/bin/bash
set -eu
clang_delta="/home/avj/clones/cvise/master/build/clang_delta/clang_delta"
passes="aggregate-to-scalar callexpr-to-value param-to-global param-to-local remove-enum-member-value remove-namespace remove-nested-function remove-unused-function replace-callexpr simplify-callexpr union-to-struct local-to-global param-to-global"
file=unit.cpp
pass_no=1
g++ -Wall -Werror -Wno-unused -c $file
for pass_name in $passes; do
ret=0
${clang_delta} --transformation=${pass_name} --counter=${pass_no} ${file} || ret=$?
if [ $ret -ne 139 ]; then
exit 1
fi
done
echo "All failed"
exit 0
# EOF
That is, for all of passes aggregate-to-scalar
, param-to-local
, remove-nested-function
, local-to-global
, union-to-struct
, remove-namespace
and param-to-global
, clang_delta
crashes.
seen with 2.1.0, on https://gcc.gnu.org/PR98107.
$ ls -l tmp*|wc -l
51
using
g++ -c -fchecking=2 --param=hash-table-verification-limit=1000 foo.ii
in the interestingness test.
For the following file:
struct a {
a(a &b) = default;
};
and arguments:
clang_delta --transformation=param-to-local --counter=1 unit.cpp
clang_delta
crashes as follows:
clang_delta: ../clang_delta/RewriteUtils.cpp:764: bool RewriteUtils::addLocalVarToFunc(const string&, clang::FunctionDecl*): Assertion `Body && "NULL body for a function definition!"' failed.
Again from @mgorny.
Traceback (most recent call last):
File "/home/avj/cvise/build/cvise.py", line 229, in <module>
reducer.reduce(pass_group, skip_initial=args.skip_initial_passes)
File "/home/avj/cvise/build/cvise/cvise.py", line 124, in reduce
self._run_additional_passes(pass_group["first"])
File "/home/avj/cvise/build/cvise/cvise.py", line 151, in _run_additional_passes
self.test_manager.run_pass(p)
File "/home/avj/cvise/build/cvise/utils/testing.py", line 460, in run_pass
success_env = self.run_parallel_tests()
File "/home/avj/cvise/build/cvise/utils/testing.py", line 387, in run_parallel_tests
quit_loop = self.process_done_futures()
File "/home/avj/cvise/build/cvise/utils/testing.py", line 320, in process_done_futures
if test_env.success:
AttributeError: 'NoneType' object has no attribute 'success'
Found on following code (t.cpp):
#include <variant>
#include <vector>
struct A {
std::basic_string<wchar_t> m;
};
struct B {
B(A a);
std::variant<A> m;
};
struct C : B {
using B::B;
};
void f() {
std::vector<C> c{
};
c.push_back(C{
A{
}
}
);
}
Launch script:
py -3.6 G:\llvm\build-cvise\cvise.py --clang-delta-std c++17 --n 1 --print-diff checkIfBugExists.bat t.cpp
The error occurs within ClangPass::remove-unused-function
checkIfBugExists.bat
SET CVISE_BUG_DETECTED=0
SET CVISE_BUG_IRREPRODUCABLE=1
setlocal enableDelayedExpansion
echo "Current dir: %cd%"
echo "Copied file: %1"
C:\Path\To\LLVM_7_0_0\bin\clang-cl.exe /c /EHsc -w /std:c++17 t.cpp > compilation.out 2> compilation.err
if %ERRORLEVEL% EQU 0 (
echo "Compilation success"
exit /b %CVISE_BUG_IRREPRODUCABLE%
)
echo "Failed to compile"
findstr /r /c:"Wrote crash dump file" .\compilation.err
if %ERRORLEVEL% EQU 0 (
echo "Found crash message"
exit /b %CVISE_BUG_DETECTED%
) else (
echo "Not found"
exit /b %CVISE_BUG_IRREPRODUCABLE%
)
Callstack of delta.exe (Compiled with LLVM 10.0.0)
issue_debug_notification(message) Line 28 C++
__acrt_report_runtime_error(message) Line 154 C++
abort() Line 61 C++
common_assert_to_stderr<wchar_t>(expression, file_name, line_number) Line 187 C++
common_assert<wchar_t>(expression, file_name, line_number, return_address) Line 420 C++
_wassert(expression, file_name, line_number) Line 444 C++
> RemoveUnusedFunction::doRewriting() Line 307 C++
RemoveUnusedFunction::HandleTranslationUnit(Ctx) Line 284 C++
clang::ParseAST(S, PrintStats, SkipFunctionBodies) Line 178 C++
TransformationManager::doTransformation(ErrorMsg, ErrorCode) Line 308 C++
main(argc, argv) Line 249 C++
Issue is in void RemoveUnusedFunction::doRewriting()
where TheFunctionDecl
seems to be nullptr. ToCounter
is on -1.
For fun I'm playing with OpenBSD 6.9 on an old PowerPC G5 Mac.
I successfully built cvise but a basic invocation crashes:
cvise --version
Segmentation fault (core dumped)
Here's the backtrace:
builder3$ gdb python3 /home/builder/external/cvise-install/bin/python3.8.core
(gdb) bt
#0 0xe06cb15c in do_mkvalue () from /usr/local/lib/libpython3.8.so.0.0
#1 0xe06cb3ac in do_mktuple () from /usr/local/lib/libpython3.8.so.0.0
#2 0xe06ca9c0 in do_mkvalue () from /usr/local/lib/libpython3.8.so.0.0
#3 0xe06c9b08 in Py_BuildValue () from /usr/local/lib/libpython3.8.so.0.0
#4 0xe08fc1bc in psutil_proc_oneshot_info () from /usr/local/lib/python3.8/site-packages/psutil/_psutil_bsd.cpython-38.so
#5 0xe05744a0 in cfunction_call_varargs () from /usr/local/lib/libpython3.8.so.0.0
#6 0xe0575458 in PyCFunction_Call () from /usr/local/lib/libpython3.8.so.0.0
#7 0xe0573af8 in _PyObject_MakeTpCall () from /usr/local/lib/libpython3.8.so.0.0
#8 0xe068662c in call_function () from /usr/local/lib/libpython3.8.so.0.0
#9 0xe06836a0 in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.8.so.0.0
#10 0xe067d584 in PyEval_EvalFrameEx () from /usr/local/lib/libpython3.8.so.0.0
#11 0xe0574910 in function_code_fastcall () from /usr/local/lib/libpython3.8.so.0.0
#12 0xe0574ad4 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.8.so.0.0
#13 0xe0686568 in call_function () from /usr/local/lib/libpython3.8.so.0.0
#14 0xe0683724 in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.8.so.0.0
#15 0xe0687424 in _PyEval_EvalCodeWithName () from /usr/local/lib/libpython3.8.so.0.0
#16 0xe0574b54 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.8.so.0.0
#17 0xe0686568 in call_function () from /usr/local/lib/libpython3.8.so.0.0
#18 0xe0683684 in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.8.so.0.0
#19 0xe067d584 in PyEval_EvalFrameEx () from /usr/local/lib/libpython3.8.so.0.0
#20 0xe0574910 in function_code_fastcall () from /usr/local/lib/libpython3.8.so.0.0
#21 0xe0574ad4 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.8.so.0.0
#22 0xe0574070 in PyVectorcall_Call () from /usr/local/lib/libpython3.8.so.0.0
#23 0xe05741fc in PyObject_Call () from /usr/local/lib/libpython3.8.so.0.0
#24 0xe0683948 in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.8.so.0.0
#25 0xe0687424 in _PyEval_EvalCodeWithName () from /usr/local/lib/libpython3.8.so.0.0
#26 0xe0574b54 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.8.so.0.0
#27 0xe0686568 in call_function () from /usr/local/lib/libpython3.8.so.0.0
#28 0xe0683684 in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.8.so.0.0
#29 0xe067d584 in PyEval_EvalFrameEx () from /usr/local/lib/libpython3.8.so.0.0
#30 0xe0574910 in function_code_fastcall () from /usr/local/lib/libpython3.8.so.0.0
#31 0xe0574ad4 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.8.so.0.0
#32 0xe0686568 in call_function () from /usr/local/lib/libpython3.8.so.0.0
#33 0xe0683684 in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.8.so.0.0
#34 0xe0687424 in _PyEval_EvalCodeWithName () from /usr/local/lib/libpython3.8.so.0.0
#35 0xe0574b54 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.8.so.0.0
#36 0xe0686568 in call_function () from /usr/local/lib/libpython3.8.so.0.0
#37 0xe0683684 in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.8.so.0.0
#38 0xe0687424 in _PyEval_EvalCodeWithName () from /usr/local/lib/libpython3.8.so.0.0
#39 0xe0574b54 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.8.so.0.0
#40 0xe05738bc in _PyObject_FastCallDict () from /usr/local/lib/libpython3.8.so.0.0
#41 0xe05759ac in _PyObject_Call_Prepend () from /usr/local/lib/libpython3.8.so.0.0
#42 0xe05fbeac in slot_tp_init () from /usr/local/lib/libpython3.8.so.0.0
#43 0xe05f5500 in type_call () from /usr/local/lib/libpython3.8.so.0.0
#44 0xe0573af8 in _PyObject_MakeTpCall () from /usr/local/lib/libpython3.8.so.0.0
#45 0xe068662c in call_function () from /usr/local/lib/libpython3.8.so.0.0
#46 0xe06836a0 in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.8.so.0.0
#47 0xe067d584 in PyEval_EvalFrameEx () from /usr/local/lib/libpython3.8.so.0.0
#48 0xe0574910 in function_code_fastcall () from /usr/local/lib/libpython3.8.so.0.0
#49 0xe0574ad4 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.8.so.0.0
#50 0xe0686568 in call_function () from /usr/local/lib/libpython3.8.so.0.0
#51 0xe0683724 in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.8.so.0.0
#52 0xe0687424 in _PyEval_EvalCodeWithName () from /usr/local/lib/libpython3.8.so.0.0
#53 0xe067d358 in PyEval_EvalCode () from /usr/local/lib/libpython3.8.so.0.0
#54 0xe06dd0c0 in run_mod () from /usr/local/lib/libpython3.8.so.0.0
#55 0xe06dabb4 in PyRun_SimpleFileExFlags () from /usr/local/lib/libpython3.8.so.0.0
#56 0xe06da5f0 in PyRun_AnyFileExFlags () from /usr/local/lib/libpython3.8.so.0.0
#57 0xe07044f4 in Py_RunMain () from /usr/local/lib/libpython3.8.so.0.0
#58 0xe0704a14 in pymain_main () from /usr/local/lib/libpython3.8.so.0.0
#59 0xe0704ba4 in Py_BytesMain () from /usr/local/lib/libpython3.8.so.0.0
#60 0x249206b4 in main () from /usr/local/bin/python3
Hopefully that means something to you... otherwise, it does repro, so I could try whatever...
For the following file:
struct a {
template <typename aj> void b(aj) {}
};
auto c() -> a;
auto d() {
c().b([] {
a d = c();
return d;
});
}
cvise
appears to get stuck in an "infinite" loop running:
clang_delta --transformation=remove-nested-function --counter=<idx>
If you run clang_delta --transformation=remove-nested-function --counter=1
manually 10 times (with the new input being the last output), then you get something like this:
struct a {
template <typename aj> void b(aj) {}
};
auto c() -> a;
auto d() {
c().b([] {
a __trans_tmp_10 = c();
a __trans_tmp_9 = __trans_tmp_10;
a __trans_tmp_8 = __trans_tmp_9;
a __trans_tmp_7 = __trans_tmp_8;
a __trans_tmp_6 = __trans_tmp_7;
a __trans_tmp_5 = __trans_tmp_6;
a __trans_tmp_4 = __trans_tmp_5;
a __trans_tmp_3 = __trans_tmp_4;
a __trans_tmp_2 = __trans_tmp_3;
a __trans_tmp_1 = __trans_tmp_2;
a d = __trans_tmp_1;
return d;
});
}
While I was cvising
one file, this count got up to 5812 before I killed it; the tail of the cvise
log looked like this:
01:27:39 INFO (-545.8%, 608185 bytes, 6636 lines)
01:27:40 INFO (-545.9%, 608282 bytes, 6637 lines)
01:27:41 INFO (-546.0%, 608379 bytes, 6638 lines)
01:27:42 INFO (-546.1%, 608476 bytes, 6639 lines)
01:27:43 INFO (-546.2%, 608573 bytes, 6640 lines)
01:27:44 INFO (-546.3%, 608670 bytes, 6641 lines)
01:27:45 INFO (-546.4%, 608767 bytes, 6642 lines)
01:27:46 INFO (-546.5%, 608864 bytes, 6643 lines)
01:27:48 INFO (-546.6%, 608961 bytes, 6644 lines)
01:27:49 INFO (-546.7%, 609058 bytes, 6645 lines)
01:27:50 INFO (-546.8%, 609155 bytes, 6646 lines)
01:27:51 INFO (-546.9%, 609252 bytes, 6647 lines)
01:27:52 INFO (-547.0%, 609349 bytes, 6648 lines)
01:27:53 INFO (-547.1%, 609446 bytes, 6649 lines)
01:27:54 INFO (-547.2%, 609543 bytes, 6650 lines)
01:27:56 INFO (-547.3%, 609640 bytes, 6651 lines)
01:27:57 INFO (-547.5%, 609737 bytes, 6652 lines)
01:27:58 INFO (-547.6%, 609834 bytes, 6653 lines)
01:27:59 INFO (-547.7%, 609931 bytes, 6654 lines)
01:28:00 INFO (-547.8%, 610028 bytes, 6655 lines)
01:28:01 INFO (-547.9%, 610125 bytes, 6656 lines)
01:28:02 INFO (-548.0%, 610222 bytes, 6657 lines)
01:28:03 INFO (-548.1%, 610319 bytes, 6658 lines)
01:28:05 INFO (-548.2%, 610416 bytes, 6659 lines)
01:28:06 INFO (-548.3%, 610513 bytes, 6660 lines)
and cvise
first stated doing --transformation=remove-nested-function
at 00:06:27
:
00:06:16 INFO ===< ClangPass::remove-namespace >===
00:06:20 INFO ===< ClangPass::aggregate-to-scalar >===
00:06:20 INFO ===< ClangPass::param-to-global >===
00:06:25 INFO ===< ClangPass::param-to-local >===
00:06:27 INFO ===< ClangPass::remove-nested-function >===
and tried no other passes -- that is, it spent 1h21m doing remove-nested-function
with the file slowly getting larger every time.
I just tried running cvise on macOS and it dies with:
AttributeError: 'Process' object has no attribute 'cpu_affinity'
. Commenting out those lines allows me to continue.
https://psutil.readthedocs.io/en/latest/#psutil.Process.cpu_affinity says Availability: Linux, Windows, FreeBSD
, so I guess the call could be guarded with a sys.platform
check?
From @mgorny:
Oh, and this really needs to be either purely informational, or come with a switch to disable the checks. Otherwise, you're going to prevent people from building binary packages without installing runtime dependencies.
I've been cvising some C++14 code recently, and ended-up with quite a few cvise_bug_*
folders (38 currently).
Is there a way for me to cvise
these failures?
Basically, what I'm asking is this: if I see
ClangPass::remove-nested-function has encountered a bug:
pass error
state: 5
in PASS_BUG_INFO.TXT
, is it possible for me to:
? If so, then I can cvise the inputs to provide reduced failures.
Sadly, I can't share the original files as-is, because they're proprietary code; if I could cvise
then, I can probably make files I can share.
In #44, functionality was introduced that allowed the user to run clang_delta
(with --transformation=replace-function-def-with-decl
), but to say "do not change this routine".
We have a few extensions that are worth implementing:
--query-instances
to not only print the instance count, but also the names of the possible routines (which then allows a user to know which name to use as part of --preserve-routine
).A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.