Nanos6 is a runtime that implements the OmpSs-2 parallel programming model, developed by the System Tools and Advanced Runtimes (STAR) group at the Barcelona Supercomputing Center.
License: GNU General Public License v3.0
Hello,
in the taskification of a code with OmpSs-2 I have encountered an error producing a segfault when the instructions inside a task pass an object by reference, e.g.
for (int i=0;i<N;i++)
{
#pragma oss task firstprivate(i)
{
Params params;
function(..., i, params, ...);
}
}
#pragma oss taskwait
where the definition of function is like void function(...,int i, Params ¶ms). Now, in some cases this gives the correct behaviour, in other cases a segfault is produced, with a stack trace like:
Stack trace (most recent call last) in thread 21375:
#16 Object "[0xffffffffffffffff]", at 0xffffffffffffffff, in
#15 Object "/lib64/libc.so.6", at 0x2ac48370a96c, in clone
#14 Object "/lib64/libpthread.so.0", at 0x2ac48200cea4, in
#13 Object "/gpfs/users/massimof/nanos6/install/lib/libnanos6-optimized-discrete.so.0.0.0", at 0x2ac48597605f, in
#12 Object "/gpfs/users/massimof/nanos6/install/lib/libnanos6-optimized-discrete.so.0.0.0", at 0x2ac485982e8a, in
#11 Object "/gpfs/users/massimof/nanos6/install/lib/libnanos6-optimized-discrete.so.0.0.0", at 0x2ac48597cffb, in
#10 Object "/gpfs/users/massimof/nanos6/install/lib/libnanos6-optimized-discrete.so.0.0.0", at 0x2ac48597cb65, in
#9 Object "/gpfs/workdir/massimof/test_OmpSs-2/./smilei", at 0x7302f8, in
#8 Object "/gpfs/workdir/massimof/test_OmpSs-2/./smilei", at 0x6f6abe, in main
#7 Object "/gpfs/workdir/massimof/test_OmpSs-2/./smilei", at 0x646744, in VectorPatch::dynamics(Params&, SmileiMPI*, SimWindow*, RadiationTables&, MultiphotonBreitWheelerTables&, double, Timers&, int)
#6 Object "/gpfs/users/massimof/nanos6/install/lib/libnanos6-optimized-discrete.so.0.0.0", at 0x2ac485957cd8, in AddTask::submitTask(Task*, Task*, bool)
#5 Object "/gpfs/users/massimof/nanos6/install/lib/libnanos6-optimized-discrete.so.0.0.0", at 0x2ac48597fbde, in
#4 Object "/gpfs/workdir/massimof/test_OmpSs-2/./smilei", at 0x5d6d7c, in Params::~Params()
#3 Object "/lib64/libc.so.6", at 0x2ac48368d298, in
#2 Object "/lib64/libc.so.6", at 0x2ac483684ed6, in
#1 Object "/lib64/libc.so.6", at 0x2ac483643a77, in abort
#0 Object "/lib64/libc.so.6", at 0x2ac483642387, in gsignal
This looks like an error at the generation of the task, right?
The code was compiled using clang built with LLVM-OmpSs. Simple tests of tasks with cout<<"Task A"<<endl; work perfectly.
A workaround was to define auxiliary variables, copies of those variables contained in the object params that were necessary to function, i.e. passing those variables to function without using params in its inputs. In some cases this can be an easy fix, but in other cases I should pass a lot a variables contained in params, restructuring functions in the code. So, are there any limitations / solutions when passing object to functions inside tasks? Or maybe there is something off in my build of Nanos6?
After executing ./configure (with boost 1.70) make returns the following error:
CXXLD libnanos6-optimized-linear-regions-fragmented.la
/tmp/ccf6usLA.ltrans0.ltrans.o: In function `std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream(std::_Ios_Openmode) [clone .constprop.194]':
<artificial>:(.text+0x4c): undefined reference to `VTT for std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >'
<artificial>:(.text+0x78): undefined reference to `vtable for std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >'
<artificial>:(.text+0xc4): undefined reference to `vtable for std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >'
[...]
<artificial>:(.text.unlikely+0x204): undefined reference to `std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_ostringstream()'
<artificial>:(.text.unlikely+0x264): undefined reference to `std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_ostringstream()'
ccollect2: error: ld returned 1 exit status
make[2]: *** [libnanos6-optimized-linear-regions-fragmented.la] Error 1
make[2]: Leaving directory `/u/aragagnin/src/ompss-2-2019.11.2/nanos6-2.3.2'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/u/aragagnin/src/ompss-2-2019.11.2/nanos6-2.3.2'
make: *** [all] Error 2
what could I be missing that prevents the linking of libnanos6-optimized-linear-regions-fragmented.la?
I see that --without-memkind --without-pqos --without-dlb --without-cuda --without-babeltrace2 aren't accepted by configure
Log here: https://786717.bugs.gentoo.org/attachment.cgi?id=703788
libtool: compile: x86_64-pc-linux-gnu-g++ -DHAVE_CONFIG_H -I. -I/usr/include -DBOOST_ENABLE_ASSERT_DEBUG_HANDLER -pthread -I/usr/include -I./src/executors/workflow/cluster/null -I./src/cluster/null -I./src/cluster/offloading/null -I./src/hardware/cluster/null -I./src/hardware/cluster/memory/null -I./src/executors/workflow/null -I./src/support/chronometers/std -I./src/executors/threads/cpu-managers/default -I./src/memory/allocator/malloc -I./src/memory/vmm/smp -I./src/instrument/null -Ino/include -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -O0 -g3 -fstack-protector-all -I./src -I./src -I./api -I. -faligned-new -I./src/executors/threads/kernel-level -I./src/instrument/support/introspection/ElfUtils -I./src/instrument/support/backtrace/backtrace -I./src/dependencies/linear-regions -I./src/dependencies/linear-regions-fragmented -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -c src/hardware-counters/rapl/RAPLHardwareCounters.cpp -fPIC -DPIC -o src/hardware-counters/rapl/.libs/libnanos6_debug_regions_la-RAPLHardwareCounters.o
In file included from src/hardware/places/MemoryPlace.hpp:12,
from src/hardware/places/NUMAPlace.hpp:10,
from src/hardware/places/NUMAPlace.cpp:7:
./src/memory/AddressSpace.hpp:27:9: error: ‘size_t’ does not name a type
27 | size_t getMemoryPlacesCount() const
| ^~~~~~
./src/memory/AddressSpace.hpp:1:1: note: ‘size_t’ is defined in header ‘<cstddef>’; did you forget to ‘#include <cstddef>’?
+++ |+#include <cstddef>
1 | /*
I've tried to run the tests but I got this
>>> Test phase: sys-cluster/nanos6-2.6-r1
make -j12 -l12 check
make check-recursive
make[1]: Entering directory '/var/tmp/portage/sys-cluster/nanos6-2.6-r1/work/nanos6-version-2.6'
Making check in .
make[2]: Entering directory '/var/tmp/portage/sys-cluster/nanos6-2.6-r1/work/nanos6-version-2.6'
make[2]: Leaving directory '/var/tmp/portage/sys-cluster/nanos6-2.6-r1/work/nanos6-version-2.6'
Making check in commands
make[2]: Entering directory '/var/tmp/portage/sys-cluster/nanos6-2.6-r1/work/nanos6-version-2.6/commands'
make[2]: Nothing to be done for 'check'.
make[2]: Leaving directory '/var/tmp/portage/sys-cluster/nanos6-2.6-r1/work/nanos6-version-2.6/commands'
Making check in tests/directive_based/clang
make[2]: Entering directory '/var/tmp/portage/sys-cluster/nanos6-2.6-r1/work/nanos6-version-2.6/tests/directive_based/clang'
make
make[3]: Entering directory '/var/tmp/portage/sys-cluster/nanos6-2.6-r1/work/nanos6-version-2.6/tests/directive_based/clang'
make[3]: Nothing to be done for 'all'.
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/nanos6-2.6-r1/work/nanos6-version-2.6/tests/directive_based/clang'
make check-TESTS
make[3]: Entering directory '/var/tmp/portage/sys-cluster/nanos6-2.6-r1/work/nanos6-version-2.6/tests/directive_based/clang'
make[4]: Entering directory '/var/tmp/portage/sys-cluster/nanos6-2.6-r1/work/nanos6-version-2.6/tests/directive_based/clang'
�[0;32m============================================================================�[m
�[0;32mTestsuite summary for nanos6 2.6�[m
�[0;32m============================================================================�[m
# TOTAL: 0
# PASS: 0
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
�[0;32m============================================================================�[m
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/nanos6-2.6-r1/work/nanos6-version-2.6/tests/directive_based/clang'
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/nanos6-2.6-r1/work/nanos6-version-2.6/tests/directive_based/clang'
make[2]: Leaving directory '/var/tmp/portage/sys-cluster/nanos6-2.6-r1/work/nanos6-version-2.6/tests/directive_based/clang'
Making check in tests/directive_based/mercurium
make[2]: Entering directory '/var/tmp/portage/sys-cluster/nanos6-2.6-r1/work/nanos6-version-2.6/tests/directive_based/mercurium'
make blocking.mercurium.test events.mercurium.test events-dep.mercurium.test onready.mercurium.test onready-events.mercurium.test scheduling-wait-for.mercurium.test fibonacci.mercurium.test dep-nonest.mercurium.test dep-early-release.mercurium.test dep-er-and-weak.mercurium.test if0.mercurium.test dep-wait.mercurium.test simple-commutative.mercurium.test commutative-stencil.mercurium.test task-for-multiaxpy.mercurium.test task-for-dep-multiaxpy.mercurium.test task-for-nonpod.mercurium.test task-for-nqueens.mercurium.test taskloop-multiaxpy.mercurium.test taskloop-dep-multiaxpy.mercurium.test taskloop-nested-dep-multiaxpy.mercurium.test taskloop-nonpod.mercurium.test taskloop-nqueens.mercurium.test taskloop-for-multiaxpy.mercurium.test taskloop-for-dep-multiaxpy.mercurium.test taskloop-for-nested-dep-multiaxpy.mercurium.test taskloop-for-nonpod.mercurium.test taskloop-for-nqueens.mercurium.test taskloop-for-reduction.mercurium.test blocking.mercurium.debug.test events.mercurium.debug.test events-dep.mercurium.debug.test onready.mercurium.debug.test onready-events.mercurium.debug.test scheduling-wait-for.mercurium.debug.test fibonacci.mercurium.debug.test dep-nonest.mercurium.debug.test dep-early-release.mercurium.debug.test dep-er-and-weak.mercurium.debug.test if0.mercurium.debug.test dep-wait.mercurium.debug.test simple-commutative.mercurium.debug.test commutative-stencil.mercurium.debug.test task-for-multiaxpy.mercurium.debug.test task-for-dep-multiaxpy.mercurium.debug.test task-for-nonpod.mercurium.debug.test task-for-nqueens.mercurium.debug.test taskloop-multiaxpy.mercurium.debug.test taskloop-dep-multiaxpy.mercurium.debug.test taskloop-nested-dep-multiaxpy.mercurium.debug.test taskloop-nonpod.mercurium.debug.test taskloop-nqueens.mercurium.debug.test taskloop-for-multiaxpy.mercurium.debug.test taskloop-for-dep-multiaxpy.mercurium.debug.test taskloop-for-nested-dep-multiaxpy.mercurium.debug.test taskloop-for-nonpod.mercurium.debug.test taskloop-for-nqueens.mercurium.debug.test taskloop-for-reduction.mercurium.debug.test critical.mercurium.test critical.mercurium.debug.test lr-nonest.mercurium.test lr-nonest-upgrades.mercurium.test lr-early-release.mercurium.test lr-er-and-weak.mercurium.test lr-release.mercurium.test lr-nonest.mercurium.debug.test lr-nonest-upgrades.mercurium.debug.test lr-early-release.mercurium.debug.test lr-er-and-weak.mercurium.debug.test lr-release.mercurium.debug.test red-firstprivate.mercurium.test red-nonest.mercurium.test red-nest.mercurium.test red-nest-other.mercurium.test red-nqueens.mercurium.test red-stress.mercurium.test red-firstprivate.mercurium.debug.test red-nonest.mercurium.debug.test red-nest.mercurium.debug.test red-nest-other.mercurium.debug.test red-nqueens.mercurium.debug.test red-stress.mercurium.debug.test discrete-deps.mercurium.test discrete-deps-nonest.mercurium.test discrete-deps-nonest.mercurium.test discrete-deps-early-release.mercurium.test discrete-deps-er-and-weak.mercurium.test discrete-deps-wait.mercurium.test discrete-release.mercurium.test discrete-simple-commutative.mercurium.test discrete-red-stress.mercurium.test discrete-taskloop-multiaxpy.mercurium.test discrete-taskloop-dep-multiaxpy.mercurium.test discrete-taskloop-nested-dep-multiaxpy.mercurium.test discrete-taskloop-nonpod.mercurium.test discrete-taskloop-nqueens.mercurium.test discrete-taskloop-for-multiaxpy.mercurium.test discrete-taskloop-for-dep-multiaxpy.mercurium.test discrete-taskloop-for-nested-dep-multiaxpy.mercurium.test discrete-taskloop-for-nonpod.mercurium.test discrete-taskloop-for-nqueens.mercurium.test discrete-taskloop-for-reduction.mercurium.test discrete-deps.mercurium.debug.test discrete-deps-nonest.mercurium.debug.test discrete-deps-nonest.mercurium.debug.test discrete-deps-early-release.mercurium.debug.test discrete-deps-er-and-weak.mercurium.debug.test discrete-deps-wait.mercurium.debug.test discrete-release.mercurium.debug.test discrete-simple-commutative.mercurium.debug.test discrete-red-stress.mercurium.debug.test discrete-taskloop-multiaxpy.mercurium.debug.test discrete-taskloop-dep-multiaxpy.mercurium.debug.test discrete-taskloop-nested-dep-multiaxpy.mercurium.debug.test discrete-taskloop-nonpod.mercurium.debug.test discrete-taskloop-nqueens.mercurium.debug.test discrete-taskloop-for-multiaxpy.mercurium.debug.test discrete-taskloop-for-dep-multiaxpy.mercurium.debug.test discrete-taskloop-for-nested-dep-multiaxpy.mercurium.debug.test discrete-taskloop-for-nonpod.mercurium.debug.test discrete-taskloop-for-nqueens.mercurium.debug.test discrete-taskloop-for-reduction.mercurium.debug.test numa-allocations.mercurium.test numa-auto.mercurium.test numa-bitmask-manip.mercurium.test numa-irregular-allocations.mercurium.test numa-off.mercurium.test numa-on.mercurium.test numa-wildcards.mercurium.test numa-allocations.mercurium.debug.test numa-auto.mercurium.debug.test numa-bitmask-manip.mercurium.debug.test numa-irregular-allocations.mercurium.debug.test numa-off.mercurium.debug.test numa-on.mercurium.debug.test numa-wildcards.mercurium.debug.test dlb-cpu-management.mercurium.test dlb-cpu-sharing-active-process.mercurium.test dlb-cpu-sharing-passive-process.mercurium.test dlb-cpu-sharing.mercurium.test dlb-cpu-management.mercurium.debug.test dlb-cpu-sharing-active-process.mercurium.debug.test dlb-cpu-sharing-passive-process.mercurium.debug.test dlb-cpu-sharing.mercurium.debug.test
make[3]: Entering directory '/var/tmp/portage/sys-cluster/nanos6-2.6-r1/work/nanos6-version-2.6/tests/directive_based/mercurium'
/usr/bin/mcxx --ompss-2 -std=c++03 --parallel -DHAVE_CONFIG_H -I. -I../../.. -I/usr/include -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -O0 -g3 -fstack-protector-all -I../../../tests -pthread --no-default-nanos6-inc --no-default-nanos6-lib -I../../../api -I../../.. -include nanos6.h -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -c -o ../blocking/blocking_mercurium_debug_test-blocking.o `test -f '../blocking/blocking.cpp' || echo './'`../blocking/blocking.cpp
/usr/bin/mcxx --ompss-2 -std=c++03 --parallel -DHAVE_CONFIG_H -I. -I../../.. -I/usr/include -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -O0 -g3 -fstack-protector-all -I../../../tests -pthread --no-default-nanos6-inc --no-default-nanos6-lib -I../../../api -I../../.. -include nanos6.h -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -c -o ../events/events_mercurium_debug_test-events.o `test -f '../events/events.cpp' || echo './'`../events/events.cpp
/usr/bin/mcxx --ompss-2 -std=c++03 --parallel -DHAVE_CONFIG_H -I. -I../../.. -I/usr/include -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -O0 -g3 -fstack-protector-all -I../../../tests -pthread --no-default-nanos6-inc --no-default-nanos6-lib -I../../../api -I../../.. -include nanos6.h -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -c -o ../events/events_dep_mercurium_debug_test-events-dep.o `test -f '../events/events-dep.cpp' || echo './'`../events/events-dep.cpp
/usr/bin/mcxx --ompss-2 -std=c++03 --parallel -DHAVE_CONFIG_H -I. -I../../.. -I/usr/include -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -O0 -g3 -fstack-protector-all -I../../../tests -pthread --no-default-nanos6-inc --no-default-nanos6-lib -I../../../api -I../../.. -include nanos6.h -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -c -o ../onready/onready_mercurium_debug_test-onready.o `test -f '../onready/onready.cpp' || echo './'`../onready/onready.cpp
/usr/bin/mcxx --ompss-2 -std=c++03 --parallel -DHAVE_CONFIG_H -I. -I../../.. -I/usr/include -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -O0 -g3 -fstack-protector-all -I../../../tests -pthread --no-default-nanos6-inc --no-default-nanos6-lib -I../../../api -I../../.. -include nanos6.h -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -c -o ../onready/onready_events_mercurium_debug_test-onready-events.o `test -f '../onready/onready-events.cpp' || echo './'`../onready/onready-events.cpp
/usr/bin/mcxx --ompss-2 -std=c++03 --parallel -DHAVE_CONFIG_H -I. -I../../.. -I/usr/include -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -O0 -g3 -fstack-protector-all -I../../../tests -pthread --no-default-nanos6-inc --no-default-nanos6-lib -I../../../api -I../../.. -include nanos6.h -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -c -o ../scheduling/scheduling_wait_for_mercurium_debug_test-scheduling-wait-for.o `test -f '../scheduling/scheduling-wait-for.cpp' || echo './'`../scheduling/scheduling-wait-for.cpp
/usr/bin/mcxx --ompss-2 -std=c++03 --parallel -DHAVE_CONFIG_H -I. -I../../.. -I/usr/include -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -O0 -g3 -fstack-protector-all -I../../../tests -pthread --no-default-nanos6-inc --no-default-nanos6-lib -I../../../api -I../../.. -include nanos6.h -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -c -o ../fibonacci/fibonacci_mercurium_debug_test-fibonacci.o `test -f '../fibonacci/fibonacci.cpp' || echo './'`../fibonacci/fibonacci.cpp
/usr/bin/mcxx --ompss-2 -std=c++03 --parallel -DHAVE_CONFIG_H -I. -I../../.. -DHAVE_CONCURRENT_SUPPORT -DHAVE_DLB -I/usr/include -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -O0 -g3 -fstack-protector-all -I../../../tests -pthread --no-default-nanos6-inc --no-default-nanos6-lib -I../../../api -I../../.. -include nanos6.h -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -c -o ../dependencies/dep_nonest_mercurium_debug_test-dep-nonest.o `test -f '../dependencies/dep-nonest.cpp' || echo './'`../dependencies/dep-nonest.cpp
/usr/bin/mcxx --ompss-2 -std=c++03 --parallel -DHAVE_CONFIG_H -I. -I../../.. -I/usr/include -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -O0 -g3 -fstack-protector-all -I../../../tests -pthread --no-default-nanos6-inc --no-default-nanos6-lib -I../../../api -I../../.. -include nanos6.h -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -c -o ../dependencies/dep_early_release_mercurium_debug_test-dep-early-release.o `test -f '../dependencies/dep-early-release.cpp' || echo './'`../dependencies/dep-early-release.cpp
/usr/bin/mcxx --ompss-2 -std=c++03 --parallel -DHAVE_CONFIG_H -I. -I../../.. -I/usr/include -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -O0 -g3 -fstack-protector-all -I../../../tests -pthread --no-default-nanos6-inc --no-default-nanos6-lib -I../../../api -I../../.. -include nanos6.h -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -c -o ../dependencies/dep_er_and_weak_mercurium_debug_test-dep-er-and-weak.o `test -f '../dependencies/dep-er-and-weak.cpp' || echo './'`../dependencies/dep-er-and-weak.cpp
/usr/bin/mcxx --ompss-2 -std=c++03 --parallel -DHAVE_CONFIG_H -I. -I../../.. -I/usr/include -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -O0 -g3 -fstack-protector-all -I../../../tests -pthread --no-default-nanos6-inc --no-default-nanos6-lib -I../../../api -I../../.. -include nanos6.h -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -c -o ../dependencies/if0_mercurium_debug_test-if0.o `test -f '../dependencies/if0.cpp' || echo './'`../dependencies/if0.cpp
/usr/bin/mcxx --ompss-2 -std=c++03 --parallel -DHAVE_CONFIG_H -I. -I../../.. -I/usr/include -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -O0 -g3 -fstack-protector-all -I../../../tests -pthread --no-default-nanos6-inc --no-default-nanos6-lib -I../../../api -I../../.. -include nanos6.h -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -c -o ../dependencies/dep_wait_mercurium_debug_test-dep-wait.o `test -f '../dependencies/dep-wait.cpp' || echo './'`../dependencies/dep-wait.cpp
Welcome to Extrae 3.8.3
Extrae: Application has been linked or preloaded with Extrae, BUT neither EXTRAE_ON nor EXTRAE_CONFIG_FILE are set!
Welcome to Extrae 3.8.3
Extrae: Application has been linked or preloaded with Extrae, BUT neither EXTRAE_ON nor EXTRAE_CONFIG_FILE are set!
Welcome to Extrae 3.8.3
Extrae: Application has been linked or preloaded with Extrae, BUT neither EXTRAE_ON nor EXTRAE_CONFIG_FILE are set!
Welcome to Extrae 3.8.3
Extrae: Application has been linked or preloaded with Extrae, BUT neither EXTRAE_ON nor EXTRAE_CONFIG_FILE are set!
Welcome to Extrae 3.8.3
Extrae: Application has been linked or preloaded with Extrae, BUT neither EXTRAE_ON nor EXTRAE_CONFIG_FILE are set!
Welcome to Extrae 3.8.3
Welcome to Extrae 3.8.3
Extrae: Application has been linked or preloaded with Extrae, BUT neither EXTRAE_ON nor EXTRAE_CONFIG_FILE are set!
Extrae: Application has been linked or preloaded with Extrae, BUT neither EXTRAE_ON nor EXTRAE_CONFIG_FILE are set!
Welcome to Extrae 3.8.3
Extrae: Application has been linked or preloaded with Extrae, BUT neither EXTRAE_ON nor EXTRAE_CONFIG_FILE are set!
Welcome to Extrae 3.8.3
Extrae: Application has been linked or preloaded with Extrae, BUT neither EXTRAE_ON nor EXTRAE_CONFIG_FILE are set!
Welcome to Extrae 3.8.3
Extrae: Application has been linked or preloaded with Extrae, BUT neither EXTRAE_ON nor EXTRAE_CONFIG_FILE are set!
Welcome to Extrae 3.8.3
Extrae: Application has been linked or preloaded with Extrae, BUT neither EXTRAE_ON nor EXTRAE_CONFIG_FILE are set!
Welcome to Extrae 3.8.3
Extrae: Application has been linked or preloaded with Extrae, BUT neither EXTRAE_ON nor EXTRAE_CONFIG_FILE are set!
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_iterator_base_funcs.h:151:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: called name '__builtin_is_constant_evaluated' not found in the current scope
make[3]: *** [Makefile:5267: ../dependencies/dep_wait_mercurium_debug_test-dep-wait.o] Error 1
make[3]: *** Waiting for unfinished jobs....
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
make[3]: *** [Makefile:5995: ../fibonacci/fibonacci_mercurium_debug_test-fibonacci.o] Error 1
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: called name '__builtin_is_constant_evaluated' not found in the current scope
make[3]: *** [Makefile:6415: ../onready/onready_mercurium_debug_test-onready.o] Error 1
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_tree.h:1216:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_tree.h:1216:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_tree.h:1223:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/stl_tree.h:1223:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: called name '__builtin_is_constant_evaluated' not found in the current scope
make[3]: *** [Makefile:5939: ../events/events_dep_mercurium_debug_test-events-dep.o] Error 1
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1039:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/basic_string.h:1058:11: error: called name '__builtin_is_constant_evaluated' not found in the current scope
make[3]: *** [Makefile:5211: ../dependencies/dep_er_and_weak_mercurium_debug_test-dep-er-and-weak.o] Error 1
make[3]: *** [Makefile:5183: ../dependencies/dep_early_release_mercurium_debug_test-dep-early-release.o] Error 1
make[3]: *** [Makefile:6023: ../dependencies/if0_mercurium_debug_test-if0.o] Error 1
make[3]: *** [Makefile:5043: ../blocking/blocking_mercurium_debug_test-blocking.o] Error 1
make[3]: *** [Makefile:5967: ../events/events_mercurium_debug_test-events.o] Error 1
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/bits/streambuf_iterator.h:466:16: error: called name '__builtin_is_constant_evaluated' not found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: called name '__builtin_is_constant_evaluated' not found in the current scope
make[3]: *** [Makefile:6387: ../onready/onready_events_mercurium_debug_test-onready-events.o] Error 1
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: type name '__builtin_is_constant_evaluated' has not been found in the current scope
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/g++-v11/istream:809:13: error: called name '__builtin_is_constant_evaluated' not found in the current scope
make[3]: *** [Makefile:6611: ../scheduling/scheduling_wait_for_mercurium_debug_test-scheduling-wait-for.o] Error 1
make[3]: *** [Makefile:5239: ../dependencies/dep_nonest_mercurium_debug_test-dep-nonest.o] Error 1
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/nanos6-2.6-r1/work/nanos6-version-2.6/tests/directive_based/mercurium'
make[2]: *** [Makefile:7330: check-am] Error 2
make[2]: Leaving directory '/var/tmp/portage/sys-cluster/nanos6-2.6-r1/work/nanos6-version-2.6/tests/directive_based/mercurium'
make[1]: *** [Makefile:40652: check-recursive] Error 1
make[1]: Leaving directory '/var/tmp/portage/sys-cluster/nanos6-2.6-r1/work/nanos6-version-2.6'
make: *** [Makefile:40949: check] Error 2
Hello,
in this minimal example code I have two for loops on the iterator ibin. Each for loop is executing taskA or taskB for the correspondent ibin. The execution of taskB on a given ibin is dependent on the execution of taskA on the same ibin. Thus a firstprivate(ibin) data sharing attribute is necessary to ensure the correctness of the procedure.
To express these dependencies, an array of dependencies has_done_taskA[ibin] or similar is used, to include the corresponding array element in the depend clause. Now, in the minimal example code this is done in 4 different ways:
1 - bin_has_done_taskA array is allocated statically in a function containing tasks;
2 - same as 1, but the array is defined dynamically;
3 - bin_has_done_taskA array is a member of a class and is allocated statically, the tasks are called within a class method;
4 - same as 4, but the array is allocated dynamically in the constructor of the class.
Methods 1,2,3 compile and work correctly, but method 4 gives this error:
error: the data-sharing 'firstprivate' conflicts with 'shared' required by the dependency
#pragma oss task firstprivate(ibin) out(bin_has_done_taskA[ibin])
^~~~
Reading the section Data Sharing Attributes in the documentation (https://pm.bsc.es/ftp/ompss-2/doc/spec/OmpSs-2-Specification.pdf) I can't find an explanation for this different behaviour. For example, if Dynamic storage duration objects are shared (e.g., malloc, new,...). as stated, shouldn't method 2 also fail?
In the code I want to taskify I use a class as in method 4, but I can easily change the procedure to method 3, hoping that allocating a dependency array each time does not worsen the performances. In any case I would like to understand better when I cannot use firstprivate with a dependency array.
Here is my minimal example code:
#include <stdio.h>
#include <iostream>
using namespace std;
void executeTasksStaticDependencyArray(int Nbin){ //compiles
int bin_has_done_taskA[Nbin];
for (int ibin = 0;ibin<Nbin;ibin++){
#pragma oss task firstprivate(ibin) out(bin_has_done_taskA[ibin])
cout<<"task0 from ibin="<<ibin<<endl;
}
for (int ibin = 0;ibin<Nbin;ibin++){
#pragma oss task firstprivate(ibin) in(bin_has_done_taskA[ibin])
cout<<"task1 from ibin="<<ibin<<endl;
}
#pragma oss taskwait
}
void executeTasksDynamicDependencyArray(int Nbin){ //compiles
int *bin_has_done_taskA;
bin_has_done_taskA = new int[Nbin];
for (int ibin = 0;ibin<Nbin;ibin++){
#pragma oss task firstprivate(ibin) out(bin_has_done_taskA[ibin])
cout<<"taskA from ibin="<<ibin<<endl;
}
for (int ibin = 0;ibin<Nbin;ibin++){
#pragma oss task firstprivate(ibin) in(bin_has_done_taskA[ibin])
cout<<"taskB from ibin="<<ibin<<endl;
}
#pragma oss taskwait
}
class testClass{
public:
testClass(int N_bins){
Nbin=N_bins;
bin_has_done_taskA = new int[Nbin];
};
~testClass(){delete bin_has_done_taskA;}
void classExecuteTasksStaticDependencyArray(){// compiles
int this_bin_has_done_taskA[Nbin];
for (int ibin = 0;ibin<Nbin;ibin++){
#pragma oss task firstprivate(ibin) out(this_bin_has_done_taskA[ibin])
cout<<"taskA from ibin="<<ibin<<endl;
}
for (int ibin = 0;ibin<Nbin;ibin++){
#pragma oss task firstprivate(ibin) in(this_bin_has_done_taskA[ibin])
cout<<"taskB from ibin="<<ibin<<endl;
}
#pragma oss taskwait
}
void classExecuteTasksDynamicDependencyArray(){ // does not compile
for (int ibin = 0;ibin<Nbin;ibin++){ // firstprivate gives an error
#pragma oss task firstprivate(ibin) out(bin_has_done_taskA[ibin])
cout<<"taskA from ibin="<<ibin<<endl;
}
for (int ibin = 0;ibin<Nbin;ibin++){ // firstprivate gives an error
#pragma oss task firstprivate(ibin) in(bin_has_done_taskA[ibin])
cout<<"taskB from ibin="<<ibin<<endl;
}
#pragma oss taskwait
}
int Nbin;
int *bin_has_done_taskA;
};
int main(int argc, char *argv[])
{
int Nbin = 3;
cout<<"Doing it with static allocation"<<endl;
executeTasksStaticDependencyArray(Nbin);
cout<<"Doing it with dynamic allocation"<<endl;
executeTasksDynamicDependencyArray(Nbin);
cout<<"Doing it with a class and static allocation"<<endl;
testClass test_Class(Nbin);
test_Class.classExecuteTasksStaticDependencyArray();
cout<<"Doing it with a class and dynamic allocation"<<endl;
test_Class.classExecuteTasksDynamicDependencyArray();
return 0;
}
libtool: compile: x86_64-pc-linux-gnu-g++ -DHAVE_CONFIG_H -I. -I/usr/include -DBOOST_ENABLE_ASSERT_DEBUG_HANDLER -pthread -I/usr/include -I./src/executors/threads/cpu-managers/default -I/usr/include -DHAVE_PQOS -I./src/memory/allocator/malloc -I./src/instrument/ctf -I/usr/include -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -O0 -g3 -fstack-protector-all -I./src -I./src -I./api -I. -faligned-new -I./src/executors/threads/kernel-level -I./src/instrument/support/introspection/ElfUtils -I./src/instrument/support/backtrace/libunwind -I./src/dependencies/linear-regions -I./src/dependencies/discrete -DDISCRETE_DEPS -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -c src/executors/threads/CPUManager.cpp -fPIC -DPIC -o src/executors/threads/.libs/libnanos6_debug_discrete_ctf_la-CPUManager.o /bin/sh ./libtool --tag=CXX --mode=compile x86_64-pc-linux-gnu-g++ -DHAVE_CONFIG_H -I. -I/usr/include -DBOOST_ENABLE_ASSERT_DEBUG_HANDLER -pthread -I/usr/include -I./src/executors/threads/cpu-managers/default -I/usr/include -DH
AVE_PQOS -I./src/memory/allocator/malloc -I./src/instrument/ctf -I/usr/include -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -O0 -g3 -fstack-protector-all -I./src -I./src -I./api -I. -faligned-new -I./src/executors/threads/kernel-level -I./src/instrument/support/introspection/ElfUtils -I./src/instrument/support/backtrace/libunwind -I./src/dependencies/linear-regions -I./src/dependencies/discrete -DDISCRETE_DEPS -Wall -Wextra -Wdisabled-opti
mization -Wshadow -fvisibility=hidden -c -o src/executors/threads/libnanos6_debug_discrete_ctf_la-CPUManagerInterface.lo `test -f 'src/executors/threads/CPUManagerInterface.cpp' || echo './'`src/executors/threads/CPUManagerInterface.cpplibtool: compile: x86_64-pc-linux-gnu-g++ -DHAVE_CONFIG_H -I. -I/usr/include -DBOOST_ENABLE_ASSERT_DEBUG_HANDLER -pthread -I/usr/include -I./src/executors/threads/cpu-managers/default -I/usr/include -DHAVE_PQOS -I./src/memory/allocator
/malloc -I./src/instrument/ctf -I/usr/include -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -O0 -g3 -fstack-protector-all -I./src -I./src -I./api -I. -faligned-new -I./src/executors/threads/kernel-level -I./src/inst
rument/support/introspection/ElfUtils -I./src/instrument/support/backtrace/libunwind -I./src/dependencies/linear-regions -I./src/dependencies/discrete -DDISCRETE_DEPS -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -c
src/executors/threads/CPUManagerInterface.cpp -fPIC -DPIC -o src/executors/threads/.libs/libnanos6_debug_discrete_ctf_la-CPUManagerInterface.o
In file included from src/hardware-counters/pqos/PQoSHardwareCounters.cpp:11:
src/hardware-counters/pqos/PQoSHardwareCounters.hpp:58:28: error: ‘string’ in namespace ‘std’ does not name a type
58 | const std::string &,
| ^~~~~~
src/hardware-counters/pqos/PQoSHardwareCounters.hpp:17:1: note: ‘std::string’ is defined in header ‘<string>’; did you forget to ‘#include <string>’?
16 | #include "hardware-counters/ThreadHardwareCountersInterface.hpp"
+++ |+#include <string>
17 |
/bin/sh ./libtool --tag=CXX --mode=compile x86_64-pc-linux-gnu-g++ -DHAVE_CONFIG_H -I. -I/usr/include -DBOOST_ENABLE_ASSERT_DEBUG_HANDLER -pthread -I/usr/include -I./src/executors/threads/cpu-managers/default -I/usr/include -DH
AVE_PQOS -I./src/memory/allocator/malloc -I./src/instrument/ctf -I/usr/include -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -O0 -g3 -fstack-protector-all -I./src -I./src -I./api -I. -faligned-new -I./src/executo
rs/threads/kernel-level -I./src/instrument/support/introspection/ElfUtils -I./src/instrument/support/backtrace/libunwind -I./src/dependencies/linear-regions -I./src/dependencies/discrete -DDISCRETE_DEPS -Wall -Wextra -Wdisabled-opti
mization -Wshadow -fvisibility=hidden -c -o src/executors/threads/libnanos6_debug_discrete_ctf_la-ThreadManager.lo `test -f 'src/executors/threads/ThreadManager.cpp' || echo './'`src/executors/threads/ThreadManager.cpp
libtool: compile: x86_64-pc-linux-gnu-g++ -DHAVE_CONFIG_H -I. -I/usr/include -DBOOST_ENABLE_ASSERT_DEBUG_HANDLER -pthread -I/usr/include -I./src/executors/threads/cpu-managers/default -I/usr/include -DHAVE_PQOS -I./src/memory/allocator
/malloc -I./src/instrument/ctf -I/usr/include -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -O0 -g3 -fstack-protector-all -I./src -I./src -I./api -I. -faligned-new -I./src/executors/threads/kernel-level -I./src/inst
rument/support/introspection/ElfUtils -I./src/instrument/support/backtrace/libunwind -I./src/dependencies/linear-regions -I./src/dependencies/discrete -DDISCRETE_DEPS -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -c
src/executors/threads/ThreadManager.cpp -fPIC -DPIC -o src/executors/threads/.libs/libnanos6_debug_discrete_ctf_la-ThreadManager.o
/bin/sh ./libtool --tag=CXX --mode=compile x86_64-pc-linux-gnu-g++ -DHAVE_CONFIG_H -I. -I/usr/include -DBOOST_ENABLE_ASSERT_DEBUG_HANDLER -pthread -I/usr/include -I./src/executors/threads/cpu-managers/default -I/usr/include -DH
AVE_PQOS -I./src/memory/allocator/malloc -I./src/instrument/ctf -I/usr/include -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -O0 -g3 -fstack-protector-all -I./src -I./src -I./api -I. -faligned-new -I./src/executo
rs/threads/kernel-level -I./src/instrument/support/introspection/ElfUtils -I./src/instrument/support/backtrace/libunwind -I./src/dependencies/linear-regions -I./src/dependencies/discrete -DDISCRETE_DEPS -Wall -Wextra -Wdisabled-opti
mization -Wshadow -fvisibility=hidden -c -o src/executors/threads/libnanos6_debug_discrete_ctf_la-WorkerThread.lo `test -f 'src/executors/threads/WorkerThread.cpp' || echo './'`src/executors/threads/WorkerThread.cpp
libtool: compile: x86_64-pc-linux-gnu-g++ -DHAVE_CONFIG_H -I. -I/usr/include -DBOOST_ENABLE_ASSERT_DEBUG_HANDLER -pthread -I/usr/include -I./src/executors/threads/cpu-managers/default -I/usr/include -DHAVE_PQOS -I./src/memory/allocator
/malloc -I./src/instrument/ctf -I/usr/include -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -O0 -g3 -fstack-protector-all -I./src -I./src -I./api -I. -faligned-new -I./src/executors/threads/kernel-level -I./src/inst
rument/support/introspection/ElfUtils -I./src/instrument/support/backtrace/libunwind -I./src/dependencies/linear-regions -I./src/dependencies/discrete -DDISCRETE_DEPS -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -c
src/executors/threads/WorkerThread.cpp -fPIC -DPIC -o src/executors/threads/.libs/libnanos6_debug_discrete_ctf_la-WorkerThread.o
/bin/sh ./libtool --tag=CXX --mode=compile x86_64-pc-linux-gnu-g++ -DHAVE_CONFIG_H -I. -I/usr/include -DBOOST_ENABLE_ASSERT_DEBUG_HANDLER -pthread -I/usr/include -I./src/executors/threads/cpu-managers/default -I/usr/include -DH
AVE_PQOS -I./src/memory/allocator/malloc -I./src/instrument/ctf -I/usr/include -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -O0 -g3 -fstack-protector-all -I./src -I./src -I./api -I. -faligned-new -I./src/executo
rs/threads/kernel-level -I./src/instrument/support/introspection/ElfUtils -I./src/instrument/support/backtrace/libunwind -I./src/dependencies/linear-regions -I./src/dependencies/discrete -DDISCRETE_DEPS -Wall -Wextra -Wdisabled-opti
mization -Wshadow -fvisibility=hidden -c -o src/executors/threads/cpu-managers/default/libnanos6_debug_discrete_ctf_la-DefaultCPUManager.lo `test -f 'src/executors/threads/cpu-managers/default/DefaultCPUManager.cpp' || echo './'`src/exe
cutors/threads/cpu-managers/default/DefaultCPUManager.cpp
libtool: compile: x86_64-pc-linux-gnu-g++ -DHAVE_CONFIG_H -I. -I/usr/include -DBOOST_ENABLE_ASSERT_DEBUG_HANDLER -pthread -I/usr/include -I./src/executors/threads/cpu-managers/default -I/usr/include -DHAVE_PQOS -I./src/memory/allocator
/malloc -I./src/instrument/ctf -I/usr/include -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -O0 -g3 -fstack-protector-all -I./src -I./src -I./api -I. -faligned-new -I./src/executors/threads/kernel-level -I./src/inst
rument/support/introspection/ElfUtils -I./src/instrument/support/backtrace/libunwind -I./src/dependencies/linear-regions -I./src/dependencies/discrete -DDISCRETE_DEPS -Wall -Wextra -Wdisabled-optimization -Wshadow -fvisibility=hidden -c
src/executors/threads/cpu-managers/default/DefaultCPUManager.cpp -fPIC -DPIC -o src/executors/threads/cpu-managers/default/.libs/libnanos6_debug_discrete_ctf_la-DefaultCPUManager.o
src/hardware-counters/pqos/PQoSHardwareCounters.cpp:20:1: error: no declaration matches ‘PQoSHardwareCounters::PQoSHardwareCounters(bool, const string&, std::vector<HWCounters::counters_t>&)’
20 | PQoSHardwareCounters::PQoSHardwareCounters(
| ^~~~~~~~~~~~~~~~~~~~
In file included from src/hardware-counters/pqos/PQoSHardwareCounters.cpp:11:
src/hardware-counters/pqos/PQoSHardwareCounters.hpp:19:7: note: candidates are: ‘constexpr PQoSHardwareCounters::PQoSHardwareCounters(const PQoSHardwareCounters&)’
19 | class PQoSHardwareCounters : public HardwareCountersInterface {
| ^~~~~~~~~~~~~~~~~~~~
src/hardware-counters/pqos/PQoSHardwareCounters.hpp:56:9: note: ‘PQoSHardwareCounters::PQoSHardwareCounters(bool, const int&, std::vector<HWCounters::counters_t>&)’
56 | PQoSHardwareCounters(
| ^~~~~~~~~~~~~~~~~~~~
src/hardware-counters/pqos/PQoSHardwareCounters.hpp:19:7: note: ‘class PQoSHardwareCounters’ defined here
19 | class PQoSHardwareCounters : public HardwareCountersInterface {
| ^~~~~~~~~~~~~~~~~~~~
Is it possible to use system jemalloc?
Why it need jemalloc built with the nanos6_je_ prefix?
I am launching my taskified code on a cluster with SLURM, but I don't see a sensitive reduction of the time spent on the parallelized part increasing the cpus-per-task. The physical results on my benchmark seem to be the same, so it's only a problem of efficient parallelization.
I have used the same taskification strategy used with other task programming systems (OpenMP task, Eventify), so from those tests I know that normally the execution time should decrease with the number of CPU cores in my benchmark. With OpenMP I have to set also the environment variable for the number of threads; with Eventify it is sufficient to increase the cpus-per-task in the submission script. So I am wondering if I am missing something in my submission script for the job. Is there a minimal test I can run to check the scaling of the execution time with my installation, environment, cluster?
My submission script:
#!/bin/bash
#SBATCH --job-name=smileiompss
#SBATCH --output=smilei.log
#SBATCH --time=01:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --nodes=1
#SBATCH --partition=cpu_short
# Clean and to Load the same modules at the compilation phases
module purge
module load zlib/1.2.9/gcc-9.2.0
module load anaconda2/2019.10/gcc-9.2.0
module load gcc/11.2.0/gcc-4.8.5
module load hdf5/1.10.7/gcc-9.2.0-openmpi
module load openmpi/4.0.2/gcc-9.2.0
export INSTALLATION_PREFIX_NANOS6=/gpfs/users/massimof/nanos6/install
export HDF5_ROOT_DIR=/gpfs/softs/spack/opt/spack/linux-centos7-cascadelake/gcc-9.2.0/hdf5-1.10.7-j6enx3ruksp5mjo5ed6lhywrzozihamq
export LD_LIBRARY_PATH=/gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install/lib:$LD_LIBRARY_PATH
export I_MPI_CXX=/gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install/bin/clang++
export OMPI_CXX=/gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install/bin/clang++
# echo of commands
set -x
# To compute in the submission directory
cd ${SLURM_SUBMIT_DIR}
# execution
srun ./smilei tst2d_02_radiation_pressure_acc.py
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.