uwhustle / hustle Goto Github PK

View Code? Open in Web Editor NEW

41.0 8.0 6.0 14.14 MB

In-memory, columnar, arrow-based database.

License: Apache License 2.0

Ruby 0.01% CMake 1.67% Shell 0.62% C++ 94.06% Python 0.79% C 2.87%

database columnar apache-arrow sqlite parallel-processing olap htap in-memory-database

hustle's People

Contributors

Stargazers

Watchers

Forkers

gindachen srsuryadev ichn-hu mprammer fytzzh chuanlei

hustle's Issues

DBTable::InsertRecords - Can we loop the columns then the rows?

Can we loop the column type then over rows? If the lambda function is not inlined, we may pay a huge cost here.

Originally posted by @GindaChen in #105 (comment)

Clean up build script for C++17/20

As mentioned in #99, we need to clean up the build script such that building in Ubuntu and macOS, with different language, can be easily done.

Major goals include:

Insulate brew dependency on macOS if the compiler is g++. Almost all tools in brew are built with clang, and if we need to use g++ we need to build most of them under the cmake toolchain. This is pretty doable.
Clean the install script and allow language version as an easy input. For example, ./build.sh --cpp20 should give us a full build on C++20. Suggestions are welcomed.

Doxygen Style Comments

Not all of my comments follow the Doxygen style. Comments also need to be added to the .cpp files.

NNG shared ptr

I am going to write my comments here so we have a history of the conversation.

Sending a shared pointer
https://github.com/UWQuickstep/hustle/blob/f2fa7582b2314ce0956aa61ded1ef6b36f3bcd92/src/main.cpp#L28

What is happening in this case, namely that the Block object contained in the shared pointer is deallocated before the receiver receives it, is the expected behavior. A shared pointer is alive while some in scope piece of code has a "name" for that shared pointer. (https://abseil.io/tips/55)

In the send/receive code when you write the value of the shared pointer to the queue and before the receiver receives it the reference counter goes to 0 and the Block object is deallocated.
There is no way for nng to know that what you are sending is a shared pointer, it never gets that information, bytes are sent.

How is this communication system going to be used? What data are we going to send? Who is going to own the data we send/receive? (maybe these questions will help you/us resolve this problem)

One solution could be to have the blocks registered with a block manager or some other structure that does not go out of scope.
Does nng have a smarter send/receive pair of methods that are aware of the type of content send?

Raw Pointer Implementation
https://github.com/UWQuickstep/hustle/blob/44511ebceb4d399a2fa8687d9d0ca1a6218edf7a/src/main.cpp#L12

I am seeing that the pointer changes, but there is no additional call to the Block constructor (right?). Can you verify with Valgrind or another memory profiler that the memory usage is doubled? This is indeed very weird.

HashAggregate

This issue extends #57 to make sure HashAggregate delivers the following:

Replace the std::unordered_map hash tables to phmap (ideally) for better performance.
Add a toggle in AggregateStrategy to flip between HashAggregate and Aggregate. We can also introduce this toggle in ssb_workload.cc and ssb_workload_lip.cc to compare the performance.
Handle hash conflict more elegantly. (Using phmap and custom hash_combine function can eliminate the problem).
Design an abstraction over the hash table, and remove some redundant type-checking codes of the arrow arrays.
Refactor arrow_compute_wrapper and add some helpers to optimize the array construction.
Eliminate most data copy between std:: and arrow::.
No memory leak. Free all memory after the Finish().
Make code readable: use less auto

Operator plan reuse - (Query Compilation)

ExecutionPlan - clear and reset (set/any simple interface initialize)
Unit tests - (Older SSB benchmark on creating a plan with basic operators - create similar tests for few queries -> see if this works ->

Create plan for the query
Submit the plan to the scheduler
Clear and reinitialize the plan
Resubmit and see if it still returns the correct result.

Understand at the high-level about the techniques used sqlite3 (sqlite3_prepare_v2(), sqlite3_step(), sqlite3_clear_bindings(), sqlite3_reset())

Reference links
Sample Manual construction of the operator plan ->

hustle/src/benchmark/ssb_workload_lip.cc

Line 734 in df14c96

SelectBuildHash s_select_op(0, s, s_result_in, s_select_result_out,

Execution Plan -> https://github.com/UWHustle/hustle/blob/df14c968e91e64babf15478268eed517fa82768b/src/execution/execution_plan.h

Install hustle failed: cannot find arrow library

When installing hustle on macOS with the following procedure (in README.md):

Installed brew
./install_requirements.sh and ./install_arrow.sh. After the step, I see hustle/arrow directory.
mkdir build && cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE .. .

This failes with the following log:

-- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE) 
CMake Error at src/FindArrow.cmake:122 (message):
  Could not find the Arrow library.  Looked for headers in , and for libs in
Call Stack (most recent call first):
  src/CMakeLists.txt:21 (find_package)

Question

cmake -v on my machine is 3.18.2 instead of 3.15. I think we didn't specify the cmake version in Brewfile. Shall we stick to that cmake version, or whatever an upper version is okay?
Am I missing a flag for header / lib path for cmake to search for arrow?

Full log

/Applications/CLion.app/Contents/bin/cmake/mac/bin/cmake -DCMAKE_BUILD_TYPE=Debug -G "CodeBlocks - Unix Makefiles" /Users/ginda/Documents/GitHub/hustle
-- The C compiler identification is AppleClang 11.0.3.11030032
-- The CXX compiler identification is AppleClang 11.0.3.11030032
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Found PythonInterp: /usr/bin/python (found version "2.7.16") 
-- Looking for C++ include unistd.h
-- Looking for C++ include unistd.h - found
-- Looking for C++ include stdint.h
-- Looking for C++ include stdint.h - found
-- Looking for C++ include inttypes.h
-- Looking for C++ include inttypes.h - found
-- Looking for C++ include sys/types.h
-- Looking for C++ include sys/types.h - found
-- Looking for C++ include sys/stat.h
-- Looking for C++ include sys/stat.h - found
-- Looking for C++ include fnmatch.h
-- Looking for C++ include fnmatch.h - found
-- Looking for C++ include stddef.h
-- Looking for C++ include stddef.h - found
-- Check size of uint32_t
-- Check size of uint32_t - done
-- Looking for strtoll
-- Looking for strtoll - found
-- Check size of pthread_rwlock_t
-- Check size of pthread_rwlock_t - done
-- Performing Test COMPILER_HAS_WNO_UNUSED_LOCAL_TYPEDEF
-- Performing Test COMPILER_HAS_WNO_UNUSED_LOCAL_TYPEDEF - Success
-- Looking for C++ include dlfcn.h
-- Looking for C++ include dlfcn.h - found
-- Looking for C++ include execinfo.h
-- Looking for C++ include execinfo.h - found
-- Looking for C++ include glob.h
-- Looking for C++ include glob.h - found
-- Looking for C++ include libunwind.h
-- Looking for C++ include libunwind.h - found
-- Looking for C++ include memory.h
-- Looking for C++ include memory.h - found
-- Looking for C++ include pwd.h
-- Looking for C++ include pwd.h - found
-- Looking for C++ include stdlib.h
-- Looking for C++ include stdlib.h - found
-- Looking for C++ include strings.h
-- Looking for C++ include strings.h - found
-- Looking for C++ include string.h
-- Looking for C++ include string.h - found
-- Looking for C++ include syscall.h
-- Looking for C++ include syscall.h - not found
-- Looking for C++ include syslog.h
-- Looking for C++ include syslog.h - found
-- Looking for C++ include sys/syscall.h
-- Looking for C++ include sys/syscall.h - found
-- Looking for C++ include sys/time.h
-- Looking for C++ include sys/time.h - found
-- Looking for C++ include sys/ucontext.h
-- Looking for C++ include sys/ucontext.h - found
-- Looking for C++ include sys/utsname.h
-- Looking for C++ include sys/utsname.h - found
-- Looking for C++ include ucontext.h
-- Looking for C++ include ucontext.h - not found
-- Looking for C++ include unwind.h
-- Looking for C++ include unwind.h - found
-- Looking for fcntl
-- Looking for fcntl - found
-- Looking for sigalstack
-- Looking for sigalstack - not found
-- Check size of void*
-- Check size of void* - done
-- Performing Test HAVE___BUILTIN_EXPECT
-- Performing Test HAVE___BUILTIN_EXPECT - Success
-- Performing Test HAVE___ATTRIBUTE__
-- Performing Test HAVE___ATTRIBUTE__ - Success
-- Performing Test HAVE___SYNC_VAL_COMPARE_AND_SWAP
-- Performing Test HAVE___SYNC_VAL_COMPARE_AND_SWAP - Success
-- Checking program counter fetch from ucontext_t member: uc_mcontext.gregs[REG_EIP]
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.gregs[REG_RIP]
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.sc_ip
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.uc_regs->gregs[PT_NIP]
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.gregs[R15]
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.arm_pc
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.mc_eip
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.mc_rip
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.__gregs[_REG_EIP]
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.__gregs[_REG_RIP]
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext->ss.eip
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext->__ss.__eip
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext->ss.rip
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext->__ss.__rip
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Success
-- Found program counter field in ucontext_t: uc_mcontext->__ss.__rip
CMake Warning (dev) at /Applications/CLion.app/Contents/bin/cmake/mac/share/cmake-3.17/Modules/FindPackageHandleStandardArgs.cmake:272 (message):
  The package name passed to `find_package_handle_standard_args` (PkgConfig)
  does not match the name of the calling package (Arrow).  This can lead to
  problems in calling code that expects `find_package` result variables
  (e.g., `_FOUND`) to follow a certain pattern.
Call Stack (most recent call first):
  /Applications/CLion.app/Contents/bin/cmake/mac/share/cmake-3.17/Modules/FindPkgConfig.cmake:45 (find_package_handle_standard_args)
  src/FindArrow.cmake:10 (include)
  src/CMakeLists.txt:21 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE) 
CMake Error at src/FindArrow.cmake:122 (message):
  Could not find the Arrow library.  Looked for headers in , and for libs in
Call Stack (most recent call first):
  src/CMakeLists.txt:21 (find_package)


-- Configuring incomplete, errors occurred!
See also "/Users/ginda/Documents/GitHub/hustle/cmake-build-debug/CMakeFiles/CMakeOutput.log".
See also "/Users/ginda/Documents/GitHub/hustle/cmake-build-debug/CMakeFiles/CMakeError.log".

[Failed to reload]

HashAggregate can halt on SSB::Q11

The new hash aggregate operator introduced in #57 can handle most of the queries in our SSB workload.

Except for q11 - it can undeterministically throw the following error:

Assertion failed: (PolicyTraits::apply(FindElement{*this}, *iterator_at(i)) == iterator_at(i) && "constructed value does not match the lookup key"), function emplace_at, file /Users/ginda/Documents/GitHub/hustle/src/utils/parallel_hashmap/phmap.h, line 1999.
[1]    51407 abort      ./hustle_src_benchmark_main

and when it doesn't throw the error, the result seems matching the output using normal aggregate.

SMA

This issue is about adding Small Materialize Aggregates (SMAs) on all numeric columns and then using these SMAs to "skip" blocks during query processing. The initial scope can be limited to:

Min, Max on numeric column types (ints and floats)
Would be great to cover dates too (just min and max)

Note, we keep SMAs in a memory structure outside the Arrow data block. If you want to look at the Quickstep code,

here is how a user could explicitly request creating an SMA, and
the SMAIndexSubBlock code.
In Quickstep a block of data could have sub-blocks. Thus, a block may have data subblocks (actually a subblock for each column) and an SMA subblock within the master block. Storing SMAs (we also had support for cache-efficient B-tree, aka. CSBtree) inside the block made memory management difficult, and we'd rather do this outside in a separate "index cache" layer.

We want to design a "memoization" cache (MC) structure, so just before we start to run a select operation on a block, we:

Check for "hits" in the MC.
If there is a hit in the MC, it returns a 4-tuple: (the block number, a checksum, the columnid/name, and a "payload"). This last part is the "answer" to the function being checked, e.g. could this block contain the value 100 for the column foo. The payload could be in a protobuf format, but please feel free to make a design decision on this aspect.
Then, before using the functional cache payload, we verify the checksum on the block. For example, if we are evaluating a predicate foo > 100, for blockid = 13, and the min-max SMA for foo is [-14, 100), then we know that we can potentially skip block 13. To confirm that our metadata is current, we check the checksum for block 13 (which should be stored with the block and updated on any change).

Thus, the functional cache can be an independent way to "accelerate" operators. It is an asynchronous data structure, so it can be built/updated lazily. For example, if new blocks are added to a table, then some blocks may have SMAs in the MC and others may not. For the latter, we have to scan the relevant columns in the block, but for hits in the MC, we may be able to skip scanning the block all together.

For now, the "answer" in the MC is a min-max SMA, but the "answer" in the 4-tuple could be a bloom filter. (Our design of the MC should leave room for this extension).

Ideally, we can use the MC to do more, and also store query expression results for pairs for block. So, in the 4-tuple, the third part could be more general than a column "expression." It can be generalized to be a relational algebraic expression (in a canonical data structure representation in the code), such as: σ_{foo > 100}.

We can generalize the expression in the MC even further to be a canonical representation for any relational algebra. With that extension, we can encode a complex function like π_{R.foo, S.bar} [(σ_{R.foo > 100}) ⋈_{R.foobar = S.foobar} (S)]. Now, when evaluating this expression between an R block, r101, and an S block, s202, if an answer is present in the MC (and the checksums for both blocks match), we can just return the answer from the MC. No need to scan the data in the block(s).

More design is clearly needed for this more extensive use of MC, but it can be thought of as a "learning" layer and we can optimize what is being cached (the MC data structure likely has a limited memory budget) based on some utility function that is a combination of how frequently this entry is used and how expensive it is to compute. MC is a buffer pool for relational algebraic expression results at the block (pairs, triple, ...) level. This MC extension could be the Spring'21 784 project for the Hustle team members.

Perhaps @srsuryadev can help start this work and lead the design, and then pull in @mprammer2 in a pair-programming style on certain aspects. For example, @srsuryadev could design and implement the approach above with a min-max SMA as the "answer", and @mprammer2 could add the bloom filter. The 784 group could build the full-fledged MC feature.

Better Flag Handling

Consider Gflags: https://gflags.github.io/gflags/

In general, a more centralized approach to setting default behavior may be relevant.

Make resolver resilient to query without predicate grouping

Problem

Currently, hustle resolver expects, the predicates from the same table are grouped to construct the individual predicate tree for each select operator for the table.

Example,

SELECT c_city, s_city, d_year, SUM(lo_revenue) AS revenue
FROM customer, lineorder, supplier, ddate
WHERE lo_custkey = c_custkey
  AND lo_suppkey = s_suppkey
  AND lo_orderdate = d_datekey
  AND (c_city = 'UNITED KI1' OR c_city = 'UNITED KI5')
  AND (s_city = 'UNITED KI1' OR s_city = 'UNITED KI5')
  AND d_yearmonth = 'Dec1997'
GROUP BY c_city, s_city, d_year
ORDER BY d_year, revenue;

Make the resolver more generic to handle the case, even when the predicates are not grouped explicitly in the query.

Example,

SELECT c_city, s_city, d_year, SUM(lo_revenue) AS revenue
FROM customer, lineorder, supplier, ddate
WHERE lo_custkey = c_custkey
  AND lo_suppkey = s_suppkey
  AND lo_orderdate = d_datekey
  AND c_city = 'UNITED KI1' OR c_city = 'UNITED KI5'
  AND s_city = 'UNITED KI1' OR s_city = 'UNITED KI5'
  AND d_yearmonth = 'Dec1997'
GROUP BY c_city, s_city, d_year
ORDER BY d_year, revenue;

Note: The paranthesis are not used in the query which is used by the sqlite parser to generate a subtree for that table alone.

Type System

The operator code uses switch blocks in several places to properly handle different data types. We should have a type system that replaces conditionals with polymorphism (also see this repo) or we could use templates.

Select Refactoring

Handle more types than just UINT8 and INT64.
Migrate from arrow's CompareOperator to a native operator that directly supports BETWEEN and NOT_EQUAL.

SQL Issues

List of issues that need to be fixed for making the Hustle wider SQL query range.

Detect reordering in the join condition based on the num_rows
Support for Aliases in the query
Support for getting access to the output table programmatically
Support for queries without joins and aggregation

Error Handling

We don't have an agreed-upon way to handle errors. We should have a separate infrastructure to handle errors, so that we can use throw to catch the stack-trace of an error.

HashAggregate could error on q11

The new hash aggregate operator introduced in #57 can handle most of the queries in our SSB workload.

Except for q11 - it can undeterministically throw the following error:

Assertion failed: (PolicyTraits::apply(FindElement{*this}, *iterator_at(i)) == iterator_at(i) && "constructed value does not match the lookup key"), function emplace_at, file /Users/ginda/Documents/GitHub/hustle/src/utils/parallel_hashmap/phmap.h, line 1999.
[1]    51407 abort      ./hustle_src_benchmark_main

and when it doesn't throw the error, the result seems matching the output using normal aggregate.

Freeze ssb-dbgen version

Got the following error due to an update to the ssb-dbgen:

> sh ./scripts/ssb/gen_benchmark_data.sh 1
...
CMake Error at CMakeLists.txt:163 (set_property):
  C_STANDARD property may not be appended.


CMake Error at CMakeLists.txt:163 (set_property):
  C_STANDARD property may not be appended.

@srsuryadev I think we can either

Upgrade CMake to 3.20, or
Freeze a ssb-gen version

Error handling

Google released a library for returning and handling error messages some time ago. Now that hustle is growing maybe such a solution would be beneficial to make the code cleaner and handling/propagation of errors easier. We had discussed this in the past, but never went through with it (this library was not available at the time).

You can find more information here: https://abseil.io/docs/cpp/guides/status

What do you think? Would such a convention for error handling play nice with arrow?

Experimental: Bump Hustle to CPP20

As mentioned ini #94, C++20 gives us promising solution to metaprogram easily and without loss of performance. To explore this possibility, we need to bump Hustle to CPP20.

I'm experimenting inside the following system:

Compiler: g++-10.2.0
OS: MacOS Big Sur (11.1)
Changes the CMakeFile to standard 20: set(CMAKE_CXX_STANDARD 20)

The following problems needs to be addressed:

[Solved] Cereal.hpp complaints at compile time.
[Solved] sqlite_utils.h:38 complaints at compile time.
[Solved] arrow linker error. Solved by upgrading to latest arrow ini git repo. (Haven't used the release yet)
[Solved] absl compiler error: abseil/abseil-cpp#722
sqlite3: Undefined references.
(Mac) Unmached mangled name reference cause linker error.

Mangled name reference error

The dump fiile libarrow.a.dump.zip does not contains the mangled name that compiler generates. For example,

"__ZNK5arrow6Status8ToStringB5cxx11Ev", referenced from:
      __Z15evaluate_statusRKN5arrow6StatusEPKci in libhustle_src_storage.a(util.cc.o)

is produced by the compiler, but the closest we got is

U __ZNK5arrow6Status8ToStringEv

The complete compiler error g++-10

====================[ Build | hustle_select_test | Debug-GPP17 ]================
/Applications/CLion.app/Contents/bin/cmake/mac/bin/cmake --build /Users/gindachen/Project/hustle/cmake-build-debug-gpp17 --target hustle_select_test -- -j 9
-- Checking program counter fetch from ucontext_t member: uc_mcontext.gregs[REG_EIP]
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.gregs[REG_RIP]
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.sc_ip
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.uc_regs->gregs[PT_NIP]
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.gregs[R15]
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.arm_pc
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.mc_eip
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.mc_rip
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.__gregs[_REG_EIP]
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.__gregs[_REG_RIP]
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext->ss.eip
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext->__ss.__eip
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext->ss.rip
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext->__ss.__rip
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Success
-- Found program counter field in ucontext_t: uc_mcontext->__ss.__rip
-- Arrow SO and ABI version: 300
-- Arrow full SO version: 300.0.0
-- Found the Arrow core library: /usr/local/lib/libarrow.dylib
-- Found the Arrow Python library: ARROW_PYTHON_LIB_PATH-NOTFOUND
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/gindachen/Project/hustle/cmake-build-debug-gpp17
[  1%] Built target absl_log_severity
[  3%] Built target hustle_src_scheduler_SchedulerTypedefs
[  3%] Built target hustle_src_utils_meta_FunctionTraits
[  3%] Built target gflags_nothreads-static
[  5%] Built target absl_spinlock_wait
[  7%] Built target gtest
[  7%] Built target absl_exponential_biased
[  9%] Built target hustle_src_utils_Macros
[ 13%] Built target glog
[ 15%] Built target hustle_src_utils_ThreadSafeQueue
[ 15%] Built target hustle_src_utils_ParallelHashmap
[ 17%] Built target hustle_src_utils_Config
[ 17%] Built target hustle_src_utils_ContainerUtil
[ 17%] Built target absl_city
[ 19%] Built target absl_civil_time
[ 19%] Built target hustle_src_type
[ 19%] Built target absl_int128
[ 23%] Built target absl_time_zone
[ 27%] Built target hustle_src_scheduler_WorkerMessage
[ 27%] Built target hustle_src_threading_ThreadUtil
[ 29%] Built target hustle_src_scheduler_SchedulerMessage
[ 29%] Built target hustle_src_threading_Mutex
[ 31%] Built target hustle_src_scheduler_SchedulerFlags
[ 33%] Built target hustle_src_scheduler_Node
[ 35%] Built target hustle_src_utils_SyncStream
[ 35%] Built target hustle_src_scheduler_SchedulerInterface
[ 37%] Built target hustle_src_scheduler_TaskName
[ 37%] Built target hustle_src_utils_EventProfiler
[ 37%] Built target gtest_main
[ 39%] Built target gmock
[ 39%] Built target absl_raw_logging_internal
[ 39%] Built target hustle_src_scheduler_TaskDescription
[ 43%] Built target absl_bad_optional_access
[ 43%] Built target absl_throw_delegate
[ 47%] Built target absl_base
[ 47%] Built target absl_bad_variant_access
[ 50%] Built target absl_strings_internal
[ 52%] Built target absl_debugging_internal
[ 54%] Built target hustle_src_scheduler_Task
[ 54%] Built target absl_malloc_internal
[ 54%] Built target absl_demangle_internal
[ 56%] Built target absl_stacktrace
[ 64%] Built target absl_strings
[ 64%] Built target hustle_src_scheduler_Worker
[ 64%] Built target hustle_src_optimizer_ExecutionPlan
[ 64%] Built target absl_graphcycles_internal
[ 68%] Built target absl_symbolize
[ 70%] Built target absl_cord
[ 70%] Built target absl_hash
[ 72%] Built target absl_time
[ 72%] Built target hustle_src_scheduler_Scheduler
[ 76%] Built target absl_synchronization
[ 78%] Built target hustle_src_utils_ArrowComputeWrappers
[ 80%] Built target absl_hashtablez_sampler
[ 80%] Built target absl_raw_hash_set
[ 88%] Built target hustle_src_operators
[ 92%] Built target hustle_src_storage
[ 94%] Built target hustle_src_resolver
[ 96%] Built target hustleDB
[ 98%] Built target hustle_src_catalog
[100%] Built target sqlite_utils
[100%] Built target sqlite3
[100%] Linking CXX executable hustle_select_test
ld: warning: dylib (/usr/local/lib/libconfig++.dylib) was built for newer macOS version (11.0) than being linked (10.15)
ld: warning: dylib (/usr/local/lib/libarrow.dylib) was built for newer macOS version (11.0) than being linked (10.15)
ld: warning: dylib (/usr/local/gcc-10.1.0/lib/libstdc++.dylib) was built for newer macOS version (10.16) than being linked (10.15)
Undefined symbols for architecture x86_64:
  "__ZN5arrow11RecordBatch4MakeESt10shared_ptrINS_6SchemaEExSt6vectorIS1_INS_9ArrayDataEESaIS6_EE", referenced from:
      __ZN6hustle7storage5Block11get_recordsEv in libhustle_src_storage.a(util.cc.o)
      __Z17copy_record_batchRKSt10shared_ptrIN5arrow11RecordBatchEE in libhustle_src_storage.a(util.cc.o)
  "__ZN5arrow12ArrayBuilder6FinishEPSt10shared_ptrINS_5ArrayEE", referenced from:
      __ZN39SelectTestFixture_SingleSelectTest_Test8TestBodyEv in select_test.cc.o
      __ZN36SelectTestFixture_AndSelectTest_Test8TestBodyEv in select_test.cc.o
      __ZN35SelectTestFixture_OrSelectTest_Test8TestBodyEv in select_test.cc.o
      __ZN49SelectTestFixture_SingleSelectManyBlocksTest_Test8TestBodyEv in select_test.cc.o
      __ZN46SelectTestFixture_AndSelectManyBlocksTest_Test8TestBodyEv in select_test.cc.o
      __ZZN6hustle7Context13apply_indicesEPNS_4TaskEN5arrow5DatumES4_S4_RS4_ENKUlS2_E_clES2_ in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
  "__ZN5arrow12ChunkedArrayC1ESt6vectorISt10shared_ptrINS_5ArrayEESaIS4_EE", referenced from:
      __ZN9__gnu_cxx13new_allocatorIN5arrow12ChunkedArrayEE9constructIS2_JRSt6vectorISt10shared_ptrINS1_5ArrayEESaIS8_EEEEEvPT_DpOT0_ in select_test.cc.o
      __ZN9__gnu_cxx13new_allocatorIN5arrow12ChunkedArrayEE9constructIS2_JRSt6vectorISt10shared_ptrINS1_5ArrayEESaIS8_EEEEEvPT_DpOT0_ in libhustle_src_storage.a(util.cc.o)
      __ZN9__gnu_cxx13new_allocatorIN5arrow12ChunkedArrayEE9constructIS2_JRSt6vectorISt10shared_ptrINS1_5ArrayEESaIS8_EEEEEvPT_DpOT0_ in libhustle_src_operators.a(select.cc.o)
      __ZN9__gnu_cxx13new_allocatorIN5arrow12ChunkedArrayEE9constructIS2_JRSt6vectorISt10shared_ptrINS1_5ArrayEESaIS8_EEEEEvPT_DpOT0_ in libhustle_src_operators.a(lazy_table.cc.o)
      __ZN5arrow12ChunkedArrayC1ESt10shared_ptrINS_5ArrayEE in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN9__gnu_cxx13new_allocatorIN5arrow12ChunkedArrayEE9constructIS2_JRSt6vectorISt10shared_ptrINS1_5ArrayEESaIS8_EEEEEvPT_DpOT0_ in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
  "__ZN5arrow12StringScalarC1ENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE", referenced from:
      __ZN9__gnu_cxx13new_allocatorIN5arrow12StringScalarEE9constructIS2_JRA3_KcEEEvPT_DpOT0_ in select_test.cc.o
  "__ZN5arrow13SchemaBuilder8AddFieldERKSt10shared_ptrINS_5FieldEE", referenced from:
      __Z11make_schemaRKN6hustle7catalog11TableSchemaE in libhustle_src_storage.a(util.cc.o)
      __ZN6hustle9operators14OperatorResult11materializeERKSt6vectorINS0_15ColumnReferenceESaIS3_EEb in libhustle_src_operators.a(operator_result.cc.o)
  "__ZN5arrow15MakeArrayOfNullERKSt10shared_ptrINS_8DataTypeEExPNS_10MemoryPoolE", referenced from:
      __Z21array_to_chunkedarraySt10shared_ptrIN5arrow5ArrayEEi in libhustle_src_storage.a(util.cc.o)
      __ZN6hustle7Context26apply_indices_internal_strERKSt10shared_ptrIN5arrow12ChunkedArrayEERKS1_INS2_5ArrayEESA_i in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN6hustle7Context23apply_indices_internal2IxEEvRKSt10shared_ptrIN5arrow12ChunkedArrayEEPPKT_RKS2_INS3_5ArrayEESF_SF_i in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN6hustle7Context22apply_indices_internalIxEEvRKSt10shared_ptrIN5arrow12ChunkedArrayEEPPKT_RKS2_INS3_5ArrayEESF_i in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN6hustle7Context23apply_indices_internal2IjEEvRKSt10shared_ptrIN5arrow12ChunkedArrayEEPPKT_RKS2_INS3_5ArrayEESF_SF_i in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN6hustle7Context22apply_indices_internalIjEEvRKSt10shared_ptrIN5arrow12ChunkedArrayEEPPKT_RKS2_INS3_5ArrayEESF_i in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
  "__ZN5arrow18RecordBatchBuilder4MakeERKSt10shared_ptrINS_6SchemaEEPNS_10MemoryPoolEPSt10unique_ptrIS0_St14default_deleteIS0_EE", referenced from:
      __Z18read_from_csv_filePKcSt10shared_ptrIN5arrow6SchemaEEib in libhustle_src_storage.a(util.cc.o)
  "__ZN5arrow2io12ReadableFile4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPNS_10MemoryPoolE", referenced from:
      __Z14read_from_filePKcbS0_ in libhustle_src_storage.a(util.cc.o)
  "__ZN5arrow2io16FileOutputStream4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb", referenced from:
      __Z13write_to_filePKcRN6hustle7storage7DBTableE in libhustle_src_storage.a(util.cc.o)
  "__ZN5arrow3ipc13NewFileWriterEPNS_2io12OutputStreamERKSt10shared_ptrINS_6SchemaEERKNS0_15IpcWriteOptionsERKS4_IKNS_16KeyValueMetadataEE", referenced from:
      __Z13write_to_filePKcRN6hustle7storage7DBTableE in libhustle_src_storage.a(util.cc.o)
  "__ZN5arrow3ipc21RecordBatchFileReader4OpenERKSt10shared_ptrINS_2io16RandomAccessFileEERKNS0_14IpcReadOptionsE", referenced from:
      __Z14read_from_filePKcbS0_ in libhustle_src_storage.a(util.cc.o)
  "__ZN5arrow4util6detail19StringStreamWrapper3strB5cxx11Ev", referenced from:
      __ZN5arrow4util13StringBuilderIJRA46_KcRxRA2_S2_EEENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEDpOT_ in select_test.cc.o
      __ZN5arrow4util13StringBuilderIJRA36_KcRxRA19_S2_S5_RA2_S2_EEENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEDpOT_ in select_test.cc.o
      __ZN5arrow4util13StringBuilderIJRA50_KcxRA22_S2_RxEEENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEDpOT_ in select_test.cc.o
      __ZN5arrow4util13StringBuilderIJRA46_KcRxRA2_S2_EEENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEDpOT_ in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN5arrow4util13StringBuilderIJRA36_KcRxRA19_S2_S5_RA2_S2_EEENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEDpOT_ in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN5arrow4util13StringBuilderIJRA24_KcEEENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEDpOT_ in libhustle_src_storage.a(sma.cc.o)
  "__ZN5arrow5DatumC1ERKSt10shared_ptrINS_5ArrayEE", referenced from:
      __ZN6hustle9operators6Select6FilterERKSt10shared_ptrINS_7storage5BlockEERKS2_INS0_9PredicateEE in libhustle_src_operators.a(select.cc.o)
      __ZN5arrow5DatumC1INS_12BooleanArrayEvEERKSt10shared_ptrIT_E in libhustle_src_operators.a(select.cc.o)
      __ZZZN6hustle7Context5matchEPNS_4TaskERKN5arrow5DatumES6_ENKUlS2_E_clES2_ENKUlvE_clEv in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN6hustle7Context18apply_filter_blockERKSt10shared_ptrIN5arrow5ArrayEES6_RSt6vectorIS4_SaIS4_EE in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZZN6hustle7Context12apply_filterEPNS_4TaskERKN5arrow5DatumES6_RS4_ENKUlS2_E_clES2_ in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN6hustle7storage3SmaC2ERKSt10shared_ptrIN5arrow5ArrayEE in libhustle_src_storage.a(sma.cc.o)
      __ZN6hustle7storage3SmaC1ERKSt10shared_ptrIN5arrow5ArrayEE in libhustle_src_storage.a(sma.cc.o)
      ...
  "__ZN5arrow5DatumC1ESt10shared_ptrINS_12ChunkedArrayEE", referenced from:
      __ZN9__gnu_cxx13new_allocatorIN6hustle9operators9LazyTableEE9constructIS3_JRSt10shared_ptrINS1_7storage7DBTableEERS6_IN5arrow12ChunkedArrayEENSB_5DatumESF_EEEvPT_DpOT0_ in libhustle_src_operators.a(select.cc.o)
      __ZN6hustle9operators9LazyTable17MaterializeColumnEi in libhustle_src_operators.a(lazy_table.cc.o)
      __ZN6hustle9operators9LazyTable17MaterializeColumnEPNS_4TaskEiRN5arrow5DatumE in libhustle_src_operators.a(lazy_table.cc.o)
      __ZZN6hustle7Context13apply_indicesEPNS_4TaskEN5arrow5DatumES4_S4_RS4_ENKUlS2_E0_clES2_ in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
  "__ZN5arrow5fieldENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt10shared_ptrINS_8DataTypeEEbS6_IKNS_16KeyValueMetadataEE", referenced from:
      __ZN17SelectTestFixture5SetUpEv in select_test.cc.o
      __Z11make_schemaRKN6hustle7catalog11TableSchemaE in libhustle_src_storage.a(util.cc.o)
  "__ZN5arrow6StatusC1ENS_10StatusCodeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE", referenced from:
      __ZN5arrow6Status8FromArgsIJRA46_KcRxRA2_S2_EEES0_NS_10StatusCodeEDpOT_ in select_test.cc.o
      __ZN5arrow6Status8FromArgsIJRA36_KcRxRA19_S2_S5_RA2_S2_EEES0_NS_10StatusCodeEDpOT_ in select_test.cc.o
      __ZN5arrow6Status8FromArgsIJRA50_KcxRA22_S2_RxEEES0_NS_10StatusCodeEDpOT_ in select_test.cc.o
      __ZN5arrow6Status8FromArgsIJRA46_KcRxRA2_S2_EEES0_NS_10StatusCodeEDpOT_ in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN5arrow6Status8FromArgsIJRA36_KcRxRA19_S2_S5_RA2_S2_EEES0_NS_10StatusCodeEDpOT_ in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN5arrow6Status8FromArgsIJRA24_KcEEES0_NS_10StatusCodeEDpOT_ in libhustle_src_storage.a(sma.cc.o)
  "__ZN5arrow6schemaESt6vectorISt10shared_ptrINS_5FieldEESaIS3_EES1_IKNS_16KeyValueMetadataEE", referenced from:
      __ZN17SelectTestFixture5SetUpEv in select_test.cc.o
  "__ZN5arrow9ArrayData4MakeERKSt10shared_ptrINS_8DataTypeEExSt6vectorIS1_INS_6BufferEESaIS8_EExx", referenced from:
      __ZN5arrow14NumericBuilderINS_10DoubleTypeEE14FinishInternalEPSt10shared_ptrINS_9ArrayDataEE in select_test.cc.o
      __ZN5arrow14NumericBuilderINS_9Int64TypeEE14FinishInternalEPSt10shared_ptrINS_9ArrayDataEE in select_test.cc.o
      __ZN5arrow17BaseBinaryBuilderINS_10BinaryTypeEE14FinishInternalEPSt10shared_ptrINS_9ArrayDataEE in select_test.cc.o
      __Z17copy_record_batchRKSt10shared_ptrIN5arrow11RecordBatchEE in libhustle_src_storage.a(util.cc.o)
      __ZN6hustle7storage5BlockC2EiRKSt10shared_ptrIN5arrow6SchemaEEib in libhustle_src_storage.a(block.cc.o)
      __ZN6hustle7storage5BlockC1EiRKSt10shared_ptrIN5arrow6SchemaEEib in libhustle_src_storage.a(block.cc.o)
      __ZN6hustle7storage5BlockC2EiSt10shared_ptrIN5arrow11RecordBatchEEib in libhustle_src_storage.a(block.cc.o)
      ...
  "__ZN5arrow9MakeArrayERKSt10shared_ptrINS_9ArrayDataEE", referenced from:
      __ZNK6hustle7storage5Block10get_columnEi in select_test.cc.o
      __ZNK6hustle7storage5Block18get_column_by_nameERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE in libhustle_src_operators.a(select.cc.o)
      __ZNK6hustle7storage5Block10get_columnEi in libhustle_src_storage.a(ma_block.cc.o)
      __ZN6hustle7storage5Block9out_blockEPvPFiS2_iPPcS4_E in libhustle_src_storage.a(block.cc.o)
      __ZN6hustle7storage5Block5printEv in libhustle_src_storage.a(block.cc.o)
      __ZNK6hustle7storage5Block16get_valid_columnEv in libhustle_src_storage.a(table.cc.o)
      __ZNK6hustle7storage5Block10get_columnEi in libhustle_src_operators.a(lazy_table.cc.o)
      ...
  "__ZNK5arrow5Array6EqualsERKSt10shared_ptrIS0_ERKNS_12EqualOptionsE", referenced from:
      __ZN39SelectTestFixture_SingleSelectTest_Test8TestBodyEv in select_test.cc.o
      __ZN36SelectTestFixture_AndSelectTest_Test8TestBodyEv in select_test.cc.o
      __ZN35SelectTestFixture_OrSelectTest_Test8TestBodyEv in select_test.cc.o
      __ZN49SelectTestFixture_SingleSelectManyBlocksTest_Test8TestBodyEv in select_test.cc.o
      __ZN46SelectTestFixture_AndSelectManyBlocksTest_Test8TestBodyEv in select_test.cc.o
  "__ZNK5arrow6Schema11field_namesB5cxx11Ev", referenced from:
      __ZN6hustle7storage5Block9out_blockEPvPFiS2_iPPcS4_E in libhustle_src_storage.a(block.cc.o)
  "__ZNK5arrow6Schema13GetFieldIndexERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE", referenced from:
      __ZNK6hustle7storage5Block18get_column_by_nameERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE in libhustle_src_operators.a(select.cc.o)
      __ZN6hustle7storage21MetadataAttachedBlock14SearchMetadataERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN5arrow5DatumENSA_7compute15CompareOperatorE in libhustle_src_operators.a(select.cc.o)
      __ZN6hustle9operators9LazyTable17MaterializeColumnENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE in libhustle_src_operators.a(lazy_table.cc.o)
      __ZN6hustle9operators9LazyTable17MaterializeColumnEPNS_4TaskENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERN5arrow5DatumE in libhustle_src_operators.a(lazy_table.cc.o)
  "__ZNK5arrow6Schema14GetFieldByNameERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE", referenced from:
      __ZN6hustle9operators14OperatorResult11materializeERKSt6vectorINS0_15ColumnReferenceESaIS3_EEb in libhustle_src_operators.a(operator_result.cc.o)
  "__ZNK5arrow6Status8ToStringB5cxx11Ev", referenced from:
      __Z15evaluate_statusRKN5arrow6StatusEPKci in libhustle_src_storage.a(util.cc.o)
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status
make[3]: *** [src/operators/hustle_select_test] Error 1
make[2]: *** [src/operators/CMakeFiles/hustle_select_test.dir/all] Error 2
make[1]: *** [src/operators/CMakeFiles/hustle_select_test.dir/rule] Error 2
make: *** [hustle_select_test] Error 2

`sqlite` Failure message

[ 93%] Linking CXX executable hustle_util_test
../../third_party/sqlite3/libsqlite3.a(sqlite3.c.o): In function `sqlite3LeaveMutexAndCloseZombie':
sqlite3.c:(.text+0x4ba12): undefined reference to `hustle_memlog_free'
../../third_party/sqlite3/libsqlite3.a(sqlite3.c.o): In function `sqlite3_open':
sqlite3.c:(.text+0x6804c): undefined reference to `hustle_memlog_initialize'
../../third_party/sqlite3/libsqlite3.a(sqlite3.c.o): In function `sqlite3_open_v2':
sqlite3.c:(.text+0x68085): undefined reference to `hustle_memlog_initialize'
../../third_party/sqlite3/libsqlite3.a(sqlite3.c.o): In function `sqlite3VdbeHalt':
sqlite3.c:(.text+0x6fe8b): undefined reference to `hustle_memlog_clear'
sqlite3.c:(.text+0x7086c): undefined reference to `hustle_memlog_update_db'
../../third_party/sqlite3/libsqlite3.a(sqlite3.c.o): In function `sqlite3VdbeExec':
sqlite3.c:(.text+0xa8134): undefined reference to `memlog_add_column_change'
sqlite3.c:(.text+0xa903f): undefined reference to `hustle_memlog_create_record'
sqlite3.c:(.text+0xa905d): undefined reference to `hustle_memlog_insert_record'
sqlite3.c:(.text+0xa94c0): undefined reference to `memlog_remove_table_mapping'
sqlite3.c:(.text+0xaa82f): undefined reference to `hustle_memlog_create_record'
sqlite3.c:(.text+0xaa842): undefined reference to `hustle_memlog_insert_record'
sqlite3.c:(.text+0xab217): undefined reference to `hustle_memlog_create_record'
sqlite3.c:(.text+0xab22a): undefined reference to `hustle_memlog_insert_record'
sqlite3.c:(.text+0xad2c3): undefined reference to `hustle_memlog_update_db'
../../third_party/sqlite3/libsqlite3.a(sqlite3.c.o): In function `sqlite3_exec':
sqlite3.c:(.text+0xb255a): undefined reference to `resolveSelect'
../../third_party/sqlite3/libsqlite3.a(sqlite3.c.o): In function `sqlite3EndTable':
sqlite3.c:(.text+0xd0026): undefined reference to `memlog_add_table_mapping'
collect2: error: ld returned 1 exit status
src/storage/CMakeFiles/hustle_util_test.dir/build.make:156: recipe for target 'src/storage/hustle_util_test' failed
make[2]: *** [src/storage/hustle_util_test] Error 1
CMakeFiles/Makefile2:5177: recipe for target 'src/storage/CMakeFiles/hustle_util_test.dir/all' failed
make[1]: *** [src/storage/CMakeFiles/hustle_util_test.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

Solved logs

`sqlite_utils` Failure message

In file included from /Users/gindachen/Projects/hustle/src/utils/sqlite_utils.cc:18:
/Users/gindachen/Projects/hustle/src/utils/sqlite_utils.h:29:29: error: 'string' in namespace 'std' does not name a type
   29 | void load_tables(const std::string &sqlitePath, std::vector<std::string> tables);
      |                             ^~~~~~
/Users/gindachen/Projects/hustle/src/utils/sqlite_utils.h:23:1: note: 'std::string' is defined in header '<string>'; did you forget to '#include <string>'?
   22 | #include <vector>
  +++ |+#include <string>
   23 | 
/Users/gindachen/Projects/hustle/src/utils/sqlite_utils.h:29:66: error: 'string' is not a member of 'std'
   29 | void load_tables(const std::string &sqlitePath, std::vector<std::string> tables);
      |                                                                  ^~~~~~
/Users/gindachen/Projects/hustle/src/utils/sqlite_utils.h:29:66: note: 'std::string' is defined in header '<string>'; did you forget to '#include <string>'?
/Users/gindachen/Projects/hustle/src/utils/sqlite_utils.h:29:72: error: template argument 1 is invalid
   29 | void load_tables(const std::string &sqlitePath, std::vector<std::string> tables);
      |                                                                        ^
/Users/gindachen/Projects/hustle/src/utils/sqlite_utils.h:29:72: error: template argument 2 is invalid
/Users/gindachen/Projects/hustle/src/utils/sqlite_utils.h:33:38: error: 'string' in namespace 'std' does not name a type
   33 | bool execute_sqlite_query(const std::string &sqlitePath,
      |                                      ^~~~~~
/Users/gindachen/Projects/hustle/src/utils/sqlite_utils.h:33:33: note: 'std::string' is defined in header '<string>'; did you forget to '#include <string>'?
   33 | bool execute_sqlite_query(const std::string &sqlitePath,
      |                                 ^~~
/Users/gindachen/Projects/hustle/src/utils/sqlite_utils.h:34:38: error: 'string' in namespace 'std' does not name a type
   34 |                           const std::string &sql);
      |                                      ^~~~~~
/Users/gindachen/Projects/hustle/src/utils/sqlite_utils.h:34:33: note: 'std::string' is defined in header '<string>'; did you forget to '#include <string>'?
   34 |                           const std::string &sql);
      |                                 ^~~
/Users/gindachen/Projects/hustle/src/utils/sqlite_utils.h:38:6: error: 'string' in namespace 'std' does not name a type
   38 | std::string execute_sqlite_result(const std::string &sqlitePath,
      |      ^~~~~~
/Users/gindachen/Projects/hustle/src/utils/sqlite_utils.h:38:1: note: 'std::string' is defined in header '<string>'; did you forget to '#include <string>'?
   38 | std::string execute_sqlite_result(const std::string &sqlitePath,
      | ^~~

Arrow Linker Error

Undefined symbols for architecture x86_64:
  "__ZN5arrow10MapBuilderC1EPNS_10MemoryPoolERKSt10shared_ptrINS_12ArrayBuilderEES7_b", referenced from:
      __ZSt12construct_atIN5arrow10MapBuilderEJPNS0_10MemoryPoolERSt10shared_ptrINS0_12ArrayBuilderEES7_EEDTgsnwcvPvLi0E_T_pispcl7declvalIT0_EEEEPS9_DpOSA_ in libhustle_src_type.a(type_helper.cc.o)
  "__ZN5arrow10dictionaryERKSt10shared_ptrINS_8DataTypeEES4_b", referenced from:
      __ZN6hustle7details10TestFieldsEN5arrow4Type4typeE in libhustle_src_type.a(type_helper.cc.o)
      __ZNK5arrow8internal21DictionaryBuilderBaseINS_18AdaptiveIntBuilderENS_9Int32TypeEE4typeEv in libhustle_src_type.a(type_helper.cc.o)
      __ZNK5arrow8internal21DictionaryBuilderBaseINS_18AdaptiveIntBuilderENS_8Int8TypeEE4typeEv in libhustle_src_type.a(type_helper.cc.o)
  "__ZN5arrow10large_listERKSt10shared_ptrINS_8DataTypeEE", referenced from:
      __ZN6hustle7details10TestFieldsEN5arrow4Type4typeE in libhustle_src_type.a(type_helper.cc.o)
  "__ZN5arrow11RecordBatch4MakeESt10shared_ptrINS_6SchemaEExSt6vectorIS1_INS_9ArrayDataEESaIS6_EE", referenced from:
      __ZN6hustle7storage5Block11get_recordsEv in libhustle_src_storage.a(util.cc.o)
      __Z17copy_record_batchRKSt10shared_ptrIN5arrow11RecordBatchEE in libhustle_src_storage.a(util.cc.o)
  "__ZN5arrow11dense_unionESt6vectorISt10shared_ptrINS_5FieldEESaIS3_EES0_IaSaIaEE", referenced from:
      __ZN6hustle7details10TestFieldsEN5arrow4Type4typeE in libhustle_src_type.a(type_helper.cc.o)
  "__ZN5arrow12ArrayBuilder6FinishEPSt10shared_ptrINS_5ArrayEE", referenced from:
      __ZN38HashAggregateTestFixture_MeanTest_Test8TestBodyEv in hash_aggregate_test.cc.o
      __ZN37HashAggregateTestFixture_SumTest_Test8TestBodyEv in hash_aggregate_test.cc.o
      __ZN39HashAggregateTestFixture_CountTest_Test8TestBodyEv in hash_aggregate_test.cc.o
      __ZN48HashAggregateTestFixture_SumWithGroupByTest_Test8TestBodyEv in hash_aggregate_test.cc.o
      __ZN55HashAggregateTestFixture_SumWithGroupByOrderByTest_Test8TestBodyEv in hash_aggregate_test.cc.o
      __ZN6hustle9operators13HashAggregate6FinishEPNS_4TaskE in libhustle_src_operators.a(hash_aggregate.cc.o)
      __ZZN6hustle7Context13apply_indicesEPNS_4TaskEN5arrow5DatumES4_S4_RS4_ENKUlS2_E_clES2_ in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      ...
  "__ZN5arrow12ChunkedArrayC1ESt6vectorISt10shared_ptrINS_5ArrayEESaIS4_EE", referenced from:
      __ZSt12construct_atIN5arrow12ChunkedArrayEJRSt6vectorISt10shared_ptrINS0_5ArrayEESaIS5_EEEEDTgsnwcvPvLi0E_T_pispcl7declvalIT0_EEEEPSA_DpOSB_ in hash_aggregate_test.cc.o
      __ZSt12construct_atIN5arrow12ChunkedArrayEJRSt6vectorISt10shared_ptrINS0_5ArrayEESaIS5_EEEEDTgsnwcvPvLi0E_T_pispcl7declvalIT0_EEEEPSA_DpOSB_ in libhustle_src_storage.a(util.cc.o)
      __ZSt12construct_atIN5arrow12ChunkedArrayEJRSt6vectorISt10shared_ptrINS0_5ArrayEESaIS5_EEEEDTgsnwcvPvLi0E_T_pispcl7declvalIT0_EEEEPSA_DpOSB_ in libhustle_src_operators.a(lazy_table.cc.o)
      __ZN5arrow12ChunkedArrayC1ESt10shared_ptrINS_5ArrayEE in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZSt12construct_atIN5arrow12ChunkedArrayEJRSt6vectorISt10shared_ptrINS0_5ArrayEESaIS5_EEEEDTgsnwcvPvLi0E_T_pispcl7declvalIT0_EEEEPSA_DpOSB_ in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
  "__ZN5arrow12sparse_unionESt6vectorISt10shared_ptrINS_5FieldEESaIS3_EES0_IaSaIaEE", referenced from:
      __ZN6hustle7details10TestFieldsEN5arrow4Type4typeE in libhustle_src_type.a(type_helper.cc.o)
  "__ZN5arrow13SchemaBuilder8AddFieldERKSt10shared_ptrINS_5FieldEE", referenced from:
      __Z11make_schemaRKN6hustle7catalog11TableSchemaE in libhustle_src_storage.a(util.cc.o)
      __ZN6hustle9operators13HashAggregate12OutputSchemaENS0_15AggregateKernelERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE in libhustle_src_operators.a(hash_aggregate.cc.o)
      __ZN6hustle9operators14OperatorResult11materializeERKSt6vectorINS0_15ColumnReferenceESaIS3_EEb in libhustle_src_operators.a(operator_result.cc.o)
  "__ZN5arrow13SchemaBuilder9AddFieldsERKSt6vectorISt10shared_ptrINS_5FieldEESaIS4_EE", referenced from:
      __ZN6hustle9operators13HashAggregate12OutputSchemaENS0_15AggregateKernelERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE in libhustle_src_operators.a(hash_aggregate.cc.o)
  "__ZN5arrow13StructBuilderC1ERKSt10shared_ptrINS_8DataTypeEEPNS_10MemoryPoolESt6vectorIS1_INS_12ArrayBuilderEESaISA_EE", referenced from:
      __ZSt12construct_atIN5arrow13StructBuilderEJRSt10shared_ptrINS0_8DataTypeEEPNS0_10MemoryPoolESt6vectorIS2_INS0_12ArrayBuilderEESaISA_EEEEDTgsnwcvPvLi0E_T_pispcl7declvalIT0_EEEEPSE_DpOSF_ in libhustle_src_operators.a(hash_aggregate.cc.o)
      __ZSt12construct_atIN5arrow13StructBuilderEJRSt10shared_ptrINS0_10StructTypeEEPNS0_10MemoryPoolERSt6vectorIS2_INS0_12ArrayBuilderEESaISA_EEEEDTgsnwcvPvLi0E_T_pispcl7declvalIT0_EEEEPSF_DpOSG_ in libhustle_src_type.a(type_helper.cc.o)
  "__ZN5arrow15MakeArrayOfNullERKSt10shared_ptrINS_8DataTypeEExPNS_10MemoryPoolE", referenced from:
      __Z21array_to_chunkedarraySt10shared_ptrIN5arrow5ArrayEEi in libhustle_src_storage.a(util.cc.o)
      __ZN6hustle7Context26apply_indices_internal_strERKSt10shared_ptrIN5arrow12ChunkedArrayEERKS1_INS2_5ArrayEESA_i in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN6hustle7Context23apply_indices_internal2IxEEvRKSt10shared_ptrIN5arrow12ChunkedArrayEEPPKT_RKS2_INS3_5ArrayEESF_SF_i in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN6hustle7Context22apply_indices_internalIxEEvRKSt10shared_ptrIN5arrow12ChunkedArrayEEPPKT_RKS2_INS3_5ArrayEESF_i in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN6hustle7Context23apply_indices_internal2IjEEvRKSt10shared_ptrIN5arrow12ChunkedArrayEEPPKT_RKS2_INS3_5ArrayEESF_SF_i in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN6hustle7Context22apply_indices_internalIjEEvRKSt10shared_ptrIN5arrow12ChunkedArrayEEPPKT_RKS2_INS3_5ArrayEESF_i in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
  "__ZN5arrow15fixed_size_listERKSt10shared_ptrINS_8DataTypeEEi", referenced from:
      __ZN6hustle7details10TestFieldsEN5arrow4Type4typeE in libhustle_src_type.a(type_helper.cc.o)
  "__ZN5arrow17BasicUnionBuilder14FinishInternalEPSt10shared_ptrINS_9ArrayDataEE", referenced from:
      __ZTVN5arrow18SparseUnionBuilderE in libhustle_src_type.a(type_helper.cc.o)
  "__ZN5arrow17BasicUnionBuilderC2EPNS_10MemoryPoolERKSt6vectorISt10shared_ptrINS_12ArrayBuilderEESaIS6_EERKS4_INS_8DataTypeEE", referenced from:
      __ZN5arrow18SparseUnionBuilderC1EPNS_10MemoryPoolERKSt6vectorISt10shared_ptrINS_12ArrayBuilderEESaIS6_EERKS4_INS_8DataTypeEE in libhustle_src_type.a(type_helper.cc.o)
  "__ZN5arrow17Decimal128BuilderC1ERKSt10shared_ptrINS_8DataTypeEEPNS_10MemoryPoolE", referenced from:
      __ZSt12construct_atIN5arrow17Decimal128BuilderEJRSt10shared_ptrINS0_8DataTypeEEPNS0_10MemoryPoolEEEDTgsnwcvPvLi0E_T_pispcl7declvalIT0_EEEEPS9_DpOSA_ in libhustle_src_type.a(type_helper.cc.o)
  "__ZN5arrow18AdaptiveIntBuilder14FinishInternalEPSt10shared_ptrINS_9ArrayDataEE", referenced from:
      __ZN5arrow8internal21DictionaryBuilderBaseINS_18AdaptiveIntBuilderENS_9Int32TypeEE20FinishWithDictOffsetExPSt10shared_ptrINS_9ArrayDataEES8_ in libhustle_src_type.a(type_helper.cc.o)
      __ZN5arrow8internal21DictionaryBuilderBaseINS_18AdaptiveIntBuilderENS_8Int8TypeEE20FinishWithDictOffsetExPSt10shared_ptrINS_9ArrayDataEES8_ in libhustle_src_type.a(type_helper.cc.o)
  "__ZN5arrow18RecordBatchBuilder4MakeERKSt10shared_ptrINS_6SchemaEEPNS_10MemoryPoolEPSt10unique_ptrIS0_St14default_deleteIS0_EE", referenced from:
      __Z18read_from_csv_filePKcSt10shared_ptrIN5arrow6SchemaEEib in libhustle_src_storage.a(util.cc.o)
  "__ZN5arrow20FixedSizeListBuilderC1EPNS_10MemoryPoolERKSt10shared_ptrINS_12ArrayBuilderEERKS3_INS_8DataTypeEE", referenced from:
      __ZSt12construct_atIN5arrow20FixedSizeListBuilderEJPNS0_10MemoryPoolERSt10shared_ptrINS0_12ArrayBuilderEERS4_INS0_8DataTypeEEEEDTgsnwcvPvLi0E_T_pispcl7declvalIT0_EEEEPSC_DpOSD_ in libhustle_src_type.a(type_helper.cc.o)
  "__ZN5arrow22FixedSizeBinaryBuilderC1ERKSt10shared_ptrINS_8DataTypeEEPNS_10MemoryPoolE", referenced from:
      __ZSt12construct_atIN5arrow22FixedSizeBinaryBuilderEJRSt10shared_ptrINS0_8DataTypeEEPNS0_10MemoryPoolEEEDTgsnwcvPvLi0E_T_pispcl7declvalIT0_EEEEPS9_DpOSA_ in libhustle_src_type.a(type_helper.cc.o)
  "__ZN5arrow2io12ReadableFile4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPNS_10MemoryPoolE", referenced from:
      __Z14read_from_filePKcbS0_ in libhustle_src_storage.a(util.cc.o)
  "__ZN5arrow2io16FileOutputStream4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb", referenced from:
      __Z13write_to_filePKcRN6hustle7storage7DBTableE in libhustle_src_storage.a(util.cc.o)
  "__ZN5arrow3ipc13NewFileWriterEPNS_2io12OutputStreamERKSt10shared_ptrINS_6SchemaEERKNS0_15IpcWriteOptionsERKS4_IKNS_16KeyValueMetadataEE", referenced from:
      __Z13write_to_filePKcRN6hustle7storage7DBTableE in libhustle_src_storage.a(util.cc.o)
  "__ZN5arrow3ipc21RecordBatchFileReader4OpenERKSt10shared_ptrINS_2io16RandomAccessFileEERKNS0_14IpcReadOptionsE", referenced from:
      __Z14read_from_filePKcbS0_ in libhustle_src_storage.a(util.cc.o)
  "__ZN5arrow3mapESt10shared_ptrINS_8DataTypeEES2_b", referenced from:
      __ZN6hustle7details10TestFieldsEN5arrow4Type4typeE in libhustle_src_type.a(type_helper.cc.o)
  "__ZN5arrow4listERKSt10shared_ptrINS_8DataTypeEE", referenced from:
      __ZN6hustle7details10TestFieldsEN5arrow4Type4typeE in libhustle_src_type.a(type_helper.cc.o)
  "__ZN5arrow4util6detail19StringStreamWrapper3strB5cxx11Ev", referenced from:
      __ZN5arrow4util13StringBuilderIJRA46_KcRxRA2_S2_EEENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEDpOT_ in hash_aggregate_test.cc.o
      __ZN5arrow4util13StringBuilderIJRA36_KcRxRA19_S2_S5_RA2_S2_EEENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEDpOT_ in hash_aggregate_test.cc.o
      __ZN5arrow4util13StringBuilderIJRA32_KcxRA14_S2_RxEEENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEDpOT_ in hash_aggregate_test.cc.o
      __ZN5arrow4util13StringBuilderIJRA50_KcxRA22_S2_RxEEENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEDpOT_ in hash_aggregate_test.cc.o
      __ZN5arrow4util13StringBuilderIJRA46_KcRxRA2_S2_EEENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEDpOT_ in libhustle_src_operators.a(hash_aggregate.cc.o)
      __ZN5arrow4util13StringBuilderIJRA36_KcRxRA19_S2_S5_RA2_S2_EEENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEDpOT_ in libhustle_src_operators.a(hash_aggregate.cc.o)
      __ZN5arrow4util13StringBuilderIJRA32_KcxRA14_S2_RxEEENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEDpOT_ in libhustle_src_operators.a(hash_aggregate.cc.o)
      ...
  "__ZN5arrow5DatumC1ERKSt10shared_ptrINS_5ArrayEE", referenced from:
      __ZNO5arrow6ResultISt10shared_ptrINS_5ArrayEEE5ValueINS_5DatumEvEENS_6StatusEPT_ in libhustle_src_operators.a(hash_aggregate.cc.o)
      __ZN6hustle9operators10Expression8EvaluateEPNS_4TaskEi in libhustle_src_operators.a(expression.cc.o)
      __ZN6hustle9operators10Expression12ExecuteBlockIN5arrow12BooleanArrayEbEENS3_5DatumEiSt10shared_ptrINS3_5ArrayEES8_S8_ in libhustle_src_operators.a(expression.cc.o)
      __ZN6hustle9operators10Expression12ExecuteBlockIN5arrow12NumericArrayINS3_8Int8TypeEEEaEENS3_5DatumEiSt10shared_ptrINS3_5ArrayEESA_SA_ in libhustle_src_operators.a(expression.cc.o)
      __ZN6hustle9operators10Expression12ExecuteBlockIN5arrow12NumericArrayINS3_9Int16TypeEEEsEENS3_5DatumEiSt10shared_ptrINS3_5ArrayEESA_SA_ in libhustle_src_operators.a(expression.cc.o)
      __ZN6hustle9operators10Expression12ExecuteBlockIN5arrow12NumericArrayINS3_9Int32TypeEEEiEENS3_5DatumEiSt10shared_ptrINS3_5ArrayEESA_SA_ in libhustle_src_operators.a(expression.cc.o)
      __ZN6hustle9operators10Expression12ExecuteBlockIN5arrow12NumericArrayINS3_9Int64TypeEEExEENS3_5DatumEiSt10shared_ptrINS3_5ArrayEESA_SA_ in libhustle_src_operators.a(expression.cc.o)
      ...
  "__ZN5arrow5DatumC1ESt10shared_ptrINS_12ChunkedArrayEE", referenced from:
      __ZN6hustle9operators9LazyTable17MaterializeColumnEi in libhustle_src_operators.a(lazy_table.cc.o)
      __ZN6hustle9operators9LazyTable17MaterializeColumnEPNS_4TaskEiRN5arrow5DatumE in libhustle_src_operators.a(lazy_table.cc.o)
      __ZZN6hustle7Context13apply_indicesEPNS_4TaskEN5arrow5DatumES4_S4_RS4_ENKUlS2_E0_clES2_ in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
  "__ZN5arrow5fieldENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt10shared_ptrINS_8DataTypeEEbS6_IKNS_16KeyValueMetadataEE", referenced from:
      __ZN24HashAggregateTestFixture5SetUpEv in hash_aggregate_test.cc.o
      __Z11make_schemaRKN6hustle7catalog11TableSchemaE in libhustle_src_storage.a(util.cc.o)
      __ZN6hustle9operators13HashAggregate12OutputSchemaENS0_15AggregateKernelERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE in libhustle_src_operators.a(hash_aggregate.cc.o)
      __ZN6hustle7details10TestFieldsEN5arrow4Type4typeE in libhustle_src_type.a(type_helper.cc.o)
  "__ZN5arrow6StatusC1ENS_10StatusCodeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE", referenced from:
      __ZN5arrow6Status8FromArgsIJRA46_KcRxRA2_S2_EEES0_NS_10StatusCodeEDpOT_ in hash_aggregate_test.cc.o
      __ZN5arrow6Status8FromArgsIJRA36_KcRxRA19_S2_S5_RA2_S2_EEES0_NS_10StatusCodeEDpOT_ in hash_aggregate_test.cc.o
      __ZN5arrow6Status8FromArgsIJRA32_KcxRA14_S2_RxEEES0_NS_10StatusCodeEDpOT_ in hash_aggregate_test.cc.o
      __ZN5arrow6Status8FromArgsIJRA50_KcxRA22_S2_RxEEES0_NS_10StatusCodeEDpOT_ in hash_aggregate_test.cc.o
      __ZN5arrow6Status8FromArgsIJRA46_KcRxRA2_S2_EEES0_NS_10StatusCodeEDpOT_ in libhustle_src_operators.a(hash_aggregate.cc.o)
      __ZN5arrow6Status8FromArgsIJRA36_KcRxRA19_S2_S5_RA2_S2_EEES0_NS_10StatusCodeEDpOT_ in libhustle_src_operators.a(hash_aggregate.cc.o)
      __ZN5arrow6Status8FromArgsIJRA32_KcxRA14_S2_RxEEES0_NS_10StatusCodeEDpOT_ in libhustle_src_operators.a(hash_aggregate.cc.o)
      ...
  "__ZN5arrow6schemaESt6vectorISt10shared_ptrINS_5FieldEESaIS3_EES1_IKNS_16KeyValueMetadataEE", referenced from:
      __ZN24HashAggregateTestFixture5SetUpEv in hash_aggregate_test.cc.o
  "__ZN5arrow7struct_ERKSt6vectorISt10shared_ptrINS_5FieldEESaIS3_EE", referenced from:
      __ZN6hustle9operators13HashAggregate10InitializeEPNS_4TaskE in libhustle_src_operators.a(hash_aggregate.cc.o)
      __ZN6hustle7details10TestFieldsEN5arrow4Type4typeE in libhustle_src_type.a(type_helper.cc.o)
  "__ZN5arrow8internal14DieWithMessageERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE", referenced from:
      __ZN5arrow6ResultISt10shared_ptrINS_12ArrayBuilderEEEC1ERKNS_6StatusE in libhustle_src_type.a(type_helper.cc.o)
  "__ZN5arrow8internal19DictionaryMemoTable12GetArrayDataExPSt10shared_ptrINS_9ArrayDataEE", referenced from:
      __ZN5arrow8internal21DictionaryBuilderBaseINS_18AdaptiveIntBuilderENS_9Int32TypeEE20FinishWithDictOffsetExPSt10shared_ptrINS_9ArrayDataEES8_ in libhustle_src_type.a(type_helper.cc.o)
      __ZN5arrow8internal21DictionaryBuilderBaseINS_18AdaptiveIntBuilderENS_8Int8TypeEE20FinishWithDictOffsetExPSt10shared_ptrINS_9ArrayDataEES8_ in libhustle_src_type.a(type_helper.cc.o)
  "__ZN5arrow8internal19DictionaryMemoTableC1EPNS_10MemoryPoolERKSt10shared_ptrINS_8DataTypeEE", referenced from:
      __ZN5arrow8internal21DictionaryBuilderBaseINS_18AdaptiveIntBuilderENS_8Int8TypeEEC2IS3_EENSt9enable_ifIXntsrSt10is_base_ofINS_19FixedSizeBinaryTypeET_E5valueERKSt10shared_ptrINS_8DataTypeEEE4typeEPNS_10MemoryPoolE in libhustle_src_type.a(type_helper.cc.o)
      __ZN5arrow8internal21DictionaryBuilderBaseINS_18AdaptiveIntBuilderENS_9Int32TypeEEC2IS3_EENSt9enable_ifIXntsrSt10is_base_ofINS_19FixedSizeBinaryTypeET_E5valueERKSt10shared_ptrINS_8DataTypeEEE4typeEPNS_10MemoryPoolE in libhustle_src_type.a(type_helper.cc.o)
  "__ZN5arrow9ArrayData4MakeERKSt10shared_ptrINS_8DataTypeEExSt6vectorIS1_INS_6BufferEESaIS8_EES6_IS1_IS0_ESaISB_EExx", referenced from:
      __ZN5arrow15BaseListBuilderINS_13LargeListTypeEE14FinishInternalEPSt10shared_ptrINS_9ArrayDataEE in libhustle_src_type.a(type_helper.cc.o)
      __ZN5arrow15BaseListBuilderINS_8ListTypeEE14FinishInternalEPSt10shared_ptrINS_9ArrayDataEE in libhustle_src_type.a(type_helper.cc.o)
  "__ZN5arrow9ArrayData4MakeERKSt10shared_ptrINS_8DataTypeEExSt6vectorIS1_INS_6BufferEESaIS8_EExx", referenced from:
      __ZN5arrow14NumericBuilderINS_10DoubleTypeEE14FinishInternalEPSt10shared_ptrINS_9ArrayDataEE in hash_aggregate_test.cc.o
      __ZN5arrow14NumericBuilderINS_9Int64TypeEE14FinishInternalEPSt10shared_ptrINS_9ArrayDataEE in hash_aggregate_test.cc.o
      __ZN5arrow17BaseBinaryBuilderINS_10BinaryTypeEE14FinishInternalEPSt10shared_ptrINS_9ArrayDataEE in hash_aggregate_test.cc.o
      __Z17copy_record_batchRKSt10shared_ptrIN5arrow11RecordBatchEE in libhustle_src_storage.a(util.cc.o)
      __ZN5arrow14NumericBuilderINS_10DoubleTypeEE14FinishInternalEPSt10shared_ptrINS_9ArrayDataEE in libhustle_src_operators.a(hash_aggregate.cc.o)
      __ZN5arrow14NumericBuilderINS_9Int64TypeEE14FinishInternalEPSt10shared_ptrINS_9ArrayDataEE in libhustle_src_operators.a(hash_aggregate.cc.o)
      __ZN5arrow17BaseBinaryBuilderINS_10BinaryTypeEE14FinishInternalEPSt10shared_ptrINS_9ArrayDataEE in libhustle_src_operators.a(hash_aggregate.cc.o)
      ...
  "__ZN5arrow9MakeArrayERKSt10shared_ptrINS_9ArrayDataEE", referenced from:
      __ZNK6hustle7storage5Block10get_columnEi in hash_aggregate_test.cc.o
      __ZN6hustle7storage5Block9out_blockEPvPFiS2_iPPcS4_E in libhustle_src_storage.a(block.cc.o)
      __ZN6hustle7storage5Block5printEv in libhustle_src_storage.a(block.cc.o)
      __ZNK6hustle7storage5Block16get_valid_columnEv in libhustle_src_storage.a(table.cc.o)
      __ZNK6hustle7storage5Block10get_columnEi in libhustle_src_operators.a(lazy_table.cc.o)
      __ZN6hustle7Context26apply_indices_internal_strERKSt10shared_ptrIN5arrow12ChunkedArrayEERKS1_INS2_5ArrayEESA_i in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN6hustle7Context23apply_indices_internal2IxEEvRKSt10shared_ptrIN5arrow12ChunkedArrayEEPPKT_RKS2_INS3_5ArrayEESF_SF_i in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      ...
  "__ZNK5arrow5Array6EqualsERKSt10shared_ptrIS0_ERKNS_12EqualOptionsE", referenced from:
      __ZN38HashAggregateTestFixture_MeanTest_Test8TestBodyEv in hash_aggregate_test.cc.o
      __ZN37HashAggregateTestFixture_SumTest_Test8TestBodyEv in hash_aggregate_test.cc.o
      __ZN39HashAggregateTestFixture_CountTest_Test8TestBodyEv in hash_aggregate_test.cc.o
      __ZN55HashAggregateTestFixture_SumWithGroupByOrderByTest_Test8TestBodyEv in hash_aggregate_test.cc.o
  "__ZNK5arrow5Field8WithTypeERKSt10shared_ptrINS_8DataTypeEE", referenced from:
      __ZN5arrow15BaseListBuilderINS_8ListTypeEEC2EPNS_10MemoryPoolERKSt10shared_ptrINS_12ArrayBuilderEERKS5_INS_8DataTypeEE in libhustle_src_type.a(type_helper.cc.o)
      __ZN5arrow15BaseListBuilderINS_13LargeListTypeEEC2EPNS_10MemoryPoolERKSt10shared_ptrINS_12ArrayBuilderEERKS5_INS_8DataTypeEE in libhustle_src_type.a(type_helper.cc.o)
      __ZNK5arrow15BaseListBuilderINS_13LargeListTypeEE4typeEv in libhustle_src_type.a(type_helper.cc.o)
      __ZNK5arrow15BaseListBuilderINS_8ListTypeEE4typeEv in libhustle_src_type.a(type_helper.cc.o)
  "__ZNK5arrow6Schema11field_namesB5cxx11Ev", referenced from:
      __ZN6hustle7storage5Block9out_blockEPvPFiS2_iPPcS4_E in libhustle_src_storage.a(block.cc.o)
  "__ZNK5arrow6Schema13GetFieldIndexERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE", referenced from:
      __ZN6hustle9operators9LazyTable17MaterializeColumnENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE in libhustle_src_operators.a(lazy_table.cc.o)
      __ZN6hustle9operators9LazyTable17MaterializeColumnEPNS_4TaskENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERN5arrow5DatumE in libhustle_src_operators.a(lazy_table.cc.o)
  "__ZNK5arrow6Schema14GetFieldByNameERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE", referenced from:
      __ZN6hustle9operators13HashAggregate10InitializeEPNS_4TaskE in libhustle_src_operators.a(hash_aggregate.cc.o)
      __ZN6hustle9operators14OperatorResult11materializeERKSt6vectorINS0_15ColumnReferenceESaIS3_EEb in libhustle_src_operators.a(operator_result.cc.o)
  "__ZNK5arrow6Status8ToStringB5cxx11Ev", referenced from:
      __Z15evaluate_statusRKN5arrow6StatusEPKci in libhustle_src_storage.a(util.cc.o)
      __ZN5arrow6ResultISt10shared_ptrINS_12ArrayBuilderEEEC1ERKNS_6StatusE in libhustle_src_type.a(type_helper.cc.o)
  "__ZNK5arrow8DataType18ComputeFingerprintB5cxx11Ev", referenced from:
      __ZTVN5arrow12BaseListTypeE in libhustle_src_type.a(type_helper.cc.o)
      __ZTVN5arrow10NestedTypeE in libhustle_src_type.a(type_helper.cc.o)
  "__ZNK5arrow8DataType26ComputeMetadataFingerprintB5cxx11Ev", referenced from:
      __ZTVN5arrow12BaseListTypeE in libhustle_src_type.a(type_helper.cc.o)
      __ZTVN5arrow10NestedTypeE in libhustle_src_type.a(type_helper.cc.o)
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status
make[3]: *** [src/operators/hustle_hash_aggregate_test] Error 1
make[2]: *** [src/operators/CMakeFiles/hustle_hash_aggregate_test.dir/all] Error 2
make[1]: *** [src/operators/CMakeFiles/hustle_hash_aggregate_test.dir/rule] Error 2
make: *** [hustle_hash_aggregate_test] Error 2

Select Operator

Select operator takes in arrow's table as an input and produces a filtered arrow table as an output. It should be able to support multiple predicates connected through ands and ors. The predicates could be on fields of different data types. Gandiva's expression tree evaluation should be used.

Use Gandiva to write a select operator with one clause of int32 type
Support multiple clauses
Support clauses of different data types

blink tree加锁问题

您好，我在https://stackoverflow.com/questions/58478390/blink-tree-insert-operation看到您回答了一个blink tree的问题。我不明白的是，为什么在parent层向右移动的时候，需要先拿右边的锁，然后再释放左边的锁。PostgreSQL实现blink tree的时候，没有采用这种chain lock的方式，而是直接先放左边的，然后拿右边的锁。我的邮箱是mengqingzhong.gm[at]gmail.com，期待您的回复。

Interface-based Block Hierarchy

The current metadata-attached block implementation is somewhat clunky overall, and may cause issues if further block extensions are implemented. Look into an interface-based system to make the overall design significantly more compartmentalized.

Build Problem

There is a problem with compiling the Catalog. I the following errors from linking sqlite3:
https://app.circleci.com/pipelines/github/UWQuickstep/hustle/12/workflows/4eae7dc5-21cd-4d7a-a00f-771f0d09917b/jobs/13. (The build report is from CircleCI)

Any ideas @lichengxihuang?

User-Defined Type in Arrow

support custom data types (i.e. IP addrress was the running example at the time)

@johnchronis Im actually pretty confident that we can use the arrow::Extention type in arrow to implement IP Address. In fact, this mysterious Extention type is designed for user-defined type. The extention type can, in itself, defines its relationship with other types (ideally, of course, using what we have proposed ini #94).

In fact, arrow had a test IPv4 type in their code base:
https://github.com/apache/arrow/blob/2d140c3eeecca3ff7823edc8c9562ebd6a1c336a/cpp/src/arrow/array/array_view_test.cc#L55-L78

We can open up an issue to address user-defined type - will need to experiment a little bit on this.

Originally posted by @GindaChen in #28 (comment)

Multithreaded Predicate execution

Currently, in the select operator, we do multithreading between different batches (i.e a set of blocks). In this task, we will experiment and see if we are getting speed-up by doing much fine multithreaded execution.

Do multithreaded execution of the leaf nodes (i.e comparison operator execution) in the predicate tree.
Have this feature togglable so that it can enabled and disabled easily using macros.
Compare the SSB benchmark performance by enabling it and disabling it.
Add further robust unit tests to the Select operators

Ref:
Comparison operator call,

hustle/src/operators/select.cc

Line 95 in df14c96

return Filter(block, node->predicate_);

Sample ref for multithreaded implementation,

hustle/src/operators/select.cc

Line 55 in df14c96

ctx->spawnTask(CreateTaskChain(

Multi-threaded Select operator

As the first step towards a multi-threaded implementation of Hustle operators, I will make selections multi-threaded. This will set the stage for multi-threaded implementations of the other operators. I'm seeing if we can reuse any code from the mini-version of Quickstep that UW-Madison submitted for the SIGMOD 2018 contest.

Performance Statistics for Metadata

Per-table counters that record the effectiveness of metadata units. Must be thread-safe.

(Mac) Unmached mangled name reference cause linker error in g++

Procedure to reproduce

Use the gcc@10 on your machine: /usr/local/Cellar/gcc/10.2.0_4/bin/g++-10 ~~Install the g++10 on mac. This requires manual compile, and hopefully we can host it on brew some day... Call @GindaChen if you want a pre-built g++.~~
Check out arrow (3.0.0) and build with the new g++ compiler. Example:

git clone https://github.com/apache/arrow.git
cd arrow
git checkout apache-arrow-3.0.0
mkdir -p cpp/release
cd cpp/release

cmake  -DARROW_COMPUTE=ON \
-DARROW_DEPENDENCY_SOURCE=AUTO \
-DCMAKE_BUILD_TYPE=Debug \
-DCMAKE_C_COMPILER=/path/to/gcc-10.1 \    # <- Your gcc10 Compiler path
-DCMAKE_CXX_COMPILER=/path/to/g++-10.1 \  # <- Your g++10 Compiler path
-DCMAKE_CXX_STANDARD_REQUIRED=ON \
-DCMAKE_CXX_STANDARD=17 ..
make -j install

Create a new toolchain item and use that toolchain in CMake. Also specify the cmake flags:

-DCMAKE_BUILD_TYPE=Debug  
-DCMAKE_C_COMPILER=gcc-10.1    # <- Your gcc10 Compiler path
-DCMAKE_CXX_COMPILER=g++-10.1  # <- Your g++10 Compiler path
-DCMAKE_CXX_STANDARD_REQUIRED=ON 
-DCMAKE_CXX_STANDARD=17

Do the all ctest compile in CLion.

Error Description

The dump fiile libarrow.a.dump.zip does not contains the mangled name that compiler generates. For example,

"__ZNK5arrow6Status8ToStringB5cxx11Ev", referenced from:
      __Z15evaluate_statusRKN5arrow6StatusEPKci in libhustle_src_storage.a(util.cc.o)

is produced by the compiler, but the closest we got is

U __ZNK5arrow6Status8ToStringEv

To compare, the one built using g++ (first one) has an infix B5cxx11 that the second one (in libarrow) does not. libarrow is bulit by clang (and will fail if built by g++), and is possible that it does not carry with some information in the ABI.

# Hustle-referenced symbol
__ZNK5arrow6Status8ToStringB5cxx11Ev
# Arrow symbol, produced by the dump
__ZNK5arrow6Status8ToStringEv

Complete Dump

libarrow.a dump file. Produced by

nm libarrow.a > ~/libarrow.a.dump

The complete compiler error g++-10

====================[ Build | hustle_select_test | Debug-GPP17 ]================
/Applications/CLion.app/Contents/bin/cmake/mac/bin/cmake --build /Users/gindachen/Project/hustle/cmake-build-debug-gpp17 --target hustle_select_test -- -j 9
-- Checking program counter fetch from ucontext_t member: uc_mcontext.gregs[REG_EIP]
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.gregs[REG_RIP]
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.sc_ip
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.uc_regs->gregs[PT_NIP]
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.gregs[R15]
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.arm_pc
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.mc_eip
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.mc_rip
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.__gregs[_REG_EIP]
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext.__gregs[_REG_RIP]
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext->ss.eip
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext->__ss.__eip
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext->ss.rip
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Failed
-- Checking program counter fetch from ucontext_t member: uc_mcontext->__ss.__rip
-- Performing Test PC_FROM_UCONTEXT_COMPILES
-- Performing Test PC_FROM_UCONTEXT_COMPILES - Success
-- Found program counter field in ucontext_t: uc_mcontext->__ss.__rip
-- Arrow SO and ABI version: 300
-- Arrow full SO version: 300.0.0
-- Found the Arrow core library: /usr/local/lib/libarrow.dylib
-- Found the Arrow Python library: ARROW_PYTHON_LIB_PATH-NOTFOUND
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/gindachen/Project/hustle/cmake-build-debug-gpp17
[  1%] Built target absl_log_severity
[  3%] Built target hustle_src_scheduler_SchedulerTypedefs
[  3%] Built target hustle_src_utils_meta_FunctionTraits
[  3%] Built target gflags_nothreads-static
[  5%] Built target absl_spinlock_wait
[  7%] Built target gtest
[  7%] Built target absl_exponential_biased
[  9%] Built target hustle_src_utils_Macros
[ 13%] Built target glog
[ 15%] Built target hustle_src_utils_ThreadSafeQueue
[ 15%] Built target hustle_src_utils_ParallelHashmap
[ 17%] Built target hustle_src_utils_Config
[ 17%] Built target hustle_src_utils_ContainerUtil
[ 17%] Built target absl_city
[ 19%] Built target absl_civil_time
[ 19%] Built target hustle_src_type
[ 19%] Built target absl_int128
[ 23%] Built target absl_time_zone
[ 27%] Built target hustle_src_scheduler_WorkerMessage
[ 27%] Built target hustle_src_threading_ThreadUtil
[ 29%] Built target hustle_src_scheduler_SchedulerMessage
[ 29%] Built target hustle_src_threading_Mutex
[ 31%] Built target hustle_src_scheduler_SchedulerFlags
[ 33%] Built target hustle_src_scheduler_Node
[ 35%] Built target hustle_src_utils_SyncStream
[ 35%] Built target hustle_src_scheduler_SchedulerInterface
[ 37%] Built target hustle_src_scheduler_TaskName
[ 37%] Built target hustle_src_utils_EventProfiler
[ 37%] Built target gtest_main
[ 39%] Built target gmock
[ 39%] Built target absl_raw_logging_internal
[ 39%] Built target hustle_src_scheduler_TaskDescription
[ 43%] Built target absl_bad_optional_access
[ 43%] Built target absl_throw_delegate
[ 47%] Built target absl_base
[ 47%] Built target absl_bad_variant_access
[ 50%] Built target absl_strings_internal
[ 52%] Built target absl_debugging_internal
[ 54%] Built target hustle_src_scheduler_Task
[ 54%] Built target absl_malloc_internal
[ 54%] Built target absl_demangle_internal
[ 56%] Built target absl_stacktrace
[ 64%] Built target absl_strings
[ 64%] Built target hustle_src_scheduler_Worker
[ 64%] Built target hustle_src_optimizer_ExecutionPlan
[ 64%] Built target absl_graphcycles_internal
[ 68%] Built target absl_symbolize
[ 70%] Built target absl_cord
[ 70%] Built target absl_hash
[ 72%] Built target absl_time
[ 72%] Built target hustle_src_scheduler_Scheduler
[ 76%] Built target absl_synchronization
[ 78%] Built target hustle_src_utils_ArrowComputeWrappers
[ 80%] Built target absl_hashtablez_sampler
[ 80%] Built target absl_raw_hash_set
[ 88%] Built target hustle_src_operators
[ 92%] Built target hustle_src_storage
[ 94%] Built target hustle_src_resolver
[ 96%] Built target hustleDB
[ 98%] Built target hustle_src_catalog
[100%] Built target sqlite_utils
[100%] Built target sqlite3
[100%] Linking CXX executable hustle_select_test
ld: warning: dylib (/usr/local/lib/libconfig++.dylib) was built for newer macOS version (11.0) than being linked (10.15)
ld: warning: dylib (/usr/local/lib/libarrow.dylib) was built for newer macOS version (11.0) than being linked (10.15)
ld: warning: dylib (/usr/local/gcc-10.1.0/lib/libstdc++.dylib) was built for newer macOS version (10.16) than being linked (10.15)
Undefined symbols for architecture x86_64:
  "__ZN5arrow11RecordBatch4MakeESt10shared_ptrINS_6SchemaEExSt6vectorIS1_INS_9ArrayDataEESaIS6_EE", referenced from:
      __ZN6hustle7storage5Block11get_recordsEv in libhustle_src_storage.a(util.cc.o)
      __Z17copy_record_batchRKSt10shared_ptrIN5arrow11RecordBatchEE in libhustle_src_storage.a(util.cc.o)
  "__ZN5arrow12ArrayBuilder6FinishEPSt10shared_ptrINS_5ArrayEE", referenced from:
      __ZN39SelectTestFixture_SingleSelectTest_Test8TestBodyEv in select_test.cc.o
      __ZN36SelectTestFixture_AndSelectTest_Test8TestBodyEv in select_test.cc.o
      __ZN35SelectTestFixture_OrSelectTest_Test8TestBodyEv in select_test.cc.o
      __ZN49SelectTestFixture_SingleSelectManyBlocksTest_Test8TestBodyEv in select_test.cc.o
      __ZN46SelectTestFixture_AndSelectManyBlocksTest_Test8TestBodyEv in select_test.cc.o
      __ZZN6hustle7Context13apply_indicesEPNS_4TaskEN5arrow5DatumES4_S4_RS4_ENKUlS2_E_clES2_ in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
  "__ZN5arrow12ChunkedArrayC1ESt6vectorISt10shared_ptrINS_5ArrayEESaIS4_EE", referenced from:
      __ZN9__gnu_cxx13new_allocatorIN5arrow12ChunkedArrayEE9constructIS2_JRSt6vectorISt10shared_ptrINS1_5ArrayEESaIS8_EEEEEvPT_DpOT0_ in select_test.cc.o
      __ZN9__gnu_cxx13new_allocatorIN5arrow12ChunkedArrayEE9constructIS2_JRSt6vectorISt10shared_ptrINS1_5ArrayEESaIS8_EEEEEvPT_DpOT0_ in libhustle_src_storage.a(util.cc.o)
      __ZN9__gnu_cxx13new_allocatorIN5arrow12ChunkedArrayEE9constructIS2_JRSt6vectorISt10shared_ptrINS1_5ArrayEESaIS8_EEEEEvPT_DpOT0_ in libhustle_src_operators.a(select.cc.o)
      __ZN9__gnu_cxx13new_allocatorIN5arrow12ChunkedArrayEE9constructIS2_JRSt6vectorISt10shared_ptrINS1_5ArrayEESaIS8_EEEEEvPT_DpOT0_ in libhustle_src_operators.a(lazy_table.cc.o)
      __ZN5arrow12ChunkedArrayC1ESt10shared_ptrINS_5ArrayEE in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN9__gnu_cxx13new_allocatorIN5arrow12ChunkedArrayEE9constructIS2_JRSt6vectorISt10shared_ptrINS1_5ArrayEESaIS8_EEEEEvPT_DpOT0_ in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
  "__ZN5arrow12StringScalarC1ENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE", referenced from:
      __ZN9__gnu_cxx13new_allocatorIN5arrow12StringScalarEE9constructIS2_JRA3_KcEEEvPT_DpOT0_ in select_test.cc.o
  "__ZN5arrow13SchemaBuilder8AddFieldERKSt10shared_ptrINS_5FieldEE", referenced from:
      __Z11make_schemaRKN6hustle7catalog11TableSchemaE in libhustle_src_storage.a(util.cc.o)
      __ZN6hustle9operators14OperatorResult11materializeERKSt6vectorINS0_15ColumnReferenceESaIS3_EEb in libhustle_src_operators.a(operator_result.cc.o)
  "__ZN5arrow15MakeArrayOfNullERKSt10shared_ptrINS_8DataTypeEExPNS_10MemoryPoolE", referenced from:
      __Z21array_to_chunkedarraySt10shared_ptrIN5arrow5ArrayEEi in libhustle_src_storage.a(util.cc.o)
      __ZN6hustle7Context26apply_indices_internal_strERKSt10shared_ptrIN5arrow12ChunkedArrayEERKS1_INS2_5ArrayEESA_i in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN6hustle7Context23apply_indices_internal2IxEEvRKSt10shared_ptrIN5arrow12ChunkedArrayEEPPKT_RKS2_INS3_5ArrayEESF_SF_i in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN6hustle7Context22apply_indices_internalIxEEvRKSt10shared_ptrIN5arrow12ChunkedArrayEEPPKT_RKS2_INS3_5ArrayEESF_i in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN6hustle7Context23apply_indices_internal2IjEEvRKSt10shared_ptrIN5arrow12ChunkedArrayEEPPKT_RKS2_INS3_5ArrayEESF_SF_i in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN6hustle7Context22apply_indices_internalIjEEvRKSt10shared_ptrIN5arrow12ChunkedArrayEEPPKT_RKS2_INS3_5ArrayEESF_i in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
  "__ZN5arrow18RecordBatchBuilder4MakeERKSt10shared_ptrINS_6SchemaEEPNS_10MemoryPoolEPSt10unique_ptrIS0_St14default_deleteIS0_EE", referenced from:
      __Z18read_from_csv_filePKcSt10shared_ptrIN5arrow6SchemaEEib in libhustle_src_storage.a(util.cc.o)
  "__ZN5arrow2io12ReadableFile4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPNS_10MemoryPoolE", referenced from:
      __Z14read_from_filePKcbS0_ in libhustle_src_storage.a(util.cc.o)
  "__ZN5arrow2io16FileOutputStream4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb", referenced from:
      __Z13write_to_filePKcRN6hustle7storage7DBTableE in libhustle_src_storage.a(util.cc.o)
  "__ZN5arrow3ipc13NewFileWriterEPNS_2io12OutputStreamERKSt10shared_ptrINS_6SchemaEERKNS0_15IpcWriteOptionsERKS4_IKNS_16KeyValueMetadataEE", referenced from:
      __Z13write_to_filePKcRN6hustle7storage7DBTableE in libhustle_src_storage.a(util.cc.o)
  "__ZN5arrow3ipc21RecordBatchFileReader4OpenERKSt10shared_ptrINS_2io16RandomAccessFileEERKNS0_14IpcReadOptionsE", referenced from:
      __Z14read_from_filePKcbS0_ in libhustle_src_storage.a(util.cc.o)
  "__ZN5arrow4util6detail19StringStreamWrapper3strB5cxx11Ev", referenced from:
      __ZN5arrow4util13StringBuilderIJRA46_KcRxRA2_S2_EEENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEDpOT_ in select_test.cc.o
      __ZN5arrow4util13StringBuilderIJRA36_KcRxRA19_S2_S5_RA2_S2_EEENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEDpOT_ in select_test.cc.o
      __ZN5arrow4util13StringBuilderIJRA50_KcxRA22_S2_RxEEENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEDpOT_ in select_test.cc.o
      __ZN5arrow4util13StringBuilderIJRA46_KcRxRA2_S2_EEENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEDpOT_ in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN5arrow4util13StringBuilderIJRA36_KcRxRA19_S2_S5_RA2_S2_EEENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEDpOT_ in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN5arrow4util13StringBuilderIJRA24_KcEEENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEDpOT_ in libhustle_src_storage.a(sma.cc.o)
  "__ZN5arrow5DatumC1ERKSt10shared_ptrINS_5ArrayEE", referenced from:
      __ZN6hustle9operators6Select6FilterERKSt10shared_ptrINS_7storage5BlockEERKS2_INS0_9PredicateEE in libhustle_src_operators.a(select.cc.o)
      __ZN5arrow5DatumC1INS_12BooleanArrayEvEERKSt10shared_ptrIT_E in libhustle_src_operators.a(select.cc.o)
      __ZZZN6hustle7Context5matchEPNS_4TaskERKN5arrow5DatumES6_ENKUlS2_E_clES2_ENKUlvE_clEv in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN6hustle7Context18apply_filter_blockERKSt10shared_ptrIN5arrow5ArrayEES6_RSt6vectorIS4_SaIS4_EE in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZZN6hustle7Context12apply_filterEPNS_4TaskERKN5arrow5DatumES6_RS4_ENKUlS2_E_clES2_ in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN6hustle7storage3SmaC2ERKSt10shared_ptrIN5arrow5ArrayEE in libhustle_src_storage.a(sma.cc.o)
      __ZN6hustle7storage3SmaC1ERKSt10shared_ptrIN5arrow5ArrayEE in libhustle_src_storage.a(sma.cc.o)
      ...
  "__ZN5arrow5DatumC1ESt10shared_ptrINS_12ChunkedArrayEE", referenced from:
      __ZN9__gnu_cxx13new_allocatorIN6hustle9operators9LazyTableEE9constructIS3_JRSt10shared_ptrINS1_7storage7DBTableEERS6_IN5arrow12ChunkedArrayEENSB_5DatumESF_EEEvPT_DpOT0_ in libhustle_src_operators.a(select.cc.o)
      __ZN6hustle9operators9LazyTable17MaterializeColumnEi in libhustle_src_operators.a(lazy_table.cc.o)
      __ZN6hustle9operators9LazyTable17MaterializeColumnEPNS_4TaskEiRN5arrow5DatumE in libhustle_src_operators.a(lazy_table.cc.o)
      __ZZN6hustle7Context13apply_indicesEPNS_4TaskEN5arrow5DatumES4_S4_RS4_ENKUlS2_E0_clES2_ in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
  "__ZN5arrow5fieldENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt10shared_ptrINS_8DataTypeEEbS6_IKNS_16KeyValueMetadataEE", referenced from:
      __ZN17SelectTestFixture5SetUpEv in select_test.cc.o
      __Z11make_schemaRKN6hustle7catalog11TableSchemaE in libhustle_src_storage.a(util.cc.o)
  "__ZN5arrow6StatusC1ENS_10StatusCodeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE", referenced from:
      __ZN5arrow6Status8FromArgsIJRA46_KcRxRA2_S2_EEES0_NS_10StatusCodeEDpOT_ in select_test.cc.o
      __ZN5arrow6Status8FromArgsIJRA36_KcRxRA19_S2_S5_RA2_S2_EEES0_NS_10StatusCodeEDpOT_ in select_test.cc.o
      __ZN5arrow6Status8FromArgsIJRA50_KcxRA22_S2_RxEEES0_NS_10StatusCodeEDpOT_ in select_test.cc.o
      __ZN5arrow6Status8FromArgsIJRA46_KcRxRA2_S2_EEES0_NS_10StatusCodeEDpOT_ in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN5arrow6Status8FromArgsIJRA36_KcRxRA19_S2_S5_RA2_S2_EEES0_NS_10StatusCodeEDpOT_ in libhustle_src_utils_ArrowComputeWrappers.a(arrow_compute_wrappers.cc.o)
      __ZN5arrow6Status8FromArgsIJRA24_KcEEES0_NS_10StatusCodeEDpOT_ in libhustle_src_storage.a(sma.cc.o)
  "__ZN5arrow6schemaESt6vectorISt10shared_ptrINS_5FieldEESaIS3_EES1_IKNS_16KeyValueMetadataEE", referenced from:
      __ZN17SelectTestFixture5SetUpEv in select_test.cc.o
  "__ZN5arrow9ArrayData4MakeERKSt10shared_ptrINS_8DataTypeEExSt6vectorIS1_INS_6BufferEESaIS8_EExx", referenced from:
      __ZN5arrow14NumericBuilderINS_10DoubleTypeEE14FinishInternalEPSt10shared_ptrINS_9ArrayDataEE in select_test.cc.o
      __ZN5arrow14NumericBuilderINS_9Int64TypeEE14FinishInternalEPSt10shared_ptrINS_9ArrayDataEE in select_test.cc.o
      __ZN5arrow17BaseBinaryBuilderINS_10BinaryTypeEE14FinishInternalEPSt10shared_ptrINS_9ArrayDataEE in select_test.cc.o
      __Z17copy_record_batchRKSt10shared_ptrIN5arrow11RecordBatchEE in libhustle_src_storage.a(util.cc.o)
      __ZN6hustle7storage5BlockC2EiRKSt10shared_ptrIN5arrow6SchemaEEib in libhustle_src_storage.a(block.cc.o)
      __ZN6hustle7storage5BlockC1EiRKSt10shared_ptrIN5arrow6SchemaEEib in libhustle_src_storage.a(block.cc.o)
      __ZN6hustle7storage5BlockC2EiSt10shared_ptrIN5arrow11RecordBatchEEib in libhustle_src_storage.a(block.cc.o)
      ...
  "__ZN5arrow9MakeArrayERKSt10shared_ptrINS_9ArrayDataEE", referenced from:
      __ZNK6hustle7storage5Block10get_columnEi in select_test.cc.o
      __ZNK6hustle7storage5Block18get_column_by_nameERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE in libhustle_src_operators.a(select.cc.o)
      __ZNK6hustle7storage5Block10get_columnEi in libhustle_src_storage.a(ma_block.cc.o)
      __ZN6hustle7storage5Block9out_blockEPvPFiS2_iPPcS4_E in libhustle_src_storage.a(block.cc.o)
      __ZN6hustle7storage5Block5printEv in libhustle_src_storage.a(block.cc.o)
      __ZNK6hustle7storage5Block16get_valid_columnEv in libhustle_src_storage.a(table.cc.o)
      __ZNK6hustle7storage5Block10get_columnEi in libhustle_src_operators.a(lazy_table.cc.o)
      ...
  "__ZNK5arrow5Array6EqualsERKSt10shared_ptrIS0_ERKNS_12EqualOptionsE", referenced from:
      __ZN39SelectTestFixture_SingleSelectTest_Test8TestBodyEv in select_test.cc.o
      __ZN36SelectTestFixture_AndSelectTest_Test8TestBodyEv in select_test.cc.o
      __ZN35SelectTestFixture_OrSelectTest_Test8TestBodyEv in select_test.cc.o
      __ZN49SelectTestFixture_SingleSelectManyBlocksTest_Test8TestBodyEv in select_test.cc.o
      __ZN46SelectTestFixture_AndSelectManyBlocksTest_Test8TestBodyEv in select_test.cc.o
  "__ZNK5arrow6Schema11field_namesB5cxx11Ev", referenced from:
      __ZN6hustle7storage5Block9out_blockEPvPFiS2_iPPcS4_E in libhustle_src_storage.a(block.cc.o)
  "__ZNK5arrow6Schema13GetFieldIndexERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE", referenced from:
      __ZNK6hustle7storage5Block18get_column_by_nameERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE in libhustle_src_operators.a(select.cc.o)
      __ZN6hustle7storage21MetadataAttachedBlock14SearchMetadataERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN5arrow5DatumENSA_7compute15CompareOperatorE in libhustle_src_operators.a(select.cc.o)
      __ZN6hustle9operators9LazyTable17MaterializeColumnENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE in libhustle_src_operators.a(lazy_table.cc.o)
      __ZN6hustle9operators9LazyTable17MaterializeColumnEPNS_4TaskENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERN5arrow5DatumE in libhustle_src_operators.a(lazy_table.cc.o)
  "__ZNK5arrow6Schema14GetFieldByNameERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE", referenced from:
      __ZN6hustle9operators14OperatorResult11materializeERKSt6vectorINS0_15ColumnReferenceESaIS3_EEb in libhustle_src_operators.a(operator_result.cc.o)
  "__ZNK5arrow6Status8ToStringB5cxx11Ev", referenced from:
      __Z15evaluate_statusRKN5arrow6StatusEPKci in libhustle_src_storage.a(util.cc.o)
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status
make[3]: *** [src/operators/hustle_select_test] Error 1
make[2]: *** [src/operators/CMakeFiles/hustle_select_test.dir/all] Error 2
make[1]: *** [src/operators/CMakeFiles/hustle_select_test.dir/rule] Error 2
make: *** [hustle_select_test] Error 2

Operators Module (Except Tests)
Storage Module (Except Tests)
Resolver and Catalog and all test files
Integrate with CircleCI

Flaky Tests

The parser, catalog, resolver, HustleDB tests are flaky and fail with a seg fault.
(The test are currently disabled)
To reproduce run all tests with ctest.