Engine columnar crash with infinity core dump loop

[2peter3]

Engine columnar crash with infinity core dump loop

[2peter3]

As an additional note, I can mention that simultaneous replacements with just one value work without any issues. Therefore, there are likely problems when multiple replacements are made at the same time, including with concurrent connections.

[2peter3]

manticoresoftware/columnar#20

i think it is related to this

[tomatolog]

could you upload your index and binlog as described in the manual

We could reproduce this crash locally and fix it or confirms that is was already fixed at manticoresoftware/columnar#20

[2peter3]

@tomatolog i have uploaded the data to manticore/write-only/issue-2176/dataFolder also @sanikolaev got a script via slack with a small file dump to reproduce it.

[2peter3]

fyi the bug is also presented on earlier versions:

Manticore 6.0.0 8de9df201@230206 (columnar 2.0.0 a7c703d@230130) (secondary 2.0.0 a7c703d@230130)

Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with Clang 15.0.4
Configured with flags: Configured with these definitions: -DDISTR_BUILD=bookworm -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmariadb.so.3 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore/data -DFULL_SHARE_DIR=/usr/share/manticore
Built on Linux x86_64 (bookworm) (cross-compiled)
Stack bottom = 0x7f636c05dc80, thread stack size = 0x20000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x1)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x1, stack=0x7f636c060000, stacksize=0x20000)
Trying system backtrace:
begin of system symbols:
/usr/bin/searchd(_Z12sphBacktraceib+0x22a)[0x555e82ca27ea]
/usr/bin/searchd(_ZN11CrashLogger11HandleCrashEi+0x355)[0x555e82b64fc5]
/lib/x86_64-linux-gnu/libc.so.6(+0x3c050)[0x7f639dc5b050]
/usr/bin/searchd(_ZN13LibcCIHash_fn4HashEPKhim+0x30)[0x555e83089540]
/usr/share/manticore/modules/lib_manticore_columnar.so(_ZN8columnar13Packer_Hash_c6AddDocEPKhi+0x20)[0x7f639d7f6d60]
/usr/share/manticore/modules/lib_manticore_columnar.so(_ZN8columnar9Builder_c7SetAttrEiPKhi+0x40)[0x7f639db94690]
/usr/bin/searchd(_Z15SetColumnarAttri8ESphAttrPN8columnar9Builder_iERSt10unique_ptrINS0_10Iterator_iESt14default_deleteIS4_EERN3sph8Vector_TIlNS9_13DefaultCopy_TIlEENS9_14DefaultRelimitENS9_16DefaultStorage_TIlEEEE+0x125)[0x555e83098d85]
/usr/bin/searchd(_ZNK9RtIndex_c15WriteAttributesER21SaveDiskDataContext_tR10CSphString+0x6b2)[0x555e82f4adc2]
/usr/bin/searchd(_ZNK9RtIndex_c12SaveDiskDataEPKcRK11VecTraits_TI17CSphRefcountedPtrIK11RtSegment_tEERK12ChunkStats_tR10CSphString+0x16e)[0x555e82f4ce8e]
/usr/bin/searchd(_ZN9RtIndex_c13SaveDiskChunkEbbb+0x5ba)[0x555e82f48f9a]
/usr/bin/searchd(_ZZN7Threads11CoRoutine_c13CreateContextESt8functionIFvvEE11VecTraits_TIhEENUlN5boost7context6detail10transfer_tEE_8__invokeES9_+0x1c)[0x555e8346e3dc]
/usr/bin/searchd(make_fcontext+0x2f)[0x555e8348d08f]
Trying boost backtrace:
 0# sphBacktrace(int, bool) in /usr/bin/searchd
 1# CrashLogger::HandleCrash(int) in /usr/bin/searchd
 2# 0x00007F639DC5B050 in /lib/x86_64-linux-gnu/libc.so.6
 3# LibcCIHash_fn::Hash(unsigned char const*, int, unsigned long) in /usr/bin/searchd
 4# columnar::Packer_Hash_c::AddDoc(unsigned char const*, int) in /usr/share/manticore/modules/lib_manticore_columnar.so
 5# columnar::Builder_c::SetAttr(int, unsigned char const*, int) in /usr/share/manticore/modules/lib_manticore_columnar.so
 6# SetColumnarAttr(int, ESphAttr, columnar::Builder_i*, std::unique_ptr<columnar::Iterator_i, std::default_delete<columnar::Iterator_i> >&, sph::Vector_T<long, sph::DefaultCopy_T<long>, sph::DefaultRelimit, sph::DefaultStorage_T<long> >&) in /usr/bin/searchd
 7# RtIndex_c::WriteAttributes(SaveDiskDataContext_t&, CSphString&) const in /usr/bin/searchd
 8# RtIndex_c::SaveDiskData(char const*, VecTraits_T<CSphRefcountedPtr<RtSegment_t const> > const&, ChunkStats_t const&, CSphString&) const in /usr/bin/searchd
 9# RtIndex_c::SaveDiskChunk(bool, bool, bool) in /usr/bin/searchd
10# Threads::CoRoutine_c::CreateContext(std::function<void ()>, VecTraits_T<unsigned char>)::{lambda(boost::context::detail::transfer_t)#1}::__invoke(boost::context::detail::transfer_t) in /usr/bin/searchd
11# make_fcontext in /usr/bin/searchd

[2peter3]

I've discovered something else: it seems that the error only occurs in connection with multiple values in REPLACE INTO t (...) VALUES (1), (2), (3...) when the data type is "string". If the values are also relatively long, it crashes the database. This happens regardless of whether it is done through MySQL or via a bulk JSON API. The only situation where multiple replacements work well is when the data type is "text". Thus, there seems to be an issue with "string" data types, "concurrency", and long values in columnar databases.

[sanikolaev]

MRE

mysql -P9315 -h0 -e "drop table if exists t; CREATE TABLE t(ip string attribute) engine='columnar'";

curl -s -H 'Content-type: application/x-ndjson' 0:9316/bulk --data-binary '{"replace": {"index": "t", "id": 1, "doc": {"ip":"xx.xxx.xxx.xx"}}}
{"replace": {"index": "t", "id": 2, "doc": {"ip":"xx.xxx.xx.xx"}}}
{"replace": {"index": "t", "id": 1, "doc": {"ip":"xx.xxx.xxx.xx"}}}'

mysql -P9315 -h0 -e "flush ramchunk t"

Backtrace:

 0# sphBacktrace(int, bool) in searchd
 1# CrashLogger::HandleCrash(int) in searchd
 2# 0x00007F036F1BE520 in /lib/x86_64-linux-gnu/libc.so.6
 3# LibcCIHash_fn::Hash(unsigned char const*, int, unsigned long) in searchd
 4# columnar::Packer_Hash_c::AddDoc(unsigned char const*, int) in /usr/share/manticore/modules/lib_manticore_columnar.so
 5# columnar::Builder_c::SetAttr(int, unsigned char const*, int) in /usr/share/manticore/modules/lib_manticore_columnar.so
 6# SetColumnarAttr(int, ESphAttr, columnar::Builder_i*, std::unique_ptr<columnar::Iterator_i, std::default_delete<columnar::Iterator_i> >&, unsigned int, sph::Vector_T<long, sph::DefaultCopy_T<long>, sph::DefaultRelimit, sph::DefaultStorage_T<long> >&) in searchd
 7# RtIndex_c::WriteAttributes(SaveDiskDataContext_t&, CSphString&) const in searchd
 8# RtIndex_c::SaveDiskData(char const*, VecTraits_T<CSphRefcountedPtr<RtSegment_t const> > const&, ChunkStats_t const&, CSphString&) const in searchd
 9# RtIndex_c::SaveDiskChunk(bool, bool, bool) in searchd
10# RtIndex_c::ForceDiskChunk() in searchd
11# HandleMysqlFlushRamchunk(RowBuffer_i&, SqlStmt_t const&) in searchd
12# ClientSession_c::Execute(std::pair<char const*, int>, RowBuffer_i&) in searchd
13# SqlServe(std::unique_ptr<AsyncNetBuffer_c, std::default_delete<AsyncNetBuffer_c> >) in searchd
14# MultiServe(std::unique_ptr<AsyncNetBuffer_c, std::default_delete<AsyncNetBuffer_c> >, std::pair<int, unsigned short>, Proto_e) in searchd
15# 0x000055B1DFE40BAF in searchd
16# Threads::CoRoutine_c::CreateContext(std::function<void ()>, std::pair<boost::context::stack_context, Threads::StackFlavour_E>)::{lambda(boost::context::detail::transfer_t)#1}::__invoke(boost::context::detail::transfer_t) in searchd
17# make_fcontext in searchd

Reproduced with:

Manticore 6.2.13 cc9294cd6@24051604 dev (columnar 2.2.5 0c18998@240424) (secondary 2.2.5 0c18998@240424) (knn 2.2.5 0c18998@240424)

Config:

searchd {
    listen = 9315:mysql
    listen = 9316
    buddy_path =
    log = searchd.log
    pid_file = searchd.pid
    data_dir = data
    binlog_path =
}

Notes:

• Note, there are duplicate ids in the payload. It doesn't crash w/o that.
• It doesn't crash w/o engine='columnar'
• It doesn't crash if the same is done via mysql:

   drop table if exists t; 
   CREATE TABLE t(ip string attribute) engine='columnar';
   insert into t values(1, 'xx.xxx.xxx.xx'),(2,'xx.xxx.xx.xx'),(1, 'xx.xxx.xxx.xx');
   flush ramchunk t;
  

[tomatolog]

I can not reproduce the crash using MRE from the comment with either

Manticore 6.2.13 cc9294cd6@24051604 dev (columnar 2.2.5 0c18998@240424) (secondary 2.2.5 0c18998@240424) (knn 2.2.5 0c18998@240424)

or master headr versions of daemon and MCL at windows box and linux at the dev2 box.

However I can reproduce the crash replaing binlog user provided and see that crash persists at the mater version of daemon and MCL. Will investigate the crash further

[sanikolaev]

I can not reproduce the crash using MRE from the #2176 (comment) with either

Strange. I can easily do it - https://youtu.be/oWmM1JeM9wY

[tomatolog]

the crash was fixed at f6c433a

You need to reindex the data from scratch with the daemon from the development version to get the crash fixed

[tomatolog]

some issue still left at #2209