aerospike/aerospike-client-c

Aerospike C Client

aerospike-client-c's Issues

segfault at as_node_get

In cluster of 5, one client ( out of 5 ), which was async writing 2 ops to binA and binB segfaulted. gdb backtrace:

(gdb) bt
#0  as_node_get (master=true, replica=<optimized out>, digest=0x7fa5d1bcd581 "\317\330~e\341\220\327\026V\024\002\310\253\017V9\362\352\323\276", ns=0x7fa5d1bcd4f1 "test", 
    cluster=0x0) at src/include/aerospike/as_cluster.h:427
#1  as_event_command_node_init (as=0x602940 <as>, node_out=<synthetic pointer>, replica=<optimized out>, key=0x7fa5d1bcd4f0, err=0x7fa5d1bcd5a0)
    at src/main/aerospike/aerospike_key.c:70
#2  aerospike_key_operate_async (as=as@entry=0x602940 <as>, err=err@entry=0x7fa5d1bcd5a0, policy=policy@entry=0x7fa5d1bcd430, key=key@entry=0x7fa5d1bcd4f0, 
    ops=ops@entry=0x7fa5d1bcd3f0, listener=listener@entry=0x401e67 <write_listener>, udata=udata@entry=0x7ffcfa4b5850, event_loop=0xda3010, 
    pipe_listener=pipe_listener@entry=0x0) at src/main/aerospike/aerospike_key.c:743
#3  0x0000000000401dca in write_record (event_loop=<optimized out>, counter=0x7ffcfa4b5850) at asyncwrite.c:299
#4  0x00007fa5d33db853 in as_event_error_callback (cmd=cmd@entry=0xdbe610, err=err@entry=0x7fa5d1bcda40) at src/main/aerospike/as_event.c:431
#5  0x00007fa5d33dbccb in as_event_timeout (cmd=0xdbe610) at src/main/aerospike/as_event.c:513
#6  0x00007fa5d31973a6 in ev_invoke_pending (loop=0x7fa5cc0008c0) at ev.c:3314
#7  0x00007fa5d319a9d4 in ev_run (loop=0x7fa5cc0008c0, flags=flags@entry=0) at ev.c:3717
#8  0x0000000000401ad2 in ev_loop (flags=0, loop=<optimized out>) at /usr/local/include/ev.h:835
#9  loop_thread (udata=0x7ffcfa4b5830) at asyncwrite.c:217
#10 0x00007fa5d28aa0a4 in start_thread (arg=0x7fa5d1bce700) at pthread_create.c:309
#11 0x00007fa5d1cb787d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) frame 0 
#0  as_node_get (master=true, replica=<optimized out>, digest=0x7fa5d1bcd581 "\317\330~e\341\220\327\026V\024\002\310\253\017V9\362\352\323\276", ns=0x7fa5d1bcd4f1 "test", 
    cluster=0x0) at src/include/aerospike/as_cluster.h:427
427		if (cluster->shm_info) {
(gdb) list
422	as_node_get(as_cluster* cluster, const char* ns, const uint8_t* digest, as_policy_replica replica, bool master)
423	{
424	#ifdef AS_TEST_PROXY
425		return as_node_get_random(cluster);
426	#else
427		if (cluster->shm_info) {
428			return as_shm_node_get(cluster, ns, digest, replica, master);
429		}
430		else {
431			as_partition_table* table = as_cluster_get_partition_table(cluster, ns);
(gdb)

Server: Aerospike Server Community edition 3.11.0.1
Client: C 4.1.3

Let me know if I need to provide more information.

Pkg-config configuration is not included with the tarballs

pc file is not present in both macos and linux environments.

as_record bins reallocation

Hi guys,

The documentation (http://www.aerospike.com/apidocs/c/df/dd6/structas__record.html) says that as_record_init caller could pass nbins == 0 if the number of bins to initialize is unknown. I guess it implies that the desired number of bins would be allocated later (like in std::vector). But in fact, there is no reallocation logic in as_record_bin_forupdate.

I'm wondering whether it's an issue or this behavior is considered to be correct.

Compile error on ubuntu14

cc -Isrc/include -Imodules/common/target/Linux-x86_64/include -Imodules/mod-lua/target/Linux-x86_64/include -Imodules/lua/src -std=gnu99 -g -Wall -fPIC -O3 -fno-common -fno-strict-aliasing -D_FILE_OFFSET_BITS=64 -D_REENTRANT -D_GNU_SOURCE -march=nocona -finline-functions -rdynamic -o target/Linux-x86_64/obj/aerospike/as_tls.o -c src/main/aerospike/as_tls.c
src/main/aerospike/as_tls.c: In function \u2018as_tls_thread_cleanup\u2019:
src/main/aerospike/as_tls.c:369:2: warning: \u2018ERR_remove_state\u2019 is deprecated (declared at /usr/local/include/openssl/err.h:248) [-Wdeprecated-declarations]
ERR_remove_state(0);
^
src/main/aerospike/as_tls.c: In function \u2018verify_callback\u2019:
src/main/aerospike/as_tls.c:417:25: error: dereferencing pointer to incomplete type
if (current_cert == ctx->cert) {
^
src/main/aerospike/as_tls.c:426:39: error: dereferencing pointer to incomplete type
bool matched = as_tls_match_name(ctx->cert, hostname, allow_wildcard);
^
src/main/aerospike/as_tls.c: In function \u2018as_tls_context_setup\u2019:
src/main/aerospike/as_tls.c:493:3: warning: implicit declaration of function \u2018SSLv3_client_method\u2019 [-Wimplicit-function-declaration]
method = SSLv3_client_method();
^
src/main/aerospike/as_tls.c:493:10: warning: assignment makes pointer from integer without a cast [enabled by default]
method = SSLv3_client_method();
^
src/main/aerospike/as_tls.c:496:3: warning: \u2018TLSv1_client_method\u2019 is deprecated (declared at /usr/local/include/openssl/ssl.h:1591) [-Wdeprecated-declarations]
method = TLSv1_client_method();
^
src/main/aerospike/as_tls.c:499:3: warning: \u2018TLSv1_1_client_method\u2019 is deprecated (declared at /usr/local/include/openssl/ssl.h:1597) [-Wdeprecated-declarations]
method = TLSv1_1_client_method();
^
src/main/aerospike/as_tls.c:502:3: warning: \u2018TLSv1_2_client_method\u2019 is deprecated (declared at /usr/local/include/openssl/ssl.h:1603) [-Wdeprecated-declarations]
method = TLSv1_2_client_method();
^
src/main/aerospike/as_tls.c: At top level:
src/main/aerospike/as_tls.c:255:1: warning: \u2018pthreads_locking_callback\u2019 defined but not used [-Wunused-function]
pthreads_locking_callback(int mode, int type, const char *file, int line)
^
src/main/aerospike/as_tls.c:265:1: warning: \u2018pthreads_thread_id\u2019 defined but not used [-Wunused-function]
pthreads_thread_id(CRYPTO_THREADID *tid)
^
make: *** [target/Linux-x86_64/obj/aerospike/as_tls.o] Error

crash

Both linux and MacOSX, at src/main/citrusleaf/citrusleaf.c:1382.
as->cluster is not checked before referencing

(gdb) bt
#0 do_the_full_monte (asc=0x0, info1=info1@entry=3, info2=info2@entry=0, info3=info3@entry=0, ns=ns@entry=0x7ffff5c01cc1 "test",

set=set@entry=0x0, key=key@entry=0x0, digest=0x7ffff5c01d51, values=values@entry=0x7ffff51b79e0, 

operator=operator@entry=CL_OP_READ, operations=operations@entry=0x0, n_values=n_values@entry=0x7ffff51b79c0, 

cl_gen=cl_gen@entry=0x7ffff51b79a0, cl_w_p=cl_w_p@entry=0x7ffff51b7950, trid=trid@entry=0x7ffff51b7940, 

setname_r=setname_r@entry=0x0, call=call@entry=0x0, cl_ttl=cl_ttl@entry=0x7ffff51b79b0) at src/main/citrusleaf/citrusleaf.c:1382

#1 0x00007ffff59ce8ff in citrusleaf_get_all_digest_getsetname (asc=, ns=ns@entry=0x7ffff5c01cc1 "test",

digest=<optimized out>, values=values@entry=0x7ffff51b79e0, n_values=n_values@entry=0x7ffff51b79c0, 

timeout_ms=timeout_ms@entry=1000, cl_gen=cl_gen@entry=0x7ffff51b79a0, setname=setname@entry=0x0, 

cl_ttl=cl_ttl@entry=0x7ffff51b79b0) at src/main/citrusleaf/citrusleaf.c:1772

#2 0x00007ffff59cf6bb in aerospike_key_get (as=0x7ffff5c00920, err=0x7ffff51b7a50, policy=, key=0x7ffff5c01cc0,

rec=0x7ffff51b7a48) at src/main/aerospike/aerospike_key.c:89

cf_atomic.h is not compatible with POWERPC

I am working on porting aerospike to POWERPC and fc_atomic.h has embedded x86 assembly. I changed 2 header files, 3 Makefiles, and updated the ck subdirectory to get the workload to compile for POWER. I would like to check in this code so I can run it with PerfKitBenchmarker.
I followed the process outlined by PerfKitBenchmarker/CONTRIBUTING.md to push code but I got a permission denied error. How should I go about submitting this code? Thanks!

mlpesant@tul65p1:~/aerospike-client-c$ git push origin ppc64
Username for 'https://github.com': mlpesant
Password for 'https://[email protected]': 
remote: Permission to aerospike/aerospike-client-c.git denied to mlpesant.
fatal: unable to access 'https://github.com/aerospike/aerospike-client-c.git/': The requested URL returned error: 403

Scan policy issue (fail_on_cluster_change)

Hi all!

It seems that I found a bug in SDK.
I have several process writting data to the cluster permanently.
I try to scan cluster in the following way:

.....
bool scan_cb(const as_val* p_val, void* udata)
{
   if (! p_val) {
      LOG("scan callback returned null - scan is complete");
      return true;
   }
   as_record* p_rec = as_record_fromval(p_val);
   if (! p_rec) {
      LOG("scan callback returned non-as_record object");
      return true;
   }
   LOG("Got record!");
   return true;
}


void walk_interests () {
   as_scan scan;
   as_error err;
   as_scan_init(&scan, "database", "userdata");

//initialise scan policy:
   as_policy_scan policy;
   as_policy_scan_init(&policy);
   policy.fail_on_cluster_change=false;

   LOG("starting scan all ...");

   if (aerospike_scan_foreach(&as, &err, &policy, &scan, scan_cb, NULL) !=
         AEROSPIKE_OK) {
      LOG("aerospike_scan_foreach() returned %d - %s", err.code, err.message);
      as_scan_destroy(&scan);
      exit(-1);
   }
   LOG("scan finished");
}

The program generates the following output, regardless of the value of the fail_on_cluster_change field:

starting scan all ...
aerospike_scan_foreach() returned 504 - AEROSPIKE_ERR_CLUSTER_CHANGE

Is that a bug or there is an error in this listing?

Cannot compile this for HP UX

bash-4.3# make
Make: Must be a separator on rules line 5. Stop.

Error comes from project/settings.mk file

BUILD VARIABLES

export CFLAGS=-Wall ------------------> Line 5
export LDFLAGS=" "
export ARFLAGS=" "

Has anyone tried to compile the client or server on HPUX or AIX platforms ? Am I missing something here ?

Compile library for iOS SDK

Hello.
I compile your 'aerospace-client-c' (library 'libaerospike.a') for iOS SDK, but I have an error

What is wrong?

Tarballs on github are incomplete

Thanks to usage of git submodules ...

Build fails with
find: ‘/var/tmp/portage/dev-libs/libaerospike-4.0.4/work/aerospike-client-c-4.0.4/modules/common/src/include/aerospike/ck’: No such file or directory
since obviously that directory in the tarballs is empty.

Would be very convenient to have proper release tarballs available ...

Parsing record with ldt

Hello,
I have found that getting record for key,to which was previously added a large list is returning

1. LDTCONTROLBIN
2. llist_bin

with AS_NIL value types
related topic on aerospike forum: https://discuss.aerospike.com/t/parsing-record-with-ldt/2264/1

Usage in C++ code requires wrapping headers in extern "C"

We include the Aerospike client libraries in a C++ project. In order to avoid linker errors, every file that includes an #include directive for an Aerospike header needs to wrap those in an extern "C" block like this:

// Aerospike -- WARNING - extern "C" required!
extern "C" {
#include <aerospike/aerospike.h>
#include <aerospike/aerospike_key.h>
#include <aerospike/aerospike_query.h>
#include <aerospike/aerospike_lmap.h>
#include <aerospike/aerospike_index.h>
}

Could the Aerospike client library header files be made C++-safe by including this in the headers themselves?

Error when trying to dlopen libaerospike.so

I am trying to build a python client and use cffi to call libaerospike.so from python. However, I am getting this error when I try to load the lib:

File "/home/mvalle/pyve01/local/lib/python2.7/site-packages/cffi/api.py", line 117, in dlopen
lib, function_cache = _make_ffi_library(self, name, flags)
File "/home/mvalle/pyve01/local/lib/python2.7/site-packages/cffi/api.py", line 399, in _make_ffi_library
backendlib = backend.load_library(name, flags)
OSError: cannot load library libaerospike.so: /home/mvalle/workspaces/s1mbi0se/aerospike-client-c/target/Linux-x86_64/lib/libaerospike.so: undefined symbol: lua_gettop

I tried the debian 7 package and also tried to build from source. The error keeps raising.

Libevent support

Hello, guys!

Do you plan to support libevent-based async engine? It would be very nice, because libevent is as popular as libev or libuv.
Now I use that one http://www.aerospike.com/docs/client/libevent/ , but it's closed-source and it looks a little bit outdated/frozen.

Regards

Can't compile with make -j >1

I haven't been able to isolate the culprit but it's fairly easy to reproduce, just try to build using make -j4 and see.

Trouble building on Mac with Xcode

The README.md file could make life much easier for Mac users by providing a separate Mac section, which contains a ready-to-use shell script, as shown at the end of this report.

Using macOS 10.12.3 with Xcode 8.2.1, I tried to follow the instructions in the README.md file, as follows.

Start with this:

brew install openssl

Homebrew complains and suggests doing this, which doesn't seem to help (more later):

brew install --force openssl

Do this:

git clone --depth=1 https://github.com/aerospike/aerospike-client-c.git
cd aerospike-client-c
git submodule update --init
open project/client.xcworkspace

In Xcode, try to build aerospike-test (which is not the default when Xcode opens). Xcode complains 'lua.h' file not found

brew install lua

In Xcode, try to build aerospike-test. Xcode complains 'openssl/ssl.h' file not found

Do this by hand to get around brew's refusal to do so (above, despite the --force argument):

# All this is required for Xcode but not for make.
pushd /usr/local/include
ln -sf ../opt/openssl/include/openssl .
cd /usr/local/lib
ln -sf ../opt/openssl/lib/libcrypto.dylib \
       ../opt/openssl/lib/libcrypto.1.0.0.dylib .
ln -sf ../opt/openssl/lib/libssl.dylib \
       ../opt/openssl/lib/libssl.1.0.0.dylib    .     
popd

In Xcode, try to build aerospike-test. Xcode complains about lua symbols. Brian says that this is because the lua installed by Homebrew is too recent. So do this:

brew uninstall lua

In Xcode, try to build aerospike-test. Xcode complains 'lua.h' file not found again.

Do this workaround, so that Xcode can see the lua that comes with the client from the repo. The README.md doesn't say that even if you're going to be building with Xcode, you nevertheless have to run make. This was not at all obvious.

# First we have to run make to get the lua includes and libs ready for use.
make
# so that they can be put in /usr/local/

Incidentally, I wasted some time because I didn't expect make to build lua in its source tree:

lrwxrwx---  1 yost  wheel      14 2017-02-05.22:16:50 modules/lua/src/luaconf.h -> luaconf.h.orig
-rw-rw----  1 yost  wheel    9048 2017-02-05.22:16:52 modules/lua/src/lapi.o
-rw-rw----  1 yost  wheel    9120 2017-02-05.22:16:52 modules/lua/src/lauxlib.o
-rw-rw----  1 yost  wheel   16196 2017-02-05.22:16:52 modules/lua/src/lbaselib.o
-rw-rw----  1 yost  wheel   25160 2017-02-05.22:16:52 modules/lua/src/lcode.o
-rw-rw----  1 yost  wheel    1528 2017-02-05.22:16:53 modules/lua/src/ldblib.o
-rw-rw----  1 yost  wheel    4848 2017-02-05.22:16:53 modules/lua/src/ldebug.o
-rw-rw----  1 yost  wheel    1432 2017-02-05.22:16:53 modules/lua/src/ldo.o
-rw-rw----  1 yost  wheel   11816 2017-02-05.22:16:53 modules/lua/src/ldump.o
-rw-rw----  1 yost  wheel    8400 2017-02-05.22:16:53 modules/lua/src/lfunc.o
-rw-rw----  1 yost  wheel    3160 2017-02-05.22:16:53 modules/lua/src/lgc.o
-rw-rw----  1 yost  wheel    2596 2017-02-05.22:16:53 modules/lua/src/linit.o
-rw-rw----  1 yost  wheel    7164 2017-02-05.22:16:54 modules/lua/src/liolib.o
-rw-rw----  1 yost  wheel    2128 2017-02-05.22:16:54 modules/lua/src/llex.o
-rw-rw----  1 yost  wheel    8292 2017-02-05.22:16:54 modules/lua/src/lmathlib.o
-rw-rw----  1 yost  wheel    1776 2017-02-05.22:16:54 modules/lua/src/lmem.o
-rw-rw----  1 yost  wheel    3640 2017-02-05.22:16:54 modules/lua/src/loadlib.o
-rw-rw----  1 yost  wheel   22828 2017-02-05.22:16:54 modules/lua/src/lobject.o
-rw-rw----  1 yost  wheel    1656 2017-02-05.22:16:55 modules/lua/src/lopcodes.o
-rw-rw----  1 yost  wheel   13608 2017-02-05.22:16:55 modules/lua/src/loslib.o
-rw-rw----  1 yost  wheel    8008 2017-02-05.22:16:55 modules/lua/src/lparser.o
-rw-rw----  1 yost  wheel    8820 2017-02-05.22:16:55 modules/lua/src/lstate.o
-rw-rw----  1 yost  wheel   15496 2017-02-05.22:16:55 modules/lua/src/lstring.o
-rw-rw----  1 yost  wheel   11688 2017-02-05.22:16:55 modules/lua/src/lstrlib.o
-rw-rw----  1 yost  wheel   18328 2017-02-05.22:16:55 modules/lua/src/ltable.o
-rw-rw----  1 yost  wheel   14980 2017-02-05.22:16:55 modules/lua/src/ltablib.o
-rw-rw----  1 yost  wheel    5432 2017-02-05.22:16:56 modules/lua/src/ltm.o
-rw-rw----  1 yost  wheel    9572 2017-02-05.22:16:56 modules/lua/src/lua.o
-rw-rw----  1 yost  wheel    7536 2017-02-05.22:16:56 modules/lua/src/luac.o
-rw-rw----  1 yost  wheel   18460 2017-02-05.22:16:56 modules/lua/src/lundump.o
-rw-rw----  1 yost  wheel   10728 2017-02-05.22:16:56 modules/lua/src/lvm.o
-rw-rw----  1 yost  wheel    1768 2017-02-05.22:16:56 modules/lua/src/lzio.o
-rw-rw----  1 yost  wheel    4996 2017-02-05.22:16:57 modules/lua/src/print.o
-rwxrwx---  1 yost  wheel  180796 2017-02-05.22:16:56 modules/lua/src/lua
-rw-rw----  1 yost  wheel  278232 2017-02-05.22:16:56 modules/lua/src/liblua.a
-rwxrwx---  1 yost  wheel  127332 2017-02-05.22:16:57 modules/lua/src/luac

Now that make has built the right version of lua, do this:

cp -p modules/lua/src/liblua.a          /usr/local/lib      
cp -p modules/lua/src/lua.h \
      modules/lua/src/luaconf.h \
      modules/lua/src/luaconf.h.orig \
      modules/lua/src/lauxlib.h \
      modules/lua/src/lualib.h          /usr/local/include 

In Xcode, try to build aerospike-test. Xcode complains ld: library not found for -luv.

Do this (even though the README.md file implies that it's optional):

brew install libuv
brew install libev

In Xcode, try to build aerospike-test. Build Succeeded! (but with many warnings)

Xcode has many hard-coded paths to libs (mostly in red):

Only the liblua.a and libev.a references work. The others don't work for path and version reasons. In any case, if these references are going to be here and made to work, they should probably be shared, either in the aerospike project or in a new aerospike-libs project.

Here are all of my workarounds, collected in one script:

if [[ $cleanfirst ]] ; then
  echo cleaning...
  brew uninstall libuv
  brew uninstall libev
  brew uninstall openssl
  rm -f /usr/local/include/openssl
  rm -f /usr/local/lib/lib{ssl,crypto}*.dylib  
  rm -f /usr/local/include/lua.h \
        /usr/local/include/luaconf.h \
        /usr/local/include/lualib.h \
        /usr/local/include/lauxlib.h \
        /usr/local/lib/liblua.a
  rm -rf aerospike-client-c
fi
  
# Both are required for Xcode; neither is required for make.
brew install libuv
brew install libev

# I had to do --force. Maybe others won't.
brew install --force openssl
(
    # All this is required for Xcode but not for make.
    cd /usr/local/include
    ln -sf ../opt/openssl/include/openssl .
    cd /usr/local/lib
    ln -sf ../opt/openssl/lib/libcrypto.dylib \
           ../opt/openssl/lib/libcrypto.1.0.0.dylib .
    ln -sf ../opt/openssl/lib/libssl.dylib \
           ../opt/openssl/lib/libssl.1.0.0.dylib    .     
)

git clone --depth=1 [email protected]:aerospike/aerospike-client-c.git
cd aerospike-client-c
git submodule update --init

# make -j doesn't work.
make

# 
cp -p modules/lua/src/liblua.a          /usr/local/lib      
cp -p modules/lua/src/lua.h \
      modules/lua/src/luaconf.h \
      modules/lua/src/luaconf.h.orig \
      modules/lua/src/lauxlib.h \
      modules/lua/src/lualib.h          /usr/local/include 

open project/client.xcworkspace

Compilation on ppc64le

Hi,

Any plan to allow compilation of aerospike client, without lua, on ppc64le ?

Regards,

Bertrand

New feature request: Alpine

Hello,

Any plans about Alpine support?

Connect to bad cluster after historic fall when using shm

I have run into issue, when client connect to another cluster because of fall (sigsegv) in past:

I'm connecting to a cluster with shm enabled:

cfg.use_shm = true;
cfg.shm_takeover_threshold_sec=1;

Let's connect to first cluster and fall due to sigsegv:

$> ./sigsegv 10.0.23.9
zsh: segmentation fault (core dumped)  ./sigsegv 10.0.23.9

shared memory is allocated:

$> ipcs -m | grep 0xa5
0xa5000000 51642414   jarda      666        263224

Now let's connect to another cluster with repaired binary (same config, but another cluster):

$> ./info 10.0.17.89

aerospike_info_any response ok: service 10.0.23.9:3000
aerospike_info_any response ok: service 10.0.23.9:3000
aerospike_info_any response ok: service 10.0.23.9:3000
aerospike_info_any response ok: service 10.0.23.9:3000
aerospike_info_any response ok: service 10.0.23.9:3000

The cluster, that is answering is the old one (10.0.23.9), but I wanted to seed cluster from 10.0.17.89.
I understand first line - there is a shared memory, so client doesn't need to ask cluster. But in config I set cfg.shm_takeover_threshold_sec=1, so after 1 s, there should be new cluster reached.

Source codes of binaries: https://gist.github.com/jarda-manana/053be1411c24ae4cbdcf, C Client Library 3.1.22

Ability to control (de)serialization on all requests

This is specially useful/efficient when using the C client as a basis to other language clients, as in the Python client. By being able to bypass the deserialization on the requests the python library can deserialize the msgpack directly into python objects, pretty much halving memory allocations and iterations. The same applies for serialization.

Right now you can only control complex type deserialization on scan requests via as_scan.deserialize_list_map = false. It'd be great to support the same for get and query, the infrastructure it's already in place for that. We just need a way to pass this down to the last param of as_command_parse_bins.

As for serialization on put, there's no real infrastructure in place for it, but we can easily add two new types, as_rawlist and as_rawmap that just point their buffer to the already serialized msgpack data. This can even go into the external client.

Am I missing something? Is this something you guys want?

aerospike_key_operate will leak memory.

leaking memory at aerospike_key_operate if pass the as_map/as_arraylist value to as_operations_add_write function.

==46149== 8,192 bytes in 1 blocks are definitely lost in loss record 116 of 116
==46149==    at 0x6DFB: malloc (in /usr/local/Cellar/valgrind/3.9.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==46149==    by 0x100049744: cf_malloc (in ./tools/memcheck)
==46149==    by 0x10003E498: as_msgpack_serializer_serialize (in ./tools/memcheck)
==46149==    by 0x100023103: as_serializer_serialize (in ./tools/memcheck)
==46149==    by 0x100022E60: asval_to_clobject (in ./tools/memcheck)
==46149==    by 0x100023157: asbinvalue_to_clobject (in ./tools/memcheck)
==46149==    by 0x100026C0C: aerospike_key_operate (in ./tools/memcheck)
==46149==    by 0x100002A72: operate_lua (in ./tools/memcheck)
==46149==    by 0x11732: luaD_precall (in /usr/local/Cellar/lua/5.1.5/lib/liblua.5.1.5.dylib)
==46149==    by 0x1A676: luaV_execute (in /usr/local/Cellar/lua/5.1.5/lib/liblua.5.1.5.dylib)
==46149==    by 0x11A99: luaD_call (in /usr/local/Cellar/lua/5.1.5/lib/liblua.5.1.5.dylib)
==46149==    by 0x11160: luaD_rawrunprotected (in /usr/local/Cellar/lua/5.1.5/lib/liblua.5.1.5.dylib)
==46149== 
==46149== LEAK SUMMARY:
==46149==    definitely lost: 8,240 bytes in 4 blocks
==46149==    indirectly lost: 4,857 bytes in 18 blocks
==46149==      possibly lost: 0 bytes in 0 blocks
==46149==    still reachable: 9,840 bytes in 19 blocks
==46149==         suppressed: 25,403 bytes in 381 blocks
==46149== Reachable blocks (those to which a pointer was found) are not shown.
==46149== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==46149== 

Fetching keys with scan?

Using the C API, the key member of as_record does not seem to be populated in the scan callback. Is there a way to project the key value during a scan?

Should 4.0.7 build on CentOS 5?

Yes, I understand that CentOS 5 is ancient, but I have lots of old servers... Should I be able to build 4.0.7 on CentOS 5? I'm getting the following errors at the end of make package:

Requires(interp): /bin/sh /bin/sh
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
Requires(pre): /bin/sh
Requires(post): /bin/sh
Requires: ld-linux-x86-64.so.2()(64bit) ld-linux-x86-64.so.2(GLIBC_2.3)(64bit) libc.so.6()(64bit) libc.so.6(GLIBC_2.2.5)(64bit) libc.so.6(GLIBC_2.3)(64bit) libc.so.6(GLIBC_2.3.2)(64bit) rtld(GNU_HASH)
Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/user/aerospike-client-c/stage/BUILD
D:  execv(/usr/lib/rpm/check-files) pid 29755
D:  waitpid(29755) rc 29755 status 0
error: cannot create /home/user/aerospike-client-c/stage/RPMS/x86_64: No such file or directory
D: fini      040775  4 (   0,   0)         0 /opt/aerospike/client
D: fini      040775  3 (   0,   0)         0 /opt/aerospike/client/sys
D: fini      040775  3 (   0,   0)         0 /opt/aerospike/client/sys/udf
D: fini      040775  2 (   0,   0)         0 /opt/aerospike/client/sys/udf/lua
D: fini      100664  1 (   0,   0)      5623 /opt/aerospike/client/sys/udf/lua/aerospike.lua
D: fini      100664  1 (   0,   0)      3211 /opt/aerospike/client/sys/udf/lua/as.lua
D: fini      100664  1 (   0,   0)      8862 /opt/aerospike/client/sys/udf/lua/stream_ops.lua
D: fini      040775  3 (   0,   0)         0 /opt/aerospike/client/usr
D: fini      040775  3 (   0,   0)         0 /opt/aerospike/client/usr/udf
D: fini      040775  2 (   0,   0)         0 /opt/aerospike/client/usr/udf/lua
D: fini      100664  1 (   0,   0)   4288746 /usr/lib/libaerospike.a
D: fini      100775  1 (   0,   0)   2250897 /usr/lib/libaerospike.so
GZDIO:     205 writes,  6559216 total bytes in 7.395009 secs
D: May free Score board((nil))
error: Could not open /home/user/aerospike-client-c/stage/RPMS/x86_64/aerospike-client-c-4.0.7-1.el5.x86_64.rpm: No such file or directory


RPM build errors:
    cannot create /home/user/aerospike-client-c/stage/RPMS/x86_64: No such file or directory
    Could not open /home/user/aerospike-client-c/stage/RPMS/x86_64/aerospike-client-c-4.0.7-1.el5.x86_64.rpm: No such file or directory
D: May free Score board((nil))
make: *** [package] Error 1
[[email protected] aerospike-client-c]$

Full log attached:
centos5_build.log.txt

server connection error

I had a quick question on handling server connection errors. I have a persistent connection to the aerospike server. I keep doing read/writes.
Is there a specific error code which gets set when the server becomes unavailable ? We do scaling and shrinking of aerospike servers and when this error happens I can disconnect and reconnect back to the active list at that point in time.
I was thinking I can look for - AEROSPIKE_ERR_CONNECTION code which gets set in as_error.code ?
But I see that when the initial connection fails due to down server, the code is set to - AEROSPIKE_ERR_CLIENT ?

Could you please clarify. Thank you.

Bug in seting timeout in aerospike_key_operate

Hi, there seems to be a bug in setting operation timeout.

// ...
as_error error;
as_policy_operate policy;
as_policy_operate_init(&policy);
policy.timeout = 900000; // timeout in ms
policy.retry = AS_POLICY_RETRY_ONCE;
if (aerospike_key_operate(&as, &error, &policy, &key, &ops, nullptr) != AEROSPIKE_OK)
{
    std::lock_guard<std::mutex> _{mutex};
    std::cerr << "Oerr(" << error.code << ") " << error.message << " at [" << error.file << ":" << error.line << "]" << std::endl;
}
// ...

results in error AFTER about 1 SECOND:
Oerr(9) Client timeout: timeout=900000 iterations=3 failedNodes=0 failedConns=3 at [src/main/aerospike/as_command.c:446]

This error happens mainly when we use a lot of threads (~1000) to achieve parallel writes. Obviously the network is saturated so we would like to set a bigger timeout.

We are using Aerospike C client 3.1.18 on debian wheezy.

Auto include git submodules in tar.gz / zip releases (To ease building of packages)

Hi -

We're required to run 'git submodule update --init' to update this libraries dependencies before we build. This normally wouldn't be an issue as I can do that in my RPM spec file in the prep section. However this cannot be done when I download the source tar ball. I get this error:

Hence:
$ git submodule update --init
fatal: Not a git repository (or any of the parent directories): .git

So that means I shouldn't bother downloading the source releases, as they don't contain all the needed bits to build a package.

I would recommend that you include these submodules in your source releases for the aerospike-client-c project. This way when I build my RPM I can identify the RPM version to an exact source release version. Where-as now I wouldn't exactly know what source version of common, lua-core, and mod-lua were when I build this package.

Close this ticket if you like as this really isn't an issue, but I thought the owners of this project should know they're making it difficult to build sane package releases.

SIGSEGV in as_batch_async_parse_records

We are able to consistently produce this crash under load in production but haven't been able to reproduce it locally.

Seems to be occurring in memcmp(digest, record->key.digest.value, AS_DIGEST_VALUE_SIZE) in aerospike_batch.c:164

Here is the stack trace of the SIGSEGV:

* [  7] libc-2.19.so              0x7f3910b54f96
    -----------------------------------------------------------------------
    Signal: [11] SIGSEGV (segmentation violation)
    Reason: SEGV_MAPERR (address not mapped to object)
    Signal address: 0x7f3970e4ed41
    -----------------------------------------------------------------------

  [  8] bee-stinger               as_batch_async_parse_records (src/main/aerospike/aerospike_batch.c:164)
  [  9] bee-stinger               as_uv_command_read.part.6 (src/main/aerospike/as_event_uv.c:287)
  [ 10] bee-stinger               uv__read (src/unix/stream.c:1178)
  [ 11] bee-stinger               uv__stream_io (src/unix/stream.c:1241)
  [ 12] bee-stinger               uv__io_poll (src/unix/linux-core.c:345)
  [ 13] bee-stinger               uv_run (src/unix/core.c:351)
  [ 14] bee-stinger               as_uv_worker (src/main/aerospike/as_event_uv.c:130)
  [ 15] libpthread-2.19.so        start_thread

Something to check: leaks, etc

Hi,

Here is a partial result of cppcheck run on this project:

[benchmarks/src/main/latency.c:142]: (error) Memory leak: array
[demos/ad_demo_cpp/src/main/ad_udf.cpp:209]: (error) Resource leak: fptr
[examples/utils/src/main/example_utils.c:579]: (error) Resource leak: file
[modules/common/src/test/types/types_bytes.c:43]: (error) Buffer is accessed out of bounds.
[modules/luajit/src/host/minilua.c:2936]: (error) No pair for character ("). Can't process file. File is either invalid or unicode, which is currently not supported.
[modules/luajit/src/lj_trace.c:774]: (error) Signed integer overflow for expression '(24004567864)+1'.
[modules/luajit/src/lj_trace.c:774]: (error) Signed integer overflow for expression '(19665386104)+1'.
[modules/mod-lua/src/main/mod_lua.c:243]: (error) Buffer is accessed out of bounds.
[modules/mod-lua/src/main/mod_lua.c:254]: (error) Buffer is accessed out of bounds.
[modules/mod-lua/src/main/mod_lua.c:299]: (error) Buffer is accessed out of bounds.
[modules/mod-lua/src/test/validation/validation_basics.c:70]: (error) Resource leak: file
[src/main/aerospike/as_query.c:296]: (error) va_list 'ap' was opened but not closed by va_end().
[src/test/util/udf.c:51]: (error) Resource leak: file
[src/test/util/udf.c:83]: (error) Resource leak: file

I'm experiencing memory leak with operate calls in python client: aerospike/aerospike-client-python#108
It does not seem that the reported leaks by the cppcheck are related to that leak, but maybe it can help for something else.
Thanks

Bus error: 10 using as_query.

I have a query that attempts to grab 1000 records at a time:

    as_query_init(&query_ctx->query, ns, set);
    as_query_where_init(&query_ctx->query, 1);
    as_query_where(&query_ctx->query, key, integer_range(start, end));
    if (aerospike_query_foreach(&query_ctx->as, &err, NULL, &query, callback, NULL) != AEROSPIKE_OK) {
        fprintf(stderr, "error(%d) %s at [%s:%d]\n", err.code, err.message, err.file, err.line);
        cleanup(query_ctx);
        return 1;
    }
    cleanup(query_ctx);
    return 0;

If I run it with a limit of 300 or below it runs, over that it starts failing regardless of what my callback is. The start/end are also long long int (in my test start = 1113032000, end = 1113033000).

Running through valgrind I get:
==49900== Process terminating with default action of signal 10 (SIGBUS)
==49900== Non-existent physical address at address 0xB0106000
==49900== at 0x2A967: as_integer_init (in /usr/local/lib/libaerospike.dylib)
==49900== by 0x18B73: as_bin_init_int64 (in /usr/local/lib/libaerospike.dylib)
==49900== by 0x27A6B: as_record_set_int64 (in /usr/local/lib/libaerospike.dylib)
==49900== by 0x1A068: clbin_to_asrecord (in /usr/local/lib/libaerospike.dylib)
==49900== by 0x1A17A: clbins_to_asrecord (in /usr/local/lib/libaerospike.dylib)
==49900== by 0x14AA0: cl_query_worker_do (in /usr/local/lib/libaerospike.dylib)
==49900== by 0x143D6: cl_query_worker (in /usr/local/lib/libaerospike.dylib)
==49900== by 0x3C3898: _pthread_body (in /usr/lib/system/libsystem_pthread.dylib)
==49900== by 0x3C3729: _pthread_start (in /usr/lib/system/libsystem_pthread.dylib)
==49900== by 0x3C7FC8: thread_start (in /usr/lib/system/libsystem_pthread.dylib)

Lastly, each record is structured like this:

ip: (int64)
counter1: (int)
counter2: (int)
counterN: (int)
and so forth. counter1..N is an undetermined length and only the ip is really int64 since the max theoretical ip4 is 4294967295.

In the mean time, I can possibly get away with 300 200 100 records at a time but it greatly reduces the speed of my processes (the C program just ideals around and consumes stdin returning the data as needed back to a bode.js module).

Cannot compile release aerospike-client-c-3.1.25 coming from tar.gz archive

Hi,

I downloaded the tar.gz archive for the release 3.1.25 with wget but because it does not contain a .git folder I cannot execute git submodule update --init and thus compile the project. Is it possible to add this folder in the release?

I did it by hand and the compilation worked.

Also, make package does not work on Ubuntu 14.04: "Invalid platform ubuntu14.04". Maybe it is expected.

Thanks,
Victor

API cause core dumps by referencing data that can be released by user

The following code make core dump:

char *addr = *(String::Utf8Value(addrVal->ToString()));
int port = portVal->Int32Value();
as_config_add_host(&asConfig, addr, port);

The code is plain and simple. But we have a problem. char *addr is on stack and released on function exit. Why aerospike C client cannot make a deep copy of the hostname if it needs it for long time? The following code solve the problem, but is ugly:

char *addr = *(String::Utf8Value(addrVal->ToString()));
int port = portVal->Int32Value();
as_config_add_host(&asConfig, strdup(addr), port);

The macro as_bins_inita might allocate not enough memory

There is a dangerous calculation of the memory size in the macro 'as_bins_inita':

#define as_bins_inita(__bins, __capacity) \\
        (__bins)->_free = false;\\
        (__bins)->capacity = __capacity;\\
        (__bins)->size = 0;\\
        (__bins)->entries = (as_bin *) alloca(sizeof(as_bin) * __capacity);

__capacity might be an expression (e.g. 5 + vec.size()). In this case, the size of allocated memory will be calculated incorrect:

 (__bins)->entries = (as_bin *) alloca(sizeof(as_bin) * 5 + vec.size()); // The order will be: ( (sizeof(as_bin) * 5) + vec.size() )

The correct version might look like this:

#define as_bins_inita(__bins, __capacity) \
        (__bins)->_free = false;\
        (__bins)->capacity = __capacity;\
        (__bins)->size = 0;\
        (__bins)->entries = (as_bin *) alloca(sizeof(as_bin) * ((__bins)->capacity));

Warnings on compiling

Hello guys. First of all thanks for your work on client-c. I have an issues when compiling query_examples:

warning: the comparison will always evaluate as ‘true’ for the address of ‘query’ will never be NULL [-Waddress]

This could prevent compiling when all warnings treated as errors.

aerospike-client-c does not resize thread pool size if needed

First, here is some background. We use the Client library as part of our client database backend of our application. This is the first context of utilisation of the Client library. The default value of the thread pool size is set to 16. Each process of our application had 30+ running threads linked to the Aerospike backend default configuration. Our application spawns client processes and hit the maximum thread limit because of the overall number of threads present on the machine.

We needed to reduce the number of threads in this first context. We then reduced the thread pool size to one thread while initialising Aerospike in our application, static initialisation.

The Client library is used in a second context where we observed hanging behaviour while examining the poolstats because of the load inherent to a single work thread executing the queries. In this second context, we would expect more threads to be created to cope with the load. Furthermore, we noticed that, in the Client library, the thread pool size is not resized with the function as_thread_pool_resize from the library aerospike-common.

Why is the thread pool not resize upon load balancing needs? We would like the Client library to resize the thread pool size if needed.

Details: We run our application on CentOS 7.2 with the following packages:

Name        : aerospike-client-c
Version     : 4.0.6

Name        : aerospike-client-c-devel
Version     : 4.0.6

Name        : aerospike-server-community
Version     : 3.8.4

Name        : aerospike-tools
Version     : 3.8.3

aerospike_key_put_async segfault at high load.

Hello all! segfault after AEROSPIKE_ERR_TIMEOUT error in function aerospike_key_put_async. What can be wrong?

#0 as_event_command_node_init (err=err@entry=0x7fffe77fd550,
key=key@entry=0x7fffe77fd4a0,
replica=replica@entry=AS_POLICY_REPLICA_MASTER,
node_out=node_out@entry=0x7fffe77fd3e8, as=)
at src/main/aerospike/aerospike_key.c:70
#1 0x00000000004147ea in aerospike_key_put_async_ex (as=0x683fc0 <g_as>,
err=0x7fffe77fd550, policy=0x6840c4 <g_as+260>, key=0x7fffe77fd4a0,
rec=0x699f20,
listener=0x409570 <Write_Listener(as_error_s*, void*, as_event_loop*)>,
udata=udata@entry=0x69a418, event_loop=event_loop@entry=0x69a840,
pipe_listener=pipe_listener@entry=0x0, length=length@entry=0x0,
comp_length=comp_length@entry=0x0)
at src/main/aerospike/aerospike_key.c:424
#2 0x0000000000414e1f in aerospike_key_put_async (as=,
err=, policy=, key=,
rec=, listener=, udata=0x69a418,
event_loop=0x69a840, pipe_listener=0x0)
at src/main/aerospike/aerospike_key.c:522

You can also repeat the error for the utility benchmark (args.async_max_commands > 500).

Random malloc: *** error for object 0x7fca304091c0: incorrect checksum for freed object - object was probably modified after being freed.

I have been getting some weird resource release errors on the Aerospike C Client on MacOSX Yosemite running a Vagrant build of Aerospike 3.5.14.

I am running the following codes to insert into an lmap in
Aerospike 3.5.14 Community Edition with C Client 3.1.18

//add them into database
    std::map<newton::InternalLabelType, int>::const_iterator end = lbl_count.end();
    for(std::map<newton::InternalLabelType, int>::const_iterator it = lbl_count.begin(); it != end; ++it) {
        as_ldt lmap;
        as_ldt_init(&lmap, "images", AS_LDT_LMAP, NULL);

        as_string rec_guid;
        as_string_init(&rec_guid, (char*)guid.c_str(), false);

        //create key based on cluster number
        as_key key;
        as_key_init_int64(&key, "iq", "clusters", it->first);

        as_integer count;
        as_integer_init(&count, it->second);

        as_policy_apply policy;
        as_policy_apply_init(&policy);
        policy.key = AS_POLICY_KEY_SEND;

        as_error err;
        if(aerospike_lmap_put(m_as, &err, &policy, &key, &lmap,
                              (const as_val*)&rec_guid, (const as_val*)&count) != AEROSPIKE_OK) {
            fprintf(stderr, "aerospike_lmap_put() returned %d - %s", err.code, err.message);
        }

        as_key_destroy(&key);
        as_ldt_destroy(&lmap);
        as_string_destroy(&rec_guid);
        as_integer_destroy(&count);
    }

An other function calls the loop which usually run around 500 to 1000 times
lbl_count is a map that looks like this

{
1: 3,
2: 55,
3: 100
}

The insertion runs normal and breaks randomly at different points issuing an error:
malloc: * error for object 0xc6a3970: pointer being freed was not allocated

Debugging through lldb shows that the error occurred in aerospike_lmap_put(),
Here is the context of the code

// =======================================================================
// PUT
// =======================================================================
as_status aerospike_lmap_put(
    aerospike * as, as_error * err, const as_policy_apply * policy,
    const as_key * key, const as_ldt * ldt,
    const as_val * mkey, const as_val * mval)
{

//... codes 
    as_val* p_return_val = NULL;
    aerospike_key_apply(
        as, err, policy, key, DEFAULT_LMAP_PACKAGE, LDT_MAP_OP_PUT,
        (as_list *)&arglist, &p_return_val);

    as_arraylist_destroy(&arglist);

    if (ldt_parse_error(err) != AEROSPIKE_OK) {
        return err->code;
    }

    if (!p_return_val) {
        return as_error_set(err, AEROSPIKE_ERR_LDT_INTERNAL,
                "no value returned from server");
    }
    int64_t ival = as_integer_getorelse(as_integer_fromval(p_return_val), -1);
    as_val_destroy(p_return_val); //error happens here 

    if (ival == -1) {
        return as_error_set(err, AEROSPIKE_ERR_LDT_INTERNAL,
                "value returned from server not parse-able");
    }

    return err->code;
}

The error happens at line 97 as_val_destroy(p_return_val); where the p_return_val is attempted to be freed.

In the as_val_destroy() code, the error is traced to line 162

as_val * as_val_val_destroy(as_val * v)
{
    if ( v == NULL || !v->count ) return v;
    // if we reach the last reference, call the destructor, and free
    if ( 0 == cf_atomic32_decr(&(v->count)) ) {
        as_val_destroy_callbacks[ v->type ](v);     
        if ( v->free ) {
            cf_free(v); //error happens here
        }
        v = NULL;
    }
    return v;
}

I have tested the code on ubuntu 14.04 gcc 4.8.2, and it completes without any issues.
If anyone can provide some help to this will be greatly appreciated!

Leak when overwriting a bin.

It seems that when a bin is overwritten in a record, it is not destroyed properly.
By looking at the code I'm not sure why it happens as it seems that all the executions path are using as_record_bin_forupdate which should free the previous value if any.

It seems that the problem happen with as_list. I'll try to comment on the ticket with a test casse when I can.

Another crash, now in aerospike_key_put

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff59b6700 (LWP 22194)]
asval_to_clobject (val=0xf736b54a, obj=obj@entry=0x7ffff59b5d90) at src/main/aerospike/_shim.c:171
171 src/main/aerospike/_shim.c: No such file or directory.
(gdb) bt
#0 asval_to_clobject (val=0xf736b54a, obj=obj@entry=0x7ffff59b5d90)

at src/main/aerospike/_shim.c:171

#1 0x00007ffff59e67fa in asbinvalue_to_clobject (binval=,

obj=obj@entry=0x7ffff59b5d90) at src/main/aerospike/_shim.c:224

#2 0x00007ffff59e682d in asbin_to_clbin (as=as@entry=0x7fffffffdd80, cl=cl@entry=0x7ffff59b5d80)

at src/main/aerospike/_shim.c:231

#3 0x00007ffff59e6861 in asrecord_to_clbins (rec=rec@entry=0x7fffec0026a0,

bins=bins@entry=0x7ffff59b5d80, nbins=nbins@entry=1) at src/main/aerospike/_shim.c:238

#4 0x00007ffff59cdb5b in aerospike_key_put (as=0x7ffff5c005e8, err=0x7fffec0021d8,

policy=<optimized out>, key=0x7fffec0025f8, rec=0x7fffec0026a0)
at src/main/aerospike/aerospike_key.c:332

SSLv3_client_method is not available on Debian Stretch

Debian Stretch builds openssl with no-sslv3-method flag, hence SSLv3_client_method is no longer available, which is used here ( https://github.com/aerospike/aerospike-client-c/blob/master/src/main/aerospike/as_tls.c#L493 ). So linker fails when compiling against it:
[..] libaerospike.so: undefined reference to 'SSLv3_client_method'

No (public) API to close loops individually

As the public API goes, I can close all loops with as_event_close_loops, but not individually each of them. The immediate problem is that when I attach a uv loop by as_event_set_external_loop, the uv loop is attached some handles that prevents it to exit automatically; but if I call as_event_close_loops, all are closed, of course, which means as loops attached to other uv loops are affected.

Having a look at what is done by as_event_close_loops, this should be easy to support (and sort of "supported" in as_event_internal.h). I did not submit a PR directly because this involves changes to the API, so I would like you to decide that.

malloc return checks

Hi,

I am using the C client for aerospike. I had a quick question, I see that in various places the cf_malloc() is not checked for null returns. (example as_command.c) this causes a crash when we run out of memory. Any thoughts ?

Thanks.

With use_shm=true client prints info request to stdout

With shared memory enabled, the client sometimes dumps what looks like an info request string prefixed with "#" to stdout:

$ ./test
Connected to cluster
#node
partition-generation
services

The string after the "#" matches the info request string defined in INFO_STR_CHECK at as_node.c:592.

This is the simple test program with which I am able to reproduce the issue:

#include "aerospike/aerospike.h"

int main()
{
    as_error err;

    as_config config;
    as_config_init(&config);
    as_config_add_host(&config, "192.168.33.10", 3000);
    config.use_shm = true;
    config.shm_key = 0xa5000000;

    aerospike as;
    aerospike_init(&as, &config);
    if (aerospike_connect(&as, &err) != AEROSPIKE_OK) {
        fprintf(stderr, "error(%d) %s at [%s:%d]", err.code, err.message, err.file, err.line);
    }

    printf("Connected to cluster\n");
    sleep(60);

    if (aerospike_close(&as, &err) != AEROSPIKE_OK) {
        fprintf(stderr, "error(%d) %s at [%s:%d]", err.code, err.message, err.file, err.line);
    }
}

The issue seems to only occur if I terminate the program early, before the call to aerospike_close, and then run it again. The string also doesn't get dumped immediately; it seems to happen once the lock on the shared memory segment expires.

3/3 clients segfaulted when attempted to connect

Client version: 4.1.3 C Client Library with libev support.

3 clients async querying database every second segfaulted around the same time ( after roughly 30 hours running ):

• Node 2 --> 2017-01-15 08:36:10
• Node 1 --> 2017-01-15 08:36:22
• Node 3 --> 2017-01-15 08:38:05

Node 1 - BB9A0C114EDE000 ( running Aerospike-Server-3.10.1.1, libc6 2.19-18+deb8u4 ):

(gdb) bt
#0  __find_specmb (
    format=0x7f85c0ddd4be <fmt> "%u.%u.%u.%u")
    at printf-parse.h:108
#1  _IO_vfprintf_internal (s=s@entry=0x7ffdd5e255e0, 
    format=format@entry=0x7f85c0ddd4be <fmt> "%u.%u.%u.%u", ap=ap@entry=0x7ffdd5e25708) at vfprintf.c:1315
#2  0x00007f85c0ce33fb in __IO_vsprintf (
    string=0x7ffdd5e257e0 "", 
    format=0x7f85c0ddd4be <fmt> "%u.%u.%u.%u", 
    args=args@entry=0x7ffdd5e25708) at iovsprintf.c:42
#3  0x00007f85c0cc7eb7 in __sprintf (
    s=s@entry=0x7ffdd5e257e0 "", 
    format=format@entry=0x7f85c0ddd4be <fmt> "%u.%u.%u.%u") at sprintf.c:32
#4  0x00007f85c0d7c2df in inet_ntop4 (
    src=src@entry=0x71e174 "\300\250\001\001", 
    dst=dst@entry=0x7ffdd5e25970 "|", size=size@entry=64)
    at inet_ntop.c:99
#5  0x00007f85c0d7c531 in __GI_inet_ntop (af=af@entry=2, 
    src=src@entry=0x71e174, 
    dst=dst@entry=0x7ffdd5e25970 "|", size=size@entry=64)
    at inet_ntop.c:67
#6  0x00007f85c247a9e0 in as_address_name (
    addr=addr@entry=0x71e170, 
    name=name@entry=0x7ffdd5e25970 "|", 
    size=size@entry=64)
    at src/main/aerospike/as_address.c:32
#7  0x00007f85c2488a92 in as_node_add_address (
    node=node@entry=0x71fcc0, addr=addr@entry=0x71e170)
    at src/main/aerospike/as_node.c:168
#8  0x00007f85c2488ddd in as_node_create (
    cluster=cluster@entry=0x71c370, 
    hostname=0x7ffdd5e260fb "192.168.1.1", 
    tls_name=0x7ffdd5e260f9 "", port=<optimized out>, 
    is_alias=<optimized out>, addr=addr@entry=0x71e170, 
    node_info=node_info@entry=0x7ffdd5e25ad0)
    at src/main/aerospike/as_node.c:96
#9  0x00007f85c248c874 in as_peers_create_node (
    addr=0x71e170, node_info=0x7ffdd5e25ad0, 
    is_alias=<optimized out>, host=0x7ffdd5e25fd0, 
    cluster=0x71c370, peers=0x7ffdd5e2a390)
    at src/main/aerospike/as_peers.c:106
#10 as_peers_validate_node (
    peers=peers@entry=0x7ffdd5e2a390, 
    cluster=cluster@entry=0x71c370, 
    host=host@entry=0x7ffdd5e25fd0, 
    expected_name=expected_name@entry=0x7ffdd5e260e9 "BB9A0C114EDE000", 
    is_peers_protocol=is_peers_protocol@entry=true)
    at src/main/aerospike/as_peers.c:170
#11 0x00007f85c248d1c2 in as_peers_parse_peers (
    peers=peers@entry=0x7ffdd5e2a390, 
    err=err@entry=0x7ffdd5e2a3d0, 
    cluster=cluster@entry=0x71c370, 
    node=node@entry=0x71d910, 
    buf=0x7ffdd5e260e0 "5,3000,[[BB9A0C114EDE000")
    at src/main/aerospike/as_peers.c:472
#12 0x00007f85c2489786 in as_node_process_peers (
    peers=0x7ffdd5e2a390, values=0x7ffdd5e260b0, 
    node=0x71d910, err=0x7ffdd5e2a3d0, cluster=0x71c370)
    at src/main/aerospike/as_node.c:578
#13 as_node_refresh_peers (
    cluster=cluster@entry=0x71c370, 
    err=err@entry=0x7ffdd5e2a3d0, 
    node=node@entry=0x71d910, 
    peers=peers@entry=0x7ffdd5e2a390)
    at src/main/aerospike/as_node.c:631
#14 0x00007f85c2481a44 in as_cluster_tend (
    cluster=cluster@entry=0x71c370, 
    err=err@entry=0x7ffdd6623070, 
    enable_seed_warnings=enable_seed_warnings@entry=true, 
    config_change=config_change@entry=false)
    at src/main/aerospike/as_cluster.c:553
#15 0x00007f85c2482261 in as_wait_till_stabilized (
    err=0x7ffdd6623070, cluster=0x71c370)
    at src/main/aerospike/as_cluster.c:629
#16 as_cluster_init (cluster=cluster@entry=0x71c370, 
    err=err@entry=0x7ffdd6623070, 
    fail_if_not_connected=<optimized out>)
    at src/main/aerospike/as_cluster.c:749
#17 0x00007f85c2482b0a in as_cluster_create (
    config=config@entry=0x6024d0 <as+16>, 
    err=err@entry=0x7ffdd6623070, 
    cluster_out=cluster_out@entry=0x6024c8 <as+8>)
    at src/main/aerospike/as_cluster.c:982
#18 0x00007f85c2468614 in aerospike_connect (
    as=as@entry=0x6024c0 <as>, 
    err=err@entry=0x7ffdd6623070)
    at src/main/aerospike/aerospike.c:164
#19 0x000000000040121b in main (argc=<optimized out>, 
    argv=<optimized out>) at scan_query.c:82

Node 2 - BB9755A597AC40C ( running Aerospike-Server-3.11.0.1, libc6 2.19-18+deb8u4 ):

(gdb) bt
#0  __find_specmb (format=0x7fe9563334be <fmt> "%u.%u.%u.%u")
    at printf-parse.h:108
#1  _IO_vfprintf_internal (s=s@entry=0x7ffdf3e255e0, 
    format=format@entry=0x7fe9563334be <fmt> "%u.%u.%u.%u", 
    ap=ap@entry=0x7ffdf3e25708) at vfprintf.c:1315
#2  0x00007fe9562393fb in __IO_vsprintf (string=0x7ffdf3e257e0 "", 
    format=0x7fe9563334be <fmt> "%u.%u.%u.%u", 
    args=args@entry=0x7ffdf3e25708) at iovsprintf.c:42
#3  0x00007fe95621deb7 in __sprintf (s=s@entry=0x7ffdf3e257e0 "", 
    format=format@entry=0x7fe9563334be <fmt> "%u.%u.%u.%u")
    at sprintf.c:32
#4  0x00007fe9562d22df in inet_ntop4 (
    src=src@entry=0x21c0ea4 "\300\250\001\002", 
    dst=dst@entry=0x7ffdf3e25970 "|", size=size@entry=64)
    at inet_ntop.c:99
#5  0x00007fe9562d2531 in __GI_inet_ntop (af=af@entry=2, 
    src=src@entry=0x21c0ea4, dst=dst@entry=0x7ffdf3e25970 "|", 
    size=size@entry=64) at inet_ntop.c:67
#6  0x00007fe9579d09e0 in as_address_name (
    addr=addr@entry=0x21c0ea0, name=name@entry=0x7ffdf3e25970 "|", 
    size=size@entry=64) at src/main/aerospike/as_address.c:32
#7  0x00007fe9579dea92 in as_node_add_address (
    node=node@entry=0x21c0ec0, addr=addr@entry=0x21c0ea0)
    at src/main/aerospike/as_node.c:168
#8  0x00007fe9579deddd in as_node_create (
    cluster=cluster@entry=0x21bd170, 
    hostname=0x7ffdf3e260fb "192.168.1.2", 
    tls_name=0x7ffdf3e260f9 "", port=<optimized out>, 
    is_alias=<optimized out>, addr=addr@entry=0x21c0ea0, 
    node_info=node_info@entry=0x7ffdf3e25ad0)
    at src/main/aerospike/as_node.c:96
#9  0x00007fe9579e2874 in as_peers_create_node (addr=0x21c0ea0, 
    node_info=0x7ffdf3e25ad0, is_alias=<optimized out>, 
    host=0x7ffdf3e25fd0, cluster=0x21bd170, peers=0x7ffdf3e2a390)
    at src/main/aerospike/as_peers.c:106
#10 as_peers_validate_node (peers=peers@entry=0x7ffdf3e2a390, 
    cluster=cluster@entry=0x21bd170, 
    host=host@entry=0x7ffdf3e25fd0, 
    expected_name=expected_name@entry=0x7ffdf3e260e9 "BB9755A597AC40C", is_peers_protocol=is_peers_protocol@entry=true)
    at src/main/aerospike/as_peers.c:170
#11 0x00007fe9579e31c2 in as_peers_parse_peers (
    peers=peers@entry=0x7ffdf3e2a390, err=err@entry=0x7ffdf3e2a3d0, 
    cluster=cluster@entry=0x21bd170, node=node@entry=0x21bc240, 
    buf=0x7ffdf3e260e0 "7,3000,[[BB9755A597AC40C")
    at src/main/aerospike/as_peers.c:472
#12 0x00007fe9579df786 in as_node_process_peers (
    peers=0x7ffdf3e2a390, values=0x7ffdf3e260b0, node=0x21bc240, 
    err=0x7ffdf3e2a3d0, cluster=0x21bd170)
    at src/main/aerospike/as_node.c:578
#13 as_node_refresh_peers (cluster=cluster@entry=0x21bd170, 
    err=err@entry=0x7ffdf3e2a3d0, node=node@entry=0x21bc240, 
    peers=peers@entry=0x7ffdf3e2a390)
    at src/main/aerospike/as_node.c:631
#14 0x00007fe9579d7a44 in as_cluster_tend (
    cluster=cluster@entry=0x21bd170, err=err@entry=0x7ffdf46237f0, 
    enable_seed_warnings=enable_seed_warnings@entry=true, 
    config_change=config_change@entry=false)
    at src/main/aerospike/as_cluster.c:553
#15 0x00007fe9579d8261 in as_wait_till_stabilized (
    err=0x7ffdf46237f0, cluster=0x21bd170)
    at src/main/aerospike/as_cluster.c:629
#16 as_cluster_init (cluster=cluster@entry=0x21bd170, 
    err=err@entry=0x7ffdf46237f0, 
    fail_if_not_connected=<optimized out>)
    at src/main/aerospike/as_cluster.c:749
#17 0x00007fe9579d8b0a in as_cluster_create (
    config=config@entry=0x6024d0 <as+16>, 
    err=err@entry=0x7ffdf46237f0, 
    cluster_out=cluster_out@entry=0x6024c8 <as+8>)
    at src/main/aerospike/as_cluster.c:982
#18 0x00007fe9579be614 in aerospike_connect (
    as=as@entry=0x6024c0 <as>, err=err@entry=0x7ffdf46237f0)
    at src/main/aerospike/aerospike.c:164
#19 0x000000000040121b in main (argc=<optimized out>, 
    argv=<optimized out>) at scan_query.c:82

Node 3 - BB9D12C7CBAE290 ( running Aerospike-Server-3.10.1.1, libc6 version2.19-18+deb8u3 ) :

(gdb) bt
#0  __GI__IO_default_xsputn (f=0x7ffe04e325f0, data=0x7fc58fd2b756 <fmt>, 
    n=0) at genops.c:447
#1  0x00007fc58fc0b28d in _IO_vfprintf_internal (
    s=s@entry=0x7ffe04e325f0, 
    format=format@entry=0x7fc58fd2b756 <fmt> "%u.%u.%u.%u", 
    ap=ap@entry=0x7ffe04e32718) at vfprintf.c:1323
#2  0x00007fc58fc313fb in __IO_vsprintf (string=0x7ffe04e327f0 "", 
    format=0x7fc58fd2b756 <fmt> "%u.%u.%u.%u", 
    args=args@entry=0x7ffe04e32718) at iovsprintf.c:42
#3  0x00007fc58fc15eb7 in __sprintf (s=s@entry=0x7ffe04e327f0 "", 
    format=format@entry=0x7fc58fd2b756 <fmt> "%u.%u.%u.%u")
    at sprintf.c:32
#4  0x00007fc58fcca2df in inet_ntop4 (
    src=src@entry=0x1da4f34 "\300\250\001\001", 
    dst=dst@entry=0x7ffe04e32980 "|", size=size@entry=64)
    at inet_ntop.c:99
#5  0x00007fc58fcca531 in __GI_inet_ntop (af=af@entry=2, 
    src=src@entry=0x1da4f34, dst=dst@entry=0x7ffe04e32980 "|", 
    size=size@entry=64) at inet_ntop.c:67
#6  0x00007fc5913c79e0 in as_address_name (addr=addr@entry=0x1da4f30, 
    name=name@entry=0x7ffe04e32980 "|", size=size@entry=64)
    at src/main/aerospike/as_address.c:32
#7  0x00007fc5913d5a92 in as_node_add_address (node=node@entry=0x1da6e10, 
    addr=addr@entry=0x1da4f30) at src/main/aerospike/as_node.c:168
#8  0x00007fc5913d5ddd in as_node_create (
    cluster=cluster@entry=0x1da3680, 
    hostname=0x7ffe04e3310b "192.168.1.1", tls_name=0x7ffe04e33109 "", 
    port=<optimized out>, is_alias=<optimized out>, 
    addr=addr@entry=0x1da4f30, node_info=node_info@entry=0x7ffe04e32ae0)
    at src/main/aerospike/as_node.c:96
#9  0x00007fc5913d9874 in as_peers_create_node (addr=0x1da4f30, 
    node_info=0x7ffe04e32ae0, is_alias=<optimized out>, 
    host=0x7ffe04e32fe0, cluster=0x1da3680, peers=0x7ffe04e373a0)
    at src/main/aerospike/as_peers.c:106
#10 as_peers_validate_node (peers=peers@entry=0x7ffe04e373a0, 
    cluster=cluster@entry=0x1da3680, host=host@entry=0x7ffe04e32fe0, 
    expected_name=expected_name@entry=0x7ffe04e330f9 "BB9A0C114EDE000", 
    is_peers_protocol=is_peers_protocol@entry=true)
    at src/main/aerospike/as_peers.c:170
#11 0x00007fc5913da1c2 in as_peers_parse_peers (
    peers=peers@entry=0x7ffe04e373a0, err=err@entry=0x7ffe04e373e0, 
    cluster=cluster@entry=0x1da3680, node=node@entry=0x1da5240, 
    buf=0x7ffe04e330f0 "7,3000,[[BB9A0C114EDE000")
    at src/main/aerospike/as_peers.c:472
#12 0x00007fc5913d6786 in as_node_process_peers (peers=0x7ffe04e373a0, 
    values=0x7ffe04e330c0, node=0x1da5240, err=0x7ffe04e373e0, 
    cluster=0x1da3680) at src/main/aerospike/as_node.c:578
#13 as_node_refresh_peers (cluster=cluster@entry=0x1da3680, 
    err=err@entry=0x7ffe04e373e0, node=node@entry=0x1da5240, 
    peers=peers@entry=0x7ffe04e373a0) at src/main/aerospike/as_node.c:631
#14 0x00007fc5913cea44 in as_cluster_tend (
    cluster=cluster@entry=0x1da3680, err=err@entry=0x7ffe05631300, 
    enable_seed_warnings=enable_seed_warnings@entry=true, 
    config_change=config_change@entry=false)
    at src/main/aerospike/as_cluster.c:553
#15 0x00007fc5913cf261 in as_wait_till_stabilized (err=0x7ffe05631300, 
    cluster=0x1da3680) at src/main/aerospike/as_cluster.c:629
#16 as_cluster_init (cluster=cluster@entry=0x1da3680, 
    err=err@entry=0x7ffe05631300, fail_if_not_connected=<optimized out>)
    at src/main/aerospike/as_cluster.c:749
#17 0x00007fc5913cfb0a in as_cluster_create (
    config=config@entry=0x6024d0 <as+16>, err=err@entry=0x7ffe05631300, 
    cluster_out=cluster_out@entry=0x6024c8 <as+8>)
    at src/main/aerospike/as_cluster.c:982
#18 0x00007fc5913b5614 in aerospike_connect (as=as@entry=0x6024c0 <as>, 
    err=err@entry=0x7ffe05631300) at src/main/aerospike/aerospike.c:164
#19 0x000000000040121b in main (argc=<optimized out>, 
    argv=<optimized out>) at scan_query.c:82

Cleanup is not finished on return from aerospike_close when using libuv

The uv code queues up closing of connections but the ev code closes immediately. As a result, when using libuv, a return from aerospike_close still leaves open connections to be closed, which will be executed on later loop iterations; but at this point it is unsafe to call as_event_close_loops from one of the loop threads, because a loop will be closed immediately if detected to be on the same thread as the caller, and pending connections will be leaked.

Btw. in uv the aerospike_close -> aerospike_destroy -> as_event_close_loops process gives me a segfault in trying to unlock a mutex at the end; I am not sure if these are related, but if I skip that thread == current thread check and blindly queue up the close command, it works fine.

Segment fault in as_event_loop_get

Hello All,

I am working on Aerospike Async C APIs. For async event handling, I am creating a single thread using
as_event_loop_get. The complete async code is available in async_client.txt file attached herewith.

Whenever I run the program, I keep on getting segment faults.
async_client.txt

The async client program creates a pipe between parent and child process. The child process keeps on adding key+values on a pipe. Parent reads from pipe and inserts key into Aerospike KV store.

As I was continuously running into segment faults, tried debugging the code and realized the function as_event_loop_get tries to access invalid pointer. The problem seems to be with the next pointer used in circular link list.

To make sure, as_event_loop_get is really a problem. I wrote simple event loop programm attached as ev_loop.txt
ev_loop.txt

Please have a look. I really appreciate any help that you can provide.

Thanks and Regards,
Prasad

Valgrind with LDT: Conditional jump or move depends on uninitialised value

When running valgrind on code that uses LDT (in our case map), there is an occurrence of an initialised read on an as_error struct.

The error itself seems to be fairly harmless (caused by not initialising the message member of the struct to 0), but causes our valgrind tests to fail:

==12284== Memcheck, a memory error detector
==12284== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
==12284== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
==12284== Command: ./test
==12284==
==12284== Conditional jump or move depends on uninitialised value(s)
==12284==    at 0x4E4D4CE: ldt_parse_error (_ldt.c:23)
==12284==    by 0x4E521DE: aerospike_lmap_put (aerospike_lmap.c:88)
==12284==    by 0x41EDAF: AsEngine::addToSet(std::string const&, std::string const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&) (AsEngine.cpp:316)
==12284==    by 0x41F740: AsEngine::addRef(std::string const&, std::string const&, std::string const&) (AsEngine.cpp:376)
==12284==    by 0x416F15: AsSet::storeKeyRef(std::string const&) (AsSet.cpp:125)
==12284==    by 0x416A2F: AsSet::putString(std::string const&, std::string const&) (AsSet.cpp:63)
==12284==    by 0x414FB2: main (main.cpp:32)
==12284==
==12284== Conditional jump or move depends on uninitialised value(s)
==12284==    at 0x4E4D4CE: ldt_parse_error (_ldt.c:23)
==12284==    by 0x4E51349: aerospike_lmap_destroy (aerospike_lmap.c:462)
==12284==    by 0x41F02F: AsEngine::addToSet(std::string const&, std::string const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&) (AsEngine.cpp:327)
==12284==    by 0x41F740: AsEngine::addRef(std::string const&, std::string const&, std::string const&) (AsEngine.cpp:376)
==12284==    by 0x416F15: AsSet::storeKeyRef(std::string const&) (AsSet.cpp:125)
==12284==    by 0x416A2F: AsSet::putString(std::string const&, std::string const&) (AsSet.cpp:63)
==12284==    by 0x414FB2: main (main.cpp:32)
==12284==

We have a workaround by using as_error err = {}; in our code, but we prefer to have this fixed to the LDT in the library.

Leak in as_batch_read_execute

Hello, everyone! I'm writing a c++ wrapper around your library, and after I've implemented some batch operations using aerospike_batch_read_async, I've noticed continuously increasing memory consumption of my application in production environment. Here is how it looks like:

It is heaptrack_gui visualization, and lowest rasing area is memory allocated by as_vector_increase_capacity (plateau on right side is because of very low rps at night). Also heaptrack was able to tell where allocations occure:

I'm sorry for screenshots, but profiling took about day, and log takes over 6GB.
It seems that sometimes uneven distribution of keys among nodes (in case of n_nodes >= 2) occurs, which leads to overflowing capacity of some as_batch_node.offsets (which are allocated on stack or heap, depending on n_keys value), which leads to implicit copying to the heap, and nobody takes care of that memory. I've made pull request and I'd be appreciate if you take a look at it. I currently run my application with patched library and it looks like problem has gone ;)

libev and libevent build speed comparison.

Hello. The performance test for building a utility benchmarks with the libev library gave the following results:
time ./benchmarks_libev -h 127.0.0.1 -p 3000 -a -n test -s newtest -k 1000000 -o B:256 -w RU,50 -t 3000000 -T 10 -z 1
real 0m26.936s
user 0m5.948s
sys 0m20.984s

However, the assembly with the libevent library is much slower:
time ./benchmarks_libevent -h 127.0.0.1 -p 3000 -a -n test -s newtest -k 1000000 -o B:256 -w RU,50 -t 3000000 -T 10 -z 1
real 0m36.975s
user 0m8.884s
sys 0m27.948s

Why such a big time difference?

A potential memory leak

https://github.com/aerospike/aerospike-client-c/blob/master/src/main/aerospike/as_info.c#L255
If the read above failed, response would leak.

No deep copy of `as_config_s` in `aerospike_init` after making `hosts` member as a pointer instead of C-array in commit 6b5a82f8

After recent changes in 6b5a82f8 in file as_config.h, member hosts changed from this:

as_config_host hosts[AS_CONFIG_HOSTS_SIZE];

to this:

as_vector* hosts;

But aerospike_init copies config using simple memcpy:

static aerospike*
aerospike_defaults(aerospike* as, bool free, as_config* config)
{
    as->_free = free;
    as->cluster = NULL;
    if ( config != NULL ) {
        memcpy(&as->config, config, sizeof(as_config));
    }

aerospike*
aerospike_init(aerospike* as, as_config* config)
{
    if ( !as ) return as;
    return aerospike_defaults(as, false, config);
}

As a result of that two places have the same pointer to hosts which may lead to double free due to diluting ownership of pointer to hosts.

This is effectively happen when as_cluster_create destroys hosts:

// Destroy config hosts vector after transferring data to cluster.
as_vector_destroy(config->hosts);

(this change from the same commit).

Then we destroy (in our library) separately our config (which is supposedly deeply copied) after we call aerospike_init. And here crash happens (in our library).

We will do workaround about this case in our library. But I think it's also should be fixed in your library as well (i.e. don't use memcpy to copy config in aerospike_defaults).

Also, I suggest as_vector_destroy in as_cluster_create should be replaced by some specific destructor similar to how hosts destroyed in as_config_destroy since it's not just as_vector_destroy.