Segmentation fault during batchGet

[aerospike]

Segmentation fault during batchGet

[GayathriKaliyamoorthy]

We have identified the reason for the segfault. Soon we will make a release and update you. Thanks for your patience.

Thanks

[GayathriKaliyamoorthy]

We have made an official release with the fix for crash in batchGet API, [email protected]. Please try the latest version and let us know if that fixes the crash.

Thanks.

[rasmusx]

Hello,

In updated aerospike nodejs module (v 1.0.29) I still get the same segmentation fault.

[vivekkrbajpai]

Thanks @GayathriKaliyamoorthy for the update i will check it and revert you back.

[GayathriKaliyamoorthy]

@vivekkrbajpai, @rasmusx Is your crash issue solved with the latest release 1.0.34, please do let us know so that we can close this issue.

Thanks

[vivekkrbajpai]

Hey @GayathriKaliyamoorthy i have tested the latest build 1.0.34 it contains the same segfault during batchGet.

[ryanwitt]

I've just tested with 1.0.35 and it looks like it is still segfaulting for me as well. Switched back to 1.0.27.

Hi, I'm having the same issue with 1.0.28, then updated to 1.0.35 and same error. Any news on this?

[vivekkrbajpai]

I think you should go back to 1.0.27 it doesn't have this issue.

Regards

Vivek

Sent from Mailbox

On Thu, Apr 9, 2015 at 2:15 AM, fdnieves [email protected] wrote:

Hi, I'm having the same issue with 1.0.28, then updated to 1.0.35 and same error. Any news on this?

Reply to this email directly or view it on GitHub:
#40 (comment)

I'm trying it right now, thanks for the tip.

However they should notice this bug so they fix it again on the latest build.

[wchu-citrusleaf]

Sorry for the inconvenience. Would be great if folks can add more source code and stack trace, to make sure we are looking in the right direction

[wiszman]

I managed to get a stack trace using segfault-handler node module. Hopefully this helps.

PID 3705 received SIGSEGV for address: 0x4
/home/ec2-user/node_modules/segfault-handler/build/Release/segfault-handler.node(+0xfef)[0x7f1da9744fef]
/lib64/libpthread.so.0(+0xfae0)[0x7f1db0dc2ae0]
/lib64/libc.so.6(memcpy+0x37)[0x7f1db0ab2bb7]
/home/ec2-user/node_modules/aerospike/build/Release/aerospike.node(+0x35d89)[0x7f1da997bd89]
/home/ec2-user/node_modules/aerospike/build/Release/aerospike.node(aerospike_batch_get+0x21)[0x7f1da997c236]
/home/ec2-user/node_modules/aerospike/build/Release/aerospike.node(+0x1d3c8)[0x7f1da99633c8]
node[0x95e2f0]
node[0x953be1]
/lib64/libpthread.so.0(+0x7ddb)[0x7f1db0dbaddb]
/lib64/libc.so.6(clone+0x6d)[0x7f1db0b097bd]
Aborted

[GayathriKaliyamoorthy]

We are looking into the issue. We will resolve it as early as possible and give you some updates on this.

[GayathriKaliyamoorthy]

I found one potential issue, where if any of the key values passed into the batch is undefined, the batch API crashes. We will release with the fix for this in next release. But I would like to know more details about how the application invokes the batch API. It would help us improve by making sure our test cases cover all such scenario before we make the next official release.

Thanks.

[vivekkrbajpai]

@GayathriKaliyamoorthy i hv tested it in ec2 which have amazon linux. My application code is simple as included in example.

Use Case Scenario : I hv 3 keys to query in a single batch which leads me these cases.

Case 1: May be all three exist in the db.
Case 2: May be none of them exist in db.
Case 3: Few of them available in db.

I am not getting any segfault with 1.0.27 rest all the version lead me a segfault including the latest one 1.0.35.

[GayathriKaliyamoorthy]

Hi
We fixed a bug related to this segfault in 1.0.37. Could you check this and please let us know your feedback.

Thanks

[ryanwitt]

Can still reproduce:

PID 17276 received SIGSEGV for address: 0x4
/home/ec2-user/node_modules/segfault-handler/build/Release/segfault-handler.node(+0xfef)[0x7fa5ccbb0fef]
/lib64/libpthread.so.0(+0xfae0)[0x7fa5d4d6eae0]
/lib64/libc.so.6(memcpy+0x37)[0x7fa5d4a5ebb7]
/home/ec2-user/node_modules/aerospike/build/Release/aerospike.node(+0x35119)[0x7fa5cd770119]
/home/ec2-user/node_modules/aerospike/build/Release/aerospike.node(aerospike_batch_get+0x21)[0x7fa5cd7705c6]
/home/ec2-user/node_modules/aerospike/build/Release/aerospike.node(+0x1cc58)[0x7fa5cd757c58]
/usr/local/bin/node[0x95e2f0]
/usr/local/bin/node[0x953be1]
/lib64/libpthread.so.0(+0x7ddb)[0x7fa5d4d66ddb]
/lib64/libc.so.6(clone+0x6d)[0x7fa5d4ab57bd]

$ npm list |grep aerospike
├─┬ [email protected]
$ node --version
v0.10.29

What else can I do to help?

[ryanwitt]

Just saw 1.0.38! Will test.

[GayathriKaliyamoorthy]

Hi @ryanwitt

Is it possible for you to give access to your environment, to figure out this segfault.

[ryanwitt]

@GayathriKaliyamoorthy yup, let's work out a time this week. Should I email support@?

[GayathriKaliyamoorthy]

@ryanwitt. That would be great. Also if you send the credentials to access your environment, I can start digging the issue right away. :).

Thanks

[bbulkow]

@ryanwitt Are you still having problems? Can we get access to your machines?

[ryanwitt]

Yup, just got the traceback below on 1.0.40. Seems to happen on every version past 1.0.27.

I'll message you at your github profile email to setup access.

PID 15986 received SIGSEGV for address: 0x4
/home/ec2-user/node_modules/segfault-handler/build/Release/segfault-handler.node(+0xfef)[0x7f20f079dfef]
/lib64/libpthread.so.0(+0xfae0)[0x7f20f8779ae0]
/lib64/libc.so.6(memcpy+0x37)[0x7f20f8469bb7]
/home/ec2-user/node_modules/aerospike/build/Release/aerospike.node(+0x36a99)[0x7f20f135ba99]
/home/ec2-user/node_modules/aerospike/build/Release/aerospike.node(aerospike_batch_get+0x21)[0x7f20f135bf46]
/home/ec2-user/node_modules/aerospike/build/Release/aerospike.node(+0x1ce78)[0x7f20f1341e78]
/usr/local/bin/node[0x95e2f0]
/usr/local/bin/node[0x953be1]
/lib64/libpthread.so.0(+0x7ddb)[0x7f20f8771ddb]
/lib64/libc.so.6(clone+0x6d)[0x7f20f84c07bd]

[vivekkrbajpai]

Hey @GayathriKaliyamoorthy I just tested the version 1.0.40 in our production severs and it seems like the old bug is still there.
Here is the system configuration.

AWS ami id : ami-e8427a80/ami-978d91fe(Amazon linux)
EC2 instance : c3.xlarge /c3.large
Nodejs : 0.10.38/0.10.26

[GayathriKaliyamoorthy]

Could you give us the information about the size of batch, (number of records to be read in the batch request)? Can it happen in your logic that, the size is less than the size of Aerospike cluster?

We found a potential bug in cases where batch size is less than cluster size. We fixed it and made an official release. Please upgrade it to the latest 1.0.42 and let us know if this fixes your crash.

Thanks.

[GayathriKaliyamoorthy]

@vivekkrbajpai @ryanwitt
Could you confirm, if the latest release solves the batchGet issue?

Thanks

[vivekkrbajpai]

Hey @GayathriKaliyamoorthy, The latest release fixed the batchGet issue 👍

[ryanwitt]

Looks like 1.0.43 fixes this for us! Thanks @GayathriKaliyamoorthy!