1.0.36 memory leak

[aerospike]

1.0.36 memory leak

[woozyking]

We noticed the same thing.

I spent 2 days already thinking it was our app. But then the logic has not altered much on the server side.

I'll try downgrading aerospike module and see if it helps.

Downgraded aerospike module to 1.0.35 and the abnormal memory growth of the app process has stopped.

On the bright side, this issue pushed me to refactored the code base of the app and it's much cleaner ;)

[GayathriKaliyamoorthy]

We ran our longevity tests with node versions v0.12.* and v0.10.*
Both the longevity tests ran successfully. And also, I ran them under valgrind. I didn't see a considerable leak between the two versions. Only difference I observed was, the rate of growth in memory usage from the top output. In case the node v0.10.* the process memory reached 1.7GB after a while and it remained the same. The process did not crash. In case of node v0.12.* the process memory reached 1.9GB little earlier and stabilized at 1.9Gb. Did not cross beyond that. I'll update you with more details at the earliest possible.

One more question. Do you use Buffers / Blob data in your application. In that case, V8 in node version v0.12.* (The internal engine used by nodejs) allocates memory in large blocks(typically 4K or 8K).

Thanks.

[jenslukas]

Experienced the same issue with 1.0.36 and node 0.10.36.
I could track down the leak to a simple get request. Afterwards also tried to run against a clean aerospike set with the same outcome. The node process goes up to 1.7 GB within a few minutes and overall response time of the application then increases from average 10ms up to 500ms and more. It does not crash but response time is too high.

I downgraded to 1.0.35 and now the memory stays around 200 MB.

We store JSON objects as value with size being less then 1kb.

[woozyking]

@GayathriKaliyamoorthy the app that we're running does pretty much just one thing, batchGet a bunch of keys, strip out other metadata and keep only bins values (using underscore _.compact(_.pluck(results, 'record')), where results being what the batchGet method returns), pass them straight to browser through socket.io, and keep/refresh a copy of those bin values in one Object. We recursively call the same above routine with setTimeout every N interval.

To illustrate, this is our module as that performs the batchGet (with logic to lazily connect to Aerospike cluster if it's not connected).

var as = (function(aerospike) {
  var _client = aerospike.client({
    hosts: [{
      addr: process.env.AS || 'localhost',
      port: parseInt(process.env.AS_PORT || 3000, 10)
    }]
  });
  var _connected = false;
  const NO_RECONNECT = [
    aerospike.status.AEROSPIKE_ERR_TIMEOUT
  ];

  function _connect(callback) {
    if (!_connected) {
      _client.connect(function(err) {
        if (err.code === aerospike.status.AEROSPIKE_OK) {
          _connected = true;
          callback(null);
        } else {
          callback(err);
        }
      });
    }
  }

  function _close() {
    try {
      _client.close();
      console.log(new Date().toISOString() + ' :: Aerospike Closed');
    } catch(ignore) {
      //
    }

    _connected = false;
  }

  function _batchGet(keys, callback) {
    _client.batchGet(keys, function(err, results) {
      if (err.code === aerospike.status.AEROSPIKE_OK) {
        callback(null, results);
      } else {
        if (NO_RECONNECT.indexOf(err.code) < 0) {
          _close();
        }

        callback(err, null);
      }
    });
  }

  return {
    connect: _connect,
    close: _close,
    batchGet: function(keys, callback) {
      if (!_connected) {
        _connect(function(err) {
          if (err) {
            callback(err, null);
          } else {
            _batchGet(keys, callback);
          }
        });
      } else {
        _batchGet(keys, callback);
      }
    }
  };
})(require('aerospike'));

module.exports = as;

We then build a recursive routine to run as.batchGet every N interval

// see as module above
var as = require('./as');
// the cache
var LAST_VALUE = {};
// underscore
var _ = require('underscore');

function routine() { 
  // bunch of keys
  // var keys = ...
  as.batchGet(keys, function(err, results) {
    if (!err) {
      var records = _.compact(_.pluck(results, 'record'));
      // send records to browser through socket.io
      // ...
      // keep/refresh cache
      LAST_VALUE = records;
    } else {
      console.error((new Date()).toISOString() + ' :: ' + err.message);
    }

    setTimeout(routine, process.env.GET_INTERVAL || 1000);
  });
}

// start the recursive routine
routine();

I was hesitant on using setTimeout and recursion to perform this but I couldn't find any better ways (maybe using async module but I'll explore that later).

Also, please point out if my as module has any flaws. We basically want to have a crash free way of continuously batchGet said keys and auto-reconnects when needed.

[GayathriKaliyamoorthy]

Hi,

We identified the reason for both memory leak and batchGet failure. We will make an official release to the npm repository within next 2 days.

Thanks.

[woozyking]

👍

[haze-mobicow]

Great news! If you need the client to be tested before you push , I can run it through our use case and see if the memory leaks is gone..

[GayathriKaliyamoorthy]

Hi,

Thank you very much. I am planning to fork the aerospike-client-nodejs repository and put in all the changes in the forked repo. You can clone that and use it for testing.
Another way is I can mail the complete repository with the code changes. If you prefer the later, please share your email id. Otherwise the changes will be available in the fork.

Thanks.

[Hamper]

1.0.37 fixes this memory leak for me, thanks