chriso/redback

A high-level Redis library

redback's Issues

Catching redis connection errors

If I was using the Redis library directly, I would catch connection errors like so:

var redis = require("redis"),
    client = redis.createClient();

client.on("error", function (err) {
    // No Redis server instance running?
    console.log("Error " + err);
});

How can I achieve this when I'm using redback? I don't appear to have access to the underlying Redis object.

Thanks - Jules

redis/hiredis dependencies update

I'm not sure about your plans, so leave here text proposal, instead of pull request.

1. redis dependency can be updated to 0.8.1 - it's faster, and eliminates need to use hiredis.
2. hiredis could be removed:
   • not actual with node_redis 0.8.1
   • no binary dependency - more convenient
   • if someone really need hiredis, he can install it separately, on higher level

Pub/Sub example results in exception from node-redis

When using the example code for pub/sub (slightly modified to have a valid message)

var channel = redback.createChannel('chat').subscribe();

//To received messages
channel.on('message', function (msg) {
   console.log(msg);
});

//To send messages
channel.publish("hello world");

An exception is thrown:

/Users/addisonj/Projects/augment/node_modules/redback/node_modules/redis/index.js:523
            throw err;
                  ^
Error: Connection in pub/sub mode, only pub/sub commands may be used
    at RedisClient.send_command (/Users/addisonj/Projects/augment/node_modules/redback/node_modules/redis/index.js:726:15)
    at RedisClient.send_offline_queue (/Users/addisonj/Projects/augment/node_modules/redback/node_modules/redis/index.js:378:34)
    at RedisClient.on_ready (/Users/addisonj/Projects/augment/node_modules/redback/node_modules/redis/index.js:308:14)
    at RedisClient.on_info_cmd (/Users/addisonj/Projects/augment/node_modules/redback/node_modules/redis/index.js:341:14)
    at RedisClient.ready_check.send_anyway (/Users/addisonj/Projects/augment/node_modules/redback/node_modules/redis/index.js:365:14)
    at try_callback (/Users/addisonj/Projects/augment/node_modules/redback/node_modules/redis/index.js:520:9)
    at RedisClient.return_reply (/Users/addisonj/Projects/augment/node_modules/redback/node_modules/redis/index.js:590:13)
    at ReplyParser.RedisClient.init_parser (/Users/addisonj/Projects/augment/node_modules/redback/node_modules/redis/index.js:263:14)
    at ReplyParser.EventEmitter.emit (events.js:96:17)
    at ReplyParser.send_reply (/Users/addisonj/Projects/augment/node_modules/redback/node_modules/redis/lib/parser/javascript.js:297:10)

It seems a client can't be reused for both publish and subscribe and a new connection is needed. As redback is a high level, perhaps when subscribe is called, it creates an new client to listen on?

I shall try and see if I can get it working and open a PR.

Thanks

Emmiters for ready/error on createClient

Hello!

Couldn't find where to get the emmiters, if it's available for when the client is ready or have an error, like on node_redis "on" events.

I know I can instantiate the redis directly and use it, but then I lose the namespace facility, which is awesome.

Is there any other way?

Link to blog post on README.md is broken

The link to http://chris6f.com/rate-limiting-with-redis does not work.

Is there a mirror somewhere or another relevant page that could explain the algorithm?

Meanwhile, you can use https://web.archive.org/web/20131220125148/https://chris6f.com/rate-limiting-with-redis

is it possible to add two new api for SortedSet?

hi chriso,
firstly thank you for your work very much.
is it possible to add two new api for SortedSet?
the scene is as follow:
if we have a list of element and we want to get the score or rank for them, now we can only get one by one.

if there is an API which accept a list of element and query to redis with pipeline, it will only query to redis one time.

is it possible?

thanks a lot!

redback doesnt support select

Hi,

I might be missing something here but previously, talking directly to the node_redis client, I used to do this:

coffeescript

redis = require "redis"
client = redis.createClient()
client.select app.REDIS_DB

(where app is an express obj).

Refactoring this to redback reveals that the select method isn't supported on the created client.

redback = require('redback').createClient()
redback.select app.REDIS_DB
redback

TypeError: Object [object Object] has no method 'select'

Cheers,
Nick

authentication

One thing that I found lacking in this library was support for node_redis' authentication.
Sadly, this is a requirement at the work server and as such, I was unable to use this library.

As far as I can see, if this project utilizes node_redis, then maybe you can filter an auth method from redback to client.auth(password,callback)from node_redis

Test for RateLimit

I'm considering using the RateLimit structure in my current project. However, I don't see any test cases for RateLimit in the repository. Is the RateLimit structure well tested? Is the test case coming in soon?

CappedList.unshift makes no sense

What is the purpose of the CappedList.unshift? It does nothing as soon as a list get max length.

Random subset of followers/following

Hi,

I'm using redback to save a SocialGraph (like twitter) and its working great.
Thanks for creating this project. Its been a great help!

One thing that I think its missing is a way to get only a random subset of followers and/or following.
This is one of twitter functionalities: It shows in the right panel (next to the timeline) a subset of 5 followers and 5 following.

From what I gathered this cannot be done using sets in redis, because you can only get one random element each time (using the SRANDMEMBER command), so that leaves doing this on the client side, and getting for instance 5 random elements from a 1000 array in javascript is not the best way to go.
Do you know another way to do this?

I've already asked in the redis google groups if there are other ways to do this.
http://groups.google.com/group/redis-db/browse_thread/thread/a58699d22de43456

Thanks for your help.
Best regards,
Hugo
P.S: Sorry for adding this as an issue, but I didn't found a better place to ask this question.

Ratelimiting questions

Hey this is not an issue with your rate-limiting module, I am writing to ask why you chose the design decisions that you chose. I have read your blog post https://chris6f.com/rate-limiting-with-redis many times over, and have done some reading on consistant hashing but I'm still very confused about some of your decisions.

First of all, when you increment a key, you mention that you clear the next two buckets in the hash - Why do you do that? Doesn't this loose data?

Second, why do floor((timestamp % bucket span) / bucket interval) I get the timestamp & bucket decides what bucket it should go to, but why divide by the bucket interval.

Third (this is an implementation question) if I want to implement this for an hour would I set Bucket span to 3600? What about subject expiry, should that also be set to an hour? Also, how much does the bucket interval matter in this situation, you wrote that the bigger it is the less space it takes on price of accuracy. but i don't get why...

can we reuse variable this way

Hi,
i got following code, setRedisKeys will be called by a timer every hour, is this ok?

var myhash

function setRedisKeys() {
myhash = redback.createHash(dateString)
}

SLRU/LRU advanced data structures

Pushing this one out of my brain FIFO. Had a think about it the other day and thought it would be a pretty decent idea

InstanceOf comparison for redis client always fails

We create our own redis client so that we can select the DB we'd like to use as well as authorize against it. From some initial investigation it looks like the following check when creating a new redback instance is flawed based on the way Node performs instanceof comparisons.

this.client = arguments[0] instanceof redis.RedisClient

We've found that this only evaluates to true if you create a redis client using the same redis npm module that redback uses. If you create an instance using the npm module installed for the project, then you always get false. I can create a more thorough example if needed.

redback.use() throws Redis connection to 127.0.0.1:[object Object] failed - EAFNOSUPPORT, Address family not supported by protocol family

Hi!

This code:

var redis = require('redis'),
   redback = require('redback'),
   rc = redis.createClient();

console.log(rc instanceof redis.RedisClient); // True
redback.use(rc);

throws this:

Error: Redis connection to 127.0.0.1:[object Object] failed - EAFNOSUPPORT, Address family not supported by protocol family
at Socket.<anonymous> (/usr/local/lib/node_modules/redback/node_modules/redis/index.js:123:28)
at Socket.emit (events.js:64:17)
at Array.<anonymous> (net.js:836:27)
at EventEmitter._tickCallback (node.js:126:26)

It appears to be a problem with the Redback constructor:

this.client = arguments[0] instanceof redis.RedisClient ?
    arguments[0] : redis.createClient.apply(this, arguments);

where arguments[0] instanceof redis.RedisClient does not return true.

This occurs using node_redis 0.6.7, redback 0.2.7 and node 0.4.11.

Cheers,

Adam

list of 'classware'

hey i love this library - really shows off the value and power of Redis as a data modelling library.

It would be great to build up a list of users Redback objects - perhaps a page in the wiki ?

namespace is undefined

var Redback = exports.Redback = function () {
this.client = arguments[0] instanceof redis.RedisClient ?
arguments[0] : redis.createClient.apply(this, arguments);
var options = arguments[1] instanceof Object ?
arguments[1] : arguments[2];
this.namespace = options.namespace || '';
}

Line 47 in Redback.js

options.namespace is undefined and causes a crash.

I solved it temporarily by using

var redis = require('redis').createClient();
var redback = require('redback').use(redis, {namespace: "LOL"});

too old version in npm (0.2.8)

1. Could you publish 0.3.0? Redback looks solid, but it would be nice to have latest updates without cheating with direct links to master.
2. Could you start adding tags to your releases? It would greately simplify understanding, what happens with sources. Now it's difficult to find exact commit, that was published to npm. Here is example https://github.com/nodeca/mincer/tags .

Ratelimit issue

There's cases in which an "expired" bucket may get re-used before it was reset to 0.

This will happen for any client that didn't increment the hitcount for the duration of at least 2 "buckets"
before incrementing a bucket again.

For the default settings (bucket span of 10 minutes) picture a client that sends requests only every 10 minutes. They will end up in the same bucket everytime, which will never be cleared. This bug would affect any client that has pauses for longer than 10 seconds though.

A possible fix (afaik) for this would be to save a string with "timestamp:count" in each bucket instead and use a lua script to increment the bucket instead of using HINCRYBY. Another fix would be to keep the timestamp in a separate field "bucketNumber:timestamp = timestamp", which would still require a lua script for incrementing (because we can't get around checking the timestamp before incrementing).

Edit: A third solution (which is how I'm doing rate limiting right now) is to use individual redis keys for buckets which each have their own TTL. This might not be the optimal solution if you have many active buckets though (I only use 2 with an ever-increasing bucket counter).

link to annotated source does not work

host, port order are wrong according to the code

This works for me:

redback = require('redback').createClient cfg.REDIS.PORT, cfg.REDIS.HOST

However, this throws Error: Redis connection to 6379:localhost failed - connect ENOENT:

redback = require('redback').createClient cfg.REDIS.HOST, cfg.REDIS.PORT

I'm thinking this probably is a result of the fact that the redis client specifies port, host in it's constructor, and you probably pass these values through when constructing a redback client.

Unicode support for search and index

Right now the FullText structure is completely useless for non-english characters due to the assumption such as in the line of code here:

redback/lib/advanced_structures/FullText.js

Line 176 in e0a2453

.replace(/[^a-zA-Z0-9'\s\r\n\t-]/g, '')

Passing non-english characters to extractWords completely remove them (incorrectly) rendering the FullText structure useless for indexing any kind of non-English data.

I have a module for splitting words here: https://github.com/chakrit/node-icu-wordsplit

But it requires the full libicu to be installed separately which may not be ideal for everyone. However, it is still better than using the regexes there.

Not exactly sure about what is the best fix but thought I should raise the issue up here so you guys are aware of them atleast.

Would love to hear your thoughts on this as I'm planning to write a better libicu bindings for node as well in the hope that I get to see this kind of (international) issue fixed for good across the node community.

Ratelimit question

code

var redback = require('redback').createClient();
var ratelimit = redback.createRateLimit('requests', {
    bucket_span: 6,
    bucket_interval: 1,
    subject_expiry: 12
});

// if request 1 times per 3 seconds
setInterval(function () {
    ratelimit.add('127.0.0.1');
}, 3000);

redis monitor

1475139642.585969 [0 127.0.0.1:50241] "multi"
1475139642.585994 [0 127.0.0.1:50241] "hincrby" "requests:127.0.0.1" "0" "1"
1475139642.586005 [0 127.0.0.1:50241] "hdel" "requests:127.0.0.1" "1"
1475139642.586010 [0 127.0.0.1:50241] "hdel" "requests:127.0.0.1" "2"
1475139642.586015 [0 127.0.0.1:50241] "expire" "requests:127.0.0.1" "12"
1475139642.586023 [0 127.0.0.1:50241] "exec"
1475139645.592772 [0 127.0.0.1:50241] "multi"
1475139645.592794 [0 127.0.0.1:50241] "hincrby" "requests:127.0.0.1" "3" "1"
1475139645.592804 [0 127.0.0.1:50241] "hdel" "requests:127.0.0.1" "4"
1475139645.592809 [0 127.0.0.1:50241] "hdel" "requests:127.0.0.1" "5"
1475139645.592814 [0 127.0.0.1:50241] "expire" "requests:127.0.0.1" "12"
1475139645.592822 [0 127.0.0.1:50241] "exec"
1475139648.594619 [0 127.0.0.1:50241] "multi"
1475139648.594651 [0 127.0.0.1:50241] "hincrby" "requests:127.0.0.1" "0" "1"
1475139648.594669 [0 127.0.0.1:50241] "hdel" "requests:127.0.0.1" "1"
1475139648.594677 [0 127.0.0.1:50241] "hdel" "requests:127.0.0.1" "2"
1475139648.594685 [0 127.0.0.1:50241] "expire" "requests:127.0.0.1" "12"
1475139648.594698 [0 127.0.0.1:50241] "exec"
1475139651.597944 [0 127.0.0.1:50241] "multi"
1475139651.597967 [0 127.0.0.1:50241] "hincrby" "requests:127.0.0.1" "3" "1"
1475139651.597978 [0 127.0.0.1:50241] "hdel" "requests:127.0.0.1" "4"
1475139651.597983 [0 127.0.0.1:50241] "hdel" "requests:127.0.0.1" "5"
1475139651.597987 [0 127.0.0.1:50241] "expire" "requests:127.0.0.1" "12"
1475139651.597995 [0 127.0.0.1:50241] "exec"

bucket 0 and 3 will not be deleted.
now I'd like to get the count on bucket 0, it's 2.
actually I just hit 1 times in this second, or in last 2 seconds.

Website --- ?

Is there a website hosting the API documentation? I cannot find a link for any. This would be a nice way to get an overview of the API, without having to clone the library.

Errors in SocialGraph isFollowing and follow methods

Hi,

I'm trying to use the library in a express.js application that I'm building but I keep getting the same errors:
Whenever I try to use the isFollowing method I get the error:

throw e; // process.nextTick error, or 'error' event on first tick
^
Error: Error: ERR wrong number of arguments for 'sismember' command
at HiredisReplyParser. (/.../node_modules/redback/node_modules/redis/index.js:75:27)
at HiredisReplyParser.emit (events.js:64:17)
at HiredisReplyParser.execute (/.../node_modules/redback/node_modules/redis/lib/parser/hiredis.js:33:22)
at RedisClient.on_data (/.../node_modules/redback/node_modules/redis/index.js:325:27)
at Socket. (/.../node_modules/redback/node_modules/redis/index.js:90:14)
at Socket.emit (events.js:64:17)
at Socket._onReadable (net.js:672:14)
at IOWatcher.onReadable as callback

And if I try to call the follow method the following error is showed:
throw e; // process.nextTick error, or 'error' event on first tick
^
TypeError: string is not a function
at String.CALL_NON_FUNCTION (native)
at Object.callback (/.../node_modules/redback/node_modules/redis/index.js:737:13)
at RedisClient.return_reply (/.../node_modules/redback/node_modules/redis/index.js:384:29)
at HiredisReplyParser. (/.../node_modules/redback/node_modules/redis/index.js:78:14)
at HiredisReplyParser.emit (events.js:64:17)
at HiredisReplyParser.execute (/.../node_modules/redback/node_modules/redis/lib/parser/hiredis.js:35:22)
at RedisClient.on_data (/.../node_modules/redback/node_modules/redis/index.js:325:27)
at Socket. (/.../node_modules/redback/node_modules/redis/index.js:90:14)

Source code:

// profile page.
app.get('/profile/:id', requiresLogin, function(req, res){

var logged_user = req.session.user;

User.findById(req.params.id, function(err, user) {
    if (err) {
        console(err);
    }

    var user_social_graph = redback.createSocialGraph(logged_user._id);

    if(logged_user._id == user._id) {
        res.local("my_profile", true);
    } else {
        res.local("my_profile", false);
        //res.local("following", user_social_graph.isFollowing(user._id));
    }
    res.render('profile', {
        locals: {
            user: user
    }});
});

});

// follow user.
app.get('/follow/:id', requiresLogin, function(req, res){

var logged_user = req.session.user;

var social_graph = redback.createSocialGraph(logged_user._id);
social_graph.follow(req.params.id);

});

// unfollow user.
app.get('/unfollow/:id', requiresLogin, function(req, res){

var logged_user = req.session.user;

var social_graph = redback.createSocialGraph(logged_user._id);
social_graph.unfollow(req.params.id);

});

Can you help me? Am I doing something wrong?

Thanks in advance.
Best regards,
Migrate

callbacks

Hi!

mranney's node_redis supports optional callbacks. However, the logic of determining whether the trailing undefined means "no callback", or this undefined means the last argument to a multi-argument redis command, is ambiguous.

Suppose we call List::next(). The resulting command passed to redis client's send_command is .lpop with arguments ['id-of-the-list', undefined]. This undefined is treated as argument to lpop and chokes the client. This pattern is widely used in redback's methods.

I propose to change the pattern to always wrap real command arguments in an array, which is supported by redis client as well and can even be faster (tests?). That way redis client is always given two arguments -- array of args ready to be applied, and truly optional callback.

update: in fact, #2 is an illustration of the issue.

update2: to clarify what i mean -- that's how to easily cope with optional callbacks: https://github.com/muchmala/rq/blob/master/lib/clients/storage.js

TIA,
--Vladimir