loopbackio/loopback-connector-redis

EXPERIMENTAL: Redis connector for LoopBack.

This project provides early access to advanced or experimental functionality. It may lack usability, completeness, documentation, and robustness, and may be outdated.

However, StrongLoop supports this project. Community users, please report bugs on GitHub.

loopback-connector-redis

The official Redis connector for the LoopBack framework.

Usage

Install dependencies

Install the required dependencies via NPM:

npm install --save loopback-connector-redis
npm install --save loopback-datasource-juggler

This connector depends on loopback-datasource-juggler.

Configure a Redis datasource

In your code, declare Redis as a datasource:

var DataSource = require('loopback-datasource-juggler).DataSource;

var ds = new DataSource('redis');
...

Testing

1. Start the Redis server (using the default port)
2. Run npm test

WARNING

Existing data in database 0 may be deleted

loopback-connector-redis's Issues

Null becomes type string after save

'should convert undefined to null on save' test fails

No indexes found for <fieldname> impossible to sort and filter using redis connector

Hi,

I discovered loopback few weeks ago and I find it amazing ! However I need your help to solve my problem on redis connector:
The 'where' filter does not work despite indexing the fields I want to query on .. indeed, when I'm trying to call model.find({where: {field: true}, limit: 10}, function ...) it does not consider my query at all and give me an array of the ten first model insertions ...

However, the same query in the loopback explorer is working well ... even if i'm just a JavaScript beginner I think there's something wrong in the nodeAPI redis adapter

Could someone light my mind about how using redis-adapter to avoid problem ? I know that this project is under development and I accept any thug solution ;)

Thank you to strongloop / loopback team and all contributors for this awesome product !

Where filter not working.

My model:

{
  "name": "Cart",
  "plural": "carts",
  "base": "PersistedModel",
  "idInjection": true,
  "options": {
    "validateUpsert": true
  },
  "mixins": {
    "Timestamp": true
  },
  "properties": {
    "customerId": {
      "type": "string",
      "required": true,
      "index": {
        "unique": true
      } 
    }
  },
  "validations": [],
  "relations": {
    "lines": {
      "type": "hasMany",
      "model": "CartLine",
      "foreignKey": "cartId"
    },
    "customer": {
      "type": "belongsTo",
      "model": "Customer",
      "foreignKey": "customerId"
    }
  },
  "acls": [],
  "methods": {}
}

When doing:

 Cart
 .findOne({
      where: {
           customerId: "58147cab5185d50f002d5a70"
      }
 })

The result is null.

The record exists:

imported.test.js is failing

imported.test.js is failing

Use ESLint instead of JSHint

Configure this project to use ESLint instead of JSHint.

"idInjection": false doesn't work

I am trying set idInjection to false, so that I can set some other field as ID. Below is my
user-votes.json
{ "name": "user-votes-redis", "base": "PersistedModel", "idInjection": false, "options": { "validateUpsert": true }, "properties": { "user-name": { "type": "string", "id": true, "required": true }, "poll-id": { "type": [ "number" ] } }, "validations": [], "relations": {}, "acls": [], "methods": {} }
I get below error when i try to do following operations.

GET: /user-votes

{ "error": { "name": "TypeError", "status": 500, "message": "Cannot read property 'type' of undefined", "stack": "TypeError: Cannot read property 'type' of undefined\n at /Users/akumar/repos/loopback-testing/node_modules/loopback-connector-redis/lib/redis.js:553:19\n at Array.forEach (native)\n at BridgeToRedis.all (/Users/akumar/repos/loopback-testing/node_modules/loopback-connector-redis/lib/redis.js:547:10)\n at /Users/akumar/repos/loopback-testing/node_modules/loopback-datasource-juggler/lib/dao.js:1964:21\n at doNotify (/Users/akumar/repos/loopback-testing/node_modules/loopback-datasource-juggler/lib/observer.js:98:49)\n at doNotify (/Users/akumar/repos/loopback-testing/node_modules/loopback-datasource-juggler/lib/observer.js:98:49)\n at doNotify (/Users/akumar/repos/loopback-testing/node_modules/loopback-datasource-juggler/lib/observer.js:98:49)\n at doNotify (/Users/akumar/repos/loopback-testing/node_modules/loopback-datasource-juggler/lib/observer.js:98:49)\n at Function.ObserverMixin._notifyBaseObservers (/Users/akumar/repos/loopback-testing/node_modules/loopback-datasource-juggler/lib/observer.js:121:5)\n at Function.ObserverMixin.notifyObserversOf (/Users/akumar/repos/loopback-testing/node_modules/loopback-datasource-juggler/lib/observer.js:96:8)" } }

POST:(works)

Seems to be working but can't find the index on redis.

GET :/user-votes/{id}
'{
"error": {
"name": "Error",
"status": 500,
"message": "user-votes-redis: no indexes found for user-name impossible to sort and filter using redis connector",
"stack": "Error: user-votes-redis: no indexes found for user-name impossible to sort and filter using redis connector\n at BridgeToRedis.all (/Users/akumar/repos/loopback-testing/node_modules/loopback-connector-redis/lib/redis.js:510:13)\n at /Users/akumar/repos/loopback-testing/node_modules/loopback-datasource-juggler/lib/dao.js:1964:21\n at doNotify (/Users/akumar/repos/loopback-testing/node_modules/loopback-datasource-juggler/lib/observer.js:98:49)\n at doNotify (/Users/akumar/repos/loopback-testing/node_modules/loopback-datasource-juggler/lib/observer.js:98:49)\n at doNotify (/Users/akumar/repos/loopback-testing/node_modules/loopback-datasource-juggler/lib/observer.js:98:49)\n at doNotify (/Users/akumar/repos/loopback-testing/node_modules/loopback-datasource-juggler/lib/observer.js:98:49)\n at Function.ObserverMixin._notifyBaseObservers (/Users/akumar/repos/loopback-testing/node_modules/loopback-datasource-juggler/lib/observer.js:121:5)\n at Function.ObserverMixin.notifyObserversOf (/Users/akumar/repos/loopback-testing/node_modules/loopback-datasource-juggler/lib/observer.js:96:8)\n at Function.ObserverMixin._notifyBaseObservers (/Users/akumar/repos/loopback-testing/node_modules/loopback-datasource-juggler/lib/observer.js:119:15)\n at Function.ObserverMixin.notifyObserversOf (/Users/akumar/repos/loopback-testing/node_modules/loopback-datasource-juggler/lib/observer.js:96:8)"
}
}'

`database` will cause error

i create a config file datasources.local.js

module.exports = {
  redis: {
    connector: "redis",
    database: "redisdb",
    host: process.env.REDIS_PORT_6379_TCP_ADDR,
    port: process.env.REDIS_PORT_6379_TCP_PORT
  }
}

then i got a error:

/app/user$ node .
Web server listening at: http://0.0.0.0:3000
Browse your REST API at http://0.0.0.0:3000/explorer
Connection fails:  Error: ERR invalid DB index
    at Error (native)
    at HiredisReplyParser.execute (/app/user/node_modules/loopback-connector-redis/node_modules/redis/lib/parser/hiredis.js:30:33)
    at RedisClient.on_data (/app/user/node_modules/loopback-connector-redis/node_modules/redis/index.js:547:27)
    at Socket.<anonymous> (/app/user/node_modules/loopback-connector-redis/node_modules/redis/index.js:102:14)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at readableAddChunk (_stream_readable.js:172:18)
    at Socket.Readable.push (_stream_readable.js:130:10)
    at TCP.onread (net.js:542:20)
It will be retried for the next request.
/app/user/node_modules/loopback-connector-redis/node_modules/redis/index.js:575
                throw callback_err;
                ^

Error: ERR invalid DB index
    at Error (native)
    at HiredisReplyParser.execute (/app/user/node_modules/loopback-connector-redis/node_modules/redis/lib/parser/hiredis.js:30:33)
    at RedisClient.on_data (/app/user/node_modules/loopback-connector-redis/node_modules/redis/index.js:547:27)
    at Socket.<anonymous> (/app/user/node_modules/loopback-connector-redis/node_modules/redis/index.js:102:14)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at readableAddChunk (_stream_readable.js:172:18)
    at Socket.Readable.push (_stream_readable.js:130:10)
    at TCP.onread (net.js:542:20)

then i remove database key, the error gone.

The function destroyAll fails with error

When running destroyAll this error is returned:

     TypeError: Cannot read property 'length' of undefined
      at BridgeToRedis.all (lib/redis.js:513:18)
      at BridgeToRedis.destroyAll (lib/redis.js:631:10)

Broken CLA Link

Bug or feature request

• Bug
• Feature request

Description of feature (or steps to reproduce if bug)

The CLA link in CONTRIBUTING.md is broken.

Link to sample repo to reproduce issue (if bug)

N/A

Expected result

The CLA link should points to a valid web page.

Actual result (if bug)

The CLA link points to an non-existent web page:

Additional information (Node.js version, LoopBack version, etc)

Support ioredis

The ioredis is more full featured redis client with support of cluster, sentinel, etc. Is there any plan to support ioredis as the underlying redis client?

strong-globalize can't process error object

Bug or feature request

• Bug
• Feature request

Description of feature (or steps to reproduce if bug)

1. Create a LB app
2. Add redis datasource following steps outlined in https://loopback.io/doc/en/lb2/Redis-connector.html
3. Change the port to something that is not right for the datasource

Expected result

graceful error like how loopback-connector-mysqlconnector does it:

| => node .
Web server listening at: http://0.0.0.0:3000
Browse your REST API at http://0.0.0.0:3000/explorer
Connection fails: Error: connect ECONNREFUSED 0.0.0.0:3307
It will be retried for the next request.
events.js:141
      throw er; // Unhandled 'error' event
      ^

Error: connect ECONNREFUSED 0.0.0.0:3307
    at Object.exports._errnoException (util.js:907:11)
    at exports._exceptionWithHostPort (util.js:930:20)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1078:14)
    --------------------

Actual result (if bug)

 => node .
Web server listening at: http://0.0.0.0:3000
Browse your REST API at http://0.0.0.0:3000/explorer
/Users/badmike/mysql65repo/redis38/node_modules/strong-globalize/lib/helper.js:740
      matched = (name.indexOf(header) === 0);
                      ^

TypeError: name.indexOf is not a function
    at /Users/badmike/mysql65repo/redis38/node_modules/strong-globalize/lib/helper.js:740:23
    at Array.forEach (native)
    at headerIncluded (/Users/badmike/mysql65repo/redis38/node_modules/strong-globalize/lib/helper.js:738:20)
    at Object.hashKeys (/Users/badmike/mysql65repo/redis38/node_modules/strong-globalize/lib/helper.js:109:12)
    at formatMessage (/Users/badmike/mysql65repo/redis38/node_modules/strong-globalize/lib/globalize.js:96:14)
    at packMessage (/Users/badmike/mysql65repo/redis38/node_modules/strong-globalize/lib/globalize.js:201:17)
    at Object.rfc5424 (/Users/badmike/mysql65repo/redis38/node_modules/strong-globalize/lib/globalize.js:224:10)
    at StrongGlobalize.log (/Users/badmike/mysql65repo/redis38/node_modules/strong-globalize/index.js:185:20)
    at RedisClient.<anonymous> (/Users/badmike/mysql65repo/redis38/node_modules/loopback-connector-redis/lib/redis.js:56:7)
    at emitOne (events.js:77:13)
    at RedisClient.emit (events.js:169:7)
    at RedisClient.on_error (/Users/badmike/mysql65repo/redis38/node_modules/redis/index.js:196:10)
    at Socket.<anonymous> (/Users/badmike/mysql65repo/redis38/node_modules/redis/index.js:106:14)
    at emitOne (events.js:77:13)
    at Socket.emit (events.js:169:7)
    at emitErrorNT (net.js:1269:8)

Additional information (Node.js version, LoopBack version, etc)

| => lb --version
2.2.0 ([email protected] [email protected])
___________________    | ~/mysql65repo/redis38 @ biniams-mbp (badmike)
| => node -v
v4.8.0
___________________    | ~/mysql65repo/redis38 @ biniams-mbp (badmike)
| => npm -v
3.10.10

The issue happens because https://github.com/strongloop/strong-globalize/blob/master/lib/helper.js#L730-#L735 expects name to be a string, but our downstream dep node_redis sends an error object along https://github.com/NodeRedis/node_redis/blob/v0.12.1/index.js#L196 (btw we should look into updating that dep because it is currently on 2.7.1).

destroyById crashes the app when model with the id does not exist

Whenever I try to use destroyById with an ID that does not exist, Redis connector throws an error, which ultimately crashes the application. There is an error event handler inside the connector but that just generates another error, which is not handled.

I debugged the issue a bit and the error happens after the following line on redis.js returns [null] array:
return handleKeys(null, [model + ':' + filter.where.id]);

Later, the returned array is iterated and since result is null, an error is generated :

      results.forEach(function(result) {
        tasks.push(function(done) { br.destroy(model, result.id, done); });
      });

[TypeError: Cannot read property 'id' of null] 'error@context': {}

Later, this is what crashes the app:

TypeError: name.indexOf is not a function
at C:\source\x\node_modules\loopback-connector-redis\node_modules\strong-globalize\lib\helper.js:739:23

Lost relation between AccessToken and User when using AccessToken stored in redis

Description

Can't use AccessToken Model stored in redis datasource.
Actually, we can store the access_token datas in database, but it loses the relation to User model.

Lines cause the problem (bug)

In: https://github.com/strongloop/loopback/blob/master/common/models/access-token.js
Lines: 242 to 249.

In line: User = AccessToken.registry.findModel(this.principalType);

I found that User is undefined somehow.

Additional information

Version:
loopback 3.14.0

Other questions:

• what are the differences between this repo and: https://github.com/strongloop/loopback-connector-kv-redis ?
• Is this repo still maintained or not?
  Thank you.

Upserting an existing records returns all fields as strings

It will only display itself when you use the upsert route while specifying an ID:

This will not do it:

PUT /hvacs { installed: true, enabled: false}
---
{
  "installed": true,
  "enabled": false
}

But when you pass the ID to modify an existing:

PUT /hvacs { id: 1, installed: true, enabled: false}
---
{
  "installed": "true",
  "enabled": "false"
}

Even if the model doesn't already exist:

PUT /hvacs { id: 999, installed: true}
{
  "installed": "true",
  "id": 999
}

How to set Redis database number?

Out of the box, a Redis instance supports 16 logical databases. These databases are effectively siloed off from one another, and when you run a command in one database it doesn’t affect any of the data stored in other databases in your Redis instance. Redis databases are numbered from 0 to 15 and, by default, you connect to database 0 when you connect to your Redis instance. However, you can change the database you’re using with the select command after you connect:
127.0.0.1:6379 > select 15
127.0.0.1:6379[15] > ...

So my question is next: how can I set this db number for my redis instance in loopback? I saw that in redis connector config there is property called 'database' but I am not sure if it's the property I am looking for

Bad value stored for a foreign key that is a string.

Bug or feature request

• Bug
• Feature request

Description of feature (or steps to reproduce if bug)

In the linked repository there are two models: Profile and Activity. Profiles are stored in ElasticSearch, activities are stored in Redis. A profile has many activities, therefore Loopback automatically injects the property profile_id into the class Activity.

To reproduce the bug follow the steps:

1. Start the server and open the API explorer
2. Create a new profile, you get a string identifier
3. Use the profile ID to create a new Activity, the response should be:

{
  "id": "UUID",
  "name": "name",
  "profile_id": "AV3GqNDiTUO_Ow-hB5vl"
}

4. Getting all the activities you have:

[
  {
    "id": "UUID",
    "name": "name",
    "profile_id": "\"AV3GqNDiTUO_Ow-hB5vl\""
  }
]

5. Getting all the activities through the nested route GET /profiles/{id}/activities the result is an empty array.

Link to sample repo to reproduce issue (if bug)

https://github.com/TwistedLogic/loopback-connector-redis-issue

Expected result

The route GET /profiles/{id}/activities should return all the activities related to the profile identified by {id}

Actual result (if bug)

An empty array is returned by the API

Additional information (Node.js version, LoopBack version, etc)

The profile_id value in the keystore is "\"AV3GqNDiTUO_Ow-hB5vl\"" and not "AV3GqNDiTUO_Ow-hB5vl".

The problem is in the function BridgeToRedis.prototype.forDb, lines 198-200 in current master:

if (!p[i]) {
    data[i] = JSON.stringify(data[i]);
    continue;
}

JSON.stringify("AV3GqNDiTUO_Ow-hB5vl") === "\"AV3GqNDiTUO_Ow-hB5vl\""

I suppose that the connector doesn't consider the magic property profile_id that loopback automatically injects into the model.

Adding the foreign key to the Activity model like so:

"profile_id": {
  "type": "string",
  "index": true,
  "required": true
}

makes everything work fine, but I believe this is just a workaround.

crashes the application on connection-errors

Bug or feature request

• Bug

Description steps to reproduce bug

Use redis as datasource for a model and set an invalid hostname for redis (thereby simulating a non-reachable redis instance).

Expected result

error-message in console

Actual result (if bug)

application crashes:

Error: Redis connection to redis:6379 failed - getaddrinfo EAI_AGAIN redis:6379
    at RedisClient.flush_and_error (/app/node_modules/redis/index.js:149:13)
    at RedisClient.on_error (/app/node_modules/redis/index.js:191:10)
    at Socket.<anonymous> (/app/node_modules/redis/index.js:106:14)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at connectErrorNT (net.js:1022:8)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickDomainCallback (internal/process/next_tick.js:122:9)
/app/node_modules/strong-globalize/lib/helper.js:740
      matched = (name.indexOf(header) === 0);
                      ^

TypeError: name.indexOf is not a function
    at /app/node_modules/strong-globalize/lib/helper.js:740:23
    at Array.forEach (native)
    at headerIncluded (/app/node_modules/strong-globalize/lib/helper.js:738:20)
    at Object.hashKeys (/app/node_modules/strong-globalize/lib/helper.js:109:12)
    at formatMessage (/app/node_modules/strong-globalize/lib/globalize.js:96:14)
    at packMessage (/app/node_modules/strong-globalize/lib/globalize.js:201:17)
    at Object.rfc5424 (/app/node_modules/strong-globalize/lib/globalize.js:224:10)
    at StrongGlobalize.log (/app/node_modules/strong-globalize/index.js:185:20)
    at RedisClient.<anonymous> (/app/node_modules/loopback-connector-redis/lib/redis.js:56:7)
    at emitOne (events.js:96:13)
    at RedisClient.emit (events.js:188:7)
    at RedisClient.on_error (/app/node_modules/redis/index.js:196:10)
    at Socket.<anonymous> (/app/node_modules/redis/index.js:106:14)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at connectErrorNT (net.js:1022:8)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickDomainCallback (internal/process/next_tick.js:122:9)

Additional information (Node.js version, LoopBack version, etc)

Node.js 6.9.2
Loopback 2.22.0
loopback-connector-redis 0.1.0

Docs

Hi,

I really could not get how this connector works. There istn a doc explaining how to use or in tests the example is not something that can be used in the real world.

The thing is, in redis we need to have a key to find or store a value. So it should have something like:

• RedisModel.find("mykeyname") and not the id of the saved model.

If we take the tests it saves the key and value in same string, and to find this results it uses a generated ID

• https://github.com/strongloop/loopback-connector-redis/blob/master/test/json-parsing.test.js#L13

If I use a filter like the example, It return a empty result.

• RedisModel.find( {where: {name: key}},function(val){

I could see that it is saving in redis:

1. "id:RedisModel"
2. "i:RedisModel:name:veiculosmotos"
3. "RedisModel:2"
4. "RedisModel:1"
5. "RedisModel:4"
6. "RedisModel:3"
7. "s:RedisModel"

Error: ERR wrong number of arguments for HMSET

Bug or feature request

• [x ] Bug
• Feature request

Description of feature (or steps to reproduce if bug)

redis package is old '0.12.1' which has bug related to HMSET
Bug Mentioned : Fix multi.hmset key not being type converted if used with an object and key not being a string on https://github.com/NodeRedis/node_redis/releases/tag/v.2.1.0

Link to sample repo to reproduce issue (if bug)

Latest Version and even the older version will have issue since redis dependencies is on older version i.e. 0.12.1

Expected result

No Error when using redis

Actual result (if bug)

When running application with SLC package manager error is thrown when using redis.
Error: ERR wrong number of arguments for HMSET when

Additional information (Node.js version, LoopBack version, etc)

Node - Any version
SLC - v5.0.1 / v6.0.3
NPM - 4.3.0 / 2.15.11 / 1.4.29

Re-enable test cases

Re-enable test cases and fix them to get CI pass

"TypeError: Cannot read property 'package' of undefined" while adding datasource

Hi there,

I'm new to LoopbackJS but already a fan, and I'm trying to use Redis as a datasource.
I successfully installed Redis and loopback-connector-redis, however when I try to add it as a datasource (slc loopback:datasource) I run into the following error:

? Enter the data-source name: redisDB
? Select the connector for redis: other
? Enter the connector name without the loopback-connector- prefix: redis
events.js:154
throw er; // Unhandled 'error' event
^

TypeError: Cannot read property 'package' of undefined
at module.exports.yeoman.generators.Base.extend.installConnector (/usr/local/lib/node_modules/strongloop/node_modules/generator-loopback/datasource/index.js:202:24)
at /usr/local/lib/node_modules/strongloop/node_modules/yeoman-generator/lib/base.js:430:16
at processImmediate as _immediateCallback

Am I missing something?

loopback-connector-redis and include filter

I'm trying to create a hasOne relation between a mysql table with a model that uses a redis connector.

When you are querying only 1 element it works but when you query several elements, the redis indexes doesnt work properly and then return only the result of the first table without the relation.

Thank you for this awsome product.

edit: i fixed it creating the other way round relation. now I have a belongsTo in the redis side and this works.

edit2: im not 100% sure but its something with the indexes. if you cant recreate this i will try to be more specific.

edit 3: my version is 0.0.3 that is what is in npm.

Is the transactional feature implemented ?

Hello,

hello, does this connector include the transactional feature of Redis ?

Thanks in advance