Add redis adapter

[rom-rb]

Add redis adapter

[dreoliv]

I was looking into this issue, and it is showing itself more challenging than I thought. So I thought I should probably ask some clarification first 😄.

I have used Ohm as a Redis ORM and redis-rb to perform atomic operations. I don't know and haven't found any libraries that provide a query interface for storing/retrieving records from redis similar to Squeel or Moped. Are you aware of any? If not, I'll probably strip down Ohm and write one myself based on Ohm's code.

[solnic]

There's no need to write something that's similar to Sequel. You just use redis-rb directly (take a look at axiom-redis-adapter for some inspiration.

The adapter could provide a convenient query DSL if you want or you could built the adapter on top of Ohm. You are free to do whatever you want. ROM's relations have access to whatever interface the adapter exposes. The only interface that's required is #each and yielding hash-like objects on the dataset object that's passed into ROM::Relation + it must support chaining like Sequel::Dataset which means that every operation must return an instance of the same kind, that's why for mongo adapter I had to create a custom dataset class.

My plan was to simply port axiom-redis-adapter to rom-redis-adapter for a good start and take it from there.

[dreoliv]

Thanks for the suggestion. I've been experimenting with Ohm, but since it is designed to be an ORM in itself, I kept running into awkward situations that required workarounds I was not happy with.

axiom-redis-adapter seems simple enough, I'll go with that 😄

[solnic]

Yes, using Ohm would be like using ActiveRecord behind a ROM adapter which would be awkward. Maybe there's a piece in Ohm that you could use, I'm not sure, something that could be extracted from Ohm. Maybe we should talk to @soveran :)

[dreoliv]

I looked into it, Ohm is fairly simple (couldn't expect anything different from @soveran). My initial thought was to use Ohms code, removing the object mapping and adding a record-query interface. That would allow us to use it like a hash store, with all the mature indexing/searching features Ohm provides.

[solnic]

This sounds like a good plan

[soveran]

Hello guys, I don't know exactly how an adapter should be in order to work with ROM, so what I'm writing is based on what I saw in axiom-redis-adapter. That said, I'm not even familiar with axiom, so take my comments with a hundred grains of salt.

I think the approach in axiom-redis-adapter can be improved, but it's not trivial. It uses the KEYS command for building relations, but that command is pretty much forbidden for production environments. The recommended way to achieve that would be by using sets to build indices, and above all any update or delete should update both the records and the indices atomically. If the plan is to start from that library and build a ROM adapter, I would strongly suggest keeping those details in mind. Otherwise, the user will find out about his performance issues when it's too uncomfortable to change (that is, when the database grows to the point that executing KEYS creates a peformance penalty that causes other connections to timeout, or just to run very slowly).

Something I'm thinking could be of use is the Lua scripts used by Ohm in order to save and delete records. There are some Ohm ports that use those scripts, they have proved useful for making cross language compatible clients. You can find them here, and you can see how they are used in Ohm itself, but also in other clients like Gohm (Go), redis3m (C++), or Ohm.lua (Lua).

Even though it looks like Ohm does a lot, it's very small (600 loc) and straightforward because of the extraction of those two operations. If you guys want to use them, or if you want more information about how that works, just let me know. I'm not sure if the documentation is clear enough, so if there's something missing I will be glad to update it. Even if you don't find a good use for those scripts or for Ohm itself in this context, I would still like to help you with this endeavor, so please feel free to ask me.

Looking a bit more into the problem, I think Ohm could be used directly, as it provides easy access to low level features. I guess we can discuss that too if you guys agree.

[solnic]

@soveran thanks! This is a lot of useful information.

ROM will work with anything that provides an array of hash-like objects. That's the only true requirement. Everything else is just extra db-specific functionality. Maybe just using Ohm under the hood is a good idea as long as we can use it's lower level features and skip the actual hash->object mapping as that part is handled by ROM. This is pretty much how Sequel is used in ROM - its adapter exposes Sequel::Dataset interface and you do whatever you want with it.

One thing I need to make very clear is that ROM does not provide any abstract query interface, that's why writing adapters is very very simple. ROM relations give you access to the interface provided by the underlaying db library. I would expect this library to be redis-rb in case of redis adapter, but maybe it's too low level, I really don't know yet. An adapter can provide a decorator object that adds more convenient ways of working with a given database though but it will always be adapter-specific and there will never be any common abstract interface that every adapter must implement.

[dreoliv]

Thanks @soveran for taking the time to respond and for being so thorough!

I'll take a proper look at Ohm's source tonight and see if I can use the low-level features as @solnic mentioned.

[bryanp]

Hello, all! I'd like to jump in and help write a Redis adapter for ROM. It ties in with some things I've been working on, and while Ohm is fantastic I'd like to be able to use Redis via ROM.

I've reviewed the existing adapters (primarily rom-sql) and have a decent handle on how to go about building a rom-redis adapter based on the information in this thread. I'd like to start down the path of building the adapter on redis-rb.

Anything to discuss before I start slinging code?

[solnic]

@bryanp hey! that's awesome news. I'm a total redis n00b so I can only help you with ROM-specific stuff.

The best starting point is to make it possible to configure a connection with default options and come up with a basic dataset class. If it can be an array-like object you can use a helper module like described in #86. If you can come up with some interface that connects to redis and returns dataset objects then you've got a working adapter done. The rest is just expanding the interface with features (like some query DSL, command objects etc.).

Please also remember that we are very open to improve adapter interface because the goal is to be able to really easily build any adapter. If you see something fishy please tell about it. If something feels too hard, tell about it too. We already managed to build basic functionality for sql, mongo and file-based (as we call them) adapters like csv and yaml. Redis is super important as it's a key-value store which is a different type of a database.

I'm hanging out on gitter on a daily basis so ping me if you need some help or just report issues if something is not working for you.

[bryanp]

We'll make a good team then! I'm a ROM noob but more experienced in Redis :)

Will do some preliminary work in the next few days and shoot you a note when there's something to look at.

[solnic]

Sounds like a good plan :)

[WattsInABox]

Lemme know if you need any help on this, @bryanp or whoever else is working on this. I'm pretty experienced with Redis and I have a huge desire to get it put into ROM.

[bryanp]

Hi @BillyWatson that would be awesome, actually. Haven't had a chance to do much the last little bit. I certainly want to help, but also don't want to be a blocker for anyone else.

[solnic]

@BillyWatson @bryanp would it be helpful if I started the project with some basic structure so that we have a redis connection setup in place?

[WattsInABox]

Sure, @solnic that'd be a huge help to a guy fairly new to ROM, like myself

[bryanp]

@solnic That'd be great!

[solnic]

OK I'll try to kick it off tomorrow :)

[solnic]

@billdueber @bryanp I just pushed rom-rb/rom-redis to github with some super basic stuff working. I've no idea what the strategy should be so I just did the simplest thing that could possibly work for a good start :)

oh and sorry it took so long...

@soveran I would appreciate if you could take a look too :)

[WattsInABox]

Awesome, thanks!

[bryanp]

@BillyWatson have you started on an aspect of this? would be good to coordinate somehow.

and thanks @solnic!

[WattsInABox]

I haven't started yet, no. I've been trying to finish the project that needs this without it so that I could see where it would fit in. Did you have a place to start?

[bryanp]

Same here, actually. I was looking to drive it based on needed features, which to start with would just be basic CRUD operations. That would likely descend into more complex queries.

[WattsInABox]

Well it looks like from my previous research that the CUD portions aren't as well defined in ROM yet so I was planning on just sticking with implementing good read interfaces for now. What do you think?

[solnic]

CUD is handled by commands and the interface is very simple, every adapter can provide Commands namespace with Create, Update and Delete classes that inherit from ROM::Commands::* base classes. You can take a look at memory adapter, this is really simple stuff. The only interface that's required is #execute which receives various args depending on the type of command.

[WattsInABox]

I guess I misspoke. "Well-defined" clearly isn't right, but some of the docs said that CUD is "in-flux" or something of the sort. Is that not correct?

[solnic]

It is getting more stable. Adding more adapters contributes to this process that's why it would be fantastic to get it working for redis too :)

On 23 Feb 2015, at 21:47, Billy Watson [email protected] wrote:

I guess I misspoke. "Well-defined" clearly isn't right, but some of the docs said that CUD is "in-flux" or something of the sort. Is that not correct?

—
Reply to this email directly or view it on GitHub.

[bryanp]

Cool; I'll start with CR. Getting really close to needing this so I should start in the next couple days.

[gotar]

Redis adapter exists: https://github.com/rom-rb/rom-redis this issue should still be open?

[solnic]

Yes because this is still help-wanted :)

[artemeff]

@solnic hi! In current version you're implement storing data in JSON, but what if we will be store it as is? Just like array of values – user can choose right structure and we can respond it's relation as collection of simple values.

And I can take this adapter, btw :)

[solnic]

@artemeff hey :) we should support any format of course, I just started with JSON for a simple PoC version. I'm not a redis user so I don't know how this adapter should work, unfortunately :/