Giter Site home page Giter Site logo

moneta's Introduction

Moneta: A unified interface for key/value stores

Gem Version Build Status Code Climate Flattr this git repo

Moneta provides a standard interface for interacting with various kinds of key/value stores. Moneta supports the well-known NoSQL and document based stores.

A short overview of the features:

  • Supports a lot of backends with consistent behaviour (See below)
  • Allows a full configuration of the serialization -> compression -> adapter stack using proxies (Similar to Rack middlewares)
    • Configurable serialization via Moneta::Transformer proxy (Marshal/JSON/YAML and many more)
    • Configurable value compression via Moneta::Transformer proxy (Zlib, Snappy, LZMA, ...)
    • Configurable key transformation via Moneta::Transformer proxy
  • Expiration for all stores (Added via proxy Moneta::Expires if not supported natively)
  • Atomic operations
    • Atomic incrementation and decrementation for most stores (Method #increment and #decrement)
    • Atomic creation of entries (Method #create)
    • Shared/distributed database-wide synchronization primitives Moneta::Mutex and Moneta::Semaphore
  • Includes a simple pure-ruby key/value server (Moneta::Server) and client (Moneta::Adapters::Client)
  • Integration with Rails, Rack/Rack-Cache, Sinatra, Padrino and Ramaze.

If you are not yet convinced, you might ask why? What are the goals of the project?

  • Get people started quickly with key/value stores! Therefore all the adapters are included in the gem and you are ready to go. Tilt does the same for template languages.
  • Make it easy to compare different key/value stores and benchmark them
  • To hide a lot of different and maybe complex APIs behind one well-designed and simple Moneta API
  • Give people a starting point or example code to start working with their favourite key/value store. Feel free to copy code, please mention Moneta then :)
  • Create a reusable piece of code, since similar things are solved over and over again (Rails brings its own cache stores, and many frameworks do the same...)

Moneta is tested thoroughly using GitHub Actions.

Getting started

Install Moneta via Rubygems

$ gem install moneta

or add it to your Gemfile

gem 'moneta'

Now you are ready to go:

require 'moneta'

# Create a simple file store
store = Moneta.new(:File, dir: 'moneta')

# Store some entries
store['key'] = 'value'

# Read entry
store.key?('key') # returns true
store['key'] # returns 'value'

store.close

Links

In case you are wondering, Moneta uses Semantic Versioning since v1.0.0.

Supported backends

Out of the box, it supports the following backends. Use the backend name symbol in the Moneta constructor (e.g. Moneta.new(:Memory)).

Some of the backends are not exactly based on key/value stores, e.g. the relational ones. These are useful if you already use the corresponding backend in your application. You get a key/value store for free then without installing any additional services and you still have the possibility to upgrade to a real key/value store.

Backend feature matrix

NOTE: The backend matrix is much more readable on rubydoc.info than on github. Go there!

AdapterRequired gemsMRI support1JRuby support1Multi-thread safe2Multi-process safe3Atomic increment4Atomic create5Native expires6PersistentKey TraversalBulk read7Bulk write8Description
Persistent stores
Mongomongo✓✓✓✓✓✓✓✓✓✓✓MongoDB database
Redisredis✓✓✓✓✓✓✓✓✓✓✓Redis database
ActiveRecordactiverecord✓✓✓✓✓✓✗✓✓✓✓ActiveRecord ORM
File-✓✓✓✓✓✓✗✓✓✗✗File store
LMDBlmdb✓✗✓✓✓✓✗✓✓✓✓Symas Lightning Memory-Mapped Database (LMDB)
Sequelsequel✓✓✓✓✓✓✗✓✓✓✓Sequel ORM
TokyoTyranttokyotyrant or ruby-tokyotyrant✓✓✗✓✓✓✗✓✗✓✗TokyoTyrant database
PStore-✓✗✗✓9✓✓✗✓✓✓✓PStore store
YAML-✓✓✗✓9✓✓✗✓✓✗✗YAML store
Sqlitesqlite3✓✗?✓9✓✓✗✓✓✓✓Sqlite3 database
Daybreakdaybreak✓✓✗✓✓✓✗✓✓✗✓Incredibly fast pure-ruby key/value store Daybreak
DBM-✓✗✗✗✓✓✗✓✓✓✓Berkeley DB using DBM interface or NDBM (Depends on Ruby environment)
GDBMffi-gdbm on JRuby✓✓✗✗✓✓✗✓✓✓✓GDBM database
LevelDBleveldb✓✗✗✗✓✓✗✓✓✓✓LevelDB database
SDBM-✓✗✗✗✓✓✗✓✓✓✓SDBM database
TDBtdb✓✗✗✗✓✓✗✓✓✗✗TDB database
KyotoCabinetkyotocabinet-ruby or kyotocabinet-ruby-reanimated✓✗✗✗✓✓✗✓✓✓✓KyotoCabinet database
TokyoCabinettokyocabinet✓✗✗✗✓✓✗✓✓✗✗TokyoCabinet database
DataMapperdm-core, dm-migrations✓✗✓✓✗✓✗✓✗✗✗DataMapper ORM
Couchfaraday, multi_json✓✓✗✓✗✓✗✓✓✓✓CouchDB database
HBasehbaserb✗✗?✓✓✗✗✓✗✗✗HBase database
Cassandracassandra✓✓?✓✗✗✓✓✓✓✓Cassandra distributed database
LocalMemCachelocalmemcache✓✗✓✓✗✗✗✓✗✗✗LocalMemCache database
Fogfog✓✓?✓✗✗✗✓✗✗✗Fog cloud store
Riakriak-client✗✗✗✓✗✗✗✓✗✗✗Riak database
Non-persistent stores
MemcachedDallidalli✓✓✓✓✓✓✓✗10✗✓✓Memcached database with Dalli library
Memcacheddalli or memcached✓?11?11✓✓✓✓✗10✗?11?11Memcached database
MemcachedNativememcached✓✗✗✓✓✓✓✗10✗✗✗Memcached database with native library
Cookie-✓✓✗✓12✓✓✓✗✓✗✗Cookie in memory store
LRUHash-✓✓✗✓12✓✓✗✗✓✗✗LRU memory store
Memory-✓✓✗✓12✓✓✗✗✓✓✓Memory store
Null-✓✓✓✓✗✗✗✗✗✗✗No database
Network clients
Client-✓✓✗✓?13?13?13?13?13✗✗Moneta client adapter
RestClient-✓✓✗✓✗✗✗?13✗✗✗Moneta REST client adapter
  1. Indicates that the adapter is expected to work on this platform. Most adapters will at least work on MRI, but some are curently considered unstable, in which case they are not supported on any platform.
  2. Make adapters thread-safe by using Moneta::Lock or by passing the option threadsafe: true to Moneta#new. There is also Moneta::Pool which can be used to share a store between multiple threads if the store is multi-process safe. I recommend to add the option :threadsafe to ensure thread-safety since for example under JRuby and Rubinius even the basic datastructures are not thread safe due to the lack of a global interpreter lock (GIL). This differs from MRI where some adapters might appear thread safe already but only due to the GIL.
  3. Share a Moneta store between multiple processes using Moneta::Shared (See below).
  4. If a store provides atomic increment it can be used with Moneta::Semaphore. You can add weak #increment support using the Moneta::WeakIncrement proxy.
  5. If a store provides atomic creation it can be used with Moneta::Mutex. You can add weak #create support using the Moneta::WeakCreate proxy.
  6. Add expiration support by using Moneta::Expires or by passing the option expires: true to Moneta#new.
  7. This indicates that there is some performance gain when fetching multiple values at once using #values_at/#fetch_values or #slice. For instance, the MGET instruction in Redis, or the ability to retrieve several rows in one query in SQL.
  8. This indicates that there is some performance gain when storing multiple key/value pairs at once using #merge!/#update.
  9. Sqlite/YAML/PStore are multiprocess safe, but the performance suffers badly since the whole database file must be locked for writing. Use a key/value server if you want multiprocess concurrency!
  10. There are some servers which use the memcached protocol but which are persistent (e.g. MemcacheDB, Kai, IronCache, Roma, Flare and Kumofs)
  11. This feature is only available if the dalli backend is selected
  12. Store is multi-process safe because it is an in-memory store, values are not shared between multiple processes
  13. Depends on server

Proxies

In addition it supports proxies (Similar to Rack middlewares) which add additional features to storage backends:

  • Moneta::Proxy and Moneta::Wrapper are the proxy base classes.
  • Moneta::Cache combine two stores, one as backend and one as cache (e.g. Moneta::Adapters::File + Moneta::Adapters::LRUHash). Add it in the builder using use(:Cache) {}.
  • Moneta::Expires to add expiration support to stores which don't support it natively. Add it in the builder using use :Expires.
  • Moneta::Fallback use a store as a fallback when exceptions occur (by default the :Null adapter is used so that an error results in a no-op). Add it to the builder using use(:Fallback, rescue: IOError)
  • Moneta::Lock to make store thread safe. Add it in the builder using use :Lock.
  • Moneta::Logger to log database accesses. Add it in the builder using use :Logger.
  • Moneta::Pool to create a pool of stores as a means of making the store thread safe. Add it in the builder using use(:Pool, min: 2, max: 4, ttl: 60, timeout: 5) {}.
  • Moneta::Shared to share a store between multiple processes. Add it in the builder using use(:Shared) {}.
  • Moneta::Stack to stack multiple stores (Read returns result from first where the key is found, writes go to all stores). Add it in the builder using use(:Stack) {}.
  • Moneta::Transformer transforms keys and values (Marshal, YAML, JSON, Base64, MD5, ...). Add it in the builder using use :Transformer.
  • Moneta::WeakIncrement and Moneta::WeakCreate to add #create and #increment support without atomicity (weak) to stores which don't support it.
  • Moneta::WeakEachKey to add key traversal to stores that don't support it, with the important caveat that only those keys previously seen by this proxy will be traversed.

Check the YARD documentation for more information and examples.

Serializers and compressors (Moneta::Transformer)

Supported serializers:

  • BEncode (:bencode)
  • BERT (:bert)
  • BSON (:bson)
  • JSON (:json)
  • Marshal (:marshal)
  • MessagePack (:msgpack)
  • Ox (:ox)
  • PHP (:php)
  • TNetStrings (:tnet)
  • YAML (:yaml)

Supported value compressors:

  • Bzip2 (:bzip2)
  • LZ4 (:lz4)
  • LZMA (:lzma)
  • LZO (:lzo)
  • Snappy (:snappy)
  • QuickLZ (:quicklz)
  • Zlib (:zlib)

Supported encoders:

  • Base64 (RFC 2045; :base64)
  • URL-safe Base64 (RFC 4648; :urlsafe_base64)
  • Url escape (:escape)
  • Hexadecimal (:hex)
  • QP (:qp)
  • UUEncode (:uuencode)

Special transformers:

  • Digests (MD5, Shas, CityHash, ...)
  • Add prefix to keys (:prefix)
  • HMAC to verify values (:hmac, useful for Rack::MonetaCookies)

Moneta API

The Moneta API is purposely extremely similar to the Hash API with a few minor additions. Every method takes also a optional option hash. In order so support an identical API across stores, Moneta does not support partial matches.

#initialize(options)                      options differs per-store, and is used to set up the store.

#[](key)                                  retrieve a key. If the key is not available, return nil.

#load(key, options = {})                  retrieve a key. If the key is not available, return nil.

#fetch(key, options = {}, &block)         retrieve a key. If the key is not available, execute the
                                          block and return its return value.

#fetch(key, value, options = {})          retrieve a key. If the key is not available, return the value,

#[]=(key, value)                          set a value for a key. If the key is already used, clobber it.
                                          keys set using []= will never expire.

#store(key, value, options = {})          same as []=, but you can supply options.

#delete(key, options = {})                delete the key from the store and return the current value.

#key?(key, options = {})                  true if the key exists, false if it does not.

#increment(key, amount = 1, options = {}) increment numeric value. This is an atomic operation
                                          which is not supported by all stores. Returns current value.

#decrement(key, amount = 1, options = {}) increment numeric value. This is an atomic operation
                                          which is not supported by all stores. Returns current value.
                                          This is just syntactic sugar for incrementing with a negative value.

#create(key, value, options = {})         create entry. This is an atomic operation which is not supported by all stores.
                                          Returns true if the value was created.

#values_at(*keys, **options)              retrieve multiple keys. Returns an array of equal length to the keys.
                                          Each entry in the array is either the value corresponding to the key
                                          in the same position, or nil if the key is not available.

#fetch_values(*keys, **options, &block)   retrieve multiple keys. Return is identical to values_at, except that
                                          when a block is given it will be called once for each key that is not
                                          available, and the return value of the block will be used in place of
                                          nil in the array.

#slice(*keys, **options)                  retrieve multiple keys. Returns an enumerable of key-value pairs,
                                          one for each of the supplied keys that is present in the store.

#merge!(pairs, options = {})              set values for multiple keys. "pairs" must be an enumerable of
                                          key-value pairs to be stored. Any existing keys will be clobbered.

#merge!(pairs, options = {}, &block)      set values for multiple keys. For each existing key, execute the block
                                          passing the key, existing value and new value, and store the return
                                          value.

#update(pairs, options = {}, &block)      same as merge!

#each_key                                 return an enumerable which will yield all keys in the store, one at a
                                          time. This method is present if and only if the store supports the
                                          :each_key feature.

#each_key(&block)                         yield all keys in the store to the block, one at a time. Again, this
                                          method is present if and only if the store supports the :each_key
                                          feature.

#clear(options = {})                      clear all keys in this store.

#close                                    close database connection.

#features                                 return array of features, e.g. [:create, :expires, :increment]

#supports?(feature)                       returns true if store supports a given feature

Creating a Store

There is a simple interface to create a store using Moneta.new. You will get automatic key and value serialization which is provided by Moneta::Transformer. This allows you to store arbitrary Ruby objects. You can tune some options when you call Moneta.new. However for very fine tuning use Moneta.build.

store = Moneta.new(:Memcached, server: 'localhost:11211')
store['key'] = 'value'
store['hash_key'] = {a: 1, b: 2}
store['object_key'] = MarshallableRubyObject.new

If you want to have control over the proxies, you have to use Moneta.build:

store = Moneta.build do
  # Adds expires proxy
  use :Expires

  # Transform key using Marshal and Base64 and value using Marshal
  use :Transformer, key: [:marshal, :base64], value: :marshal

  # IMPORTANT: adapter must be defined last for the builder to function properly.

  # Memory backend
  adapter :Memory
end

You can also directly access the underlying adapters if you don't want to use the Moneta stack.

db = Moneta::Adapters::File.new(dir: 'directory')
db['key'] = {a: 1, b: 2} # This will fail since you can only store Strings

# However for Mongo and Couch this works
# The hash will be mapped directly to a Mongo/Couch document.
db = Moneta::Adapters::Couch.new
db['key'] = {a: 1, b: 2}

db = Moneta::Adapters::Mongo.new
db['key'] = {a: 1, b: 2}

Expiration

The Cassandra, Memcached, Redis and Mongo backends support expiration natively.

cache = Moneta::Adapters::Memcached.new

# Or using the builder...
cache = Moneta.build do
  adapter :Memcached
end

# Expires in 60 seconds
cache.store(key, value, expires: 60)

# Never expire
cache.store(key, value, expires: 0)
cache.store(key, value, expires: false)

# Update expires time if value is found
cache.load(key, expires: 30)
cache.key?(key, expires: 30)

# Or remove the expiration if found
cache.load(key, expires: false)
cache.key?(key, expires: 0)

You can add the expires feature to other backends using the Moneta::Expires proxy. But be aware that expired values are not deleted automatically if they are not looked up.

# Using the :expires option
cache = Moneta.new(:File, dir: '...', expires: true)

# or manually by using the proxy...
cache = Moneta::Expires.new(Moneta::Adapters::File.new(dir: '...'))

# or using the builder...
cache = Moneta.build do
  use :Expires
  adapter :File, dir: '...'
end

Key traversal

Where supported by the store's backend, it is possible to traverse the keys in the store using the #each_key method. Support for this can be tested by calling store.supports?(:each_key), or checking for the presence of :each_key in store.features.

store.each_key # returns an Enumerable
store.each_key do |key|
  store.load(key)  # read operations are supported within the block
  store[key] = "x" # behaviour of write operations is undefined
end

Atomic operations

Atomic incrementation and raw access

The stores support the #increment which allows atomic increments of unsigned integer values. If you increment a non existing value, it will be created. If you increment a non integer value an exception will be raised.

store.increment('counter')     # returns 1, counter created
store.increment('counter')     # returns 2
store.increment('counter', -1) # returns 1
store.increment('counter', 13) # returns 14
store.increment('counter', 0)  # returns 14
store.decrement('counter')     # returns 13
store['name'] = 'Moneta'
store.increment('name')        # raises an Exception

If you want to access the counter value you have to use raw access to the datastore. This is only important if you have a Moneta::Transformer somewhere in your proxy stack which transforms the values e.g. with Marshal.

store.increment('counter')          # returns 1, counter created
store.load('counter', raw: true) # returns 1

store.store('counter', '10', raw: true)
store.increment('counter') # returns 11

Fortunately there is a nicer way to do this using some syntactic sugar!

store.increment('counter') # returns 1, counter created
store.raw['counter']       # returns 1
store.raw.load('counter')  # returns 1

store.raw['counter'] = '10'
store.increment('counter') # returns 11

You can also keep the raw store in a variable and use it like this:

counters = store.raw

counters.increment('counter') # returns 1, counter created
counters['counter']           # returns 1
counters.load('counter')      # returns 1

counters['counter'] = '10'
counters.increment('counter') # returns 11

Atomic create

The stores support the #create which allows atomic creation of entries. #create returns true if the value was created.

store.create('key', 'value') # returns true
store.create('key', 'other value') # returns false

Atomic bulk operations

All stores support storage and retrieval of multiple keys using #values_at/#fetch_values/#slice and #merge!/#update. Wherever possible, these operations are performed atomically. When this is not possible, the #load and #store methods are called once for each key.

store.merge!('key1' => 'value1', 'key2' => 'value2') # stores two keys
store.values_at('key1', 'key2', 'key3') # returns ['value1', 'value2', nil]
store.fetch('key1', 'key3') { |k| k + ' missing' } # returns ['key1', 'key3 missing']
store.slice('key1', 'key2', 'key3') # returns enumerable of ['key1', 'value1'], ['key2', 'value2']

store.merge!('key2' => 'new value2', 'key3' => 'value3') do |key, value, new_value|
  [value, new_value].join('+')
end # stores "value3" and "value2+new value2"

Shared/distributed synchronization primitives

Moneta provides shared/distributed synchronization primitives which are shared database-wide between all clients.

Moneta::Mutex allows a single thread to enter a critical section.

mutex = Moneta::Mutex.new(store, 'mutex_key')

mutex.synchronize do
   mutex.locked? # returns true

   # Synchronized access to counter
   store['counter'] += 1
end

begin
  mutex.lock
  mutex.locked? # returns true
  # ...
ensure
  mutex.unlock
end

Moneta::Semaphore allows max_concurrent threads to enter a critical section.

semaphore = Moneta::Semaphore.new(store, 'semaphore_counter', max_concurrent)

semaphore.synchronize do
   semaphore.locked? # returns true
   # ...
end

begin
  semaphore.enter
  semaphore.locked? # returns true
  # ...
ensure
  semaphore.leave
end

Weak atomic operations

If an underlying adapter doesn't provide atomic #create or #increment and #decrement you can use the proxies Moneta::WeakIncrement and Moneta::WeakCreate to add support without atomicity.

But then you have to ensure that the store is not shared by multiple processes and thread-safety is provided by Moneta::Lock.

Syntactic sugar and option merger

For raw data access as described before the class Moneta::OptionMerger is used. It works like this:

# All methods after 'with' get the options passed
store.with(raw: true).load('key')

# You can also specify the methods
store.with(raw: true, only: :load).load('key')
store.with(raw: true, except: [:key?, :increment]).load('key')

# Syntactic sugar for raw access
store.raw.load('key')

# Access substore where all keys get a prefix
substore = store.prefix('sub')
substore['key'] = 'value'
store['key']    # returns nil
store['subkey'] # returns 'value'

# Set expiration time for all keys
short_lived_store = store.expires(60)
short_lived_store['key'] = 'value'

Add proxies to existing store

You can add proxies to an existing store. This is useful if you want to compress only a few values for example.

compressed_store = store.with(prefix: 'compressed') do
  use :Transformer, value: :zlib
end

store['key'] = 'this value will not be compressed'
compressed_store['key'] = 'value will be compressed'

Framework Integration

Inspired by redis-store there exist integration classes for Rails and Rack/Rack-Cache. You can also use all the Rack middlewares together with Rails and the Sinatra framework. There exist the following integration classes:

  • Rack, Rails and Sinatra
    • Rack::Session::Moneta is a Rack middleware to use Moneta for storing sessions
    • Rack::MonetaStore is a Rack middleware which places a Moneta store in the environment and enables per-request caching
    • Rack::MonetaCookies is a Rack middleware which uses Moneta to store cookies
    • Rack::MonetaRest is a Rack application which exposes a Moneta store via REST/HTTP
    • Rack::Cache::Moneta provides meta and entity stores for Rack-Cache
  • Rails
    • ActionDispatch::Session::MonetaStore is a Rails middleware to use Moneta for storing sessions
    • ActiveSupport::Cache::MonetaStore is a Rails cache implementation which uses a Moneta store as backend
  • Ramaze
    • Ramaze::Cache::Moneta is integrated into the Ramaze project and allows Ramaze to use Moneta as caching store
  • Padrino adopted Moneta to replace their cache stores in padrino-cache.

Rack

Session store

You can use Moneta as a Rack session store. Use it in your config.ru like this:

require 'rack/session/moneta'

# Use only the adapter name
use Rack::Session::Moneta, store: :Redis

# Use Moneta.new
use Rack::Session::Moneta, store: Moneta.new(:Memory, expires: true)

# Set rack options
use Rack::Session::Moneta, key: 'rack.session',
domain: 'foo.com',
path: '/',
expire_after: 2592000,
store: Moneta.new(:Memory, expires: true)

# Use the Moneta builder
use Rack::Session::Moneta do
  use :Expires
  adapter :Memory
end

Moneta middleware

There is a simple middleware which places a Moneta store in the Rack environment at env['rack.moneta_store']. It supports per-request caching if you add the option cache: true. Use it in your config.ru like this:

require 'rack/moneta_store'

# Add Rack::MonetaStore somewhere in your rack stack
use Rack::MonetaStore, :Memory, cache: true

run lambda { |env|
  env['rack.moneta_store'] # is a Moneta store with per-request caching
}

# Pass it a block like the one passed to Moneta.build
use Rack::MonetaStore do
  use :Transformer, value: :zlib
  adapter :Cookie
end

run lambda { |env|
  env['rack.moneta_store'] # is a Moneta store without caching
}

REST server

If you want to expose your Moneta key/value store via HTTP, you can use the Rack/Moneta REST service. Use it in your config.ru like this:

require 'rack/moneta_rest'

map '/moneta' do
  run Rack::MonetaRest.new(:Memory)
end

# Or pass it a block like the one passed to Moneta.build
run Rack::MonetaRest.new do
  use :Transformer, value: :zlib
  adapter :Memory
end

Rack-Cache

You can use Moneta as a Rack-Cache store. Use it in your config.ru like this:

require 'rack/cache/moneta'

use Rack::Cache,
      metastore:   'moneta://Memory?expires=true',
      entitystore: 'moneta://Memory?expires=true'

# Or used named Moneta stores
Rack::Cache::Moneta['named_metastore'] = Moneta.build do
  use :Expires
  adapter :Memory
end
use Rack::Cache,
      metastore: 'moneta://named_metastore',
      entity_store: 'moneta://named_entitystore'

Cookies

Use Moneta to store cookies in Rack. It uses the Moneta::Adapters::Cookie. You might wonder what the purpose of this store or Rack middleware is: It makes it possible to use all the transformers on the cookies (e.g. :prefix, :marshal and :hmac for value verification).

require 'rack/moneta_cookies'

use Rack::MonetaCookies, domain: 'example.com', path: '/path'
run lambda { |env|
  req = Rack::Request.new(env)
  req.cookies #=> is now a Moneta store!
  env['rack.request.cookie_hash'] #=> is now a Moneta store!
  req.cookies['key'] #=> retrieves 'key'
  req.cookies['key'] = 'value' #=> sets 'key'
  req.cookies.delete('key') #=> removes 'key'
  [200, {}, []]
}

Rails

Session store

Add the session store in your application configuration config/environments/*.rb.

require 'action_dispatch/middleware/session/moneta_store'

# Only by adapter name
config.cache_store :moneta_store, store: :Memory

# Use Moneta.new
config.cache_store :moneta_store, store: Moneta.new(:Memory)

# Use the Moneta builder
config.cache_store :moneta_store, store: Moneta.build do
  use :Expires
  adapter :Memory
end

Cache store

Add the cache store in your application configuration config/environments/*.rb. Unfortunately the Moneta cache store doesn't support matchers. If you need these features use a different server-specific implementation.

require 'active_support/cache/moneta_store'

# Only by adapter name
config.cache_store :moneta_store, store: :Memory

# Use Moneta.new
config.cache_store :moneta_store, store: Moneta.new(:Memory)

# Use the Moneta builder
config.cache_store :moneta_store, store: Moneta.build do
  use :Expires
  adapter :Memory
end

Padrino

Padrino adopted Moneta to replace their cache stores in padrino-cache. You use it like this

# Global Padrino caching
# Don't forget the expires: [true, Integer] if you want expiration support!
Padrino.cache = Moneta.new(:Memory, expires: true)

# Application caching
# Don't forget the expires: [true, Integer] if you want expiration support!
set :cache, Moneta.new(:Memory, expires: true)

Advanced

Build your own key value server

You can use Moneta to build your own key/value server which is shared between multiple processes. If you run the following code in two different processes, they will share the same data which will also be persistet in the database shared.db.

require 'moneta'

store = Moneta.build do
  use :Transformer, key: :marshal, value: :marshal
  use :Shared do
    use :Cache do
      cache do
        adapter :LRUHash
      end
      backend do
        adapter :GDBM, file: 'shared.db'
      end
    end
  end
end

If you want to go further, you might want to take a look at Moneta::Server and Moneta::Adapters::Client which are used by Moneta::Shared and provide the networking communication. But be aware that they are experimental and subjected to change. They provide an acceptable performance (for being ruby only), but don't have a stable protocol yet.

You might wonder why I didn't use DRb to implement server and client - in fact my first versions used it, but with much worse performance and it was real fun to implement the networking directly :) There is still much room for improvement and experiments, try EventMachine, try Kgio, ...

ToyStore ORM

If you want something more advanced to handle your objects and relations, use John Nunemaker's ToyStore which works together with Moneta. Assuming that Person is a ToyStore::Object you can add persistence using Moneta as follows:

# Use the Moneta Redis backend
Person.adapter :memory, Moneta.new(:Redis)

Testing and Benchmarks

Testing is done using GitHub Actions. Currently we support MRI Ruby >= 2.4.0 (but not yet 3.x) and the JRuby >= 9.2.9.0. MRI 2.3.0 should mostly still work, but is no longer tested in CI.

Benchmarks for each store are done on Travis-CI for each build. At the time of writing, benchmarks still need to be migrated from Travis to GitHub Actions. Take a look there to compare the speed of the different key value stores for different key/value sizes and size distributions. Feel free to add your own configurations! The impact of Moneta should be minimal since it is only a thin layer on top of the different stores.

How to contribute?

Always feel free to open an issue on https://github.com/moneta-rb/moneta/issues if something doesn't work as you expect it to work. Feedback is also very welcome!

My only request about patches is that you please try to test them before submitting.

Contribute an adapter

If you want support for another adapter you can at first at it to the list of missing adapters at #16

If you choose to implement an adapter please also add tests. Please check also if anything in .github/workflows needs changes, for example if you need to start additional services.

Check if the default settings in Moneta#new are appropriate for your adapter. If not specify a better one.

Don't forget to edit the README.md and the CHANGES.

Alternatives

  • Horcrux: Used at github, supports batch operations but only Memcached backend
  • ActiveSupport::Cache::Store: The Rails cache store abstraction
  • ToyStore: ORM mapper for key/value stores
  • ToyStore Adapter: Adapter to key/value stores used by ToyStore, Moneta can be used directly with the ToyStore Memory adapter
  • Cache: Rubygem cache wraps Memcached and Redis
  • Ramaze::Cache: Cache stores of the Ramaze framework with support for LocalMemCache, Memcached, Sequel, Redis, ...

Authors

moneta's People

Contributors

aeden avatar aemadrid avatar asppsa avatar atoxio avatar benschwarz avatar brutuscat avatar dsrw avatar erithmetic avatar greyblake avatar hamptonmakes avatar hannesg avatar jarib avatar jaymitchell avatar jcharaoui avatar korkey128k avatar laktek avatar malmckay avatar minad avatar nbibler avatar olleolleolle avatar orien avatar piotrmurach avatar raxoft avatar rthbound avatar skrobul avatar timogoebel avatar tomk32 avatar tony612 avatar wycats avatar xaviershay avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

hannesg rsanders jvoorhis vireshas stakach hptt piotrmurach rthbound simlegate atoxio bjhaid mimosa semipermeable gabrielg billiam pombredanne dariocravero itswindtw mobilecommons malmckay thethanghn wehal3001 faustoct1 nbibler newaltcoin jaymitchell scalp42 mimperatore rezpe lingochamp veryfatboy sunxk spnkr venkatd zfjoy520 rettinohusker51 asppsa abargnesi maniacs-db giiko halida drsharp digideskio karna1995 daniilfranks shidel-dev timogoebel eugen0329 skrobul wjordan spikewilliams mallozup anakinj prasadsurase olleolleolle alexanderkraemer brutuscat raxoft salsify mbrukman tupalo goltergaul awesomedatatool botp beewill bvicenzo jgnagy jbampton gogainda olegykz git-naren jesterxd valscion aks terceiro petergoldstein gsmetal pklot sriedel bearerpipelinetest korkey128k anarcat tomk32 djberube mgarcia-fsl jcharaoui andrewrobison

moneta's Issues

Cannot store any value using moneta/sequel/H2

Hi,

I've been trying the following gems and jar files on jruby-1.6.8 (in 1.8 mode):
moneta (0.7.19)
sequel (3.48.0)
h2-1.3.173.jar from h2database.com (a SQL database)

When trying the following code snippet, I get a serialization error. What is the proper way to configure the transformer in moneta to insert properly a blob in Sequel?

Code snippet:

require 'moneta'
require 'sequel'
require 'h2-1.3.173.jar'
store = Moneta.new(:Sequel, :db => 'jdbc:h2:foo', :table => 'tbl_moneta')
store['key'] = 'value'

Error log:
Sequel::DatabaseError: NativeException: org.h2.jdbc.JdbcSQLException: Hexadecimal string contains non-hex character: "BAgiCnZhbHVl"; SQL statement:
INSERT INTO "TBL_MONETA" ("K", "V") VALUES ('a2V5', 'BAgiCnZhbHVl') -- ('a2V5', 'BAgiCnZhbHVl') [90004-173]

Adapters missing from Rack::Session::Moneta?

Hello all,

I'm trying to use the :MongoMoped adapter with Rack::Session::Moneta, but I'm getting uninitialized constant errors:

use Rack::Session::Moneta, store: :MongoMoped

When I inspect Moneta::Adapters module, I can see a number of other adapters included, but not :MongoMoped:

...
Moneta::Adapters::Memcached
Moneta::Adapters::MemcachedDalli
Moneta::Adapters::MemcachedNative
Moneta::Adapters::Memory
Moneta::Adapters::Mongo
Moneta::Adapters::Null
...

So, what happened to MongoMoped and MongoOffical? Do I have to require them separately? Does Rack::Session::Moneta even support these adapters?

Add missing tests on travis

  • localmemcache is untested:

Could not allocate memory/mmap errors occur always. I think the problem is that the travis machines are 32 bit, this is mentioned on the localmemcache page.

  • hbase is untested since it is not installed on travis

Key/value iterator (#each, #keys)

Hi,

I'm currently using Moneta as a backend for I18n and it's missing a keys method interface needed by that gem as to what's the minimum a backend should support to comply.

I reckon it might be of good use on other scenarios anyway, however, is there any reason why it isn't available already?

I'll be happy to work on a patch if this makes sense.

Thanks,
Darío

PS: I'm patching it right now for the Memory adapter as follows:

module Moneta
  class Proxy
    def keys
      adapter.keys
    end
  end

  module HashAdapter
    # Expand on https://github.com/minad/moneta/blob/master/lib/moneta/mixins.rb
    def keys
      @backend.keys
    end
  end
end

Fix tests for other ruby implementations

Currently only 1.9.3 and 1.8.7 are marked as supported.

  • Jruby works mostly but some gems are not supported there. This check is done in the Gemfile. Check if there are equivalent Jruby gems.
  • Rubinius sometimes fails badly. I am not very familiar with rubinius, so I cannot help here.

Support for set-if-not-exists.

Hi

I think we could really use this to avoid race conditions ( like in the session generator ). Most backends support this out-of-the box anyway. I'm not sure about the syntax. My idea so far:

moneta.store('key','value', :update => false )

TODO:

  • try-out new todo lists
  • implement basic specs
  • add method to Defaults, Proxy, Transformer
  • activerecord ( too tricky with exceptions )
  • Cassandra
  • Client
  • cookie ( not supported by backend )
  • CouchDB ( doesn't allow create-if-not-exists for named docs afaik )
  • DBM ( ruby binding lacks support )
  • File ( tricky as File.rename can delete files on newer rubies )
  • Fog ( not supported by backend )
  • GDBM ( ruby binding lacks support )
  • HBase ( not supported by backend )
  • LevelDB ( not supported by backend )
  • Localmemcache ( not supported by backend )
  • LRUHash
  • Memcached
  • Memory
  • MongoDB
  • Null
  • PStore ( get & set in transaction )
  • Redis
  • Restclient ( difficult semantic! http header? )
  • Riak ( not sure if command is available, but riak is eventually cosistent anyway so no strong guarantee it works )
  • SDBM ( ruby binding lacks support )
  • Sequel ( too tricky with exceptions )
  • Sqlite
  • TDB ( tdb.store )
  • Tokyo
  • YAML via PStore

Test cassandra on travis

You have to define some kind of schema for cassandra. I haven't used cassandra before, so maybe an expert could take over from here.

Missing key/value stores

This is a list of currently missing key/value stores. Please comment if you know another one that you might want to use. Implementations are also greatly appreciated!

Some implementations are not yet finished but are already in the staging branch. They need testing and fixes!

Gems available:

Relatively unknown:

Interesting databases which currently lack a ruby binding, but claim to be very fast:

Relational:

  • RDBI (gem rdbi)
  • DBI (gem dbi)
  • DataObjects (gem data_objects)
  • JDBC

Special:

  • AWS adapter using the amazon library instead of Fog
  • FTP (stdlib net/ftp)
  • RCS (Git, Mercurial, SVN, ...)
  • IMAP
  • LDAP
  • TOML (https://github.com/mojombo/toml)
  • Configuration files (e.g. gem configtoolkit, ...)

For inspiration: http://nosql-database.org/, https://github.com/datamapper/dm-core/wiki/Adapters

Check if connection is open/ready

I've noticed I can't determine from Moneta if the data store (or connection to it) is ready for requests or been closed. I end up needing to pull the backend object out of the Moneta adapter and directly query the backend for its status. Since I'm no longer using Moneta's abstraction, it means my code is no longer portable between backends.

Is this a planned feature, or perhaps there is another approach I overlooked?

More documentation

Documentation missing:

  • Juno.new with options
  • Options of all adapters and proxies
  • Document missing methods

Missing key/value stores

This is a list of currently missing key/value stores. Please comment if you know another one that you might want to use. Implementations are also greatly appreciated!

Some implementations are not yet finished but are already in the staging branch. They need testing and fixes!

Gems available:

Relatively unknown:

Interesting databases which currently lack a ruby binding, but claim to be very fast:

Relational:

  • RDBI (gem rdbi)
  • DBI (gem dbi)
  • DataObjects (gem data_objects)
  • JDBC

Special:

  • AWS adapter using the amazon library instead of Fog
  • FTP (stdlib net/ftp)
  • RCS (Git, Mercurial, SVN, ...)
  • IMAP
  • LDAP
  • TOML (https://github.com/mojombo/toml)
  • Configuration files (e.g. gem configtoolkit, ...)

For inspiration: http://nosql-database.org/, https://github.com/datamapper/dm-core/wiki/Adapters

Order of adapter and Transformer is important in Moneta.build routine 

Moneta.build do
  use :Transformer, :key => :marshal, :value => :marshal
  adapter :Redis
end

Success!

Moneta.build do
  adapter :Redis
  use :Transformer, :key => :marshal, :value => :marshal
end

Fails! with

moneta/transformer.rb:35:in `new': Option :key or :value is required (ArgumentError)

It's somehow reading the adapter as an array and then calling Transformer new without a specified adapter. Not sure why, but that's what is happening in Moneta Builder.

klass.new(options, &block)

is being called instead of

klass.new(stores.last, options, &block)

in

def build
  adapter = @proxies.first
  if Array === adapter
    klass, options, block = adapter
    adapter = klass.new(options, &block)
  end
  @proxies[1..-1].inject([adapter]) do |stores, proxy|
    klass, options, block = proxy
    stores << klass.new(stores.last, options, &block)
  end
end

TokyoTyrant and HBase packing/unpacking

TokyoTyrant and HBase add 0 byte to distinguish integer values from string values. Do you have a better idea? Otherwise move pack and unpack somehow to utils.

Setting cookie expiry date in Rack

So, I can't wrap my head around this: I'm using DataMapper as a "storage adapter" inside a Sinatra/Rack app. I was able to configure all the relevant cookie fields (like domain, path, etc.) but never the expires.

Here's how I'm initializing Moneta and hooking it up with Rack:

use Rack::Session::Moneta, {
    domain: settings.cookies['domain'],
    path:   settings.cookies['path'],
    key:    settings.cookies['key'],
    secret: settings.cookies['secret'],
    secure: settings.cookies['secure'],
    store: Moneta.new(:DataMapper, {
      repository: :default,
      setup: "mysql://#{dbc[:un]}:#{dbc[:pw]}@#{dbc[:host]}:#{dbc[:port]}/#{dbc[:db]}"
    })
  }

How do I get my cookies to expire after a certain amount of time? Right now they're acting like session-cookies, causing people to re-login everytime they visit the page.

Cassandra keyspace creation fails

@greyblake: The cassandra tests worked after I added the cassandra schema file as you suggested. See https://github.com/minad/juno/blob/4aaea72be9a9fd616825699bc3f8fd131b69d418/spec/juno.cassandra

But then I tried to change the cassandra adapter in such a way that it creates the keyspace automatically if it doesn't exist, to make juno easier to use. See https://github.com/minad/juno/blob/master/lib/juno/adapters/cassandra.rb#L26

Now it doesn't work anymore. On ruby 1.8.7 it fails with "Keyspace doesn't exist", see https://travis-ci.org/minad/juno/jobs/3696450

On 1.9.3 it fails with "SimpleStrategy requires a replication_factor strategy option", see https://travis-ci.org/minad/juno/jobs/3696452

Do you have any ideas? Are you using Cassandra in production somewhere?

Fyi, I took the code to create the keyspace etc from https://github.com/bmuller/ankusa/blob/master/lib/ankusa/cassandra_storage.rb

Support for increment method

Not done:

  • Cassandra
  • DataMapper
  • HBase

Not supported:

  • Riak
  • Localmemcache
  • Mongo
  • Couch

Done:

  • Sequel
  • File
  • ActiveRecord
  • DBM/GDBM/SDBM
  • Cookie
  • Memory
  • PStore/YAML
  • Sqlite
  • LRUHash
  • Redis
  • Memcached
  • TokyoCabinet
  • LevelDB

Improve Transformer

I think the Transformer class could need some improvements. When I started it, I just saw it as functions which are invertible for values and not invertible for keys. But now it is getting chaotic...

When I see it now, there are multiple types of functions:

Values:

  1. Serializers (Marshal, JSON, ...)
  2. Compressors (Zlib, QuickLZ, ...)
  3. String encoders (Base64, ...)

Keys:

  1. Serializers (Marshal, JSON, ...)
  2. Digests
  3. String encoders (Escape, Base64, ...)
  4. Misc (Spread special for file storage, Prefix)

In the future there could come things like HMAC and encryption.

So they build some "chain" and we could for example enforce that serializers have to come always before the rest, and the special string manipulation has to come at the end.

It could also be interesting to refactor the Transformer in two separate classes:

  • InvertibleTransformer with methods #forward and #back
  • NotInvertibleTransformer with method #call

Transformer could use these classes then to build the known hash proxy. This feature could be useful without using the hash key/value thing, maybe it even deserves an own gem maybe with some functionality to register new functions.

@hannesg: What do you think?

examples of using in multi process environments?

Hi.

I want to use moneta in a multi threaded multi process environment to manage counters and locks. I plan on using redis as a back end in production but just using a LRU in development.

The document is not clear on whether I should use the shared builder( which seems to run a daemon?) or if I should use the mutexes manually or maybe just rely on redis to handle the contention.

Is there some howto someplace that describes how to use it like this?

Thanks.

Think about batch operations

batch_load(*keys), batch_store(key_value_hash), batch_delete(*keys) which could have optimized implementation for some backends

Horcrux supports that. Maybe this is too specific and not useful in most cases. Maybe one shouldn't use Juno if one needs such an optimization.

Git backend

It would be nice to have a backend which accesses a git repository.

Force persistant sync / Intentional cache busting

A lot of data stores and libraries cache, but there's no way that I can find to bust the cache or force a reload. If I make a change with one process or thread, it may take time to propagate to another, or in some cases not at all. To get around Moneta not supporting this feature, I have to dig out the backend and run commands on it directly. This means my code is no longer portable between backends, and voids a big chunk of Moneta's usefulness for me.

Is there another way to do this?

Add logger or aspect proxy

  • Add a proxy which fires events for each store operation change (Maybe use ruby stdlib Observable). Would this be useful, what do you think?
  • Add a logger proxy which logs store accesses for debugging purposes, this could use the Observer proxy if there is one.

Fix travis tests

Some tests are crashing ruby on travis. We have to split the tests and group them somehow to find failing tests easier. I have no idea how to refactor it :(

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.