roma / roma Goto Github PK

View Code? Open in Web Editor NEW

328.0 25.0 40.0 1.81 MB

ROMA: A Distributed Key-Value Store in Ruby

Home Page: http://roma-kvs.org/

License: GNU General Public License v3.0

Ruby 100.00%

roma ruby memcached rakuten kvs

roma's Introduction

ROMA - A Distributed Key-Value Store in Ruby

ROMA is one of the data storing systems for distributed key-value stores.
It is a completely decentralized distributed system that consists of multiple processes, called nodes, on several machines. It is based on pure P2P architecture
like a distributed hash table, thus it provides high availability and scalability.

ROMA is written in Ruby. However, following choices are available to access to ROMA.

Client libraries of Ruby and Java are available.
ROMA protocol is compatible with memcached text-based one so that
any memcached client libraries allows users to interact with ROMA.

More information is here.

Documentation

Refer to ROMA documentations

Requirements

Ruby >= 2.1.0

Installation

Install ROMA

You can simply install ROMA and dependency libralies by using a gem command of Ruby as follows.

$ gem install roma

Make routing files

ROMA is required to make the routing files before starting up.
The routing file is stored the routing information of each processes.

$ mkroute localhost_10001 localhost_10002 --replication_in_host

If succeeded, two new files which named localhost_10001.route and localhost_10002.route created in the current directory.
Refer to Commands for more detail information about Shell Commands.

Start up ROMA

Run two processes by using a romad.rb program as follows:

$ romad localhost -p 10001 -d --replication_in_host
$ romad localhost -p 10002 -d --replication_in_host

Refer to Shell Commands for more detail information about Shell Commands.

Usage

Like memcached, you can connect to ROMA with telnet. Connect to the ROMA process that you ran above.

$ telnet localhost 10001

You can interact with ROMA in the same way of memcached commands.

set foo 0 0 3 <return>
bar <return>
STORED
get foo <return>
VALUE foo 0 3
bar
END

Refer to Commands for more detail information about ROMA Commands.

Contributing

If you would like to contribute, please...

Fork and git clone it.
Install gems required for development.
Make changes in a branch & add unit tests.
Run Unit Test

bundle exec rake (if unit test fails, run it again - it's fickle).
Specify STORAGE to test test cases related to storages such as groonga, sqlite3 and dbm.

Create a pull request.

Contributions, improvements, comments and suggestions are welcome!

Promoters

Roma is promoted by Rakuten, Inc. and Rakuten Institute of Technology.

roma's People

Contributors

Stargazers

Watchers

roma's Issues

Add 'groonga' option to mkconfig

Currently, mkconfig.rb has 2 storage options (RubyHash and TokyoCabinet).
ROMA v1.1.0 can use 'groonga' as storage software.
So mkconfig needs to add groonga option.

add progress rate of data accumulation tool

Now user can not know when data accumulation will be finished in case of large data.

mkconfig make directories and set full path to configuration file instead of place holder

mkconfig makes configuration file including place holder.
I think it is better to make configuration file based on current directory(default) and set base directory with command line.

add check logic of TokyoCabinet file flag

Sometimes Tokyocabinet file will be broken by unexpected Error.
We want to check function of TokyoCabinet healt to Quickly notice for physical error.

[snapshot] Enable join, recover and other process during snapshot

How do I install this on my Mac?

Please update the README.md

Which versions of ruby are supported?

rubygems.rog means that the gem roma supports all versions of ruby since required ruby versions is '>= 0'.

Is it true? if no, supported versions of ruby should be specified in roma.gemspec.

Add explanation of "plugin_map_count"

http://roma-kvs.org/commands.html

Add command check logic by use jaro-winkler

enable to check log level on stats

Add explanation of "plugin_debug"

create virtual console

Enable join with data

Add monitoring function like a "top" command

develop data accumulation tool

divide vnodes count within secondary node

Now ROMA's stat of vnodes count have just primary node and secondary node.
So all vnodes (without primary) combined to secondary nodes.

So I will make secondary2, secondary3....

Add explanation of "plugin_storage"

http://roma-kvs.org/commands.html

Wrap up monitoring points

investigate ruby 2.2

enable to check failover on stats

Benchmark test between memcached & ROMA-RH

Test for v1.2.0 release

Add explanation of "plugin_test"

cluster replication for hot standby

Add explanation of "plugin_gui"

Make ruby motion application

make new command (rbalse + release)

At first, rbalse was unofficial command.
So it is not confirm when executing.
Add confirmation like a balse?
Or make new command alias for example "single_shutdown"?
Or make new command which include both function of "rbalse" and "release".
This tickets will start investigation and discussing.
After decide specification, develop actually and do not forget add test.

routing.secondary do not increase when execute "join" after "release" action

routing.secondary did not increase when execute "join" after "release" action.
If routing.short_vnodes is 0 in the cluster, routing.secondary will not increase on new node during join action.

$ bin/romad --replication_in_host localhost -p 11213 --config config.rb -j localhost_11211 -d
$ echo "stat primary|secondary" | nc localhost 11213
routing.primary 170
routing.secondary 0
END

1.1.0RC1 has ruby/server/ directory

roma-1.1.0RC1.tar.gz has both lib/ and ruby/server/lib/ directory. It seems that ROMA migrated to lib/ from ruby/server/lib/ by #26. But the archive has ruby/server/lib.

And Groonga storage exists only ruby/server/lib/. Could you confirm it?

adjust config file to groonga

in case of selection groonga for storage, mkconfig process remove unnecessary value.
(like a bnum, xmsiz, opts, dfunit)

[plugin]plugin_gui.rb does not resolve routing directory.

I think plugin_gui.rb does not resolve routing directory when there is other file and directory under routing direcotry.

bash$ ls -la /your/path/app/routing
192.168.xxx.xxx_10001.route
192.168.xxx.xxx_10001.route.1
192.168.xxx.xxx_10001.route.2
192.168.xxx.xxx_10002.route
192.168.xxx.xxx_10002.route.1
192.168.xxx.xxx_10002.route.2
hoge_direcotry ## this is bad directory.

If there is "hoge_direcotry" under your routing path, Gladiator does not work well.
Gladiator is ROMA GUI.
https://github.com/roma/gladiator

move information from googledoc

Add explanation of "plugin_map"

duplicate plugin_storage.rb in mkconfig

When making config.rb, plugin_storage.rb is descripted twice.

After
+----------------------------------------------------------------------------------------------------------------+
|STORAGE_CLASS: Roma::Storage::TCStorage |
|STORAGE_OPTION: bnum=5000000#xmsiz=375809638#opts=d#dfunit=10 |
|PLUGIN_FILES: ["plugin_storage.rb", "plugin_storage.rb", "plugin_gui.rb", "plugin_alist.rb", "plugin_debug.rb"] |
+----------------------------------------------------------------------------------------------------------------+

Mkconfig is finish.

If you need, change directory path about LOG, RTTABLE, STORAGE, WB and other setting.

enable to check expired time of key

Add "--replication-in-host" option to rename "--enabled_repeathost" to "--replication-in-host"

"--enabled_repeathost" option is making confusion now. We will rename "--enabled_repeathost" to "--replication-in-host" in the long term, so I add "--replication-in-host" option at first.
This option has same function of "--enabled_repeathost".
We will remain the "--enabled_repeathost" option in the short term.

New command line tool without 'telnet'

Usability improvement
Provision to prohibit using 'telnet' in our DC

add config version's check

config.rb's contents are changed sometimes, so it have possibility to happen error if keep using too old config file.
So we have to check version's logic.

Is there any release plan or milestone for the next release?

I've received many notifications from merging PRs and closing issues.
But I could not understand where roma is moving ahead since many PRs don't have description.

I think release plan or milestone for the next release would help users understand its direction roma is moving ahead.

Make virtual console for ROMA

decide virtual console specification

develop data consistency check tool

I have developed and merged it.
#111

add plugin ongoin(without rebooting ROMA)

Gemfile is not working

I tried bundle install on roma directory but I faced error during tokyocabinet installation.

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    /usr/local/rbenv/versions/2.1.5/bin/ruby extconf.rb
setting variables ...
  $CFLAGS = -I. -I/usr/local/include -Wall $(cflags) -O2
  $LDFLAGS = -L. -L/usr/local/rbenv/versions/2.1.5/lib  -fstack-protector -rdynamic -Wl,-export-dynamic -L. -L/usr/local/lib
  $libs =  -ltokyocabinet -lz -lbz2 -lpthread -lm -lc
checking for tcutil.h... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
        --with-opt-dir
        --without-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=/usr/local/rbenv/versions/2.1.5/bin/ruby
        --with-tokyocabinet-dir
        --without-tokyocabinet-dir
        --with-tokyocabinet-include
        --without-tokyocabinet-include=${tokyocabinet-dir}/include
        --with-tokyocabinet-lib
        --without-tokyocabinet-lib=${tokyocabinet-dir}/lib
/usr/local/rbenv/versions/2.1.5/lib/ruby/2.1.0/mkmf.rb:456:in `try_do': The compiler failed to generate an executable file. (RuntimeError)
You have to install development tools first.
        from /usr/local/rbenv/versions/2.1.5/lib/ruby/2.1.0/mkmf.rb:587:in `try_cpp'
        from /usr/local/rbenv/versions/2.1.5/lib/ruby/2.1.0/mkmf.rb:1067:in `block in have_header'
        from /usr/local/rbenv/versions/2.1.5/lib/ruby/2.1.0/mkmf.rb:918:in `block in checking_for'
        from /usr/local/rbenv/versions/2.1.5/lib/ruby/2.1.0/mkmf.rb:351:in `block (2 levels) in postpone'
        from /usr/local/rbenv/versions/2.1.5/lib/ruby/2.1.0/mkmf.rb:321:in `open'
        from /usr/local/rbenv/versions/2.1.5/lib/ruby/2.1.0/mkmf.rb:351:in `block in postpone'
        from /usr/local/rbenv/versions/2.1.5/lib/ruby/2.1.0/mkmf.rb:321:in `open'
        from /usr/local/rbenv/versions/2.1.5/lib/ruby/2.1.0/mkmf.rb:347:in `postpone'
        from /usr/local/rbenv/versions/2.1.5/lib/ruby/2.1.0/mkmf.rb:917:in `checking_for'
        from /usr/local/rbenv/versions/2.1.5/lib/ruby/2.1.0/mkmf.rb:1066:in `have_header'
        from extconf.rb:28:in `<main>'

extconf failed, exit code 1

Gem files will remain installed in /usr/local/roma/roma_git_master/.tarrtest/ruby/2.1.0/bundler/gems/tokyocabinet-ruby-1904c4785b45 for inspection.
Results logged to /usr/local/roma/roma_git_master/.tarrtest/ruby/2.1.0/bundler/gems/extensions/x86_64-linux/2.1.0-static/tokyocabinet-ruby-1904c4785b45/gem_make.out
An error occurred while installing tokyocabinet (1.31.a), and Bundler cannot continue.