drecom/activerecord-turntable

ActiveRecord Sharding Plugin

activerecord-turntable's Issues

Reduce connection pool size

Operations that access all connection pools (i,e. clear_active_connections!) might be slowed down if applications having hundreds of model classes, because Turntable adds connection_pools(PoolProxy) object for each model classes.

Support acts_as_paranoid

Thread safety

Sequencer ignores config and updates master shard

Haven't figured out the whole thing yet but the config for sequencer is completely ignored.

The code below is where it gets ignored.

lib/active_record/turntable/cluster.rb:16

# setup sequencer
if (seq = (@options[:seq] || @config[:seq])) && seq[:type] == :mysql
  @seq_shard = SeqShard.new(seq)
end

There is no :type inside seq, there is only a sequence name like user_seq.
I tried doing something like @seq_shard = SeqShard.new(seq["user_seq"]) but that raised an error below.

/usr/lib64/ruby/gems/2.2.0/gems/activerecord-turntable-2.1.1/lib/active_record/turntable/shard.rb:43:in create_connection_class': undefined method with_indifferent_access' for nil:NilClass (NoMethodError)

Thoughts?

Support acts_as_archive

Support activerecord-import

CannotSpecifyShardError isn't raised on `exists?` queries in spite of `raise_on_not_specified_shard_query` being true

`ClusterHelperMethods.weighted_random_shards_with` raises NoMethodError

Occurred on this commit: 0e0a03d

Support multi-server DB?

In fact, If we need sharding DB then we must have multi-server host as below.

• Server MySQL 1: DB1-->DB100 (included user table)
• Server MySQL 2: DB101-->DB200
  ...
  But I see database.yml config file with one only host DB.

    connection_spec: &spec
      adapter: mysql2
      encoding: utf8
      reconnect: false
      pool: 5
      username: root
      password: root
      socket: /tmp/mysql.sock

Is it support multi-server DB?

Please visit the link.

https://medium.com/@Pinterest_Engineering/sharding-pinterest-how-we-scaled-our-mysql-fleet-3f341e96ca6f

undefined local variable doing rake db:migrate

Using Rails 4 and Turntable gem 2.1.1 (and 2.2.0), doing a rake db:migrate leads to the following error:

NameError: undefined local variable or method config' for main:Object ~/.rvm/gems/ruby-2.1.5/gems/activerecord-turntable-2.1.1/lib/active_record/turntable/railties/databases.rake:81:inblock (4 levels) in <top (required)>'

After performing migrations, it calls dump, which has this line:
shard_configs.merge!(config["seq"]) if config["seq"]

The variable 'config' is not set. 'current_config' is set. In other methods in the file, 'config' is initialized as:
config = ActiveRecord::Base.configurations[Rails.env]

Should config be set or should it be using current_config or do I not have something configured properly in turntable.yml or database.yml?

Undefined `*cluster*_transaction` method when models are lazy loading(development)

turntable DSL creates cluster_methods, but on development env, modesl are lazy loaded.

we should initialize clusters before model class loading.

Rails4 brakes class_to_pool, owner_to_pool hacking

On Rails4, class_to_pool and owner_to_pool implementations are using per process hashes.

When using Unicorn on preload_app=true env, the change causes turntable connection proxies broken.

Turntable installation error

An error occurred when I executed active_record:turntable:install after rails new and bundled activerecord-turntable.

versions

• turntable: current master (587d982)
• rails: 5.1.4
• ruby: 2.4.3

error

$ bundle exec rails g active_record:turntable:install
/path/to/activerecord-turntable-587d9825a957/lib/active_record/turntable/railtie.rb:24:in `exist?': no implicit conversion of nil into String (TypeError)
        from /path/to/activerecord-turntable-587d9825a957/lib/active_record/turntable/railtie.rb:24:in `block (2 levels) in <class:Railtie>'
        from /path/to/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:69:in `instance_eval'
        from /path/to/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:69:in `block in execute_hook'
        from /path/to/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:60:in `with_execution_control'
        from /path/to/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:65:in `execute_hook'
        from /path/to/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:50:in `block in run_load_hooks'
        from /path/to/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:49:in `each'
        from /path/to/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:49:in `run_load_hooks'
        from /path/to/gems/activerecord-5.1.4/lib/active_record/base.rb:326:in `<module:ActiveRecord>'
        from /path/to/gems/activerecord-5.1.4/lib/active_record/base.rb:25:in `<top (required)>'
        from /path/to/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `require'
        from /path/to/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `block in require'
        from /path/to/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:258:in `load_dependency'
        from /path/to/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `require'
        from /path/to/gems/spring-2.0.2/lib/spring/application.rb:356:in `active_record_configured?'
        from /path/to/gems/spring-2.0.2/lib/spring/application.rb:273:in `disconnect_database'
        from /path/to/gems/spring-2.0.2/lib/spring/application.rb:107:in `preload'
        from /path/to/gems/spring-2.0.2/lib/spring/application.rb:153:in `serve'
        from /path/to/gems/spring-2.0.2/lib/spring/application.rb:141:in `block in run'
        from /path/to/gems/spring-2.0.2/lib/spring/application.rb:135:in `loop'
        from /path/to/gems/spring-2.0.2/lib/spring/application.rb:135:in `run'
        from /path/to/gems/spring-2.0.2/lib/spring/application/boot.rb:19:in `<top (required)>'
        from /path/to/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /path/to/rubygems/core_ext/kernel_require.rb:55:in `require'
        from -e:1:in `<main>'

Parallelize query execution

STI subclasses of turntable enabled parents raises ConnectionNotEstablished Error

How to reproduce

class UserItem < ActiveRecord::Base
  turntable :user_cluster, :user_id
  sequencer :user_seq
end

class UserEquipmentItem < UserItem; end

> UserEquipmentItem.connection 
# => ActiveRecord::ConnectionNotEstablished: No connection pool with id turntable_pool_proxy::UserEquipmentItem found.