Giter Site home page Giter Site logo

standby's Introduction

Standby - Read from standby databases for ActiveRecord (formerly Slavery)

Build Status

Standby is a simple, easy to use gem for ActiveRecord that enables conservative reading from standby databases, which means it won't automatically redirect all SELECTs to standbys.

Instead, you can do Standby.on_standby { User.count } to send a particular query to a standby.

Background: Probably your app started off with one single database. As it grows, you would upgrade to a primary-standby (or master-slave) replication for redundancy. At this point, all queries still go to the primary and standbys are just backups. With that configuration, it's tempting to run some long-running queries on one of the standbys. And that's exactly what Standby does.

  • Conservative - Safe by default. Installing Standby won't change your app's current behavior.
  • Future proof - No dirty hacks. Simply works as a proxy for ActiveRecord::Base.connection.
  • Simple code - Intentionally small. You can read the entire source and completely stay in control.

Standby works with ActiveRecord 3 or later.

Install

Add this line to your application's Gemfile:

gem 'standby'

And create standby configs for each environment.

development:
  database: myapp_development

development_standby:
  database: myapp_development

By convention, config keys with [env]_standby are automatically used for standby reads.

Notice that we just copied the settings of development to development_standby. For development and test, it's actually recommended as probably you don't want to have replicating multiple databases on your machine. Two connections to the same identical database should be fine for testing purpose.

In case you prefer DRYer definition, YAML's aliasing and key merging might help.

common: &common
  adapter: mysql2
  username: root
  database: myapp_development

development:
  <<: *common

development_standby:
  <<: *common

Optionally, you can use a database url for your connections:

development: postgres://root:@localhost:5432/myapp_development
development_standby: postgres://root:@localhost:5432/myapp_development_standby

At this point, Standby does nothing. Run tests and confirm that nothing is broken.

Usage

To start using Standby, you need to add Standby.on_standby in your code. Queries in the Standby.on_standby block run on the standby.

Standby.on_standby { User.count }   # => runs on standby
Standby.on_standby(:two) { User.count }  # => runs on another standby configured as `development_standby_two`

You can nest on_standby and on_primary interchangeably. The following code works as expected.

Standby.on_standby do
  ...
  Standby.on_primary do
    ...
  end
  ...
end

Alternatively, you may call on_standby directly on the scope, so that the query will be read from standby when it's executed.

User.on_standby.where(active: true).count

Caveat: pluck is not supported by the scope syntax, you still need Standby.on_standby in this case.

Read-only user

For an extra safeguard, it is recommended to use a read-only user for standby access.

development_standby:
  <<: *common
  username: readonly

With MySQL, GRANT SELECT creates a read-only user.

GRANT SELECT ON *.* TO 'readonly'@'localhost';

With this user, writes on a standby should raise an exception.

Standby.on_standby { User.create }  # => ActiveRecord::StatementInvalid: Mysql2::Error: INSERT command denied...

With Postgres you can set the entire database to be readonly:

ALTER DATABASE myapp_development_standby SET default_transaction_read_only = true;

It is a good idea to confirm this behavior in your test code as well.

Disable temporarily

You can quickly disable standby reads by dropping the following line in config/initializers/standby.rb.

Standby.disabled = true

With this line, Standby stops connection switching and all queries go to the primary.

This may be useful when one of the primary or the standby goes down. You would rewrite database.yml to make all queries go to the surviving database, until you restore or rebuild the failed one.

Transactional fixtures

When use_transactional_fixtures is set to true, it's NOT recommended to write to the database besides fixtures, since the standby connection is not aware of changes performed in the primary connection due to transaction isolation.

In that case, you are suggested to disable Standby in the test environment by putting the following in test/test_helper.rb (or spec/spec_helper.rb for RSpec users):

Standby.disabled = true

Upgrading from version 3 to version 4

The gem name has been changed from slavery to standby.

Update your Gemfile

gem 'standby'

Then

  • Replace Slavery with Standby, on_slave with on_standby, and on_master with on_primary
  • Update keys in database.yml (e.g. development_slave to development_standby)

Upgrading from version 2 to version 3

Please note that Standby.spec_key= method has been removed from version 3.

Support for non-Rails apps

If you're using ActiveRecord in a non-Rails app (e.g. Sinatra), be sure to set RACK_ENV environment variable in the boot sequence, then:

require 'standby'

ActiveRecord::Base.configurations = {
  'development' =>          { adapter: 'mysql2', ... },
  'development_standby' =>  { adapter: 'mysql2', ... }
}
ActiveRecord::Base.establish_connection(:development)

Changelog

  • v4.0.0: Rename gem from Slavery to Standby
  • v3.0.0: Support for multiple standby targets (@punchh)
  • v2.1.0: Debug log support / Database URL support / Rails 3.2 & 4.0 compatibility (Thanks to @citrus)
  • v2.0.0: Rails 5 support

standby's People

Contributors

agis avatar asanghi avatar citrus avatar coryodaniel avatar dsinn avatar gaurish avatar jayuen avatar jrforrest avatar kenn avatar loveeachday avatar pnixx avatar wildfiler avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

standby's Issues

ActiveRecord relation inverse_of broken

Hello, I know that this repository is not maintained anymore but I still prefer to warn other developers of the finding we went through.

Whenever we use the standby gem, the file lib/standby/active_record/relation.rb will break the inverse_of your ActiveRecord models.

Let's say you have those two models:

class Note < ApplicationRecord
  belongs_to :musician
end
class Musician < ApplicationRecord
  has_many :notes
end

If you do something like this:

  musician = Notes.last.musician
  musician.name
  #=> 'Bar'
  musician.attributes = { name: 'Foo' }
  musician.notes.last.musician.name
  #=> 'Bar' # Here, we expect it to be 'Foo'

The in-memory change of your model is not reflected anymore through the inverse_of.
You will actually see a call done to the DB to fetch again the musician coming from your last note.

When we comment the lib/standby/active_record/relation.rb file, it works again.

Cheers,

Active Record 4.0.0 error raised when calling to establish_connection: Anonymous class is not allowed

Hi,
Firstly, thank you for creating such a great gem!

Active Record 4.0.0 does not allow an anonymous class to call establish_connection. See this merged pull request:
https://github.com/rails/rails/pull/9002/files
Related to this issue:
rails/rails#8934

When running tests for the slavery gem against Active Record 4.0.0, you'll get 3 failures with the following trace:
RuntimeError:
Anonymous class is not allowed.
# ./lib/slavery.rb:90:in block in slave_connection_holder' # ./lib/slavery.rb:83:ininitialize'
# ./lib/slavery.rb:83:in new' # ./lib/slavery.rb:83:inslave_connection_holder'
# ./lib/slavery.rb:78:in slave_connection' # ./lib/slavery.rb:57:inconnection_with_slavery'
# ./lib/slavery/relation.rb:18:in calculate_with_slavery' # ./spec/slavery_spec.rb:81:inblock (4 levels) in <top (required)>'
# ./lib/slavery.rb:33:in run' # ./lib/slavery.rb:23:inon_slave'
# ./spec/slavery_spec.rb:81:in `block (3 levels) in <top (required)>'

I have written a simple workaround, which I can submit as a pull request, if you'd like:
https://github.com/darrenlevy/slavery/commit/8bc53030b199b33944d406f634af623e141f5d47

Specs Broken?

Hello,
Are the specs broken?

Please see below:

/Users/gaurish/.rvm/rubies/ruby-2.3.1/bin/ruby -I/Users/gaurish/.rvm/gems/ruby-2.3.1/gems/rspec-core-3.5.4/lib:/Users/gaurish/.rvm/gems/ruby-2.3.1/gems/rspec-support-3.5.0/lib /Users/gaurish/.rvm/gems/ruby-2.3.1/gems/rspec-core-3.5.4/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb
DEPRECATION WARNING: alias_method_chain is deprecated. Please, use Module#prepend instead. From module, you can access the original method using super. (called from <class:Relation> at /Users/gaurish/code/repo/slavery/lib/slavery/relation.rb:22)
DEPRECATION WARNING: alias_method_chain is deprecated. Please, use Module#prepend instead. From module, you can access the original method using super. (called from <class:Relation> at /Users/gaurish/code/repo/slavery/lib/slavery/relation.rb:23)
DEPRECATION WARNING: alias_method_chain is deprecated. Please, use Module#prepend instead. From module, you can access the original method using super. (called from singleton class at /Users/gaurish/code/repo/slavery/lib/slavery.rb:13)
.F....FFF

Failures:

  1) Slavery returns value from block
     Failure/Error: calculate_without_slavery(operation, column_name, options)

     ArgumentError:
       wrong number of arguments (given 3, expected 2)
     # /Users/gaurish/.rvm/gems/ruby-2.3.1/gems/activerecord-5.0.0.1/lib/active_record/relation/calculations.rb:110:in `calculate'
     # ./lib/slavery/relation.rb:18:in `calculate_with_slavery'
     # /Users/gaurish/.rvm/gems/ruby-2.3.1/gems/activerecord-5.0.0.1/lib/active_record/relation/calculations.rb:40:in `count'
     # /Users/gaurish/.rvm/gems/ruby-2.3.1/gems/activerecord-5.0.0.1/lib/active_record/querying.rb:13:in `count'
     # ./spec/slavery_spec.rb:14:in `block (3 levels) in <top (required)>'
     # ./lib/slavery.rb:41:in `run'
     # ./lib/slavery.rb:35:in `on_master'
     # ./spec/slavery_spec.rb:14:in `block (2 levels) in <top (required)>'

  2) Slavery works with scopes
     Failure/Error: calculate_without_slavery(operation, column_name, options)

     ArgumentError:
       wrong number of arguments (given 3, expected 2)
     # /Users/gaurish/.rvm/gems/ruby-2.3.1/gems/activerecord-5.0.0.1/lib/active_record/relation/calculations.rb:110:in `calculate'
     # ./lib/slavery/relation.rb:18:in `calculate_with_slavery'
     # /Users/gaurish/.rvm/gems/ruby-2.3.1/gems/activerecord-5.0.0.1/lib/active_record/relation/calculations.rb:40:in `count'
     # /Users/gaurish/.rvm/gems/ruby-2.3.1/gems/activerecord-5.0.0.1/lib/active_record/querying.rb:13:in `count'
     # ./spec/slavery_spec.rb:72:in `block (2 levels) in <top (required)>'

  3) Slavery configuration connects to master if slave configuration not specified
     Failure/Error: calculate_without_slavery(operation, column_name, options)

     ArgumentError:
       wrong number of arguments (given 3, expected 2)
     # /Users/gaurish/.rvm/gems/ruby-2.3.1/gems/activerecord-5.0.0.1/lib/active_record/relation/calculations.rb:110:in `calculate'
     # ./lib/slavery/relation.rb:18:in `calculate_with_slavery'
     # /Users/gaurish/.rvm/gems/ruby-2.3.1/gems/activerecord-5.0.0.1/lib/active_record/relation/calculations.rb:40:in `count'
     # /Users/gaurish/.rvm/gems/ruby-2.3.1/gems/activerecord-5.0.0.1/lib/active_record/querying.rb:13:in `count'
     # ./spec/slavery_spec.rb:98:in `block (4 levels) in <top (required)>'
     # ./lib/slavery.rb:41:in `run'
     # ./lib/slavery.rb:31:in `on_slave'
     # ./spec/slavery_spec.rb:98:in `block (3 levels) in <top (required)>'

  4) Slavery configuration raises error when no configuration found
     Failure/Error: expect { Slavery.on_slave { User.count } }.to raise_error(Slavery::Error)

       expected Slavery::Error, got #<ArgumentError: wrong number of arguments (given 3, expected 2)> with backtrace:
         # /Users/gaurish/.rvm/gems/ruby-2.3.1/gems/activerecord-5.0.0.1/lib/active_record/relation/calculations.rb:110:in `calculate'
         # ./lib/slavery/relation.rb:18:in `calculate_with_slavery'
         # /Users/gaurish/.rvm/gems/ruby-2.3.1/gems/activerecord-5.0.0.1/lib/active_record/relation/calculations.rb:40:in `count'
         # /Users/gaurish/.rvm/gems/ruby-2.3.1/gems/activerecord-5.0.0.1/lib/active_record/querying.rb:13:in `count'
         # ./spec/slavery_spec.rb:105:in `block (5 levels) in <top (required)>'
         # ./lib/slavery.rb:41:in `run'
         # ./lib/slavery.rb:31:in `on_slave'
         # ./spec/slavery_spec.rb:105:in `block (4 levels) in <top (required)>'
         # ./spec/slavery_spec.rb:105:in `block (3 levels) in <top (required)>'
     # ./spec/slavery_spec.rb:105:in `block (3 levels) in <top (required)>'

Finished in 0.04407 seconds (files took 0.53068 seconds to load)
9 examples, 4 failures

Failed examples:

rspec ./spec/slavery_spec.rb:13 # Slavery returns value from block
rspec ./spec/slavery_spec.rb:71 # Slavery works with scopes
rspec ./spec/slavery_spec.rb:95 # Slavery configuration connects to master if slave configuration not specified
rspec ./spec/slavery_spec.rb:101 # Slavery configuration raises error when no configuration found

`Standby#on_standby` can block unnecessarily

Standby::Base.inside_transaction? uses ActiveRecord::Base.connection on the primary to check whether access to the standby is occurring from within a transaction. ActiveRecord::Base.connection looks for a cached connection belonging to the current thread and, if it doesn't find one, attempts to checkout a connection from the connection pool. If the connection pool is exhausted, it blocks for up to ConnectionPool#checkout_timeout (default 5 seconds).

I believe this can be avoided if the current thread does not already have an active connection to the primary. In that case, I assume it is not possible to be in a transaction.

Why Slavery is not reading from the slave db.

I am setting up slavery, and pointing development_slave to an empty database. Then I fire up rails c and run Slavery.on_slave { User.find(3) }. What I expect is, it ends up with ActiveRecord::NotFound but it find the correct record (it has to be from my master db). And I could see this log coming from development.log: [slave] User Load (36.4ms) SELECT users.* FROM usersWHEREusers.id = 3 LIMIT 1, indicating it goes to the Slave.

And I could be sure Slavery.disabled is nil.

Here's my database.yml:

default: &default
  adapter: mysql2
  encoding: utf8mb4
  charset: utf8mb4
  collation: utf8mb4_general_ci
  reconnect: false
  pool: 16
  username: root
  password: admin
  host: localhost
  strict: false

development: &development
  <<: *default
  host: <%= ENV.fetch("MYSQL_HOST", "127.0.0.1") %>
  port: <%= ENV.fetch("MYSQL_PORT", 3306) %>
  username: <%= ENV.fetch("MYSQL_USERNAME", "root") %>
  password: <%= ENV.fetch("MYSQL_PASSWORD", "admin") %>
  database: <%= ENV.fetch("MYSQL_DATABASE", "maleskine") %>
  strict: false

development_slave:
  <<: *default
  host: 127.0.0.1
  port: 3306
  username: root
  password: admin
  database: maleskine_test
  strict: false

Is there anything that I've done wrong?

Query on slave returning wrong results in test

Hey,

After switching some queries to be sent to the slaves, I've noticed a specific test failing due to the on_slave scope returning wrong results:

[2] pry(#<RSpec::ExampleGroups::ShopsController::GETIndex>)> Shop.where("created_on > ?", 5.days.ago)
[master]  Shop Load (1.2ms)  SELECT `shops`.* FROM `shops` WHERE (created_on > '2017-01-14 11:54:33.639784')
 # => [#<Shop:0x007fa103931290>]
# returns the `Shop` record as expected, test succeeds

[3] pry(#<RSpec::ExampleGroups::ShopsController::GETIndex>)> Shop.on_slave.where("created_on > ?", 5.days.ago)
[slave]  Shop Load (0.8ms)  SELECT `shops`.* FROM `shops` WHERE (created_on > '2017-01-14 11:56:41.160893')
# => [] 
# test fails

Oddly enough, Slavery.on_slave still sends the queries to the master:

[4] pry(#<RSpec::ExampleGroups::ShopsController::GETIndex>)> Slavery.on_slave { Shop.where("created_on > ?", 5.days.ago) }
[master]  Shop Load (0.9ms)  SELECT `shops`.* FROM `shops` WHERE (created_on > '2017-01-14 11:58:43.070048')
# => [#<Shop:0x007fa103931290>]

The failing test is roughly the following:

  describe "foo" do
    let!(:shop) { FactoryGirl.create(:shop) }
    
    it "foooooo" do
      shop.update_attribute :created_on, 3.days.ago
      shop.update_attribute :updated_on, 3.days.ago

      get :index, { :since => 5.days.ago }
      expect(assigns(:shops)).to eq([shop])
    end
  end

I've also tried reducing the query to Shop.count/Shop.on_slave.count, same issue: the result is 1 vs. 0.

Any ideas?

Thanks

Best way to handle fallback

In your opinion, what would be the best way to handle fallback to primary in case the replicas are down?
Let's say the replica is restarted or not available, or just when ActiveRecord cant connect or a timeout occurs. Similar to Standby.disabled = true but coming from a rescue block.

Consider changes to master/slave terminology and project name

I understand the terms master and slave have been used in database design for decades, but I believe there are replacement terms that aren't politically charged and historically problematic, such as "leader/follower", or "primary/replica". Other communities have run into these issues and made common sense changes, such as Django, Drupal, CouchDB, etc.

I love the Ruby community and would like to see it lead on issues such as this, to be pro-active about problematic uses of language as a barrier for people of color or other marginalised communities from joining the community and feeling like they are included.

If you agree with me I'll be happy to help the migration to the new terminology and will submit a PR. There is also some good advice over on StackOverflow about the steps to take when renaming a gem.

How disable Slavery for single model?

I have disable reading from slave for single model, for example Action.

class Action < ActiveRecord::Base
  self.slavery_dasabled = true
end

Slavery.on_slave do
  users = User.count
  User.each do |user|
    actions = user.actions.count
  end
end

already initialized constant SlaverySlaveConnectionHolder

After setting up a very basic slave in development I tried running a simple Slavery.on_slave{ User.count } and I get a recursive slavery-3.0.0/lib/slavery/connection_holder.rb:22: warning: already initialized constant SlaverySlaveConnectionHolder error

I am on rails 4.2.10 with ruby 2.3.3p222

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.