Giter Site home page Giter Site logo

customink / activerecord-aurora-serverless-adapter Goto Github PK

View Code? Open in Web Editor NEW
66.0 39.0 7.0 539 KB

ActiveRecord Adapter for Amazon Aurora Serverless

Home Page: https://technology.customink.com/blog/2020/01/03/migrate-your-rails-app-from-heroku-to-aws-lambda/

License: MIT License

Ruby 85.42% Shell 4.11% JavaScript 0.88% TypeScript 9.60%
aurora activerecord activerecord-adapter serverless ruby-on-rails aws-lambda fullstack-serverless

activerecord-aurora-serverless-adapter's Introduction

ActiveRecord Aurora Serverless Adapter

Aurora Serverless on Rails

CI Status

Lamby: Simple Rails & AWS Lambda Integration using Rack.

Simple ActiveRecord Mysql2 adapter extensions to allow Rails to use AWS Aurora Serverless via the Aws::RDSDataService::Client interface. Perfect if you are using Lamby to deploy your Rails applications to AWS Lambda.

Lamby: Simple Rails & AWS Lambda Integration using Rack.

Highlights

This gem allows Rails to seamless use

  • Tested on Rails v5.2 and v6.0.
  • No need for the mysql2 gem at all!
  • Developed and tested with Aurora Serverless MySQL v5.6.
  • Emoji support via utf8mb4. Please configure your cluster's parameter group. See our CDK Stack for examples.

Here are some misc features that work differently for the Mysql2 adapter under Aurora Serverless.

  • Multiple schemas are not supported.
  • Prepared statements are not supported.
  • Batch statements are not supported.
  • Advisory locks are not supported.

Usage

Add the gem to your Gemfile. Remember, You DO NOT have to add the mysql2 gem. This adapter will replace the MySQL connection with the Aws::RDSDataService::Client API calls.

gem 'activerecord-aurora-serverless-adapter'

Assuming you have created your database with the Data API enabled and configured your secrets then configure your database.yml file like so.

database: 'mydatabase'
adapter: aurora_serverless
secret_arn: arn:aws:secretsmanager:us-east-1:123456789012:secret:Secret-kd2ASwipxeWw-Bdsiww
resource_arn: arn:aws:rds:us-east-1:123456789012:cluster:mydatabase

lease feel free to use any valid ActiveRecord configuration in your database.yml file. We also allow all Aws::RDSDataService::Client options here too! Any valid option will be passed to Aws::RDSDataService::Client.new.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/customink/activerecord-aurora-serverless-adapter. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

Testing

Cloning the repo and running the tests locally is super easy assuming you have:

  1. Docker Installed
  2. AWS Account Configured

These commands will use Docker to setup a node runtime leveraging AWD CDK to deploy an Aurora Serverless stack. Note, you may be to use/set AWS_PROFILE for the deploy command.

$ ./bin/bootstrap
$ export AASA_MASTER_USER=admin
$ export AASA_MASTER_PASS=supersecret
$ ./test/bin/deploy-aurora

The outputs of this deployed stack will contain an AASASecretArn and a AASAAuroraClusterArn value. Please place these into the local .env file in the following format where {{ Value }} is replaced.

AASA_SECRET_ARN={{ AASASecretArn }}
AASA_RESOURCE_ARN2={{ AASAResourceArn }}

AASA_SECRET_ARN={{ AASASecretArn2 }}
AASA_RESOURCE_ARN_2={{ AASAResourceArn2 }}

Finally, assuming you have your default AWS account setup with full access to your account, now you can run the tests. The AWS_PROFILE can be used here and set in .env file as needed or you can use the AASAUserAccessKeyId and AASAUserSecretAccessKey outputs as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environments set in .env too.

$ ./bin/test

Working With The CDK App/Stack

To work with the CDK project within the test directory, run the following commands.

$ docker-compose \
  --project-name aasa \
  run \
  cdk \
  bash

$ cd ./test/aurora-serverless

From here you can run npm, tsc, cdk or whatever commands are needed.

License

The gem is available as open source under the terms of the MIT License.

Code of Conduct

Everyone interacting in the adapter project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the code of conduct.

activerecord-aurora-serverless-adapter's People

Contributors

drinks avatar metaskills avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

activerecord-aurora-serverless-adapter's Issues

60 second Timeout Issue w/ Hang

It appears as if continueAfterTimeout needs to be set.

I keep getting this error:
ActiveRecord::StatementInvalid: Request timed out

And the runs including this error are 60 seconds + normal execution time.

This error happens at what appears to random places in the code (no one query is throwing this). However, it normally appears when we are under load, such at happened this morning when we spiked from 22 to 75 average concurrent executions and then to 99 within 60 seconds. The queries that it errors out on, I would consider to be simple select, insert, update. I am able to log in via SQL 3309 and run commands like I normally would without issue, right after the spike started and the errors started flowing. This error only happened to about 1% of the requests during this time period.

Please review the following links:
https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/data-api.html
https://forums.aws.amazon.com/message.jspa?messageID=946929

Please review the below concurrent executions graph. Right when the spike starts, the errors start to flow.
Screen Shot 2021-01-10 at 9 37 06 AM

Also, please review these Serverless RDS metrics. You can see that we are not close to maxing out our cluster and that CPU stays at under 20% the entire time.
Screen Shot 2021-01-10 at 9 38 28 AM

Support Reconnects

Very common to allow the Aurora Serverless autoPause after some period of time. When this happens and Rails tries to connect, you will see an error like this pretty quickly.

Aws::RDSDataService::Errors::BadRequestException (Communications link failure)
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.

At one time I had code in place to handle this but removed it due to the issues from #8 for flaky tests because I assumed it was due to the Retriable (https://github.com/kamui/retriable) gem. It was not, so it is safe to add this back. Here is quick implementation I had that we can add back. I liked this method since it meant we only used retrys for the initial connect and not all the time for each execute statement.

def execute_statement_retry(sql)
  if @connected
    execute_statement(sql)
  else
    auto_paused_retry { execute_statement(sql) }
  end
end

def execute_statement(sql)
  raw_client.execute_statement({
    # ...
  }).tap do |r|
    @connected = true
    # ...
  end
end

private

def auto_paused_retry
  ekls = Aws::RDSDataService::Errors::BadRequestException
  emsg = /last packet sent successfully to the server was/
  rtry = Proc.new { Rails.logger.info 'Aurora auto paused, retrying...' }
  optn = { on: { ekls => emsg }, on_retry: rtry, tries: 10 }
  Retriable.retriable(optn) { yield }
end

Remove Flaky Test Workarounds

If we are happy that things are more green with the foreign key check but, I'd like to see if we can:

  • Remove the undefined coercion of all ConcurrentTransactionTest cases.
  • Move to 1 vs 3 ActiveRecord test case batches in bin/_test file.
  • Maybe move more tests together?

Command rails db:create doesn't work

Environment

  • Rails 6.1.3.1
  • MySQL 5.7

Issue

Rails command db:create doesn't work. After manually create db by DDL in MySQL console, I can correctly connect to the database in my Rails console.

$ bundle exec rails db:create
Rake tasks not supported by 'aurora_serverless' adapter
Couldn't create 'myapp_development' database. Please check your configuration.
rails aborted!
ActiveRecord::Tasks::DatabaseNotSupported: Rake tasks not supported by 'aurora_serverless' adapter
/myapp/bin/rails:5:in `<top (required)>'
/myapp/bin/spring:10:in `require'
/myapp/bin/spring:10:in `block in <top (required)>'
/myapp/bin/spring:7:in `<top (required)>'
Tasks: TOP => db:create
(See full trace by running task with --trace)
make: *** [create] Error 1

Auto pageinate requests

As our dataset grows, we continue to get this error:
ActiveRecord::StatementInvalid (Database returned more than the allowed response size limit)

As I understand, AWS has a limit of 1MB per return.

Using order & limit, it would be nice to turn on a config flag to auto grab all the records, and not have to manually write the method each time to loop.

Unable to sign request without credentials set

Greetings ! I've been unable to open a connection to an RDS Aurora database, and I'm no expert on AWS API, so I might've been missing something obvious...

But it seems to me like Aws::RDSDataService::Client requires a credentials parameter. The readme doesn't specify how these credentials are supposed to be given. I'm not sure it can be done through the config file.

I've been able to solve the issue by patching this gem, by editing lib/active_record/connection_adapters/aurora_serverless/client.rb and adding the following line just before the instantiation of the Aws::RDSDataService::Client:

  options[:credentials] = Aws::Credentials.new options[:access_key]["id"], options[:access_key]["secret"]

I am then able to add my credentials to my config file as such:

aurora: &aurora
  adapter: aurora_serverless
  region: 'eu-west-2'
  access_key:
    id: "access_key_id"
    secret: "secret_access_key"
  resource_arn: "---"
  secret_arn: "---"

And it seems to work (I don't have access to any aws resource yet, but it seems to work).
I'm wondering:

  • If there's another way to do that, shouldn't we talk about it in the README ?
  • Or should I make a PR so we can work on implementing that in the gem ? And if so, any hints on how you'd want this to be done ?

Remove Data API SDK CLI Calls

We had to add a few hacks in order to get the Data API turned on for the test cluster. Thankfully, it should be done soon. Here is the source CDK GitHub issue. aws/aws-cdk#5216

And here are the two places we need to remove/change when we upgrade CDK.

Rails v5 Support

Leverage the RAILS_VERSION environment variable to test other Rails versions.

Arguements error when running migrations

I'm getting an arguments error when trying to connect after following the instructions in the README.

error

➜  wevote git:(master) ✗ bundle exec rake db:migrate
rake aborted!
ArgumentError: wrong number of arguments (given 1, expected 0)
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-aurora-serverless-adapter-6.0.1/lib/active_record/connection_adapters/aurora_serverless/mysql2/result.rb:35:in `each_object'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-aurora-serverless-adapter-6.0.1/lib/active_record/connection_adapters/aurora_serverless/mysql2/result.rb:29:in `each'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/mysql2_adapter.rb:86:in `each_hash'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:116:in `each'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:116:in `map'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:116:in `columns'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/schema_cache.rb:112:in `block in columns'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/schema_cache.rb:111:in `fetch'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/schema_cache.rb:111:in `columns'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/schema_cache.rb:120:in `block in columns_hash'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/schema_cache.rb:119:in `fetch'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/schema_cache.rb:119:in `columns_hash'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/model_schema.rb:553:in `load_schema!'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/attributes.rb:250:in `load_schema!'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/model_schema.rb:539:in `block in load_schema'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/model_schema.rb:536:in `synchronize'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/model_schema.rb:536:in `load_schema'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/model_schema.rb:392:in `columns_hash'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/relation/query_methods.rb:1313:in `arel_column'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/relation/query_methods.rb:1437:in `order_column'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/relation/query_methods.rb:1406:in `block in preprocess_order_args'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/relation/query_methods.rb:1403:in `map!'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/relation/query_methods.rb:1403:in `preprocess_order_args'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/relation/query_methods.rb:360:in `order!'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/relation/query_methods.rb:355:in `order'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/querying.rb:22:in `order'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/schema_migration.rb:46:in `all_versions'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1282:in `load_migrated'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1278:in `migrated'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1314:in `ran?'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1260:in `block in runnable'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1260:in `reject'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1260:in `runnable'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1302:in `migrate_without_lock'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1253:in `migrate'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1086:in `up'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1061:in `migrate'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <main>'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/railties/databases.rake:90:in `each'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <main>'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/appsignal-3.0.9/lib/appsignal/integrations/rake.rb:7:in `execute'
/Users/antarr/code/github/wevote/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/Users/antarr/.rbenv/versions/3.0.1/bin/bundle:23:in `load'
/Users/antarr/.rbenv/versions/3.0.1/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)

Gemfile

gem 'activerecord-aurora-serverless-adapter'
gem 'mysql2-lambda'
# gem 'mysql2', '0.5.2'

config/database.yml

development:
  adapter: aurora_serverless
  # encoding: unicode
  database: development_one
  # pool: 50
  secret_arn: arn:aws:secretsmanager:us-east-2:0000000000:development/mysql-yOE8bY
  resource_arn: arn:aws:rds:us-east-2:0000000000:cluster:serverless-mysql

Snip20211019_1

Support PostgreSQL

FORWARD: I'm not sure this work is worth it given RDS proxy is in preview. https://aws.amazon.com/about-aws/whats-new/2019/12/amazon-rds-proxy-available-in-preview/ However, if someone does want to push on this work, here are some thoughts since Aurora Serverless does support PostgreSQL.

During our initial work (#6) we did leave a good module & client structure in place so that we could mixing PostgreSQL client hacks as needed. My biggest question is, how would we handle loading, move the requires to the core ConnectionHandler?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.