Giter Site home page Giter Site logo

frozen_record's Introduction

FrozenRecord

Build Status Gem Version

Active Record-like interface for read only access to static data files of reasonable size.

Installation

Add this line to your application's Gemfile:

gem 'frozen_record'

And then execute:

$ bundle

Or install it yourself as:

$ gem install frozen_record

Models definition

Just like with Active Record, your models need to inherits from FrozenRecord::Base:

class Country < FrozenRecord::Base
end

But you also have to specify in which directory your data files are located. You can either do it globally

FrozenRecord::Base.base_path = '/path/to/some/directory'

Or per model:

class Country < FrozenRecord::Base
  self.base_path = '/path/to/some/directory'
end

FrozenRecord has two built-in backends, for JSON and YAML. Backends are classes that know how to load records from a static file.

The default backend is YAML and it expects a file that looks like this:

- id: 'se'
  name: 'Sweden'
  region: 'Europe'
  language: 'Swedish'
  population: 10420000
- id: 'de'
  name: 'Germany'
  region: 'Europe'
  language: 'German'
  population: 83200000

#

You can also specify a custom backend:

class Country < FrozenRecord::Base
  self.backend = FrozenRecord::Backends::Json
end

Custom backends

A custom backend must implement the methods filename and load as follows:

module MyCustomBackend
  extend self

  def filename(model_name)
    # Returns the file name as a String
  end

  def load(file_path)
    # Reads file and returns records as an Array of Hash objects
  end
end

Query interface

FrozenRecord aim to replicate only modern Active Record querying interface, and only the non "string typed" ones.

# Supported query interfaces
Country.
  where(region: 'Europe').
  where.not(language: 'English').
  where(population: 10_000_000..).
  order(id: :desc).
  limit(10).
  offset(2).
  pluck(:name)

# Non-supported query interfaces
Country.
  where('region = "Europe" AND language != "English"').
  order('id DESC')

Scopes

Basic scope :symbol, lambda syntax is now supported in addition to class method syntax.

class Country
  scope :european, -> { where(continent: 'Europe' ) }

  def self.republics
    where(king: nil)
  end

  def self.part_of_nato
    where(nato: true)
  end
end

Country.european.republics.part_of_nato.order(id: :desc)

Supported query methods

  • where
  • where.not
  • order
  • limit
  • offset

Supported finder methods

  • find
  • first
  • last
  • to_a
  • exists?

Supported calculation methods

  • count
  • pluck
  • ids
  • minimum
  • maximum
  • sum
  • average

Indexing

Querying is implemented as a simple linear search (O(n)). However if you are using Frozen Record with larger datasets, or are querying a collection repeatedly, you can define indices for faster access.

class Country < FrozenRecord::Base
  add_index :name, unique: true
  add_index :continent
end

Composite index keys are not supported.

The primary key isn't indexed by default.

Rich Types

The attribute method can be used to provide a custom class to convert an attribute to a richer type. The class must implement a load class method that takes the raw attribute value and returns the deserialized value (similar to ActiveRecord serialization).

class ContinentString < String
  class << self
    alias_method :load, :new
  end
end

Size = Struct.new(:length, :width, :depth) do
  def self.load(value) # value is lxwxd eg: "23x12x5"
    new(*value.split('x'))
  end
end

class Country < FrozenRecord::Base
  attribute :continent, ContinentString
  attribute :size, Size
end

Limitations

Frozen Record is not meant to operate on large unindexed datasets.

To ensure that it doesn't happen by accident, you can set FrozenRecord::Base.max_records_scan = 500 (or whatever limit makes sense to you), in your development and test environments. This setting will cause Frozen Record to raise an error if it has to scan more than max_records_scan records. This property can also be set on a per model basis.

Configuration

Reloading

By default the YAML files are parsed once and then cached in memory. But in development you might want changes to be reflected without having to restart your application.

For such cases you can set auto_reloading to true either globally or on a model basis:

FrozenRecord::Base.auto_reloading = true # Activate reloading for all models
Country.auto_reloading # Activate reloading for `Country` only

Testing

Testing your FrozenRecord-backed models with test fixtures is made easier with:

require 'frozen_record/test_helper'

# During test/spec setup
test_fixtures_base_path = 'alternate/fixture/path'
FrozenRecord::TestHelper.load_fixture(Country, test_fixtures_base_path)

# During test/spec teardown
FrozenRecord::TestHelper.unload_fixtures

Here's a Rails-specific example:

require 'test_helper'
require 'frozen_record/test_helper'

class CountryTest < ActiveSupport::TestCase
  setup do
    test_fixtures_base_path = Rails.root.join('test/support/fixtures')
    FrozenRecord::TestHelper.load_fixture(Country, test_fixtures_base_path)
  end

  teardown do
    FrozenRecord::TestHelper.unload_fixtures
  end

  test 'countries have a valid name' do
  # ...

Contributors

FrozenRecord is a from scratch reimplementation of a Shopify project from 2007 named YamlRecord. So thanks to:

frozen_record's People

Contributors

0xflotus avatar andyw8 avatar bdewater avatar bernardoamc avatar bslobodin avatar byroot avatar casperisfine avatar cbothner avatar chrisbutcher avatar christianblais avatar dazworrall avatar dependabot[bot] avatar gmalette avatar jtgrenz avatar katayounhillier avatar kmcphillips avatar larouxn avatar ltk avatar lucasuyezu avatar morriar avatar natmorcos avatar olleolleolle avatar petergoldstein avatar rafaelfranca avatar ragalie avatar richardplatel avatar sandstrom avatar selesse avatar serioushaircut avatar ttasanen avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

frozen_record's Issues

Random failures in version 0.26.1

When running tests in Bloodhound's main branch I'm seeing random test failures after upgrading to version 0.26.1.

Example test with backtrace

TrackerTest#test_#poll_works_for_Platform_carriers
    Minitest::UnexpectedError: NoMethodError: undefined method `[]' for nil:NilClass

      record = @index[value]
                     ^^^^^^^
    ~/.gem/ruby/3.1.1/gems/frozen_record-0.26.1/lib/frozen_record/index.rb:64 in `lookup`
    ~/.gem/ruby/3.1.1/gems/frozen_record-0.26.1/lib/frozen_record/base.rb:140 in `block in find_by`
    ~/.gem/ruby/3.1.1/gems/frozen_record-0.26.1/lib/frozen_record/base.rb:137 in `each`
    ~/.gem/ruby/3.1.1/gems/frozen_record-0.26.1/lib/frozen_record/base.rb:137 in `find_by`
    ~/.gem/ruby/3.1.1/gems/frozen_record-0.26.1/lib/frozen_record/base.rb:132 in `find`
    app/models/shipify/carrier.rb:76 in `block in providers`
    app/models/shipify/carrier.rb:76 in `map`
    app/models/shipify/carrier.rb:76 in `providers`
    app/models/tracker.rb:79 in `poll`
    ~/.gem/ruby/3.1.1/gems/sorbet-runtime-0.5.10217/lib/types/private/methods/call_validation_2_7.rb:654 in `bind_call`
    ~/.gem/ruby/3.1.1/gems/sorbet-runtime-0.5.10217/lib/types/private/methods/call_validation_2_7.rb:654 in `block in create_validator_method_medium0`
    test/models/tracker_test.rb:118 in `block in <class::TrackerTest>`
  Rerun: dev test test/models/tracker_test.rb:111

Fails in this method

https://github.com/Shopify/bloodhound/blob/main/app/models/shipify/carrier.rb#L73-L77

    def providers
      return [] if provider_ids.nil?

      provider_ids.map { |id| ShipifyProvider.find(id) }
    end

Can't replicate it in CI so I believe it has to do with eager loading, since it's disabled in development and test when in CI.

Better handle larger databases

FrozenRecord was never meant to handle "large" datasets. However it's not always evident what a large dataset is, and it's also easy for a dataset to grow over time and severely impact performance.

It's also often not obvious when an index could help.

Reject slow queries in test

For the discoverability of performance issues, I think we should add an option "slow query" limit, that can be turned on in test environments. e.g. FrozenRecord.max_scan_size = 1000 that would immediately raise if you do a query that end up scanning more than 1k records.

This should prompt owners to either find a proper index, or to use something other than FrozenRecord.

That one is totally trivial to implement.

Sqlite backend?

For cases where there's just no possible indexes, we could explore a more efficient storage. I've seen some app generate static sqlite dbs on deploy, the usability was a bit shit, but I think we could have a FrozenRecord backend that pretty much does this automatically with little to no change to the interface.

That one need some research, it may or may not work.

cc @alexcwatt

Can an ActiveRecord reference a FrozenRecord?

Suppose you use FrozenRecord to define a class named Department with 12 instances, each with a unique "id". Then you want the class Person to belong_to a Department. Is that possible?

If so, can you show an example of the syntax in the two classes to make that happen?

If not, is that on your roadmap?

Advanced filtering

I'm in the process of migrating a bunch of models that are backed by YAML files. We had some homegrown code that basically did what frozen record does, but not as elegant as this library.

One thing that has cropped up though, is some filtering limitations.

We have one model with scopes that does greater-than/less-than comparisons of time.

More specifically, we're doing something like this with our homegrown code:

scope :in_use, lambda {
  current_time = Time.now
  records.find_all { |c| !c.withdrawn_at || c.withdrawn_at > current_time }
}

I haven't figured out a way of moving this over to FrozenRecord scopes, since the query interface doesn't support a gt operator in where.

So one idea I wanted to throw out there:

How about adding support for .find_by_lambda(lambda { |r| … }) or .find_all { |r| … }, for those edge cases where the regular querying interface isn't enough?

Is this something you'd be open to exploring? In that case, I can submit a more fullfledged example of what the API could look like.

Release 1.0

This seem like an excellent and mature project.

How about releasing 1.0? To signal that it's mature and ready for external use.

(also minor OCD-wish; how about disabling wiki and project tabs, since they aren't used)

Support other source formats like json?

Do you have any plans to support other source formats than YAML? This nice little gem does exactly what I need, but now I just have to convert json files to yaml in my current project.

I was looking at the source and seems it would be pretty easy to add support for other source formats as well..

I would not tie this strictly to yaml and json, but maybe allow user to pass a proc that could return any Enumerable with the data? But still defaulting to YAML.load_file

If you are interested I could hack something together and submit a PR?

Acceptable use cases?

Hi @byroot,

Thanks a lot for your work. 💪

We also use a notion of FrozenRecord at my work, but I have a hard time figuring out when to use them, when not and providing clear guidelines.
I would be very interested in your opinion on this, if you have some time to spare.

Thanks in advance!

[Discussion] API to declare a prefiltered scope.

Let me start with the current use case and how we are dealing with different scopes.

Right now we have a model and we want to search records from this model in two distinct ways. To make this more clear here is the way we are doing it:

def MyClass
  def self.search(search_term)
    PrefixSearchScope.new(self).search(search_term)
  end

  def self.prefix_search
    @prefix_search ||= PrefixSearch.new(all, fields: [:label])
  end

  class PrefixSearchScope < FrozenRecord::Scope
    def initialize(*)
      @search_term = nil
      super
    end

    def search!(search_term)
      @search_term = search_term
      self
    end

    def search(search_term)
      spawn.search!(search_term)
    end

    protected

    def select_records(records)
      if @search_term.present?
        super(@klass.prefix_search.find(@search_term))
      else
        super(records)
      end
    end
  end
end

It would be interesting to be able to declare something like .with_scoped_subset in our class that returns a FrozenRecord::Scope for a predefined set of records instead of calling .load_records.

Example:

def self.with_scoped_subset(subset)
  # Returns a `FrozenRecord::Scope` over `subset`
end

def self.search(search_term)
  search_results = prefix_search.find(search_term)
  with_scoped_subset(search_results)
end

[Enhancement] Forcing a reload of records

I recently wanted to write a test for a FrozenRecord model that uses different fixtures (and therefore different backing YAML files) in tests than in production. This was harder than I'd like it to be and it forced me to set an instance variable from the outside to force FrozenRecord to reload the records.

In my test class:

class MyModelTest < ActiveSupport::TestCase
  test ".complicated_finder_method does something" do
    with_test_data do
      assert_equal 1, MyModel.complicated_finder_method(...).size
    end
  end

  def with_test_data(name)
      MyModel.stubs(:file_path).returns(Rails.root.join('test/files/data/#{name}.yml'))
      # need to reset the records to force it to read the file again
      MyModel.instance_variable_set(:@records, nil)
      MyModel.load_records
      yield
    ensure
      MyModel.unstub(:file_path)
      MyModel.instance_variable_set(:@records, nil)
      MyModel.load_records
    end
end

I can live with stubbing the public file_path method but I don't like the MyModel.instance_variable_set(:@records, nil). Any chance we could get support for this use case into FrozenRecord?

Release 1.0

This is a stable and popular project. I'd recommend signaling that, by releasing 1.0!

image

Support for splitting a FrozenRecord across multiple files?

Hi @byroot / @casperisfine, been a while!

At my new place we've been thinking about a use case with a few very large records (with very long list properties) and it feels kind of natural to put each record in its own file, e.g. something like:

config/frozen_records/role/viewer.yml
config/frozen_records/role/editor.yml
config/frozen_records/role/admin.yml
...

where the actual ruby class would be class Role < FrozenRecord::Base.

This is sort-of / hackily achievable already by writing a custom backend, but the backend API and main gem code still assumes there's a single backing file. I guess you could also fake it with ERB code that uses ruby to iterate / load / inline all the files in the directory, but it runs into similar issues with reloading in dev, etc.

Thoughts on the use case? Any interest or tips for a PR for something like this, if we move forward with the idea? I don't know how deep the single-file assumption is baked right now.

cc @qr8r

Lazy loading of records

Hey!

First of all, thanks for your gem!

This change 64871cb have broken our app, because of our custom backend have nontrivial records loader using dependencies related to ActionView and so on which are not fully loaded at application boot time. And either the application boot time is increased too. So my question is - what do you this about adding an option to a configuration which will be taken into account when this initializer is executed?

Add changelog/releases notes

For better integration with things like Dependabot, it'd be great if a Changelog or Release notes could be provided with new Frozen Record releases.

At the moment developers have to grok commits and/or PR's to understand the changes introduced in each version which is slow and subjective.

Support for find with a block is missing

FrozenRecord.Scope#find, in frozen_record/scope.rb, does not support that a block is passed to it

    def find(id)
      raise RecordNotFound, "Can't lookup record without ID" unless id

      scope = self
      if @limit || @offset
        scope = limit(nil).offset(nil)
      end
      scope.find_by_id(id) or raise RecordNotFound, "Couldn't find a record with ID = #{id.inspect}"
    end

It seems it should, as ActiveRecord.FinderMethods#find in active_record/relation/finder_methods.rb

    def find(*args)
     return super if block_given?
     find_with_ids(*args)
   end

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.