evadne/etso

Ecto 3 adapter allowing use of Ecto schemas held in ETS tables

Etso

Etso is an ETS adapter, allowing you to use Ecto schemas with ETS tables.

Within this library, a bare-bones Ecto Adapter is provided. The Adapter transparently spins up ETS tables for each Ecto Repo and Schema combination. The tables are publicly accessible to enable concurrency, and tracked by reference to ensure encapsulation. Each ETS table is spun up by a dedicated Table Server under a shared Dynamic Supervisor.

For a detailed look as to what is available, check out Northwind Repo Test.

Highlights & Benefits

Key highlights of this library are:

• It transparently handles translations between Ecto.Schema structs and ETS tuples.
• It knows how to form ETS Match Specifications for simple queries.

Key points to consider when adopting this library are:

• It is suitable for rapid retrieval of simply formatted data, for example presets that do not change frequently.
• It is not suitable for use as a full-fledged data layer unless your requirements are simple.

Feature Coverage

The following features are working:

• Basic query scenarios (C / R / U / D)
  • Insert all (c:Ecto.Repo.insert_all/3, etc.)
  • Insert one (c:Ecto.Repo.insert/2, etc.)
  • Get all (c:Ecto.Repo.all/2, etc.)
  • Get by ID (c:Ecto.Repo.get/3, etc.)
  • Get where (Ecto.Query.where/3, etc.)
  • Delete by ID (c:Ecto.Repo.delete/2, etc.)
• Selects
• Assocs
• Preloads

The Northwind Repo Test should give you a good idea of what’s included.

Missing Features

The following features, for example, are missing:

• Composite primary keys
• Dynamic Repos (c:Ecto.Repo.put_dynamic_repo/1)
• Aggregates (dynamic / static)
• Joins
• Windows
• Transactions
• Migrations
• Locking

Installation

Using Etso is a two-step process. First, include it in your application’s dependencies:

defp deps do
  [
    {:etso, "~> 1.1.0"}
  ]
end

Afterwards, create a new Ecto.Repo, which uses Etso.Adapter:

defmodule MyApp.Repo do
  @otp_app Mix.Project.config()[:app]
  use Ecto.Repo, otp_app: @otp_app, adapter: Etso.Adapter
end

Once this is done, you can use any Schema against the Repo normally, as you would with any other Repo. Check out the Northwind modules for an example.

Utilisation

Originally, Etso was created to answer the question of whether ETS and Ecto can be married together. It has since found some uses in production applications, serving as a Data Repository for pre-defined nested content. This proved invaluable for rapid iteration.

If you have other Use Cases for this library, please add it here with a Pull Request.

• The Erlef Website is using Etso to cache and query news and event posts that are parsed and imported into the Repo on application startup.

Further Note

This repository is extracted from a prior project ETS Playground, which was created to support a session at ElixirConf EU 2019, Leveraging ETS Effectively.

Specifically, this library was created to illustrate the point that ETS can serve as a scalable storage layer for data which changes infrequently. Check out the Northwind Importer for an example.

Acknowledgements

This project contains a copy of data obtained from the Northwind database, which is owned by Microsoft. It is included for demonstration and testing purposes only, and is excluded from the distribution. The Author thanks Microsoft Corporation for the inspiration.

The Author also wishes to thank the following individuals:

• Wojtek Mach, for the inspiration regarding creation of an Ecto adapter for ETS.

• Steven Holdsworth, for initial concept proofing and refinement.

• Igor Kopestenski, for initial reviews.

• David Schainker, for initial reviews, and for finding uses for this library.

• Masanori Iwasaki, for support of for…in queries.

• Kohei Noda, for fixing nested {:orelse, …} queries.

• William Martins, for fixing primary key unicity check issues.

• Soichiro Nishizawa, for providing the implementation of sorted results.

• Doug W., for providing insights into parallel preloads.

• Atanda Rasheed, for support of JSON extract paths.

etso's Issues

Implement Ecto.Adapter.Schema.insert_all/8

As subject.

Issue after updating Ecto to version 3.8.0

Hi,

I am using the Logflare_logger_backend which itself is using Etso for batching log events.

After Ecto was updated to version 3.8.0 Etso started throwing this error:

 ** (UndefinedFunctionError) function Etso.Adapter.Meta.fetch/2 is undefined (Etso.Adapter.Meta does not implement the Access behaviour)
        (etso 0.1.6) Etso.Adapter.Meta.fetch(%{__struct__: Etso.Adapter.Meta, adapter: Etso.Adapter, cache: #Reference<0.2578754468.1338638337.64788>, pid: #PID<0.3611.0>, repo: LogflareLogger.Repo}, :stacktrace)
        (elixir 1.13.3) lib/access.ex:285: Access.get/3
        (ecto 3.8.0) lib/ecto/repo/supervisor.ex:161: Ecto.Repo.Supervisor.tuplet/2
        (logflare_logger_backend 0.11.0) lib/repo.ex:2: LogflareLogger.Repo.all/2
        (logflare_logger_backend 0.11.0) lib/logflare_logger/batch_cache.ex:103: LogflareLogger.BatchCache.events_in_flight/0
        (logflare_logger_backend 0.11.0) lib/logflare_logger/http_backend.ex:56: LogflareLogger.HttpBackend.handle_info/2
        (stdlib 3.15) gen_event.erl:627: :gen_event.server_update/4
        (stdlib 3.15) gen_event.erl:609: :gen_event.server_notify/4
        (stdlib 3.15) gen_event.erl:398: :gen_event.handle_msg/6
        (stdlib 3.15) proc_lib.erl:226: :proc_lib.init_p_do_apply/3

Ecto 3.7.2 is the last version not creating this issue.

Etso in phoenix : Expected Etso.Adapter to implement Ecto.Adapter.Storage

👋 Hello

I'm trying to use Etso inside my Phoenix app.

So basically I replaced the adapter inside the Repo

defmodule MyApp.Repo do
  use Ecto.Repo,
    otp_app: :my_app,
    adapter: Etso.Adapter
end

But during compilation I got this error

** (Mix) Expected Etso.Adapter to implement Ecto.Adapter.Storage in order to create storage for MyApp.Repo

Thx

Is Etso compartible with DETS?

Hello,
Can Etso be used with DETS? If yes, is Ecto Changeset/Update operation available? Only the Insert, Get, Delete features appears to have been listed.

Return unique constraint errors for custom primary keys

Hi there! I've been using Etso for a while now and it's a great solution to get an Ecto based project up and running in no time.

One thing that I'm missing is to be able to get the error that happens when trying to insert a duplicate entry. In other words, I get a changeset that is invalid, however, the errors property is empty. The impact is that I'm unable to show this error information on a form.

I've been dabbling a litte with the implementation in order to check if it's possible, and here are my findings.

First, the empty error is justified by the following line:

etso/lib/etso/adapter/behaviour/schema.ex

Line 24 in f25bdea

if ets_result, do: {:ok, []}, else: {:invalid, []}

If I change that to this:

diff --git a/lib/etso/adapter/behaviour/schema.ex b/lib/etso/adapter/behaviour/schema.ex
index 6fe3992..fd8f7e1 100644
--- a/lib/etso/adapter/behaviour/schema.ex
+++ b/lib/etso/adapter/behaviour/schema.ex
@@ -21,7 +21,7 @@ defmodule Etso.Adapter.Behaviour.Schema do
     ets_field_names = TableStructure.field_names(schema)
     ets_changes = TableStructure.fields_to_tuple(ets_field_names, fields)
     ets_result = :ets.insert_new(ets_table, ets_changes)
-    if ets_result, do: {:ok, []}, else: {:invalid, []}
+    if ets_result, do: {:ok, []}, else: {:invalid, [unique: "primary_key"]}
   end
 
   def update(%{repo: repo}, %{schema: schema}, fields, filters, [], _) do

Now I start to receive the error:

  @primary_key{:key, :string, autogenerate: false}
  
  def changeset(schema, params \\ %{}) do
    schema
    |> unique_constraint(:key, name: :primary_key)
  end

    %{
      key: [
        {"has already been taken",
         [constraint: :unique, constraint_name: "primary_key"]}
      ]
    }

---

What are your thoughts on that, is that something you wish to support?

Many to Many Support

Hello,

I've created a minimal test to show how the library does not currently support many_to_many in Ecto schemas using the Northwind Demo. Here's the output:

I've also created a PR in hopes that I could get some guidance on how we might go about adding support for this feature.

The issue seems to be in that the build conditions function head is hard coded here:

etso/lib/etso/ets/match_specification.ex

Line 57 in f25bdea

defp build_condition(field_names, _, {{:., [], [{:&, [], [0]}, field_name]}, [], []}) do

However it's not clear how I might change this to support many_to_many.

Please let me know if I can provide more information.

Thanks :)

like support in Etso?

Hello! thanks for Etso!

I was wondering if it would make sense to implement https://hexdocs.pm/ecto/Ecto.Query.API.html#like/2 in Etso to have basic search capabilities.

A "trick" for prefix search could be something like

iex(55)> :ets.new(:test, [:named_table])
:test
iex(56)> :ets.insert(:test, {'key1', 'like'})
true
iex(57)> :ets.insert(:test, {'key2', 'like2'})
true
iex(58)> :ets.insert(:test, {'key3', 'will like'})
true
iex(59)> :ets.select(:test, [{{'key1' ++ :_, :_}, [], [:"$_"]}])
[{'key1', 'like'}]
iex(60)> :ets.select(:test, [{{'key2' ++ :_, :_}, [], [:"$_"]}])
[{'key2', 'like2'}]
iex(61)> :ets.select(:test, [{{'key' ++ :_, :_}, [], [:"$_"]}]) 
[{'key2', 'like2'}, {'key1', 'like'}, {'key3', 'will like'}]
iex(62)> :ets.select(:test, [{{:_, 'will' ++ :_}, [], [:"$_"]}])         
[{'key3', 'will like'}]
iex(63)> :ets.select(:test, [{{:_, 'like' ++ :_}, [], [:"$_"]}])
[{'key2', 'like2'}, {'key1', 'like'}]

that would satisfy "a_string%", however for "%like" might be trickier

UndefinedFunctionError with ecto 3.8.1

After upgrading ecto_sql and ecto from 3.7.1 to 3.8.1 I have the following error :

2022-05-05T11:42:12Z app[e71f06cd] cdg [info]** (stop) {:EXIT, {%UndefinedFunctionError{arity: 2, function: :fetch, message: nil, module: Etso.Adapter.Meta, reason: "Etso.Adapter.Meta does not implement the Access behaviour"}, [{Etso.Adapter.Meta, :fetch, [%{__struct__: Etso.Adapter.Meta, adapter: Etso.Adapter, cache: #Reference<0.604452022.765329409.233143>, pid: #PID<0.4743.0>, repo: LogflareLogger.Repo}, :stacktrace], []}, {Access, :get, 3, [file: 'lib/access.ex', line: 285]}, {Ecto.Repo.Supervisor, :tuplet, 2, [file: 'lib/ecto/repo/supervisor.ex', line: 161]}, {LogflareLogger.Repo, :insert!, 2, [file: 'lib/repo.ex', line: 2]}, {LogflareLogger.BatchCache, :put, 2, [file: 'lib/logflare_logger/batch_cache.ex', line: 17]}, {LogflareLogger.HttpBackend, :handle_event, 2, [file: 'lib/logflare_logger/http_backend.ex', line: 41]}, {:gen_event, :server_update, 4, [file: 'gen_event.erl', line: 627]}, {:gen_event, :server_notify, 4, [file: 'gen_event.erl', line: 609]}]}}
2022-05-05T11:42:12Z app[e71f06cd] cdg [info]Last message: {:gen_event_EXIT, LogflareLogger.HttpBackend, {:EXIT, {%UndefinedFunctionError{arity: 2, function: :fetch, message: nil, module: Etso.Adapter.Meta, reason: "Etso.Adapter.Meta does not implement the Access behaviour"}, [{Etso.Adapter.Meta, :fetch, [%{__struct__: Etso.Adapter.Meta, adapter: Etso.Adapter, cache: #Reference<0.604452022.765329409.233143>, pid: #PID<0.4743.0>, repo: LogflareLogger.Repo}, :stacktrace], []}, {Access, :get, 3, [file: 'lib/access.ex', line: 285]}, {Ecto.Repo.Supervisor, :tuplet, 2, [file: 'lib/ecto/repo/supervisor.ex', line: 161]}, {LogflareLogger.Repo, :insert!, 2, [file: 'lib/repo.ex', line: 2]}, {LogflareLogger.BatchCache, :put, 2, [file: 'lib/logflare_logger/batch_cache.ex', line: 17]}, {LogflareLogger.HttpBackend, :handle_event, 2, [file: 'lib/logflare_logger/http_backend.ex', line: 41]}, {:gen_event, :server_update, 4, [file: 'gen_event.erl', line: 627]}, {:gen_event, :server_notify, 4, [file: 'gen_event.erl', line: 609]}]}}}
2022-05-05T11:42:12Z app[e71f06cd] cdg [info]11:42:12.527 [error] :gen_event handler LogflareLogger.HttpBackend installed in Logger terminating
2022-05-05T11:42:12Z app[e71f06cd] cdg [info]** (UndefinedFunctionError) function Etso.Adapter.Meta.fetch/2 is undefined (Etso.Adapter.Meta does not implement the Access behaviour)
2022-05-05T11:42:12Z app[e71f06cd] cdg [info]    (etso 0.1.6) Etso.Adapter.Meta.fetch(%{__struct__: Etso.Adapter.Meta, adapter: Etso.Adapter, cache: #Reference<0.604452022.765329409.233143>, pid: #PID<0.4743.0>, repo: LogflareLogger.Repo}, :stacktrace)
2022-05-05T11:42:12Z app[e71f06cd] cdg [info]    (elixir 1.13.4) lib/access.ex:285: Access.get/3
2022-05-05T11:42:12Z app[e71f06cd] cdg [info]    (ecto 3.8.1) lib/ecto/repo/supervisor.ex:161: Ecto.Repo.Supervisor.tuplet/2
2022-05-05T11:42:12Z app[e71f06cd] cdg [info]    (logflare_logger_backend 0.11.0) lib/repo.ex:2: LogflareLogger.Repo.all/2
2022-05-05T11:42:12Z app[e71f06cd] cdg [info]    (logflare_logger_backend 0.11.0) lib/logflare_logger/batch_cache.ex:103: LogflareLogger.BatchCache.events_in_flight/0
2022-05-05T11:42:12Z app[e71f06cd] cdg [info]    (logflare_logger_backend 0.11.0) lib/logflare_logger/http_backend.ex:56: LogflareLogger.HttpBackend.handle_info/2
2022-05-05T11:42:12Z app[e71f06cd] cdg [info]    (stdlib 3.17.2) gen_event.erl:627: :gen_event.server_update/4
2022-05-05T11:42:12Z app[e71f06cd] cdg [info]    (stdlib 3.17.2) gen_event.erl:609: :gen_event.server_notify/4
2022-05-05T11:42:12Z app[e71f06cd] cdg [info]Last message: :in_flight_check

Support null check is_nil

Currently it raises the error

no function clause matching in Etso.ETS.MatchSpecification.build_condition/3

lib/etso/ets/match_specification.ex

  defmacrop guard_operator(:!=), do: :"/="
  defmacrop guard_operator(:<=), do: :"=<"
  defmacrop guard_operator(operator), do: operator
  for operator <- ~w(== != < > <= >= and or)a do
    defp build_condition(field_names, params, {unquote(operator), [], [lhs, rhs]}) do
      lhs_condition = build_condition(field_names, params, lhs)
      rhs_condition = build_condition(field_names, params, rhs)
      {guard_operator(unquote(operator)), lhs_condition, rhs_condition}
    end
  end

https://hexdocs.pm/ecto/Ecto.Query.API.html#is_nil/1

Can you provide a License?

This is a really useful library, and I have forked to make some changes for a personal project but then hit the stumbling block that there is no license.

Can you please provide a license that would allow me to use this?

Thank you!

OTP 26 and ecto version

Seems need to use more updated version of ecto such as 3.11 in order to solve the problem of OTP 26!

** (Kernel.TypespecError) lib/ecto/query.ex:430: type dynamic/0 is a built-in type and it cannot be redefined.

Reference:
https://brianmeeker.me/2023/05/24/otp-26-ecto-dynamic-error/

Relax Ecto Version Constraint

The current version constraint (3.8.3) is limiting because one can't take advantage of new ecto features because you can't update the installed version (even though new versions of ecto have no changes that affect Etso). So, changing from "~> 3.8.3" to "~> 3.8" fixes this problem. Please treat this as important.

References:

etso/mix.exs

Line 30 in cae311e

{:ecto, "~> 3.8.3"},