sheharyarn/memento

Simple + Powerful interface to the Mnesia Distributed Database 💾

Simple but Powerful Elixir interface to the Erlang Mnesia Database
_{Mnesia. Memento. Get it?}

• 😀 Easy to Use: Provides a simple & intuitive API for working with Mnesia
• ⚡️ Real-time: Has extremely fast real-time data searches, even across many nodes
• 💪 Powerful Queries: on top of Erlang's MatchSpec and QLC, that are much easier to use
• 📓 Detailed Documentation: and examples for all methods on HexDocs
• 💾 Persistent: Schema can be coherently kept on disc & in memory
• 🌐 Distributed: Data can easily be replicated on several nodes
• 🌀 Atomic: A series of operations can be grouped in to a single atomic transaction
• 🔍 Focused: Encourages good patterns by omitting dirty calls to the database
• 🔧 Mnesia Compatible: You can still use Mnesia methods for Schemas and Tables created by Memento
• ❄️ No Dependencies: Zero external dependencies; only uses the built-in Mnesia module
• ⛅️ MIT Licensed: Free for personal and commercial use

Memento is an extremely easy-to-use and powerful wrapper in Elixir that makes it intuitive to work with Mnesia, the Erlang Distributed Realtime Database. The original Mnesia API in Erlang is convoluted, unorganized and combined with the complex MatchSpec and QLC query language, is hard to work with in Elixir, especially for beginners. Memento attempts to define a simple API to work with schemas, removing the majority of complexity associated with it.

Installation

Add :memento to your list of dependencies in your Mix file:

def deps do
  [{:memento, "~> 0.3.2"}]
end

If your Elixir version is 1.3 or lower, also add it to your applications list:

def application do
  [applications: [:memento]]
end

It's preferable to only add :memento and not :mnesia along with it. This will ensure that that OTP calls to Mnesia go through the Supervisor spec specified in Memento.

Configuration

It is highly recommended that a custom path to the Mnesia database location is specified, even on the local :dev environment (You can add .mnesia to your .gitignore):

# config/config.exs
config :mnesia,
  dir: '.mnesia/#{Mix.env}/#{node()}'        # Notice the single quotes

Usage

You start by defining a Module as a Memento Table by specifying its attributes, type and other options. At least two attributes are required, where the first one is the primary-key of the table. A simple definition looks like this:

defmodule Blog.Author do
  use Memento.Table, attributes: [:username, :fullname]
end

A slightly more complex definition that uses more options, could look like this:

defmodule Blog.Post do
  use Memento.Table,
    attributes: [:id, :title, :content, :status, :author],
    index: [:status, :author],
    type: :ordered_set,
    autoincrement: true


  # You can also define other methods
  # or helper functions in the module
end

Once you have defined your schemas, you need to create them before you can interact with them:

Memento.Table.create!(Blog.Author)
Memento.Table.create!(Blog.Post)

See the Memento.Table documentation for detailed examples and more information about all the options.

CRUD Operations & Queries

Once a Table has been created, you can perform read/write/delete operations on their records. An API for all of these operations is exposed in the Memento.Query module, but these methods can't be called directly. Instead, they must always be called inside a Memento.Transaction:

Memento.transaction! fn ->
  Memento.Query.all(Blog.Author)
end

# => [
#  %Blog.Author{username: :sye,     fullname: "Sheharyar Naseer"},
#  %Blog.Author{username: :jeanne,  fullname: "Jeanne Bolding"},
#  %Blog.Author{username: :pshore,  fullname: "Paul Shore"},
# ]

For the sake of succinctness, transactions are ignored in most of the examples below, but they are still required. Here's a quick overview of all the basic operations:

# Get all records in a Table
Memento.Query.all(Post)

# Get a specific record by its primary key
Memento.Query.read(Post, id)
Memento.Query.read(Author, username)

# Write a record
Memento.Query.write(%Author{username: :sarah, name: "Sarah Molton"})

# Delete a record by primary key
Memento.Query.delete(Post, id)
Memento.Query.delete(Author, username)

# Delete a record by passing the full object
Memento.Query.delete_record(%Author{username: :pshore, name: "Paul Shore"})

For more complex read operations, Memento exposes a select/3 method that lets you chain conditions using a simplified version of the Erlang MatchSpec. This is what some queries would look like for a Movie table:

• Get all movies named "Rush":

  Memento.Query.select(Movie, {:==, :title, "Rush"})

• Get all movies directed by Tarantino before the year 2000:

  guards = [
    {:==, :director, "Quentin Tarantino"},
    {:<, :year, 2000},
  ]
  Memento.Query.select(Movie, guards)

See Query.select/3 for more information about the guard operators and detailed examples.

Persisting to Disk

Setting up disk persistence in Mnesia has always been a bit weird. It involves stopping the application, creating schemas on disk, restarting the application and then creating the tables with certain options. Here are the steps you need to take to do all of that:

# List of nodes where you want to persist
nodes = [ node() ]

# Create the schema
Memento.stop
Memento.Schema.create(nodes)
Memento.start

# Create your tables with disc_copies (only the ones you want persisted on disk)
Memento.Table.create!(TableA, disc_copies: nodes)
Memento.Table.create!(TableB, disc_copies: nodes)
Memento.Table.create!(TableC)

This needs to be done only once and not every time the application starts. It also makes sense to create a helper function or mix task that does this for you. You can see a sample implementation here.

Roadmap

• Memento
  • start/stop
  • info
  • system_info
  • Application
  • Config Vars
• Memento.Table
  • Create/Delete helpers
  • clear_table
  • table_info
  • wait
  • Ecto-like DSL
  • Migration Support
• Memento.Query
  • Integration with Memento.Table
  • match/select
  • read/write/delete
  • first/next/prev/all_keys
  • test matchspec
  • continue/1 for select continuations
  • autoincrement
  • Helper use macro
• Memento.Transaction
  • Simple/Synchronous
  • Bang versions
  • inside?
  • abort
  • Lock Helpers
• Memento.Schema
  • create/delete
  • print (schema/1)
• Memento.Collection
  • Easy Helpers
  • Custom DSL
• Mix Tasks

FAQ

1. Why Memento/Mnesia?

In most applications, some kind of data storage mechanism is needed, but this usually means relying on some sort of external dependency or program. Memento should be used in situations when it might not always make sense in an Application to do this (e.g. the data is ephemeral, the project needs to be kept light-weight, you need a simple data store that persists across application restarts, data-code decoupling is not important etc.).

2. When shouldn't I use Memento/Mnesia?

Like mentioned in the previous point, Memento/Mnesia has specific use-cases and it might not always make sense to use it. This is usually when you don't want to couple your code and database, and want to allow independent or external accesses to transformation of your data. In such circumstances, you should always prefer using some other datastore (like Redis, Postgres, etc.).

3. Isn't there already an 'Amnesia' library?

I've been a long-time user of the Amnesia package, but with the recent releases of Elixir (1.5 & above), the library has started to show its age. Amnesia's dependence on the the exquisite package has caused a lot of compilation problems, and it's complex macro-intensive structure hasn't made it easy to fix them either. The library itself doesn't even compile in Elixir 1.7+ so I finally decided to write my own after I desperately needed to update my Mnesia-based projects.

Memento is meant to be an extremely lightweight wrapper for Mnesia, providing a very easy set of helpers and forcing good decisions by avoiding the "dirty" methods.

4. Are there any other projects that are using Memento?

Memento is a new package so there aren't many Open Source examples available. Que is another library that uses Memento for background job processing and storing the state of these Jobs. If your project uses Memento, feel free to send in a pull-request so it can be mentioned here.

Contributing

• Fork, Enhance, Send PR
• Lock issues with any bugs or feature requests
• Implement something from Roadmap
• Spread the word ❤️

License

This package is available as open source under the terms of the MIT License.

memento's Issues

How to update specific keys in a row without overwriting the rest?

I’ve created a module of CRUD operations, mostly one-liners, but I’m stuck at creating a general purpose update function.
Goals:
1. Update the value of any key or keys in the row, independently of each other.
2. Insert and or update the :updated_at key.
3. Only update the keys specified by the update functions arguments

My approach so far has been something along these lines (pseudo code):

defmodule Todo do
  use Memento.Table,
    attributes: [:id, :name, :priority, :updated_at],
    type: :ordered_set,
    autoincrement: true

# .....

  def update(id, changes = %{}) do
    Memento.transaction do
      case get_by_id(id) do
        %Todo{} = todo ->
          todo
          |> adjust( changes)
          |> Map.put!(:updated_at, NaiveDateTime.utc_now()
 
        _ ->
          {:error, :not_found}
      end
    end
    
    defp adjust(%Todo{} = todo, changes) do
       todo
         |> Map.update!(changes))
         |> Todo.write()
    end

    def get_by_id(id) do
          Memento.transaction(fn -> Memento.Query.read(Todo, id) end)
    end

Elixir school demonstrates updates with Mnesia.write/1 but this overwrites the whole row.

The other solutions I found are either over my head or in Erlang:

• https://stackoverflow.com/questions/63048415/elixir-mnesia-what-is-the-most-concise-way-to-update-a-set-of-value-in-an-el
• https://stackoverflow.com/questions/1820996/erlang-mnesia-updating-a-single-field-value-in-a-row

Table Fragmentation?

Are there any plans to support table fragmentation?

Problems creating Mnesia database because the node is not set for distillery release commands.

I tried to create Distillery release commands, but gave up because the node is not set. Even if I replace the database path or move the files, the database is not usable when the application is started with the proper node name.

There may or may not have been issues with mix commands as well. I did not test them as extensively, but I am using the same base function for both.

Is there a way to create a usable Mnesia database for the node real_node_name@real_host_name when the current node is nonode@nohost in a Distillery release task? Also, being able to batch create Mnesia databases for a list of nodes might be useful, although I do not think that changes anything given a solution for one node.

EDIT

I wound up using the following function to set set the node name for the task.

def set_node_name_for_task(node_name \\ nil) do
  node_name
  |> Kernel.||(System.get_env("NODE_NAME"))
  |> Kernel.||("nonode@nohost")
  |> String.to_atom
  |> Node.start
end

Combined with the following change to the :mnesia configuration, release tasks can be used to create databases. This only works for a single node at a time, however.

config :mnesia,
  dir: '/etc/.mnesia/#{Mix.env}/#{System.get_env("NODE_NAME")}'        # Notice the single quotes

Maybe it is worth adding a note to the documentation?

Support composite primary keys

Didn't find anywhere if memento supports composite primary keys, mnesia does support it would it be possible to bring that into memento ?

Using memento with Phoenix releases

Hello

Im trying to build a release of an application that uses Memento.
When running the app with mix phx.server everything works as expected.
When I try to create a production release, I get an error when trying to access the store.
%Memento.Error{message: "Transaction Failed with: {:no_exists, Sigfox.DeviceState}"}

Any help would be much appreciated.

Thanks

Checking if records exist with Memento

I am using memento to wrap Mnesia functionality into my application.

I want to check if a record with a certain id exists and if not perform certain actions.

Documentation says read should return nil when record doesn’t exist. Howerver, the Memento.transation wrapper aborts with a:not_existsexception therefore my application cannot proceed.

Here’s how my function looks like:

def find_conversation(user_id) do
    Memento.transaction! fn ->
      user_id
      |> read
      |> to_conversation
    end
end

defp to_conversation(nil), do: nil
  defp to_conversation(%@store{} = conversation) do
    struct(Conversation, Map.from_struct(conversation))
  end

Complex queries on nested maps using match spec

A question was posted on the ElixirForum today:

I’m using @sheharyarn’s Memento for integrating Mnesia for my Phoenix App. Having little to no experience in erlang, where should I look in the documentation for complex queries, I am storing a map in one of my column and I need to fetch by querying in that map, I can do this in Ecto using fragments but I’m wondering how to do the same using Mnesia.

The Erlang Matchspec is very confusing, especially so for beginners. It becomes even harder to write them when nested maps are involved. Quite often, I also forget how to use them for many scenarios and have to read the Erlang docs on match_spec again.

This was one of the very reasons I decided to write the Memento library in the first place. While I won't add support for an API to directly query nested maps in the package as of yet, I'll use this issue to cover some of the advanced queries for now.

Memento select_raw error

Calling Memento.Query.select_raw(...) leads to an error when attempting to load the results.

** (FunctionClauseError) no function clause matching in Memento.Query.Data.load/1    
    
    The following arguments were given to Memento.Query.Data.load/1:
    
        # 1
        "a"
    
    Attempted function clauses (showing 1 out of 1):
    
        def load(data) when is_tuple(data)
    
    (memento 0.3.2) lib/memento/query/data.ex:55: Memento.Query.Data.load/1
    (elixir 1.13.0) lib/enum.ex:1593: Enum."-map/2-lists^map/1-0-"/2
    (mnesia 4.20.3) mnesia_tm.erl:842: :mnesia_tm.apply_fun/3
    (mnesia 4.20.3) mnesia_tm.erl:818: :mnesia_tm.execute_transaction/5
    (memento 0.3.2) lib/memento/transaction.ex:71: Memento.Transaction.execute/2

Is there a way to search for a pattern?

Hi there,
I am using Memento for a small project. It works great but I am lacking a small thing. Is there a way to search for a pattern ?

For example I have list of Movies containing word "Rush". So in a traditional database, we will do something like

where title like '%Rush%'

Is there something similar with Mnesia?

Regards,

Select by date

Hi! First of all, thanks for this library, is awesome!
I have a little problem querying Mnesia by date: I need to get every record where :creation_date is less than 24 hours ago.

I am writing the following method:

    def select_by_date do
     date = Timex.shift(NaiveDateTime.utc_now, hours: -24) 
     run_query(
        {:<, :created_at, date}
      )
    end

    defp run_query(pattern) do
      Memento.transaction! fn ->
        @store 
        |> Memento.Query.select(pattern)
      end
    end

For the run_querymethod, I've followed your example.

When I run the select_by_date method, I get the following error:

iex(1)> delete_outdated_config
** (Memento.MnesiaException) Mnesia operation failed
   Bad or invalid argument, possibly bad type
   Mnesia Error: {:badarg, [Engine.Mnesia.Db.Config, [{{Engine.Mnesia.Db.Config, :"$1", :"$2", :"$3", :"$4", :"$5", :"$6", :"$7"}, [{:<, :"$6", ~N[2019-01-23 16:50:45.690805]}], [:"$_"]}]]}
    (memento) lib/memento/query/query.ex:580: Memento.Query.select_raw/3
    (mnesia) mnesia_tm.erl:836: :mnesia_tm.apply_fun/3
    (mnesia) mnesia_tm.erl:812: :mnesia_tm.execute_transaction/5
    (memento) lib/memento/transaction.ex:71: Memento.Transaction.execute/2
    (memento) lib/memento/transaction.ex:84: Memento.Transaction.execute!/2

Is there a way to compare dates during the query?

Getting Mnesia Operation Failed cyclic

I'm using que without persistence. Currently after upgrading Que from 0.5.0 to 0.7.0 which changed dependency from mnesia to memento, I'm getting these errors:

Elixir.Memento.MnesiaException: Mnesia operation failed
   {:cyclic, :"[email protected]", {Que.Persistence.Mnesia.DB.Jobs, :______WHOLETABLE_____}, :read, :read, {:tid, 138, #PID<0.5537.0>}}
   Mnesia Error: {:cyclic, :"[email protected]", {Que.Persistence.Mnesia.DB.Jobs, :______WHOLETABLE_____}, :read, :read, {:tid, 138, #PID<0.5537.0>}}
  File "lib/memento/query/query.ex", line 728, in Memento.Query.autoincrement_key_for/1
  File "lib/memento/query/query.ex", line 701, in Memento.Query.prepare_record_for_write!/2
  File "lib/memento/query/query.ex", line 250, in Memento.Query.write/2
  File "lib/que/persistence/mnesia/db.ex", line 144, in anonymous fn/1 in Que.Persistence.Mnesia.DB.Jobs.update_job/1
  File "mnesia_tm.erl", line 836, in :mnesia_tm.apply_fun/3
  File "mnesia_tm.erl", line 812, in :mnesia_tm.execute_transaction/5
  File "lib/memento/transaction.ex", line 71, in Memento.Transaction.execute/2
  File "lib/memento/transaction.ex", line 84, in Memento.Transaction.execute!/2
  Module "Elixir.Que.Server", in Que.Server.init/1

Seems like there's a cyclic error in the transaction

Add Distribution Helpers

Related to #15 and #17.

Add more mnesia wrapper methods to easily add new cluster nodes to the mnesia config and change schema, copy type, waiting for tables to get ready and more.

• Connect Nodes
  • Memento.add_nodes(nodes)
• Schema wait
• Schema copy
  • Memento.Schema.set_storage_type(node, mode)
• Table copy
  • Memento.Table.create_copy(Jobs, node, mode)
  • Memento.Table.delete_copy(Jobs, node)
  • Memento.Table.move_copy(Jobs, from_node, to_node)
  • Memento.Table.set_storage_type(Jobs, node, mode)

Distributed Memento (w/libcluster)

@sheharyarn this is a real cool effort you've got here. 👏 Nice work.

How do I link up two nodes together using Memento? I don't see any change config or add table copy logic so far. Am I correct that I should drop out to :mnesia to do it?

Eg (w/two node RAM instances)... first node:

Memento.start
Memento.Table.create!(MyLovelyTable)
#:mnesia.wait_for_tables?

Second node:

Memento.start
:mnesia.change_config(:extra_db_nodes, Node.list())
:mnesia.add_table_copy(MyLovelyTable, node())
#:mnesia.wait_for_tables?

How does pagination work?

Hi, I really like this library and have been using it to build a prototype application. I am wondering if there is a way to paginate?

How to persist data on disk?

Maybe I did get something wrong, but how do I persist data on disk across restarts? I reread the documentation several times, tried to reuse :mnesia.create_table options... And nothing - data exists only in RAM or error is raised.

Migrations method

Hey @sheharyarn

I understand migrations are not fully supported yet, is there a way to make it work? Let's say I add a field i want it to have the new fields able to be stored. Any procedure / steps you recommend to manually do migrations?

Expand Macro arguments

Add support for sigils and module attributes by expanding the arguments given to the Memento macros. See this stackoverflow question.

Does memento support disc_only as well as disc_copies?

Hello,
According to mnesia documentation. Mnesia has the ability to support, ram, disc_only as well as disc_copies.

ram_copies

This option makes it so all data is stored exclusively in ETS, so memory only. Memory should be limited to a theoretical 4GB (and practically around 3GB) for virtual machines compiled on 32 bits, but this limit is pushed further away on 64 bits virtual machines, assuming there is more than 4GB of memory available.

disc_only_copies

This option means that the data is stored only in DETS. Disc only, and as such the storage is limited to DETS' 2GB limit.

disc_copies

This option means that the data is stored both in ETS and on disk, so both memory and the hard disk. disc_copies tables are not limited by DETS limits, as Mnesia uses a complex system of transaction logs and checkpoints that allow to create a disk-based backup of the table in memory.

Learn You Some Erlang for Mnesia
I was wondering if Memento had the same support. I'm referencing this section of the mnesia documentation. Thank you!

No function clause when the record does not exist

I noticed that if a query fails to return any records, I get the following error:
** (FunctionClauseError) no function clause matching in Memento.Query.coerce_records/1
(memento) lib/memento/query/query.ex:663: Memento.Query.coerce_records(:"$end_of_table")

However calling Mnesia directly, I get the standard message of:
{:atomic, []} that I can easily pattern match on.

Data is not always flushed to disk

Original post by @cpilka, moved from #3:

---

I have a weird issue with disc persistence. Data is flushed to disc just for a certain amount of records (few hundred). Single inserts are not written to disc. Just wonder if there's a :disc_only_copies in Memento that would write all data straight to mnesia files.

This one doesn't write to disc:

Memento.transaction! fn ->
  Memento.Query.write(record)
end

This one does:

Memento.transaction! fn ->
  for _ <- 1..100 do
    Memento.Query.write(record)
  end
end

Is there any setting that tells Mnesia to flush the memory part always to disc and not to flush reaching certain buffer? Or whatever causes this issue ...

Data replication across nodes with Peerage

Question sent to me via email:

How would I guarantee that Memento/Mnesia is replicating data across nodes, I’m using something called peerage which does node discovery, so at boot the nodes may not all be discovered.

Switch to Github Actions CI

Due to the shutdown of travis-ci.org, all existing pipelines will be dead soon, so it makes sense to switch to Github Actions, which will remain free for OS projects.