thefirstavenger/ets

:ets, the Elixir way

ETS

:ets, the Elixir way

ETS is a set of Elixir modules that wrap Erlang Term Storage (:ets).

Current Features

• ETS.Set - wraps :set and :ordered_set
• ETS.Bag - wraps :bag and :duplicate_bag
• ETS.KeyValueSet - extension of ETS.Set that abstracts away tuple and key index concepts into simple key/value inputs/outputs.
• Most used functions from :ets replicated for all wrappers
• Returns {:error, reason} tuples (or raises in ! versions) for:
  • :table_not_found
  • :table_already_exists
  • :key_not_found
  • :invalid_record (when inserting non-tuples)
  • :record_too_small (tuple smaller than keypos index)
  • :position_out_of_bounds (lookup with a pos > length of one of the results)
  • :invalid_select_spec
  • :write_protected - trying to write to a protected or private table from a different process than the owner
  • :read_protected - trying to read from a private table from a different process than the owner

Design Goals

The purpose of this package is to improve the developer experience when both learning and interacting with Erlang Term Storage.

This will be accomplished by:

• Conforming to Elixir standards:
  • Two versions of all functions:
    • Main function (e.g. get) returns {:ok, return}/{:error, reason} tuples.
    • Bang function (e.g. get!) returns unwrapped value or raises on :error.
  • All options specified via keyword list.
• Wrapping unhelpful ArgumentError's with appropriate error returns.
  • Avoid adding performance overhead by using try/rescue instead of pre-validation
  • On rescue, try to determine what went wrong (e.g. missing table) and return appropriate error
  • Fall back to {:error, :unknown_error} (logging details) if unable to determine reason.
• Appropriate error returns/raises when encountering $end_of_table.
• Providing Elixir friendly documentation.
• Providing ETS.Set and ETS.Bag modules with appropriate function signatures and error handling.
  • ETS.Set.get returns a single item (or nil/provided default) instead of list as sets never have multiple records for a key.
• Providing abstractions on top of the two base modules for specific usages
  • ETS.KeyValueSet abstracts away the concept of tuple records, replacing it with standard key/value interactions.

Changes

For a list of changes, see the changelog

Usage

Creating ETS Tables

ETS Tables can be created using the new function of the appropriate module, either ETS.Set (for ordered and unordered sets) or ETS.Bag (for duplicate or non-duplicate bags). See module documentation for more examples and documentation, including a guide on What type of ETS table should I use?.

Create Examples

iex> {:ok, set} = Set.new(ordered: true, keypos: 3, read_concurrency: true, compressed: false)
iex> Set.info!(set)[:read_concurrency]
true

# Named :ets tables via the name keyword
iex> {:ok, set} = Set.new(name: :my_ets_table)
iex> Set.info!(set)[:name]
:my_ets_table
iex> {:ok, set} = Set.wrap_existing(:my_ets_table)
iex> set = Set.wrap_existing!(:my_ets_table)

Adding/Updating/Retrieving records in Sets

To add records to an ETS table, use put or put_new with a tuple record or a list of tuple records. put will overwrite existing records with the same key. put_new not insert if the key already exists. When passing a list of tuple records, all records are inserted in an atomic and isolated manner, but with put_new no records are inserted if at least one existing key is found.

Set Examples

iex> set = Set.new!(ordered: true)
iex> |> Set.put!({:a, :b})
iex> |> Set.put!({:a, :c}) # Overwrites entry from previous line
iex> |> Set.put!({:c, :d})
iex> Set.get(:a)
{:ok, {:a, :c}}
iex> Set.to_list(set)
{:ok, [{:a, :c}, {:c, :d}]}

iex> Set.new!(ordered: true)
iex> |> Set.put!({:a, :b})
iex> |> Set.put_new!({:a, :c}) # Doesn't insert due to key :a already existing
iex> |> Set.to_list!()
[{:a, :b}]

Bag Examples

iex> bag = Bag.new!()
iex> |> Bag.add!({:a, :b})
iex> |> Bag.add!({:a, :c})
iex> |> Bag.add!({:a, :c}) # Adds dude to duplicate: true
iex> |> Bag.add!({:c, :d})
iex> Bag.lookup(set, :a)
{:ok, [{:a, :b}, {:a, :c}, {:a, :c}]}
iex> Bag.to_list(bag)
{:ok, [{:a, :b}, {:a, :c}, {:a, :c}, {:c, :d}]}
iex> Bag.add_new!(bag, {:a, :z}) # Doesn't add due to key :a already existing
iex> Bag.to_list(bag)
{:ok, [{:a, :b}, {:a, :c}, {:a, :c}, {:c, :d}]}

iex> bag = Bag.new!(duplicate: false)
iex> |> Bag.add!({:a, :b})
iex> |> Bag.add!({:a, :c})
iex> |> Bag.add!({:a, :c}) # Doesn't add dude to duplicate: false
iex> |> Bag.add!({:c, :d})
iex> Bag.lookup(bag, :a)
{:ok, [{:a, :b}, {:a, :c}]}
iex> Bag.to_list(bag)
{:ok, [{:a, :b}, {:a, :c}, {:c, :d}]}

Current Progress

Base Modules

• ETS
  • All
• ETS.Set
  • Put (insert)
  • Get (lookup)
  • Get Element
  • Delete
  • Delete All
  • First
  • Next
  • Last
  • Previous
  • Match
  • Select
  • Select Delete
  • Has Key (Member)
  • Info
  • Delete
  • To List (tab2list)
  • Wrap
• ETS.Bag
  • Add (insert)
  • Lookup
  • Lookup Element
  • Delete
  • Delete All
  • Match
  • Select
  • Select Delete
  • Has Key (Member)
  • Info
  • Delete
  • To List (tab2list)
  • Wrap

Abstractions

• ETS.KeyValueSet
  • New
  • Wrap Existing
  • Put
  • Put New
  • Get
  • Info
  • Get Table
  • First
  • Last
  • Next
  • Previous
  • Has Key
  • Delete
  • Delete Key
  • Delete All
  • To List

Installation

ETS can be installed by adding ets to your list of dependencies in mix.exs:

def deps do
  [
    {:ets, "~> 0.9.0"}
  ]
end

Docs can be found at https://hexdocs.pm/ets.

Contributing

Contributions welcome. Specifically looking to:

• Add remainder of functions, (See Erlang Docs).
• Discover and add zero-impact recovery for any additional possible :ets ArgumentErrors.

ets's Issues

How to work with named tables?

Currently, every function needs as first argument a struct. But when I create a named_table i want to use this functions with an atom(). Is there a reason why all functions only accept structs as the first argument and not like the erlang ets an atom()?

Support `fetch/2`, `fetch!/2` for sets

It'd be nice to have a variant of ETS.Set.get/2,3 for when you want to know if a value was not found in ETS, without raising (a la ETS.Set.get!/2).

This is important if you are using ETS as a read-through cache: when looking up a value, you need to know if it was present in the cache before performing a potentially expensive operation. If that expensive operation returns nil today, there is no API provided to differentiate between the two cases.

---

A common access pattern in Elixir (see Map, Keyword) that enables this is to implement a fetch/2 that returns an ok tuple or error atom:

def fetch(source, key) do
  case lookup(source, key) do
    <found_value> -> {:ok, value}
    <not_found> -> :error
  end
end

Then you can implement get/2,3 and fetch!/2 on top of them:

def get(source, key, default \\ nil) do
  case fetch(source, key) do
    {:ok, value} -> {:ok, value}
    :error -> default
  end
end

def fetch!(source, key) do
  case fetch(source, key) do
    {:ok, value} -> {:ok, value}
    :error -> raise
  end
end

I propose we do this for ETS.Set (keeping get!/2 for backwards compatibility) for a more idiomatic Map-like API. The actual error tuple handling would have to be a little different to align with the consistent two-tuple errors, but the principle is the same.

I'm happy to add a PR for this, just wanted to solicit your thoughts as I get into it.

RFC - lazily getting a stream of keys from an ETS table

Hello,

Do you think it would be useful to add stream_keys() and stream_values() functions that respectively return all keys/values from an ETS table? I'm currently doing this to lazily handle a large ETS table as follows:

alias Ets.Set.KeyValueSet, as: EtsMap

emap = a_very_large_instance_of_EtsMap

table = emap |> EtsMap.get_table!()

keys = # this would become stream_keys()
  Stream.iterate(:ets.first(table), &:ets.next(table, &1))
  |> Stream.take_while(fn :"$end_of_table" -> false; _else -> true end)

values = # this would become stream_values()
  keys
  |> Stream.map(&EtsMap.get!(emap, &1))

If you think this would be useful, I will submit a PR with tests (written per TDD, of course).

Thanks for your consideration.

Blame ets errors directlty in Elixir stdlib

Hi @TheFirstAvenger!

Thanks for showing me this library doing ElixirConf. I have been thinking about it and I realized that the part related to the error messages could be added directly to Elixir itself!

Elixir exceptions have a blame mechanism, which is used to add more information. We can use the blame mechanism for ArgumentError to see if the error came from ets and provide more information. We already do this for erlang:apply/3, for example:

https://github.com/elixir-lang/elixir/blob/master/lib/elixir/lib/exception.ex#L685

What do you think? Would you be interested in sending a PR? We could start with something simple, such as blaming ets:lookup, and then build it from there.

Add a guide to choosing table type?

What would you think of adding a guide to choosing a table type to the README? Something that conveys this:

Need unique keys?

• No
  • Bag
    • Need unique values and willing to have slower insertion?
      • No
        • Bag.new()
      • Yes
        • Bag.new(duplicate: false)
• Yes
  • Set
    • Need ordered keys and willing to have slower insertion?
      • No
        • Set.new()
      • Yes
        • Set.new(ordered: true)

It could be a nested list like this, or I could make an ASCII flowchart for it.