Giter Site home page Giter Site logo

ex_audit's Introduction

ExAudit

Ecto auditing library that transparently tracks changes and can revert them.

ExAudit plugs right into your ecto repositories and hooks all the data mutating Ecto.Repo functions to track changes to entities in your database.

Features

  • Wraps Ecto.Repo, no need to change your existing codebase to start tracking changes
  • Creates +- diffs of the casted structs. Custom types are automatically supported.
  • Ships with functions to review the history of a struct and roll back changes
  • Allows custom ID types and custom fields in the version schema
  • Tracks associated entities when they're created, updated or deleted in a single Repo call
  • Recursively tracks cascading deletions

Usage

ExAudit replaces some functions in your repo module:

  • insert/2
  • insert!/2
  • update/2
  • update!/2
  • delete/2
  • delete!/2

All changes to the database made with these functions will automatically be tracked.

Also, new functions are added to the repository:

  • history/2: lists all versions of the given struct ordered from oldest to newest
  • revert/2: rolls the referenced entity back to the state it was before the given version was changed

With this API, you should be able to enable auditing across your entire application easily.

If for some reason ExAudit does not track a change, you can manually add it with ExAudit.Tracking.track_change(module, adapter, action, changeset, resulting_struct, opts).

In the same module, there are a few other functions you might find useful to roll custom tracking.

Setup

Add ex_audit to your list of dependencies:

def deps do
  [
    {:ex_audit, "~> 0.9"}
  ]
end

For older ecto versions than 3.2, check out what to do in the Ecto Versions section.

You have to hook ExAudit.Repo to your repo:

defmodule MyApp.Repo do
  use Ecto.Repo,
    otp_app: :my_app,
    adapter: Ecto.Adapters.Postgres

  use ExAudit.Repo
end

Configuration

You have to tell ExAudit which schemas to track and the module of your version schema.

In your config.exs, write something like this:

config :ex_audit,
  ecto_repos: [MyApp.Repo],
  version_schema: MyApp.Version,
  tracked_schemas: [
    MyApp.Accounts.User,
    MyApp.BlogPost,
    MyApp.Comment
  ]

Optionally, you can tell ExAudit to treat certain structs as primitives and not record internal changes for the struct. Add these under the key :primitive_structs in your config. So for example, if you configured Date to be treated as a primitive:

config :ex_audit,
  ecto_repos: [MyApp.Repo],
  version_schema: MyApp.Version,
  tracked_schemas: [
    MyApp.Accounts.User,
    MyApp.BlogPost,
    MyApp.Comment
  ],
  primitive_structs: [
    Date
  ]

then the patch would record the entire Date struct as a change:

{:primitive_change, ~D[2000-01-01], ~D[2000-01-18]}

instead of descending into the struct to find the individual part that changed:

{:changed, %{day: {:changed, {:primitive_change, 1, 18}}}}

Version Schema and Migration

You need to copy the migration and the schema module for the versions table. This allows you to add custom fields to the table or decide which type to use for the primary key.

version.ex

defmodule MyApp.Version do
  use Ecto.Schema
  import Ecto.Changeset

  schema "versions" do
    # The patch in Erlang External Term Format
    field :patch, ExAudit.Type.Patch

    # supports UUID and other types as well
    field :entity_id, :integer

    # name of the table the entity is in
    field :entity_schema, ExAudit.Type.Schema

    # type of the action that has happened to the entity (created, updated, deleted)
    field :action, ExAudit.Type.Action

    # when has this happened
    field :recorded_at, :utc_datetime

    # was this change part of a rollback?
    field :rollback, :boolean, default: false

    # custom fields
    belongs_to :user, MyApp.Accounts.User
  end

  def changeset(struct, params \\ %{}) do
    struct
    |> cast(params, [:patch, :entity_id, :entity_schema, :action, :recorded_at, :rollback])
    |> cast(params, [:user_id]) # custom fields
  end
end

create_version_table.exs

defmodule MyApp.Migrations.AddVersions do
  use Ecto.Migration

  def change do
    create table(:versions) do
      # The patch in Erlang External Term Format
      add :patch, :binary

      # supports UUID and other types as well
      add :entity_id, :integer

      # name of the table the entity is in
      add :entity_schema, :string

      # type of the action that has happened to the entity (created, updated, deleted)
      add :action, :string

      # when has this happened
      add :recorded_at, :utc_datetime

      # was this change part of a rollback?
      add :rollback, :boolean, default: false

      # optional fields that you can define yourself
      # for example, it's a good idea to track who did the change
      add :user_id, references(:users, on_update: :update_all, on_delete: :nilify_all)
    end

    # create this if you are going to have more than a hundred of thousands of versions
    create index(:versions, [:entity_schema, :entity_id])
  end
end

Recording custom data

If you want to track custom data such as the user id, you can simply pass a keyword list with that data to the :ex_audit_custom option in any Repo function:

MyApp.Repo.insert(changeset, ex_audit_custom: [user_id: conn.assigns.current_user.id])

Of course it is tedious to upgrade your entire codebase just to track the user ID for example, so you can also pass this data in a plug:

defmodule MyApp.ExAuditPlug do
  def init(_) do
    nil
  end

  def call(conn, _) do
    ExAudit.track(user_id: conn.assigns.current_user.id)
    conn
  end
end

In the background, ExAudit.track will remember the PID it was called from and attaches the passed data to that PID. In most cases, the conn process will call the Repo functions, so ExAudit can get the data from that PID again deeper in the plug tree.

In some cases where it is not possible to call the Repo function from the conn process, you have to pass the custom data manually via the options described above.

Examples for data you might want to track additionally:

  • User ID
  • API Key ID
  • Message from the user describing what she changed

Ecto versions

For ecto 2.x, use {:ex_audit, "~> 0.5"}

For ecto 3.0, upgrade ecto to 3.1

For ecto 3.1, use {:ex_audit, "~> 0.6"}

For ecto 3.1.2 or higher, upgrade ecto to 3.2

For ecto 3.2, use {:ex_audit, "~> 0.7"}

More

The documentation is available at https://hexdocs.pm/ex_audit.

Check out ZENNER IoT Solutions, makers of the ELEMENT IoT platform.

ex_audit's People

Contributors

bolte-17 avatar brunohkbx avatar edgurgel avatar hisapy avatar jvoegele avatar mcasper avatar mindreframer avatar narrowtux avatar nduitz avatar rauann avatar revati avatar samhamilton avatar simonprev avatar tapajos avatar vasspilka avatar youalreadydid avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ex_audit's Issues

Version 0.7 on hex.pm does not contain the same code as described in github

Hi!

The code I get from hex.pm does not contain the changes to ExAudit.Repo as here on github.
All the docs and code here on github refers to version 0.7, but that is clearly not available when code is fetched from hex.pm

I am using {:ex_audit, "~> 0.7"} in my mix.exs file.

First 10 lines of code in the code I got from hex.pm for ExAudit.Repo:

defmodule ExAudit.Repo do
  @moduledoc """
  Replaces Ecto.Repo to be able to keep track of changes made to entities in the repo.
  Changes made with the following functions are tracked, other function calls must be manually tracked:
   * `insert`, `insert!`
   * `update`, `update!`
   * `delete`, `delete!`
  ### Shared options
  All normal Ecto.Repo options will work the same, however, there are additional options specific to ex_audit:
   * `:ex_audit_custom` - Keyword list of custom data that should be placed in new version entries. Entries in this

First 10 lines here on github:

defmodule ExAudit.Repo do
  @moduledoc """
  Adds ExAudit version tracking to your Ecto.Repo actions. The following functions are
  extended to detect if the given struct or changeset is in the list of :tracked_schemas
  given in :ex_audit config:
    insert: 2,
    update: 2,
    insert_or_update: 2,
    delete: 2,

Conflicting behaviors using ExAudit.Repo

Using ExAudit.Repo properly after usage of Ecto.Repo:

defmodule PINWaste.Repo do
  use Ecto.Repo,
    otp_app: :pin_waste,
    adapter: Ecto.Adapters.Postgres

  use ExAudit.Repo

This causes the following warning:
warning: conflicting behaviours found. function default_options/1 is required by Ecto.Repo and ExAudit.Repo

Getting microseconds error

Hello, I am trying to integrate ex_audit wit my application using more or less the default configuration.

However when I try to save a model with changes I get the following error:

:utc_datetime expects microseconds to be empty, got: #DateTime<2019-05-02 05:19:39.935000Z>
Use `DateTime.truncate(utc_datetime, :second)` (available in Elixir v1.6+) to remove microseconds.

lib/ecto/type.ex
  end
  defp check_no_usec!(%{microsecond: {0, 0}} = datetime, _kind), do: datetime
  defp check_no_usec!(%struct{} = datetime, kind) do
    raise ArgumentError, """
    #{inspect(kind)} expects microseconds to be empty, got: #{inspect(datetime)}
    Use `#{inspect(struct)}.truncate(#{kind}, :second)` (available in Elixir v1.6+) to remove microseconds.
    """
  end

Any suggestions on how to fix ?

Here are the versions I use:

...  
  ecto 3.1.1
  ecto_sql 3.1.1
  ex_audit 0.6.0
  phoenix 1.4.3
  phoenix_ecto 4.0.0

Also here's the whole ST:

 ecto lib/ecto/type.ex:1224 Ecto.Type.check_no_usec!/2
 ecto lib/ecto/type.ex:412 Ecto.Type.dump_utc_datetime/1
 ecto lib/ecto/type.ex:817 Ecto.Type.process_dumpers/3
 ecto lib/ecto/repo/schema.ex:925 Ecto.Repo.Schema.dump_field!/6
 ecto lib/ecto/repo/schema.ex:109 anonymous fn/5 in Ecto.Repo.Schema.init_mapper/3
 elixir lib/enum.ex:1437 anonymous fn/3 in Enum.map_reduce/3
 stdlib maps.erl:257 :maps.fold_1/3
 elixir lib/enum.ex:1956 Enum.map_reduce/3
 ecto lib/ecto/repo/schema.ex:81 anonymous fn/5 in Ecto.Repo.Schema.extract_header_and_fields/5
 elixir lib/enum.ex:1431 Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3
 ecto lib/ecto/repo/schema.ex:80 Ecto.Repo.Schema.extract_header_and_fields/5
 ecto lib/ecto/repo/schema.ex:44 Ecto.Repo.Schema.do_insert_all/6
 ex_audit lib/repo/schema.ex:34 anonymous fn/4 in ExAudit.Schema.update/4
 ex_audit lib/repo/schema.ex:155 ExAudit.Schema.run_in_multi/4
 ecto lib/ecto/multi.ex:579 Ecto.Multi.apply_operation/5
 elixir lib/enum.ex:1940 Enum."-reduce/3-lists^foldl/2-0-"/3
 ecto lib/ecto/multi.ex:563 anonymous fn/5 in Ecto.Multi.apply_operations/5
 ecto_sql lib/ecto/adapters/sql.ex:874 anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
 db_connection lib/db_connection.ex:1415 DBConnection.run_transaction/4
 ecto lib/ecto/repo/transaction.ex:15 Ecto.Repo.Transaction.transaction/4

Thanks !

Compilation error with Ecto 3.1.2 and higher

I'm getting a compilation error when i use ex_audit with ecto 3.1.2 or higher.

== Compilation error in file example/repo.ex ==
** (MatchError) no match of right hand side value: {:ex_audit, Ecto.Adapters.Postgres, ExAudit.Test.Repo, [Ecto.Adapter, Ecto.Adapter.Migration, Ecto.Adapter.Queryable, Ecto.Adapter.Schema, Ecto.Adapter.Transaction, Ecto.Adapter.Storage, Ecto.Adapter.Structure]}
    example/repo.ex:2: (module)
    (stdlib) erl_eval.erl:680: :erl_eval.do_apply/6

With ecto 3.1.1 everything seems fine.

To reproduce it:

  • clone this repo
  • pin ecto to at least 3.1.2
  • run mix deps.get
  • run tests

About maintaining this library.

Hello, i know this is oss and i'm very glad about you made this library. Its has been handy for me. I truly grateful that you have done the work and created it. But i see that there are some maintenance latency.

Few PR have been idle for long time. Its an issue thats needs fixing (IMHO). Currently there are few PRs that have been merged in other forks:

This creates divergence, as one has some PRs, other have different PRs. With time each one will diverge more and more. While all forks and main library will work on base level, at some point one will have features z,x,c and other will have x,c,v creating case where no library have z and v features together.

As a developer how can i know which one is the main library? I will require thorough research from user pow.

My question is: are you willing to add some other developer as maintainers? That could go through/merge/create releases PRs for this library? If you are willing i'm would like to step forth and try my best to help you.

Im i no way trying to disrespect you. As it is free software you don't have any obligation to maintain it more than you are doing it currently. If thats the case, i personally would appreciate open communication about it. And then i most likely would try my best to create feature full fork. And try to maintain separately, but i rather collaborate.

Either case, have a great day 👍🏼

Tracking schema with associations

Hello,

I track schemas items, reservations, and items_reservations. I want to show all Item changes, including the item reservations, when any item_reservation is inserted or deleted.

I start the query from versions and searching for entity_schema is items, and entity_id is the current item id. After that I need to join all 'items_reservations' versions to know when item was reserved or deleted from reservation.

Do you have any solution or idea, how can I make this?

Thanks

(Postgrex.Error) ERROR 23502 (not_null_violation) null value in column "id" of relation "versions" violates not-null constraint

[debug] QUERY OK source="orders" db=0.6ms
UPDATE "orders" SET "status" = $1, "updated_at" = $2 WHERE "id" = $3 [:active, ~N[2024-02-08 11:55:30], "f1b82004-7560-40d6-b98f-3909319a15da"]
↳ Logistic.Orders.update_order/2, at: lib/logistic/orders.ex:78
[debug] QUERY ERROR source="versions" db=0.6ms
INSERT INTO "versions" ("action","patch","recorded_at","entity_id","entity_schema") VALUES ($1,$2,$3,$4,$5) [:updated, %{status: {:changed, {:primitive_change, :delivered, :active}}, updated_at: {:changed, %{second: {:changed, {:primitive_change, 23, 30}}, day: {:changed, {:primitive_change, 6, 8}}, minute: {:changed, {:primitive_change, 58, 55}}, hour: {:changed, {:primitive_change, 14, 11}}}}}, ~U[2024-02-08 11:55:30Z], "f1b82004-7560-40d6-b98f-3909319a15da", Logistic.Orders.Order]
↳ ExAudit.Schema.augment_transaction/3, at: lib/repo/schema.ex:146
[debug] QUERY OK db=0.3ms
rollback []
↳ ExAudit.Schema.augment_transaction/3, at: lib/repo/schema.ex:146
[error] GenServer #PID<0.9328.0> terminating
** (Postgrex.Error) ERROR 23502 (not_null_violation) null value in column "id" of relation "versions" violates not-null constraint

table: versions
column: id

Failing row contains (null, \x8374000000027706737461747573680277076368616e676564680377107072..., f1b82004-7560-40d6-b98f-3909319a15da, orders, updated, 2024-02-08 11:55:30, f, null, null, null, null, null).
(ecto_sql 3.11.1) lib/ecto/adapters/sql.ex:1054: Ecto.Adapters.SQL.raise_sql_call_error/1
(ecto_sql 3.11.1) lib/ecto/adapters/sql.ex:925: Ecto.Adapters.SQL.insert_all/9
(ecto 3.11.1) lib/ecto/repo/schema.ex:59: Ecto.Repo.Schema.do_insert_all/7
(ex_audit 0.10.0) lib/repo/schema.ex:34: anonymous fn/5 in ExAudit.Schema.update/4
(ex_audit 0.10.0) lib/repo/schema.ex:155: ExAudit.Schema.run_in_multi/4
(ecto 3.11.1) lib/ecto/multi.ex:883: Ecto.Multi.apply_operation/5
(elixir 1.16.0) lib/enum.ex:2528: Enum."-reduce/3-lists^foldl/2-0-"/3
(ecto 3.11.1) lib/ecto/multi.ex:856: anonymous fn/5 in Ecto.Multi.apply_operations/5
(ecto_sql 3.11.1) lib/ecto/adapters/sql.ex:1358: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
(db_connection 2.6.0) lib/db_connection.ex:1710: DBConnection.run_transaction/4
(ecto 3.11.1) lib/ecto/repo/transaction.ex:18: Ecto.Repo.Transaction.transaction/4
(ex_audit 0.10.0) lib/repo/schema.ex:146: ExAudit.Schema.augment_transaction/3
(logistic 0.1.0) lib/logistic/orders.ex:78: Logistic.Orders.update_order/2
(logistic 0.1.0) lib/logistic_web/live/dashboard_operator/order_live/form_component.ex:180: LogisticWeb.DashboardOperator.OrderLive.FormComponent.handle_event/3
(phoenix_live_view 0.20.2) lib/phoenix_live_view/channel.ex:716: anonymous fn/4 in Phoenix.LiveView.Channel.inner_component_handle_event/4

(phoenix_live_view 0.20.2) lib/phoenix_live_view/diff.ex:211: Phoenix.LiveView.Diff.write_component/4
(phoenix_live_view 0.20.2) lib/phoenix_live_view/channel.ex:639: Phoenix.LiveView.Channel.component_handle_event/6
(stdlib 5.2) gen_server.erl:1095: :gen_server.try_handle_info/3
(stdlib 5.2) gen_server.erl:1183: :gen_server.handle_msg/6

config :logistic, Logistic.Repo, migration_primary_key: [name: :id, type: :uuid]
defmodule Logistic.Schema do
@moduledoc false
defmacro using(_) do
quote do
use Ecto.Schema
@primary_key {:id, Ecto.UUID, autogenerate: true}
@foreign_key_type Ecto.UUID
end
end
end

default_options/1 can't be overridden

Hello,

I've defined in MyApp.Repo the function default_options/1. Unfortunately ExAudit prevents overriding that function. At least it's not used. When I replace

    defmodule MyApp.Repo  do
      use ExAudit.Repo

by

    defmodule MyApp.Repo do
      use Ecto.Repo

it's working correctly.

Moreover when compiling the ExAudit dependency I get warning: conflicting behaviours found. function default_options/1 is required by Ecto.Repo and ExAudit.Repo (in module MyApp.Repo).

support for different Version schemf for multi repo.

with the way current config loading, it looks like there is no way to use ex_audit for different Version schema for multiple Ecto.Repo?

Will it be something that we will consider adding support for ex_audit?

I started a branch from my fork(https://github.com/AltoFinancial/ex_audit/tree/kafai--support-multi-repos). Please let me know your opinion on it. (Sorry for the code quality. I was rushing and I will further improve it if it's what we think should add for this library)

Handling Decimal changes

This is part observation, part feature request, part wondering if I'm missing something. ExAudit was easy to set up and works great, however we store dollar amounts as Decimals. Since these are represented as structs with :coef, :sign, and :exp keys, ExAudit only records what changed in the struct. For example: total: {:changed, %{coef: {:changed, {:primitive_change, 401, 153400}}}}.

I've struggled to figure out how to use that data when printing out a change summary. Even if I look at two consecutive Version records, I don't have enough data to represent a Decimal. I'd have to start with the initial :added struct and apply each Version's change to it and show the use the last two revisions to show the old and new Decimal value.

Instead, I am adding a custom field to my Version and I am storing all the before/after Decimals in a map in there. This also has its frustrations however, as I have a main record (purchase order) as well as a has_many (order items) that get recorded at the same time. I don't see a way to tell ExAudit.track which Version record to put the custom data on. I have it recording the field on the main record, but it seems I have to record all the custom data in that Version, rather than being able to put the order items' decimal changes in its Version record.

I can make this work in a pinch but was curious if there's some other approach I'm missing. Also, it seems like it would be useful to be able to record certain types, like decimals, as full struct diffs or before and after structs.

Prefix support

Hi, I am actively using PG schemas and prefix options for Repo.insert(%{}, prefix: "my_shema") in my current multi-tenant application. Any plans on support the prefix option for schema? I can contribute if you are open.

Currently, I can add versions inside every schema, and it works fine on insert and update, but for delete, I got the trouble:

** (Postgres.Error) ERROR 42P01 (undefined_table) relation "versions" does not exist

    query: INSERT INTO "versions" ("account_id","action","entity_id","entity_schema","id","patch","recorded_at") VALUES ($1,$2,$3,$4,$5,$6,$7)
    (ecto_sql 3.6.2) lib/ecto/adapters/sql.ex:760: Ecto.Adapters.SQL.raise_sql_call_error/1
    (ecto_sql 3.6.2) lib/ecto/adapters/sql.ex:667: Ecto.Adapters.SQL.insert_all/9
    (ecto 3.6.2) lib/ecto/repo/schema.ex:58: Ecto.Repo.Schema.do_insert_all/7
    (ex_audit 0.9.0) lib/repo/schema.ex:72: anonymous fn/4 in ExAudit.Schema.delete/4
    (ex_audit 0.9.0) lib/repo/schema.ex:155: ExAudit.Schema.run_in_multi/4
    (ecto 3.6.2) lib/ecto/multi.ex:716: Ecto.Multi.apply_operation/5
    (elixir 1.12.1) lib/enum.ex:2356: Enum."-reduce/3-lists^foldl/2-0-"/3
    (ecto 3.6.2) lib/ecto/multi.ex:690: anonymous fn/5 in Ecto.Multi.apply_operations/5
    (ecto_sql 3.6.2) lib/ecto/adapters/sql.ex:1017: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4

Maybe somehow it's possible to disable delete tracking for now? I can also contribute these options as well.

p.s. Thanks for doing this amazing open-source solution that should simplify my work a lot

Documentation/Suggestion: Add example of how to manually use track_changes?

We started using the library and on a specific scenario, we have to use Ecto.Multi .
Looking through the documentation/code, the usage of ExAudit.Tracking.track_change implies that user will build the resulting_struct through Ecto.Repo.Schema and that is clear unless you go through the codebase.
Also, Ecto Documentation does not expose Ecto.Repo.Schema which makes the usage trickier.

btw, plugin works great on basic operations.

A way to use the original Repo

It would be great if we can use the modified Repo of ex_audit as a separate module that calls the original Repo, so it will not completely replace the Repo, something like VersionedRepo. If parts of the app are potentially versioned (for instance a CMS), and other parts are not (the user-facing app), as in our case, I'd like to not have the performance impact of versioning on the user-facing app when I know versioning is not involved there.

Tracking non persistent events

Hello,

What would be the suggested way of (ab)using this library so that events that don't persist changes in the database are tracked along with those that do?

For example, track every "show" action for certain entities.

If this is a feature that could be scoped within this library I'd appreciate any pointers so I can submit a patch for review.

The simplest approach I can think of would be to record an empty Version.patch with a new (custom?) ExAudit.Type.Action. Would that make any sense?

Error when upgrading to ecto_sql 3.2

I'm getting this error when using version 3.2 of ecto_sql:

** (UndefinedFunctionError) function NeopagSchema.Repo.prepare_query/3 is undefined or private, but the behaviour Ecto.Repo expects it to be present (neopag_schema) NeopagSchema.Repo.prepare_query(:all, #Ecto.Query<from c0 in NeopagSchema.Store.ChargingSetting, select: c0>, []) (ecto) lib/ecto/repo/queryable.ex:159: Ecto.Repo.Queryable.execute/4 (ecto) lib/ecto/repo/queryable.ex:17: Ecto.Repo.Queryable.all/3

I've tried to redefine the prepare_query function in the module ExAudit.Repo with no success.

Support for Repo.insert_or_update

I am experimenting with ExAudit for an application that makes heavy use of the Repo.insert_or_update function. Unfortunately, ExAudit does not currently capture and record the resulting insert or update operation.

I have verified that Repo.insert and Repo.update operations are both properly recorded in the versions table.

Getting some errors on existing code

Hi Guys. I setup ExAudit and it seems is running fine, but some modules stopped working after replacing the original Repo by ExAudit.

For example. Aggregations are not working anymore.

assert MyApp.Repo.aggregate(MyApp.SessionEvent, :count, :id) == 0
** (UndefinedFunctionError) function nil.id/0 is undefined

I am using Ecto "3.4.5" and I had to setup override: true to be compatible with ExAudit, but I think there's something wrong.

Does anyone can help me understand this?

New error prevent tracked schemas to write into DB

This happens while inserts and updates.
I wonder what wrong here, since this problem appeared this week.
You know how to fix this?

[error] GenServer #PID<0.960.0> terminating
** (FunctionClauseError) no function clause matching in Ecto.Repo.Schema.do_update/4
    (ecto 3.9.1) Ecto.Repo.Schema.do_update(MyApp.Repo, MyApp.Repo, #Ecto.Changeset<action: nil, changes: %{gender: "Female"}, errors: [], data: #MyApp.Accounts.User<>, valid?: true>, [ex_audit_custom: []])
    (ex_audit 0.9.0) lib/repo/schema.ex:30: anonymous fn/4 in ExAudit.Schema.update/4
    (ex_audit 0.9.0) lib/repo/schema.ex:155: ExAudit.Schema.run_in_multi/4
    (ecto 3.9.1) lib/ecto/multi.ex:801: Ecto.Multi.apply_operation/5
    (elixir 1.12.3) lib/enum.ex:2385: Enum."-reduce/3-lists^foldl/2-0-"/3
    (ecto 3.9.1) lib/ecto/multi.ex:775: anonymous fn/5 in Ecto.Multi.apply_operations/5
    (ecto_sql 3.9.0) lib/ecto/adapters/sql.ex:1190: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
    (db_connection 2.4.2) lib/db_connection.ex:1562: DBConnection.run_transaction/4
    (ecto 3.9.1) lib/ecto/repo/transaction.ex:18: Ecto.Repo.Transaction.transaction/4
    (ex_audit 0.9.0) lib/repo/schema.ex:146: ExAudit.Schema.augment_transaction/3
    ````

Increased logging after switching to using ExAudit.Repo

In a Phoenix app I maintain when I switch our sole Repo from using Ecto.Repo to ExAudit.Repo, I see innumerable log lines after booting up the app locally:

2020-05-29T16:05:28.559Z [DEBUG] QUERY OK db=0.1ms
begin []
2020-05-29T16:05:28.559Z [DEBUG] QUERY OK db=0.1ms
begin []
2020-05-29T16:05:28.559Z [DEBUG] QUERY OK db=0.2ms
begin []
2020-05-29T16:05:28.559Z [DEBUG] QUERY OK db=0.2ms
begin []
2020-05-29T16:05:28.559Z [DEBUG] QUERY OK db=0.2ms
begin []
2020-05-29T16:05:28.560Z [DEBUG] QUERY OK db=0.5ms
commit []
2020-05-29T16:05:28.560Z [DEBUG] QUERY OK db=0.7ms
commit []
2020-05-29T16:05:28.560Z [DEBUG] QUERY OK db=0.5ms
commit []
2020-05-29T16:05:28.560Z [DEBUG] QUERY OK db=0.6ms
commit []
2020-05-29T16:05:28.560Z [DEBUG] QUERY OK db=0.8ms
commit []

My config is set up with an empty list for tracked_schemas and besides that just a version_schema set.

Is this happening to anyone else and/or expected?

Getting error when using ExAudit.Repo

Hy Guys. Thank you for the library.

I put this clause into my code: use ExAudit.Repo

After compiling I get this error: key :otp_app not found in: []

Does anyone know what's happening?

Tracking deletion of nested association in parent model update

Lets say that I have an Article entity with a has_many :comments in its schema. Imagine that I'm going through a comments listings in my UI and pick one of them to be deleted but also the UI allows me to update other attributes of the Article entity altogether. When I hit the submit button to actually update the entire article entity and its nested associations, will the current implementation detect the missing comment and include the comment deletion in the versions that will be inserted from this update operation? This question also applies to has_one associations too.

Is this kind of logic currently supported? I went over the source code, but apparently I didn't spot the logic that would handle that kind of scenario.

(Sorry to bring this doubt as an issue, but I didn't find a better way to make sure about the existence of this feature)

Support for insert_all

insert_all does not seem to be supported. The relevant code in ExAudit.Schema module looks like a WIP ?

defmodule ExAudit.Schema do
  def insert_all(module, adapter, schema_or_source, entries, opts) do
    # TODO!
    opts = augment_opts(opts)
    Ecto.Repo.Schema.insert_all(module, adapter, schema_or_source, entries, opts)
  end

  ...

end

Viewing entities at a version, without reverting in the database

Hi and thanks for the useful library!

I would like to allow my users to preview the entity before a specific version. It seems that currently one would have to use the revert/2 function on the Repo, but this would actually rollback the changes in the database as well.

Would you be open to accepting a PR that essentially breaks Queryable.revert/3 in half, extracting the first half in to a function like entity_at_version/3 (happy to discuss better names, maybe preview/3?). This function would simply return the result of Enum.reduce(versions, struct, &_revert/2), which is currently in Queryable.revert/3, without changing anything in the database. We could then expose this function as well, on the Repo.

Thoughts?

But does not write to the database ex_audit_custom

def insert_versions(module, changes, opts)

[
%{
action: :updated,
patch: %{
status: {:changed, {:primitive_change, :active, :delivered}},
updated_at: {:changed,
%{
second: {:changed, {:primitive_change, 34, 58}},
minute: {:changed, {:primitive_change, 10, 19}}
}}
},
entity_id: "f1b82004-7560-40d6-b98f-3909319a15da",
entity_schema: Logistic.Orders.Order
}
]
Logistic.Repo
[
ex_audit_custom: [operator_id: "2840b6a9-3386-414d-bc90-1151154ab575"],
stacktrace: [
{Ecto.Repo.Supervisor, :tuplet, 2,
[file: ~c"lib/ecto/repo/supervisor.ex", line: 163]},
{Logistic.Repo, :update, 2,
[file: ~c"deps/ex_audit/lib/repo/repo.ex", line: 119]},
{Logistic.Orders, :update_order, 2,
[file: ~c"lib/logistic/orders.ex", line: 78]},
{LogisticWeb.DashboardOperator.OrderLive.FormComponent, :handle_event, 3,
[
file: ~c"lib/logistic_web/live/dashboard_operator/order_live/form_component.ex",
line: 180
]},
{Phoenix.LiveView.Channel, :"-inner_component_handle_event/4-fun-0-", 4,
[file: ~c"lib/phoenix_live_view/channel.ex", line: 716]},
{:telemetry, :span, 3,
[
file: ~c"/logistic/deps/telemetry/src/telemetry.erl",
line: 321
]},
{Phoenix.LiveView.Diff, :write_component, 4,
[file: ~c"lib/phoenix_live_view/diff.ex", line: 211]},
{Phoenix.LiveView.Channel, :component_handle_event, 6,
[file: ~c"lib/phoenix_live_view/channel.ex", line: 639]},
{:gen_server, :try_handle_info, 3, [file: ~c"gen_server.erl", line: 1095]},
{:gen_server, :handle_msg, 6, [file: ~c"gen_server.erl", line: 1183]},
{:proc_lib, :wake_up, 3, [file: ~c"proc_lib.erl", line: 251]}
]
]

[debug] QUERY OK source="versions" db=3.3ms
INSERT INTO "versions" ("action","patch","recorded_at","entity_id","entity_schema") VALUES ($1,$2,$3,$4,$5) [:updated, %{status: {:changed, {:primitive_change, :active, :delivered}}, updated_at: {:changed, %{second: {:changed, {:primitive_change, 34, 58}}, minute: {:changed, {:primitive_change, 10, 19}}}}}, ~U[2024-02-09 11:19:58Z], "f1b82004-7560-40d6-b98f-3909319a15da", Logistic.Orders.Order]

ex_audit_custom: [operator_id: "2840b6a9-3386-414d-bc90-1151154ab575"], --- OK

But does not write to the database

version_schema() |> apply(:changeset, [empty_version_schema, change]) |> Map.get(:changes)

#Ecto.Changeset<
action: nil,
changes: %{
action: :updated,
patch: %{
status: {:changed, {:primitive_change, :active, :win}},
updated_at: {:changed,
%{
second: {:changed, {:primitive_change, 22, 9}},
minute: {:changed, {:primitive_change, 0, 6}}
}}
},
recorded_at: ~U[2024-02-09 12:06:09Z],
entity_id: "ff7908c4-118f-4c04-acf5-1712b0a2f4d9",
entity_schema: Logistic.Orders.Order
},
errors: [operator_id: {"is invalid", [type: :id, validation: :cast]}],
data: #Logistic.Version<>,
valid?: false

Support for multi schema versioning

Hi!

I have a need to track changes of entities in different schemas.

Currently, we define version_schema in config :ex_audit. But this forces a single version table in a given schema. Is there a way to have multiple version tables, one per schema?

Is this something of interest to somebody else?

Thanks for all the hard work.

A few questions

Awesome module! Came across this, and this is very similar to something I wanted to build for a long time. I am working on an Elixir content management system based on GraphQL / React and Ecto (will be open source soon) and I might integrate this. I have a few questions:

  • How does it deal with adding and removing fields in the schema?
  • How to deal with data migrations (in migrations you cannot really use schema names as they might change or be removed)
  • Would it be possible to also add a version that is not yet applied to the DB (like a draft).

Store patch as map(json)

Any plans to add a serialization strategy for Patch type? It would be great to view and most important query changes.

Upserts support

Options such as on_conflict, conflict_target and returning for upserts are passed down to insert_versions and cause a crash since the target is non existent for the Version schema.

defmodule ExAudit.Test.Repo.Migrations.UniqueConstraint do
  use Ecto.Migration

  def change do
    alter table(:users) do
      add(:unique_field, :string)
    end

    create(unique_index(:users, [:unique_field]))
  end
end
test "on conflict" do
    user = Repo.insert!(%User{name: "Admin", email: "[email protected]", unique_field: "foo"})
    user2 = %User{name: "Should not change", email: "[email protected]", unique_field: "foo"}
    # this will crash
    Repo.insert!(user2, on_conflict: {:replace, [:email]}, conflict_target: :unique_field, returning: true)
end

I think it'd require dropping specific opts in insert_versions and introducing a new action type. I haven't figured out yet how to qualify upsert as either update or insert -- the most tricky bit is calculating diffs to support the rollback feature.

Recording custom data with Plug does not work in Phoenix LiveView App

Hello,

thanks a lot for your library!

I've started using it in a Phoenix Live App. Everything works, except recording custom data like an actor_id using a Plug. I've added a Plug as described in the README. Then I've integrated it into a pipeline in my router. The Plug is definitely executed with the right information.
Non-LiveView controller actions include the custom data. But LiveView actions not. Probably because of the websocket. Any idea how to adapt that for LiveView? Thank a lot!

Using ex_audit breaks `returning` option for Repo functions

The insert and update functions in Ecto accept a returning option to specify specific fields to return after the operation. Passing this option to an ExAudit repo results in the following error.

** (KeyError) key :access_id not found in: %{action: {:action, ExAudit.Type.Action}, entity_id: {:entity_id, :integer}, entity_schema: {:entity_schema, ExAudit.Type.Schema}, id: {:id, :id}, patch: {:patch, ExAudit.Type.Patch}, recorded_at: {:recorded_at, :utc_datetime}, rollback: {:rollback, :boolean}, user_id: {:user_id, :id}}
    :erlang.map_get(:access_id, %{action: {:action, ExAudit.Type.Action}, entity_id: {:entity_id, :integer}, entity_schema: {:entity_schema, ExAudit.Type.Schema}, id: {:id, :id}, patch: {:patch, ExAudit.Type.Patch}, recorded_at: {:recorded_at, :utc_datetime}, rollback: {:rollback, :boolean}, user_id: {:user_id, :id}})
    (ecto 3.10.0) lib/ecto/repo/schema.ex:611: anonymous fn/3 in Ecto.Repo.Schema.fields_to_sources/2
    (elixir 1.14.3) lib/enum.ex:2468: Enum."-reduce/3-lists^foldl/2-0-"/3
    (ecto 3.10.0) lib/ecto/repo/schema.ex:45: Ecto.Repo.Schema.do_insert_all/7
    (ex_audit 0.10.0) lib/repo/schema.ex:16: anonymous fn/5 in ExAudit.Schema.insert/4
    (ex_audit 0.10.0) lib/repo/schema.ex:155: ExAudit.Schema.run_in_multi/4
    (ecto 3.10.0) lib/ecto/multi.ex:832: Ecto.Multi.apply_operation/5
    (elixir 1.14.3) lib/enum.ex:2468: Enum."-reduce/3-lists^foldl/2-0-"/3
    (ecto 3.10.0) lib/ecto/multi.ex:806: anonymous fn/5 in Ecto.Multi.apply_operations/5
    (ecto_sql 3.10.0) lib/ecto/adapters/sql.ex:1203: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
    (db_connection 2.5.0) lib/db_connection.ex:1630: DBConnection.run_transaction/4
    (ecto 3.10.0) lib/ecto/repo/transaction.ex:18: Ecto.Repo.Transaction.transaction/4
    (ex_audit 0.10.0) lib/repo/schema.ex:146: ExAudit.Schema.augment_transaction/3

Loss of significant error messages

Wrapping all calls to insert, update, etc causes valuable information to be lost about normal database transactions.

With a schema that uses Changeset.foreign_key_constraint/3 or Changeset.no_assoc_constraint, the messages are lost.

def prepare_delete(category) do
  category
  |> change
  |> no_assoc_constraint(:topics)
end



# Without using ExAudit.Repo

Category
|> Repo.get(id)
|> Category.prepare_delete
|> Repo.delete

# {:error,
#  #Ecto.Changeset<action: :delete, changes: %{},
#   errors: [topics: {"are still associated with this entry", []}],
#   data: #Category<>, valid?: false>}


# Using ExAudit.Repo

Category
|> Repo.get(id)
|> Category.prepare_delete
|> Repo.delete

# {:error, :rollback}

I realize this is not a straightforward (or even possible) thing to fix, but it should be very well documented that database errors and constraint messages are lost. I have a few places where there are multiple database-enforced constraints, and when using the provided Repo module, I have no way of knowing which constraint failed

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.