Migration guide to RailsEventStore::JSONClient

[tomasc]

Migration guide to RailsEventStore::JSONClient

[lukaszreszke]

Hey @tomasc, thanks for reaching out.

JSON/JSONB column support for PostgreSQL is nothing new in RailsEventStore. Event serialization formats are described here

RailsEventStore::JSONClient is a result of our experience working with RailsEventStore in projects using the PostgreSQL database. However, the same configuration was possible to build before version 2.9.0 of RES.

We don't plan to develop a migration script for the reasons mentioned above. However, there's a blog post by @pawelpacana that you might find useful (https://blog.arkency.com/how-to-migrate-large-database-tables-without-a-headache/), it should help you build and execute the migration script.

Cheers,
Łukasz

[mostlyobvious]

Is that even possible? Would you be able to briefly outline a migration script?

What you want here is changing data and metadata column types in event_store_events table, along with converting its contents to new format.

You can do it either offline (with downtime, stopping new writes) or mostly online (simultaneous writing in two tables, backfilling missing record and finally performing table switch).

In any case you'd probably need two instances of RailsEventStore::Client at hand. One to read existing events from storage, configured for current database columns and used serialization method (i.e. binary + YAML). Second one to be used in new "writes", transforming event objects by the rules of new serializer.

Example of backfilling:

# given new table for changed events
class CreateEventStoreEventsOfTheFuture < ActiveRecord::Migration[4.2]
  def change
    create_table(:event_store_events_v2, id: :bigserial, force: false) do |t|
      t.references  :event,       null: false, type: :uuid
      t.string      :event_type,  null: false
      t.jsonb       :metadata
      t.jsonb       :data,        null: false
      t.datetime    :created_at,  null: false
      t.datetime    :valid_at,    null: true
    end
    add_index :event_store_events_v2, :event_id, unique: true
    add_index :event_store_events_v2, :created_at
    add_index :event_store_events_v2, :valid_at
    add_index :event_store_events_v2, :event_type
  end
end

# and the model to operate on this table
class EventV2 < ::ActiveRecord::Base
  self.primary_key = :id
  self.table_name = "event_store_events_v2"
end

# and the glue needed to work on the new table
repository = 
  RubyEventStore::ActiveRecord::EventRepository.new(
    serializer: RubyEventStore::NULL, # as seen in RailsEventStore::JSONClient internals
    model_factory: ->{ [EventV2, RubyEventStore::ActiveRecord::EventInStream] }
  )

# then having both clients
current_client = RailsEventStore::Client.new     # YAML, binary
future_client  = RailsEventStore::JSONClient.new(repository: repository) # JSON, jsonb

# we can transform in Ruby from one serialization format to another
current_client.read.in_batches_of(1000).each do |event| 
  future_client.append(event)
end

Results — sequence numbers, event ids and timestamps intact; data and metadata converted:

migrate_data_and_metadata_development=# SELECT * from event_store_events;
 id |               event_id               |     event_type     |                                                        metadata                                                        |                                                                                                                                                                                                      data                                                                                                                                                                                                      |         created_at         | valid_at
----+--------------------------------------+--------------------+------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------+----------
  1 | ccc0bd55-9dc8-4376-9b54-9b828851a1c6 | SomethingSomething | \x2d2d2d0a3a636f7272656c6174696f6e5f69643a2032346336323530312d643932352d346630662d393965612d6164373465643362373562310a | \x2d2d2d207b7d0a                                                                                                                                                                                                                                                                                                                                                                                               | 2023-02-02 15:57:43.385879 |
  2 | 0a39eb80-a8e8-40b6-b98c-614939860edd | SomethingSomething | \x2d2d2d0a3a636f7272656c6174696f6e5f69643a2038623430303861342d363435392d343637352d626536392d3463396638353335303034350a | \x2d2d2d0a3a746f6461793a20323032332d30322d30320a                                                                                                                                                                                                                                                                                                                                                               | 2023-02-02 16:03:00.979878 |
  3 | 20207ea0-d953-477c-8cc1-9441f3d24bea | SomethingSomething | \x2d2d2d0a3a636f7272656c6174696f6e5f69643a2037343230393063642d613864322d343431362d623430632d3331623932663265323738630a | \x2d2d2d0a3a746f6d6f72726f773a2021727562792f6f626a6563743a416374697665537570706f72743a3a54696d65576974685a6f6e650a20207574633a20323032332d30322d30332031363a31363a32302e333838363333303030205a0a20207a6f6e653a2021727562792f6f626a6563743a416374697665537570706f72743a3a54696d655a6f6e650a202020206e616d653a204574632f5554430a202074696d653a20323032332d30322d30332031363a31363a32302e333838363333303030205a0a | 2023-02-02 16:16:20.390303 |
(3 rows)

migrate_data_and_metadata_development=# SELECT * from event_store_events_v2;
 id |               event_id               |     event_type     |                                                                                           metadata                                                                                           |                    data                     |         created_at         | valid_at
----+--------------------------------------+--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------+----------------------------+----------
  1 | ccc0bd55-9dc8-4376-9b54-9b828851a1c6 | SomethingSomething | {"types": {"data": {}, "metadata": {"correlation_id": ["Symbol", "String"]}}, "correlation_id": "24c62501-d925-4f0f-99ea-ad74ed3b75b1"}                                                      | {}                                          | 2023-02-02 15:57:43.385879 |
  2 | 0a39eb80-a8e8-40b6-b98c-614939860edd | SomethingSomething | {"types": {"data": {"today": ["Symbol", "Date"]}, "metadata": {"correlation_id": ["Symbol", "String"]}}, "correlation_id": "8b4008a4-6459-4675-be69-4c9f85350045"}                           | {"today": "2023-02-02"}                     | 2023-02-02 16:03:00.979878 |
  3 | 20207ea0-d953-477c-8cc1-9441f3d24bea | SomethingSomething | {"types": {"data": {"tomorrow": ["Symbol", "ActiveSupport::TimeWithZone"]}, "metadata": {"correlation_id": ["Symbol", "String"]}}, "correlation_id": "742090cd-a8d2-4416-b40c-31b92f2e278c"} | {"tomorrow": "2023-02-03T16:16:20.388633Z"} | 2023-02-02 16:16:20.390303 |
(3 rows)

Writing simultaneously to both current and future event_store_events tables can be tricky. You'd like to retain sequence numbers in the future table as they are in the current table (id column). But you probably won't be able to use database triggers, as described.

Whatever the method to double-write, here's useful snippet to get raw data and metadata from Event object, suitable for SQL INSERT:

# as seen in RailsEventStore::JSONClient internals
future_mapper = 
          RubyEventStore::Mappers::InstrumentedMapper.new(
            RubyEventStore::Mappers::PipelineMapper.new(
              RubyEventStore::Mappers::Pipeline.new(
                RubyEventStore::Mappers::Transformation::PreserveTypes
                  .new
                  .register(Symbol, serializer: ->(v) { v.to_s }, deserializer: ->(v) { v.to_sym })
                  .register(
                    Time,
                    serializer: ->(v) { v.iso8601(RubyEventStore::TIMESTAMP_PRECISION) },
                    deserializer: ->(v) { Time.iso8601(v) }
                  )
                  .register(
                    ActiveSupport::TimeWithZone,
                    serializer: ->(v) { v.iso8601(RubyEventStore::TIMESTAMP_PRECISION) },
                    deserializer: ->(v) { Time.iso8601(v).in_time_zone },
                    stored_type: ->(*) { "ActiveSupport::TimeWithZone" }
                  )
                  .register(Date, serializer: ->(v) { v.iso8601 }, deserializer: ->(v) { Date.iso8601(v) })
                  .register(DateTime, serializer: ->(v) { v.iso8601 }, deserializer: ->(v) { DateTime.iso8601(v) })
                  .register(BigDecimal, serializer: ->(v) { v.to_s }, deserializer: ->(v) { BigDecimal(v) }),
                RubyEventStore::Mappers::Transformation::SymbolizeMetadataKeys.new
              )
            ),
            ActiveSupport::Notifications
          )

# extract raw values

event
=>
#<RubyEventStore::Event:0x000000010aec7740
 @data={:tomorrow=>Fri, 03 Feb 2023 16:16:20.388633000 UTC +00:00},
 @event_id="20207ea0-d953-477c-8cc1-9441f3d24bea",
 @metadata=#<RubyEventStore::Metadata:0x000000010aec7718 @h={:correlation_id=>"742090cd-a8d2-4416-b40c-31b92f2e278c", :timestamp=>2023-02-02 16:16:20.390303 UTC, :valid_at=>2023-02-02 16:16:20.390303 UTC, :event_type=>"SomethingSomething"}>>

record = future_mapper.event_to_record(event)
=>
#<RubyEventStore::Record:0x000000010ae05230
 @data={"tomorrow"=>"2023-02-03T16:16:20.388633Z"},
 @event_id="20207ea0-d953-477c-8cc1-9441f3d24bea",
 @event_type="SomethingSomething",
 @metadata={:correlation_id=>"742090cd-a8d2-4416-b40c-31b92f2e278c", :event_type=>"SomethingSomething", :types=>{:data=>{:tomorrow=>["Symbol", "ActiveSupport::TimeWithZone"]}, :metadata=>{:correlation_id=>["Symbol", "String"], :event_type=>["Symbol", "String"]}}},
 @serialized_records={},
 @timestamp=2023-02-02 16:16:20.390303 UTC,
 @valid_at=2023-02-02 16:16:20.390303 UTC>

serialized_record = record.serialize(JSON)
=>
#<RubyEventStore::SerializedRecord:0x000000010b184500
 @data="{\"tomorrow\":\"2023-02-03T16:16:20.388633Z\"}",
 @event_id="20207ea0-d953-477c-8cc1-9441f3d24bea",
 @event_type="SomethingSomething",
 @metadata="{\"correlation_id\":\"742090cd-a8d2-4416-b40c-31b92f2e278c\",\"event_type\":\"SomethingSomething\",\"types\":{\"data\":{\"tomorrow\":[\"Symbol\",\"ActiveSupport::TimeWithZone\"]},\"metadata\":{\"correlation_id\":[\"Symbol\",\"String\"],\"event_type\":[\"Symbol\",\"String\"]}}}",
 @timestamp="2023-02-02T16:16:20.390303Z",
 @valid_at="2023-02-02T16:16:20.390303Z">

serialized_record.data
=> "{\"tomorrow\":\"2023-02-03T16:16:20.388633Z\"}"

serialized_record.metadata 
=> "{\"correlation_id\":\"742090cd-a8d2-4416-b40c-31b92f2e278c\",\"event_type\":\"SomethingSomething\",\"types\":{\"data\":{\"tomorrow\":[\"Symbol\",\"ActiveSupport::TimeWithZone\"]},\"metadata\":{\"correlation_id\":[\"Symbol\",\"String\"],\"event_type\":[\"Symbol\",\"String\"]}}}"

I hope this helps a bit.

[mostlyobvious]

We also offer commercial support with RailsEventStore. We can help you with writing and performing such migration on your codebase and environment.

Let's connect at [email protected] if you're interested 🙂