mapbox/dynamodb-migrator

For migration and cleanup operations on your DynamoDB table

dynamodb-migrator's Issues

Pass config to a setup or preflight function

Many migration scripts are meant to target a very specific table and could lead to Bad Things 😞 if run on others.

I've currently hacked some sanity checks for the table being targeted (and bailing out) by inspecting process.argv from my migrate module but it would be cleaner/more reliable for a setup or preflight function to be called with these options (method/region/table might be sufficient?)

I can take a quick pass at implementing this if this sounds reasonable.

cc @rclark @emilymcafee

Dyno.deserialize doesn't like non-wire-format JSON

The change I made in #3 (use Dyno.deserialize, expecting wire-format from a backup instead of line-deliminated JSON) doesn't play nice with feeding in a list of keys that look like { id: "foo", collection: "foo:bar"}.

I think the migrator should continue support for line-deliminated JSON, especially in stream mode (i.e. for "give me a list of keys to target, and I'll do stuff to them"). Maybe this should be an additional option passed only in stream mode, like a --json flag or something. Since I messed this up in the first place I can take this on, let me know what you think @rclark.

filter objects before the job is run

Right now we store all objects in a buffer while we wait to run them.

For many migrations, you will want to filter out objects that don't need to be migrated.

We should consider supporting a filter function that takes the object and MUST return true or false.

This filter should run before we add the object to the queue.

Filtering script

Provide a simple shell script that reads a backup file, runs a user-provided function over each line, and pipes the output into another file.

Improve or drop automatic logging

Issues #6 and #10 bring up some problems with logging, but we also have serious memory buffering issues with it during large migrations. I'm going to take a pass through to see if it can be improved, but there's a pretty good chance we can't without impacting the way folks expect to be able to write logs in their migration scripts.

We may have to fall back to just console.log + piping to a file as part of the bash command.

Expose the table name being migrated

The table name being operated on is not exposed at runtime through the dyno object. If developers are using TypeScript, the table name is a required part of the update expression. So, to avoid the compilation error, developers must provide their migrations another way to get the table name being worked on - duplicating it with the command line arguments used to invoke dynamodb-migrator.

If the dyno object made the table name string available, we'd be able to avoid this extra step.

Use segmented scans to improve concurrency rate

http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB.html#scan-property

@rclark - possible fix for high cpu migrations running at low concurrencies?

support query mode

Right now we support scan and stream. While we have filter functionality to let you transform a stream so the migration will only run on some items in the export, we can only use this where a database export is usable.

To support filtered results where an export is not viable, we should consider supporting the query operation.

This would require the migration file to export a query attribute. I'm torn on if it should be a function that accepts a dyno instance or a JSON object that describes the dynamo query.

log file location

I think this is supposed to be logged at the beginning of a migration, but right now it's logged at the end. It would be useful to have this up front.

naming log files

It would be awesome to get a timestamp on log files, and maybe other descriptive information, too.

--rate should work for stream mode

The rate flag only works right now when you're running in scan mode. stream + --rate = crash