diamonddao/discord-etl

This program takes the output from `discord-bot` then transforms and loads into a database

discord-etl's Issues

Include Threads in Datamodel

Discord threads seem to be important to distinguish. Currently, our bot parses them and stores them as channels.

In order to make thread data easy to query, they should be included in our datamodel.

To stay consistent with the rest of the model, these will have an entity table and a history table.

Refactor to pull from a timestamp onward

Refactor ETL to store last parsed timestamp, then on future parses only look over data from that timestamp onwards.

Refactor Reactions Datamodel

Refactor datamodel (in database and figma file) to include a reaction_history table as well as a reaction entity table.

Then refactor ETL accordingly.

reaction entity properties:

• userId
• messageId
• reactionAddedTimestamp

Then all other data, such as "deleted", "emoji", etc can go under the reaction_history table. This stays consistant with the rest of the datamodel.