Extremely slow load times on Windows due to "Slow SessionWorkflow loop"

[slatinsky]

Extremely slow load times on Windows due to "Slow SessionWorkflow loop"

[slatinsky]

If you think that the process is stuck, please reopen the issue.

There is a tradeoff between the speed of preprocessing and the RAM we need. There are people with gigabytes of export data - that just doesn't fit in their system RAM. I have chosen this speed tradeoff. The preprocessing is still 50-times faster than the time needed to export it from Discord.

[walllable]

Alright, noted - thanks for your time and the explanation, that makes sense to me.

[slatinsky]

Hello,

Slow SessionWorkflow loop messages are completely normal - if you see them, you know that data is pushed to mongodb database and the process is not stuck.

The frontend is not just an simple viewer. The data needs to preprocessed due to some design choices in DCE, where some data is not exported directly, but can be inferred from other data (for example emojis in the message content are not exported, but can be recomputed from reactions,... ).

Also the viewer needs to pick the best data fields if the same message was exported multiple times.

And of course the database is needed for search to work.

The slow start should happen only the first run (+ if you upgrade the program). Else it will detect already processed files and skip them - speeding up the start.

If you are not patient enough, go to http://127.0.0.1:21011 in your browser to view partial results