Local messages store

[vladimiry]

Local messages store

[vladimiry]

Here is an animated demo of the work in progress "local cache" feature:

• You can see that app is reactive. Means it reacts to the mails/folders/contacts changing and applies the respective patch to the local cache (local database), and it does it efficiently.
• Enabling "local store" in the account setting adds the database/online toggle button near to the account handle. The button has two states, database view, and online view:
  
• It's hard to notice on a small amount of data, but when the actual syncing is happening the view toggle button is blinking in green color - an indication of the data fetching and database patch forming activity.
  

This is a WIP demo, a feature is not yet ready for release, even in the beta channel, but the app version shown on the gif animation is in master branch already. Initially, this feature is going to be available only for the Tutanota account types.

[vladimiry]

Emails export feature has been enabled with v2.0.0-beta.4 release.

[vladimiry]

I was exploring existing Node.js compatible embedded databases with built-in encryption support and have not really found solutions that would keep metadata encrypted as well. Means, it's possible to encrypt values of specific columns/fields, but metadata remains unencrypted. Metadata is for example information like how many rows in the database you have, what is the columns set, empty/filled cells, etc. But such information in some case can also be considered as sensitive.

So I'm considering the following approach. App keeps all the messages in memory. App flushes these messages to the encrypted file with some interval and probably on some triggers. This file would be a brick of bytes, fully encrypted, including metadata, similar to how settings.bin is being currently saved (it's a very small file though). When app starts, it loads the saved file into a memory and process continues as described above. At this stage, attachments are not going to be saved, and so it's not going to be a very memory consuming thing. Later it would be possible to introduce an encrypted binary file storage and fetch files into it in a scheduled manner.

Database encryption key would be generated once and stored in settings.bin. A feature of changing this key can be added later in along with re-encrypting the database with a new key. The encryption algorithm is going to be AES 256 CBC with randomly generated IV on every file saving.

[vladimiry]

I was exploring Tutanota's entity event batch stream and have realized how to efficiently utilize it in app. So it's going to be a complete offline emails storage, including the state syncing part. Messages fetching and syncing is already pushed into the master branch.

[vladimiry]

Encrypted database persistence has just landed into the master branch. So backend logic is completely done for Tutanota and it works efficiently enough. Which means building minimal UI around the local messages store is only remaining blocker for releasing the feature. Alternatively, I'm considering releasing the feature only after the Protonmail also gets backend logic done. I've not even yet started researching Protonmail's Rest API though.

[vladimiry]

I think before jumping into the UI building I will add contacts syncing. So emails, folders, and contacts will be stored in the local database.

[vladimiry]

Contacts storing is now in the master branch, in addition to the mails and folders.

[vladimiry]

I was exploring the possibility to enable conversation view mode for local store viewer, which is in general a demanded and not yet supported feature by the official Web Client, ref1 / ref2. And it looks like a quite possible thing to do with still remaining issues to sort out though (see the weird, 2nd-level, grayed out removed message):

Conversation shown on the example screenshot is 3 level hierarchy conversation. Hierarchical tree-like displaying is possible because Tutanota keeps the conversation structure using nodes with parent/previous reference model.

[vladimiry]

v2.0.0-beta.1 release has been published.

[vladimiry]

Feature got enabled for protonmail accounts with v2.0.0-beta.2 release.

[vladimiry]

Closing the issue as resolved. The final v2 release is going to be published soon. There will be new issues open if I decide to start working on the email attachments syncing/exporting or search in database view mode features.

[vladimiry]

If needed, getting and storing attachments can be added later, as an individual background job.

At this stage, attachments are not going to be saved, and so it's not going to be a very memory consuming thing. Later it would be possible to introduce an encrypted binary file storage and fetch files into it in a scheduled manner.

v4.7.0 is going to optionally enable the live attachments loading during the export process. So full batch export will be happening, including the attachments content.