Pagination and data size limits

[smallbets]

Pagination and data size limits

[j-berman]

@Saunved

Suppose a user's database has over 100mb data. As far as I'm aware, the changeHandler picks up all items available in the database. Does this mean all 100mb will get downloaded on the client side at some point?

Yep, when you call openDatabase, the client loads all items in that database into memory.

Couple things to keep in mind that can help you work around this:

1. A user can create many databases as they want. So both of your suggestions are possible to do on your end (Pagination by date ranges, or number of items). For example, Database X contains January 2019 - March 2019, Database Y contains April 2019 - June 2019, etc. We’ve had thoughts about implementing a library that can smoothly handle more complex logic like this.

2. The client does not load files into memory when you open a database. You need to call getFile to load a file into memory, and you can load byte ranges at a time if you want. So if larger items can be stored as files instead, this would be useful.

We understand that the above still may not be ideal for apps where individual users need to store lots of data that needs to be filtered/sorted/queried. We built Userbase in this way to optimize for end-to-end encryption and simplicity. We felt the current approach we settled on had the best set of tradeoffs, that it would be simplest, both to end developers, and to us implementing, and that it would be useful enough to satisfy the largest number of use cases for traditional web apps.

We also benchmarked Userbase a while back, and found it took ~0.3 seconds to fetch and decrypt 1000 todos in a database, ~0.8 seconds for 10k todos, and ~6.4 seconds for 100k (2.9 GHz Macbook Pro with 100 mb/s network). This is likely inaccurate in Userbase's current form, but this performance (and faster) is certainly possible with our current framework. We would like to spend more time optimizing in the future for sure.

[shamblesides]

A couple of my users actually do have databases in the 10's of MB. I'll ask and see how it's going for them.

[shamblesides]

From that user:

"Its way better now, we had [a chat] with over 500 pages and then we did a new one wich is on 223 pages right now. Since we changed and made a new one we dont have much problems anymore, just tiny disconnects sometimes"

Regarding the first one that was having problems:

"... it fails to connect or is still reconnecting after refreshing a few times, that can take a few mins until it works again."

It's possible that this was actually related to an auxiliary service I was using that was unrelated to userbase, though.

[Saunved]

@shamblesides @j-berman Thanks a lot for the really helpful responses and helping me understand how things work at the moment. The data load times are pretty fast, so that isn't an issue at all.

I think if necessary, one can implement pagination based on date ranges. It's a good enough solution for me at the moment.

Thanks a lot again!
Feel free to close the issue :)

[shamblesides]

I could probably provide a somewhat-adequate description of how the database compaction step works but @j-berman would probably do a better job.

[j-berman]

@shamblesides

"... it fails to connect or is still reconnecting after refreshing a few times, that can take a few mins until it works again."

We do still have some kinks to iron out to prevent this! There is more room for improvement on this front for sure.