[RFC] Add support for multiple databases

[apyrgio]

[RFC] Add support for multiple databases

[andreasnomikos]

Why allow segmentation of the db based on the app and not allow sharding of each table so that both database nodes are symmetric?

[apyrgio]

Database sharding is surely interesting, but it requires a Django model that is aware of this as well as being free from cross-database relationships. At this point, our models have neither, which is why the proposed solution is app-based.

Once we have an easy way to manage multiple databases though, the next step will be examining database sharding.

[andreasnomikos]

Yes you are absolutely right that the model has to be somewhat aware of the shading system as you would have to generate guid from the application layer and decide the sharding mechanism (usually a modulo on the guid) from there. Regarding cross-database relationship if you mean foreign keys enforced at the database level it is a common practice that such relationships are enforced at the application layer in large scale applications as the other way clearly does not scale. If your current concerns are scaling you will not get much milage out of the proposed solution. It might provide some flexibility in the sort run, but a big installation would hit problems fast. (I don't have more context or numbers on your current biggest installation so this was just a suggestion for a more long term approach). Thanks.

[apyrgio]

Thanks for your input! Pretty valid points all in all.

Now, I'd say that until we examine database sharding and shape our models accordingly, we need to at least provide to the end-users a simple way to have many databases, as this can affect people that are currently installing Synnefo. This is a hard limitation that needs an immediate solution.

Also, as you may have noticed, we have not touched the issue of scaling here, since this should be the subject of a future RFC that would explicitly address it. The main point of this RFC is to provide a quick and intuitive way to handle multiple databases. If any point we've raised contradicts with the above or has room for improvement, then we'd be glad to know.

Thanks,
Alex

P.S.

(I don't have more context or numbers on your current biggest installation so this was just a
suggestion for a more long term approach)

This is out of the context of this RFC but you can get an idea about the numbers of our biggest installation here. If I'm not mistaken, the number of active users is about 6,500.

[apyrgio]

A solution for this issue (and the issue of database transactions) has been added in the release branch of Synnefo.