Support HS upgrade tests

[matrix-org]

Support HS upgrade tests

[richvdh]

Firstly: I love the idea of running upgrade tests; I wish we had something like that for Synapse. Do you run it as part of your CI?

To answer the question here: I agree that it does feel a bit feature-creepy; but given that Complement has most of the framework needed to make it happen, it seems like a bit of a shame for every HS impl to have to implement it from scratch.

I think you can say much the same of Homerunner. Generally, having a collection of commands powered by the internals of Complement doesn't feel like a terrible thing to me.

So basically, +1?

[kegsay]

Yeah CI runs the last published semver release and then the pushed commit over the top, as it takes a while to run every single release on CI all the time. This catches thinkos in upgrade scripts but isn't capable of catching everything as sometimes data corruption happens in earlier versions which then affects the next database migration. For example, dendrite 0.1 had corrupted state snapshots which then caused problems when those rooms made on 0.1 were migrated to HEAD. For that, we need to run everything from the beginning, which we'll just do as a pre-release checklist.

I'll go ahead then with this and see what we end up with.

[ShadowJonathan]

Why should this be part of complement?

Complement, in my eyes, about black-box compliance testing, there's nothing that the spec does to address, acknowledge, see, or even know about server upgrades, as that is a completely implementation-specific concept.

What exactly should complement test for? That servers dont crash when they upgrade? How would that exactly be Complement's job to execute? Why can't that just be a standard external tool that's executed separately in a homeserver's CI pipeline instead?

In my opinion, this is heavily feature creep, this should be its standalone tool, or banished to ./cmd/.

It could maybe be something along the lines of (calling it) semver-hs-db-upgrade, which is then fed a list of "significant" homeserver versions (for synapse: the versions where database upgrades happen), and then produce and execute every variation of A->B upgrades between them.