Can you tell us about the architecture you used?

[nerdsinspace]

Can you tell us about the architecture you used?

[leijurv]

Also, why didn't you use something like Kubernetes and make it truly scalable? Anyways, cool project.

Laughed out loud, that's not how that works at all

How were you able to manage query performance and storage optimizations with the amount of data you were processing? I would assume it's a pretty big feat considering you have two terabytes worth of data in a single database. Did you host a managed database with DO, or did you provision a droplet with block storage and just manage your own installation?

Bare metal was the only way to go. Just a 2TB nvme, with hand-tuned ZFS on top of that, and hand-tuned Postgres on top of that, and hand-tuned queries and indexes on top of that. Queries to the biggest table, blocks (over ten billion rows), to fetch old data about coordinates, would peak at over 30,000 SELECTs per second (sustained average over minutes).

[mwalden2004]

Laughed out loud, that's not how that works at all

Yeah, probably should have phrased that better. I'm just curious as to why you wouldn't containerize it and just put it on something like Kubernetes to make it easy to port into multiple servers. My expertise is not in Minecraft, so I'm just curious about that.

Bare metal was the only way to go. Just a 2TB nvme, with hand-tuned ZFS on top of that, and hand-tuned Postgres on top of that, and hand-tuned queries and indexes on top of that. Queries to the biggest table, blocks (over ten billion rows), to fetch old data about coordinates, would peak at over 30,000 SELECTs per second (sustained average over minutes).

That's really cool, great job again.

[leijurv]

I'm just curious as to why you wouldn't containerize it

Depends on what you mean by it. From context, I thought you were referring to the Postgres but perhaps not? Postgres doesn't have good multi-master, and given the real-time write-heavy workload of the block downloader and tracker it wouldn't make much sense to have a follower or replica (no matter if it's a replica slot or a wal shipper type scenario). We DID ship wal logs though, that's how the backups were done.

But we absolutely did use containers, there are something like a dozen different things running in docker. We just used docker-compose since kubernetes doesn't have much to offer on top of it.