High-availability Production Configuration

[internetarchive]

High-availability Production Configuration

[bnewbold]

Partial progress has been made:

• haproxy configured in a single-node configuration
• using uptimerobot for status page (https://status.fatcat.wiki) and it is working better than previous attempts (labstatus, cabot, etc)

Still to be done:

• keepalived for multi-node haproxy
• more monitoring and alerting
• move fatcat ES usage to multi-node cluster (shared with scholar index)
• postgres replication (and update to postgres 13)

[bnewbold]

Status update:

• search is replicated, and upgraded to elasticsearch 7.10
• virtual IP managed by keepalived is configured with a second haproxy node. have not switched over DNS to the virtual IP yet
• have not updated to postgresql 13 or configured postgresql replication yet, but hardware is ready for it
• need to implement "read-only mode" in web interface and API (or just figure out the correct configuration for this behavior)

[bnewbold]

More updates:

• upgraded to postgresql 13
• upgraded to ubuntu focal
• "read-only mode" banner and database configuration documented

The last remaining step is adding hot-standby (but non-synchronous) binary-mode (WAL) postgresql replication, with the replica being read-only by default (manual failover).