iresharma/reach-file-server

A file server written in python using flask interacting with cloudflare R2 and a messaging relay to Kafka for data duplication over at glacier

Reach File Server

A simple file server for handling files in Reach, also a relay for kafka to support data duplication into glacier.

reach-file-server's Issues

Figure out data duplication

This is an early item to plan about handling data duplication, current plan is

• use clodflare r2 as as our primary blob storage
• setup a message queue (kafka/rabbitmq)
• use the file handling server to create events on to the kafka queue
• create a kafka downstream that pulls these blobs from r2 and dupes them into aws glacier

This gives us an advantage on data deletion as well because we can configure file handling server to delete blob from r2 immediately but mark items as deleted in db and them run a cron job/queue event that enforces blob to to be deleted from glacier after set amount of days. This can be another paid feature(called recycle bin).

data structure for storage, versioning and review system

setup kafka

Setup a kafka queue on railway or AWS

setup R2