divinerapier/weaver

A distributed object file system inspired by the paper Beaver which was published by Facebook Inc. at 2010.

Weaver

Weaver 是一个基于 Facebook Inc. 在 2010 发表的论文 开发的一个分布式对象文件系统。

组件

WeaverGateway

作为所有请求的入口。

WeaverDirectory

1. 维护逻辑卷到物理卷的映射关系；
2. 负责对读写数据进行负载均衡；
3. 维护逻辑卷的空闲空间；
4. 维护图片对应的逻辑卷；

WeaverCache

Cache 服务是一个分布式缓存，可以使用数据的 ID 对内容进行索引。当缓存命中失败时，负责从后端 Store 服务获取数据。

WeaverStore

每一个 Store 服务维护多个物理卷。每个物理卷存储了百万级别的数据。具体来说，一个物理卷可以被认为是一个超大的文件(比如100G)，被保存在物理磁盘上的某个位置。可以通过图片所在物理卷及偏移量，大小等信息快速读取数据。

Architecture

weaver's Issues

retry on failure. eg: add a error type: RetriableError(Box<Error>)

https://github.com/divinerapier/onyxiafs/blob/624fd26b6f41701172aac56ccd36842b32809a6f/libonyxia/src/directory.rs#L74-L79

---

This issue was generated by todo based on a TODO comment in 624fd26 when #36 was merged. cc @divinerapier.

set magic number and compute check sum

https://github.com/divinerapier/onyxiafs/blob/dadc104a009cb159cc0b9a7c60c5a3984e2be24b/libonyxia/src/needle/mod.rs#L105-L110

---

This issue was generated by todo based on a TODO comment in dadc104. It's been assigned to @divinerapier because they committed the code.

range hashset not in a random order

https://github.com/divinerapier/onyxiafs/blob/4c3585d1245c71910e7ae3e5c50d6292f7ecd3bb/libonyxia/src/directory.rs#L182-L185

---

This issue was generated by todo based on a TODO comment in 4c3585d when #42 was merged. cc @divinerapier.

supports write-ahead log

https://github.com/divinerapier/onyxiafs/blob/befc64ff3089538a25ed1c844d856175ddbdffdd/libonyxia/src/volume.rs#L253-L258

---

This issue was generated by todo based on a TODO comment in befc64f when #24 was merged. cc @divinerapier.

load indexes and open related physical volume

https://github.com/divinerapier/onyxiafs/blob/cce95421008f682e9483ef786cca129422d49be0/libonyxia/src/directory.rs#L30-L35

---

This issue was generated by todo based on a TODO comment in cce9542 when #12 was merged. cc @divinerapier.

check file if exists

https://github.com/divinerapier/onyxiafs/blob/b20f1e623ace8866dc11e6eaa34f035a4d4c16dd/libonyxia/src/volume.rs#L58-L63

---

This issue was generated by todo based on a TODO comment in b20f1e6 when #21 was merged. cc @divinerapier.

volume should be a trait

read needle footer

https://github.com/divinerapier/onyxiafs/blob/a6cb64977c321b86ae45bf20c6a3ff55ec2c93b7/libonyxia/src/store/volume.rs#L434-L437

---

This issue was generated by todo based on a TODO comment in a6cb649. It's been assigned to @divinerapier because they committed the code.

extract index content

https://github.com/divinerapier/onyxiafs/blob/0d63810604507b202d437a6c7f7d9355e79e5b00/libonyxia/src/volume.rs#L132-L137

---

This issue was generated by todo based on a TODO comment in 0d63810 when #15 was merged. cc @divinerapier.

use clap to handle commands

https://github.com/divinerapier/onyxiafs/blob/fbed1db62845ccbec41a27755ca49c693e445207/onyxia-store/src/main.rs#L15-L18

---

This issue was generated by todo based on a TODO comment in fbed1db when #55 was merged. cc @divinerapier.

use a min-heap to store volumes? tuple(id, remain_length)

https://github.com/divinerapier/onyxiafs/blob/624fd26b6f41701172aac56ccd36842b32809a6f/libonyxia/src/directory.rs#L16-L19

---

This issue was generated by todo based on a TODO comment in 624fd26 when #36 was merged. cc @divinerapier.

optimize copying index

https://github.com/divinerapier/onyxiafs/blob/0d63810604507b202d437a6c7f7d9355e79e5b00/libonyxia/src/directory.rs#L40-L45

---

This issue was generated by todo based on a TODO comment in 0d63810 when #15 was merged. cc @divinerapier.

using thread pool

https://github.com/divinerapier/onyxiafs/blob/1466d5260269d14cfb5aa415c0e754cf40dbbcee/libonyxia/src/volume.rs#L310-L315

---

This issue was generated by todo based on a TODO comment in 1466d52 when #30 was merged. cc @divinerapier.

error should be handled instead of calling unwrap

https://github.com/divinerapier/onyxiafs/blob/cce95421008f682e9483ef786cca129422d49be0/libonyxia/src/volume.rs#L56-L61

---

This issue was generated by todo based on a TODO comment in cce9542 when #12 was merged. cc @divinerapier.

test read multiparts

https://github.com/divinerapier/onyxiafs/blob/624fd26b6f41701172aac56ccd36842b32809a6f/libonyxia/src/directory.rs#L302-L306

---

This issue was generated by todo based on a TODO comment in 624fd26 when #36 was merged. cc @divinerapier.

check if length of volume data matches to index

https://github.com/divinerapier/onyxiafs/blob/befc64ff3089538a25ed1c844d856175ddbdffdd/libonyxia/src/volume.rs#L118-L123

---

This issue was generated by todo based on a TODO comment in befc64f when #24 was merged. cc @divinerapier.