bazil/bazil

Bazil is a file system that lets your data reside where it is most convenient for it to reside.

Bazil -- a distributed, weakly-connected, file system

Bazil is a file system that lets your data reside where it is most convenient for it to reside.

The top-level package, bazil.org/bazil, is the executable for the Bazil file system.

Please see the main website at http://bazil.org/ for more.

Announcement: http://bazil.org/2014/04/24/introducing-bazil.html

Documentation: http://bazil.org/doc/

bazil's Issues

Compress chunks, maybe

• compress with snappy or lzo before hashing
• still do venti-style zero trim? it has nice properties.
• do I need to worry about compressed size revealing things and piercing crypto

Keep track of last writer, time of write

Each direntry would locally store

• peerID of peer who last made a write
• wallclock time when this happened

This would be updated last-writer-wins style when resolving syncs (action==Copy).

Protect against filenames with "/" "\x00" and paths with "\x00" at every boundary

• implement flagx.Filename, flagx.Pathname
• protect peer rpc
• protect control rpc

installation error

I'm trying to install Bazil, and I got the following error while go getting Bazil.

go version

go version go1.5 linux/amd64

go get bazil.org/bazil

# bazil.org/bazil/util/grpcedtls
../bazil/util/grpcedtls/auth.go:24: cannot use (*Authenticator)(nil) (type *Authenticator) as type credentials.TransportAuthenticator in assignment:
    *Authenticator does not implement credentials.TransportAuthenticator (wrong type for ClientHandshake method)
        have ClientHandshake(string, net.Conn, time.Duration) (net.Conn, error)
        want ClientHandshake(string, net.Conn, time.Duration) (net.Conn, credentials.AuthInfo, error)

What to do with pending entries (conflicts etc) in subtree delete

They become inaccessible.

If we did materialise them in the dir, rm -rf would resolve them naturally. To support rename(2) for resolving, we'd need to pick real (non-dynamic) inodes for them.

What to do about pending non-conflicts, waiting for a file to close or data to be pinned? Too noisy to materialize. Maybe after a delay? But then we need to solve this, anyway..

Include conflicts(+postponed syncs) when sending a sync

Define the VolumeSyncPullItem lexicographic list of names to allow duplicates.

Node.js client?

I have been interested on filesystems development from some years ago (and in fact I won a prize with my own one, PirannaFS :-) ) and always I was a bit concern about the problems that bazil try to fix, with solutions like mhddfs and so. Now for NodeOS I was willing to have each user its own $HOME on a remote server using WebDAV so their data is globally available and their local hard disk working as a cache... just as bazil is doing but with exteroids, so having something better available, why to write it from scratch? :-D So my question is, do you know if there's somewhere a Node.js client, or where should I start looking for to create my own one? :-)

Delayed Release op from kernel can make tests flaky by forcing sync to go to pending

The tests do things like read (expect "one"), sync, read again (expect "two"). If the Release from the first open+read+close is delayed, the sync can run at a time where there's still an open Handle, and get pushed to pending.

We need a way to wait for a Handle to go away, or to wait for all pending clocks to be resolved, or something.

Reasonable Sync semantics for platforms without InvalidateEntry (OSXFUSE 2.x)

OSXFUSE 2.x does not support invalidation.

Thus, on OS X, incoming sync deleting files cannot invalidate kernel dentry cache, thus cat foo keeps working even after foo has been removed.

Maybe add a secondary mode where if !Protocol().HasInvalidate(), postpone syncs if node is in dir.active? Move reconciliation from Release to Forget?

Currently, the server will log FUSE invalidate error with EINVAL.

Perhaps a miracle will happen and OSXFUSE 3.x will see a stable release before this becomes relevant.

Project status

Hello,
What is the future of this project? Does anyone use it in his notebook?

There is https://syncthing.net/ , any other alternatives?

Installation error: undefined: grpc.WithInsecure

Hello, trying to install Bazil via instructions on bazil.org, but ran into this issue:

~/D/o/gopath$ go get google.golang.org/grpc
~/D/o/gopath$ cd $GOPATH/src/google.golang.org/grpc
~/D/o/g/s/g/grpc:master$ git remote add bazil https://github.com/bazil/grpc-go
~/D/o/g/s/g/grpc:master$ git fetch bazil
remote: Counting objects: 36, done.
remote: Total 36 (delta 28), reused 28 (delta 28), pack-reused 8
Unpacking objects: 100% (36/36), done.
From https://github.com/bazil/grpc-go
 * [new branch]      auth       -> bazil/auth
 * [new branch]      auth-1     -> bazil/auth-1
 * [new branch]      auth-2     -> bazil/auth-2
 * [new branch]      auth-3     -> bazil/auth-3
 * [new branch]      master     -> bazil/master
~/D/o/g/s/g/grpc:master$ git checkout -b auth bazil/auth
Branch auth set up to track remote branch auth from bazil by rebasing.
Switched to a new branch 'auth'
~/D/o/g/s/g/grpc:auth$ go get bazil.org/bazil
# bazil.org/bazil/util/grpcunix
../../bazil.org/bazil/util/grpcunix/grpcunix.go:18: undefined: grpc.WithInsecure
# bazil.org/bazil/util/grpcedtls
../../bazil.org/bazil/util/grpcedtls/auth.go:38: undefined: credentials.AuthInfo
../../bazil.org/bazil/util/grpcedtls/auth.go:42: undefined: credentials.AuthInfo
../../bazil.org/bazil/util/grpcedtls/auth.go:63: undefined: credentials.AuthInfo

On OSX, Go 1.6.2.

util/grpcunix: Way to stop Dial from retrying on permanent errors

Right now it just tries again, so I set the timeout aggressively low, but really, it's a UNIX domain socket: if it's not working, just return an error. Timeout() == false but how to communicate it out of / to the grpc framework straigtjacket?

How to identify volume & node through FUSE mount

We want to use current directory and paths passed to the CLI as input for things like bazil sync [PATH...], bazil pin PATH..., and so on. To do this, we need a reliable way to

1. detect that we're inside a bazil fs mount
2. communicate the right file to the server

All of this in presence of bind mounts, namespaces, and other programs mutating the tree.

Inodes are unique, if we know what volume we're looking at.

Cannot search files on Finder

I'm using "bazil.org/fuse", "bazil.org/fuse/fs" packages to create my own file system but mounted file systems are not searchable through Finder. Finder tries to find items but is says nothing but Loading and spinning loader icon. Do I miss something or it's some kind of bug?

Here is my fuse.MountOptions

• fuse.FSName("samplevolume")
• fuse.Subtype("sample")
• fuse.VolumeName("Sample Volume")
• fuse.AsyncRead()
• fuse.ReadOnly()

and I'm optionally adding fuse.LocalVolume() option depending on flag but nothing changed.

Thanks for packages BTW :)

How do you get play.golang.org examples to show in godoc?

Not an issue but I can't seem to find any information out there and it is working for you ! Mind sharing how?

ReadAll not working

I would expect this sample to work, but cat $MNT/file.txt returns nothing (empty file):
https://gist.github.com/bpowers/d90bbb48845fc3e825f1

I'm running linux 3.17-rc1, but have also tried with 3.16.1. Thoughts?

Subtree Rename results in incorrect clocks

The old side becomes a tombstone, the new side is harder. The bug is that we move foreign Create clocks to new files. This leads them to falsely dominating files actually created remotely.

(we use foo^ to mean elementwise max of the foo vectors in the clock, e.g. mod^)

• we need a create^ mod^ operation
• the operation needs to be atomic for a whole subtree and O(1) (to preserve unixy semantics of rename(2))
• done naively, that would conflict with later child creations with other peers getting synced in. clocks are a||b, no monotonic growth
• introduce a rename epoch
• can it use the same epoch counter as syncs? must dir.Rename call cleanEpoch? or use a separate move epoch counter? is it a level below epoch, <epoch,age>?
• it's purely local, so just single epoch, no need to store peerid or have vector

if child.renameEpoch < parent.renameEpoch {
    then merge parent clock to child
}

• the merge is a elementwise max merge not a replace, create^ mod^ (0 sync is default for new entries anyway, no need to merge)

mv foo bar
touch bar/a/b

• sync must always descend the tree, not start from random inode

Alternative design: maybe it's a separate inode:name bucket (just flag as dirty, unflag when children are flagged)?

Sketch for a test:

peer10: echo one >a
sync
peer11: echo two >b && mv b a
sync

Ponder:

A: mkdir foo
sync
A: mv foo bar
B: touch foo/a
sync

Ponder mv foo bar:

|       | create | mod  | sync | E5 | create                              | mod  | sync |
|-------+--------+------+------+----+-------------------------------------+------+------|
| foo/  | e1     | p:e3 | p:s1 |    | e1                                  | e5   | s1   |
| foo/a | e2     | p:e2 | p:s2 |    | e2                                  | e5   | s2   |
| foo/b | e3     | p:e3 | p:s3 |    | e3                                  | e5   | s3   |
|       |        |      |      |    |                                     |      |      |
| bar/  |        |      |      |    | e5                                  | e5   | 0    |
| bar/a |        |      |      |    | (e5)                                | (e5) | (0)  |
| bar/b |        |      |      |    | (e5)                                | (e5) | (0)  |
|       |        |      |      |    | ^ non-existing use parents info,    |      |      |
|       |        |      |      |    | when the rename epoch rules trigger |      |      |