objectfs / objectfs Goto Github PK

View Code? Open in Web Editor NEW

28.0 5.0 3.0 473 KB

A file system with the power of an object store.

License: Apache License 2.0

Python 95.51% Shell 4.49%

aws-s3 openstack-swift fuse-filesystem google-cloud-storage fuse s3 posix-compliant

objectfs's Introduction

ObjectFS

A file system with the power of an object store.

ObjectFS allows you to mount a generic object store as a file system. It is POSIX complete and compatible with AWS S3, Google Cloud Storage, OpenStack Swift and other S3-based object stores.

Usage

Make file-system

./objectfs_cli make <filesystem-name>

Mount file-system

./objectfs_cli mount <filesystem-name> <mount-point>

List all file-systems

./objectfs_cli list

Delete file-system

./objectfs_cli delete <filesystem-name>

Architecture

ObjectFS is a file system which uses object storage as a backend. It's goal is to provide:

Access data via both file system and object interfaces
Support multiple object stores
Portable across multiple operating systems

The high-level architecture of ObjectFS:

A stack overview of ObjectFS:

A file in ObjectFS:

Contributing

Please submit bug reports as git issues and label them as a bug. If you have feature requests, questions please free to open git issues for the same and label them appropriately. You can also contribute code or bug fixes by opening a pull request.

objectfs's People

Contributors

Stargazers

Watchers

Forkers

zhuguoliang maniaabdi effygal

objectfs's Issues

Refactor objectfs_operations.py

We have multiple existing configurations (no cache, cache, multi-part) and more coming. Creating a class per configuration will be easier to manage and debug. Some methods which have significant overlap can be inherited.

atime and mtime are not updated

This is an issue inherited from llfuse which in turn inherits this from FUSE.

Daemonize the FUSE process

Mount an existing object-store to file-system

Ability to mount an existing object-store with objects as a file-system
Write a organizer script which groups different objects based on Content-Type/file-extensions
Add the FS meta-data for these objects in key-value store

List existing file systems

Currently, we base this of bucket list which is incomplete. There are three possibilities:

Both metadata exists and bucket exists. File system present
Metadata does not exist but bucket exists. Uninitialized file system
Metadata exists but no bucket. Zombie file system.

Implement inode metadata field with hashmap in Redis

Some orphan inodes are not deleted from the object store

This issue was discovered by running configure on file bench and activating caching
Suspect there is an edge case which is not being handled for deleting objects

Redis connection_pool causes single client

The singleton class implementation over the redis connection pool is not working as expected. It causes only a single connection to be opened to the remote redis. The expected behavior is to pen upto max_connections set by the user. By default this is 10.

The go-around for now is to use individual redis client connections which work as expected but don't have the advantage of limiting the number of connections.

Asychronous deletion of objects

For large file-systems (>90,000 inodes) the time for deletion is limited by object deletion time.
Solution:

delete the file-metadata from the metadata server instantly
delete the objects later

Based on user policy
Use the object name stored in inode metadata

Run benchmarks on Redis for checking performance of sync to disk

Run benchmarks on Redis for checking performance of sync to disk #1

What are special characters in different systems?

Identify the special characters in each system.

The file-system namespace will not include the union of these characters
The delimited namespace will be an intersection of these characters

This issue was discovered when trying to boot a node using the the file system with cache activated
Suspected that some operation is unhanded in which object should be loaded but only the key is created