sudssm / daruma Goto Github PK

View Code? Open in Web Editor NEW

13.0 5.0 1.0 6.19 MB

Safety in Cloud Storage

Home Page: http://sudssm.github.io/daruma

License: GNU General Public License v3.0

Python 91.27% CSS 2.44% HTML 2.55% JavaScript 1.47% Objective-C 2.27%

daruma's Introduction

Daruma

Overview

Cloud storage is ubiquitous in our everyday lives. Businesses and consumers use services like Dropbox, Google Drive, and Box to store a wide variety of important documents, ranging from family photos to healthcare information and proprietary corporate data. However, users rarely consider the possibility of these services experiencing failures. What happens if their provider goes down or gets hacked? For many, the consequences would be catastrophic.

All cloud providers make assurances of data safety, security, and availability. However, these cannot be guaranteed because providers are a single point of failure, vulnerable to everything from hackers and software flaws to state-level intervention.

Daruma solves these problems by eliminating the need to trust any cloud provider. It combines and secures the storage of existing providers by running advanced cryptographic and redundancy algorithms on users’ computers. With no single point of failure, we make a simple but previously unattainable guarantee: no provider can read, corrupt, or delete your files -- ever.

Further, Daruma achieves this in an intuitive, familiar interface. Once a user logs in with their existing provider accounts, Daruma handles everything automatically - no new passwords or different workflows. When a provider fails, Daruma identifies the issue and fully recovers, without interrupting the user’s workflow.

Best of all, Daruma is highly space efficient and is faster and more scalable than using cloud providers in isolation. Daruma handles the complexities of security and reliability for users, allowing them to confidently utilize cloud storage without worrying about its previously inherent risks.

Installation

Currently, the Daruma executable (with all GUI features) only supports OSX El Capitan. Users of other Linux-based operating systems can use the command line version, or attempt to build their own executable. Daruma currently doesn't support Windows because one of our dependencies (liberasurecode) doesn't compile on Windows.

Command Line (OSX & Linux)

Install dependencies - Python 2.7, Pip, libnacl, libffi, haveged, liberasurecode. On Ubuntu, these needed libraries can be installed with

echo "deb http://archive.ubuntu.com/ubuntu trusty-backports main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list
sudo apt-get update
sudo apt-get -y install libnacl-dev libffi-dev haveged liberasurecode-dev

While in the project directory:

pip install --user . --process-dependency-links robustsecretsharing

Building the GUI Application (Tested on OSX El Capitan)

While in the project directory:

pip install --user -e \.\[gui] --process-dependency-links robustsecretsharing
python build.py

The newly built app will be in the dist directory.

Running

GUI

Simply execute the packaged Daruma executable. To see status badges in the OSX Finder, you may need to explicitly enable the app as a Finder Extension, which can be done in the Extensions pane of System Preferences.

Command Line

To run the command line REPL:

python driver/daruma_cli.py

Type 'help' to get started. 'add' will allow you to add new providers, 'provision' will create a new Daruma instance from the providers added, and 'load' will load an existing Daruma instance from the providers.

Testing

py.test will automatically run all available test cases.

Note that on OSX, a system message may pop up saying "Python quit unexpectedly". As long as the tests run to completion, this is expected and an indication of successful behavior.

License

This software is licensed under the GPL license. If you would like an alternative license, please contact us.

daruma's People

Contributors

Stargazers

Watchers

Forkers

cloudxtreme

daruma's Issues

Verification of Parameters

Right now bad parameters trigger errors all the way down in our supporting libraries (once we go through our secret sharing and erasure encoding code). Bounds checks should probably happen at initialization. I've added some code in SecretBox to do this for now - as a more general reminder we should more rigorously verify / sanitize / etc all possible user input later on.

Ensure that all providers are threadsafe

cli won't work otherwise

find cheaters

using the secret and the list of shares associated with it, swap in remaining shares and for each one call standard sharing (non-robust). for each share that reconstructs to the authenticated secret, consider it valid and add all others to the invalid list

gui errors

IOError when loading app creds fails

Providers

Make sure nothing bad happens when user cancels a provider auth flow.
Make sure all providers have Daruma branding

All provider operations should be done in parallel

Base provider

Note to self:
provider inits don't call connect by default; check to make sure that this doesn't cause problems anywhere

Note to @DoronShapiro:
base provider doesn't really do much - lets have a conversation offline about its usefulness

Think about removing providers

So say we have provider A in our system, and we want to remove it for some reason.

Currently, it stays in our user credentials file. So, the next time we start the program, we'll load up A, and then 'repair' it.

The solution is, when the file manager boots up - when we load the manifest - check to see if the filemanager's provider list contains elements that the manifest's provider list doesn't contain. If it does, then we should put these extra providers to the side, and ask the user explicitly if they should be added.

(Keeping the extra provider is actually more than just a ui issue; it causes the bootstrap n (and therefore the filemanager and distributor n) to be smaller than the number of providers we are working with),

clean up testing folders

clean up the foo/daruma and other auto-generated folders

make accounts for all providers

make a non-school account for box in particular and ensure that you can authenticate with all providers for demo

Daruma branding

App doesn't always quit

Clicking the quit item in the menubar app will sometimes remove the menubar item but keep python running.

Things needed for filesystem support

(Will update as new things come up)

File renames (i.e. the mv command) should just trigger a change in the manifest
We should support empty directories in the manifest

filesystem - ls and download all files upon init

GoogleDrive: put files into Daruma folder

currently, all files go into root and can be seen by the user
update put, search, etc to be specific to a Daruma folder

Storing providers in manifest

Upon connecting to an existing installation for the first time, we won't know if a provider is offline (or not connected to). We store k in the bootstrap, but not n.

I propose that we should store the list of all providers being used in the manifest. Are there any issues with this?

provider wipe parallelization

secretsharing module blocks in low-entropy settings

On freshly created Ubuntu VMs, the secret sharing tests block (reading /dev/random) until you type a bunch or install a tool like haveged.

For now, the above seem to be good workarounds, but posting as a thing to keep an eye out for later.

Static vs classmethods

change staticmethods to classmethods where appropriate

passing shares to reconstruction

always pass all available shares to reconstruction
if you only pass reconstruction_threshold shares and some of them are invalidated by the authentication scheme then we will fail to reconstruct the secret even if other honest shares were available.
pretty sure we decided on this anyway, but wanted a reminder here for when we do integration

Providers page sometimes renders funny

Working user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/601.4.4 (KHTML, like Gecko)
Strange layouts for user-agent Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 (KHTML, like Gecko)

Think about adding missing provider when we have extra provider

@DoronShapiro

consider making a system with a,b,c
then loading the system with a,b,d.
We load into readonly mode, and ask for c.

add c
Now we have a complete set! but d is actually blank.
On future operations, d will fail and go to yellow (but be repaired, if he is behaving).

We should probably detect if the filemanager's provider set is a superset of that stored in the manifest. And then... automatically reprovision? But the user may want to add more providers. Stay in ReadOnlyMode until the user reprovisions? But that overloads the meaning of ReadOnlyMode in an ugly way (since we can actually write, but we perhaps don't want to).

Non-reproducable test failures in bootstrapmanager

I got a few test failures for bootstrapmanager. When I re-ran the tests, they went away.

> py.test                                                                                                                                               16:19
==================================================================== test session starts =====================================================================
platform darwin -- Python 2.7.11, pytest-2.8.7, py-1.4.31, pluggy-0.3.1
collected 102 items

driver/tests/test_secretbox.py ..........
managers/tests/test_bootstrap_manager.py .F..FF..FF.F
managers/tests/test_distributor.py .............
managers/tests/test_file_manager.py ..........
managers/tests/test_manifest.py .................................
providers/tests/test_LocalFilesystemProvider.py ......
tools/tests/test_encryption.py ....
tools/tests/test_erasure_encoding.py .....
tools/tests/test_secret_sharing.py .....
tools/tests/test_utils.py ....

========================================================================== FAILURES ==========================================================================
_______________________________________________________________________ test_roundtrip _______________________________________________________________________

    def test_roundtrip():
        BM = BootstrapManager(providers, 2)
        BM.distribute_bootstrap(bootstrap)
>       assert BM.recover_bootstrap() == bootstrap

managers/tests/test_bootstrap_manager.py:23:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
managers/BootstrapManager.py:97: in recover_bootstrap
    bootstrap = Bootstrap.parse(string)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

string = '\x05O\x82j>\x97t[\xb3)\xaapO\x9f\xfb\xb4>\xc5\xd9\x83\xf9\xecyM\x9a&\xeeN\x14\xd1\xbd04D23BB49E1A4AF7A94B33D414648C91\x00\x00\x00\x02'

    @staticmethod
    def parse(string):
>       key, manifest_name, file_reconstruction_threshold = struct.unpack(Bootstrap.STRUCT_FORMAT, string)
E       error: unpack requires a string argument of length 68

managers/BootstrapManager.py:35: error

And then similar failures in test_multiple_sessions, test_erase_recover, test_corrupt_k_recover, test_corrupt_k_2_recover, and test_corrupt_k_but_not_fail.

google drive wipe should be fixed

to wipe all files, not just first page

google drive branding

handle invalid provider ids

use the value of n to see if a returned provider id is out of range and if so consider that provider malicious
if we see duplicate ids, take all combinations and run robust sharing - then accept the secret that has the greatest number of valid shares associated with it

Tools tests corrupt shares with a predetermined value

This causes random failures:

=================================== FAILURES ===================================
__________________________ test_malicious_ciphertext ___________________________
def test_malicious_ciphertext():
# First encrypt
plaintext = "FOO BAR woohoo!"
key = tools.encryption.generate_key()
ciphertext = tools.encryption.encrypt(plaintext, key)

    # Then corrupt ciphertext
    malicious_ciphertext = ciphertext[0:5] + 'C' + ciphertext[6:]

    # Then attempt to decrypt
    with pytest.raises(exceptions.DecryptError):

      tools.encryption.decrypt(malicious_ciphertext, key)
E Failed: DID NOT RAISE