jewel/clearskies

Open source btsync clone

clearskies's Issues

ipv6 not working correctly

Looks like host B is trying to connect to an ipv6 port but host A only opened it for ipv4:

HOST A:

tracker> Tracking with http://clearskies.tuxng.com/clearskies/track?id=ab7f710316924fed38f493aa14413442bf2db28c8542054eb50988dc1f596fef%40b44ddbaf4d241e348568a515810654c0&id=10e53688a92370ced39427d8af929217b5445aa40c3cb69c890906231f8bbd33%40b44ddbaf4d241e348568a515810654c0&tcp_port=48555&utp_port=46327

root@cereza:0:/home/piotr/devel/clearskies# lsof -Pni | grep 48555
ruby 19347 piotr 9u IPv4 1376349 0t0 TCP *:48555 (LISTEN)

HOST B:

tracker> New outgoing connection with A
tracker> Tracker discovered 10e53688a92370ced39427d8af929217b5445aa40c3cb69c890906231f8bbd33 b44ddbaf4d241e348568a515810654c0 utp A 46327
tracker> New outgoing connection with A
connection410> Opening socket to utp A 46327
connection410> Could not connect via utp to 2a02:8109:8440:8b8:f2de:f1ff:fe00:441b:46327: Address family not supported by protocol - sendto(2)
connection409> Opening socket to tcp A 48555
connection409> Could not connect via tcp to A:48555: Connection refused - connect(2)

I can look more into it tomorrow.

GUI

Hi there,
I just found a quite nice Linux GUI for Bittorrent Sync released under GPLv2, I think you guys might use parts of the code:

https://github.com/tuxpoldo/btsync-deb

Just in case you didn't know about it already.

separate implementation and protocol in different repositories

Hi

as agreed we should separate the protocol spec and implementation in different repositories.

Local IP tracking, broadcast limitations

The broadcast mechanism is not sufficient for networks which have different disjoint broadcast domains. The peer should report to the tracker it's list of known local addresses, so other peers in the same private network can readily find them, even their broadcast domains are different.

rb-inotify is broken in older ubuntus

The system version of rb-inotify keeps the global-vm-lock in older versions of Ubuntu. This may be a problem with FFI instead of rb-inotify.

The symptom is that the test as ran from rake locks up completely.

minor passphrase clarifications

The Password section starts with "Instead of generating the master key". It might be clearer to say "Instead of generating a random read-write PSK". (I don't think the term "master key" is used anywhere else?)

Second, "The SHA256 algorithm is applied twenty million times to generate the 256-bit encryption key". Does this mean the read-write PSK? Is the PSK actually 256-bits in the passphrase case, but 128-bits for random keys?

Also, I was thinking it might be good practice for users to include a unique(ish) identifier as part of their passphrase, such as an email or domain name, in order to avoid accidentally colliding with someone else's passphrase.

This is looking pretty cool -- keep it up!
-greg

segfault on ARM in debian stable

On a sheevaplug running debian wheezy, the gnutls code segfaults. I have yet to determine if this is ARM specific.

Do not use SHA-256 hashing

Because of bitcoin miners, we now have dedicated devices that can hash SHA-256 in the order of Terahash/s. Since this rate is rising fast, SHA-256 will soon be obsolete.

I don't know enough to suggest any algorithm with confidance, but I'd also avoid s-crypt as dedicated devices for that algorithm are coming this year. S-crypt-n should be fine, however.

Die properly if two instances are ran

If you run two clearskies instances you get a crashed thread, but clearskies continues to run. It should exit.

control> Thread crash: Daemon already running
control> /home/jake/repos/clearskies/lib/control_server.rb:23:in run' control> /home/jake/repos/clearskies/lib/control_server.rb:13:inblock in start'
control> /home/jake/repos/clearskies/lib/simple_thread.rb:72:in block (2 levels) in initialize' control> /home/jake/repos/clearskies/lib/simple_thread.rb:38:inglock'
control> /home/jake/repos/clearskies/lib/simple_thread.rb:70:in `block in initialize'

Donations

Hi, I think you should display a donation option more prominently. I found a bitcoin address in the clearskies_core repo, but not many people go there, and there isn't even a bitcoin logo or a colorful donate button, or something to catch the eyes.

I think this project is pretty awesome, and I think many people might support the efforts if you made it a bit easier. Adding a paypal option might not hurt either.

Collaborate with Syncthing?

See discussion at https://github.com/calmh/syncthing/issues/105

Improvement: use rolling checksums for files

If big files change often recalculating the sha256 hash and resending the whole file will be very inefficient. A rolling checksum mechanism like the one rsync uses would be much more efficient in case a large file is being modified constantly, for example when having on sync a directory where torrents are being downloaded.

http://rsync.samba.org/tech_report/node2.html

restart command throws error

./clearskies restart
./clearskies:149:in `

': uninitialized constant Daemon (NameError)

[feature] support time difference accounting in peers to make it more robust

It's a bit of a hassle to assume that all peers need to have the clocks synchronized.

Silly Question

Is there any documentation on how to use the cli? I can't seem to get the daemon to start. Every time I try to run any of the commands I get 'Daemon not running'. I ran the tests and I get:

test_0004_should detect deleted files(Scanner::finds files)
test_0002_should detect new files(Scanner::finds files)

I am not sure what to do. I am not too familiar with ruby but I would still like to contribute.

Parsing JSON before authentication opens for certain vulnerabilities

Very large JSON objects in the handshake could consume non-trivial amounts of memory and CPU on the server side.

uTP doesn't support IPv6

The uTP code needs to try and open an IPv6 SharedUDPSocket, as well as as another STUNClient and UTPSocket.

Also, the tracker protocol needs to support a separate port number for IPv6. It almost certainly will be a different port number than the IPv4 port, especially since the IPv4 port is assigned by the NAT table on the router.

Detach command throws error

./clearskies detach ../aaaaa
clearskies/lib/control_client.rb:14:in []=': can't convert Symbol into Integer (TypeError) from clearskies/lib/control_client.rb:14:inissue'
from ./clearskies:246:in `

'

Debian/Ubuntu packaging

I'd like to take a stab at packaging this for Debian/Ubuntu if nobody is working on that.
Sounds good?

No fork in JRuby

The following test cases fail because there is no fork in Jruby.

1. Error:
   test_0001_can_connect_and_send_data(GnuTLS::Session::acts as a client):
   NotImplementedError: fork is not available on this platform
   org/jruby/RubyKernel.java:1855:in fork' clearskies/test/gnutls.rb:16:inrun_test'
   org/jruby/RubyBasicObject.java:1735:in `instance_eval'

2. Error:
   test_0002_won_t_connect_with_wrong_password(GnuTLS::Session::acts as a client):
   NotImplementedError: fork is not available on this platform
   org/jruby/RubyKernel.java:1855:in fork' clearskies/test/gnutls.rb:16:inrun_test'
   org/jruby/RubyBasicObject.java:1735:in `instance_eval'

3. Error:
   test_0001_can_connect_and_send_data(GnuTLS::Session::acts as a server):
   NotImplementedError: fork is not available on this platform
   org/jruby/RubyKernel.java:1855:in fork' clearskies/test/gnutls.rb:16:inrun_test'
   org/jruby/RubyBasicObject.java:1735:in `instance_eval'

4. Error:
   test_0002_won_t_connect_with_wrong_password(GnuTLS::Session::acts as a server):
   NotImplementedError: fork is not available on this platform
   org/jruby/RubyKernel.java:1855:in fork' clearskies/test/gnutls.rb:16:inrun_test'
   org/jruby/RubyBasicObject.java:1735:in `instance_eval'

Status of project

Is the protocol being worked on any more? I note that clearskies_core is now abandoned.

Issue with file deletion policy

https://github.com/jewel/clearskies/blob/master/protocol/core.md#deleted-files explains the file deletion policy which is used to avoid ghost copies.
However, that policy has a flaw where a file which has been deleted cannot be re-added to that share.

Testing with the latest client reveals this problem (machines A and B):

1. Add file test.txt to A
2. See file being correctly copied to B
3. Delete file on A
4. File is deleted on B
5. Re-add the file on A
6. File is not copied to B

Furthermore:
7. Copy the file manually to B
8. File is automatically deleted from A and B

In my view, this problem is as severe, if not worse than ghost copies, as it results in silent loss of data.

My proposal here would be to store the date of the file which was deleted along with the file name, and use it to decide whether any given file is a ghost or a new version.

Request for clarification

Hello.

While reading the protocol draft, I didn't catch the following idea. When a user want to give full access to a share to a peer, does he have to generate a temporary access code or it's enough to share only a read-write secret (like in BT Sync)?

Thanks in advance.

Unclear what to do when access code or club is not found

In core.md:

The server then uses the username to see if it has a corresponding access code or club. If it does, it completes the connection. The password or key is given to the TLS library as lowercase hexadecimal.

It isn't clear what is supposed to happen if it does not have a corresponding access code or club.

Wire protocol chunk size

Size of chunks in the wire protocol be limited to a maximum to avoid DOS attacks on clients that reserve a buffer with the size of the chunk.

"incompatible character encodings" error with Ruby 2.0

$ ./clearskies debug
....
control_connection> Control error: incompatible character encodings: UTF-8 and ASCII-8BIT
control_connection> /home/badboy/code/clearskies/lib/access_code.rb:59:in `to_s'
/home/badboy/code/clearskies/lib/control_server.rb:115:in `handle_command'
/home/badboy/code/clearskies/lib/control_server.rb:59:in `block in serve'
/home/badboy/code/clearskies/lib/control_server.rb:53:in `loop'
/home/badboy/code/clearskies/lib/control_server.rb:53:in `serve'
/home/badboy/code/clearskies/lib/control_server.rb:40:in `block (2 levels) in run'
/home/badboy/code/clearskies/lib/simple_thread.rb:72:in `block (2 levels) in initialize'
/home/badboy/code/clearskies/lib/simple_thread.rb:38:in `glock'
/home/badboy/code/clearskies/lib/simple_thread.rb:70:in `block in initialize'

$ ./clearskies share $(pwd)/shared read-write
/home/badboy/code/clearskies/lib/control_client.rb:24:in `issue': server can't :create_access_code, says: Encoding::CompatibilityError incompatible character encodings: UTF-8 and ASCII-8BIT (RuntimeError)
    from ./clearskies:143:in `<main>'

In Ruby 2.0 the default encoding is UTF-8 for all files (if not specified differently). This way access_code fails expecting ASCII-8BIT where it is UTF-8.
You should specifiy that you're handling binary data.

Tracker protocol unclear text

"The ID and peer ID are combined into a single string separated by an "@" character, and sent as the "id" parameter."

It's not clear which ID is referring to. The ids should be clarified with an abbreviation consistent in the whole text.

Need to list required ports for firewalls

Could you add the required ports for opening up a firewall somewhere?

Daemon not starting after recent commit

Commit 9c2f4 (Bind TCP to IPv6) broke one of my Clearskies setups: Raspberry Pi on IPv4-only network.
My other setup on a desktop in a dual IPv4+IPv6 network works fine, so I can't say if the problem is the architecture of the network.

The previous commit runs fine.
Here's the error log.

12:35:21.287118246 error main> Exiting due to exception in 'main' thread: Address family not supported by protocol - socket(2)
12:35:21.287557228 error main> /home/cachapa/clearskies/lib/network.rb:20:in initialize' 12:35:21.287898214 error main> /home/cachapa/clearskies/lib/network.rb:20:innew'
12:35:21.288262200 error main> /home/cachapa/clearskies/lib/network.rb:20:in start' 12:35:21.288597186 error main> /home/cachapa/clearskies/lib/daemon.rb:62:inrun'
12:35:21.288930173 error main> /home/cachapa/clearskies/lib/daemon.rb:41:in daemonize' 12:35:21.289513149 error main> ./clearskies:147:in

'

Small issue with installation instructions

I had a bit of trouble installing Clear Skies on my Debian Unstable and Raspbian distos.

The Ubuntu packages suggested don't seem to exist in Debian, so I tried to use the gem installation method, but that method requires the "ruby-dev" package.
Googling the problem shows that this is also true for Ubuntu, so I propose adding that to the instructions to simplify the installation process for others.

My abridged instructions:
[sudo] apt-get install ruby ruby-dev
[sudo] gem install rb-inotify ffi

Cannot load GNUTLS (Windows)

I was wondering how to get this section loaded on windows, I have tried installing gnutls (libgnutls26) with cygwin, have I missed anything?

https://github.com/jewel/clearskies/blob/master/lib/gnutls.rb