Native OSX support

[philcryer]

Native OSX support

[d-a-v]

And for what's it's worth, here is a version of lsyncd with the old-kernel-api (currently used by the official lsyncd) but with a patch to allow running lsyncd without beeing root on OSX :

git clone [email protected]:d-a-v/lsyncd.git
cd lsyncd/
git tag -l
git checkout tags/release-2.1.5-last-fseventsapi-and-makefiles
sudo port install autogen autoconf automake lua asciidoc
./autogen.old-OSX-MacPort.sh
cd OSX
make

and you need to put the suid bit on the exec (chown root lsyncd ; chmod +s lsyncd). The patch included will revoke the root rights once /dev/fsevents is successfully opened.

[Complexity]

Mac OS X Launchd can invoke events when directories are changed. This would replace lsyncd and invoke rsync via a script.

[philcryer]

This sounds great, while I like lsyncd, it sounds like it's not 100% on OSX, and that 10.5 and 10.6 may have different requirements, which I'd like to avoid. To be honest I haven't done much with OSX, development wise (aside from ssh'ing in it to my Linux boxes :)) but I installed growl-notifiy the other day so I could understand how lipsync could call out via Growl when things happen instead of notify or zenity in Linux. How can launchd be scripted? I'll try and take a look at this soon, all we need is for it to watch a directory for changes, and when they occur kick off a rsync command, hey, it's not rocket surgery you know. Thanks the tip, let me know if you've done this, or have any other guides out there to consult.

[Complexity]

I've set launchd up for a few things and it's extremely simple. I'm playing around with this at the moment because I have an OS X box and an Ubuntu box which I'd like to place in sync using lipsync, so I wanted to get lipsync working on 10.6. I'll send you the results.

[Complexity]

NM I was being stupid. :)

[philcryer]

well, it's been broken before, and I think it might be again - I just reinstalled from the latest. Now we have lipsyncd watching the sync directory, but there's nothing running on the server that could tell us about new files, so what we do is cheat (a little), the install script writes out a file in /etc/cron.d to kicked off a manual run of lipsync that checks the server, and grabs whatever new things it has that we don't, automatically ever 5 minutes. the file in cron.d looks like this

#!/bin/sh -e
# Distributed under the terms of the BSD License.
# Copyright (c) 2011 Phil Cryer [email protected]
# Source https://github.com/philcryer/lipsync

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
CONF_FILE=/etc/lipsyncd

if [[ -e $CONF_FILE ]]; then
        . $CONF_FILE
fi

# run lipsync against the server every 15 minutes to catch changes from other nodes
15,30,45,00 * * * *     $USER_NAME  /usr/local/bin/lipsync

but that doesn't seem to work. When I try it as

15,30,45,00 * * * *    root /usr/local/bin/lipsync

it doesn't work either, prompting the lipsync user for a password. However, if I manually run this as root:

su phil -c "/usr/local/bin/lipsync"

It works as designed; changes from the server are sync'd to the client (adds or deletes). So I'm not sure what is broken here, from the first run, just kicking /usr/local/bin/lipsync prompts you for the users' password, even though the username should have been sourced correctly by that file...I don't see why I have to call it out first to run /usr/local/bin/lipsync, and then again within that script itself, but that's what it feels like. I'll take a look at this later tonight, thanks for pointing it out!

[Complexity]

Okydoke I have a Frankenstein bidirectional version of lipsync running via launchd on OS X. I'll prettify the code a bit and send it over so you can have a look. There's an interesting issue with password protected ssh keys but I believe there are fiddly ways around that.

[Complexity]

Hey Phil, just messing about with this and it seems to work. https://github.com/Complexity/lipsmack

[philcryer]

I need to get this tested, and have the installer seamlessly go this way if it's run on OSX -- is anyone running it currently on OSX?

[cooperaj]

Tried to adapt the techniques here but it's limited by the launchd technology. WatchFolders unfortunately does not recurse through any hierarchy. So this script will only trigger if a file changes in the top level folder (or every 300 seconds). The only way to get more depth on this is to use the actual FSEvents API which obviously entails a bit more effort.

[anemes]

So - is launchd the way forward? I recently got a mac, and just found an older thread on lsync macosx here
http://groups.google.com/group/lsyncd/browse_thread/thread/7795dcb982956e24

Anybody had luck with that?

[Complexity]

Cooperaj is right, Launchd doesn't trigger anything lower than the first directory level.

After a bit of research there is a userspace API for fsevents which is exposed to things like Python. (E.g. http://pypi.python.org/pypi/MacFSEvents/0.2.1 ) That could be a way forward. A small, regularly polled thread which only triggers the rest of the Lipsync process on true hierarchical directory modifications.

[philcryer]

This is high on the list of things for me to work on, I have vacation
until Tuesday, but plan on looking at this later in the week. Having
the same install script work with OSX would be awesome.

P

On Tue, Jul 5, 2011 at 6:35 AM, Complexity
[email protected]
wrote:

Cooperaj is right, Launchd doesn't trigger anything lower than the first directory level.

After a bit of research there is a userspace API for fsevents which is exposed to things like Python. (E.g. http://pypi.python.org/pypi/MacFSEvents/0.2.1 ) That could be a way forward. A small, regularly polled thread which only triggers the rest of the Lipsync process on true hierarchical directory modifications.

Reply to this email directly or view it on GitHub:
#7 (comment)

http://philcryer.com

[audionerd]

Has Guard been considered? It's a Ruby lib which abstracts FSEvent/inotify (even Windows Directory Change Notification) and works great on OS X.
https://github.com/guard/guard

[philcryer]

This sounds promising - I'll take a look at Guard on my iMac and see
how it works.

P

On Thu, Sep 8, 2011 at 1:20 AM, Eric Skogen
[email protected]
wrote:

Has Guard been considered? It's a Ruby lib which abstracts FSEvent/inotify (even Windows Directory Change Notification) and works great on OS X.
 https://github.com/guard/guard

Reply to this email directly or view it on GitHub:
#7 (comment)

http://philcryer.com

[zachfi]

Any update on this? I would love to see OSX supported.

[audionerd]

I haven't had time to try Guard for this unfortunately, but it can be set to run shell commands whenever a file changes:

https://github.com/hawx/guard-shell

[philcryer]

UPDATE: haha, if anyone is interested this is another place that LinkedIn spammed (not) on my behalf! The whole sorted tale is here: LinkedIn is spamming all of my Gmail contacts! http://fak3r.com/2011/10/12/linkedin-is-spamming-all-of-my-gmail-contacts/

I'd like to add you to my professional network on LinkedIn.

• Phil

Phil Cryer
Technical infrastructure engineer at Center for Library & Informatics at Woods Hole Marine Biological Laboratory
Greater St. Louis Area

Confirm that you know Phil Cryer:
https://www.linkedin.com/e/sx37d5-gtkqzvcr-5o/isd/4504899444/_GuCvnB0/?hs=false&tok=3qJk4W4Srlf4Y1

You are receiving Invitation to Connect emails. Click to unsubscribe:
http://www.linkedin.com/e/sx37d5-gtkqzvcr-5o/ql1xzIiEd2QiqszGpcWdCa_CRk_LQPXUKEewqGiJfI_nMPu0KJ3R8ajpxoDEi4q0QgyrFAjJOIfMveXPQRyrMUehfZlEFlweNcIVaD/goo/reply%2Bi-799934-9861fb798c8fcec9e2677e092c69bcb135790b48%40reply%2Egithub%2Ecom/20061/I1560656522_1/?hs=false&tok=2aG-G1UKLlf4Y1

(c) 2011 LinkedIn Corporation. 2029 Stierlin Ct, Mountain View, CA 94043, USA.

[philcryer]

it looks like lsyncd DOES work on OSX, it uses fsevent instead of inotify, as noted by the main DEV of lsyncd:

http://stackoverflow.com/questions/9713752/why-does-lsyncd-require-xnu

./configure --without-inotify --with-fsevents CFLAGS="-I /Users/calvin/xnu-1699.24.23/" LUA_CFLAGS="-I /opt/local/include" LUA_LIBS="/opt/local/lib/liblua.a"
make

would like to see this working and validate OSX client/Linux both working...

[massayoshi]

Will give this a try as soon as my mac comes back from the repair shop. I currently have a linode account running debian and what I wanna do is sync the desktops of two different macs and keep a version online. Do you think it's doable with lipsync?

[mpj]

Ok, I've tried --with-fsevent on my MBA (Mountain Lion). The problem is that you need root access to get at /dev/fsevents:

Error: Cannot access /dev/fsevents monitor! (1:Operation not permitted)

You can sudo, but it becomes horribly inconvinient, because you don't have access to your SSH keys when sudoing.

[MPV]

Did anyone get this working properly? @mpj?
I eventually copied across my SSH keys to my root account, and then it worked for me.

[d-a-v]

a fork in lsyncd using userland fsevents api (instead of kernel fsevents) is awaiting for testing, that would solve OSX problems.
some improvements are on their way but I need someone with lua skills, or just time to dive into lua, to finish it up. The author has neither no time.
https://github.com/d-a-v/lsyncd

[philcryer]

Sounds like a good idea, I haven't revisited this in some time - my idea would be to redo it in python, then attack the OSX issue, since that's always been a sticking point. Here's an idea using some ruby https://www.quora.com/How-can-I-keep-a-local-directory-on-my-Mac-in-sync-automatically-with-a-remote-directory-on-a-server-that-I-have-access-to-via-SSH

[philcryer]

More scripting around lsyncd... http://backdrift.org/recursive-inotify-scripting-with-lsyncd

[d-a-v]

Sorry I don't get you :-] and I may have been unclear.
lua is not something to try out, it is already integrated and used into current lsyncd. The osx-userland-fsevents port in my branch (which replaces the unofficial api osx-kernel-fsevents which current lsyncd uses today and which is the source of all lsyncd problems on OSX) is already implemented but at the time I wrote the port, I was stuck at the internal-lsyncd-lua-bindings. Even the author had no time to help me with that.
I'll try to find time to finish it up, but once resolved, all recurrent lsyncd on OSX problems would disappear.