facebook/watchman

Watches files and records, or triggers actions, when they change.

watchman's Issues

Watchman crash on path = .

$ mkdir /tmp/fileURkhtr
$ echo "foo" > /tmp/fileURkhtr/morx.jar
$ watchman watch /tmp/fileURkhtr
$ socat - /tmp/.watchman.dturner <<EOF
["query", "/tmp/fileURkhtr", {"expression": ["since", "c:1390499661:13064:15:3"], "fields": ["name", "ctime_f"], "path": ["."]}]
EOF
(socat exits here, because watchman has crashed)

Here's a backtrace from watchman:
$ gdb /home/dturner/watchman/watchman
(gdb) r -f
Starting program: /home/dturner/watchman/watchman -f
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff7ffa000
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff75bf700 (LWP 16489)]
[New Thread 0x7ffff657b700 (LWP 16490)]
[New Thread 0x7ffff5c79700 (LWP 16496)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff5c79700 (LWP 16496)]
w_ht_lookup (ht=0x0, key=140737085710048, val=0x7ffff5c78a28, copy=false)
at ht.c:396
396 };
(gdb) bt
#0 w_ht_lookup (ht=0x0, key=140737085710048, val=0x7ffff5c78a28, copy=false)

at ht.c:396

#1 0x0000000000406ddb in w_ht_get (ht=, key=)

at ht.c:217

#2 0x000000000040a3d1 in path_generator (ctx=,

root=<optimized out>, query=<optimized out>) at query/eval.c:252

#3 default_generators (query=0x7fffe8000ed0, root=0x7ffff697e010,

ctx=0x7ffff5c78ab0, gendata=<optimized out>) at query/eval.c:291

#4 0x000000000040a538 in w_query_execute (query=0x7fffe8000ed0,

root=0x7ffff697e010, res=0x7ffff5c78c50, 
generator=0x40a0f0 <default_generators>, gendata=0x0) at query/eval.c:364

#5 0x00000000004076f4 in cmd_query (client=0x624d60, args=)

at cmds/query.c:52

#6 0x000000000040cea0 in dispatch_command (client=0x624d60,

args=0x7fffe80008e0) at listener.c:509

#7 0x000000000040d00e in client_thread (ptr=0x624d60) at listener.c:566
#8 0x00007ffff7bc4e9a in start_thread (arg=0x7ffff5c79700)

at pthread_create.c:308

#9 0x00007ffff76b43fd in clone ()

at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112

#10 0x0000000000000000 in ?? ()

Add log rotation support

I think we ought to rotate this log once in a while. Mine is 202 MiB..

-rw------- 1 bgertzfield THEFACEBOOK\Domain Users 202409754 Oct 7 08:57 .watchman.bgertzfield.log

watchman log-level gives error

watchman --persistent log-level debug
{
"version": "2.9.3",
"log_level": "debug"
}
1393368388: tid=1053128448 failed to identify PDU:
1393368388: tid=1053128448 unable to talk to your watchman!

Trigger added immediately after watch added doesn't work

I am using a Ruby program to automate the creation of a watched directory and trigger. Adding a trigger seems to work (running the 'trigger-list' command shows it), but the trigger never fires. Running the same exact 'trigger' command to add the trigger, but directly on the terminal, works just fine.
Is it something to do with environment?

Changes from editor do not trigger watchman

I have a vm running centos. I have compiled and installed watchman.
I have this trigger:

{
"version": "2.9.9",
"triggers": [
{
"name": "compilets",
"append_files": true,
"command": [
"make"
],
"stdin": [
"name",
"exists",
"new",
"size",
"mode"
],
"expression": [
"anyof",
[
"match",
"*.ts",
"wholename"
]
]
}
]
}

If I am in the terminal and use vim to change the files this gets triggered, but if I am using notepad++ (from my windows 7 machine) this does not trigger.

Is this the expected behaviour? I thought watchman would watch for changes in the folder independently of where does changes were coming from.

This folder is a shared folder from windows to linux (virtualbox does all the mouting stuff).
I was hoping that watchman would still work :)

blocked on reading when output is piped (line buffering?)

When I pipe watchman -pj --no-pretty, it hangs reading from the socket

$ strace -- watchman -pj --server-encoding=json --no-pretty | cat
...
read(0, "[\"subscribe\", \"/tmp/foo\", \"foxy\""..., 1024) = 79
read(0, "", 1024)                       = 0
socket(PF_FILE, SOCK_STREAM, 0)         = 3
connect(3, {sa_family=AF_FILE, path="/var/facebook/watchman/fugalh"}, 110) = 0
setsockopt(3, SOL_SOCKET, SO_RCVBUF, [1048576], 4) = 0
mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa4433b9000
write(3, "[\"subscribe\",\"/tmp/foo\",\"foxy\",{"..., 72) = 72
read(3, "{\"version\":\"2.9.1\",\"subscribe\":\""..., 1048576) = 702
fstat(1, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa4434c9000
read(3, 

At that point it just hangs. Note the piping into cat; when it runs with a tty as stdout it works fine:

$ strace -- watchman -pj --server-encoding=json --no-pretty
...
read(0, "[\"subscribe\", \"/tmp/foo\", \"foxy\""..., 1024) = 79
read(0, "", 1024)                       = 0
socket(PF_FILE, SOCK_STREAM, 0)         = 3
connect(3, {sa_family=AF_FILE, path="/var/facebook/watchman/fugalh"}, 110) = 0
setsockopt(3, SOL_SOCKET, SO_RCVBUF, [1048576], 4) = 0
mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f026bcd2000
write(3, "[\"subscribe\",\"/tmp/foo\",\"foxy\",{"..., 72) = 72
read(3, "{\"version\":\"2.9.1\",\"subscribe\":\""..., 1048576) = 702
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 2), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f026bde2000
write(1, "{\"version\":\"2.9.1\",\"subscribe\":\""..., 72) = 72
write(1, "{\"version\":\"2.9.1\",\"clock\":\"c:13"..., 630) = 630
read(3, 

Smells like line buffering, but I'm not sure how that would make a difference here - we read 702 bytes from fd 3 both times, but the first time we try to read again instead of trying to write anything.

(Linux)

Watchman goes nuts and consumes lots of CPU

I'm going to apologize in advance for the uselessness of this bug report. I'll update later if I get some better info. We're using watchman with git ( https://github.com/dturner-tw/git/tree/watchman ). Occasionally, on OS X (I think 10.9.4 but I'll update the ticket if I hear otherwise), watchman goes completely nuts and consumes all of the CPU. This tends to be correlated with git commands running, unsurprisingly (probably checkout or diff). Since I personally don't use a Mac, I have to rely on reports from others. Here's a log from one of the times when this happened: https://gist.github.com/dturner-tw/d6e54782978e55241deb (he then killed watchman and went on with his life). I know this log is not from the most recent version of watchman (it's from a late June version, IIRC), so it probably doesn't have the info that you need. I've since upgraded watchman, so hopefully next time we can get a better log.

I asked the same guy to run dtruss on the misbehaving process the next time it happened, to see if it was doing anything interesting. His machine immediately froze hard, so he sent a screenshot taken with his camera:

Looking at that screenshot, I notice that it's calling psynch_cvwait, which I think probably correspond to pthread_cond_timedwait. It looks like the two calls to pthread_cond_timedwait in the code don't check to ensure that they are the only thread awake (pthread_cond_signal is allowed to wake up more than one process.). But I didn't thoroughly audit the code -- I just glanced briefly.

I'll send more info when I have it.

INSTALL requirements, man page

Hi there

I am thinking about to package watchman for Debian and what I came across are two questions;

• What are the exact dependencies? aka INSTALL file with build requirements and instructions
• Can you create a man page?

Kind regards
Sascha

make install broken

make install now seems to includes an extra /usr/local, so it tries to install to /usr/local/usr/local/....

This is the commit that breaks it, along with the line I believe to be at fault:
6ea4ed8#diff-c949f93d03f44a4217d7a138f9e2e54aR3

Failed to identify PDU

After I noticed watchman wasn't executing the script specified I tried to debug the problem.

[root@dev www]# watchman --persistent log-level debug
{
    "version": "2.8.2",
    "log_level": "debug"
}
1381439535: tid=1055180544 failed to identify PDU: 
1381439535: tid=1055180544 unable to talk to your watchman!

Can anyone help me solve this issue? There doesn't seem to be any documentation about why this happens, and it doesn't look like anyone else has encounter this either.

Examples?

I've been googling around for some examples on how to use this, but I can't find anything...

Don't get me wrong, the documentation seems extensive, I just wish it had a for examples.

Any blog posts that someone can point me to?

Better yet, wanna show me a complete example that say, watches a jsx/ folder and compiles the React components while also watching a sass/ folder and compiling to a css file?

exclude ignore_vcs dirs from ignore_dirs to avoid breaking cookies

56fcdba introduced a significant performance regression on my system (OS X 10.8.5 (12F45)) causing a since query to take over a minute:

jimp-mbp1:watchman jimp$ time echo '["query", "/Users/jimp/devtools/buck", {"since": "n:buckd", "fields":["name","exists","new"]}]' | watchman --server-encoding=json --no-pretty -j > /dev/null

real 1m0.006s
user 0m0.003s
sys 0m0.002s

The same query using e0762ed on the same machine takes a fraction of a second:

jimp-mbp1:watchman jimp$ time echo '["query", "/Users/jimp/devtools/buck", {"since": "n:buckd", "fields":["name","exists","new"]}]' | watchman --server-encoding=json --no-pretty -j > /dev/null

real 0m0.018s
user 0m0.004s
sys 0m0.003s

The regression seems to occur whether or not I have a .watchmanconfig file in the root of the watched tree.

Run a command when an apk is updated

I'm having problems to make watchman work. What I want to do is that everytime I drop an updated version of an *.apk file into a folder, I want to send that apk to my device.

Here's watchman trigger-list output:

{
    "version": "2.9.8",
    "triggers": [
        {
            "name": "sendtodevice",
            "expression": [
                "pcre",
                "/Users/myuser/apk-builds/My.android.apk"
            ],
            "command": [
                "adb",
                "install",
                "-r",
                "/Users/myuser/apk-builds/My.android.apk"
            ]
        }
    ]
}

But every time I overwrite My.android.apk, nothing happens and the log doesn't even update.

NetBSD supports statvfs

NetBSD supports statvfs (and dropped support for statfs), but the code only checks for SunOS.
As an additional complication, some members of statvfs are called slightly differently, see
http://netbsd.gw.com/cgi-bin/man-cgi?statvfs+5+NetBSD-current

The following patch makes fstype.c compile on NetBSD; please check that I chose the correct field:

--- fstype.c.orig       2014-05-20 21:14:24.000000000 +0000
+++ fstype.c
@@ -53,11 +53,15 @@ w_string_t *w_fstype(const char *path)
   }

   return w_string_new(name);
-#elif defined(sun)
+#elif HAVE_SYS_STATVFS_H
   struct statvfs sfs;

   if (statvfs(path, &sfs) == 0) {
+#ifdef __NetBSD__
+    return w_string_new(sfs.f_fstypename);
+#else
     return w_string_new(sfs.f_basetype);
+#endif
   }
 #elif HAVE_STATFS
   struct statfs sfs;

Watchman leaks open files (Linux)

I believe that this is actually a Linux kernel bug that watchman is triggering. I'm reporting it here because maybe there's a way to work around it in watchman. I'm also going to report it to LKML; I'll comment with a link to the thread once it's posted.

Here's a script:
https://gist.github.com/dturner-tw/9336584

If you run it on Linux, you'll soon (after a few hundred lines of output) see something like:
{u'version': u'2.9.3', u'error': u'unable to resolve root /tmp/tmpL0H6OT: watch(/tmp/tmpL0H6OT): inotify_init error: Too many open files'}

Here's where it gets really fun: I kill the script. Then I kill watchman. Then I delete my /tmp/.watchman.{username}* files. So I should be ready to go again. But no, next time I run the script, I get the "Too many open files" message much more quickly. And in fact, inotify is messed up generally (not just for watchman):
$ tail -f /etc/hosts >/dev/null
tail: inotify cannot be used, reverting to polling: Too many open files

(I sometimes need to do that in two or three separate windows to trigger the error)

Only rebooting will make inotify work correctly again.

Some other diagnostics:
$ cat /proc/sys/fs/inotify/max_user_instances
128
$ ls -l /proc//fd/ 2>/dev/null | grep -c anon_inode:inotify
18
$ sudo ls -l /proc//fd/ 2>/dev/null | grep -c anon_inode:inotify
32

System info:
uname -a
Linux stross 3.11.0-17-generic #18 31~precise1-Ubuntu SMP Tue Feb 4 21:25:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

I also tried on an Ubuntu 3.8.0-36-generic kernel, and on a stock 3.13.5 that I built from source (this one in virtualbox).

watchman error on deleted vcs dir

If you watch a directory that contains a vcs directory, and then later delete that directory, watchman does not detect this, so it continues to attempt to use that directory for cookies, and thus nothing works.

(This is probably an edge case, but it is worth reporting)

./configure fails on 32-bit systems

% uname -a
Linux timbuntu 3.8.0-22-generic #33-Ubuntu SMP Thu May 16 15:17:59 UTC 2013 i686 i686 i686 GNU/Linux
% gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/i686-linux-gnu/4.7/lto-wrapper
Target: i686-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.7.3-1ubuntu1' --with-bugurl=file:///usr/share/doc/gcc-4.7/README.Bugs --enable-languages=c,c++,go,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.7 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.7 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin --with-system-zlib --enable-objc-gc --enable-targets=all --with-cloog --enable-cloog-backend=ppl --disable-cloog-version-check --disable-ppl-version-check --enable-multiarch --disable-werror --with-arch-32=i686 --with-multilib-list=m32,m64,mx32 --with-tune=generic --enable-checking=release --build=i686-linux-gnu --host=i686-linux-gnu --target=i686-linux-gnu
Thread model: posix
gcc version 4.7.3 (Ubuntu/Linaro 4.7.3-1ubuntu1)

% ./configure
checking build system type... i686-pc-linux-gnu
checking host system type... i686-pc-linux-gnu
checking target system type... i686-pc-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
checking whether make sets $(MAKE)... yes
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... configure: error: in /home/tim/tmp/watchman': configure: error: cannot run C compiled programs. If you meant to cross compile, use--host'.
See `config.log' for more details

Here's config.log:

configure:3320: checking whether we are cross compiling
configure:3328: gcc -o conftest -m64 -D_REENTRANT conftest.c >&5
configure:3332: $? = 0
configure:3339: ./conftest
./configure: line 3341: ./conftest: cannot execute binary file
configure:3343: $? = 126
configure:3350: error: in /home/tim/tmp/watchman': configure:3352: error: cannot run C compiled programs. If you meant to cross compile, use--host'.
See `config.log' for more details

If I modify configure to not use -m64, configure succeeds. But then make fails. I'll put that in another issue.

Watchman should escape color codes in environment variables before logging them

Some of my environment variables contain ASCII color codes (notably, the LESS_TERMCAP_* variables). Watchman logs these, along with other environment variables, when a file trigger is activated, but doesn't seem to escape them, which results in entire blocks of the log output being displayed in random colors when I tail the log. Here's an example of a problematic log line which triggers a color change:

1407265155: tid=12767232 envp[49] LESS_TERMCAP_mb=^[01;31m

Shouldn't watchman be escaping/stripping such codes?

add trigger option to disable passing the list of changed file names to the trigger command

This is not always wanted, is there an option to prevent this? If not, would love to see one.

$PATH is very hard to set on MacOS Yosemite

Commands that I invoke through Watchman on Yosemite have a PATH set to /usr/bin:/bin:/usr/sbin:/sbin

Needless to say, very little of my toolchain is in that path.

I think that it has to do with launchd defaulting to a very restricted path http://apple.stackexchange.com/questions/106355/

The only solution I've come up with is to source ~/.bash_profile in scripts invoked from Watchman, but this means that if I invoke them normally they have an enormous path set.

watchman -h should show command list

Either show the list of commands or link to the docs

Builds fail for 2.8 and 2.8.1 on OSX

When I attempt to compile watchman I get stack traces like the following:

brew: superenv removed: -Werror -g -O2 -Wall -Wextra -Wdeclaration-after-statement -g -gdwarf-2
cc -DHAVE_CONFIG_H -I.  -I./thirdparty/jansson -I./thirdparty/jansson -Werror -D_REENTRANT -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE  -g -O2 -I/usr/local/Cellar/pcre/8.33/include -Wall -Wextra -Wdeclaration-after-statement -g -gdwarf-2 -fno-omit-frame-pointer -c -o watchman-rules.o `test -f 'rules.c' || echo './'`rules.c
brew: superenv removed: -Werror -g -O2 -Wall -Wextra -Wdeclaration-after-statement -g -gdwarf-2
root.c:1018:15: error: use of undeclared identifier 'k'
      memset(&k, 0, sizeof(k));
              ^
/usr/include/secure/_string.h:76:21: note: expanded from macro 'memset'
  ((__darwin_obsz0 (dest) != (size_t) -1)                               \
                    ^
/usr/include/secure/_common.h:38:55: note: expanded from macro '__darwin_obsz0'
#define __darwin_obsz0(object) __builtin_object_size (object, 0)
                                                      ^
root.c:1018:15: error: use of undeclared identifier 'k'
      memset(&k, 0, sizeof(k));
              ^
/usr/include/secure/_string.h:77:30: note: expanded from macro 'memset'
   ? __builtin___memset_chk (dest, val, len, __darwin_obsz0 (dest))     \
                             ^
root.c:1018:15: error: use of undeclared identifier 'k'
      memset(&k, 0, sizeof(k));
              ^
/usr/include/secure/_string.h:78:27: note: expanded from macro 'memset'
   : __inline_memset_chk (dest, val, len))
                          ^
root.c:1019:15: error: use of undeclared identifier 'k'
      EV_SET(&k, dir->wd, EVFILT_VNODE, EV_ADD|EV_CLEAR,
              ^
/usr/include/sys/event.h:105:29: note: expanded from macro 'EV_SET'
        struct kevent *__kevp__ = (kevp);       \
                                   ^
root.c:1024:32: error: use of undeclared identifier 'k'
      if (kevent(root->kq_fd, &k, 1, NULL, 0, 0)) {
                               ^
root.c:1116:1: error: function declared in block scope cannot have 'static' storage class
static void spawn_command(w_root_t *root,
^
root.c:1119:41: error: expected ';' at end of declaration
    struct watchman_rule_match *matches)
                                        ^
                                        ;
root.c:1974:1: error: function declared in block scope cannot have 'static' storage class
static bool root_check_restrict(const char *watch_path)
^
root.c:1974:56: error: expected ';' at end of declaration
static bool root_check_restrict(const char *watch_path)
                                                       ^
                                                       ;
root.c:2468:1: error: expected '}'

^
root.c:935:1: note: to match this '{'
{
^
10 errors generated.
make[1]: *** [watchman-root.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [all] Error 2
/

I am able to compile 2.6 - 2.7 fine.

return the list of new files since a specified point of time

I'm adding inotify support git using watchman. One of the things I need is to know if new files are added since a specified point of time. If a file is deleted, then created again by the time watchman is queried, then it does not count as "added". This is the key point. I don't think this can be done with current queries. Please ignore the rest if it can be done already.

This could be done outside of watchman, although doing it in watchman is easier and more effecient. Would it be ok if such a feature is added to watchman?

What I'm thinking is to add an option to "query" object to save "exists" field for each entry, and save the current clock (iow a snapshot of "exists" for the entire watched dir). A new generator could be added, where it checks if the given 'since' clock is the same as the saved clocked above, then return entries that saved_exists != exists. If the clock does not match, error out. I can try to code some thing up..

OSX FSEvents gets into a bad state and watchman stops working

At least this is the conclusion that I came to. In the logs the following can be found when watching a new dir:

2014-10-30 23:59 watchman[68156] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2014-10-30 23:59 watchman[68156] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)2014-10-30 23:59 watchman[68156] (FSEvents.framework) 

After restarting, this will be gone and watchman will work great. Maybe there is a way to get around this.

Dig in to travis ruby watchman failures

https://travis-ci.org/facebook/watchman/jobs/39762394 shows one of the failures we encountered for a while before I ended up disabling the ruby tests again.
It seems to be a travis problem more than anything, but it would be great if we can find a way to work around (or ideally resolve!) this particular problem so that we can turn the ruby tests back on again.

Unportable test(1) construct in configure script.

The standard comparison operator for test(1) is '='; only bash supports '==', not even GNU coreutils' test(1) does. Here's a patch fixing some occurrences in the configure.ac script:

--- configure.ac.orig   2014-05-20 21:14:24.000000000 +0000
+++ configure.ac
@@ -88,7 +88,7 @@ AC_ARG_WITH(pcre, [
 ])

 if test "$pcre_config" != "no" ; then
-  if test "$pcre_config" == "yes" ; then
+  if test "$pcre_config" = "yes" ; then
     pcre_config="pcre-config"
   fi

@@ -187,11 +187,11 @@ AC_CHECK_FUNCS(backtrace backtrace_symbo
 if test -n "$GCC" ; then
   CFLAGS="$CFLAGS -Wall -Wextra -Wdeclaration-after-statement -g -gdwarf-2 -fno-omit-frame-pointer"
 fi
-if test -n "$GCC" -a "$stack_protect" == "yes" ; then
+if test -n "$GCC" -a "$stack_protect" = "yes" ; then
   CFLAGS="$CFLAGS -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4"
 fi
 IRONMANCFLAGS=""
-if test -n "$GCC" -a "$lenient" == "no" ; then
+if test -n "$GCC" -a "$lenient" = "no" ; then
   IRONMANCFLAGS="-Werror"
 fi
 AC_SUBST(IRONMANCFLAGS)

add fsevents support for osx

We have some concerns about the number of open files we'll need on osx.
Evaluate using fsevents instead to see if that handles things without requiring the sysctl to be bumped

-Werror problems on NetBSD

watchman-2.9.8 doesn't compile with -Werror on NetBSD; however, -Werror is the default except if one passes --enable-lenient.

The following warnings need fixing:

root.c: In function 'consume_kqueue':
root.c:1837:36: warning: initialization makes pointer from integer without a cast [enabled by default]
       struct watchman_file *file = root->keventbuf[i].udata;
                                    ^
string.c: In function 'w_string_dup_lower':
string.c:102:5: warning: array subscript has type 'char' [-Wchar-subscripts]
     if (tolower(str->buf[i]) != str->buf[i]) {
     ^
string.c:126:5: warning: array subscript has type 'char' [-Wchar-subscripts]
     buf[i] = tolower(str->buf[i]);
     ^
string.c: In function 'w_string_new_lower':
string.c:154:5: warning: array subscript has type 'char' [-Wchar-subscripts]
     buf[i] = tolower(str[i]);
     ^
string.c: In function 'w_string_equal_caseless':
string.c:202:5: warning: array subscript has type 'char' [-Wchar-subscripts]
     if (tolower(a->buf[i]) != tolower(b->buf[i])) {
     ^
string.c:202:5: warning: array subscript has type 'char' [-Wchar-subscripts]
string.c: In function 'w_string_suffix_match':
string.c:240:5: warning: array subscript has type 'char' [-Wchar-subscripts]
     if (tolower(str->buf[base + i]) != suffix->buf[i]) {
     ^
string.c: In function 'w_string_suffix':
string.c:267:9: warning: array subscript has type 'char' [-Wchar-subscripts]
         *buf = tolower(str->buf[end]);
         ^
thirdparty/tap.c: In function '_gen_result':
thirdparty/tap.c:94:5: warning: array subscript has type 'char' [-Wchar-subscripts]
     if(!isdigit(*c) && !isspace(*c)) {
     ^
thirdparty/tap.c:94:5: warning: array subscript has type 'char' [-Wchar-subscripts]
query/suffix.c: In function 'suffix_parser':
query/suffix.c:44:5: warning: array subscript has type 'char' [-Wchar-subscripts]
     arg[i] = tolower(arg[i]);
     ^
tests/bser.c: In function 'hexdump':
tests/bser.c:34:7: warning: array subscript has type 'char' [-Wchar-subscripts]
       printf("%c", isprint(start[i]) ? start[i] : '.');
       ^

See the CAVEATS section in
http://netbsd.gw.com/cgi-bin/man-cgi?ctype+3+NetBSD-current
for an explanation what the problem with "type 'char'" is.

failed to identify PDU: n:/usr/local

1. Installed watchman from homebrew with brew install watchman
2. Did watchman watch /Users/matt/dev/src/houdini/help.documents
3. Tried to do the subscribe example from the readme:

bash-3.2$ watchman -j -p <<-EOT
> ["subscribe", "/Users/matt/dev/src/houdini/help/documents", "mysub", {
> "expression": ["allof",
> ["type", "f"],
> ["not", "empty"],
> ["suffix", "txt"]
> ],
> "fields": ["name"]
> }]
> EOT

Get the following output:

{
    "version": "2.9.3",
    "clock": "c:1395957042:22510:1:2",
    "subscribe": "mysub"
}
1395957383: tid=2066264848 failed to identify PDU: n:/usr/local
1395957383: tid=2066264848 unable to talk to your watchman!

Not sure what /usr/local has to do with anything.

unaligned accesses on RISC processors

This sort of code is not portable and tends to not work on RISC processors:

  case BSER_INT16:
      *val = *(int16_t*)(buf+1);
      return 1;
    case BSER_INT32:
      *val = *(int32_t*)(buf+1);
      return 1;
    case BSER_INT64:
      *val = *(int64_t*)(buf+1);
      return 1;

It fails with bus errors on SPARC, for example, when the alignment isn't correct.

• bser_real(), bunser() - dereferences double pointer of unknown alignment
• bser_int(), bunser_int() - dereferences 16-bit, 32-bit, and 64-bit integer pointers of unknown alignment

Handle relative paths correctly

If I say watchman watch . from some small subdirectory, watchman hangs, persumably because it interpreted the . to mean a different directory to what I expected and is crawling some giant folder somewhere like / or ~.

It would be nice to have relative paths warn, or just handle them correctly.

Documentation error/installation script error (depending on your perspective)

According to the documentation building is a simple matter of calling first autogen.sh. But there are dependencies and so if you try to run this on OSX you get this:

./autogen.sh: line 9: aclocal: command not found
./autogen.sh: line 10: autoheader: command not found
./autogen.sh: line 11: automake: command not found
./autogen.sh: line 12: autoconf: command not found

That's not very helpful.

As an aside, if you install autoconf but haven't got aclocal installed the autogen.sh script doesn't actually quit on the (presumably fatal) error that it can't run aclocal, it just continues running.

Can't get watchman to work on Mac OS X

Hello,

I'm trying to use watchman to execute a Makefile target whenever a file changes. However, for this example I'm just trying to get it to do anything at all!

I've set up some roots:

[rdio/rdio] watchman watch-list
{
    "version": "2.9.3",
    "roots": [
        "/Users/justin/Dev/rdio/rdio/web/client/Components",
        "/Users/justin/Dev/rdio/rdio/web/client/core"
    ]
}

I've set up some triggers:

[rdio/rdio] watchman trigger-list `pwd`/web/client/Components
{
    "version": "2.9.3",
    "triggers": [
        {
            "name": "component-bits",
            "command": [
                "ls",
                "-l"
            ],
            "rules": [
                {
                    "pattern": "*.js *.less *.html",
                    "include": true,
                    "negated": false
                }
            ]
        }
    ]
}

And I'm running a command to log things: watchman --persistent --server-encoding=json log-level debug

When I modify a js file, it sees the modification but never executes anything. The relevant bits of the log file are:

{
    "version": "2.9.3",
    "log": "1389754641: tid=85434368 fse_thread: add /Users/justin/Dev/rdio/rdio/web/client/Components/PlayButton/PlayButton.js 15400\n"
}
{
    "version": "2.9.3",
    "log": "1389754641: tid=84897792 add_pending: /Users/justin/Dev/rdio/rdio/web/client/Components/PlayButton/PlayButton.js\n"
}
{
    "version": "2.9.3",
    "log": "1389754641: tid=84897792 lstat(/Users/justin/Dev/rdio/rdio/web/client/Components/PlayButton/PlayButton.js) file=0x7f8a327a2890 dir=0x0\n"
}
{
    "version": "2.9.3",
    "log": "1389754641: tid=84897792 file changed exists=1 via_notify=1 stat-changed=0 isdir=0 /Users/justin/Dev/rdio/rdio/web/client/Components/PlayButton/PlayButton.js\n"
}
{
    "version": "2.9.3",
    "log": "1389754641: tid=84897792 lstat(/Users/justin/Dev/rdio/rdio/web/client/Components/PlayButton) file=0x7f8a32715330 dir=0x7f8a32703b00\n"
}
{
    "version": "2.9.3",
    "log": "1389754641: tid=84897792 file changed exists=1 via_notify=1 stat-changed=0 isdir=1 /Users/justin/Dev/rdio/rdio/web/client/Components/PlayButton\n"
}
{
    "version": "2.9.3",
    "log": "1389754641: tid=88330240 notify_thread[/Users/justin/Dev/rdio/rdio/web/client/core] assessing triggers\n"
}
{
    "version": "2.9.3",
    "log": "1389754641: tid=84897792 sub last=2  pending=3\n"
}
{
    "version": "2.9.3",
    "log": "1389754641: tid=88330240 notify_thread[/Users/justin/Dev/rdio/rdio/web/client/core] assessing triggers\n"
}
{
    "version": "2.9.3",
    "log": "1389754641: tid=84897792 notify_thread[/Users/justin/Dev/rdio/rdio/web/client/Components] assessing triggers\n"
}

After that it just continues to print lines about assessing triggers.

It appears from the source that I should eventually see some logs about processing events and the posix_spawnp line with the return code of the command. None of that seems to be happening.

Where should I start debugging this? I'm more than willing to dive into the code but I don't yet understand how the FSEvent watcher goes from adding something to the pending queue to actually spawning the appropriate processes.

Any help would be much appreciated!

OS X install instructions recommend ludicrous maxfiles setting

The install guide instructs OS X users to adjust the sysctl settings kern.maxfiles and kern.maxfilesperproc to 10 and 1 million respectively.

Those are absolutely outrageous values that can cause serious performance degradation in other apps (namely ones that use the standard linux double-fork-then-close-all-fds as often recommended, for example here: http://stackoverflow.com/questions/3095566/linux-daemonize).

I would suggest to remove this instruction entirely (possibly with a hint that in rare cases those settings might have to be tuned).

add exclusion rules to .watchmanconfig

Some trees maintain dirs that hold a large number of files that are uninteresting wrt. watching the tree.

We currently hard-code a set of dirs to ignore (the git and mercurial history). This feature is about expanding that list to a configuration option that can be placed in the watchman config.

Does not compile on Ubuntu with GCC 4.7.3

% make
make all-am
make[1]: Entering directory /home/tim/tmp/watchman' gcc -DHAVE_CONFIG_H -I. -Ithirdparty/jansson -Werror -D_REENTRANT -g -O2 -Wall -Wextra -Wdeclaration-after-statement -g -gdwarf-2 -fno-omit-frame-pointer -MT watchman-opt.o -MD -MP -MF .deps/watchman-opt.Tpo -c -o watchman-opt.otest -f 'opt.c' || echo './'opt.c opt.c: In function ‘w_getopt’: opt.c:201:19: error: ignoring return value of ‘asprintf’, declared with attribute warn_unused_result [-Werror=unused-result] cc1: all warnings being treated as errors make[1]: *** [watchman-opt.o] Error 1 make[1]: Leaving directory/home/tim/tmp/watchman'
make: *** [all] Error 2

If I hack the code to not ignore asprintf, it gets further:

ht.c: In function ‘w_ht_string_copy’:
ht.c:370:19: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
ht.c: In function ‘w_ht_string_del’:
ht.c:376:19: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
ht.c: In function ‘w_ht_string_equal’:
ht.c:381:25: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
ht.c:381:41: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
ht.c: In function ‘w_ht_string_hash’:
ht.c:386:11: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
cc1: all warnings being treated as errors
make[1]: *** [watchman-ht.o] Error 1
make[1]: Leaving directory `/home/tim/src/watchman'
make: *** [all] Error 2

It compiled great on my Mac (GCC 4.2.1).

watchman returns 0 on error

I'm not sure if this is a bug or working as intended, but I was surprised that when I ran:

echo '["query", "/some/unwatched/path", {"since": "n:somenewname"}]' | watchman -j

I received a JSON formatted error as the output, but the return code from watchman was 0. I expected errors to return a non-zero exit-status and so was silently ignoring these errors due to the 0 return code.

crash during w_root_perform_age_out on OS X

Here are some logs:
https://gist.github.com/dturner-tw/5d411b401aa665f3e136

Thread 2 Crashed:
0   watchman                        0x00000001016924e9 w_ht_first + 9
1   watchman                        0x000000010169c448 age_out_dir + 40
2   watchman                        0x000000010169c4bc age_out_dir + 156
3   watchman                        0x000000010169c275 age_out_file + 197
4   watchman                        0x000000010169c34e w_root_perform_age_out + 158
5   watchman                        0x000000010169c40b consider_age_out + 75
6   watchman                        0x000000010169e001 run_notify_thread + 1009
7   libsystem_pthread.dylib         0x00007fff92df7899 _pthread_body + 138
8   libsystem_pthread.dylib         0x00007fff92df772a _pthread_start + 137
9   libsystem_pthread.dylib         0x00007fff92dfbfc9 thread_start + 13

Let me know if there's more info we can collect.

Provide more options for registering triggers

The following options are desirable:

• set stdin, stdout, stderr or other descriptors, > or >>
• control over passing the list of changed filenames via argv (for big trees this list can exceed the command line limits of the system)
• control over passing structured data about changed files via stdin (again, can be large, and it may not need to be JSON for some traditional unixy command compositions)
• cwd, environment

To a certain extent, these can be controlled via a shell script that does the same, except for the command line length and JSON processing (unless you have installed some auxiliary json tools)

OSX: must run make twice to build json dependency

When I build this on OSX, I get the following error the first time I run make:

cc  -m64 -I/usr/local/Cellar/pcre/8.33/include -Wall -Wextra -Wdeclaration-after-statement -g -gdwarf-2 -fno-omit-frame-pointer   -o tests/bser.t tests/tests_bser_t-bser.o tests_bser_t-bser.o -L. -lwmanjson -ltap -L/usr/local/Cellar/pcre/8.33/lib -lpcre
ld: library not found for -lwmanjson
clang: error: linker command failed with exit code 1 (use -v to see invocation)

The second time I execute the make command, the library builds.

Any ideas why it wouldn't work the first time around?

[question] How to write proper upstart script for watchman

I am using ubuntu 12.10, and tried to create upstart script for watchman but the status is always stop although it is running.
Here is my current upstart watchman.conf

description "Watchman service"

start on started beanstalkd
stop on runlevel [!2345]

respawn
respawn limit 5 2

script
    exec  sudo -u root /var/bi/bin/watchman watch /var/bi/var/dbf & /var/bi/bin/watchman -- trigger /var/bi/var/dbf importPr "*.DBF" "*.dbf" -- /var/bi/bicmd importPr
end script

ruby tests failing

I'm going to disable the ruby tests while someone figures out why they're failing.

Trigger only fires last command in shell script

I have a shell script that uses rsync when certain files are modified. When I checked the log file I noticed that only the output of the last command in that shell script was written to the log file.

I used the shutdown command and added the trigger back via a JSON file. The result said it was already defined. I changed a file and it correctly executed everything as normal.

Is there a known issue of this happening or could it be something on my end. I am on OS X 10.9.4

when a file is replaced by a directory

The below commands create a file "df" first, then delete it and replace "df" with a directory. I would expect watchman to return "df" listing (i.e. df/file). Returning "df" only is probably not so helpful. I think this is a tricky case, if we can't do any better, I think the behavior should be documented somewhere (the same for replacing a directory with a file).

$ watchman watch /tmp/z
{
    "version": "2.9.3",
    "watch": "/tmp/z"
}
$ watchman find /tmp/z
{
    "version": "2.9.3",
    "clock": "c:1415969169:24704:6:3",
    "files": []
}
$ echo abc >df
$ watchman since /tmp/z/ "c:1415969169:24704:6:3"
{
    "version": "2.9.3",
    "clock": "c:1415969169:24704:6:8",
    "is_fresh_instance": false,
    "files": [
        {
            "dev": 24,
            "gid": 100,
            "name": "df",
            "exists": true,
            "size": 4,
            "mode": 33188,
            "uid": 1000,
            "mtime": 1416822103,
            "ctime": 1416822103,
            "ino": 89526671,
            "nlink": 1,
            "oclock": "c:1415969169:24704:6:5",
            "new": true,
            "cclock": "c:1415969169:24704:6:5"
        }
    ]
}
$ rm df ; mkdir df; echo df >df/file
$ watchman since /tmp/z/ "c:1415969169:24704:6:3"
{
    "version": "2.9.3",
    "clock": "c:1415969169:24704:6:12",
    "is_fresh_instance": false,
    "files": [
        {
            "dev": 24,
            "gid": 100,
            "name": "df",
            "exists": true,
            "size": 60,
            "mode": 16877,
            "uid": 1000,
            "mtime": 1416822140,
            "ctime": 1416822140,
            "ino": 89527844,
            "nlink": 2,
            "oclock": "c:1415969169:24704:6:10",
            "new": true,
            "cclock": "c:1415969169:24704:6:10"
        }
    ]
}

Can't get watchman triggers to work

(If there's an IRC channel or something that would be more appropriate for this kind of help request, let me know! I didn't see any mentioned in the README.)

I'm trying to compile and minify my less and js with make and watchman, per the Rdio post.

(Is it just me, or do none of the code snippets in that blog post work out of the box? Anyway.)

Gist with Makefile, Python script, and scrollback.

I have a Makefile. make works. make clean works. I cannot for the life of me get make watch, which uses watchman, to work. The trigger syntax is lifted directly from the README.

trigger-list finds the triggers (see scrollback in gist), so that's not the problem.

I read this issue and it seemed to suggest I had to wrap the make call in a script, since watchman triggers pass the filename as an argument, which make doesn't expect.

So I wrote make.py to do this. I can run python make.py with arbitrary arguments and it does the right thing.

But make watch still doesn't do anything. It doesn't compile after a make clean. It doesn't compile after editing a file. Watchman has been inert the whole time I've been trying to get it to work.

Help?

Alpha Windows support

@wez edited this issue to place this status at the top

Current status: Alpha for 64-bit Windows.
It builds and passes the test suite on Windows Server 2012 R2 (x64) and should work on Windows 7 and later. PCRE expression terms are not currently supported.

At this time we're looking for alpha testers. Watchman on Windows may hang, crash or otherwise be unreliable. If you encounter problems we'll need you to provide detailed information about how to trigger the problem, and we may ask you to use visual studio to attach and perform some rudimentary debugging.

The latest build is available here:
https://ci.appveyor.com/api/projects/wez/watchman/artifacts/watchman.zip?branch=master&job=Environment:+WATCHMAN_WIN7_COMPAT%3D

Extract the zip file and make sure that watchman.exe is located in your PATH.
The watchman.pdb file is provided to facilitate debugging.

If you are consuming watchman from python, you will need to build pywatchman for yourself.
The released package on https://pypi.python.org/pypi/pywatchman should work for this purpose.

If you are consuming watchman from nodejs, the existing fb-watchman package available via npm should work, but the version in master has slightly better diagnostics in case you experience hangs.

Having lots of trouble getting watchman to work

i'm trying to use watchman in a make file so i can issue make watch and have it lint my code.

JSHINT=./node_modules/jshint/bin/jshint
JSHINTFLAGS=

js_files=$(shell find src -name '*.js')
jshint: $(js_files)
    $(JSHINT) $(JSHINTFLAGS) $?
# .PHONY just tells make that these rules don't produce files. So if there is a
# file called "jshint", it won't interpret that file as the output of the
# jshint recipe.
.PHONY: jshint

watch:
    watchman watch $(shell pwd)/src
    watchman -- trigger $(shell pwd)/src remake '*.js' -- make jshint

when i do this i get triggers for the right dir, however i don't see any output on the console.

we also looked at the watchman log output to see if the files were changing, and they are.
watchman --persistent --server-encoding=json log-level debug
log at https://gist.github.com/nurey/9916489

also, when i run watchman find $(pwd)/src '*.js' i get the file back that i want to be watched.

watchman find $(pwd)/src '*.js'
{
    "version": "2.9.5",
    "clock": "c:1396363466:31774:1:46",
    "files": [
        {
            "dev": 2050,
            "gid": 1000,
            "name": "imageupload.js",
            "exists": true,
            "size": 14890,
            "mode": 33188,
            "uid": 1000,
            "mtime": 1396364671,
            "ctime": 1396364671,
            "ino": 2498402,
            "nlink": 1,
            "oclock": "c:1396363466:31774:1:33",
            "new": true,
            "cclock": "c:1396363466:31774:1:1"
        }
    ]
}

really sorry to be posting an issue like this, but i could really use some working examples. the example on your readme that uses ls -l doesn't work for my either.

watchman -j hanging

Hi,
I'm very new to watchman. I find that the simple syntax is great and easy to work with. But I need extended syntax for a couple of reasons, try to run the example script with my path replaced :

watchman -j <<-EOT
["trigger", "/path/to/root", {
  "name": "assets",
  "expression": ["pcre", "\.(js|css|c|cpp)$"],
  "command": ["make"]
}]
EOT

It doesn't seem to do anything. It just hangs there and the trigger is not in the path's trigger list. Did I do anything wrong ?

Thanks for creating such great tool :)

age-out watches

More of a question than an issue. I'm writing a node library on top of the fb-watchman node binding and I have question about cleanup:

• Do I need to watch-del the directory I watched? Or is it enough to just unsubscribe?
• What if the user has other programs using that watched directory, will I be messing that up?
• If I don't watch-del, is that a leak?

Thanks

Complete build options for Pthread API

Would you like to add the configuration script "AX_PTHREAD" to your build specification?