100% CPU usage on high (100k+) number of watches

[notify-rs]

100% CPU usage on high (100k+) number of watches

[vandenoever]

The code to set up watches recursively in inotifywatch is here:

https://github.com/rvoicilas/inotify-tools/blob/master/libinotifytools/src/inotifytools.c#L1289

[passcod]

If you run the inotifywatch program for, say, 30 seconds, how does it fare?

[passcod]

Note that 100k+ watches is probably an edge-case... Was this for testing, or are you doing this regularly?

[passcod]

As far as I can tell, the code you linked does a simple recursive directory walk. We delegate ours to the Walkdir crate, but the base implementation is pretty much the same.

[passcod]

As with @hannobraun upstream at inotify-rs, I think hitting the limit on watches is understandable and expected in this case. The bug remains that you're seeing 100% CPU usage, so that's what should be focused on, I think.

[vandenoever]

If you run the inotifywatch program for, say, 30 seconds, how does it fare?

It runs fine with barely any CPU usage.

$ time inotifywatch ~ -r -t 30
Establishing watches...
Finished establishing watches, now collecting statistics.
total  access  modify  close_write  close_nowrite  open  moved_from  moved_to  create  filename
324    114     0       0            105            105   0           0         0
....
real    0m33.266s
user    0m0.879s
sys     0m2.384s

[vandenoever]

Note that 100k+ watches is probably an edge-case... Was this for testing, or are you doing this regularly?

I'd like to run a daemon that watches all files in $HOME. Most desktop search clients do this.

With inotify, you do not need to add a watch to a file if you're watching the parent directory.

When a directory is monitored, inotify will return events for the directory itself, and for files inside the directory.

source: man inotify

[passcod]

Yes, I think we do only watch directories unless specifically asked ;) (and
even then, at first we refused to watch single files).

I'm not sure desktop searches actually do this, rather I think they keep an
index that they update by crawling regularly. On Gnome, for example, the
tracker service's logs can be seen doing that — I occasionally have to
kill it when it gets stuck on large files that change while it's looking at
them.

Even then, the 100% cpu usage is certainly a bug, and that inotifywatch
manages it just fine is a testament to that.

This may not get fixed until the great refactor of #62 and related, mind,
unless the CPU sink is identified...

On Sun, 22 May 2016, 19:17 Jos van den Oever [email protected]
wrote:

Note that 100k+ watches is probably an edge-case... Was this for testing,
or are you doing this regularly?

I'd like to run a daemon that watches all files in $HOME. Most desktop
search clients do this.

With inotify, you do not need to add a watch to a file if you're watching
the parent directory.

When a directory is monitored, inotify will return events for the
directory itself, and for files inside the directory.

source: man inotify

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#69 (comment)

[vandenoever]

From reading the code, it seems that you are putting a watch on every file. Simply adding a filter there with is_dir() would reduce the memory load and perhaps also the cpu load.

https://github.com/passcod/rsnotify/blob/master/src/inotify/mod.rs#L102

[passcod]

Ah, you're right. This must have slipped through a refactor. I'll fix
tomorrow.

On Mon, 23 May 2016, 02:37 Jos van den Oever [email protected]
wrote:

From reading the code, it seems that you are putting a watch on every
file. Simply adding a filter there with is_dir() would reduce the memory
load and perhaps also the cpu load.

https://github.com/passcod/rsnotify/blob/master/src/inotify/mod.rs#L102

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#69 (comment)

[vandenoever]

#74 changes rsnotify so it only watches directories.

[vandenoever]

I've found that the example code is wrong. There is no 100% cpu user. Here is a fixed example. I'll close this issue.

extern crate notify;
use std::sync::mpsc::Receiver;
use std::sync::mpsc::sync_channel;
use std::thread;
use notify::{RecommendedWatcher, Watcher};
use std::sync::mpsc::channel;

fn watch() -> notify::Result<()> {
  // Create a channel to receive the events.
  let (tx, rx) = channel();

  // Automatically select the best implementation for your platform.
  // You can also access each implementation directly e.g. INotifyWatcher.
  let mut watcher: RecommendedWatcher = try!(Watcher::new(tx));

  // Add a path to be watched. All files and directories at that path and
  // below will be monitored for changes.
  try!(watcher.watch("/home/oever"));    
  loop {
      // You'll probably want to do that in a loop. The type to match for is
      // notify::Event, look at src/lib.rs for details.
      match rx.recv() {
        Ok(notify::Event{path:Some(path),op:Ok(op)}) => {
            println!("{:?} {:?}", op, path);
        },
        Err(e) => println!("watch error {}", e),
        _ => ()
      }
  }
}

fn main() {
  if let Err(err) = watch() {
    println!("Error!: {:?}", err)
  }
}