Giter Site home page Giter Site logo

fabiospampinato / watcher Goto Github PK

View Code? Open in Web Editor NEW
109.0 6.0 13.0 75 KB

The file system watcher that strives for perfection, with no native dependencies and optional rename detection support.

License: MIT License

TypeScript 32.62% JavaScript 67.38%
fs file system filesystem watch watcher

watcher's Introduction

Watcher

The file system watcher that strives for perfection, with no native dependencies and optional rename detection support.

Features

  • Reliable: This library aims to handle all issues that may possibly arise when dealing with the file system, including some the most popular alternatives don't handle, like EMFILE errors.
  • Rename detection: This library can optionally detect when files and directories are renamed, which allows you to provide a better experience to your users in some cases.
  • Performant: Native recursive watching is used when available (macOS and Windows), and it's efficiently manually performed otherwise.
  • No native dependencies: Native dependencies can be painful to work with, this library uses 0 of them.
  • No bloat: Many alternative watchers ship with potentially useless and expensive features, like support for globbing, this library aims to be much leaner while still exposing the right abstractions that allow you to use globbing if you want to.
  • TypeScript-ready: This library is written in TypeScript, so types aren't an afterthought but come with the library.

Comparison

You are probably currently using one of the following alternatives for file system watching, here's how they compare against Watcher:

  • fs.watch: Node's built-in fs.watch function is essentially garbage and you never want to use it directly.
    • Cons:
      • Recursive watching is only supported starting in Node v19.1.0 on Linux.
      • Even if you only need to support environments where native recursive watching is provided, the events provided by fs.watch are completely useless as they tell you nothing about what actually happened in the file system, so you'll have to poll the file system on your own anyway.
      • There are many things that fs.watch doesn't take care of, for example watching non-existent paths is just not supported and EMFILE errors are not handled.
  • chokidar: this is the most popular file system watcher available, while it may be good enough in some cases it's not perfect.
    • Cons:
      • It requires a native dependency for efficient recursive watching under macOS, and native dependencies can be a pain to work with.
      • It doesn't watch recursively efficiently under Windows, Watcher on the other hand is built upon Node's native recursive watching capabilities for Windows.
      • It can't detect renames.
      • If you don't need features like globbing then chokidar will bloat your app bundles unnecessarely.
      • EMFILE errors are not handled properly, so if you are watching enough files chokidar will eventually just give up on them.
      • It's not very actively maintened, Watcher on the other hand strives for having 0 bugs, if you can find some we'll fix them ASAP.
    • Pros:
      • It supports handling symlinks.
      • It has some built-in support for handling temporary files written to disk while perfoming an atomic write, although ignoring them in Watcher is pretty trivial too, you can ignore them via the ignore option.
      • It can more reliably watch network attached paths, although that will lead to performance issues when watching ~lots of files.
      • It's more battle tested, although Watcher has a more comprehensive test suite and is used in production too (for example in Notable, which was using chokidar before).
  • node-watch: in some ways this library is similar to Watcher, but much less mature.
    • Cons:
      • No initial events can be emitted when starting watching.
      • Only the "update" or "remove" events are emitted, which tell you nothing about whether each event refers to a file or a directory, or whether a file got added or modified.
      • "add" and "unlink" events are not provided in some cases, like for files inside an added/deleted folder.
      • Watching non-existent paths is not supported.
      • It can't detect renames.
  • nsfw: this is a lesser known but pretty good watcher, although it comes with some major drawbacks.
    • Cons:
      • It's based on native dependencies, which can be a pain to work with, especially considering that prebuild binaries are not provided so you have to build them yourself.
      • It's not very customizable, so for example instructing the watcher to ignore some paths is not possible.
      • Everything being native makes it more difficult to contribute a PR or a test to it.
      • It's not very actively maintained.
    • Pros:
      • It adds next to 0 overhead to the rest of your app, as the watching is performed in a separate process and events are emitted in batches.
  • "perfection": if there was a "perfect" file system watcher, it would compare like this against Watcher (i.e. this is pretty much what's currently missing in Watcher):
    • Pros:
      • It would support symlinks, Watcher doesn't handle them just yet.
      • It would watch all parent directories of the watched roots, for unlink detection when those parents get unlinked, Watcher currently also watches only up-to 1 level parents, which is more than what most other watchers do though.
      • It would provide some simple and efficient APIs for adding and removing paths to watch from/to a watcher instance, Watcher currently only has some internal APIs that could be used for that but they are not production-ready yet, although closing a watcher and making a new one with the updated paths to watch works well enough in most cases.
      • It would add next to 0 overhead to the rest of your app, currenly Watcher adds some overhead to your app, but if that's significant for your use cases we would consider that to be a bug. You could potentially already spawn a separate process and do the file system watching there yourself too.
      • Potentially there are some more edge cases that should be handled too, if you know about them or can find any bug in Watcher just open an issue and we'll fix it ASAP.

Install

npm install --save watcher

Options

The following options are provided, you can use them to customize watching to your needs:

  • debounce: amount of milliseconds to debounce event emission for.
    • by default this is set to 300.
    • the higher this is the more duplicate events will be ignored automatically.
  • depth: maximum depth to watch files at.
    • by default this is set to 20.
    • this is useful for avoiding watching directories that are absurdly deep, that would probably waste resources.
  • limit: maximum number of paths to prod.
    • by default this is set to 10_000_000.
    • this is useful as a safe guard in cases where for example the user decided to watch /, perhaps by mistake.
  • ignore: optional function (or regex) that if returns true for a path it will cause that path and all its descendants to not be watched at all.
    • by default this is not set, so all paths are watched.
    • setting an ignore function can be very important for performance, you should probably ignore folders like .git and temporary files like those used when writing atomically to disk.
    • if you need globbing you'll just have to match the path passed to ignore against a glob with a globbing library of your choosing.
  • ignoreInitial: whether events for the initial scan should be ignored or not.
    • by default this is set to false, so initial events are emitted.
  • native: whether to use the native recursive watcher if available and needed.
    • by default this is set to true.
    • the native recursive watcher is only available under macOS and Windows.
    • when the native recursive watcher is used the depth option is ignored.
    • setting it to false can have a positive performance impact if you want to watch recursively a potentially very deep directory with a low depth value.
  • persistent: whether to keep the Node process running as long as the watcher is not closed.
    • by default this is set to false.
  • pollingInterval: polling is used as a last resort measure when watching non-existent paths inside non-existent directories, this controls how often polling is performed, in milliseconds.
    • by default this is set to 3000.
    • you can set it to a lower value to make the app detect events much more quickly, but don't set it too low if you are watching many paths that require polling as polling is expensive.
  • pollingTimeout: sometimes polling will fail, for example if there are too many file descriptors currently open, usually eventually polling will succeed after a few tries though, this controls the amount of milliseconds the library should keep retrying for.
    • by default this is set to 20000.
  • recursive: whether to watch recursively or not.
    • by default this is set to false.
    • this is supported under all OS'.
    • this is implemented natively by Node itself under macOS and Windows.
  • renameDetection: whether the library should attempt to detect renames and emit rename/renameDir events.
    • by default this is set to false.
    • rename detection may cause a delayed event emission, because the library may have to wait some more time for it.
    • if disabled, the raw underlying add/addDir and unlink/unlinkDir events will be emitted instead after a rename.
    • if enabled, the library will check if each pair of add/unlink or addDir/unlinkDir events are actually rename or renameDir events respectively, so it will wait for both of those events to be emitted.
    • rename detection is fairly reliable, but it is fundamentally dependent on how long the file system takes to emit the underlying raw events, if it takes longer than the set rename timeout the app won't detect the rename and will instead emit the underlying raw events.
  • renameTimeout: amount of milliseconds to wait for a potential rename/renameDir event to be detected.
    • by default this is set to 1250.
    • the higher this value is the more reliably renames will be detected, but don't set this too high, or the emission of some events could be delayed by that amount.
    • the higher this value is the longer the library will take to emit add/addDir/unlink/unlinkDir events.

Usage

Watcher returns an EventEmitter instance, so all the methods inherited from that are supported, and the API is largely event-driven.

The following events are emitted:

  • Watcher events:
    • error: Emitted whenever an error occurs.
    • ready: Emitted after the Watcher has finished instantiating itself. No events are emitted before this events, expect potentially for the error event.
    • close: Emitted when the watcher gets explicitly closed and all its watching operations are stopped. No further events will be emitted after this event.
    • all: Emitted right before a file system event is about to get emitted.
  • File system events:
    • add: Emitted when a new file is added.
    • addDir: Emitted when a new directory is added.
    • change: Emitted when an existing file gets changed, maybe its content changed, maybe its metadata changed.
    • rename: Emitted when a file gets renamed. This is only emitted when renameDetection is enabled.
    • renameDir: Emitted when a directory gets renamed. This is only emitted when renameDetection is enabled.
    • unlink: Emitted when a file gets removed from the watched tree.
    • unlinkDir: Emitted when a directory gets removed from the watched tree.

Basically it you have used chokidar in the past Watcher emits pretty much the same exact events, except that it can also emit rename/renameDir events, it doesn't provide stats objects but only paths, and in general it exposes a similar API surface, so switching from (or to) chokidar should be easy.

The following interface is provided:

type Roots = string[] | string;

type TargetEvent = 'add' | 'addDir' | 'change' | 'rename' | 'renameDir' | 'unlink' | 'unlinkDir';
type WatcherEvent = 'all' | 'close' | 'error' | 'ready';
type Event = TargetEvent | WatcherEvent;

type Options = {
  debounce?: number,
  depth?: number,
  ignore?: (( targetPath: Path ) => boolean) | RegExp,
  ignoreInitial?: boolean,
  native?: boolean,
  persistent?: boolean,
  pollingInterval?: number,
  pollingTimeout?: number,
  recursive?: boolean,
  renameDetection?: boolean,
  renameTimeout?: number
};

class Watcher {
  constructor ( roots: Roots, options?: Options, handler?: Handler ): this;
  on ( event: Event, handler: Function ): this;
  close (): void;
}

You would use the library like this:

import Watcher from 'watcher';

// Watching a single path
const watcher = new Watcher ( '/foo/bar' );

// Watching multiple paths
const watcher = new Watcher ( ['/foo/bar', '/baz/qux'] );

// Passing some options
const watcher = new Watcher ( '/foo/bar', { renameDetection: true } );

// Passing an "all" handler directly
const watcher = new Watcher ( '/foo/bar', {}, ( event, targetPath, targetPathNext ) => {} );

// Attaching the "all" handler manually
const watcher = new Watcher ( '/foo/bar' );
watcher.on ( 'all', ( event, targetPath, targetPathNext ) => { // This is what the library does internally when you pass it a handler directly
  console.log ( event ); // => could be any target event: 'add', 'addDir', 'change', 'rename', 'renameDir', 'unlink' or 'unlinkDir'
  console.log ( targetPath ); // => the file system path where the event took place, this is always provided
  console.log ( targetPathNext ); // => the file system path "targetPath" got renamed to, this is only provided on 'rename'/'renameDir' events
});

// Listening to individual events manually
const watcher = new Watcher ( '/foo/bar' );

watcher.on ( 'error', error => {
  console.log ( error instanceof Error ); // => true, "Error" instances are always provided on "error"
});
watcher.on ( 'ready', () => {
  // The app just finished instantiation and may soon emit some events
});
watcher.on ( 'close', () => {
  // The app just stopped watching and will not emit any further events
});
watcher.on ( 'all', ( event, targetPath, targetPathNext ) => {
  console.log ( event ); // => could be any target event: 'add', 'addDir', 'change', 'rename', 'renameDir', 'unlink' or 'unlinkDir'
  console.log ( targetPath ); // => the file system path where the event took place, this is always provided
  console.log ( targetPathNext ); // => the file system path "targetPath" got renamed to, this is only provided on 'rename'/'renameDir' events
});
watcher.on ( 'add', filePath => {
  console.log ( filePath ); // "filePath" just got created, or discovered by the watcher if this is an initial event
});
watcher.on ( 'addDir', directoryPath => {
  console.log ( directoryPath ); // "directoryPath" just got created, or discovered by the watcher if this is an initial event
});
watcher.on ( 'change', filePath => {
  console.log ( filePath ); // "filePath" just got modified
});
watcher.on ( 'rename', ( filePath, filePathNext ) => {
  console.log ( filePath, filePathNext ); // "filePath" got renamed to "filePathNext"
});
watcher.on ( 'renameDir', ( directoryPath, directoryPathNext ) => {
  console.log ( directoryPath, directoryPathNext ); // "directoryPath" got renamed to "directoryPathNext"
});
watcher.on ( 'unlink', filePath => {
  console.log ( filePath ); // "filePath" got deleted, or at least moved outside the watched tree
});
watcher.on ( 'unlinkDir', directoryPath => {
  console.log ( directoryPath ); // "directoryPath" got deleted, or at least moved outside the watched tree
});

// Closing the watcher once you are done with it
watcher.close ();

// Updating watched roots by closing a watcher and opening an updated one
watcher.close ();
watcher = new Watcher ( /* Updated options... */ );

Thanks

  • chokidar: for providing me a largely good-enough file system watcher for a long time.
  • node-watch: for providing a good base from which to make Watcher, and providing some good ideas for how to write good tests for it.

License

MIT © Fabio Spampinato

watcher's People

Contributors

benmccann avatar fabiospampinato avatar scenaristeur avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

watcher's Issues

TypeError: Watcher is not a constructor

Hello,

I'm trying to use the package with Electron, I'm getting this error :
TypeError: Watcher is not a constructor

Here's how I'm importing the package (because require isn't working) :
const Watcher = await import("watcher");

Any idea or solution to solve this ? Thanks

Cannot watch relative paths

I noticed that relative paths watchers don't emit events at all.

// no events
const watcher = new Watcher(this.dirname, {
  renameDetection: true
, ignoreInitial: true
, recursive: true
})

// emit events
const watcher = new Watcher(path.resolve(this.dirname), {
  renameDetection: true
, ignoreInitial: true
, recursive: true
})

Is this expected behavior?

Watcher hung on initializing windows 10

I have been using Watcher for a while without issue.

I'm initializing a watcher with 1 folder that contains 2000 folders, each with 1 sub folder that has a few files.

I put a few console logs in and saw the watcher is getting hung in tiny-readdir here: https://github.com/fabiospampinato/tiny-readdir/blob/master/src/index.ts#L35

In testing I increased the concurrency limit over 2k and it finished initialization.

No symlinks and the config is

{
  ignored: /(^|[\/\\])\../, // ignore dotfiles
  renameDetection: true,
  renameTimeout: 2000,
  recursive: true,
  ignoreInitial: true,
  persistent: true
}

TypeScript type errors on import

When I use this library I get the following when running tsc (using version 4.1.3):

node_modules/watcher/dist/watcher_handler.d.ts:7:14 - error TS2709: Cannot use namespace 'Watcher' as a type.

7     watcher: Watcher;
               ~~~~~~~

node_modules/watcher/dist/watcher_handler.d.ts:13:26 - error TS2709: Cannot use namespace 'Watcher' as a type.

13     constructor(watcher: Watcher, config: WatcherConfig, base?: WatcherHandler);
                            ~~~~~~~

I can work around this by adding skipLibCheck: true to my tsconfig.json, it really shouldn't be necessary.

I think the issue is caused by the use of:

import Watcher from './watcher';

Watcher is both the default export and the namespace.

Changing watcher_handler.d.ts so that it imports the type not the default export:

import { type as Watcher } from './watcher';

Fixes it.

create and delete event not emitted if folder was freshly created and then deleted within timeout period

My watcher config is -

{
      ignoreInitial: true,
      recursive: true,
      renameDetection: true,
      renameTimeout: 10000,
      persistent: true,
      ignore(filePath: string) {
        return !!filePath.match(/(^|[\/\\])\../);
      },
    }

Is this the correct behaviour? If i create a file and delete it within the timeout period or delete it before create event is emitted, delete is not emitted as well.

Can we get both the events to emit? Just for consistency and what if we want to track and perform some server-side updates for each of the event.

Is this just happening because renameTimeout is too high?

Watcher doesn't listening local disk such as Disk D on Windows

Hi!

When I try to watch local disk such as 'Disk D' or or similar,
I receive a list items of Disk D, but after list items of the disk it does not listening. I create a new file or folder - nothing happens, remove something - nothing...

But, when I try to watch some inner folder of 'Disk D', for example 'D:\myfolder' it was pretty works!

What confuses me is that I get 4 characters "\" after "D:" because it must be 2 symbols "D:\\" or 1 symbol "D:/"

Here is my code below:

import Watcher from 'watcher';

const watcher = new Watcher('D:\\', {
    depth: 1,
    ignored: /(^|[/\\])\../,
    persistent: true,
    renameDetection: true,
    ignoreInitial: false
});

watcher.on(`ready`, (...args) => console.log(`[ready]`, args));
watcher.on(`error`, (...args) => console.log(`[error]`, args));
watcher.on(`close`, (...args) => console.log(`[close]`, args));

watcher.on(`add`, (...args) => console.log(`[add]`, args));
watcher.on(`addDir`, (...args) => console.log(`[addDir]`, args));
watcher.on(`change`, (...args) => console.log(`[change]`, args));
watcher.on(`rename`, (...args) => console.log(`[rename]`, args));
watcher.on(`renameDir`, (...args) => console.log(`[renameDir]`, args));
watcher.on(`unlink`, (...args) => console.log(`[unlink]`, args));
watcher.on(`unlinkDir`, (...args) => console.log(`[unlinkDir]`, args));

Output:

PS C:\myProjects\node\agent> node .\test-watcher.js
[ready] []
[addDir] [ 'D:\\', undefined ]
[addDir] [ 'D:\\\\myfolder', undefined ]
[addDir] [ 'D:\\\\$RECYCLE.BIN', undefined ]
[addDir] [ 'D:\\\\Backup', undefined ]
[addDir] [ 'D:\\\\msdownld.tmp', undefined ]
[addDir] [ 'D:\\\\Others', undefined ]
[addDir] [ 'D:\\\\Projects', undefined ]
[addDir] [ 'D:\\\\Soft', undefined ]
[addDir] [ 'D:\\\\share', undefined ]
[add] [ 'D:\\\\clip.MOV', undefined ]
[add] [ 'D:\\\\MediaID.bin', undefined ]
[add] [ 'D:\\\\Thumbs.db', undefined ]

And then nothing else happens...

Seeking advice regarding integrating file watcher into an asynchronous-based application.

As the title says, I am developing a non web Node app and like most modern apps it is very asynchronous and uses async and await as well as promises. The issue I am having is how I can expose my apps' async methods to be called by some file event triggered by the file watcher which doesn't support asynchronous event handlers. I'm hoping that someone can provide some insights into finding a solution.

Performance question re: large directories

I'm wondering how well this library handles large directories.

Use case:

  • 100,000+ files
  • Not deep, maybe 5-7
  • Linux
  • Infrequent updates (days to weeks apart)
  • Changes are usually new files (maybe 100)

Is this feasible? In particular I want to avoid excessive memory usage and periodic CPU load spikes. Less cpu/memory would be preferable to timely updates.

There is a typo in example.

watcher.on ( 'addDir', directoryPath => {
  console.log ( filePath ); // "directoryPath" just got created, or discovered by the watcher if this is an initial event
});

Wait writing files to finish

Hello,
I'm coming from chokidar and it has the awaitWriteFinish option that is quite useful. It's specified in the watcher description that we can use the ignore option to wait writing (e.g you copy a large file, you want to wait it's fully copied to send the event). Could you please provide some example ?
Thank you

rename detection without timeout

It seems the current implementation depends on a timeout. Is it possible to avoid this at least in some cases such as when the native watcher is used?

TypeScript issues

Relative import paths need explicit file extensions in ECMAScript imports when '--moduleResolution' is 'node16' or 'nodenext'. Did you mean './enums.js'?
Relative import paths need explicit file extensions in ECMAScript imports when '--moduleResolution' is 'node16' or 'nodenext'. Did you mean './watcher_handler.js'?
Relative import paths need explicit file extensions in ECMAScript imports when '--moduleResolution' is 'node16' or 'nodenext'. Did you mean './watcher_locker.js'?
Relative import paths need explicit file extensions in ECMAScript imports when '--moduleResolution' is 'node16' or 'nodenext'. Did you mean './watcher_poller.js'?

... and so on.

Renaming not triggering proper events with different character cases

Hi,

When I rename a file with the same name but with different cases, such as test.txt -> Test.txt, unlike other renaming behaviors, such as test.txt -> test2.txt, the events triggered are "change + add" instead of an "unlink + add".

Occurred with:

  • Windows 10 Professional (20H2)
  • Node v12.22.1 & v14.17.2
  • Watcher v1.2.0
  • renameDetection is and must be disabled in my case but using it doesn't work either and also returns a "change + add"

Thanks!

doc: Small update to `Comparison` documentation

I love how well documented this project is. Specially the comparison section.

One minor detail:

Recursive watching is not supported under Linux, so if you need to support Linux at all you are out of luck already.

I do believe that Node.js added this support on Node 19 😋


image

Opening a file triggers change

new Watcher('/xy', { recursive: true }, () => console.log('change'));

On Windows 11, watching a directory like this is triggering a 'change' event whenever a file is opened (clicked in solution) in Jetbrains Rider.

Does somebody know something about this behaviour?

Thanks for any help.

Error [ERR_REQUIRE_ESM]: require() of ES Module

Hi,
thanks for this great tool!
I was successfully using it on a Node Js project and it was working smoothly.
I am now converting that project to use TS but now I get this error at build time:

Error [ERR_REQUIRE_ESM]: require() of ES Module .../node_ts/node_modules/watcher/dist/watcher.js
from .../dist/data.js not supported.
Instead change the require of watcher.js in .../dist/data.js to a dynamic import() which is available in all CommonJS modules.

Is this a known issue or am I doing something wrong?

best!

Reading a file will trigger a change event

Step 1:watch a folder (index.js)

import Watcher from "watcher";
const watcher = new Watcher("D:/code", {
  ignoreInitial: true,
  recursive: true,
});
watcher.on("all", (event, targetPath) => {
  console.log(event, targetPath);
});
watcher.on("ready", () => {
  console.log("watcher is ready");
});

run node index.js

Step 2:read any file in this folder(readfile.js)

import fs from "fs";
let content = fs.readFileSync("D:/code/a/b/c.js", "utf-8");
console.log(content);

run node readfile.js

Step 3:index.js 's output is:

watcher is ready
change D:/code/a/b/c.js

environment:win11 x64、node v16.20

Support for emitting "initialized" event after initial scan of watched directory

It would be nice to have an event emitted after the initial items are scanned. Since there is an ignoreInitial option, I assume this should be possible.

watcher.on ( 'initialized', () => {
  // All initial (add & addDir) events for pre-existing files & folders have just been emitted
});

If this is an enhancement that should be added, I would be happy to work on this if I can be pointed in the right direction.

No way to distinguish between file and directory in ignore callback

The ignore callback just gets a path, but it can be helpful for ignore rules to know if that path is for a directory or a file. Of course, you could do a file stat to find out, but that would presumably be very inefficient. If this information is already available to Watcher it would be good to pass it in the ignore function handler.

Unexpected behavior and some other issues

Thanks for creating a "chokidar" alternative. I'm considering moving to "watcher" at some point in the future because it handles drive root dirs like C:/ properly (doesn't emit unlink events on init), but I encountered some problems with it, as well as some unexpected behavior, while testing it:

  1. If you set, let's say, depth: 3 and recursive: true it still detects changes at deeper levels: 4, 5, 6, 7, ... and so on, which doesn't make much sense since depth is supposed to represent the scan depth limit.
  2. If you set depth option but do not specify recursive: true it won't detect any changes in the nested directories at depth > 0, which doesn't make sense either. I think it should automatically set recursive: true if you specify depth > 0. Am I misunderstanding the purpose of this property? What's the point of setting depth: 10 if it does not detect any changes at depth > 0 until you set recursive: true?
  3. If you call the function that inits the watcher multiple times in a row for different paths with a relatively high depth level, like 6, so it doesn't have a chance to finish the previous task, it doesn't close the previous watchers, it just keeps scanning the directory specified in the first function call, until the scan is finished. I think it's happening because it scans the drive at a max possible speed, and to no surprise, such a computationally intensive task blocks the thread completely, as if it was a synchronous function. I think it can be solved with a timeout function that pauses the scan every 1 ms or so, or by setting a lower highWaterMark value for the node's read stream that scans the drive, not sure.
  4. Sometimes, it starts throwing the Uncaught TypeError shown below during the drive scan (before it emits ready event). It happens 100% of the time for some paths like C:/ (system drive) past certain depth (for me it's 4) and only ~25% of the times for other paths. I just reload the app, and after a few seconds manually trigger the watcher function for some path with a button. Sometimes I get no errors and it logs ready, sometimes I get the error once before it logs ready, sometimes it throws the error multiple times:
Uncaught TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received null
    at validateString (internal/validators.js:117)
    at Object.resolve (path.js:139)
    at WatcherHandler.onWatcherChange (watcher_handler.js?a049:197)
    at FSWatcher.emit (events.js:310)
    at FSEvent.FSWatcher._handle.onchange (internal/fs/watchers.js:135)
NodeError @ internal/errors.js:254
validateString @ internal/validators.js:117
resolve @ path.js:139
onWatcherChange @ watcher_handler.js?a049:197
emit @ events.js:310
FSWatcher._handle.onchange @ internal/fs/watchers.js:135
01:56:28.412 App

I'm not sure how to catch that error. I tried adding watcher.on ( 'error', error => {}) and wrapping the whole function with a try / catch block. Neither catches the error.

Code

I init this watcher inside of a web worker in an Electron app.

dirWatcherWorker.js

import Watcher from 'watcher'
let watcher = null
let dirPath = 'C:/'

self.addEventListener('message', () => {
  initDirWatch()
})

function initDirWatch () {
  // Reset watcher
  try { watcher.close() } catch (error) {}

  // Init watcher
  watcher = new Watcher(dirPath, { 
    ignoreInitial: true, 
    depth: 5,
    recursive: true
  })

  // Init listeners
  watcher.on('all', (event, targetPath, targetPathNext) => {
    console.log('Watcher:', event, targetPath, targetPathNext)
  })
  watcher.on('error', (error) => {
    console.log('Watcher:', error)
  })
}

Environment

OS: x64, Windows 10 20H2 build 19042.662
Exec env: web worker in an Electron app v10.1.3 (Chrome 85 + Node v12.16)

Providing readdirMap only works if it has a depth of 1

Use case

I want to use tiny-readdir to build my directory map and pass it to Watcher manually using the readdirMap option.

Example

import readdir from 'tiny-readdir'
import Watcher from 'watcher'

const ignoreRegExp = /(^|[\/\\])\.(?!local\/share\/)+|node_modules/
const pathToWatch = '/var/home/aral/sandbox/domain-fs-test'

const readdirMap = await readdir (pathToWatch, {
  depth: 20,
  limit: 1_000_000,
  followSymlinks: false,
  ignore: targetPath => ignoreRegExp.exec(targetPath.replace(pathToWatch, '')) !== null
})

const watcher = new Watcher(pathToWatch, {readdirMap})

What should happen

All files in readdirMap should be added to the watcher.

What actually happens

Only the files at depth 1 are added.

Recursive

If the watcher is instantiated with recursive: true:

const watcher = new Watcher(pathToWatch, {recursive: true, readdirMap})

Then readdirMap is ignored and all files (including the ones that were filtered out of readdirMap using tiny-readdir) are added to the watcher.

I believe the culprit is this line:

if ( readdirMap && depth === 1 && rootPath in readdirMap ) { // Reusing cached data

Workaround

At this point, the easiest thing to do is to use tiny-readdir to get the initial map and then instantiate watcher using the same ignore rules, etc., and with recursive: true and take the hit of building the directory/file map twice.

Support symlinks

Support both watching symlinks themselves and following them.

Support for awaitWriteFinish

From what I can read in the documentation, moving from Chokidar to Watcher doesn't have a equal to Chokidars
awaitWriteFinish: { stabilityThreshold: 2000, pollInterval: 100 },

With Watcher being an alternative to Chokidar, it seems to be a feature that would be desired to have. It's at least one I've been using to great success not having to implement custom logic to not have a an event called before a file is actually ready to be manipulated - and therefore is still locked by Windows.

If I'm missing something or there's any alternatives that I have missed please do tell 🙏
Outside of that Watcher has seemed good so far for rename event etc.

Fails when parent dir contains files/dirs that you don't have permission for

Watcher seems to try to stat all items in the parent directory, which is a problem if you don't have permission for some of those items, which is a common situation in /tmp. I ran into this problem trying to use Watcher in a Github Action, e.g.,

const root = await fs.mkdtemp(path.resolve(os.tmpdir(), "my-"))
const watcher = new Watcher(root)
// root may be, e.g., /tmp/my-2asdfy723r4

will complain about not having access to /tmp/snap.lxd (or presumably other files in /tmp/ that you don't have access to).

The workaround I did was to create another subdirectory and watch that, but presumably Watcher should do one of:

  1. Not try to stat anything but the direct ancestors of the path you're in (so /tmp and /tmp/my-* only) - not all children of parent
  2. Gracefully handle these kinds of errors
  3. Provide a way to turn off parent watching

Question: run once and stop watching?

I'd love to use this in my build script that has a --watch option, and it works great with that, but I'd also love to not have to write a separate code path to traverse the directory tree for non-watching mode. Is it possible to tell watcher to run once on all existing files, then exit?

Depth parameter is not working properly

Issue 1:

I found a few more issues with depth parameter values:

  1. When you set depth > 4 and recursive: true, no events will be emitted
  2. When you set depth <= 4 and recursive: true, events emitted properly, though it doesn't make much sense logically, since this setup emits events up to Infinity, even though you specify level 4 as the limit. Perhaps it should throw an error if recursive and depth are both used at the same time?
  3. When you set recursive: true without specifying depth, no events will be emitted
  4. When you set recursive: true and depth: Infinity, no events will be emitted

I'm not sure what's the intended behavior.

For the following setup:

{
  depth: 10,
  recursive: true
}

I'd expected the following behavior:

limits the amount of nested levels being watched to level 10 on Linux (for performance), while watching up to Infinity on Win and MacOS using the native watchers. Though, I don't really understand the purpose of the recursive option - if its only purpose is to switch to the native watcher, why not call it useNativeWatcher instead?

Issue 2:

Also there's a separate depth problem - when you init the watcher for a drive root path (e.g. "E:/") no events are emitted as well.

Bug: Recursive watcher with depth = 1 does not work under Linux if watched folder has subfolders

Description

A Watcher with recursive: true but depth: 1 does not work under Linux if the watched folder has subfolders. Note that the same configuration does work under Windows (not tested in macOS).

Reproduction

Open the following StackBlitz:
https://stackblitz.com/edit/stackblitz-starters-bvvkgg?file=index.js

In the terminal, run

node index.js

Expected output:

recursiveNestedWatcher addDir /home/projects/stackblitz-starters-bvvkgg/nestedFolder
recursiveNestedWatcher addDir /home/projects/stackblitz-starters-bvvkgg/nestedFolder/subfolder
recursiveNestedWatcher add /home/projects/stackblitz-starters-bvvkgg/nestedFolder/foo.txt
nonRecursiveNestedWatcher addDir /home/projects/stackblitz-starters-bvvkgg/nestedFolder
nonRecursiveNestedWatcher addDir /home/projects/stackblitz-starters-bvvkgg/nestedFolder/subfolder
nonRecursiveNestedWatcher add /home/projects/stackblitz-starters-bvvkgg/nestedFolder/foo.txt
flatWatcher addDir /home/projects/stackblitz-starters-bvvkgg/flatFolder
flatWatcher add /home/projects/stackblitz-starters-bvvkgg/flatFolder/hello.txt

Actual output:

nonRecursiveNestedWatcher addDir /home/projects/stackblitz-starters-bvvkgg/nestedFolder
nonRecursiveNestedWatcher addDir /home/projects/stackblitz-starters-bvvkgg/nestedFolder/subfolder
nonRecursiveNestedWatcher add /home/projects/stackblitz-starters-bvvkgg/nestedFolder/foo.txt
flatWatcher addDir /home/projects/stackblitz-starters-bvvkgg/flatFolder
flatWatcher add /home/projects/stackblitz-starters-bvvkgg/flatFolder/hello.txt

Executed code:

import Watcher from 'watcher'

const flatWatcher = new Watcher(
  'flatFolder',
  {
    recursive: true,
    depth: 1,
    limit: Infinity,
  },
  makeHandler('flatWatcher')
)

const recursiveNestedWatcher = new Watcher(
  'nestedFolder',
  {
    recursive: true,
    depth: 1,
    limit: Infinity,
  },
  makeHandler('recursiveNestedWatcher')
)

const nonRecursiveNestedWatcher = new Watcher(
  'nestedFolder',
  {
    recursive: false,
    depth: 1,
    limit: Infinity,
  },
  makeHandler('nonRecursiveNestedWatcher')
)

function makeHandler(name) {
  return function onFile(type, targetPath) {
    console.log(name, type, targetPath)
  }
}

Cannot use with Electron anymore

I was using watcher happily with Electron until a few months ago. Then an update of electron-forge came, stuff broke, and I stopped development of the app. Now I am back, but somehow only commonjs packages seem to work now. My own packages work with both commonjs and ESM, but watcher doesn't, and has all of these little dependencies that make it a pain to convert (what is "tsex"??).

It is the only external dependency in my project, but I think I will have to remove it now and write a watcher myself instead.

Change of file not detected on macOS

In the following situation file changes are not reported:

  • macOS Monterey 12.6.1, Intel
  • options: { recursive: true, native: false, renameDetection: false }
  • watch directory A which has a subdirectory B which contains file f. Rename B to C. Changes to f are not detected anymore.

When using native: true instead, everything works as expected. I haven't tested on other operating systems.

File renamed with capitalization not returning correct event

Using the example code I tried renaming a file from foo.mp3 to Foo.mp3 and got this:

change
E:\ggmusic\foo.mp3
undefined
add
E:\ggmusic\Foo.mp3
undefined

Renaming the file completely did work correctly though:

rename
E:\ggmusic\Run Free.mp3
E:\ggmusic\foo.mp3

Tests failing (Linux and macOS)

  1. Cloned the repository

  2. Run:

    npm install
    npm run clean
    npm run compile
    npm run test

Errors

Linux

(Tested on Fedora Silverblue 37 using Node.js 19.3.0 and 18.12.1)

> [email protected] test
> fava '**/test/index.js'

✖ Watcher › watching files › should watch new files inside an initially empty deep directory
╭ Error ─╮
watchers number
──
/var/home/aral/Projects/other/watcher/test/index.js:176
╰─ t.context.hasWatchObjects ( 0, 0, 3 );
╰─╯
✖ Watcher › watching files › should watch (touched) new files inside an initially empty deep directory
╭ Error ──╮
watchers number
───
/var/home/aral/Projects/other/watcher/test/index.js:189
╰─ t.context.hasWatchObjects ( 0, 0, 3 );

(Hangs at end.)

macOS

(Tested on macOS Monterey with Node.js 19.3.0.)

> [email protected] test
> fava '**/test/index.js'

✖ Watcher › watching directories › should watch new directories inside a new deep directory (1100ms)
╭ Error ──╮
Expected "0" to be exactly "3"
───
/Users/aral/sandbox/watcher/test/index.js:364
╰─ t.context.deepEqualUnorderedChanges ( [newdir0, newdir1, newdir2] );
╰──╯

ℹ File: index.js
ℹ Tests: 183 passed, 1 failed

(Gets further to display test totals but also hangs; doesn’t exit.)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.