Giter Site home page Giter Site logo

tinkershack / fluffy Goto Github PK

View Code? Open in Web Editor NEW
32.0 6.0 9.0 115 KB

Fluffy watches, reports Linux on-disk filesystem events faithfully. Comes with a CLI framework/tool for convenience. The library, libfluffy, can be independently used in other projects.

Home Page: http://tinkershack.in

License: The Unlicense

C 97.24% Makefile 2.76%
inotify filesystem-events inotify-library notification-library watch linux freedom cli wip

fluffy's Introduction

In the spirit of true freedom, Fluffy is Unlicensed. Free as in do what ever pleases you sort of freedom and free beer as well! Fluffy believes that a piece of software is free(freedom?) only when it has severed ties with licenses, copyrights, intellectual property rights and other crap of similar kind. Attribution is a kind gesture, Fluffy appreciates it but doesn't fret if you fail to say "good dog!"

Fluffy reports on-disk filesystem events faithfully and celebrates craftsmanship.

Fluffy - A 'free' watchdog for Linux on-disk filesystems [WIP]

Fluffy is a convenient CLI tool, libfluffy is what you will find under the hood. libfluffy uses the inotify kernel subsystem.

There are challenges in using the native inotify library effectively; Michael Kerrisk provides a lucid description. Fluffy's magic cuts through all the challenges cleanly.

Why Fluffy?

There are other implementations of filesystem watchers already, why Fluffy?

  • Linux specific; Fluffy is not cross platform, but loyal to Linux.
    Linux has its tricks, *BSD has its, so does Solaris. It only makes sense to have separate implementations for each of it to fully utilize the exclusive features which aren't available across all platforms. If the features aren't very diverse, then it's only a matter of porting; though it may be painful it's relatively less painful than an all-in-one model. Besides, a platform specific library helps keep the code base clean and lean. It's simpler. If you are thinking POSIX, please think again why many of the popular operating systems aren't POSIX/SUS compliant.

  • Reports events faithfully.
    There are popular libraries/tools built already that do a poor job at reporting the events - wrong event path, erroneous event handling, erroneous event reporting, oblivious to filesystem structure/hierarchy changes(dir moves especially), can't add watches on the fly without re-executing(reinitiating) the program(library). After considering various aspects of the process, building from scratch seemed better than fixing the broken ones.

  • Add/remove watch paths on the fly.

  • A fully functional library that utilizes the native inotify kernel subsystem properly. This means, unlike few tools, the events are not limited to just file 'modifications'. Every possible event action like 'open', 'access', 'close', 'no write', 'delete', 'move' is caught. User has the flexibility(control?) to discard/process select events.

  • Freedom. NO GPL shit.
    Fluffy has three heads. It likes to flip the middle one to licensing.

Contents

libfluffy is a better choice if you are planning to use it in production. fluffy.h has the interface description and callables.
example.c provides a sample.

libfluffy code is fairly documented. fluffy.h would be good place to start the trail. From there, jump to the corresponding function definition in fluffy.c and follow the calls thereafter. Should you feel that it's a bit confusing, head to the example.c to get a sense of the flow.

API list

Please look at fluffy.h for description. This is just a list of function calls available.

Primary functions

int fluffy_init(int (*user_event_fn) (
    const struct fluffy_event_info *eventinfo,
    void *user_data), void *user_data);

int fluffy_add_watch_path(int fluffy_handle, const char *pathtoadd);

int fluffy_remove_watch_path(int fluffy_handle, const char *pathtoremove);

int fluffy_wait_until_done(int fluffy_handle);

int fluffy_no_wait(int fluffy_handle);

int fluffy_destroy(int fluffy_handle);

Helper functions

int fluffy_print_event(const struct fluffy_event_info *eventinfo,
    void *user_data);

int fluffy_reinitiate_context(int fluffy_handle);

int fluffy_reinitiate_all_contexts();

int fluffy_set_max_queued_events(const char *max_size);

int fluffy_set_max_user_instances(const char *max_size);

int fluffy_set_max_user_watches(const char *max_size);
/*
 * example.c
 */
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

#include <fluffy.h>

int
main(int argc, char *argv[])
{
	int ret = 0;
	int flhandle = 0;       /* Fluffy handle */
	int rc = EXIT_SUCCESS;	/* Return code */

	if (argc < 2) {
		fprintf(stderr, "Atleast a path required.\n");
		exit(EXIT_FAILURE);
	}

        /* /proc/sys/fs/inotify/max_user_watches */
	ret = fluffy_set_max_user_watches("524288");
	if (ret != 0) {
		fprintf(stderr, "fluffy_set_max_user_watches fail\n");
	}

        /* /proc/sys/fs/inotify/max_queued_events */
	ret = fluffy_set_max_queued_events("524288");
	if (ret != 0) {
		fprintf(stderr, "fluffy_set_max_queued_events fail\n");
	}

        /*
	 * Initiate fluffy and print events on the standard out with the help
	 * of libfluffy's print event helper function.
	 *
	 * Look at fluffy.h for help with plugfing your custom event handler
	 */
	flhandle = fluffy_init(fluffy_print_event, (void *)&flhandle);
	if (flhandle < 1) {
		fprintf(stderr, "fluffy_init fail\n");
		exit(EXIT_FAILURE);
	}

	do {
		/* Watch argv[1] */
		ret = fluffy_add_watch_path(flhandle, argv[1]);
		if (ret != 0) {
			fprintf(stderr, "fluffy_add_watch_path %s fail\n", argv[1]);
			rc = EXIT_FAILURE;
			break;
		}

		/* Let us not exit until Fluffy exits/returns */
		ret = fluffy_wait_until_done(flhandle);
		if (ret != 0) {
			fprintf(stderr, "fluffy_wait_until_done fail\n");
			rc = EXIT_FAILURE;
			break;
		}
	} while(0);

	if (rc != EXIT_SUCCESS) {
		ret = fluffy_destroy(flhandle);	/* Destroy Fluffy context */
	}

	exit(rc);
}

The above code block is available as example.c in ./fluffy/libfluffy/, run make example from within the libfluffy directory to compile the example program.

Until the documentation is completed, these Stack Overflow answers may be of some reference.

Inotify: Odd behavior with directory creations

Inotify linux watching subdirectories

How to use inotifywait to watch files within folder instead of folder

  • Linux kernel > 2.6

  • gcc

  • pkg-config

  • make

  • glib-2.0

  • glib-2.0 devel package (glib2-devel for CentOS or RPM based distros, libglib2.0-dev for Debian)

# Fork and clone this repo
# Ensure glib-2.0 has been installed
# cd to fluffy dir
make            # `make clean` to cleanup

# If you see errors, please consider creating an issue here.
# No erros? Cool, let's proceed.
make install    # `make uninstall` to uninstall

# The framework, along with the library, has been installed.

# Run help
fluffyctl --help-all

fluffy --help

# Try
# Execute fluffy first
fluffy

# Open a new terminal to execute fluffyctl
# Let's watch /var/log & /tmp
fluffyctl -w /var/log -w /tmp

# You must see some action at the terminal where fluffy is run.
# Nothing yet?
ls -l /var/log  # Still nothing? We may have a problem!

# Let's ignore /tmp, not interested watching anymore.
fluffyctl -I /tmp

# More? Let's quit fluffy so that you can start over & explore.
fluffy exit

# If you are interested only in the library, you can choose to compile 
# just the library.
cd ./libfluffy
make

# There will now be a libfluffy.a archive file(static library), you can
# link against it in your projects.

# example.c shows a simple usage of the library
make example
./fluffy-example

# Modify example.c to play around. Run `make example` when you wish
# to compile the modified example.c for testing.

fluffy usage

root@six-k:~# fluffy -h
Usage:
  fluffy [OPTION...] [exit]

Help Options:
  -h, --help                     Show help options

Application Options:
  -O, --outfile=./out.fluffy     File to print output [default:stdout] Ensure that it's NOT located within the watch path to prevent a recurring feedback loop!
  -E, --errfile=./err.fluffy     File to print errors [default:stderr] Ensure that it's NOT located within the watch path to prevent a recurring feedback loop!

fluffyctl usage

root@six-k:~# fluffyctl --help-all
Usage:
  fluffyctl [OPTION...] ["/path/to/hogwarts/kitchen"]

'fluffyctl' controls 'fluffy' program.
fluffy must be invoked before adding/removing watches.  By default all 
file system events are watched and reported. --help-events will show the 
options to modify this behaviour

Help Options:
  -h, --help                                      Show help options
  --help-all                                      Show all help options
  --help-events                                   File system events to report

When an option or more from 'events' group is passed, only those events 
will be reported. When a new invocation of fluffyctl sets any 'events' 
option, previously set events choice is discarded; overrides.

  --all                                           Watch all possible events [default]
  --access                                        Watch file access
  --modify                                        Watch file modifications
  --attrib                                        Watch metadata change
  --close-write                                   Watch closing of file opened for writing
  --close-nowrite                                 Watch closing of file/dir not opened for writing
  --open                                          Watch opening of file/dir
  --moved-from                                    Watch renames/moves: reports old file/dir name
  --moved-to                                      Watch renames/moves: reports new file/dir name
  --create                                        Watch creation of files/dirs
  --delete                                        Watch deletion of files/dirs
  --root-delete                                   Watch root path deletions
  --root-move                                     Watch root path moves/renames
  --isdir                                         Watch for events that occur against a directory
  --unmount                                       Watch for unmount of the backing filesystem ['isdir' not raised]
  --queue-overflow                                Watch for event queue overflows ['isdir' not raised]
  --ignored                                       Watch for paths ignored by Fluffy(not watched) ['isdir' not raised]
  --root-ignored                                  Watch for root paths ignored(not watched) ['isdir' not raised]
  --watch-empty                                   Watch whether all Fluffy watches are removed ['isdir' not raised]

Application Options:
  -w, --watch=/grimmauld/place/12                 Paths to watch recursively. Repeat flag for multiple paths.
  -W, --watch-glob                                Paths to watch recursively: supports wildcards. Any non-option argument passed will be considered as paths. [/hogwarts/*/towers]
  -i, --ignore=/knockturn/alley/borgin/brukes     Paths to ignore recursively. Repeat flag for multiple paths.
  -I, --ignore-glob                               Paths to ignore recursively: supports wildcards. Any non-option argument passed will be considered as paths. [/hogwarts/*/dungeons]
  -U, --max-user-watches=524288                   Upper limit on the number of watches per uid [fluffy defaults 524288]
  -Q, --max-queued-events=524288                  Upper limit on the number of events [fluffy defaults 524288]
  -z, --reinit                                    Reinitiate watch on all root paths. [Avoid unless necessary]

Fluffy event log snippet

MODIFY, /var/log/daemon.log
MODIFY, /var/log/syslog
MODIFY, /var/log/kern.log
MODIFY, /var/log/messages
GNORED, /var/log/apache2
IGNORED,        /var/log/tor
IGNORED,        /var/log/vmware
CREATE,ISDIR,   /tmp/test
ACCESS,ISDIR,   /tmp/test
ACCESS,ISDIR,   /tmp/test
CLOSE_NOWRITE,ISDIR,    /tmp/test
CREATE, /tmp/test/f1
OPEN,   /tmp/test/f1
ATTRIB, /tmp/test/f1
CLOSE_WRITE,    /tmp/test/f1
OPEN,   /tmp/test/f1
MODIFY, /tmp/test/f1
MODIFY, /tmp/test/f1
MODIFY, /tmp/test/f1
CLOSE_WRITE,    /tmp/test/f1
IGNORED,ROOT_IGNORED,WATCH_EMPTY,       /tmp

Perform actions on events from CLI

CAUTION It's recommended that you use libfluffy to perform sophisticated actions on events rather than scripting with CLI usage.

Let's consider a trivial action: ls -l the path on a MODIFY event

At terminal:1

root@six-k:/home/lab/fluffy# fluffy | \
while read events path; do \
    if echo $events | grep -qie "MODIFY"; then \
        ls -l $path; \
    fi \
done

At terminal:2

root@six-k:/opt/test2# fluffyctl -w ./
root@six-k:/opt/test2# touch f1
root@six-k:/opt/test2# ls -l
total 0
-rw-r--r-- 1 root root 0 Mar 18 19:38 f1
root@six-k:/opt/test2# echo "this is a MODIFY" | cat >> f1
root@six-k:/opt/test2# echo "this is another MODIFY" | cat >> f1
root@six-k:/opt/test2# fluffy exit

Output from terminal:1: [cont.]

root@six-k:/home/lab/fluffy# fluffy | \
> while read events path; do \
>     if echo $events | grep -qie "MODIFY"; then \
>         ls -l $path; \
>     fi \
> done
-rw-r--r-- 1 root root 17 Mar 18 19:38 /opt/test2/f1
-rw-r--r-- 1 root root 40 Mar 18 19:38 /opt/test2/f1
root@six-k:/home/lab/fluffy# 

There's still quite a few more to be done but these are the primary ones

  • Documentation WIP

  • Proper error reporting. For now, most error returns have been set -1 deliberately without any error string or value.

  • Valgrind

  • Test cases

  • Other helper functions

    • Destroy all contexts
    • Emit internal events(eg. watches set, file info) for analytics
    • Get the list of root watch paths
  • Option to terminate context thread when watch list becomes empty

  • Ability to modify callback function pointer

fluffy's People

Contributors

raamsri avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

fluffy's Issues

FAIL every time I run script

Aug 13 14:06:47 srsbsns monitorStaticFiles.sh[27121]: fluffy_set_max_user_watches fail
Aug 13 14:06:47 srsbsns monitorStaticFiles.sh[27121]: fluffy_set_max_queued_events fail

do I need to run with sudo?!

fluffy is watching at its own event logging files

fluffy is the only software that I know that can handle recursive Inotfiy file system event watches properly.
Well done!
Well, something is a little bit odd when writing STDOUT and STDERR to output files.

Observed behaviour

fluffly is always watching implicitly on its own logfiles always leaving some unwanted traces in the files.
Even if you try to to exclude the path of the output files form the file watches as fast a possible.

Assuming this tiny script to run fluffy:

#!/bin/bash

sudo fluffy -O /opt/fluffy/fluffy.log -E /opt/fluffy/fluffy.log &

sudo fluffyctl -w /opt -i /opt/fluffy 

sudo touch /opt/ready

Result

This results in the following output file: /opt/fluffy/fluffy.log

MODIFY, /opt/fluffy/fluffy.log
ACCESS,ISDIR,   /opt
MODIFY, /opt/fluffy/fluffy.log
CLOSE_NOWRITE,ISDIR,    /opt
MODIFY, /opt/fluffy/fluffy.log
IGNORED,        /opt/fluffy
CREATE, /opt/ready
OPEN,   /opt/ready
ATTRIB, /opt/ready
CLOSE_WRITE,    /opt/ready

Touching the file /opt/ready is done when the script above is done.

Expected behavior

fluffy should ignore file system events for its own output files at all.
For my understanding it does not make any sense to recognize them.

The whole situation also gets worse if one forgets to exclude the path where the output files are written from watching.
Then fluffy ends in some kind of infinite loop because accessing and writing to the files creates events which are logged to file and so on.

mq_open: No such file or directory

sudo fluffyctl -w ./
mq_open: No such file or directory

cat /etc/oracle-release
Oracle Linux Server release 7.9

uname -a
Linux phoenix122420 5.4.17-2136.315.3.el7uek.x86_64 #2 SMP Thu Dec 8 14:46:08 PST 2022 x86_64 x86_64 x86_64 GNU/Linux

make
make -C ./libfluffy/
make[1]: Entering directory 'scratch/dfelicia/fluffy-master/libfluffy'
gcc -g -I./ -Wall -Wextra -Wno-unused-parameter -Wno-format-extra-args -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -c -o fluffy.o fluffy.c
fluffy.c: In function ‘fluffy_remove_watch_path’:
fluffy.c:2273:2: warning: passing argument 2 of ‘fluffy_handle_removal’ discards ‘const’ qualifier from pointer target type [enabled by default]
reterr = fluffy_handle_removal(fluffy_handle, pathtoremove);
^
fluffy.c:1494:1: note: expected ‘char *’ but argument is of type ‘const char *’
fluffy_handle_removal(int fluffy_handle, char *removethis)
^
ar crs libfluffy.a fluffy.o
make[1]: Leaving directory `/scratch/dfelicia/fluffy-master/libfluffy'
mkdir -p ./bin
gcc -g -I./ -I./libfluffy -L./libfluffy -Wall -Wextra -Wno-unused-parameter -Wno-format-extra-args -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -c -o src/fluffy_impl.o src/fluffy_impl.c
gcc -g -I./ -I./libfluffy -L./libfluffy -Wall -Wextra -Wno-unused-parameter -Wno-format-extra-args -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -c -o src/fluffy_run.o src/fluffy_run.c
gcc -g -I./ -I./libfluffy -L./libfluffy -Wall -Wextra -Wno-unused-parameter -Wno-format-extra-args -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include src/fluffy_impl.o src/fluffy_run.o -lrt -lfluffy -lglib-2.0 -o bin/fluffy
gcc -g -I./ -I./libfluffy -L./libfluffy -Wall -Wextra -Wno-unused-parameter -Wno-format-extra-args -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -c -o src/fluffy_ctl.o src/fluffy_ctl.c
gcc -g -I./ -I./libfluffy -L./libfluffy -Wall -Wextra -Wno-unused-parameter -Wno-format-extra-args -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include src/fluffy_impl.o src/fluffy_ctl.o -lrt -lfluffy -lglib-2.0 -o bin/fluffyctl

Thank you good sir

..."Reports events faithfully.
There are popular libraries/tools built already that do a poor job at reporting the events..."

Lost a good amount of time (and almost my...hope),
fiddling around with this or that or the other - no names needed.
What the heck...

No real 'issue' here (hence closing it asap),
but after the frustration, for whatever it's worth it,
i just felt the need of saying 'thank you': simple in it's usage,
clear & end-user friendly in it's displayed results, it simply...delivers. 👍 ✌️

Queue overflow, reinitiating all watches

I'm using fluffy to watch my development directory. I then have a bash script move the newly created minified files to a server dir, where I have another different fluffy watcher compresses the files using zopfli and readies them for serving. The problem I am having is with the development directory. Whenever I start the fluffy watcher daemon I get this error,

Queue overflow, reinitiating all watches!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.