How to ensure the daemon starts, stays running, and watches the right Git repos

[tkellogg]

How to ensure the daemon starts, stays running, and watches the right Git repos

[bj0]

why don't you just install a systemd service?

[Nevin1901]

I have created a very simple vscode extension which prompts the user to both autostart dura, and also asks the user if they want to watch whatever workspace they're in. You can see the repository here. This is just a proof of concept (want to add many features in the future), so I'm curious as to what you guys think of it.

The extension is not yet on the vscode marketplace, but I can see what to do with it if everyone likes the idea.

As of the license, I can also change that. I wasn't sure which license to use, so I just used Apache 2.0 (the same one as in this repository).

Let me know what you guys think of the extension, and if we should keep it.

[rjc]

Hello,

It would be nice to see dura being a true daemon like say, ssh-agent, etc. and not simply runs in the background via Unix shell's &. Also, once started with &, the shell needs to remain open - otherwise, if dura responds to SIGHUP the process will get killed.

As to which repositories to monitor - this can be either as simple as find . -type d -name .git -prune or a static list, like myrepos uses.

When it comes to a service, I'd simply leave it up to an individual. Like I've mentioned earlier, something like ssh-agent - or any *-agent for that matter - does its job just fine when started from a user's login shell - no service necessary.

[tkellogg]

Systemd is a good idea. I neglected to mention it because systemd isn't cross platform; I'm looking to support this project with low effort. But I'm not opposed to the idea either.

[nalzok]

Does this work on my OS?

• Mac: yes
• Linux: probably
• Windows: possibly

If macOS have first-class support, then you should definitely look into launchd(8). You can use RunAtLoad to ensure the daemon starts, KeepAlive to ensure it stays running, and WatchPaths to watch the right Git repos. See launchd.plist(5) for more details.

[goodevilgenius]

I think this really should depend on the system. Most operating systems have some daemon management service.

It wouldn't be too difficult to include a systemd service file and a launchd plist file, and then accept contributions for other systems.

[mihai-stancu]

@rjc the SIGHUP can be mitigated by using nohup dura serve &

[mihai-stancu]

Autoscan proposal:

Declare a configuration flag intended to activate autoscan.
Finding all directories named .git.
Finding the .git/config file & parsing it.
If the config contains a [dura] section with an enabled = true flag, then start watching it.

[tkellogg]

@pkoch I created #25 for issues around the find | xargs approach to discovery. I like the approach, but it needs some work beyond the README.

[jhuntwork]

I agree with the comments about avoiding shell tricks to launch a service in the background and if you really want a daemon, to add in proper support.

However, on most systems if you want the service launched automatically and managed it's actually a lot less fragile to supervise a child process running in the foreground. Correctly keeping track of the running state of another, non-child, process is problematic. For best results, this should really be in the realm of supervisors, i.e., projects like supervisord or s6, or even systemd.

I would suggest that you continue to design dura primarily to launch in the foreground and log to stdout/stderr and possibly provide sample configurations to run under various supervision suites according to platform. This would make it the most flexible, even allowing nice functioning in a container, if there is a good use case for that, and leaves the issue of managing a stable long-running process to tools that already excel at that. It even lets you hook in standard log collection utilities that those supervisors may support.

On a side note, I also agree with @rjc's request to use sh over bash in your code/examples. sh is a standard guaranteed to be present, bash is not. And while bash does exist in so many places that it feels universal, it really isn't. There are many systems I manage where it is absent.

[tkellogg]

@Nevin1901 Yeah, let's see a prototype. Make your own repository and if it seems good I'll make a dura organization and add it to it.

[ChampokDas]

I haven't taken a look at any of the internals of the project at all.

I think making dura a service to be handled by a manager would be best to achieve "always running, invisible" state with minimal input from the user, can depend on it being alive or if it's unavailable, just restart it. Then you have a standard interface for plugins in a editor/IDE + terminal users.

Since I'm mainly a terminal user, I'd be completely satisfied with just having server/client model similar to how things are done with tmux, initial startup of the dura server and then further calls will be clients to it.

[tekacs]

For the serving aspect, if targeting Mac, Homebrew can take care of producing and managing a plist (and auto start) for launchd:

https://docs.brew.sh/Formula-Cookbook#launchd-plist-files

or

https://rubydoc.brew.sh/Formula#service-class_method

Homebrew makes it almost trivial (just one line), if that’s a viable method for the project.

[flypenguin]

i understood that one dura process watches all of the configured git folders, correct?

service - systemd on linux is a good idea and most know how to use it. i am using a mac for years now and actually never had anything to do with launchd daemons ... . i guess the "perfect" mac way would be an addition to the system settings, like for example macfuse, gpgsuite or witch do. (yeah, GUI, shame on me, i know ;). since this is probably a LOT of effort, the launchd thing might just be the way to go. (actually, if that system settings GUI would be a thing, i'd pay a couple of bucks in the app store for it - open source MAY earn money).

adding - i really like dura add $DIR. perfect would be if $DIR is a git repo, add this. if $DIR contains git repos (maybe just depth 1), add those.

low key :)

[rjc]

@mihai-stancu

@rjc the SIGHUP can be mitigated by using nohup dura serve &.

Sure, my point was to making it a proper daemon and not resort to Unix shell tricks to keep it running in the background :^)

As to finding all Git repositories below a certain directory, I don't necessarilly think this is a good idea having had a think about it. What if the repositories in question aren't under $HOME? What if one's using a large group NFS share? Again, a list similar to .mrconfig should suffice.

[mihai-stancu]

@rjc agreed, reinventing the concept of a system service by using various neckbeardery defeats the purpose of low maintenance.

Sure, the "autoscan" proposal can hit snags with large directories and slow filesystems, but it can be complementary to the list of directories in order to allow near zero configuration.

[rjc]

Just to disambiguate as there are two concepts being mixed up - daemon and a system service - they're not the same thing. dura should most definitely be/have the former - I think it should not be a system service as that would imply it starts monitoring Git repositories for everyone omn the system. Yes, I'm aware that (at least) systemd and launchd have a concept of user services but, apart from Linux and macOS, most systems do not. I've been banging about ssh-agent as this is the way I see a user daemon - it starts as soon as the user logs on (why would you need it to start earlier?), then either it just runs, or one can communicate with it via a socket, etc.

[pkoch]

Let's maybe add a note to the README, or the top post of this issue, for a command that scans people's homedir?

Something like find ~ -type d -name .git -prune | xargs -I= bash -c "cd =/..; dura watch"

[rjc]

For those who had missed the link I posted earlier, please have a look at the related software section on myrepos page - if not direct solutions, I'm sure you'll be able to find some interesting ideas there.

Also, I see a lot of bash in the text/examples - any chance we could agree on sh, as in POSIX shell?

[tkellogg]

@mihai-stancu I'd like dura to default to watching as much as possible. It seems like there's a lot more potential frustration if you find out after-the-fact that dura wasn't watching a repo. There's enough holes in this proposal that I'll settle for @pkoch's plan, a la #25

@rjc Correct, this is certainly a per-user daemon, not a system service. As currently implemented, it uses ~/.config/dura/config.json as a locking mechanism to ensure a single process is running (see #2). We could change that to be a viable system-level service, but I initially avoided it to cut down on corner cases around file permissions. Let's keep options open for now — focus on better installation instructions in the README and/or installers.

[tkellogg]

@jhuntwork I created #30 to print to stdout (unfortunately I worked away from that pattern last night).

What would be a good resolution to this discussion? Wiki pages describing how to setup systemd, etc? Installation scripts?

[Nevin1901]

Hi, I propose an idea of developing various IDE extensions which automatically run dura in the background. This would mean that dura would be watching files as soon as the user opens their IDE, and the user could have a better UX while using dura (imagine clicking and seeing the changes, vs going through the dura branches via the command line).

I can create a really basic vscode extension to try and better represent what I mean.

Let me know what you guys think. I'd imagine going this route would mean that the dura project would have to be turned into a GitHub organization, in order to better keep track of all the repositories.

This idea would also make dura easier to run on various systems, as you would not have to worry about system specific things, such as systemd. Instead, you could just let the IDE extension launch the executable.

[jhuntwork]

@jhuntwork I created #30 to print to stdout (unfortunately I worked away from that pattern last night).

What would be a good resolution to this discussion? Wiki pages describing how to setup systemd, etc? Installation scripts?

I think that depends on what you intend to provide and support. For example, with Linux do you expect you will create official packages for Ubuntu? If so then you'll probably want to include with the package actual service definitions for systemd. And then the question becomes, how far down that path do you want to go? Do you want to support other Linux distros, perhaps ones that don't use systemd? What about for Mac, or Windows?

For me, if the behavior of a tool is clear through its help messages (and other documents) then that is good enough to implement a service on my system, and probably for others as well. A simple example in the documentation about how to run it on a platform-independent supervisor would probably be a good enough start.

It seems like @Nevin1901 already has a good idea about managing this through a vscode extension - and while that doesn't cover every user of git, it's probably a great setup for a lot of people. Kind of proves the point too about having the core dura code be simple and run in the foreground and let something else manage it how it will.

[fritzr]

+1 for @rjc 's human-read+writable config file suggestion over a json config. This would make it super easy for users to maintain an explicit list of git repos to watch, and if an "autoscan" feature is implemented it would make it easy for users to control exactly what is scanned or whether to scan at all. Like @rjc says if $HOME is NFS-mounted, or especially if cross-fs links are followed, an automatic blanket find with no depth limit in the user's home directory every five seconds could be catastrophic. It may make sense as a default on install for some users but it would be silly not to have an easy way for users to disable/control it.

Regarding reliable startup, I don't think it's unreasonable to leave that as an exercise for the user with suitable suggestions for popular OSes, sort of like how you have it setup already. Like tmux, it wouldn't hurt if every command that isn't start implicitly runs start first. Also like tmux, and also as @rjc suggested, I vote for properly daemonizing the server process rather than relying on shell jobs.

[abrooks]

On Linux, inotify (a kernel feature with libraries in most languages, here's one for rust) will avoid polling and will efficiently generate events even for large directory trees.

[rjc]

@abrooks This has already been mentioned, with notify being better (cross-platform) library.

[JakeStanger]

Autoscan proposal:

Declare a configuration flag intended to activate autoscan. Finding all directories named .git. Finding the .git/config file & parsing it. If the config contains a [dura] section with an enabled = true flag, then start watching it.

Autoscan + systemd service feels like the best way to me personally, although not necessarily as above. Perhaps a ~/.config/dura/config.toml or whatever format, where you declare directories to recursively search:

directories = ["/home/jake/Programming", "/home/jake/aur"] # lots of git repos live in these places
max_depth = 3 # avoid scanning every directory on the system like node_modules or whatever

Perhaps directories could be opt-out either by having an ignored_directories key in the config, or with a .duraignore file?

[tkellogg]

@JakeStanger what do you think about include and exclude fields within directories?

@Nevin1901 thats great! I'd love to get feedback from other VS Code users

[JakeStanger]

what do you think about include and exclude fields within directories?

Yeah that could work, ideally I think the short-hand string version above should default to "include all", and then perhaps in situations where you need more control you could define paths (or globs?) to include/exclude.

Something like:

[[directories]]
path = "/home/jake/Programming"
exclude = ["Work"] # relative to path

[[directories]]
path = "/home/jake/aur"
include = ["my-aur-package", "modified-aur-package"]

Not sure if TOML is the best language of choice (not a huge fan myself) but it feels the most rust-y

[tkellogg]

@JakeStanger That works for me. Are you going to make an attempt at it?

[JakeStanger]

I'll try and have a crack tomorrow, will report back with progress

[richard-hajek]

My current setup is invisible and works as following:

dura.service started on user login ( can be auto installed )

[Unit]
Description=

[Service]
ExecStart=/usr/bin/dura serve
Restart=always
RestartSec=3

[Install]
WantedBy=default.target

And cd hook in my .bashrc, Can not be auto installed :(

function cd () {
	
	if [[ -d "$1" ]] && [[ -d "$1/.git" ]]; then
		(
			builtin cd "$1"
			dura watch > /dev/null &
		)
	fi

	builtin cd "$@" && ls --color=auto --group-directories-first
}

Captures 99% use cases

[richard-hajek]

I just got the best idea - Create an hourly systemd timer, and use it to search for new repos, using the locate command

[benriemer]

Hi, cool project. Do you know the watchman project from facebook?

You can create a unix domain socket for communication with the main service and every client who is monitorin a new filepath.

Maybe it helps https://facebook.github.io/watchman/docs/socket-interface.html

[tkellogg]

Thanks! I have heard of watchman, but never looked into it. On my mac, I setup launchd and that works very well, although it's not cross platform.

[tubbo]

If I were setting this up and I wanted to run dura watch . every time I cd-ed into a Git repo, I'd probably use the chpwd() hook:

# zsh
chpwd() {
  if [[ -d $PWD/.git ]]; then dura watch; fi
}

# bash
cd() {
  builtin cd "$@" || return
  if [[ -d $PWD/.git ]]; then dura watch; fi
}

So perhaps a subcommand like dura extend-shell could be run in the .zshrc or something, which would output the path to the script that would do one of the above...

# .zshrc

source "$(dura extend-shell [zsh|bash])"

There's probably a better interface for this, and you might even be able to do it as part of Homebrew's shell completion step when it installs autocompletion for whatever shell you're on.

[tkellogg]

I love this suggestion! Having a subcommand for it is a good idea too