leahneukirchen/nq

Unix command line queue utility

nq: queue utilities

These small utilities allow creating very lightweight job queue systems which require no setup, maintenance, supervision, or any long-running processes.

nq should run on any POSIX.1-2008 compliant system which also provides a working flock(2). Tested on Linux 2.6.37, Linux 4.1, OpenBSD 5.7, FreeBSD 10.1, NetBSD 7.0.2, Mac OS X 10.3 and SmartOS joyent_20160304T005100Z.

The intended purpose is ad-hoc queuing of command lines (e.g., for building several targets of a Makefile, downloading multiple files one at a time, running benchmarks in several configurations, or simply as a glorified nohup). But as any good Unix tool, it can be abused for whatever you like.

Job order is enforced by a timestamp nq gets immediately when started. Synchronization happens on file-system level. Timer resolution is milliseconds. No sub-second file system time stamps are required. Polling is not used. Exclusive execution is maintained strictly.

Enforcing job order works like this:

• every job has a flock(2)ed output file, ala ,TIMESTAMP.PID
• every job starts only after all earlier flock(2)ed files are unlocked
• Why flock(2)? Because it locks the file handle, which is shared across exec(2) with the child process (the actual job), and it will unlock when the file is closed (usually when the job terminates).

You enqueue (get it?) new jobs using nq CMDLINE.... The job ID is output (unless suppressed using -q) and nq detaches immediately, running the job in the background. STDOUT and STDERR are redirected into the log file.

nq tries hard (but does not guarantee) to ensure the log file of the currently running job has +x bit set. Thus you can use ls -F to get a quick overview of the state of your queue.

The "file extension" of the log file is actually the PID, so you can kill jobs easily. Before the job is started, it is the PID of nq, so you can cancel a queued job by killing it as well.

Due to the initial exec line in the log files, you can resubmit a job by executing it as a shell command file (i.e. running sh $jobid).

You can wait for jobs to finish using nq -w, possibly listing job IDs you want to wait for; the default is all of them. Likewise, you can test if there are jobs which need to be waited upon using -t.

By default, job IDs are per-directory, but you can set $NQDIR to put them elsewhere. Creating nq wrappers setting $NQDIR to provide different queues for different purposes is encouraged.

All these operations take worst-case quadratic time in the amount of lock files produced, so you should clean them regularly.

Examples

Build targets clean, depends, all, without occupying the terminal:

% nq make clean
% nq make depends
% nq make all
% fq
... look at output, can interrupt with C-c any time
without stopping the build ...

Simple download queue, accessible from multiple terminals:

% mkdir -p /tmp/downloads
% alias qget='NQDIR=/tmp/downloads nq wget'
% alias qwait='NQDIR=/tmp/downloads fq -q'
window1% qget http://mymirror/big1.iso
window2% qget http://mymirror/big2.iso
window3% qget http://mymirror/big3.iso
% qwait
... wait for all downloads to finish ...

As nohup replacement (The benchmark will run in background, every run gets a different output file, and the command line you ran is logged, too!):

% ssh remote
remote% nq ./run-benchmark
,14f6f3034f8.17035
remote% ^D
% ssh remote
remote% fq
... see output, fq exits when job finished ...

Assumptions

nq will only work correctly when:

• $NQDIR (respectively .) is writable.
• flock(2) works in $NQDIR (respectively .).
• gettimeofday behaves monotonic (using CLOCK_MONOTONIC would create confusing file names). Else job order can be confused and multiple tasks can run at once due to race conditions.
• No other programs put files matching ,* into $NQDIR (respectively .).

nq helpers

Two helper programs are provided:

fq outputs the log of the currently running jobs, exiting when the jobs are done. If no job is running, the output of the last job is shown. fq -a shows the output of all jobs, fq -q only shows one line per job. fq uses inotify on Linux and falls back to polling for size change else. (fq.sh is a similar tool, not quite as robust, implemented as shell-script calling tail.)

tq wraps nq and displays the fq output in a new tmux or screen window.

(A pure shell implementation of nq is provided as nq.sh. It needs flock from util-linux, and only has a timer resolution of 1s. Lock files from nq and nq.sh should not be mixed.)

Installation

Use make all to build, make install to install relative to PREFIX (/usr/local by default). The DESTDIR convention is respected. You can also just copy the binaries into your PATH.

You can use make check to run a simple test suite, if you have Perl's prove installed.

Comparison to at, batch, and task-spooler

• at runs jobs at a given time. batch runs jobs "when system load levels permit". nq and task-spooler run jobs in sequence with no regard to the system's load average.

• at and batch have 52 built-in queues: a-z and A-Z. Any directory can be a queue for nq. task-spooler can have different queues for different terminals.

• You can follow the output of an nq queue tail-style with fq.

• The syntax is different: at and batch take whole scripts from the standard input or a file; nq takes a single command as its command line arguments.

• nq doesn't rely on a daemon, and uses a directory to manage the queue. task-spooler automatically launches a daemon to manage a queue.

• task-spooler can set a maximum number of simultaneous jobs.

Copyright

nq is in the public domain.

To the extent possible under law, Leah Neukirchen [email protected] has waived all copyright and related or neighboring rights to this work.

http://creativecommons.org/publicdomain/zero/1.0/

nq's Issues

nq: When spawning many jobs, all nq processes stuck with high cpu load, no work done

We're using nq to process batch files in a neat, orderly fashion. For the most part, this works awesomely (thanks!). However, we sometimes get a burst of jobs to process, causing many dozen nq processes/jobs to be started (with the same $NQDIR), and at some point - I'm not certain when - work stops being done while all nq processes get stuck consuming significant CPU.

truss on a random one gives, with a few seconds interval, the following in repeated fashion:

flock(5,LOCK_EX)                                 = 0 (0x0)                                                                                       
close(5)                                         = 0 (0x0)                                                                                       
lseek(3,0x0,SEEK_SET)                            = 0 (0x0)                                                                                       
getdirentries(3,"\M^A\f\^C\0\0\0\0\0\^A\0\0\0\0\0"...,4096,{ 0x0 }) = 4088 (0xff8)                                                               
openat(3,",177d4c573b4.8614",O_RDWR,00)          = 5 (0x5)                                                                                       
flock(5,LOCK_EX|LOCK_NB)                         ERR#35 'Resource temporarily unavailable'                                                       

with the file it refers to being an earlier (completed) job in the same $NQDIR.

I'm not sure what to do to debug this in a meaningful fashion. I'm trying to come up with a reasonable reproduction scenario but so far it has eluded me. When this happens, we end up with a bunch of files in the $NQDIR without any content beyond the leading exec line.

Suggestions?

Fails to compile on Mac OSX 10.8.5

Make fails on osx. (works on ubuntu)

$ uname -a
Darwin varena 12.6.0 Darwin Kernel Version 12.6.0: Wed Mar 18 16:23:48 PDT 2015; root:xnu-2050.48.19~1/RELEASE_X86_64 x86_64

$ cc -v
Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn)
Target: x86_64-apple-darwin12.6.0
Thread model: posix

$ make
cc -g -Wall -O2 nq.c -o nq
nq.c:175:12: warning: implicit declaration of function 'openat' is invalid in C99
[-Wimplicit-function-declaration]
lockfd = openat(dirfd, lockfile, O_RDWR | O_APPEND);
^
nq.c:210:2: warning: implicit declaration of function 'renameat' is invalid in C99
[-Wimplicit-function-declaration]
renameat(dirfd, lockfile, dirfd, lockfile+1);
^
nq.c:249:9: warning: implicit declaration of function 'fdopendir' is invalid in C99
[-Wimplicit-function-declaration]
dir = fdopendir(dirfd);
^
nq.c:249:7: warning: incompatible integer to pointer conversion assigning to 'DIR _' from 'int'
[-Wint-conversion]
dir = fdopendir(dirfd);
^ ~~~~~~~~~~~~~~~~
4 warnings generated.
Undefined symbols for architecture x86_64:
"_fdopendir", referenced from:
_main in nq-179326.o
"_openat", referenced from:
_main in nq-179326.o
"_renameat", referenced from:
main in nq-179326.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: ** [nq] Error 1

Support a Priority Queue

Allow assigning an optional priority to jobs, using a simple algorithm for selecting jobs:

• Find the highest priority that has outstanding jobs to run
• Run the job that has been waiting the longest in this priority
• repeat

Given that the utility appears to be filesystem-based and has a shell implementation, I'd imagine a simple way to do this would be to change the NQDIR format to create sortable directories with the priority number; jobs then go into this with their timestamp. Using sort -n (and whatever equivalent for C) should allow sorting of negative directories, such that a default priority of zero indicates no preference. Negative numbers could indicate lower priority (probably more intuitive) or higher priority (matches nice/kernel)

I will try and fork and PR if I have time, but am adding this feature request in case I don't. If I do, I'll probably only be able to implement one version - either shell or C, if you'd be happy to reflect in the other if you want to add the feature at all.

nq.c: fix time_t overflow on 32-bit machines

In systems where time_t is defined as 32-bit integer, the multiplication
by 1000 to get millisecond will cause an integer overflow.

Debug log:

(gdb) b 87
Breakpoint 1 at 0x8048b4e: file nq.c, line 87.

...

Breakpoint 1, main (argc=1, argv=0xbffff954) at nq.c:87
87      ms = started.tv_sec*1000 + started.tv_usec/1000;
(gdb) n
89      while ((opt = getopt(argc, argv, "+htw")) != -1) {
(gdb) p ms
$1 = 445515412
(gdb) p started
$2 = {tv_sec = 1439259559, tv_usec = 572464}

Fix this problem by explicit integer conversion. Patch available:

From c6f0b7e91d99bec927b89ba4127e6a1b5ffa2d07 Mon Sep 17 00:00:00 2001
From: Lv Zheng <[email protected]>
Date: Tue, 11 Aug 2015 02:23:03 +0000
Subject: [PATCH] nq.c: fix time_t overflow on 32-bit machines

In systems where time_t is defined as 32-bit integer, the multiplication
by 1000 to get millisecond will cause an integer overflow.

Fix this problem by explicit integer conversion.

---
 nq.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/nq.c b/nq.c
index 86177f3..7401c2e 100644
--- a/nq.c
+++ b/nq.c
@@ -84,7 +84,7 @@ main(int argc, char *argv[])

    /* timestamp is milliseconds since epoch.  */
    gettimeofday(&started, NULL);
-   ms = started.tv_sec*1000 + started.tv_usec/1000;
+   ms = (int64_t)started.tv_sec*1000 + started.tv_usec/1000;

    while ((opt = getopt(argc, argv, "+htw")) != -1) {
        switch (opt) {
-- 
2.5.0

list all the queued task even none-executed ones?

This command $fq -qa only shows task that has been executed already.
Is there a way to list all task that has NOT been executed yet also?
Sometimes i add a lot to the queue but forgot which things i added already.

License hasn't kept up with contributors

nq/COPYING

Line 3 in 3b5ddeb

To the extent possible under law, Leah Neukirchen <[email protected]>

This will need to reference all contributors. I think collectively is fine, but IANAL.

Simultaneously running certain amount of jobs (i.e. "batching")

It seems nq is meant to serialize all single jobs enqueued in the given queue. How could one easily achieve utilization of all N cores in case there are N+ jobs to be enqued and all of them are CPU-intensive?

Note, I can't simply spawn N separate queues because each job will take different time to finish and thus I'll quickly end up with some cores still maxed out but others already finished with their queues.

add comparison to task-spooler

https://vicerveza.homeunix.net/~viric/soft/ts/ is very similar. It would be nice to have an overview of the differences against it.

Needs installation/setup instructions

The readme file is not clear on how to setup the utility and whether it's available via a package manager. If the user is asked to compile, then some basic directions and troubleshooting information would be helpful.

Thanks.

Completion of error handling

Would you like to add more error handling for return values from functions like the following?

• dup2 ⇒ main
• flock ⇒ islocked

[Feature request] Unbuffered output to the logfiles

OS: Debian 10 amd64
nq version: 0.3.1-1

Currently nq buffers the writing of queued command output to logfile, writing to it only once the buffer fills up / the task is done. When the queued task is not heavy with output, it can take a long time to see any output with fq, possibly even not seeing any output until the job is done.

Buffered writes to logfile make sense when the job in question produces a lot of output in a short timeframe, and is at the same time the preferable way of doing things performance-wise. However it is somewhat problematic for jobs that don't produce a lot of output.

It would be nice to have an option to make fq have unbuffered writes to the logfile.

support illumos with fcntl

I wrote a portable flock(1) a while back and I wanted it to work similarly on bsd, linux & illumos. I ended up reviewing different flock(2) emulation impls out there and came up with https://github.com/discoteq/flock/blob/master/src/flock.c#L26

Interested in me adding illumos support?

fq doesn't display command output with version 0.2

Tested on OS X 10.11.

With version 0.2:

bash-3.2$ nq touch TEST
,15bae181f3c.22760
bash-3.2$ nq echo hello
,15bae18353c.22763
bash-3.2$ nq echo world
,15bae183d01.22766
bash-3.2$ fq -a
==> ,15bae181f3c.22760 exec 'nq' 'touch' 'TEST'
==> ,15bae18353c.22763 exec 'nq' 'echo' 'hello'
==> ,15bae183d01.22766 exec 'nq' 'echo' 'world'
bash-3.2$

With version 0.1:

bash-3.2$ nq touch TEST
,15bae1a430f.23221
bash-3.2$ nq echo hello
,15bae1a520b.23224
bash-3.2$ nq echo world
,15bae1a6026.23227
bash-3.2$ fq -a
==> ,15bae1a430f.23221
exec 'nq' 'touch' 'TEST'


[exited with status 0.]
==> ,15bae1a520b.23224
exec 'nq' 'echo' 'hello'

hello

[exited with status 0.]
==> ,15bae1a6026.23227
exec 'nq' 'echo' 'world'

world

[exited with status 0.]

nq sudo ... does not work

hobbes@hubris:~$ fq
==> ,14f6c6fbfb5.3605
exec 'nq' 'sudo' 'apt-get' 'install' 'autoconf'

sudo: no tty present and no askpass program specified

[exited with status 1.]

Brainstorming some ideas for nq

Some half-cooked random ideas:

1. Have a "tail -f" mode for fq which starts following output close to end. Useful because sometimes I have single jobs with outputs as long as 500k lines over ssh. I'm unsure if it should be the default or not (I think I'd muscle memory to be my default). Note fq | tail -f won't work for this, although it just occurred to me that fq | less might be a good alternative.
2. Have an option for fq to follow just one process and stop. Maybe -1 (and then -2, -3, etc).
3. Have a way to list the current and future jobs. Similar to the way fq -nqa lists current an past jobs. Also a way to list all jobs past and future with a clear mark on the currently running job. This is orthogonal to -q and -a and alternative to -n so maybe an option -X (letter to be chosen) such that fq -Xq prints current and future jobs and fq -Xqa prints past and future jobs (think "-a" to mean past and "-X" to mean future). I use head -1 ,* a lot, but it prints 3 lines per job instead of one and is only good for past and future, not just current and future (BTW: fq -qan has a noticeable delay probably due to stuff that is not necessary when just printing the first line).
4. Have a clean way to interrupt a process; sending SIGTERM to the process doesn't seem to do "the right thing" in the sense that it's not equivalent to hitting control-C (e.g. sub-processes keep running). I may be wrong on this or there may be a trick I don't know.

`fq -a' doesn't pick up tasks added after it started

Such as when nq jobs are spawned in one terminal, then fq -a, then further jobs are added in that same directory in another terminal. This means `fq -a' terminates before all jobs in that directory have completed.

nq manpage incorrect

'man nq' says:
FILES
nq owns all files in NQDIR (respectively .) which start with “,” or “.,”.
but this does not seem to be true. Rather, the files are owned by the user who runs nq.

Flag to automatically delete lockfile?

It would be cool if there is a flag that deletes the lockfile automatically after the command finished with RC=0.

Can't compile on Mac OS X 10.9.5

nq sounds neat! Saw it on One Thing Well.

I don't have a need for this right now but I thought I'd report that it didn't build on my Mac OS X 10.9.5 machine (ie, Mavericks) using Apple's compiler. I think Mac's libc doesn't have fdopendir, openat, or renameat. If I were more serious about making this work, I'd see if Homebrew has whatever is missing.

$ cc -g -Wall nq.c -v
Apple LLVM version 6.0 (clang-600.0.54) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin13.4.0
Thread model: posix
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1 -triple x86_64-apple-macosx10.9.0 -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -main-file-name nq.c -mrelocation-model pic -pic-level 2 -mdisable-fp-elim -masm-verbose -munwind-tables -target-cpu core2 -target-linker-version 241.9 -v -gdwarf-2 -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0 -Wall -fdebug-compilation-dir /tmp/nq -ferror-limit 19 -fmessage-length 80 -stack-protector 1 -mstackrealign -fblocks -fobjc-runtime=macosx-10.9.0 -fencode-extended-block-signature -fdiagnostics-show-option -fcolor-diagnostics -vectorize-slp -o /var/folders/0j/nf7gs_917ss0cdldlygvnb600000gn/T/nq-77bc70.o -x c nq.c
clang -cc1 version 6.0 based upon LLVM 3.5svn default target x86_64-apple-darwin13.4.0
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
 /usr/include
 /System/Library/Frameworks (framework directory)
 /Library/Frameworks (framework directory)
End of search list.
nq.c:175:12: warning: implicit declaration of function 'openat' is invalid in
      C99 [-Wimplicit-function-declaration]
                lockfd = openat(dirfd, lockfile, O_RDWR | O_APPEND);
                         ^
nq.c:210:2: warning: implicit declaration of function 'renameat' is invalid in
      C99 [-Wimplicit-function-declaration]
        renameat(dirfd, lockfile, dirfd, lockfile+1);
        ^
nq.c:243:9: warning: implicit declaration of function 'fdopendir' is invalid in
      C99 [-Wimplicit-function-declaration]
                dir = fdopendir(dirfd);
                      ^
nq.c:243:7: warning: incompatible integer to pointer conversion assigning to
      'DIR *' from 'int' [-Wint-conversion]
                dir = fdopendir(dirfd);
                    ^ ~~~~~~~~~~~~~~~~
4 warnings generated.
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" -demangle -dynamic -arch x86_64 -macosx_version_min 10.9.0 -o a.out /var/folders/0j/nf7gs_917ss0cdldlygvnb600000gn/T/nq-77bc70.o -lSystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/lib/darwin/libclang_rt.osx.a
Undefined symbols for architecture x86_64:
  "_fdopendir", referenced from:
      _main in nq-77bc70.o
  "_openat", referenced from:
      _main in nq-77bc70.o
  "_renameat", referenced from:
      _main in nq-77bc70.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Ubuntu still has nq version 0.3

Not sure if this belongs into this issue tracker, but maybe consider this a heads-up.

One major difference between version 0.3 and the current 0.5 is that serializing of jobs in a queue, for a given NQDIR, works for each user individually in 0.3, whereas it works across all users in 0.5 .

So if you expect the latter (which is what batch queueing systems do normally), then the Ubuntu package will fail for you.

As an example, our use-case is that the software we use (called "AlphaFold") needs exclusive access to an NVidia GPU. There is only a single such GPU in our compute server, so we use a single queue (and a sngle NQDIR) to serialize all AlphaFold jobs from different users. However, with nq version 0.3 different users can concurrently run jobs, which leads to aborted jobs. With version 0.5, this works perfectly.

Not sure if there is a mechanism that tells the Ubuntu maintainers that they should update.

make check failed (could not reproduce)

The first run of make check on freshly compiled nq failed with the following output:

$ make check
prove -v ./tests
./tests .. 
1..36
# nq tests
ok - fails with no arguments
ok - succeeds enqueuing true
ok - generated a lockfile
ok - lockfile contains exec line
not ok - lockfile contains status line
Dubious, test returned 1 (wstat 256, 0x100)
Failed 32/36 subtests 

Test Summary Report
-------------------
./tests (Wstat: 256 Tests: 5 Failed: 1)
  Failed test:  5
  Non-zero exit status: 1
  Parse errors: Bad plan.  You planned 36 tests but ran 5.
Files=1, Tests=5,  1 wallclock secs ( 0.03 usr  0.01 sys +  0.00 cusr  0.01 csys =  0.05 CPU)
Result: FAIL
make: *** [check] Error 1

I was not able to reproduce this failure with the same build or the fresh one so this could probably be some race condition.
OS is Linux with 3.13.0-61-generic kernel.

fq Doesn't Appear to Work in WSL

I understand this may never work, but fq without any additional parameters does not output the currently running task or the last run task. fq -a appears to somewhat work.

This could be a limitation of WSL, but it would be nice if I could get fq working properly.

Shell-agnostic way to run nested aliases through `nq` (at least `bash` and `zsh`)?

What is the best practice way to have nq some-alias auto-resolve the alias, ideally even when some-alias is a nested alias?

And what if it is a function or a function inside an alias?

This seems to work:

alias nqs="nq "\$SHELL" -ic"

Way to go?

exec in logfile might fail.

The exec in the logfile might fail when the logfile is run, allowing the rest of the logfile to be executed. An explicit `exit', passing exec's failure on, could be added.

Add comparison to at(1) and batch(1)

First, thanks for writing this tool! It's an elegant design.

I think people who find it for the first time may wonder how it's different from POSIX at and batch. It could be useful to preempt their questions in the readme. Somebody asked on Hacker News, and I posted a five-point comparison based on my best understanding of both nq and at/batch. Feel free to use the text of that comment under your choice of license.

Homebrew or Homebrew Cask

Please make a Homebrew package or Homebrew cask for installing this using Homebrew on OSX.

Thanks!

fq blocks indefinitely when using inotify backend

I’ve noticed that fq does not actually follow command output on my machines. gdb shows that fq blocks indefinitely when trying to read from the inotify event FD. I can observe this when using Linux 4.15.5, glibc 2.26. I vaguely recall this used to work, but I’m not sure exactly when it regressed.

The blocking call is fq.c:174:

read(ifd, ibuf, sizeof ibuf);

This never returns, even when the MODIFY or CLOSE_WRITE events are triggered (as shown by observing the output file with inotifywait -m). I tried reading just 1 byte for a change, but this infblocks, too.

0.3 error: use of undeclared identifier 'LOCK_EX'

Error Message

failed: brew install --verbose --build-bottle nq

Stacktrace

        /usr/bin/sandbox-exec -f /tmp/homebrew20180307-15058-2u85ae.sb nice /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.3/bin/ruby -W0 -I /usr/local/Homebrew/Library/Homebrew -- /usr/local/Homebrew/Library/Homebrew/build.rb /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/nq.rb --build-bottle --verbose
==> Downloading https://github.com/chneukirchen/nq/archive/v0.3.tar.gz
Already downloaded: /Users/brew/Library/Caches/Homebrew/nq-0.3.tar.gz
==> Verifying nq-0.3.tar.gz checksum
tar xzf /Users/brew/Library/Caches/Homebrew/nq-0.3.tar.gz
==> make all PREFIX=/usr/local/Cellar/nq/0.3
clang -g -Wall -O2    nq.c   -o nq
clang -g -Wall -O2    fq.c   -o fq
nq.c:256:20: error: use of undeclared identifier 'LOCK_EX'
        if (flock(lockfd, LOCK_EX) < 0) {
                          ^
nq.c:290:18: error: use of undeclared identifier 'LOCK_EX'
                        if (flock(fd, LOCK_EX | LOCK_NB) == -1 &&
                                      ^
nq.c:290:28: error: use of undeclared identifier 'LOCK_NB'
                        if (flock(fd, LOCK_EX | LOCK_NB) == -1 &&
                                                ^
nq.c:294:15: error: use of undeclared identifier 'LOCK_EX'
                                flock(fd, LOCK_EX);   /* sit it out.  */
                                          ^
nq.c:319:19: error: use of undeclared identifier 'LOCK_EX'
                                if (flock(fd, LOCK_EX | LOCK_NB) == -1 &&
                                              ^
nq.c:319:29: error: use of undeclared identifier 'LOCK_NB'
                                if (flock(fd, LOCK_EX | LOCK_NB) == -1 &&
                                                        ^
nq.c:323:16: error: use of undeclared identifier 'LOCK_EX'
                                        flock(fd, LOCK_EX);   /* sit it out.  */
                                                  ^
7 errors generated.
make: *** [nq] Error 1
make: *** Waiting for unfinished jobs....

Homebrew/homebrew-core#24949

Addition of a build system generator

I suggest to reuse a higher level build system than your current small make file so that powerful checks for software features will become easier.

• CMake
• Autotools

nq does not serialize commands from different users

I installed nq on Ubuntu 20.04 . 'apt list nq' reports:
nq/focal,now 0.3.1-3 amd64 [installed]
I use nq to make sure that the GPU of a multi-user machine is accessed by only a single tensorflow (machine learning) process at a time. To this end, I created a world-writable directory /var/spool/nqdir and 'export NQDIR=/var/spool/nqdir' in /etc/bash.bashrc . Users are taught to start their jobs with 'nq command'.
This works fine as long as a user starts multiple jobs; these are indeed serialized. However today I saw that two processes, from different users, were running at the same time, and both crashed since the GPU memory is insufficient for both. Both users had used nq, and their logfiles were present in $NQDIR .
Is it a permission issue? Or is nq not supposed to work across several users who use the same NQDIR? How could that be solved? Using 'umask'? If this is a known problem it should be mentioned in the documentation; likewise for a workaround if it exists. Thanks!

compilation failure on MacOS X

:; make
cc -g -std=c89 -Wall -O2    nq.c   -o nq
nq.c:117:36: warning: format specifies type 'unsigned long' but the argument has type 'int64_t' (aka 'long long') [-Wformat]
                sprintf(lockfile, ".,%011lx.%d", ms, getpid());
                                     ~~~~~~      ^~
                                     %011llx
/usr/include/secure/_stdio.h:47:56: note: expanded from macro 'sprintf'
  __builtin___sprintf_chk (str, 0, __darwin_obsz(str), __VA_ARGS__)
                                                       ^
nq.c:151:35: warning: format specifies type 'unsigned long' but the argument has type 'int64_t' (aka 'long long') [-Wformat]
                sprintf(lockfile, ",%011lx.%d", ms, child);
                                    ~~~~~~      ^~
                                    %011llx
/usr/include/secure/_stdio.h:47:56: note: expanded from macro 'sprintf'
  __builtin___sprintf_chk (str, 0, __darwin_obsz(str), __VA_ARGS__)
                                                       ^
nq.c:182:35: warning: format specifies type 'unsigned long' but the argument has type 'int64_t' (aka 'long long') [-Wformat]
        sprintf(lockfile, ".,%011lx.%d", ms, getpid());
                             ~~~~~~      ^~
                             %011llx
/usr/include/secure/_stdio.h:47:56: note: expanded from macro 'sprintf'
  __builtin___sprintf_chk (str, 0, __darwin_obsz(str), __VA_ARGS__)
                                                       ^
nq.c:189:6: warning: implicit declaration of function 'flock' [-Wimplicit-function-declaration]
        if (flock(lockfd, LOCK_EX) < 0) {
            ^
nq.c:189:20: error: use of undeclared identifier 'LOCK_EX'
        if (flock(lockfd, LOCK_EX) < 0) {
                          ^
nq.c:214:18: error: use of undeclared identifier 'LOCK_EX'
                        if (flock(fd, LOCK_EX | LOCK_NB) == -1 &&
                                      ^
nq.c:214:28: error: use of undeclared identifier 'LOCK_NB'
                        if (flock(fd, LOCK_EX | LOCK_NB) == -1 &&
                                                ^
nq.c:218:15: error: use of undeclared identifier 'LOCK_EX'
                                flock(fd, LOCK_EX);   /* sit it out.  */
                                          ^
nq.c:243:19: error: use of undeclared identifier 'LOCK_EX'
                                if (flock(fd, LOCK_EX | LOCK_NB) == -1 &&
                                              ^
nq.c:243:29: error: use of undeclared identifier 'LOCK_NB'
                                if (flock(fd, LOCK_EX | LOCK_NB) == -1 &&
                                                        ^
nq.c:247:16: error: use of undeclared identifier 'LOCK_EX'
                                        flock(fd, LOCK_EX);   /* sit it out.  */
                                                  ^
4 warnings and 7 errors generated.
make: *** [nq] Error 1

Interestingly, LOCK_EX is defined if _POSIX_C_SOURCE is not because of #if !defined(_POSIX_C_SOURCE) || defined(_DARWIN_C_SOURCE) in sys/file.h.