{"id":13452955,"url":"https://github.com/leahneukirchen/nq","last_synced_at":"2025-05-14T19:09:31.685Z","repository":{"id":52285673,"uuid":"40008653","full_name":"leahneukirchen/nq","owner":"leahneukirchen","description":"Unix command line queue utility","archived":false,"fork":false,"pushed_at":"2024-07-03T16:09:12.000Z","size":87,"stargazers_count":2997,"open_issues_count":4,"forks_count":66,"subscribers_count":52,"default_branch":"master","last_synced_at":"2025-05-11T00:05:09.021Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/leahneukirchen.png","metadata":{"files":{"readme":"README.md","changelog":"NEWS.md","contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-07-31T14:17:30.000Z","updated_at":"2025-05-08T11:17:13.000Z","dependencies_parsed_at":"2024-11-19T14:03:46.052Z","dependency_job_id":"d7e9bf23-8286-4e90-b139-0320f38ec87e","html_url":"https://github.com/leahneukirchen/nq","commit_stats":{"total_commits":122,"total_committers":13,"mean_commits":9.384615384615385,"dds":0.3688524590163934,"last_synced_commit":"0360a0e538ac69d9719019233fb58550f9579261"},"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leahneukirchen%2Fnq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leahneukirchen%2Fnq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leahneukirchen%2Fnq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leahneukirchen%2Fnq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/leahneukirchen","download_url":"https://codeload.github.com/leahneukirchen/nq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253843217,"owners_count":21972874,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T08:00:29.226Z","updated_at":"2025-05-14T19:09:30.384Z","avatar_url":"https://github.com/leahneukirchen.png","language":"C","funding_links":[],"categories":["C","Process Manager","\u003ca name=\"core\"\u003e\u003c/a\u003ecore","others","C/C++程序设计"],"sub_categories":["Selfhosted","资源传输下载"],"readme":"## nq: queue utilities\n\nThese small utilities allow creating very lightweight job queue\nsystems which require no setup, maintenance, supervision, or any\nlong-running processes.\n\n`nq` should run on any POSIX.1-2008 compliant system which also\nprovides a working flock(2).  Tested on Linux 2.6.37, Linux 4.1,\nOpenBSD 5.7, FreeBSD 10.1, NetBSD 7.0.2, Mac OS X 10.3 and\nSmartOS joyent_20160304T005100Z.\n\nThe intended purpose is ad-hoc queuing of command lines (e.g., for\nbuilding several targets of a Makefile, downloading multiple files one\nat a time, running benchmarks in several configurations, or simply as\na glorified `nohup`). But as any good Unix tool, it can be abused for\nwhatever you like.\n\nJob order is enforced by a timestamp `nq` gets immediately when\nstarted.  Synchronization happens on file-system level.  Timer\nresolution is milliseconds.  No sub-second file system time stamps are\nrequired.  Polling is not used.  Exclusive execution is maintained\nstrictly.\n\nEnforcing job order works like this:\n- every job has a flock(2)ed output file, ala `,TIMESTAMP.PID`\n- every job starts only after all earlier flock(2)ed files are unlocked\n- Why flock(2)? Because it locks the file handle, which is shared\n  across exec(2) with the child process (the actual job), and it will\n  unlock when the file is closed (usually when the job terminates).\n\nYou enqueue (get it?) new jobs using `nq CMDLINE...`.  The job ID is\noutput (unless suppressed using `-q`) and `nq` detaches immediately,\nrunning the job in the background.  STDOUT and STDERR are redirected\ninto the log file.\n\n`nq` tries hard (but does not guarantee) to ensure the log file of the\ncurrently running job has `+x` bit set.  Thus you can use `ls -F` to get\na quick overview of the state of your queue.\n\nThe \"file extension\" of the log file is actually the PID, so you can\nkill jobs easily.  Before the job is started, it is the PID of `nq`,\nso you can cancel a queued job by killing it as well.\n\nDue to the initial `exec` line in the log files, you can resubmit a\njob by executing it as a shell command file (i.e. running `sh $jobid`).\n\nYou can wait for jobs to finish using `nq -w`, possibly listing job\nIDs you want to wait for; the default is all of them.  Likewise, you\ncan test if there are jobs which need to be waited upon using `-t`.\n\nBy default, job IDs are per-directory, but you can set `$NQDIR` to put\nthem elsewhere.  Creating `nq` wrappers setting `$NQDIR` to provide\ndifferent queues for different purposes is encouraged.\n\nAll these operations take worst-case quadratic time in the amount of\nlock files produced, so you should clean them regularly.\n\n## Examples\n\nBuild targets `clean`, `depends`, `all`, without occupying the terminal:\n\n\t% nq make clean\n\t% nq make depends\n\t% nq make all\n\t% nqtail\n\t... look at output, can interrupt with C-c any time\n\twithout stopping the build ...\n\nSimple download queue, accessible from multiple terminals:\n\n\t% mkdir -p /tmp/downloads\n\t% alias qget='NQDIR=/tmp/downloads nq wget'\n\t% alias qwait='NQDIR=/tmp/downloads nqtail -q'\n\twindow1% qget http://mymirror/big1.iso\n\twindow2% qget http://mymirror/big2.iso\n\twindow3% qget http://mymirror/big3.iso\n\t% qwait\n\t... wait for all downloads to finish ...\n\nAs `nohup` replacement (The benchmark will run in background, every run\ngets a different output file, and the command line you ran is logged,\ntoo!):\n\n\t% ssh remote\n\tremote% nq ./run-benchmark\n\t,14f6f3034f8.17035\n\tremote% ^D\n\t% ssh remote\n\tremote% nqtail\n\t... see output, nqtail exits when job finished ...\n\n## Assumptions\n\n`nq` will only work correctly when:\n- `$NQDIR` (respectively `.`) is writable.\n- `flock(2)` works in `$NQDIR` (respectively `.`).\n- `gettimeofday` behaves monotonic (using `CLOCK_MONOTONIC` would\n  create confusing file names).  Else job order can be confused and\n  multiple tasks can run at once due to race conditions.\n- No other programs put files matching `,*` into `$NQDIR` (respectively `.`).\n\n## nq helpers\n\nTwo helper programs are provided:\n\n**`nqtail`** outputs the log of the currently running jobs, exiting\nwhen the jobs are done.  If no job is running, the output of the last\njob is shown.  `nqtail -a` shows the output of all jobs, `nqtail -q`\nonly shows one line per job.  `nqtail` uses `inotify` on Linux and\nfalls back to polling for size change else.  (`nqtail.sh` is a similar\ntool, not quite as robust, implemented as shell-script calling\n`tail`.)\n\n**`nqterm`** wraps `nq` and displays the `nqtail` output in a new\n`tmux` or screen window.\n\n(A pure shell implementation of `nq` is provided as `nq.sh`.  It needs\n`flock` from util-linux, and only has a timer resolution of 1s.\nLock files from `nq` and `nq.sh` should not be mixed.)\n\n## Installation\n\nUse `make all` to build, `make install` to install relative to `PREFIX`\n(`/usr/local` by default).  The `DESTDIR` convention is respected.\nYou can also just copy the binaries into your `PATH`.\n\nYou can use `make check` to run a simple test suite, if you have\nPerl's `prove` installed.\n\n## Comparison to `at`, `batch`, and `task-spooler`\n\n* `at` runs jobs at a given time.\n  `batch` runs jobs \"when system load levels permit\".\n  `nq` and [`task-spooler`](https://vicerveza.homeunix.net/~viric/soft/ts/)\n  run jobs in sequence with no regard to the system's load average.\n\n* `at` and `batch` have 52 built-in queues: a-z and A-Z.\n  Any directory can be a queue for `nq`.\n  `task-spooler` can have different queues for different terminals.\n\n* You can follow the output of an `nq` queue tail-style with `nqtail`.\n\n* The syntax is different: `at` and `batch` take whole scripts from\n  the standard input or a file; `nq` takes a single command as its\n  command line arguments.\n\n* `nq` doesn't rely on a daemon, and uses a directory to manage the queue.\n  `task-spooler` automatically launches a daemon to manage a queue.\n\n* `task-spooler` can set a maximum number of simultaneous jobs.\n\n## Copyright\n\n`nq` is in the public domain.\n\nTo the extent possible under law,\nLeah Neukirchen \u003cleah@vuxu.org\u003e\nhas waived all copyright and related or\nneighboring rights to this work.\n\nhttp://creativecommons.org/publicdomain/zero/1.0/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleahneukirchen%2Fnq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fleahneukirchen%2Fnq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleahneukirchen%2Fnq/lists"}