{"id":13582527,"url":"https://github.com/brentp/gargs","last_synced_at":"2025-09-12T10:33:36.736Z","repository":{"id":66472178,"uuid":"62161482","full_name":"brentp/gargs","owner":"brentp","description":"better(?) xargs in go","archived":false,"fork":false,"pushed_at":"2018-01-26T23:19:06.000Z","size":77,"stargazers_count":144,"open_issues_count":5,"forks_count":9,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-31T19:44:10.867Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brentp.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-06-28T17:35:54.000Z","updated_at":"2025-03-24T19:45:31.000Z","dependencies_parsed_at":"2023-02-25T14:30:22.327Z","dependency_job_id":null,"html_url":"https://github.com/brentp/gargs","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brentp%2Fgargs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brentp%2Fgargs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brentp%2Fgargs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brentp%2Fgargs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brentp","download_url":"https://codeload.github.com/brentp/gargs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253166521,"owners_count":21864482,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T15:02:47.890Z","updated_at":"2025-05-09T00:11:25.224Z","avatar_url":"https://github.com/brentp.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"\u003c!--\nrm -rf binaries\nmkdir -p binaries/\nVERSION=0.3.8\nfor os in darwin linux windows; do\n\tGOOS=$os GOARCH=$arch go build -o binaries/gargs_${os} main.go\ndone\n--\u003e\ngargs\n=====\n\n[![Build Status](https://travis-ci.org/brentp/gargs.svg?branch=master)](https://travis-ci.org/brentp/gargs)\n\n**gargs** is like **xargs** but it addresses the following limitations in xargs:\n\n+ it keeps the output serialized (in `xargs` the output one process may be interrupted mid-line by the output from another process) even when using multiple threads\n+ easy to specify multiple arguments with number blocks ({0}, {1}, ...) and {} indicates the entire line.\n+ easy to use multiple lines to fill command-template.\n+ easy to --retry each command if it fails (e.g. due to network or other intermittent error)\n+ simple implementation\n+ allows exiting all commands when an error in one of them occurs.\n+ optionally logs all commands with successful commands prefixed by '#' so it's easy to find failed commands.\n+ simple implementation.\n+ expects a $SHELL command as the argument rather than requiring `bash -c ...`\n+ allows keeping output in order of input even when proceses finish out of order (via -o flag)\n\n\nA very simple example usage with 3 processes to echo some numbers:\n\n```\n$ seq 3 | gargs --log my.log -p 3 \"echo {0}\"\n1\n2\n3\n```\n\n`my.log` will contain the commands run and a final line '# SUCCESS' that shows all processes finished\nwithout error. This makes it easy to check if all commands ran without catching the exit code of the command.\n\nInstall\n=======\n\nDownload the appropriate binary for your system from [releases](https://github.com/brentp/gargs/releases) into your $PATH.\n\nEnvironment Variables\n=====================\n\n`GARGS_PROCESS_BUFFER`\n----------------------\n\n`GARGS_PROCESS_BUFFER` can be used to set the size of data that can be read into memory before a tmp file is used.\nIncreasing this value increases memory use and decreases disk IO. E.g. to tell `gargs` to use a tmp file only\nwhen it has read 20MB (for example if we know that most processes will generate less than that). user\n\n```\nGARGS_PROCESS_BUFFER=20000000 gargs ...\n```\n\nChanging this value will not affect the output at all, it will only change the internal decisions in `gargs`\n\n\n`GARGS_WAIT_MULTIPLIER`\n-----------------------\n\nIncreasing this value improves concurrency at the expense of memory when `-o` or `--ordered` is used.\nIt determines the size of the queue that finished processes will be pushed onto and therefore how many\nprocesses can be waiting will a single (or few) slow process are still running. If the user specified \n`-p` 10 and GARGS_WAIT_MULTIPLIER=5, then up to 49 processes can wait for a singe slow process.\nThe default value is 4.\n\nImplementation\n==============\n\n`gargs` will spawn a worker goroutine for each core requested via `-p`. It will attempt\nto read up to 1MB (settable by `GARGS_PROCESS_BUFFER` env variable) of output from each proceses\ninto memory. If it reaches an EOF (they end of the output from the process) within that 1MB,\nthen it will write that to stdout. If not, it will write to a temporary file keep memory usage:\nlow. The output from each process can then be sent to STDOUT with the only work being the actual copy of\nbytes from the temp-file to STDOUT--no waiting on the process itself.\n\nEach process is run via golang's [os/exec#Cmd](https://golang.org/pkg/os/exec/#Cmd) with\noutput sent to a pipe. There is very little overhead for this per-call; comparing `xargs` to `gargs`:\n\n```\nseq 1 5000 | xargs -I {} bash -c 'echo {}' \u003e /dev/null\nseq 1 5000 | gargs 'echo {}' \u003e /dev/null\n```\n\ngargs takes about 4.6 seconds while xargs takes 4.0 seconds.\n\n\nExample\n=======\nLet's say we have a file `t.txt` like:\n```\nchr1\t22 33\nchr2 22 33\nchr3 22\t33\nchr4\t22\t33\n```\nThat has a mixture of tabs and spaces. We can convert each line to chrom:start-end format with:\n\n```\n$ cat t.txt | gargs --sep \"\\s+\" -p 2 \"echo '{0}:{1}-{2}'\"\nchr2:22-33\nchr1:22-33\nchr3:22-33\nchr4:22-33\n```\n\nIn this case, we're using **2** processes to run this in parallel which will make more of a difference\nif we do something time-consuming rather than `echo`.\n\nNote that `{0}`, `{1}`, etc. grab the 1st, 2nd, ... values respectively. To get the entire line, use `{}`.\n\nWe can use `-n` to send multiple lines of input to each process:\n\n```\n$ seq 1 10 | gargs -n 4 \"echo {}\"\n1 2 3 4\n5 6 7 8\n9 10\n```\n\nNote that even though we send 4 arguments, we only specify the place-holder `{}` once.\nAlso it does the right thing (tm) for the last line where there are only 2 values (9, 10).\nThis works as long as the program accepting the arguments doesn't required a fixed number.\n\n\nUsage\n=====\n\nvia `gargs -h`\n```\ngargs 0.3.8\nusage: gargs [--procs PROCS] [--sep SEP] [--nlines NLINES] [--retry RETRY] [--ordered] [--verbose] [--stop-on-error] [--dry-run] [--log LOG] COMMAND\n\npositional arguments:\n  command                command template to fill and execute.\n\noptions:\n  --procs PROCS, -p PROCS\n                         number of processes to use. [default: 1]\n  --sep SEP, -s SEP      regex to split line to fill multiple template place-holders.\n  --nlines NLINES, -n NLINES\n                         lines to consume for each command. -s and -n are mutually exclusive. [default: 1]\n  --retry RETRY, -r RETRY\n                         times to retry a command if it fails (default is 0).\n  --ordered, -o          keep output in order of input.\n  --verbose, -v          print commands to stderr as they are executed.\n  --stop-on-error, -e    stop all processes on any error.\n  --dry-run, -d          print (but do not run) the commands.\n  --log LOG, -l LOG      file to log commands. Successful commands are prefixed with '#'.\n  --help, -h             display this help and exit\n  --version              display version and exit\n\n```\n\nEnvironment Variables\n=====================\n\nThe environment variable `PROCESS_I` is set to the (0-based) line (or batch of lines) number\nof the input line it is processing.\n\nFor example, this can be used create unique file names.:\n\n```\n... | gargs -p 20 'do-stuff $input \u003e $PROCESS_I.output.txt'\n```\n\n\nAPI\n===\n\nThere is also a simple API for running shell processes in the process subdirectory with documentation [here](https://godoc.org/github.com/brentp/gargs/process)\n\n[![GoDoc](https://godoc.org/github.com/brentp/gargs/process?status.png)](https://godoc.org/github.com/brentp/gargs/process)\n\n\n\nTODO\n====\n\n+ [X] final exit code is the largest of any seen exit code even with -c\n+ [X] dry-run\n+ [ ] combinations of `-n` and `--sep`.\n\n\nExtras\n======\n\nTransactional\n-------------\n\nWhile this isn't done in `gargs` per se. The user can implement their own transactional setup with something like:\n\n```\n... | gargs -p 20 \"if [[ ! -e $PROCESS_i.final ]]; then do-stuff {} \u003e $PROCESS_I.tmp \u0026\u0026 mv $PROCESS_I.tmp $PROCESS_I.final; fi\" \n```\nSince `mv` is atomic on most systems. This will only ever `do-stuff` sucessfully once. \n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrentp%2Fgargs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrentp%2Fgargs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrentp%2Fgargs/lists"}