{"id":18761304,"url":"https://github.com/patte/xstdin-rs","last_synced_at":"2025-12-04T03:30:17.082Z","repository":{"id":218190186,"uuid":"745829951","full_name":"patte/xstdin-rs","owner":"patte","description":"Like gnu xargs, but for stdin. Like gnu parallel, but keeps a set or workers running.","archived":false,"fork":false,"pushed_at":"2024-01-21T10:58:46.000Z","size":22,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-02-18T03:12:13.876Z","etag":null,"topics":["cli","parallel","rust","worker-pool","xargs"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/patte.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2024-01-20T09:16:27.000Z","updated_at":"2024-01-21T10:52:05.000Z","dependencies_parsed_at":"2024-01-20T10:39:24.939Z","dependency_job_id":null,"html_url":"https://github.com/patte/xstdin-rs","commit_stats":null,"previous_names":["patte/xstdin-rs"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patte%2Fxstdin-rs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patte%2Fxstdin-rs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patte%2Fxstdin-rs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patte%2Fxstdin-rs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/patte","download_url":"https://codeload.github.com/patte/xstdin-rs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239654852,"owners_count":19675306,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","parallel","rust","worker-pool","xargs"],"created_at":"2024-11-07T18:15:38.011Z","updated_at":"2025-12-04T03:30:16.771Z","avatar_url":"https://github.com/patte.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# xstdin\n\n[![tests](https://github.com/patte/xstdin-rs/actions/workflows/test-and-lint.yml/badge.svg)](https://github.com/patte/xstdin-rs/actions/workflows/test-and-lint.yml)\n\nCLI tool for distributing stdin to a set of long-running workers.\nBy default, distributes input in chunks of ~8KiB (always on line boundaries) to 4 workers.\nWith the `-l` flag, the input is distributed in a strict round-robin fashion, line by line to each worker (comes at a huge performance cost).\n\nLike `xargs`, but for stdin. Like `parallel`, but keeps a set or workers running.\nImagined by [paddor](https://github.com/paddor/). Developed with help by GPT-4.\n\n## Installation\n\n```bash\ncargo install --path .\n```\n\n## Usage\n```\nUsage: xstdin [-n NUM] [-b SIZE] [-l] \u003ccommand\u003e [\u003carg1\u003e \u003carg2\u003e ...]\n\nOptions:\n    -n, --workers NUM   set number of workers (default is 4)\n    -b, --buffer-size SIZE\n                        set buffer capacity (default is 8KiB)\n    -l, --line-mode     strictly distribute input by line (default\n                        buffer-size)\n    -h, --help          print this help menu\n```\n\n## Examples\n```bash\nseq 1 10 | xstdin -n 2 cat\n1\n3\n5\n7\n9\n2\n4\n6\n8\n10\n```\n\n```bash\nseq 1 10 | xstdin -l -n 2 -- ruby -e 'STDIN.each_line { |line| puts \"#$$: #{line}\" }'\n23026: 2\n23014: 1\n23026: 4\n23014: 3\n23026: 6\n23014: 5\n23026: 8\n23014: 7\n23026: 10\n23014: 9\n```\n\n## Benchmarks\nMacBook Air, M2, 2023:\n```bash\n# yes baseline\nyes | pv --rate | cat \u003e /dev/null\n[3.79GiB/s]\n\n# strict round robin\nyes | pv --rate | xstdin -l cat \u003e /dev/null\n[1.55MiB/s]\n\n# chunked round robin\nyes | pv --rate | xstdin cat \u003e /dev/null\n[2.64GiB/s]\n\n# big chunks round robin\nyes | pv --rate | xstdin -b 32000 cat \u003e /dev/null\n[3.29GiB/s]\n```\n\n```bash\n# large input\ndu -sh input_large.txt\n9.7G\tinput_large.txt\n\ntime pv --rate input_large.txt | xstdin -- cat \u003e /dev/null\n[2.57GiB/s]\n[2.24GiB/s]\npv --rate input_large.txt  0.11s user 2.10s system 50% cpu 4.343 total\nxstdin -- cat \u003e /dev/null  1.02s user 6.54s system 174% cpu 4.343 total\n\nwc -l input_large.txt \n 5219249490 input_large.txt\n\ntime pv --rate input_large.txt | xstdin -- wc -l | awk '{s+=$1} END {print s}'\n[1.67GiB/s]\n5219249490\npv --rate input_large.txt  0.11s user 2.46s system 44% cpu 5.828 total\nxstdin -- wc -l  11.05s user 4.86s system 272% cpu 5.827 total\nawk '{s+=$1} END {print s}'  0.00s user 0.00s system 0% cpu 5.827 total\n``````\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpatte%2Fxstdin-rs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpatte%2Fxstdin-rs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpatte%2Fxstdin-rs/lists"}