{"id":25883156,"url":"https://github.com/maxlath/load-balance-lines","last_synced_at":"2025-08-10T20:05:54.867Z","repository":{"id":67787898,"uuid":"132120492","full_name":"maxlath/load-balance-lines","owner":"maxlath","description":"Parallelize newline-delimited data processing by load balancing lines between multiple processes","archived":false,"fork":false,"pushed_at":"2024-07-17T16:19:15.000Z","size":155,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-08-09T10:07:19.319Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://npmjs.com/package/load-balance-lines","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/maxlath.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-05-04T09:39:26.000Z","updated_at":"2023-11-30T14:49:43.000Z","dependencies_parsed_at":null,"dependency_job_id":"89aee32d-260b-4188-bc5d-c28242ea04b7","html_url":"https://github.com/maxlath/load-balance-lines","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/maxlath/load-balance-lines","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxlath%2Fload-balance-lines","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxlath%2Fload-balance-lines/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxlath%2Fload-balance-lines/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxlath%2Fload-balance-lines/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/maxlath","download_url":"https://codeload.github.com/maxlath/load-balance-lines/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxlath%2Fload-balance-lines/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269739650,"owners_count":24467740,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-10T02:00:08.965Z","response_time":71,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-02T16:31:48.424Z","updated_at":"2025-08-10T20:05:54.802Z","avatar_url":"https://github.com/maxlath.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003ch1\u003eload-balance-lines\u003c/h1\u003e\n\n  Parallelize newline-delimited data processing by load balancing lines between multiple processes\n\n  ![htop](https://github.com/maxlath/load-balance-lines/raw/master/assets/htop.png)\n\u003c/div\u003e\n\n## Summary\n\n\u003c!-- START doctoc generated TOC please keep comment here to allow auto update --\u003e\n\u003c!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --\u003e\n\n\n- [Install](#install)\n- [Basic use](#basic-use)\n- [Simple demo](#simple-demo)\n- [Real case demo](#real-case-demo)\n- [Options](#options)\n  - [Number of processes](#number-of-processes)\n  - [Silent](#silent)\n\n\u003c!-- END doctoc generated TOC please keep comment here to allow auto update --\u003e\n\n## Install\n```sh\n# Make the executable accessible within your project npm scripts as load-balance-lines\n# or, out of npm scripts, as ./node_modules/.bin/load-balance-lines\nnpm i load-balance-lines\n# or globally\nnpm i -g load-balance-lines\n```\n\n## Basic use\n\nTake a huge pile of data with atomic data elements separated by newline breaks, typically [NDJSON](http://ndjson.org).\n\n```sh\n# Make sure your executable is... executable\nchmod +x /path/to/my/executable\n# and let's go!\ncat data.ndjson | load-balance-lines /path/to/my/executable some args\n```\nor without the cat command, using [`\u003c`](http://www.tldp.org/LDP/abs/html/io-redirection.html)\n```sh\nload-balance-lines /path/to/my/executable some args for the executable \u003c data.ndjson\n```\n\n## Simple demo\nsee [test](https://github.com/maxlath/load-balance-lines/blob/master/test/load_balance_lines.js)\n\n## Real case demo\n\nFor the needs of [wikidata-rank](https://github.com/maxlath/wikidata-rank), we need to parse a full [dump](https://www.wikidata.org/wiki/Wikidata:Database_download#JSON_dumps_.28recommended.29) of [Wikidata](https://wikidata.org)\n\n* get the latest dump (currently 31G gzipped)\n\n```sh\nwget -c https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.gz\n```\n\n* Use [nice](http://man7.org/linux/man-pages/man1/nice.1.html) to use the maximum amount of CPU possible while letting the priority to other processes\n* Use [pigz](https://zlib.net/pigz/) to decompress it using threads (drop-in replacement to the single threaded gzip)\n\n```sh\nnice pigz -d \u003c latest-all.json.gz | nice load-balance-lines /path/to/wikidata-rank/scripts/calculate_base_scores\n```\n\n## Options\n\n### Number of processes\nBy default, there will be as many processes as CPU cores, but it can be modified by setting an environment variable\n```sh\nexport LBL_PROCESSES=4 ; cat data.ndjson | load-balance-lines ./my/script\n```\n\n### Verbose\nBy default, the load balancer is silent to let stdout free for sub-processes outputs, but you can get some basic informations by setting `LBL_VERBOSE`\n```sh\nexport LBL_VERBOSE=true ; cat data.ndjson | load-balance-lines ./my/script\n```\n\n## See also\n* [GNU Parallel](https://www.gnu.org/software/parallel/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxlath%2Fload-balance-lines","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaxlath%2Fload-balance-lines","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxlath%2Fload-balance-lines/lists"}