https://github.com/maxlath/load-balance-lines
Parallelize newline-delimited data processing by load balancing lines between multiple processes
https://github.com/maxlath/load-balance-lines
Last synced: 12 months ago
JSON representation
Parallelize newline-delimited data processing by load balancing lines between multiple processes
- Host: GitHub
- URL: https://github.com/maxlath/load-balance-lines
- Owner: maxlath
- Created: 2018-05-04T09:39:26.000Z (about 8 years ago)
- Default Branch: main
- Last Pushed: 2024-07-17T16:19:15.000Z (about 2 years ago)
- Last Synced: 2025-08-09T10:07:19.319Z (12 months ago)
- Language: JavaScript
- Homepage: https://npmjs.com/package/load-balance-lines
- Size: 151 KB
- Stars: 5
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
load-balance-lines
Parallelize newline-delimited data processing by load balancing lines between multiple processes
## Summary
- [Install](#install)
- [Basic use](#basic-use)
- [Simple demo](#simple-demo)
- [Real case demo](#real-case-demo)
- [Options](#options)
- [Number of processes](#number-of-processes)
- [Silent](#silent)
## Install
```sh
# Make the executable accessible within your project npm scripts as load-balance-lines
# or, out of npm scripts, as ./node_modules/.bin/load-balance-lines
npm i load-balance-lines
# or globally
npm i -g load-balance-lines
```
## Basic use
Take a huge pile of data with atomic data elements separated by newline breaks, typically [NDJSON](http://ndjson.org).
```sh
# Make sure your executable is... executable
chmod +x /path/to/my/executable
# and let's go!
cat data.ndjson | load-balance-lines /path/to/my/executable some args
```
or without the cat command, using [`<`](http://www.tldp.org/LDP/abs/html/io-redirection.html)
```sh
load-balance-lines /path/to/my/executable some args for the executable < data.ndjson
```
## Simple demo
see [test](https://github.com/maxlath/load-balance-lines/blob/master/test/load_balance_lines.js)
## Real case demo
For the needs of [wikidata-rank](https://github.com/maxlath/wikidata-rank), we need to parse a full [dump](https://www.wikidata.org/wiki/Wikidata:Database_download#JSON_dumps_.28recommended.29) of [Wikidata](https://wikidata.org)
* get the latest dump (currently 31G gzipped)
```sh
wget -c https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.gz
```
* Use [nice](http://man7.org/linux/man-pages/man1/nice.1.html) to use the maximum amount of CPU possible while letting the priority to other processes
* Use [pigz](https://zlib.net/pigz/) to decompress it using threads (drop-in replacement to the single threaded gzip)
```sh
nice pigz -d < latest-all.json.gz | nice load-balance-lines /path/to/wikidata-rank/scripts/calculate_base_scores
```
## Options
### Number of processes
By default, there will be as many processes as CPU cores, but it can be modified by setting an environment variable
```sh
export LBL_PROCESSES=4 ; cat data.ndjson | load-balance-lines ./my/script
```
### Verbose
By default, the load balancer is silent to let stdout free for sub-processes outputs, but you can get some basic informations by setting `LBL_VERBOSE`
```sh
export LBL_VERBOSE=true ; cat data.ndjson | load-balance-lines ./my/script
```
## See also
* [GNU Parallel](https://www.gnu.org/software/parallel/)