{"id":21885035,"url":"https://github.com/clarksource/lerna-parallelism","last_synced_at":"2025-04-15T07:19:36.408Z","repository":{"id":42758700,"uuid":"275806959","full_name":"ClarkSource/lerna-parallelism","owner":"ClarkSource","description":"`lerna run` with CircleCI parallelism splitting support","archived":false,"fork":false,"pushed_at":"2023-03-05T05:07:35.000Z","size":1178,"stargazers_count":13,"open_issues_count":9,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-15T07:19:29.086Z","etag":null,"topics":["ci","circleci","lerna","lerna-monorepo","lernajs","parallelism","splitting"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ClarkSource.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-29T12:22:41.000Z","updated_at":"2023-07-24T09:53:52.000Z","dependencies_parsed_at":"2024-06-21T07:28:18.016Z","dependency_job_id":"9af88985-629a-47f2-83fa-92ca1350fe61","html_url":"https://github.com/ClarkSource/lerna-parallelism","commit_stats":{"total_commits":46,"total_committers":3,"mean_commits":"15.333333333333334","dds":0.04347826086956519,"last_synced_commit":"de05bf1413921de2d494182db65ce77f162fcdd3"},"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClarkSource%2Flerna-parallelism","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClarkSource%2Flerna-parallelism/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClarkSource%2Flerna-parallelism/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClarkSource%2Flerna-parallelism/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ClarkSource","download_url":"https://codeload.github.com/ClarkSource/lerna-parallelism/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249023746,"owners_count":21199961,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ci","circleci","lerna","lerna-monorepo","lernajs","parallelism","splitting"],"created_at":"2024-11-28T10:18:30.455Z","updated_at":"2025-04-15T07:19:36.387Z","avatar_url":"https://github.com/ClarkSource.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://www.clark.de/de/jobs\"\u003e\n    \u003cbr\u003e\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e\n    \u003cimg alt=\"CLARK\" src=\"./docs/assets/clark.svg\" height=\"40\"\u003e\n    \u003cbr\u003e\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n# lerna-parallelism\n\n[![CI](https://github.com/ClarkSource/lerna-parallelism/workflows/CI/badge.svg)](https://github.com/ClarkSource/lerna-parallelism/actions)\n[![npm version](https://badge.fury.io/js/lerna-parallelism.svg)](http://badge.fury.io/js/lerna-parallelism)\n[![Download Total](https://img.shields.io/npm/dt/lerna-parallelism.svg)](http://badge.fury.io/js/lerna-parallelism)\n[![code style: prettier](https://img.shields.io/badge/code_style-prettier-ff69b4.svg)](https://github.com/prettier/prettier)\n[![CLARK Open Source](https://img.shields.io/badge/CLARK-Open%20Source-%232B6CDE.svg)](https://www.clark.de/de/jobs)\u003cbr\u003e\n[![Dependabot enabled](https://img.shields.io/badge/dependabot-enabled-blue.svg?logo=dependabot)](https://dependabot.com/)\n[![dependencies Status](https://david-dm.org/ClarkSource/lerna-parallelism/status.svg)](https://david-dm.org/ClarkSource/lerna-parallelism)\n[![devDependencies Status](https://david-dm.org/ClarkSource/lerna-parallelism/dev-status.svg)](https://david-dm.org/ClarkSource/lerna-parallelism?type=dev)\n\n**lerna-parallelism** is an extension of [**lerna**][lerna] that adds support\nfor deterministically partitioning packages to allow parallel execution across\nmultiple workers, like [CircleCI's `parallelism` feature][circleci-parallelism].\n\n[lerna]: https://github.com/lerna/lerna\n[circleci-parallelism]: https://circleci.com/docs/2.0/parallelism-faster-jobs/\n\n- [Introduction](#Introduction)\n  - [Use Case](#Use-Case)\n  - [Quick Example](#Quick-Example)\n- [Installation](#Installation)\n- [Usage](#Usage)\n\n## Introduction\n\n### Use Case\n\n\u003e Speeding up monorepo CI pipelines via multi-worker parallelization.\n\nYou maintain a large [`lerna`][lerna] monorepo with lots of individual packages.\nIn CI you run a script / command for many or all of the packages.\n\n```sh\n# Execute the `test` script for all packages.\nlerna run \\\n  # Prefix each log line with the package name.\n  # https://github.com/lerna/lerna/tree/main/commands/run#--stream\n  --stream \\\n  # Run packages sequentially to avoid interleaved log output and resource contention.\n  # https://github.com/lerna/lerna/blob/main/core/global-options/README.md#--concurrency\n  --concurrency 1 \\\n  test\n```\n\nYour CI workflow is taking far too long, because you process all packages\nsequentially on a single CI worker instance / node / VM.\n\nYour CI service supports spinning up multiple worker instances, but you don't\nknow how tell `lerna` to distribute the workload across these instances. You\ncould use [`--scope`][lerna-scope] and hard-code all package names, but this is\ndifficult to maintain, as packages are added, renamed or removed.\n\n[lerna-scope]: https://github.com/lerna/lerna/tree/main/core/filter-options#--scope-glob\n\n### Quick Example\n\n**lerna-parallelism** makes it easy to split the workload — the packages to be\nprocessed — dynamically and deterministically across separate worker instances.\n\n```sh\nlerna-parallelism run \\\n  # Keep your `lerna` options just like before.\n  --stream \\\n  --concurrency 1 \\\n  # Divide the list of packages to process into 4 equal-sized chunks.\n  # This should be equal to the total number of worker instances.\n  --split 4 \\\n  # Take only the first chunk (zero-based).\n  # This should be the index of the individual worker instance you're running on.\n  --partition 0 \\\n  # Run the `test` script for all packages in that chunk only.\n  test\n```\n\n## Installation\n\n### Project-local Installation\n\n```sh\nyarn add -D lerna-parallelism\n\n# And then run via:\nyarn lerna-parallelism ...\n```\n\n### Global Installation\n\n```sh\nyarn global add lerna-parallelism\n# or\nvolta install lerna-parallelism\n\n# And then run via:\nlerna-parallelism ...\n```\n\n## Usage\n\n- [Global Options](#Global-Options)\n  - [Partition Count](#Partition-Count): `--split`, `--partition`\n  - [Distribution Strategy](#Distribution-Strategy)\n    - [`--distribute-by count`](#--distribute-by-count)\n    - [`--distribute-by weight`](#--distribute-by-weight)\n- [Commands](#Commands)\n  - [`lerna-parallelism run`](#lerna-parallelism-run)\n  - [Other Commands](#Other-Commands)\n\n### Global Options\n\n`lerna-parallelism` adds a few CLI options on top of `lerna`. The following\noptions are globally understood by all commands supported by\n`lerna-parallelism`.\n\nAll other command options behave just like the upstream `lerna` version of the\nrespective command.\n\n#### Partition Count\n\n\u003e Split the list of packages into `--split n` chunks and take the chunk with\n\u003e zero-based index `--partition n`.\n\n- `--split n`: The number of split partitions.\n  Defaults to [`$CIRCLE_NODE_TOTAL`][circleci-parallelism-env].\u003cbr\u003e\n  _This should be equal to the total number of worker instances._\n- `--partition n`: Which partition to execute, zero-based.\n  Defaults [`$CIRCLE_NODE_INDEX`][circleci-parallelism-env].\u003cbr\u003e\n  _This should be the index of the individual worker instance you're running on._\n\n[circleci-parallelism-env]: https://circleci.com/docs/2.0/parallelism-faster-jobs/#using-environment-variables-to-split-tests\n\n#### Distribution Strategy\n\n\u003e Algorithm to use for deciding which package goes into which partition.\n\n##### `--distribute-by count`\n\n\u003e **Default.** Split into even-sized partitions.\n\nThe packages are taken in order as emitted by `lerna` and simply split into\n[`--split n`](#Partition-Count) partitions of equal size.\n\nIf the total package count cannot be distributed evenly, some partition(s) may\nbe slightly smaller than the other partition(s).\n\n**Example:** 26 packages (named `a`–`z`) are split into `--split 4` partitions.\n\n- `--partition 0` (size = 7): `a`, `b`, `c`, `d`, `e`, `f`, `g`\n- `--partition 1` (size = 7): `h`, `i`, `j`, `k`, `l`, `m`, `n`\n- `--partition 2` (size = 6): `o`, `p`, `q`, `r`, `s`, `t`\n- `--partition 3` (size = 6): `u`, `v`, `w`, `x`, `y`, `z`\n\n##### `--distribute-by weight`\n\n\u003e Assign a weight to each package and split into even-weighted partitions.\n\nThe packages are ordered by \"weight\" in descending order and iteratively\nassigned to the current lightest partition. This results in partitions with\napproximately equal weight, but uneven size by total package count.\n\nThis strategy is useful to account for different CI runtimes per package.\n\nEach package's weight is read from the `lernaPackageWeight` property from its\n`package.json`. If missing, it defaults to `1`. You can change the property\nlookup name with the `--packageWeightKey` option, like\n`--packageWeightKey testRuntime`. It just has to be a `number`.\n\nYou're expected to add these weights yourself, where necessary. Basing the\nweights off of the package's CI runtime is sensible. Keep in mind, that you also\nneed to maintain these weights: you should adjust the weights from time to time,\ne.g. when adding new tests or build steps to a package, that increase it's\noverall CI runtime.\n\n**Example:** 9 packages are split into `--split 3` partitions using the\nfollowing weights.\n\n| Name   | `a` | `b` | `c` | `d` | `e` | `f` | `g` | `h` | `i` |\n| ------ | --: | --: | --: | --: | --: | --: | --: | --: | --: |\n| Weight |   9 |   5 |   5 |   5 |   2 |   2 |   2 |   1 |   1 |\n\n- `--partition 0` (size = 2, weight = 10): `a` (9), `h` (1)\n- `--partition 1` (size = 3, weight = 11): `b` (5), `d` (5), `i` (1)\n- `--partition 2` (size = 4, weight = 11): `c` (5), `e` (2), `f` (2), `g` (2)\n\n### Commands\n\nThe following commands from `lerna` are supported:\n\n### `lerna-parallelism run`\n\n[`lerna run`][lerna-run]\n\n[lerna-run]: https://github.com/lerna/lerna/blob/master/commands/run#readme\n\n\u003e Run an npm script in each package that contains that script.\n\nFor instance, this executes the last of four partitions. It also passes along\n`--stream` \u0026 `--concurrency 1` to prefix log lines with the package name.\n\n```sh\nlerna-parallelism run \\\n  --stream \\\n  --concurrency 1 \\\n  --split 4 \\\n  --partition 3 \\\n  test\n```\n\n### Other Commands\n\nFor some commands, like `lerna bootstrap`, splitting makes no sense. For some\nothers, it does, specifically:\n\n- `lerna changed`\n- `lerna exec`\n- `lerna list`\n- `lerna publish`\n\nIf you'd like to see support for these commands as well, feel free to submit a\npull request!\n\n## License\n\nThis project is licensed under the [ISC License](LICENSE.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclarksource%2Flerna-parallelism","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclarksource%2Flerna-parallelism","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclarksource%2Flerna-parallelism/lists"}