{"id":13442181,"url":"https://github.com/scinos/yarn-deduplicate","last_synced_at":"2025-05-11T03:42:37.513Z","repository":{"id":38353284,"uuid":"137625730","full_name":"scinos/yarn-deduplicate","owner":"scinos","description":"Deduplication tool for yarn.lock files","archived":false,"fork":false,"pushed_at":"2025-05-06T16:25:57.000Z","size":7618,"stargazers_count":1389,"open_issues_count":20,"forks_count":57,"subscribers_count":21,"default_branch":"master","last_synced_at":"2025-05-07T23:05:41.074Z","etag":null,"topics":["dedupe","duplicated-packages","duplicates","lock-file","yarn","yarn-lock"],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scinos.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.md","dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-06-17T03:18:37.000Z","updated_at":"2025-05-04T02:57:38.000Z","dependencies_parsed_at":"2023-10-04T08:43:45.235Z","dependency_job_id":"5b414c0b-1965-488e-82f1-5e9b7ef78b31","html_url":"https://github.com/scinos/yarn-deduplicate","commit_stats":{"total_commits":301,"total_committers":34,"mean_commits":8.852941176470589,"dds":0.6411960132890365,"last_synced_commit":"8a3cfc168cbd43eb09261f05a9ce514ced52b4c0"},"previous_names":["atlassian/yarn-deduplicate"],"tags_count":22,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scinos%2Fyarn-deduplicate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scinos%2Fyarn-deduplicate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scinos%2Fyarn-deduplicate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scinos%2Fyarn-deduplicate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scinos","download_url":"https://codeload.github.com/scinos/yarn-deduplicate/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252968117,"owners_count":21833251,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dedupe","duplicated-packages","duplicates","lock-file","yarn","yarn-lock"],"created_at":"2024-07-31T03:01:42.624Z","updated_at":"2025-05-07T23:05:46.116Z","avatar_url":"https://github.com/scinos.png","language":"TypeScript","funding_links":[],"categories":["HarmonyOS","TypeScript","🔧 Utilities \u0026 Miscellaneous","others"],"sub_categories":["Windows Manager"],"readme":"# yarn-deduplicate\n\nCleans up `yarn.lock` by removing duplicates.\n\nBuilds:\n[![Node.js CI](https://github.com/scinos/yarn-deduplicate/actions/workflows/node.js.yml/badge.svg)](https://github.com/scinos/yarn-deduplicate/actions/workflows/node.js.yml)\n\nThis package only works with Yarn v1. Yarn v2 supports package deduplication\n[natively](https://github.com/yarnpkg/berry/pull/1558)!\n\nA duplicate package is when two dependencies are resolved to a different version, even when a single\nversion matches the range specified in the dependencies. See the\n[Deduplication strategies](#deduplication-strategies) section for a few examples.\n\n## Installation\n\nInstall the package globally:\n\n```sh\nnpm install -g yarn-deduplicate\n```\n\nor\n\n```sh\nyarn global add yarn-deduplicate\n```\n\nThis package also works wth\n[npx](https://medium.com/@maybekatz/introducing-npx-an-npm-package-runner-55f7d4bd282b), so you\ndon't need to install it. For example, to recreate the most common scenario below with `npx`, run:\n\n```sh\nnpx yarn-deduplicate yarn.lock\n```\n\n---\n\n## Usage\n\nThe most common scenario is to run\n\n```sh\nyarn-deduplicate yarn.lock\n```\n\nThis will use the default strategy to remove duplicated packages in `yarn.lock`.\n\nIf you do not specify the yarn.lock path, it defaults to `yarn.lock`.\n\nCheck all available options with:\n\n```sh\nyarn-deduplicate --help\n```\n\n---\n\n## Duplicated packages\n\n`yarn.lock` contains a list of all the dependencies required by your project (including transitive\ndependencies), and the actual package version installed to satisfy those dependencies.\n\nFor the context of this project, a \"duplicated package\" is a package that appears on multiple nodes\nof the dependency tree with overlapping version ranges but resolved to different versions.\n\nFor example, imagine that your project directly depends on `lodash` and `babel`, and `babel` depends\non `lodash` as well. Specifically, your project depends on `lodash@^1.0.0` and `babel` depends on\n`lodash@^1.1.0`. Because how the resolution algorithm works in Yarn, you might end up with two\ndifferent copies of `lodash` (for example, version `1.0.1` and `1.2.0`) in your project, even when\n`1.2.0` will suffice to satisfy both requirements for `lodash`. That's a \"duplicated package\".\n\nIt is important to note that we do not consider duplicated packages when the version ranges don't\noverlap. For example, if your project depends on `underscore@^1.0.0` and `underscore@^2.0.0`. Your\nproject will end up with two versions of `underscore`, and `yarn-deduplicate` won't change that.\n\nWhen using `yarn-deduplicate` remember that **it will change your dependency tree**. There are\ncertain code paths that now will run with a different set of dependencies. It is highly recommended\nthat you review each change to `yarn.lock`. If the change is too big, use the flag `--packages` to\ndeduplicate them gradually.\n\n### Why is this necessary?\n\nYarn documentation seems to suggest this package shouldn't be necessary. For example, in\nhttps://classic.yarnpkg.com/en/docs/cli/dedupe/, it says\n\n\u003e The dedupe command isn’t necessary. `yarn install` will already dedupe.\n\nThis is, however, not exactly true. There are cases where yarn will _not_ deduplicate existing\npackages. For example, this scenario:\n\n- Install `libA`. It depends on `libB ^1.1.0`. At this point, the latest version of `libB` is\n  `1.1.2`, so it gets installed as a transitive dependency in your repo\n\n- After a few days, install `libC`. It also depends on `libB ^1.1.0`. But this time, the latest\n  `libB` version is `1.1.3`.\n\nIn the above scenario, you'll end up with `libB@1.1.2` and `libB@1.1.3` in your repo.\n\nFind more examples in:\n\n- [yarn-deduplicate — The Hero We Need](https://medium.com/@bnaya/yarn-deduplicate-the-hero-we-need-f4497a362128)\n- [De-duplicating yarn.lock](https://medium.com/@scinos/de-duplicating-yarn-lock-ae30be4aa41a)\n- https://github.com/yarnpkg/yarn/issues/3778\n\n### Deduplication strategies\n\n`--strategy \u003cstrategy\u003e`\n\n`highest` will try to use the highest installed version. For example, with the following\n`yarn.lock`:\n\n```text\nlibrary@^1.1.0:\n  version \"1.2.0\"\n\nlibrary@^1.2.0:\n  version \"1.2.0\"\n\nlibrary@^1.3.0:\n  version \"1.3.0\"\n```\n\nIt will deduplicate `library@^1.1.0` and `library@^1.2.0` to `1.3.0`\n\n`fewer` will try to minimize the number of installed versions by trying to deduplicate to the\nversion that satisfies most of the ranges first. For example, with the following `yarn.lock`:\n\n```text\nlibrary@*:\n  version \"2.0.0\"\n\nlibrary@\u003e=1.1.0:\n  version \"3.0.0\"\n\nlibrary@^1.2.0:\n  version \"1.2.0\"\n```\n\nIt will deduplicate `library@*` and `library@\u003e=1.1.0` to `1.2.0`.\n\nNote that this may cause some packages to be **downgraded**. Be sure to check the changelogs between\nall versions and understand the consequences of that downgrade. If unsure, don't use this strategy.\n\nIt is not recommended to use different strategies for different packages. There is no guarantee that\nthe strategy will be honored in subsequent runs of `yarn-deduplicate` unless the same set of flags\nis specified again.\n\n### Progressive deduplication\n\n`--packages \u003cpackage1\u003e \u003cpackage2\u003e \u003cpackageN\u003e`\n\nReceives a list of packages to deduplicate. It will ignore any other duplicated package not in the\nlist. This option is recommended when the number of duplicated packages in `yarn.lock` is too big to\nbe easily reviewed by a human. This will allow for a more controlled and progressive deduplication\nof `yarn.lock`.\n\n`--scopes \u003cscope1\u003e \u003cscope2\u003e \u003cscopeN\u003e`\n\nReceives a list of scopes to deduplicate. It will ignore any other duplicated package not in the\nlist. This option is recommended when deduplicating a large number of inter-dependent packages from\na single scope, such as @babel. This will allow for a more controlled and progressive deduplication\nof `yarn.lock` without specifying each package individually.\n\n### Excluding packages\n\n`--exclude \u003cpackage1\u003e \u003cpackage2\u003e \u003cpackageN`\n\n`--exclude-scopes \u003cscope1\u003e \u003cscope2\u003e \u003cscopeN\u003e`\n\nWith these commands you can exclude certain packages/scopes from the deduplication process. This is\nspecially useful if you want to apply a different strategy for a scope, for example.\n\n### Pre-release versions\n\nBy default, `yarn-deduplicate` will only match pre-release versions if they share they share the\nsame `major`, `minor` and `patch` versions (example: `^1.2.3-alpha.1` and `1.2.3-alpha.2` can be\ndeduplicated, but `^1.2.3` and `1.2.4-alpha.1` can't). This matches the behaviour of\n[semver](https://docs.npmjs.com/misc/semver#prerelease-tags).\n\nTo change this behaviour you can use the flag `--includePrerelease`. This will treat all pre-release\nversionas as if they were normal versions (`^1.2.3` and `1.2.4-alpha.1` can be deduplicated).\n\n### Usage in CI\n\nThis tool can be used as part of a CI workflow. Adding the flag `--fail` will force the process to\nexit with status 1 if there are duplicated packages. Example:\n\n```sh\n# Print the list of duplicated packages and exit with status 1\nyarn-deduplicate --list --fail\n\n# Deduplicate yarn.lock and exit with status 1 if changes were required\nyarn-deduplicate --fail\n```\n\n---\n\n## Migration guide\n\n### From 2.x to 3.x\n\nIn this version we have adopted variadic arguments from commander.js. These are the equivalent\ncommands:\n\n```sh\n#Old\nyarn-deduplicate --packages libA,libB\nyarn-deduplicate --scopes @scopeA,@scopeB\nyarn-deduplicate --exclude libA,libB\n\n#New\nyarn-deduplicate --packages libA libB\nyarn-deduplicate --scopes @scopeA @scopeB\nyarn-deduplicate --exclude libA libB\n```\n\nA consequence of this change is that if you were using one or more of the affected options (\n`--packages`, `--scopes` or `--exclude`) **and** a custom path for `yarn.lock`, you need to use `--`\nto \"stop\" package/scope/exclude parsing:\n\n```sh\nyarn-deduplicate --packages libA libB -- path/to/yarn.lock\n```\n\n### From 0.x to 1.x\n\nIn this version we have renamed the project and refactored the CLI. These are the equivalent\ncommands:\n\n#### Installation\n\n```sh\n# Old\nnpm install -g yarn-tools\n\n# New\nnpm install -g yarn-deduplicate\n```\n\n#### List duplicates\n\n```sh\n# Old\nyarn-tools list-duplicates path/to/yarn.lock\n\n# New\nyarn-deduplicate --list path/to/yarn.lock\n```\n\n### Deduplicate yarn.lock\n\n```sh\n# Old\nyarn-tools fix-duplicates path/to/yarn.lock \u003e tmp\nmv tmp path/to/yarn.lock\n\n# New\nyarn-deduplicate path/to/yarn.lock\n```\n\n### Limit packages to deduplicate yarn.lock\n\n```sh\n# Old\nyarn-tools fix-duplicates path/to/yarn.lock package1 package2\n\n\n# New\nyarn-deduplicate --packages package1,package2 path/to/yarn.lock\n```\n\n## License\n\nCopyright (c) 2022 Sergio Cinos and others. Apache 2.0 licensed, see [LICENSE.txt](LICENSE.txt)\nfile.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscinos%2Fyarn-deduplicate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscinos%2Fyarn-deduplicate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscinos%2Fyarn-deduplicate/lists"}