{"id":22903635,"url":"https://github.com/vltpkg/benchmarks","last_synced_at":"2026-04-09T07:04:59.958Z","repository":{"id":266389874,"uuid":"874468174","full_name":"vltpkg/benchmarks","owner":"vltpkg","description":"Tracking the performance of JS package managers","archived":false,"fork":false,"pushed_at":"2025-02-08T23:50:43.000Z","size":1208,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-03-25T07:38:54.112Z","etag":null,"topics":["benchmarks","bun","javascript","nodejs","npm","nx","packages","packages-manager","pnpm","turborepo","vlt","yarn"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vltpkg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-17T22:05:59.000Z","updated_at":"2025-02-08T23:50:46.000Z","dependencies_parsed_at":"2025-01-28T00:37:22.018Z","dependency_job_id":null,"html_url":"https://github.com/vltpkg/benchmarks","commit_stats":null,"previous_names":["vltpkg/benchmarks"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vltpkg%2Fbenchmarks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vltpkg%2Fbenchmarks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vltpkg%2Fbenchmarks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vltpkg%2Fbenchmarks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vltpkg","download_url":"https://codeload.github.com/vltpkg/benchmarks/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246599236,"owners_count":20803166,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarks","bun","javascript","nodejs","npm","nx","packages","packages-manager","pnpm","turborepo","vlt","yarn"],"created_at":"2024-12-14T02:38:05.158Z","updated_at":"2025-12-30T23:14:13.959Z","avatar_url":"https://github.com/vltpkg.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Package Manager Benchmarks\n\nThis repo contains a suite of fixtures \u0026 tools to track the performance of package managers. We benchmark various Node.js package managers (npm, yarn, pnpm, berry, deno, bun, vlt, nx, turbo) across different project types and scenarios.\n\n## Environment\n\nWe currently only test the latest `linux` runner which is the most common GitHub Action environment. The [standard GitHub-hosted public runner environment](https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories) specs are below:\n\n- VM: Linux\n- Processor (CPU): 4\n- Memory (RAM): 16 GB\n- Storage (SSD): 14 GB\n- Workflow label: `ubuntu-latest`\n\nWe may add Mac/Windows in the future but it will likely exponentially increase the already slow run time of the suite (**~40min**).\n\n## Overview\n\nThe benchmarks measure:\n- Per-package installation times\n- Total Project installation times\n- Run script execution performance\n- Standard deviation of results\n\n### Project Types (fixtures)\n- Next.js\n- Astro\n- Svelte\n- Vue\n- Large\n\n### Package Managers\n- npm\n- yarn\n- pnpm\n- Yarn Berry\n- Deno\n- Bun\n- VLT\n- NX\n- Turbo\n- Node.js\n\n## Configuration/Normalization\n\nWe do a best-effort job to configure each tool to behave as similar as possible to its peers but there's limitations to this standardization in many scenarios (as each tool makes decisions about its default support for security checks/validations/feature-set). As part of the normalization process, we count the number of packages - post-installation - \u0026 use that to determine the average speed relative to the number of packages installed. This strategy helps account for when there are significant discrepancies between the package manager's dependency graph resolution ([you can read/see more here](https://docs.google.com/presentation/d/1ojXF4jb_1MyGhew2LCbdrZ4e_0vYUr-7CoMJLJsHwZY/edit?usp=sharing)).\n\n#### Example:\n- **Package Manager A** installs **1,000** packages in **10s** -\u003e an avg. of **~10ms** per-package\n- **Package Manager B** installs **10** packages in **1s** -\u003e an avg. of **~100ms** per-package\n\n## Testing Package Installation\n\nThe installation tests we run today mimic a matrix of different variations (cold cache, warm cache, cold cache with lockfile, etc) for a variety of test fixtures (ie. we install the packages of a `next`, `vue`, `svelte` \u0026 `astro` starter project).\n\n#### Supported Tools\n\n- `vlt`\n- `npm`\n- `pnpm`\n- `yarn`\n- `yarn berry`\n- `deno`\n- `bun`\n\n## Testing Script Execution\n\nThis suite also tests the performance of basic script execution (ex. `npm run foo`). Notably, for any given build, test or deployment task the spawning of the process is a fraction of the overall execution time. That said, this is a commonly tracked workflow by various developer tools as it involves the common set of tasks: startup, filesystem read (`package.json`) \u0026 finally, spawning the process/command.\n\n#### Supported Tools\n\n- `vlt`\n- `npm`\n- `pnpm`\n- `yarn`\n- `yarn berry`\n- `deno`\n- `bun`\n- `node`\n- `turborepo`\n- `nx`\n\n## Running Benchmarks\n\n### Local Development\n\n- 1. Setup:\n\n  - 1.1 Install `jq` using your OS package manager\n  \n  - 1.2 Install `hyperfine`, version \u003e= 1.19.0 is required\n  \n  - 1.3 Install package managers and corepack:\n    ```bash\n    npm install -g npm@latest corepack@latest vlt@latest bun@latest deno@latest nx@latest turbo@latest\n    ```\n  \n  - 1.4 Make a new `results` folder:\n    ```bash\n    mkdir -p results\n    ```\n\n- 2. Run benchmarks:\n\n  ```bash\n  # Install and benchmark a specific project\n  bash ./scripts/benchmark.sh \u003cfixture\u003e \u003cvariation\u003e\n\n  # Example: Benchmark Next.js with cold cache\n  bash ./scripts/benchmark.sh next clean\n  ```\n\n### GitHub Actions\n\nThe benchmarks run automatically on:\n- Push to main branch\n- Manual workflow trigger\n\nThe workflow:\n1. Sets up the environment with all package managers\n2. Runs benchmarks for each project type (cold and warm cache)\n3. Executes task performance benchmarks\n4. Processes results and generates visualizations\n5. Deploys results to GitHub Pages (main branch only)\n\n## Results\n\n### Console Output\n\nEach benchmark run provides a summary in the console:\n```\n=== Project Name (cache type) ===\npackage-manager: X.XXs (stddev: X.XXs)\n...\n```\n\n### Generated Results\n\nResults are organized by date in the `YYYY-MM-DD/` directory:\n- `\u003cfixture\u003e-\u003cvariation\u003e.json`: Cold cache installation results\n- `\u003cfixture\u003e-\u003cvariation\u003e-package-count.json`: The count of packages installed for each package manager for a given fixture and variation\n- `index.html`: Interactive visualization\n\n### Visualization\n\nThe generated charts show:\n- Per-package installation times for each fixture and variation combination\n- Total installation times for each different combination\n- Run scripts execution performance\n- Standard deviation in tooltips\n- Summary table with total installation times and package counts\n\n### View Results Online\n\nResults are automatically deployed to GitHub Pages when running on the main branch:\n\n\u003chttps://vltpkg.github.io/benchmarks/\u003e\n\nEach run creates a new dated html file with its results, making it easy to track performance over time.\n\n## Contributing\n\n1. Fork the repository\n2. Create your feature branch\n3. Commit your changes\n4. Push to the branch\n5. Create a Pull Request\n\n## Local debugging\n\nYou can debug the result process scripts and the web app locally by using data\nfrom previous [GitHub Action runs of the Package Manager Benchmarks workflow](https://github.com/vltpkg/benchmarks/actions/workflows/benchmark.yaml).\n\nThose can be individually downloaded and their `benchmarks.json` files renamed\nto a `results/\u003cdate\u003e/\u003cfixture\u003e-\u003cvariation\u003e.json` file, where `\u003cdate\u003e` needs to\nmatch a folder name in the `results` folder in a `YYYY-MM-DD` pattern and\n`\u003cfixture\u003e` is one of the known project type fixtures and `\u003cvariation\u003e` is one\nof the known variations (e.g. `clean`, `cache`, `lockfile`, etc).\n\nYou may test the chart data generation script locally by running: \n\n```sh\nnode scripts/generate-chart.js \u003cdate\u003e\n```\n\nMake sure you have a `results/\u003cdate\u003e` directory with valid benchmark\nresult JSON files in it.\n\nAfter a succesful run, test the web app rendering the chart data locally\nby copying the result `results/\u003cdate\u003e/chart-data.json` file to the\nweb app folder in a `latest/` folder, e.g: `app/latest/chart-data.json`\n\n## License\n\nThis project is licensed under the BSD-2-Clause-Patent License - see the LICENSE file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvltpkg%2Fbenchmarks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvltpkg%2Fbenchmarks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvltpkg%2Fbenchmarks/lists"}