{"id":22843317,"url":"https://github.com/milahu/nodejs-hide-symlinks","last_synced_at":"2025-04-28T13:43:53.781Z","repository":{"id":109860780,"uuid":"411652392","full_name":"milahu/nodejs-hide-symlinks","owner":"milahu","description":"hide symlinks from nodejs, to implement a symlinked machine-level global NPM store","archived":false,"fork":false,"pushed_at":"2024-10-21T15:57:55.000Z","size":21,"stargazers_count":6,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-30T10:11:16.585Z","etag":null,"topics":["granular-caching","ld-preload","machine-level-npm-store","nix","nixos","nodejs","npm","pnpm","rust","symlinks"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/milahu.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"license.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-29T11:48:47.000Z","updated_at":"2025-02-20T20:47:09.000Z","dependencies_parsed_at":"2024-10-22T12:06:24.692Z","dependency_job_id":null,"html_url":"https://github.com/milahu/nodejs-hide-symlinks","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/milahu%2Fnodejs-hide-symlinks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/milahu%2Fnodejs-hide-symlinks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/milahu%2Fnodejs-hide-symlinks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/milahu%2Fnodejs-hide-symlinks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/milahu","download_url":"https://codeload.github.com/milahu/nodejs-hide-symlinks/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251321502,"owners_count":21570751,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["granular-caching","ld-preload","machine-level-npm-store","nix","nixos","nodejs","npm","pnpm","rust","symlinks"],"created_at":"2024-12-13T02:14:17.769Z","updated_at":"2025-04-28T13:43:53.756Z","avatar_url":"https://github.com/milahu.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nodejs-hide-symlinks\n\nhide symlinks from nodejs,\nto implement a symlinked machine-level global NPM store on nixos (and others)\n\n## build\n\nthis requires `rust nightly`, as specified in [rust-toolchain.toml](rust-toolchain.toml).\n`rust stable` will silently fail ...\n\n```\ncargo build\n```\n\n## use\n\n```\nLD_PRELOAD=\"target/debug/libnodejs_hide_symlinks.so\" node\n```\n\nin most cases, this will do nothing\n\nthe `libnodejs_hide_symlinks.so` program will only become active,  \nwhen `require` or `import` try to resolve a symlink to `/nix/store/`\n\nthen, the `statx` syscall is intercepted,  \nso that the symlink-source appears as a regular directory (or file)\n\nthe following `open` syscalls are intercepted,  \nso that `open(symlink-source)` is replaced with `open(symlink-target)`\n\n### sample output\n\n\nwith `libnodejs_hide_symlinks.so`\n\n\u003cpre\u003e\n\n\u003cspan color=\"turquoise\"\u003e$\u003c/span\u003e pwd\n\n/tmp/test-project\n\n\u003cspan color=\"turquoise\"\u003e$\u003c/span\u003e LD_PRELOAD=/tmp/nodejs-hide-symlinks/target/debug/libnodejs_hide_symlinks.so \\\nnode node_modules/cowsay/cli.js moooooh\n\nnodejs-hide-symlinks init /tmp/test-project\nnodejs-hide-symlinks stat node_modules\u003cspan color=\"green\"\u003e/\u003c/span\u003e\nnodejs-hide-symlinks stat node_modules/.pnpm/cowsay@1.5.0/node_modules/cowsay\u003cspan color=\"green\"\u003e/\u003c/span\u003e\nnodejs-hide-symlinks open node_modules/.pnpm/cowsay@1.5.0/node_modules/cowsay\u003cspan color=\"green\"\u003e/\u003c/span\u003epackage.json\nnodejs-hide-symlinks open node_modules/.pnpm/cowsay@1.5.0/node_modules/cowsay\u003cspan color=\"green\"\u003e/\u003c/span\u003ecli.js\nnodejs-hide-symlinks open node_modules\u003cspan color=\"green\"\u003e/\u003c/span\u003e.pnpm/cowsay@1.5.0/node_modules/yargs/package.json\nnodejs-hide-symlinks stat node_modules/.pnpm/yargs@15.4.1/node_modules/yargs\u003cspan color=\"green\"\u003e/\u003c/span\u003e\nnodejs-hide-symlinks open node_modules/.pnpm/yargs@15.4.1/node_modules/yargs\u003cspan color=\"green\"\u003e/\u003c/span\u003eindex.js\n[...]\nnodejs-hide-symlinks open node_modules/.pnpm/cowsay@1.5.0/node_modules/cowsay\u003cspan color=\"green\"\u003e/\u003c/span\u003elib/cows.js\nnodejs-hide-symlinks open node_modules/.pnpm/cowsay@1.5.0/node_modules/cowsay\u003cspan color=\"green\"\u003e/\u003c/span\u003elib/replacer.js\nnodejs-hide-symlinks open node_modules/.pnpm/cowsay@1.5.0/node_modules/cowsay\u003cspan color=\"green\"\u003e/\u003c/span\u003elib/faces.js\nnodejs-hide-symlinks open node_modules/.pnpm/cowsay@1.5.0/node_modules/cowsay\u003cspan color=\"green\"\u003e/\u003c/span\u003ecows/default.cow\n _________\n\u0026lt; moooooh \u0026gt;\n ---------\n        \\   ^__^\n         \\  (oo)\\_______\n            (__)\\       )\\/\\\n                ||----w |\n                ||     ||\n\n\u003c/pre\u003e\n\nleft of the green slashes, are the \"fake directories\",\nwhich are transparently resolved into the `/nix/store`\n\nfor example\n\n```\n$ readlink node_modules/.pnpm/cowsay@1.5.0/node_modules/cowsay\n\n/nix/store/4qjr3svb3dmmq2b2yn69y3wlz453wldn-cowsay-1.5.0.tgz-unpacked\n```\n\n### sample error\n\nall this is needed, because nodejs would follow the symlink into the `/nix/store`,  \nbut there, the package's sub-dependencies cannot be located\n\nwithout `libnodejs_hide_symlinks.so`\n\n```\n$ node node_modules/cowsay/cli.js moooooh\n\nnode:internal/modules/cjs/loader:936\n  throw err;\n  ^\n\nError: Cannot find module 'yargs'\nRequire stack:\n- /nix/store/4qjr3svb3dmmq2b2yn69y3wlz453wldn-cowsay-1.5.0.tgz-unpacked/cli.js\n```\n\n### symlinked node_modules\n\nwe stop the symlink-resolution one step before the `/nix/store`,  \nso that nodejs can resolve sub-dependencies in the local `node_modules` folder\n\n```\n$ ls -l -a node_modules/\n\ndr-xr-xr-x 35 root root 4096 Jan  1  1970 .pnpm\nlrwxrwxrwx  1 root root   38 Jan  1  1970 cowsay -\u003e .pnpm/cowsay@1.5.0/node_modules/cowsay\n\n$ tree node_modules/.pnpm/ | head -n7\n\nnode_modules/.pnpm/\n├── ansi-regex@3.0.0\n│   └── node_modules\n│       └── ansi-regex -\u003e /nix/store/a17g3kl3bb4gmwzjw9s9k4sz8k0zh4jx-ansi-regex-3.0.0.tgz-unpacked\n├── ansi-regex@5.0.1\n│   └── node_modules\n│       └── ansi-regex -\u003e /nix/store/xxmkcs4fyl4by41a9vpf8zanad9xj3pr-ansi-regex-5.0.1.tgz-unpacked\n```\n\nthis custom node_modules folder can be generated  \nwith a patched version of [npmlock2nix](https://github.com/nix-community/npmlock2nix)  \nand with the custom NPM installer [pnpm-install-only](https://github.com/milahu/pnpm-install-only)\n\n```nix\n# internal.nix\n\nrec {\n\n  # add\n  unpackNpmTgz = { url, hash }:\n    stdenv.mkDerivation {\n      #name = builtins.elemAt (builtins.match \"^(.+)\\.tgz$\" (builtins.baseNameOf url)) 0;\n      name = \"${builtins.baseNameOf url}-unpacked\";\n      src = fetchurl { inherit url hash; }; # cache the *.tgz file in /nix/store\n      phases = \"unpackPhase installPhase\";\n      installPhase = ''\n        cd ..\n        mv package $out\n      '';\n    };\n\n  # replace fetchurl with unpackNpmTgz\n  makeSource = sourceHashFunc: name: dependency:\n    # ...\n    if dependency ? resolved \u0026\u0026 dependency ? integrity then\n      #dependency // { resolved = \"file://\" + (toString (fetchurl (makeSourceAttrs name dependency))); }\n      dependency // { resolved = \"file://\" + (toString (unpackNpmTgz (makeSourceAttrs name dependency))); }\n    else /* ... */ null;\n\n  # add\n  pnpm_install_only = fetchFromGitHub {\n    # https://github.com/milahu/pnpm-install-only\n    repo = \"pnpm-install-only\";\n    owner = \"milahu\";\n    rev = \"TODO\";\n    sha256 = \"TODO\";\n  };\n\n  # in the buildPhase of node_modules, replace \"npm install\"\n  node_modules__buildPhase = ''\n          #npm install --offline --nodedir=${nodeSource nodejs}\n          \n          export NODE_preInstallLinks='${builtins.toJSON preInstallLinks}'\n          node --trace-uncaught --trace-warnings ${pnpm_install_only}/dist/index.js || {\n            echo \"ERROR failed to install NPM packages\"\n            exit 1\n          }\n        '';\n}\n```\n\n## related\n\nunmerged PR at https://github.com/nodejs/node/pull/10132\n\nhttps://github.com/nodejs/node/issues/10107\n\nhttps://github.com/nodejs/node-eps/issues/46#issuecomment-265227695\n\n\u003e Using Machine Level stores while keep dependency version resolution coupled to a given /node_modules root.\n\nhttps://github.com/nodejs/node-eps/issues/46#issuecomment-266173249\n\n\u003e \u003e When a module is found, its node_modules hierarchy includes `\u003cpath\u003e/node_modules` as well as `\u003cpath\u003e+node_modules` for all the elements in its path, starting with the path.dirname() of where it is found.\n\u003e\n\u003e Correct. However, I consider this the anm enhancement, and an entirely separate thing from 1 \u0026 2, which are just about working with symlinks generally. anm has no idea if symlinks are involved or not, it just makes using them to machine stores possible. They really should be seen is two discreet things.\n\nhttps://github.com/nodejs/node-eps/issues/46#issuecomment-266635299\n\n\u003e symlinks per se aren't needed to pull off machine-level stores. For instance, it would be an acceptable compromise to wrap commands with a shim, because npm already does this on Windows. If we're going to need to wrap top-level applications in a shim regardless, we might as well use that shim to monkey-patch module.require. Maybe everyone will boo and hiss at that idea, but I'm really a fan of keeping core lean, and being able to solve machine-level stores purely with userspace code appeals to me. Thoughts?\n\nhttps://github.com/nodejs/node-eps/issues/46#issuecomment-267643146\n\n\u003e \u003e Just to be clear, I mean this warning in the sense of \"That is going to be an exciting adventure!\", and I strongly encourage you to try it, if you've got the time and are in the mood for adventure. But it definitely won't be safe, so it shouldn't be something that npm or node-core try to do :)\n\u003e\n\u003e Thanks! And yes, it's precisely because it is crazy and out there that I'm going to try to do it in user space via monkey patching rather than by branching node core. It'll be fun. When/if I get a machine-store loader working I'll post back with a link.\n\nhttps://github.com/nodejs/node-eps/issues/46#issuecomment-277373566\n\n\u003cblockquote\u003e\n\nThe latest version of pnpm (which is 0.51.2) uses a global (machine) store and works without any changes in Node.js.\n\nWe did a lot of tweaks to make it work, but the main ones are:\n\n* files are linked (not symlinked) from the global store to the project's node_modules\n* command shims are rewritten to set the NODE_PATH env variable before running the binstubs.\n\nSo it is achievable to create a global store without changes in Node.js and without --preserve-symlinks. And performance is good enough with linking.\n\n\u003c/blockquote\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmilahu%2Fnodejs-hide-symlinks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmilahu%2Fnodejs-hide-symlinks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmilahu%2Fnodejs-hide-symlinks/lists"}