{"id":50319540,"url":"https://github.com/ryanleecode/papi-memory-repro","last_synced_at":"2026-05-29T02:30:34.400Z","repository":{"id":358766659,"uuid":"1242995452","full_name":"ryanleecode/papi-memory-repro","owner":"ryanleecode","description":null,"archived":false,"fork":false,"pushed_at":"2026-05-19T01:44:43.000Z","size":28,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-19T02:42:33.195Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ryanleecode.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-19T00:40:54.000Z","updated_at":"2026-05-19T01:44:47.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ryanleecode/papi-memory-repro","commit_stats":null,"previous_names":["ryanleecode/papi-memory-repro"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/ryanleecode/papi-memory-repro","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanleecode%2Fpapi-memory-repro","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanleecode%2Fpapi-memory-repro/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanleecode%2Fpapi-memory-repro/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanleecode%2Fpapi-memory-repro/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ryanleecode","download_url":"https://codeload.github.com/ryanleecode/papi-memory-repro/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanleecode%2Fpapi-memory-repro/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33634611,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-29T02:00:06.066Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-29T02:30:33.756Z","updated_at":"2026-05-29T02:30:34.395Z","avatar_url":"https://github.com/ryanleecode.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# `keys.dec()` allocates ~500x more memory than the raw SCALE data it decodes\n\n## Description\n\nCalling `typedApi.query.Resources.Consumers.getEntries()` on the Paseo People chain (~37k entries) causes a **2 GB RSS spike**. The same data decoded with manual hex-slicing of storage keys uses **120 MB** — a 17x difference.\n\nThe root cause is `keys.dec()` in `storage.js` line 159. It's called inside a synchronous `.map()` over all entries with no GC yield points:\n\n```js\nconst decodedValues = values.map(({ key, value }) =\u003e ({\n  keyArgs: codecs.keys.dec(key), // ← ~57 KB of transient allocations per key\n  value: codecs.value.dec(value),\n}))\n```\n\nEach `keys.dec()` call allocates ~57 KB of V8 objects to decode an 81-byte SCALE key (a single `AccountId32`). The allocation chain includes:\n\n1. `fromHex` → `toInternalBytes` creates 2x `InternalUint8Array` + `DataView`\n2. `BytesDec` does `buffer.slice` copies\n3. `codec.enc(result[i]).length` in `dynamic-builder.js` re-encodes the decoded key just to measure byte length (the hasher/key sizes are compile-time constants)\n4. SS58 encoding allocates trie nodes in `withSs58Cache`\n\n## Reproduction\n\nClone the repo and run:\n\n```bash\npnpm install\nnode --expose-gc --import tsx profile-typed.ts      # typed API — shows 2 GB spike\nnode --expose-gc --import tsx profile-optimized.ts   # manual decode — shows 120 MB\n```\n\n### Typed API output\n\n```\n[baseline                    ] RSS:   108.5 MB (+0.0 MB)\n[typed api created           ] RSS:   114.5 MB (+6.0 MB)\n\nCalling typedApi.query.Resources.Consumers.getEntries()...\n\n[before getEntries()         ] RSS:   115.8 MB (+7.3 MB)\n[after getEntries()          ] RSS:  2176.7 MB (+2068.2 MB)  returned 37035 entries\n[after gc()                  ] RSS:  2178.5 MB (+2069.9 MB)\n\n============================================================\n  SUMMARY\n============================================================\n  Chain:                   Paseo People\n  Storage:                 Resources.Consumers\n  Total entries:           37035\n  getEntries() RSS delta:  2060.9 MB\n  Per-entry overhead:      ~57.0 KB\n  Serialized result size:  11.2 MB\n============================================================\n```\n\n### Optimized output (manual hex-slice, same data)\n\n```\n[baseline                    ] RSS:   114.7 MB (+0.0 MB)\n[client created              ] RSS:   120.7 MB (+6.0 MB)\n[codecs ready                ] RSS:   138.3 MB (+23.6 MB)\n\nFetching Resources.Consumers with manual key decode...\n\n[before fetch                ] RSS:   138.4 MB (+23.7 MB)\n[after fetch                 ] RSS:   258.7 MB (+144.0 MB)  returned 37037 entries\n[after gc()                  ] RSS:   258.5 MB (+143.9 MB)\n\n============================================================\n  SUMMARY\n============================================================\n  Chain:                   Paseo People\n  Storage:                 Resources.Consumers\n  Total entries:           37037\n  Manual decode RSS delta: 120.3 MB\n  Per-entry overhead:      ~3.3 KB\n  Serialized result size:  10.8 MB\n============================================================\n```\n\n## The numbers\n\nBoth scripts produce the same decoded result. `v8.serialize()` confirms the final data is ~11 MB in both cases.\n\n| | Serialized result | RSS delta | Ratio |\n|---|---|---|---|\n| `getEntries()` (typed API) | 11.2 MB | 2,061 MB | **184x** the actual data |\n| Manual hex-slice + `value.dec()` | 10.8 MB | 120 MB | **11x** the actual data |\n\nThe 11 MB is the real cost of holding 37k decoded entries as JS objects. The 120 MB floor includes V8 object overhead (hidden classes, property arrays, string backing stores). The remaining ~1.9 GB on top is transient `keys.dec()` garbage that can never be collected — `getEntries()` calls `keys.dec()` inside a synchronous `.map()` with no GC yield points, so all 37k decode passes' allocations accumulate before V8 can reclaim anything.\n\n## Environment\n\n- `polkadot-api`: 2.1.0\n- `@polkadot-api/metadata-builders`: 0.14.1\n- `@polkadot-api/substrate-bindings`: 0.20.1\n- Node.js: 24+\n- Chain: Paseo People (`wss://paseo-people-next-rpc.polkadot.io`)\n- Storage: `Resources.Consumers` (~37k entries, Blake2_128Concat hasher, single `AccountId32` key)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fryanleecode%2Fpapi-memory-repro","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fryanleecode%2Fpapi-memory-repro","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fryanleecode%2Fpapi-memory-repro/lists"}