{"id":31879404,"url":"https://github.com/fxamacker/draft-notes-about-deduplicating-public-keys","last_synced_at":"2026-02-18T18:32:56.185Z","repository":{"id":317754890,"uuid":"1068125776","full_name":"fxamacker/draft-notes-about-deduplicating-public-keys","owner":"fxamacker","description":"Draft notes about recent work deduplicating public keys","archived":false,"fork":false,"pushed_at":"2025-10-29T18:20:28.000Z","size":82,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-13T09:19:26.309Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fxamacker.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-01T22:30:18.000Z","updated_at":"2025-10-29T18:20:32.000Z","dependencies_parsed_at":"2025-10-29T20:22:01.914Z","dependency_job_id":"063e922d-f017-410a-9f80-366acc8ca7e7","html_url":"https://github.com/fxamacker/draft-notes-about-deduplicating-public-keys","commit_stats":null,"previous_names":["fxamacker/draft-notes-about-deduplicating-public-keys"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/fxamacker/draft-notes-about-deduplicating-public-keys","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fxamacker%2Fdraft-notes-about-deduplicating-public-keys","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fxamacker%2Fdraft-notes-about-deduplicating-public-keys/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fxamacker%2Fdraft-notes-about-deduplicating-public-keys/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fxamacker%2Fdraft-notes-about-deduplicating-public-keys/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fxamacker","download_url":"https://codeload.github.com/fxamacker/draft-notes-about-deduplicating-public-keys/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fxamacker%2Fdraft-notes-about-deduplicating-public-keys/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29589579,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-18T16:55:40.614Z","status":"ssl_error","status_checked_at":"2026-02-18T16:55:37.558Z","response_time":162,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-13T00:29:06.188Z","updated_at":"2026-02-18T18:32:56.176Z","avatar_url":"https://github.com/fxamacker.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# draft-notes-about-deduplicating-public-keys\n\n## How We Reduced Payload Count to Cut 211M Hashes and 29 GB of State\n\nBy optimizing how we store account keys, we reduced execution state size by 28.8 GB (over 7 percent of the entire execution state). Perhaps more importantly, we did this by reducing the number of payloads, and we also reduced the payload count's growth rate.\n\nReducing the number of payloads matters because each payload adds different overhead (cpu, ram, disk, network) to different types of servers running databases, caches, execution state, indexers, etc. that handle payloads.\n\nAs just one example, MTrie (the execution state) incurs 192-288 bytes of overhead for each 72-78 byte payload that held an account key.  The overhead is larger than the payload size because each payload requires about 2-3 MTrie vertices (96 bytes each).  We reduced enough payloads for us to no longer need 210 million cryptographic hashes and the MTrie vertices containing them.\n\nTo put what we achieved into perspective, we found 77.6 million payloads with duplicate keys, but our improved data format reduces the number of payloads by 86.1 million without using compression (not a typo)!\n\nThese incremental improvements add up to have a cumulative impact on operational costs, uptime, and performance.  Although payload count increases daily, Execution Nodes, etc. can handle more payloads now compared to 9 months ago, without adding extra RAM.\n\nBeyond Execution Nodes, reducing payload counts and their overhead helps reduce hardware costs and energy consumption on various servers that handle payloads.  By improving resource utilization, we gain headroom to handle increased spikes in demand, reduce downtime, and reduce costs.  Memory use reduction on AN, EN, etc. are hard to measure since we are deploying unrelated projects at the same time, but we know we are loading 28.8 GB less execution state into RAM on EN, and memory usage reduction on EN in a prior migration project was reported to be a multiple of the state size reduction.\n\nUPDATE: Memory use reduction is still about -150 GB less RAM on each EN server about 7 days after the Oct. 22, 2025 mainnet spork.  This is over 5x MTrie size reduction! 🎉\n\nAt the same time, the change does not take anything away from end users. [...]\n\n## Scope\n\nThis project optimizes non-atree payloads to reduce the total number of payloads.  Specifically, this project only modifies a subset of non-atree payloads (account status and public key).\n\nMigration optimizes the existing payloads during spork.  Accounts that only have 1 account key (most mainnet accounts) cannot have duplicates, but they use a few bytes less in the new data format.\n\nRuntime optimizes new payloads by detecting duplicate pubic keys being added, and will store them in the new deduplicated format.  Unlike migration, runtime duplicate key detection is not 100% as a tradeoff for faster speed and more efficient storage overall.\n\nNOTE:  To avoid sacrificing speed, this project does not use compression libraries (e.g., LZ4, zstd, etc.).\n\n## Goals\n\nOur goal is to improve efficiency of databases, caches, indexers, MTrie, etc. that handle payloads on various nodes by reducing the number of payloads.\n\nReducing the number of payloads also potentially benefits 3rd-party servers if they handle payloads.\n\nBy improving resource utilization and gaining headroom to handle spikes in demand, etc. we can reduce risk of downtime and reduce operational costs.\n\nThis project adds to the cumulative impact of reducing payload counts.  This project exceeded each of our quantifiable goals.\n\n\u003cdetails\u003e\u003csummary\u003e 🔍 Expand for more detailed goals\u003c/summary\u003e\n\n#### Initial goals:\n- remove ~46% of all public keys stored\n- reduce memory and storage used by ~19 GB\n- reduce registers stored by ~65M\n\n#### Updated goals based on newer mainnet data and clarification about RAM:\n- remove ~48% of all public keys stored\n- reduce state size and disk used by ~20 GB each\n- reduce number of registers by ~68 million\n- reduce RAM used by EN is TBD after deployment to mainnet.  Memory use on Execution Nodes can sometimes reduce by ~3x state size reduction but that isn't guaranteed due to external factors like Go garbage collection, unrelated components/processes, OS page cache, etc.\n\n\u003c/details\u003e\n\n## Actual Results (Oct. 22, 2025 Mainnet Spork)\n\n|                  | Reduction | Notes |\n| ---------------------- |--------| --- |\n| Public Keys | 77.6 million (53.1%) | better than our goal of 46-48% 🎉 |\n| Payloads (aka Registers) | 86.1 million (16%) | better than our goal of 65-68 million 🎉 |\n| Payload Size | 8.9 GB | not estimated but better than private expectations 🎉 |\n| MTrie Vertices | 210 million (16%) | matches expected 2x-3x payload count reduction 🎉 |\n| MTrie Size | 28.8 GB (7.2%) | better than our goal of 19-20 GB 🎉 |\n| EN Checkpoint Size | 21.7 GB (6.2%) | better than our goal of 19-20 GB 🎉 |\n| EN RAM Usage | ~150 GB | not estimated but better than private expectations 🎉 |\n| DB, caches, indexers, etc. | TBD | components handling payloads on AN, EN, etc. |\n\nThe ~150 GB RAM reduction on each EN server (7 days after spork as of today) exceeded expectations.\n\nSpeedup is also better than expected, but multiple changes were deployed at the same time, so it is hard to separate.\n\n\u003cdetails\u003e\u003csummary\u003e 🔍 Expand for details\u003c/summary\u003e\n\n#### Mainnet Spork (Oct. 22, 2025)\n\nNOTE: After spork, the reverse migration test result (new -\u003e old) matched the regular migration (old -\u003e new) result.\n\nMTrie size reduction (bytes):\n- payloads: 8587120345\n- vertices: 20226276288\n- total: 28813396633 (28.8 GB)\n\nCheckpoint file size:\n- old format: 354914430454 bytes (created by reverse migration from new -\u003e old for comparison)\n- new format: 333167435599 bytes\n- reduction: 21746994855 bytes (27.1 GB)\n\nMTrie node (aka vertex) count:\n- before: 1323678476\n- after: 1112988098\n- count reduction: 210690378 (211M cryptographic hashes!)\n- size reduction (count * 96 bytes): 20226276288\n\nPayload count:\n- before: 541888998\n- after: 455650954\n- reduction: 86238044\n\nPayload size:\n- before: 272233935818\n- after: 263646815473\n- reduction (bytes): 8587120345\n\n\u003c/details\u003e\n\n### Migration Speed\n\nThe key deduplication part of the migration only took 5m36s using `nworkers=10` on m1 (within the longer running common migration framework of loading/saving checkpoints, etc. which unchanged from prior migrations).  The speed of migration in pre-spork tests using `nworkers=64` allowed us to use m1 instead of m3 server for the mainnet spork on Oct. 22, 2025 🎉.\n\n## Results Based on October 1, 2025 Mainnet Checkpoint\n\nNOTE: The new data format is designed to reduce payload count by more than the total number of duplicate keys.\n\nThere are ~77.6 million duplicate payloads and migration reduces payload count by 86.1 million payloads (not a typo).\n\nMost accounts on mainnet only have 1 account key but they still use fewer bytes in the new data format.\n\n|                  | Reduction | Notes |\n| ---------------------- |--------| --- |\n| Public Keys | 77.6 million (53.1%) | better than our goal of 46-48% 🎉 |\n| Payloads (aka Registers) | 86.1 million (16%) | better than our goal of 65-68 million 🎉 |\n| MTrie Vertices | 210 million (16%) | matches expected 2x-3x payload count reduction 🎉 |\n| MTrie Size | 28.8 GB (7.2%) | better than our goal of 19-20 GB 🎉 |\n| EN Checkpoint Size | 21.7 GB (6.2%) | better than our goal of 19-20 GB 🎉 |\n| EN RAM Usage | TBD | never estimate (can be affected by other changes) |\n| DB, caches, indexers, etc. | TBD | components handling payloads on AN, EN, etc. |\n\n\u003cdetails\u003e\u003csummary\u003e 🔍 Expand for more details.\u003c/summary\u003e\n\n|                  | Before | After | Reduction |\n| ---------------------- |--------|-------|------------|\n| Public Keys | 146,056,652 | 68,504,671 | 77,551,981 |\n| Payloads (aka Registers)   |  539,650,919 | 453,516,554 | 86,134,365 |\n| MTrie Vertices | 1,318,213,108 | 1,107,774,108 | 210,439,000 |\n| MTrie Size (bytes) | 397,495,799,485 | 368,718,670,098 | 28,777,129,387 |\n| EN Checkpoint Size (bytes) | 353,300,063,477 | 331,567,299,166 | 21,732,764,311 |\n| EN RAM Usage | TBD | TBD | never estimate (can be affected by other changes) |\n| DBs, caches, indexers, etc. | | | TBD on AN, EN, etc. |\n\nEN state size reduction:\n- before: 131821310896 + 270947341117 = 397495799485 bytes\n- after: 110777410896 + 262372355730 = 368718670098 bytes\n\n\u003c/details\u003e\n\nMTrie is the in-memory data structure containing the execution state. MTrie has vertices and payloads (atree and non-atree payloads).\n\nThis project only modifies non-atree payloads (account status and public key payloads).\n\nIf the Execution Nodes (Servers) RAM usage are reduced by less than the reduction in MTrie size, then it is likely that unrelated changes or activity is consuming extra RAM (vm configuration, OS page cache, db memory mapped files, updated components, etc.).  For example, some types of garbage collectors and databases might hold on to more RAM (to improve performance) if they detect more free RAM is available.\n\n## Design and Implementation\n\nI designed the new data format to reduce payload count by more than the total number of duplicate payloads.\n\nAdditionally, I wanted to make the new data format support efficient runtime duplicate key detection (when new keys are added by accounts).\n\nAnd since most accounts only have 1 account key (no duplicate public keys possible), the design special cases those accounts to avoid adding overhead.\n\n### Design of New Data Format\n\n### Efficiently Detecting Duplicates At Runtime\n\n### Encoding Formats\n\nThe new data format uses:\n- RLP encoding (for the data already encoded in RLP in the old format)\n- Raw bytes (for very simple new data where using a codec would be overkill)\n- RLE++ (for efficiently encoding both repeating values and non-repeating values)\n\n### RLE++ Encoding (I don't know if I'm the first person to invent this encoding)\n\nThe new data format uses a new encoding called RLE++ to efficiently encode both repeating values and non-repeating values in the same encoding.\n\nI named it RLE++ because it adds a feature to RLE ([run-length encoding](https://en.wikipedia.org/wiki/Run-length_encoding)) that supports efficient encoding of non-repeating values created by using the ++ operator (incrementing values).\n\n## Results of Optimized Migration Speed\n\n## Results of Optimized Runtime Storage to Reduce Need for Future Migrations\n\n## Work Done\n\nThis list is not exhaustive and it doesn't show time spent on design, getting familiar with existing codebase, optimizing migration speed before opening first PR, or extra testing (beyond unit tests and integration tests).\n\n- https://github.com/onflow/flow-go/pull/7738 Add account public key deduplication migration\n- https://github.com/onflow/flow-go/pull/7829 Add runtime public key duplicate detection and storage with support for account status v4 format\n- https://github.com/onflow/flow-go/pull/7834 Fix the RegisterSize() function\n- https://github.com/onflow/flow-go/pull/7835 Change report format for storage used migration from JSON to CSV\n- https://github.com/onflow/flow-go/pull/7841 Reduce register reads by using Accounts.GetRuntimeAccountPublicKey\n- https://github.com/onflow/flow-go/pull/7875 Create diff-keys utility to compare account public keys before and after migration\n- https://github.com/onflow/flow-go/pull/7878 Add option to validate account public key migration\n- https://github.com/onflow/flow-go/pull/7878 Support account status v4 in checkpoint-collect-stats util\n\n## Migration and Runtime Tests\n\n### Beyond Unit Tests and Integration Tests\n\nTests passed using older mainnet and testnet snapshots, but I test again when I need to get newer snapshots for something else. For example, here are some tests running last night using Oct 1, 2025 snapshot:\n\n\u003cimg width=\"3833\" height=\"780\" alt=\"image\" src=\"https://github.com/user-attachments/assets/e3dc931d-6d95-4bb9-b0ce-c26843ec9cbf\" /\u003e\n\n### Tests Used All Account Keys From Mainnet and Testnet\n\nFrom a storage and retrieval perpective, I tested [PR 7829](https://github.com/onflow/flow-go/pull/7829) for runtime public key handling with the same level of testing I used for the migration ([PR #7738](https://github.com/onflow/flow-go/pull/7738)).  I essentially round-trip tested every account key from both mainnet and testnet to get the expected results.\n\nDuring runtime tests, every account key from a mainnet and testnet snapshot were individually added and the stored results were compared to each corresponding account key that was stored in the original mainnet and testnet snapshot.\n\nThese extra tests confirmed the deduplicated public keys provide access to all the same account key data that existed prior to deduplication, such as the individual weight and sequence number of each previously duplicate key.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffxamacker%2Fdraft-notes-about-deduplicating-public-keys","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffxamacker%2Fdraft-notes-about-deduplicating-public-keys","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffxamacker%2Fdraft-notes-about-deduplicating-public-keys/lists"}