{"id":22771583,"url":"https://github.com/weavevm/substrate-archiver","last_synced_at":"2026-02-08T10:31:29.773Z","repository":{"id":267170441,"uuid":"898121968","full_name":"weaveVM/substrate-archiver","owner":"weaveVM","description":"Substrate Archiver is an ETL archive pipeline for Substrate networks. It's the simplest way to interface with WeaveVM's permanent data feature without smart contract redeployments","archived":false,"fork":false,"pushed_at":"2025-01-14T23:40:23.000Z","size":125,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-04-30T03:32:21.489Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/weaveVM.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-03T20:35:19.000Z","updated_at":"2025-01-14T23:40:27.000Z","dependencies_parsed_at":"2025-02-05T13:52:06.612Z","dependency_job_id":"9b05add6-f71d-4f1c-9130-b46a59cffbbd","html_url":"https://github.com/weaveVM/substrate-archiver","commit_stats":null,"previous_names":["weavevm/substrate-archiver"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/weaveVM/substrate-archiver","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weaveVM%2Fsubstrate-archiver","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weaveVM%2Fsubstrate-archiver/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weaveVM%2Fsubstrate-archiver/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weaveVM%2Fsubstrate-archiver/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/weaveVM","download_url":"https://codeload.github.com/weaveVM/substrate-archiver/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weaveVM%2Fsubstrate-archiver/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29227729,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-08T09:43:19.170Z","status":"ssl_error","status_checked_at":"2026-02-08T09:42:55.556Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-11T16:14:18.723Z","updated_at":"2026-02-08T10:31:29.755Z","avatar_url":"https://github.com/weaveVM.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://wvm.dev\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/weaveVM/.github/main/profile/bg.png\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n## About\nSubstrate Archiver is an ETL archive pipeline for [Substrate](https://substrate.io/) networks. It's the simplest way to interface with WeaveVM's permanent data feature without smart contract redeployments. Check [WeaveVM Archiver](https://github.com/weaveVM/wvm-archiver) for EVM networks archive pipeline.\n\n### WeaveVM Archiver Usage\n\nSubstrate Archiver is the ideal choice if you want to:\n\n- Interface with WeaveVM's permanent data settlement and high-throughput DA\n- Maintain your current data settlement or DA architecture\n- Have an interface with WeaveVM without rollup smart contract redeployments\n- Avoid codebase refactoring\n\n## Build \u0026 Run\n\n```bash\ngit clone https://github.com/weaveVM/substrate-archiver.git\n\ncd substrate-archiver\n\ncargo shuttle run\n```\n\n### Prerequisites \u0026 Dependencies\n\nWhile a WeaveVM Substrate node can run without web2 component dependencies, this node implementation uses [planetscale](https://planetscale.com) for cloud indexing (indexing target network block ID to WVM archive TXID) and [shuttle.rs](https://shuttle.rs) for backend hosting. Check [.env.example](./env.example) to set up your environment variables.\n\n```js\narchiver_pk=\"\" // WeaveVM archiver PK\nbackfill_pk=\"\" // WeaveVM backfill PK\nbackfill_start_block=\"0\" // it defaults to 0 (genesis), but it's dynamic, so you can specify from which block number you want to start backfilling\nnetwork=\"./networks/your_network.json\"\nps_livesync_table_name=\"LivesyncTableName\"\nps_backfill_table_name=\"BackfillTableName\"\n\nDATABASE_HOST=\"\" // planetscale\nDATABASE_USERNAME=\"\" // planetscale\nDATABASE_PASSWORD=\"\" // planetscale\n```\n\n### Add Your Network\n\nTo start archiving your network block data on WeaveVM:\n\n1. Add your network config file to the [networks](./networks/) directory.\n2. Name your config file using snake_case syntax (e.g., `your_network_name.json`).\n3. Modify properties that don't have a `wvm_` prefix in the config JSON file. Check [_template.json](./networks/_template.json) guide\n4. Fund your `archiver_address` \u0026 `backfill_address` with a sufficient amount of tWVM (1 MB costs ~ 5 cents). Check out [WVM Faucet](https://wvm.dev/faucet) to claim $tWVM. Make sure that the two addresses are distinct.\n5. Choose a unique `archive_pool_address` that's different from your `archiver_address` \u0026 `backfill_address`\n6. set `start_block` value to the most recent network's blockheight. That will facilitate the archiver to start in sync with live blockheight while, in parallel, reindexing from genesis using the `backfill_address`. \n7. Set up your PlanetScale DB according to `db_schema.sql`.\n\n#### Parallel Threads of Archiving\n\nAs mentioned previously, `archiver_address` is responsible for archiving blocks starting from the `start_block` specified in your `network.json` config file, while also keeping up with the network’s current blockheight (live sync). Meanwhile, `backfill_address` handles archiving blocks from `backfill_start_block` up to `start_block`.\n\n```txt\nbackfill thread: backfill_start_block -\u003e start_block\nlive sync thread: start_block -\u003e network's live blockheight\n```\n\n## How it works\n\nThe WeaveVM Substrate node operates as follows:\n\n1. It starts downloading the target substrate network block data from the RPC you provide in the network config file.\n2. The node begins pulling blocks from the `start_block` defined in the network's config file.\n3. The block data is then serialized in [borsh](https://borsh.io) format and compressed using Brotli.\n4. The serialized-compressed data is pushed to WeaveVM as calldata transaction from the `archiver_address` \u0026 `backfill_address` to the `archive_pool_address`.\n5. Simultaneously, the resulting TXID from pushing data to WeaveVM and the archived substrate block ID are indexed in the cloud for faster data retrieval.\n\n## Server Methods\n\nAs mentioned, PlanetScale is used for cloud indexing, which allows a WeaveVM Archiver node to expose its WeaveVM data as a RESTful API.\n\n### WeaveVM Archiver node instance info\n\n```bash\ncurl -X GET https://the_network.wvm.network/v1/info\n```\n**returns:**\n\n```rs\npub struct InfoServerResponse {\n    first_livesync_archived_block: Option\u003cu64\u003e,\n    last_livesync_archived_block: Option\u003cu64\u003e,\n    first_backfill_archived_block: Option\u003cu64\u003e,\n    last_backfill_archived_block: Option\u003cu64\u003e,\n    livesync_start_block: u64,\n    total_archived_blocks: u64,\n    blocks_behind_live_blockheight: u64,\n    archiver_balance: U256,\n    archiver_address: String,\n    backfill_address: String,\n    backfill_balance: U256,\n    network_name: String,\n    network_rpc: String,\n}\n```\n\n### WeaveVM Archiver all networks info:\n\n```bash\ncurl -X GET https://the_network.wvm.network/v1/all-networks-info\n```\n\n**returns:**\n\n```rs\nVec\u003cNetwork\u003e\n```\n\n### Retrieve the WVM archive TXID for a given substrate block ID\n\n```bash\ncurl -X GET https://the_network.wvm.network/v1/block/$BLOCK_ID\n```\n\n### Decode the WVM archived block data for a given substrate block ID (return original block data in JSON format)\n\n```bash\ncurl -X GET https://the_network.wvm.network/v1/block/raw/$BLOCK_ID\n```\n\n## License\nThis project is licensed under the [BSL 1.1 License](./LICENSE)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweavevm%2Fsubstrate-archiver","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fweavevm%2Fsubstrate-archiver","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweavevm%2Fsubstrate-archiver/lists"}