{"id":19079227,"url":"https://github.com/weavevm/wvm-archiver","last_synced_at":"2025-07-19T11:13:55.299Z","repository":{"id":252679119,"uuid":"840338030","full_name":"weaveVM/wvm-archiver","owner":"weaveVM","description":"WeaveVM Archiver is an ETL archive pipeline for EVM networks. It's the simplest way to interface with WeaveVM's permanent data feature without smart contract redeployments","archived":false,"fork":false,"pushed_at":"2025-03-28T11:10:07.000Z","size":159,"stargazers_count":6,"open_issues_count":0,"forks_count":9,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-04-30T05:41:44.227Z","etag":null,"topics":["archiver","etl"],"latest_commit_sha":null,"homepage":"https://docs.wvm.dev","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/weaveVM.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-08-09T13:32:51.000Z","updated_at":"2025-03-28T11:10:11.000Z","dependencies_parsed_at":"2024-08-11T19:12:59.819Z","dependency_job_id":"49f78ae6-eee3-4e30-a150-98663bdbfca4","html_url":"https://github.com/weaveVM/wvm-archiver","commit_stats":null,"previous_names":["weavevm/wvm-archiver"],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/weaveVM/wvm-archiver","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weaveVM%2Fwvm-archiver","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weaveVM%2Fwvm-archiver/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weaveVM%2Fwvm-archiver/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weaveVM%2Fwvm-archiver/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/weaveVM","download_url":"https://codeload.github.com/weaveVM/wvm-archiver/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weaveVM%2Fwvm-archiver/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259774602,"owners_count":22909164,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["archiver","etl"],"created_at":"2024-11-09T02:13:54.118Z","updated_at":"2025-06-14T06:34:25.895Z","avatar_url":"https://github.com/weaveVM.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://wvm.dev\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/weaveVM/.github/main/profile/bg.png\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n## About\nWeaveVM Archiver is an ETL archive pipeline for EVM networks. It's the simplest way to interface with WeaveVM's permanent data feature without smart contract redeployments.\n\n### WeaveVM Archiver Usage\n\nWeaveVM Archiver is the ideal choice if you want to:\n\n- Interface with WeaveVM's permanent data settlement and high-throughput DA\n- Maintain your current data settlement or DA architecture\n- Have an interface with WeaveVM without rollup smart contract redeployments\n- Avoid codebase refactoring\n\n## Build \u0026 Run\n\n```bash\ngit clone https://github.com/weaveVM/wvm-archiver.git\n\ncd wvm-archiver\n\nshuttle run\n```\n\n### Prerequisites \u0026 Dependencies\n\nWhile a WeaveVM Archiver node can run without web2 component dependencies, this node implementation uses [planetscale](https://planetscale.com) for cloud indexing (indexing target network block ID to WVM archive TXID) and [shuttle.dev](https://shuttle.dev) for backend hosting. Check [.env.example](./env.example) to set up your environment variables.\n\n```js\narchiver_pk=\"\" // WeaveVM archiver PK\nbackfill_pk=\"\" // WeaveVM backfill PK\nbackfill_start_block=\"0\" // it defaults to 0 (genesis), but it's dynamic, so you can specify from which block number you want to start backfilling\nnetwork=\"./networks/your_network.json\"\nps_livesync_table_name=\"LivesyncTableName\"\nps_backfill_table_name=\"BackfillTableName\"\n\nDATABASE_HOST=\"\" // planetscale\nDATABASE_USERNAME=\"\" // planetscale\nDATABASE_PASSWORD=\"\" // planetscale\n```\n\n***Read more about WeaveVM \u003c\u003e Shuttle teamup [here](https://www.shuttle.dev/blog/2025/01/17/shuttle-web3-weavevm)***\n\n### Add Your Network\n\nTo start archiving your network block data on WeaveVM:\n\n1. Add your network config file to the [networks](./networks/) directory.\n2. Name your config file using snake_case syntax (e.g., `your_network_name.json`).\n3. Modify properties that don't have a `wvm_` prefix in the config JSON file. Check [_template.json](./networks/_template.json) guide\n4. Fund your `archiver_address` \u0026 `backfill_address` with a sufficient amount of tWVM (1 MB costs ~ 5 cents). Check out [WVM Faucet](https://wvm.dev/faucet) to claim $tWVM. Make sure that the two addresses are distinct.\n5. Choose a unique `archive_pool_address` that's different from your `archiver_address` \u0026 `backfill_address`\n6. set `start_block` value to the most recent network's blockheight. That will facilitate the archiver to start in sync with live blockheight while, in parallel, reindexing from genesis using the `backfill_address`. \n7. Set up your PlanetScale DB according to `db_schema.sql`.\n\n#### Parallel Threads of Archiving\n\nAs mentioned previously, `archiver_address` is responsible for archiving blocks starting from the `start_block` specified in your `network.json` config file, while also keeping up with the network’s current blockheight (live sync). Meanwhile, `backfill_address` handles archiving blocks from `backfill_start_block` up to `start_block`.\n\n```txt\nbackfill thread: backfill_start_block -\u003e start_block\nlive sync thread: start_block -\u003e network's live blockheight\n```\n\n### RPC Proxy and Caching\n\nYou can use [eRPC](https://github.com/erpc/erpc) to cache, load-balance and failover between as many RPC endpoints and use eRPC's proxy URL in each network's config for WeaveVM. This will increase performance and resiliency and reduce RPC usage cost while fetching network's block data via WeaveVM.\n\n```bash\n# modify erpc.yaml\ncp erpc.yaml.dist erpc.yaml\ncode erpc.yaml\n\n# run docker-compose\ndocker-compose up -d\n```\n\nFinally, you can set eRPC's proxy URL in each relative network config.\n\n```optimism.json\n{\n    \"name\": \"Optimism\",\n    \"network_chain_id\": 10,\n    \"network_rpc\": \"http://erpc:4000/main/evm/10\",\n    ...\n}\n```\n\n## How it works\n\nThe WeaveVM Archiver node operates as follows:\n\n1. It starts downloading the target EVM network block data from the RPC you provide in the network config file.\n2. The node begins pulling blocks from the `start_block` defined in the network's config file.\n3. The block data is then serialized in [borsh](https://borsh.io) format and compressed using Brotli.\n4. The serialized-compressed data is pushed to WeaveVM as calldata transaction from the `archiver_address` \u0026 `backfill_address` to the `archive_pool_address`.\n5. Simultaneously, the resulting TXID from pushing data to WeaveVM and the archived EVM block ID are indexed in the cloud for faster data retrieval.\n\n## Server Methods\n\nAs mentioned, PlanetScale is used for cloud indexing, which allows a WeaveVM Archiver node to expose its WeaveVM data as a RESTful API.\n\n### WeaveVM Archiver node instance info\n\n```bash\ncurl -X GET https://the_network.wvm.network/v1/info\n```\n**returns:**\n\n```rs\npub struct InfoServerResponse {\n    first_livesync_archived_block: Option\u003cu64\u003e,\n    last_livesync_archived_block: Option\u003cu64\u003e,\n    first_backfill_archived_block: Option\u003cu64\u003e,\n    last_backfill_archived_block: Option\u003cu64\u003e,\n    livesync_start_block: u64,\n    total_archived_blocks: u64,\n    blocks_behind_live_blockheight: u64,\n    archiver_balance: U256,\n    archiver_address: String,\n    backfill_address: String,\n    backfill_balance: U256,\n    network_name: String,\n    network_chain_id: u32,\n    network_rpc: String,\n}\n```\n\n### WeaveVM Archiver all networks info:\n\n```bash\ncurl -X GET https://the_network.wvm.network/v1/all-networks-info\n```\n\n**returns:**\n\n```rs\nVec\u003cNetwork\u003e\n```\n\n### Retrieve the WVM archive TXID for a given EVM block ID\n\n```bash\ncurl -X GET https://the_network.wvm.network/v1/block/$BLOCK_ID\n```\n\n### Decode the WVM archived block data for a given EVM block ID (return original block data in JSON format)\n\n```bash\ncurl -X GET https://the_network.wvm.network/v1/block/raw/$BLOCK_ID\n```\n\n## License\nThis project is licensed under the [BSL 1.1 License](./LICENSE)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweavevm%2Fwvm-archiver","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fweavevm%2Fwvm-archiver","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweavevm%2Fwvm-archiver/lists"}