{"id":13504189,"url":"https://github.com/RCasatta/blocks_iterator","last_synced_at":"2025-03-29T19:30:45.774Z","repository":{"id":38204829,"uuid":"360521076","full_name":"RCasatta/blocks_iterator","owner":"RCasatta","description":"Iterate over bitcoin blocks","archived":false,"fork":false,"pushed_at":"2025-01-19T17:26:39.000Z","size":557,"stargazers_count":57,"open_issues_count":14,"forks_count":8,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-24T08:51:39.213Z","etag":null,"topics":["bitcoin","blocks","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RCasatta.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-22T13:02:52.000Z","updated_at":"2025-02-06T14:52:49.000Z","dependencies_parsed_at":"2024-01-18T23:44:14.894Z","dependency_job_id":"b3fa705c-a75a-4f01-8a9b-f4c197bf8bf5","html_url":"https://github.com/RCasatta/blocks_iterator","commit_stats":{"total_commits":234,"total_committers":3,"mean_commits":78.0,"dds":"0.047008547008547064","last_synced_commit":"7b419419b917572a127b4240790ef874a4d4b913"},"previous_names":[],"tags_count":26,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RCasatta%2Fblocks_iterator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RCasatta%2Fblocks_iterator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RCasatta%2Fblocks_iterator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RCasatta%2Fblocks_iterator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RCasatta","download_url":"https://codeload.github.com/RCasatta/blocks_iterator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246100539,"owners_count":20723478,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitcoin","blocks","rust"],"created_at":"2024-07-31T23:00:59.460Z","updated_at":"2025-03-29T19:30:45.768Z","avatar_url":"https://github.com/RCasatta.png","language":"Rust","funding_links":[],"categories":["Awesome Rust Bitcoin"],"sub_categories":["Rust Bitcoin Cookbook"],"readme":"[![MIT license](https://img.shields.io/github/license/RCasatta/blocks_iterator)](https://github.com/RCasatta/blocks_iterator/blob/master/LICENSE)\n[![Crates](https://img.shields.io/crates/v/blocks_iterator.svg)](https://crates.io/crates/blocks_iterator)\n[![Docs](https://img.shields.io/badge/docs.rs-blocks_iterator-green)](https://docs.rs/blocks_iterator)\n\n# Blocks iterator\n\nIterates over Bitcoin blocks, decoding data inside Bitcoin Core's blocks directory.\n\nFeatures:\n* Blocks are returned in height order, it avoids following reorgs (see [`Config::max_reorg`] parameter)\n* Blocks come with extra data [`BlockExtra`] like all block's previous outputs, it allows computing\ntransactions fee or [verifying](https://github.com/RCasatta/blocks_iterator/blob/master/cli/examples/verify.rs)\nscripts in the blockchain.\n\nNote:\n\nBitcoin Core 28.0 introduced xoring of bitcoin blocks and this project doesn't yet support reading the blocks directory when xored. You can disable xoring in core via `-blocksxor=0`.\n\n\n## Iteration modes\n\n### In rust programs\n\nUsed as a library blocks could be iterated via the [`iter()`] method like:\n\n```rust\n// \"blocks\" dir contains first 400 testnet blocks\nlet conf = blocks_iterator::Config::new(\"../blocks\", bitcoin::Network::Testnet);\nlet mut total_fee = 0u64;\nfor b in blocks_iterator::iter(conf) {\n  total_fee += b.fee().expect(\"fee available cause we are keeping prevouts\");\n}\n\n// Only a bunch of tx with fee exists on testnet with height \u003c 400\n// in blocks: 385, 387, 389, 390, 392, 394\nassert_eq!(total_fee, 450_000u64);\n```\n\nWhen the task to be performed is computational costly, like verifying spending conditions, it is\nsuggested to parallelize the execution like it's done with rayon (or similar) in the\n[verify](https://github.com/RCasatta/blocks_iterator/blob/master/cli/examples/verify.rs) example\n(note `par_bridge()` call).\n\n### Through Pipes\n\nOther than inside Rust programs, ordered blocks with previous outputs could be iterated using Unix pipes.\n\n```sh\n$ cargo build --release\n$ cargo build --release --examples\n$ ./target/release/blocks_iterator_cli --blocks-dir ~/.bitcoin/testnet3/blocks --network testnet --max-reorg 40 --stop-at-height 200000 | ./target/release/examples/with_pipe\n...\n[2023-03-31T15:01:23Z INFO  with_pipe] Max number of txs: 6287 block: 0000000000bc915505318327aa0f18568ce024702a024d7c4a3ecfe80a893d6c\n[2023-03-31T15:01:23Z INFO  with_pipe] total missing reward: 50065529986 in 100 blocks\n[2023-03-31T15:01:23Z INFO  with_pipe] most_output tx is 640e22b5ddee1f6d2d701e37877027221ba5b36027634a2e3c3ee1569b4aa179 with #outputs: 10001\n```\n\nIf you have more consumer process you can concatenate pipes by passing stdout to `PipeIterator::new` or using `tee` utility to split the stdout of blocks_iterator. The latter is better because it doesn't require re-serialization of the data.\n\n\n## Binary version\n\nWhen using pipes, data is serialized and deserialized.\n\nblocks_iterator 1.x produce serialized BlockExtra with version 0 and read version 0.\nblocks_iterator 2.x produce serialized BlockExtra with version 1 and read version 0 and 1.\nThe binary format has been changed so that version 1 can deserialize only the block bytes, without instantiating the Block struct. By doing so, light clients can use bitcoin_slices to visit the block data in faster way.\n\n## Memory requirements and performance\n\nRunning (`cargo run --release -- --network X --blocks-dir Y \u003e/dev/null`) on threadripper 1950X,\nTestnet @ 2130k, Mainnet @ 705k. Spinning disk. Take following benchmarks with a grain of salt\nsince they refer to older versions.\n\n| Network | `--skip--prevout` | `--max-reorg` | `utxo-db` | Memory | Time    |\n|---------|-------------------|---------------|----------:|-------:|--------:|\n| Mainnet | true              |           6   | no        |   33MB |  1h:00m |\n| Mainnet | false             |           6   | no        |  5.3GB |  1h:29m |\n| Mainnet | false             |           6   | 1 run     |  201MB |  9h:42m |\n| Mainnet | false             |           6   | 2 run     |  113MB |  1h:05m |\n| Testnet | true              |           40  | no        |  123MB |  3m:03s |\n| Testnet | false             |           40  | no        |  1.4GB |  8m:02s |\n| Testnet | false             |           40  | 1 run     |  247MB | 16m:12s |\n| Testnet | false             |           40  | 2 run     |  221MB |  8m:32s |\n\n## Doc\n\nTo build docs:\n\n```sh\nRUSTDOCFLAGS=\"--cfg docsrs\" cargo +nightly doc --all-features --open\n```\n\n## Examples\n\nRun examples with:\n\n```sh\ncargo run --release --example verify\n```\n\n* [heaviest](cli/examples/heaviest_pipe.rs) find the transaction with greatest weight\n* [most_output](cli/examples/most_output_pipe.rs) find the transaction with most output\n* [outputs_versions](cli/examples/outputs_versions.rs) Count outputs witness version\n* [signatures_in_witness](cli/examples/signatures_in_witness.rs) Count signatures in witness\n* [verify](cli/examples/verify.rs) verify transactions in blocks using libbitcoin-consensus. Consumers are run in parallel fashion.\n\n## Version 1.0 meaning\n\nThe `1.0` is not to be intended as *battle-tested production-ready* library, the binary format of\n`BlockExtra` changed and I wanted to highlight it with major version rollout.\n\n## Similar projects\n\n* [bitcoin-iterate](https://github.com/rustyrussell/bitcoin-iterate) this project inspired blocks_iterator, the differences are:\n  * It is C-based\n  * It's more suited for shell piping, while blocks_iterator can be used with piping but also as a rust library\n  * It doesn't give previous outputs information\n  * It is making two passage over `blocks*.dat` serially, while blocks_iterator is doing two passes in parallel.\n* [rust-bitcoin-indexer](https://github.com/dpc/rust-bitcoin-indexer) this project requires longer setup times (about 9 hours indexing) and a postgre database, once the indexing is finished it allows fast queries on relational database.\n* [electrs](https://github.com/romanz/electrs) specifically intended for the Electrum protocol\n\n\n## MSRV\n\nCheck minimum rust version run in CI, as of Aug 2023 is:\n\n* `1.60.0` without features (needs some pinning, check CI)\n* `1.67.0` with features.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRCasatta%2Fblocks_iterator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FRCasatta%2Fblocks_iterator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRCasatta%2Fblocks_iterator/lists"}