{"id":16805606,"url":"https://github.com/gcarq/rusty-blockparser","last_synced_at":"2025-05-16T04:02:38.144Z","repository":{"id":57665836,"uuid":"50385943","full_name":"gcarq/rusty-blockparser","owner":"gcarq","description":"Bitcoin Blockchain Parser written in Rust language","archived":false,"fork":false,"pushed_at":"2023-10-05T11:06:41.000Z","size":380,"stargazers_count":365,"open_issues_count":12,"forks_count":143,"subscribers_count":21,"default_branch":"master","last_synced_at":"2024-10-14T09:48:42.823Z","etag":null,"topics":["bitcoin","blockchain","litecoin","parser","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gcarq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["gcarq"]}},"created_at":"2016-01-25T22:34:48.000Z","updated_at":"2024-10-05T04:32:33.000Z","dependencies_parsed_at":"2024-10-25T18:38:34.968Z","dependency_job_id":"b42f381c-07fb-43fb-bf2b-82f0242dacd9","html_url":"https://github.com/gcarq/rusty-blockparser","commit_stats":{"total_commits":167,"total_committers":12,"mean_commits":"13.916666666666666","dds":0.4610778443113772,"last_synced_commit":"ba87bb9a2914f1bd5c2025ea0424bfd52b2e840a"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gcarq%2Frusty-blockparser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gcarq%2Frusty-blockparser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gcarq%2Frusty-blockparser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gcarq%2Frusty-blockparser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gcarq","download_url":"https://codeload.github.com/gcarq/rusty-blockparser/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254464891,"owners_count":22075570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitcoin","blockchain","litecoin","parser","rust"],"created_at":"2024-10-13T09:48:45.752Z","updated_at":"2025-05-16T04:02:37.689Z","avatar_url":"https://github.com/gcarq.png","language":"Rust","funding_links":["https://github.com/sponsors/gcarq"],"categories":[],"sub_categories":[],"readme":"# rusty-blockparser\n\n**rusty-blockparser** is a Bitcoin Blockchain Parser that enables data extraction of various types (e.g.:\nblocks, transactions, scripts, public keys / hashes, balances) and full UTXO dumps.\n\n### Supported Blockchains\n\n`Bitcoin`, `Namecoin`, `Litecoin`, `Dogecoin`, `Myriadcoin`, `Unobtanium` and `NoteBlockchain`.\n\n**IMPORTANT:** A local unpruned copy of the blockchain with intact block index and blk files, downloaded\nwith [Bitcoin Core](https://github.com/bitcoin/bitcoin) 0.15.1+ or similar clients is required.\nIf you are not sure whether your local copy is valid you can apply `--verify` to validate the blockdata and block merkle\ntrees.\n\n### Supported Transaction Types\n\nBitcoin and Bitcoin Testnet transactions are parsed using [rust-bitcoin](https://github.com/rust-bitcoin/rust-bitcoin),\nthis includes transactions of\ntype [P2SH](https://learnmeabitcoin.com/technical/script/p2sh/), [P2PKH](https://learnmeabitcoin.com/technical/script/p2pkh/), [P2PK](https://learnmeabitcoin.com/technical/script/p2pk/), [P2WSH](https://learnmeabitcoin.com/technical/script/p2wsh/), [P2WPKH](https://learnmeabitcoin.com/technical/script/p2wpkh/), [P2TR](https://learnmeabitcoin.com/technical/script/p2tr/), [P2MS](https://learnmeabitcoin.com/technical/script/p2ms/),  [OP_RETURN](https://learnmeabitcoin.com/technical/script/return/)\nand [SegWit](https://learnmeabitcoin.com/beginners/guide/segwit/).\n\nBitcoin forks (e.g.: Dogecoin, Litecoin, etc.) are evaluated via a custom script implementation which includes P2PK,\nP2PKH, P2SH, P2MS and OP_RETURN.\n\n## Analysis and Data Extraction\n\nData is being extracted via callbacks which are built on top of the core parser.\nThey can be easily extended to extract specific types of information and can be\nfound [here](https://github.com/gcarq/rusty-blockparser/tree/master/src/callbacks).\n\n#### Extract Balances of all known addresses\n\nThe command `balances` extracts the balance of all known addresses and dumps it to a csv file called `balances.csv` with\nthe following format:\n\n```\nbalances.csv: address ; balance\n```\n\n#### Extract all UTXOs along with their corresponding address balances\n\nThe command `unspentcsvdump` can be used to dump all [UTXOs](https://learnmeabitcoin.com/technical/transaction/utxo/)\nwith their corresponding address balance to a csv file called `unspent.csv`.\nThe csv file is in the following format:\n\n```csv\nunspent.csv: txid ; indexOut ; height ; value ; address\n```\n\nNOTE: The total size of the csv dump is at least 8 GiB (height 635000).\n\n#### Show OP_RETURN data\n\nThe command `opreturn` can be used to show all embedded OP_RETURN data in the terminal that contains valid UTF8.\n\n#### Extract full CSV dump\n\nThe command `csvdump` dumps all data to csv files. This data can be imported to a database for further analysis.\n**NOTE**: The total size of the csv dump is at least 731 GiB (height 635,000).\n\nThe files are in the following format:\n\n```\nblocks.csv: block_hash ; height ; version ; blocksize ; hashPrev ; hashMerkleRoot ; nTime ; nBits ; nNonce\n```\n\n```\ntransactions.csv: txid ; hashBlock ; version ; lockTime\n```\n\n```\ntx_in.csv: txid ; hashPrevOut ; indexPrevOut ; scriptSig ; sequence\n```\n\n```\ntx_out.csv: txid ; indexOut ; height ; value ; scriptPubKey ; address\n```\n\nSee the [block](https://en.bitcoin.it/wiki/Protocol_documentation#block)\nand [transaction](https://en.bitcoin.it/wiki/Protocol_documentation#tx) specifications if some of the fields are\nunclear.\nIf you want to insert the files into MySql see [sql/schema.sql](sql/schema.sql) (but be aware this hasn't been tested\nand used for quite some time now). It contains all table structures and SQL statements for bulk inserting. Also\nsee [sql/views.sql](sql/views.sql) for some query examples.\n\n#### Show blockchain statistics\n\nThe command `simplestats` can be used to show blockchain statistics, e.g.:\n\n* Show the txid of transactions that contain specific script types\n* Total numbers like number of blocks, number of transactions, biggest tx in value or size\n* Averages like block size, time between blocks, txs, inputs and outputs\n\n## Usage\n\n```\nUsage: rusty-blockparser [OPTIONS] [COMMAND]\n\nCommands:\n  unspentcsvdump  Dumps the unspent outputs to CSV file\n  csvdump         Dumps the whole blockchain into CSV files\n  simplestats     Shows various Blockchain stats\n  balances        Dumps all addresses with non-zero balance to CSV file\n  opreturn        Shows embedded OP_RETURN data that is representable as UTF8\n  help            Print this message or the help of the given subcommand(s)\n\nOptions:\n      --verify\n          Verifies merkle roots and block hashes\n  -v...\n          Increases verbosity level. Info=0, Debug=1, Trace=2 (default: 0)\n  -c, --coin \u003cNAME\u003e\n          Specify blockchain coin (default: bitcoin) [possible values: bitcoin, testnet3, namecoin, litecoin, dogecoin, myriadcoin, unobtanium, noteblockchain]\n  -d, --blockchain-dir \u003cblockchain-dir\u003e\n          Sets blockchain directory which contains blk.dat files (default: ~/.bitcoin/blocks)\n  -s, --start \u003cHEIGHT\u003e\n          Specify starting block for parsing (inclusive)\n  -e, --end \u003cHEIGHT\u003e\n          Specify last block for parsing (inclusive) (default: all known blocks)\n  -h, --help\n          Print help\n  -V, --version\n          Print version\n```\n\n## Example Usage\n\nTo make a `unspentcsvdump` of the Bitcoin blockchain your command would look like this:\n\n```\n# ./blockparser unspentcsvdump /path/to/dump/\n[6:02:53] INFO - main: Starting rusty-blockparser v0.7.0 ...\n[6:02:53] INFO - index: Reading index from ~/.bitcoin/blocks/index ...\n[6:02:54] INFO - index: Got longest chain with 639626 blocks ...\n[6:02:54] INFO - blkfile: Reading files from ~/.bitcoin/blocks ...\n[6:02:54] INFO - parser: Parsing Bitcoin blockchain (range=0..) ...\n[6:02:54] INFO - callback: Using `unspentcsvdump` with dump folder: /path/to/dump ...\n[6:03:04] INFO - parser: Status: 130885 Blocks processed. (left: 508741, avg: 13088 blocks/sec)\n...\n[10:28:47] INFO - parser: Status: 639163 Blocks processed. (left:    463, avg:    40 blocks/sec)\n[10:28:57] INFO - parser: Status: 639311 Blocks processed. (left:    315, avg:    40 blocks/sec)\n[10:29:07] INFO - parser: Status: 639452 Blocks processed. (left:    174, avg:    40 blocks/sec)\n[10:29:17] INFO - parser: Status: 639596 Blocks processed. (left:     30, avg:    40 blocks/sec)\n[10:29:19] INFO - parser: Done. Processed 639626 blocks in 266.43 minutes. (avg:    40 blocks/sec)\n[10:32:01] INFO - callback: Done.\nDumped all 639626 blocks:\n        -\u003e transactions: 549390991\n        -\u003e inputs:       1347165535\n        -\u003e outputs:      1359449320\n[10:32:01] INFO - main: Fin.\n```\n\n## Installing\n\nThis tool should run on Windows, OS X and Linux.\nAll you need is `rust` and `cargo`.\n\n```bash\ngit clone https://github.com/gcarq/rusty-blockparser.git\ncd rusty-blockparser\ncargo build --release\ncargo test\n./target/release/rusty-blockparser --help\n```\n\n**IMPORTANT:** Building with `--release` is essential for performance.\n\n*Tested on Gentoo Linux with rust-stable 1.85.0*\n\n## Memory Usage\n\nThe required memory usage depends on the used callback:\n\n* simplestats: ~100MB\n* csvdump: ~100M\n* unspentcsvdump: ~18GB\n* balances: ~18GB\n\n**NOTE:** Those values are taken from parsing to block height 639,631 (17.07.2020).\n\n## Contributing\n\nUse the issue tracker to report problems, suggestions and questions. You may also contribute by submitting pull\nrequests.\n\nIf you find this project helpful, please consider making a donation:\n`1LFidBTeg5joAqjw35ksebiNkVM8azFM1K`\n\n## Customizing the tool for your coin\n\nThe tool can easily be customized to your coin. This section outlines the changes that need to be made and is for users\nnot familiar with Rust and Blockchain. During this example the coin name used is NoCoinium.\n\n* The main change is `src/blockchain/parser/types.rs`.\n* Add a new entry `pub struct NoCoinium` similar to other definitions.\n* You will then need to add a `impl Coin for NoCoinium`. You could easily copy a previous block e.g. Bitcoin. The\n  changes you need to do are highlighted below as comments\n\n```rust\n//The name here should be the same case as defined in the pub struct line\nimpl Coin for NoCoinium {\n    fn name(\u0026self) -\u003e String {\n        // This is primarily for display. Use same case as before\n        String::from(\"NoCoinium\")\n    }\n    fn magic(\u0026self) -\u003e u32 {\n        // Magic bytes are a string of hex characters that prefix messages in the chain. \n        // To find this value, look for the fields pchMessageStart[0-3] in the file chainparams.cpp under CMainParams\n        // The value to be used here is 0x + pchMessageStart[3] + pchMessageStart[2] + pchMessageStart[1] + pchMessageStart[0]\n        // i.e. string the values in reverse.\n        0xd9b4bef9\n    }\n    fn version_id(\u0026self) -\u003e u8 {\n        // Version ID is used to identify the address prefix for Base58 encoding of the public address\n        // Found this using the stackoverflow comment - https://bitcoin.stackexchange.com/questions/62781/litecoin-constants-and-prefixes\n        // Again with chainparams.cpp and CMainParams, look for base58Prefixes[PUBKEY_ADDRESS]. Convert the decimal value to Hex and add it here\n        0x00\n    }\n    fn genesis(\u0026self) -\u003e sha256d::Hash {\n        // This is the Genesis Block hash - Get the value from consensus.hashGenesisBlock, again found in chainparams.cpp\n        sha256d::Hash::from_str(\"000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f\").unwrap()\n    }\n    fn default_folder(\u0026self) -\u003e PathBuf {\n        // This is the folder from the user's home folder to where the blocks files are found\n        // Note the case here. It is not CamelCase as most coin directories are lower case. However, use the actual folder name\n        // from your coin implementation.\n        Path::new(\".nocoinium\").join(\"blocks\")\n    }\n}\n```\n\n* Finally, tie these changes within `impl FromStr for CoinType` under `match coin_name`. The first part will be the\n  string passed as argument to the program (see bullet point below) and the name within `from()` will be the name used\n  above.\n\n```rust\n    \"bitcoin\" =\u003e Ok(CoinType::from(Bitcoin)),\n...\n\"nocoinium\" =\u003e Ok(CoinType::from(NoCoinium)),\n...\n```\n\n* The next change is in `src/main.rs`. Under the fn `parse_args()` add your coin to the array of coins. The case you use\n  here will be the same value as you pass in the arguments when executing the blockchain (using the `-c` argument)\n* Finally, add your coin name in the README.md file so others know your coin is supported\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgcarq%2Frusty-blockparser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgcarq%2Frusty-blockparser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgcarq%2Frusty-blockparser/lists"}