{"id":16116700,"url":"https://github.com/dpc/rust-bitcoin-indexer","last_synced_at":"2025-09-26T10:31:43.063Z","repository":{"id":41142668,"uuid":"155957046","full_name":"dpc/rust-bitcoin-indexer","owner":"dpc","description":"Powerful \u0026 versatile Bitcoin Indexer, in Rust","archived":false,"fork":false,"pushed_at":"2022-09-14T21:40:20.000Z","size":27920,"stargazers_count":148,"open_issues_count":8,"forks_count":34,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-09-22T08:58:32.739Z","etag":null,"topics":["bitcoin","chain-head","indexer","indexing-bitcoin","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dpc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-11-03T06:30:59.000Z","updated_at":"2025-09-01T06:44:54.000Z","dependencies_parsed_at":"2023-01-18T07:55:12.016Z","dependency_job_id":null,"html_url":"https://github.com/dpc/rust-bitcoin-indexer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dpc/rust-bitcoin-indexer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpc%2Frust-bitcoin-indexer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpc%2Frust-bitcoin-indexer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpc%2Frust-bitcoin-indexer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpc%2Frust-bitcoin-indexer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dpc","download_url":"https://codeload.github.com/dpc/rust-bitcoin-indexer/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpc%2Frust-bitcoin-indexer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":277063521,"owners_count":25754225,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-26T02:00:09.010Z","response_time":78,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitcoin","chain-head","indexer","indexing-bitcoin","rust"],"created_at":"2024-10-09T20:25:21.082Z","updated_at":"2025-09-26T10:31:42.803Z","avatar_url":"https://github.com/dpc.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Bitcoin Indexer\n\nAn experiment in creating a perfect Bitcoin Indexer, in Rust.\n\nQuery blocks using JsonRPC, dump them into Postgres in an append-only format,\nsuitable for querying, as much as event-sourcing-like handling. After reaching\nchain-head, keep indexing in real-time and handle reorgs.\n\nThis started as an educational experiment, but is quite advanced already.\n\nGoals:\n\n* simplicity and small code:\n    * easy to fork and customize\n* versatile data model:\n    * append-only log\n    * support for event sourcing\n    * support for efficient queries\n    * correct by construction\n    * always-coherent state view (i.e. atomic reorgs)\n* top-notch performance, especially during initial indexing\n\nRead [How to interact with a blockchain](https://dpc.pw/rust-bitcoin-indexer-how-to-interact-with-a-blockchain) for knowledge sharing, discoveries and design-decisions.\n\nStatus:\n\n* The codebase is very simple, design quite clean and composable and performance really good.\n* Some tests are there and everything seems quite robust, but not tested in production so far. \n* Indexing the blockchain works\n* Indexing mempool works\n\n## Comparing to alternatives\n\nPlease take with a grain of salt, and submit PRs if any information is stale or wrong.\n\n### vs [electrs](https://github.com/romanz/electrs)\n\n`electrs` uses an embedded key value store (RocksDB) to locally maintain addtitional indices. When needed it queries the fullnode itself for block/tx data. `rust-bitcoin-indexer` uses a normal relational database model, with decompressed data stored in a standalone Postgres instance that can run on a different host/cluster.\n\nEmbedded KV store approach has many advantages:\n\n* naturally good indexing performance and less disk space usage\n* all the data is always available as a raw block/tx data (while with `rust-bitcoin-indexer` one would have to modify the schema, and then extract and retro-fit it with a small piece of code).\n\nUsing relational database will allow you to execute ad-hoc queries and potentially share the db with other applications, without building a separate interface. You can add and remove indices according to your needs, etc.\n\n`rust-bitcoin-indexer` was designed to have a good idempotent and reliable events streaming/subscription data model. \n\n`electrs` is used for practical purposes, `rust-bitcoin-indexer` (at least right now) is just a neat experiment that gave good results and seems to be working quite well.\n\nLooking back, I think `electrs` model is esier, faster, and more robust approach for most applications, and I haven't used it mostly because I was unware of all the aspects of problem at hand ¯\\\\\\_(ツ)\\_/¯. However I am still very happy with results achived with `rust-bitcoin-indexer`. It's compact, relatively small (memory-usage \u0026 LoC-wise) and squizes every last drop of performance to make indexing to a general-purpose database practical. And running random SQL queries against Bitcoin's blockchain data is just very fun. `rust-bitcoin-indexer` can be also a good example how to quickly dump billions of rows worth of SQL data into Postgres.\n\n## Running\n\nInstall Rust with https://rustup.rs\n\n\n### Bitcoind node\n\nSetup Bitcoind full node, with a config similiar to this:\n\n```\n# [core]\n# Run in the background as a daemon and accept commands.\ndaemon=0\n\n# [rpc]\n# Accept command line and JSON-RPC commands.\nserver=1\n# Username for JSON-RPC connections\nrpcuser=user\n# Password for JSON-RPC connections\nrpcpassword=password\n\n# [wallet]\n# Do not load the wallet and disable wallet RPC calls.\ndisablewallet=1\nwalletbroadcast=0\n```\n\nThe only important part here is being able to access JSON-RPC interface.\n\n### Postgresql\n\nSetup Postgresql DB, with a db and user:pass that can access it. Example:\n\n```\nsudo su postgres\nexport PGPASSWORD=bitcoin-indexer\ncreateuser bitcoin-indexer\ncreatedb bitcoin-indexer bitcoin-indexer\n```\n\n### `.env` file\n\nSetup `.env` file with Postgresql and Bitcoin Core connection data. Example:\n\n```\nDATABASE_URL=postgres://bitcoin-indexer:bitcoin-indexer@localhost/bitcoin-indexer\nNODE_RPC_URL=http://someuser:somepassword@localhost:18443\n```\n\n#### Optimize DB performance for massive amount of inserts!\n\n**This one is very important!!!**\n\nIndexing from scratch will dump huge amounts of data into the DB.\nIf you don't want to wait for the initial indexing to complete for days or weeks,\nyou should carefully review this section.\n\nOn software level `pg.rs` already implements the following optimizations:\n\n* inserts are made using multi-row value insert statements;\n* multiple multi-row insert statements are batched into one transaction;\n* initial sync starts with no indices and utxo set is cached in memory;\n* once restarted, missing UTXOs are fetched from the db, but new ones\n  are still being cached in memory;\n* once restarted, only minimum indices are created (for UTXO fetching)\n* all indices are created only after reach the chain-head\n\nTune your system for best performance too:\n\n* [Consider tunning your PG instance][tune-psql] for such workloads.;\n* [Make sure your DB is on a performant file-system][perf-fs]; generally COW filesystems perform poorly\n  for databases, without providing any value; [on `btrfs` you can disable COW per directory][chattr];\n  eg. `chattr -R +C /var/lib/postgresql/9.6/`; On other FSes: disable barriers, align to SSD; you can\n  mount your fs in more risky-mode for initial sync, and revert back to safe settings\n  aftewards.\n* on my HDD ext4 `sudo tune2fs -o journal_data_writeback  /dev/\u003cpartition\u003e` some improvement, but ultimately\n  SMR disk turned out to be too slow to keep up with inserting data into \u003e 1B records table and updating\n  indices.\n\n[perf-fs]: https://www.slideshare.net/fuzzycz/postgresql-on-ext4-xfs-btrfs-and-zfs\n[tune-psql]: https://stackoverflow.com/questions/12206600/how-to-speed-up-insertion-performance-in-postgresql\n[chattr]: https://www.kossboss.com/btrfs-disabling-cow-on-a-file-or-directory-nodatacow/\n\nPossibly ask an experienced db admin if anything more can be done. We're talking\nabout inserting around billion records into 3 tables each.\n\nFor reference -  on my system, I get around 30k txs indexed per second:\n\n```\n[2019-05-24T05:20:29Z INFO  bitcoin_indexer::db::pg] Block 194369H fully indexed and commited; 99block/s; 30231tx/s\n```\n\nwhich leads to around 5 hours initial blockchain indexing time (current block height is around 577k)...\nand then just 4 additional hours to build indices.\n\nI suggest using `tx/s` metric to estimate completion.\n\n### Run\n\nNow everything should be ready. Compile and run with:\n\n```\ncargo run --release --bin bitcoin-indexer\n```\n\nAfter the initial full sync, you can also start mempool indexer:\n\n```\ncargo run --release --bin mempool-indexer\n```\n\nin a directory containing the `.env` file.\n\n#### More options\n\nYou can use `--wipe-whole-db` to wipe the db. (to be removed in the future)\n\nFor logging set env. var. `RUST_LOG` to `bitcoin_indexer=info` or refer to https://docs.rs/env_logger/0.6.0/env_logger/.\n\n\n### Some useful stuff that can be done already\n\nCheck current balance of an address:\n\n```\nbitcoin-indexer=\u003e select * from address_balances where address = '14zV5ZCqYmgyCzoVEhRVsP7SpUDVsCBz5g';                                                                                                                                          \n              address               |   value\n------------------------------------+------------\n 14zV5ZCqYmgyCzoVEhRVsP7SpUDVsCBz5g | 6138945213\n```\n\nCheck balances at a given height:\n\n```\nbitcoin-indexer=\u003e select * from address_balances_at_height WHERE address IN ('14zV5ZCqYmgyCzoVEhRVsP7SpUDVsCBz5g', '344tcgkKA97LpgzGtAprtqnNRDfo4VQQWT') AND height = 559834;\n              address               | height |   value   \n------------------------------------+--------+-----------\n 14zV5ZCqYmgyCzoVEhRVsP7SpUDVsCBz5g | 559834 | 162209091\n 344tcgkKA97LpgzGtAprtqnNRDfo4VQQWT | 559834 |         0\n```\n\nCheck txes pending in the mempool:\n\n```\nbitcoin-indexer=\u003e select * from tx_in_mempool order by (fee/weight) desc limit 5;\n              hash_id               |             hash_rest              | size | weight |  fee   | locktime | coinbase |                                hash                                |             ts\n------------------------------------+------------------------------------+------+--------+--------+----------+----------+--------------------------------------------------------------------+----------------------------\n \\x5d094cc3e4d9a1ec2d280aa9204ff8e4 | \\xf0e960ddfed324880dac6e8ab54544c6 |  191 |    764 | 562871 |   577816 | f        | \\xc64445b58a6eac0d8824d3fedd60e9f0e4f84f20a90a282deca1d9e4c34c095d | 2019-05-26 05:31:08.658908\n \\x5a8caef874d1cd657460f754a9ae6985 | \\xcce821c1789f20b665fd6240717340b6 |  213 |    855 | 182000 |        0 | f        | \\xb64073714062fd65b6209f78c121e8cc8569aea954f7607465cdd174f8ae8c5a | 2019-05-26 05:36:10.331908\n \\xab60acf5b6c7d6103e7186a9d319210b | \\xdb0baedb6c49cfe748411ea8de8425a8 |  371 |   1484 | 298600 |        0 | f        | \\xa82584dea81e4148e7cf496cdbae0bdb0b2119d3a986713e10d6c7b6f5ac60ab | 2019-05-26 05:30:57.902214\n \\x5b63601004575830ccde165a924b823e | \\xd5085b0193da8a5c2d0d1bc579a77a06 |  339 |   1356 | 163953 |        0 | f        | \\x067aa779c51b0d2d5c8ada93015b08d53e824b925a16decc305857041060635b | 2019-05-26 05:34:34.603281\n \\xaf8ce41bd3c7db9dd44fa126b5d5d386 | \\x1b0a8db2986f1124c2ebba3e9cbc9251 |  112 |    450 |  53788 |        0 | f        | \\x5192bc9c3ebaebc224116f98b28d0a1b86d3d5b526a14fd49ddbc7d31be48caf | 2019-05-26 05:37:46.736696\n(5 rows)\n```\n\nand many more. Refer to `./src/db/pg/*.sql` files for good overview of the schema and utilities.\n\n# Support\n\nIf you like and/or use this project, you can pay for it by sending Bitcoin to\n[33A9SwFHWEnwmFfgRfHu1GvSfCeDcABx93](bitcoin:33A9SwFHWEnwmFfgRfHu1GvSfCeDcABx93).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpc%2Frust-bitcoin-indexer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdpc%2Frust-bitcoin-indexer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpc%2Frust-bitcoin-indexer/lists"}