{"id":19571595,"url":"https://github.com/block-core/blockcore-indexer","last_synced_at":"2025-04-27T03:32:23.739Z","repository":{"id":37756401,"uuid":"236330457","full_name":"block-core/blockcore-indexer","owner":"block-core","description":"Blockcore Indexer scans the blockchain of Blockcore-derived chains and stores transaction information in a MongoDB database with REST API available for Block Explorers to use.","archived":false,"fork":false,"pushed_at":"2025-03-07T14:48:40.000Z","size":1015,"stargazers_count":21,"open_issues_count":29,"forks_count":22,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-19T04:36:17.786Z","etag":null,"topics":["blockchain-indexer"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/block-core.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-01-26T15:20:38.000Z","updated_at":"2025-03-07T14:48:45.000Z","dependencies_parsed_at":"2023-12-25T13:07:02.085Z","dependency_job_id":"15da42e9-f3f6-4295-8127-f24f527d7421","html_url":"https://github.com/block-core/blockcore-indexer","commit_stats":{"total_commits":246,"total_committers":9,"mean_commits":"27.333333333333332","dds":0.5772357723577235,"last_synced_commit":"0e17481faf7068faee031e99a1a7d486692d72b3"},"previous_names":[],"tags_count":82,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/block-core%2Fblockcore-indexer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/block-core%2Fblockcore-indexer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/block-core%2Fblockcore-indexer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/block-core%2Fblockcore-indexer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/block-core","download_url":"https://codeload.github.com/block-core/blockcore-indexer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250355179,"owners_count":21416862,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blockchain-indexer"],"created_at":"2024-11-11T06:19:24.127Z","updated_at":"2025-04-27T03:32:22.844Z","avatar_url":"https://github.com/block-core.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://avatars3.githubusercontent.com/u/53176002?s=200\u0026v=4\" height=\"100\" alt=\"Blockcore\" /\u003e\n  \u003c/p\u003e\n  \u003ch3 align=\"center\"\u003e\n    About Blockcore Indexer\n  \u003c/h3\u003e\n  \u003cp align=\"center\"\u003e\n    Basic and easy to use block indexer\n  \u003c/p\u003e\n  \u003cp align=\"center\"\u003e\n      \u003ca href=\"https://github.com/block-core/blockcore-indexer/actions\"\u003e\u003cimg src=\"https://github.com/block-core/blockcore-indexer/workflows/Pull%20Request/badge.svg\" /\u003e\u003c/a\u003e\n      \u003ca href=\"https://github.com/block-core/blockcore-indexer/actions\"\u003e\u003cimg src=\"https://github.com/block-core/blockcore-indexer/workflows/Build%20and%20Release%20Binaries/badge.svg\" /\u003e\u003c/a\u003e\n      \u003ca href=\"https://github.com/block-core/blockcore-indexer/actions\"\u003e\u003cimg src=\"https://github.com/block-core/blockcore-indexer/workflows/Build%20and%20Release%20Docker%20Image/badge.svg\" /\u003e\u003c/a\u003e\n  \u003c/p\u003e\n\u003c/p\u003e\n\n# Blockcore Indexer\n\nBlockcore Indexer scans the blockchain of Blockcore-derived chains and stores transaction/address information in a MongoDB database with REST API available for Block Explorers to use.\n\nBlockcore Indexer API can be searched by segwit addresses and Cold-Staking (hot and cold key) script types.\n\n## How to run\n\nThere are multiple ways you can run the indexer software, either from source code, from binaries or using docker.\n\nWhen you run the indexer, you can either run it with a global configuration hosted by Blockcore, or with a custom local configuration.\n\n\n\n### Docker (manual)\n\n```sh\n# Run interactively and read globla configuration for CITY\ndocker run -it blockcore/indexer:latest --CHAIN=CITY\n```\nWhen this is run, the following configuration is downloaded and applied:\n\n[CITY.json](https://github.com/block-core/chaininfo/blob/master/chains/CITY.json)\n\nThe configuration is used by multiple software, not just Indexer, and the Indexer part is the \"Indexer\". For most global configuration, it's only this configuration:\n\n```json\n  \"Indexer\": {\n    \"StoreRawTransactions\": true\n  },\n```\n\nThe default configurations are read from [appsetting.json](src/Blockcore.Indexer/appsettings.json), which has these fields:\n\n```json\n   \"Indexer\": {\n      \"ConnectionString\": \"mongodb://{Symbol}-mongo\",\n      \"DatabaseNameSubfix\": true,\n\n      \"RpcDomain\": \"{Symbol}-chain\",\n      \"RpcSecure\": false,\n      \"RpcUser\": \"rpcuser\",\n      \"RpcPassword\": \"rpcpassword\",\n\n      // Notification parameters\n      \"NotifyUrl\": \"https://apiurl\",\n      \"NotifyBatchCount\": 0,\n\n      // Syncing parameters\n      \"SyncBlockchain\": true,\n      \"SyncMemoryPool\": true,\n      \"DbBatchSize\": 10000000,\n      \"DbBatchCount\" : 10000,\n      \"ParallelRequestsToTransactionRpc\": 50,\n      \"DetailedTrace\": 0,\n      \"MaxItemsInQueue\": 10,\n      \"SyncInterval\": 5,\n      \"AverageInterval\": 10,\n\n      // Store the trx hex in mongo storage or read it from RPC\n      \"StoreRawTransactions\": true,\n      \"NumberOfPullerTasksForIBD\" : 5,\n      \"MaxItemsInBlockingCollection\" : 1000\n   }\n```\nThe most important setting if you're running a manual setup, is the `ConnectionString` which by default attempts to connect to another docker container with the name `city-mongo` in this instance. The other important setting is `RpcDomain`, which is the fullnode that has the blockchain data. Ensure you have configured the correct `RpcUser` and `RpcPassword`.\n\nTo override these settings, you can provide them through environment variables or parameters like the following:\n\n```sh\ndocker run -it blockcore/indexer:latest --chain=CITY -e \"Indexer:ConnectionString=mongodb://127.0.0.1:27017\" -e \"Indexer:RpcDomain=127.0.0.1\" -e \"Indexer:RpcUser=rpcuser1\" -e \"Indexer:RpcPassword=rpcpassword1\"\n```\n\n## Run from source\n\n```sh\ndotnet run --project src/Blockcore.Indexer/Blockcore.Indexer.csproj --chain=CITY --Indexer:ConnectionString=mongodb://127.0.0.1:27017 --Indexer:RpcDomain=127.0.0.1 --Indexer:RpcUser=rpcuser1 --Indexer:RpcPassword=rpcpassword1\n```\n\nYou can also run the node from source like this, from the `blockcore` repository:\n\n```sh\n# It is important to set server=1 which is off by default, to enable RPC.\ndotnet run --project src/Node/Blockcore.Node/Blockcore.Node.csproj --CHAIN=CITY -server=1 -rpcuser=rpcuser1 -rpcpassword=rpcpassword1\n```\n\nWhen starting it the node, you should see something like:\n\n```\ninfo: Blockcore.Features.RPC.RPCFeature[0]\n      RPC listening on:\n      http://[::1]:4334/\n      http://127.0.0.1:4334/\n```\n\n## Usage examples\n\nIf you want to quickly learn how to use the indexer for a custom solution, such as an block explorer, please look at our [Blockcore Explorer](https://github.com/block-core/blockcore-explorer) source code on how to implement paging and other features.\n\n## Compatibility\n\nWhile we do our best to keep compatibility of APIs going forward, we will continue to change and improve our APIs that can result in breaking changes for consumers of the APIs.\n\nWe will attempt to avoid breaking changes within major releases.\n\nThe 0.0.X releases of Blockcore Indexer is not compatible with the 0.1.X.\n\nAll our technologies are available on docker, so can easily be upgraded and downgraded when there are compatibility issues.\n\n### Technologies\n- .NET Core\n- Blockcore Platform\n- Running a full Bitcoin/Blockcore node\n- Running a MongoDB instance as indexing storage\n- REST API that can be consumed by Block Explorer\n\n#### DB schema\nCan be found here:  \nhttps://github.com/block-core/blockcore-indexer/blob/master/src/Blockcore.Indexer/doc/dbschema.md\n\n#### API\nOpenAPI http://[server-url]:[port]/docs/\n\n## Development\n\nTo get started working on the indexer is super easy. All you need is a locally running node that has RPC enabled and a MongoDB instance.\n\nThe process of configuration flows like this:\n\n1. Startup looks at what chain it should start, this is always in the form of ticker symbol, such as BTC or CITY.\n2. Configuration is read from appsettings.json.\n3. Configuration is downloaded from https://chains.blockcore.net/\n4. Configuration is read from appsettings.Development.json.\n\nOut of the box, the configuration when you run from Visual Studio will connect to a local running node and local MongoDB instance.\n\nWhen you have the node running, pick it from the dropdown menu (green play button) in Visual Studio and run. The indexer should start indexing the blocks and store them in your local MongoDB instance.\n\n1. Download [Blockcore Reference Node](https://github.com/block-core/blockcore-nodes/releases).\n2. Download [MongoDB](https://www.mongodb.com/).\n3. Install MongoDB\n4. Run your reference node of choice, with these parameters: \n\n```\n-server -rpcallowip=127.0.0.1 -rpcbind=127.0.0.1 -rpcpassword=rpcpassword -rpcuser=rpcuser\n```\n\n5. As soon as you have connections and your node is starting to download blocks, you can start the Blockcore Indexer.\n\n*Happy debugging and coding!*\n\n### Improved performance for initial sync and indexing\n\nRunning the node and indexer from Visual Studio will involve a reduction in performance on initial syncing of the blockchain and indexing the data. For optimal performance, \nuse the dotnet from terminal:\n\n```sh\n# Run the node and let it sync to tip\ndotnet run Blockcore.Node.csproj --chain=STRAX -txindex=1  -server -iprangefiltering=0 -rpcallowip=127.0.0.1 -rpcbind=127.0.0.1 -rpcpassword=rpcpassword -rpcuser=rpcuser -testnet\n\n# Run the indexer\nset ASPNETCORE_ENVIRONMENT=Development\ndotnet run Blockcore.Indexer.csproj --chain=TSTRAX --no-launch-profile\n```\n\n### Local MongoDB database server on docker\n\nThere is a basic docker-compose.yml that can be used to start a MongoDB database server on docker. Navigate to the docker/database folder and start the container in background mode:\n\n```\nsudo docker-compose up -d\n```\n\n## Release Process\n\n1. New changes to the codebase must come as pull requests. This will trigger the [pull-request.yml](.github/workflows/pull-request.yml) workflow.\n\n2. When a pull request is merged to master, this will trigger [build.yml](.github/workflows/build.yml). Build will produce a draft release, or update existing.\n\n3. After manual testing and verification of the draft release (which contains binaries created by build), a project responsible can release the draft release to the public, either as a release or pre-release.\n\n4. The [release.yml](.github/workflows/release.yml) workflow picks up the release events, and builds the [docker image](src/Blockcore.Indexer/Dockerfile.Release) based on the newly released binary packages.\n\n5. Newly built and released container can then be installed using either :latest tag (not adviseable) or the specific version (advised)\n\n```sh\ndocker pull blockcore/indexer:latest\n```\n\n```sh\ndocker pull blockcore/indexer:0.0.6\n```\n\n## Troubleshooting\n\n\n```\nStatusCode = Unauthorized Error =  Blockcore.Indexer.Client.BitcoinClientException: StatusCode='Unauthorized' Error=Error (0)\n   at Blockcore.Indexer.Client.BitcoinClient.HandleError(String body, HttpResponseMessage response, Nullable`1 statusCode) in /home/runner/work/blockcore-indexer/blockcore-indexer/src/Blockcore.Indexer/Client/BitcoinClient.cs:line 617\n```\n\nIssue: This error log is often related to caller IP not being part of the \"rpcallowip\" list.\n\n\n```\nBlockcore.Indexer.Client.BitcoinCommunicationException: Daemon Failed Url = 'http://city-chain:4334/'\n ---\u003e System.AggregateException: One or more errors occurred. (Connection refused)\n ---\u003e System.Net.Http.HttpRequestException: Connection refused\n ---\u003e System.Net.Sockets.SocketException (111): Connection refused\n```\n\nIssue: This happens when the DNS name is not accessible.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblock-core%2Fblockcore-indexer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblock-core%2Fblockcore-indexer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblock-core%2Fblockcore-indexer/lists"}