{"id":18235714,"url":"https://github.com/Bitcoin-com/slp-indexer","last_synced_at":"2025-04-04T04:30:25.525Z","repository":{"id":39946501,"uuid":"235024935","full_name":"Bitcoin-com/slp-indexer","owner":"Bitcoin-com","description":"Bitcoin.com slp-indexer","archived":false,"fork":false,"pushed_at":"2023-07-05T20:44:00.000Z","size":3289,"stargazers_count":20,"open_issues_count":6,"forks_count":13,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-03-27T14:55:05.575Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Bitcoin-com.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-01-20T05:23:14.000Z","updated_at":"2024-11-04T06:30:56.000Z","dependencies_parsed_at":"2024-11-27T15:32:29.189Z","dependency_job_id":null,"html_url":"https://github.com/Bitcoin-com/slp-indexer","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bitcoin-com%2Fslp-indexer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bitcoin-com%2Fslp-indexer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bitcoin-com%2Fslp-indexer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bitcoin-com%2Fslp-indexer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Bitcoin-com","download_url":"https://codeload.github.com/Bitcoin-com/slp-indexer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247123069,"owners_count":20887258,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-05T00:02:44.878Z","updated_at":"2025-04-04T04:30:24.651Z","avatar_url":"https://github.com/Bitcoin-com.png","language":"Java","funding_links":[],"categories":["Simple Ledger Protocol (SLP Token)"],"sub_categories":["Libraries"],"readme":"# Bitcoin.com SLP-Indexer\n**last updated:** 2019-12-16\n\n* 1. [What is our SLP-indexer?](#what)\n* 2. [Architecture](#architecture)\n* 3. [Performance](#performance)\n* 4. [Feature improvements](#improvements)\n* 5. [Setup](#setup)\n* 6. [How do you query for data?](#query)\n\n##  1. \u003ca name='what'\u003e\u003c/a\u003eWhat is our SLP-Indexer?\nBitcoin.com SLP-indexer is a service that listens to the Bitcoin Cash blockchain for SLP-token transactions.\nWhen it finds an SLP transaction it will parse, validate and save it in the database for easy and fast access.\n\nThis indexer is built to scale and for high performance services requiring high uptime and scalability. At Bitcoin.com we use this indexer\nto handle SLP transactions to our native wallet across several global server locations. The health-check system makes it is easy\nto setup advanced DNS-failover strategies in AWS or any other cloud provider.\n\nWhy should you run this indexer compared to the alternatives on the market? Our top priority during the development of this indexer was to support\nbusinesses with requirements on stability and scalability. It was meant to be a full enterprise solution for companies to support high volume\nSLP trading on an exchange where you need to rely on a close to 100% uptime. It also has the ability to make updates and bug fixes without any downtime\nwhich was a strong businesses requirement.\n\n##  2. \u003ca name='architecture'\u003e\u003c/a\u003eArchitecture\nThe service consist of one writer and several readers services. We use MongoDb as the storage and add indexes on\nthe fields that we need to query. The reason we have split readers and writers into separate services is that we want to have the ability\nto scale our readers separately.\n\nIt is not recommended to have more than one writers as they will read the same data from the blockchain and then\ninsert the same items creating a heavier load on your MongoDb. It is possible to run more reader if you really\nneed redundancy in case of failure.\n\nYou can deploy as many readers as you want and we recommend at least 2 readers for redundancy. Deploying more readers\nwill allow the api to handle more requests. But if your MongoDb cluster cannot handle the API load you will not increase\nperformance. A good idea is if you need a lot of readers to also scale the MongoDb cluster.\n\nIt is also recommended to run your readers on a separate machine than the writer. In the case of a DDOS or heavy load on your\nreaders you don't run the risk of also crashing the writer.\n\nWe also recommend you run two or more nodes close to the location of the writer and connect your writer only to those nodes.\nThis gives you full control over the nodes it connects to and also speeds up the time the indexer picks up a transaction from the time it was\nseen in the mempool of the node. We recommend two nodes since it gives you redundancy if one node goes down or has trouble.\n\n![architecture](assets/basic_architecture.png)\n\n## 3. \u003ca name='performance'\u003e\u003c/a\u003e Performance\nWhen we tested the performance of the indexer we used loader.io to generate requests on the api. We ran test-suits to\nfigure out the performance levels and also response times with heavy load.\n\nThe graph below shows one of our tests running with 1 m5.large handling requests from loader.io. As you can see it \ncan handle a lot of clients per second and if more is needed then we can deploy another instance. \n\n![graph](assets/graph.jpg)\n\nCurrently in Bitcoin.com production we run 2 t3.medium which gives us the performance needed to serve request \nfrom our Bitcoin.com wallet and rest.bitcoin.com without any issues. If we need more performance\nwe either deploy more machines to spread the load or deploy bigger machines. We are using auto-scaling groups to scale up/down\nas needed if the load increases or decreases.\n\n## 4. \u003ca name='improvements'\u003e\u003c/a\u003e Future improvements\nThere is still some improvements that could be made to this project to scale it even further.\n\n* **Enable full-indexing support** Currently the indexer filters out SLP only transactions. It has the ability to index the entire blockchain into mongodb\nwhich would give you the ability to query for normal BCH transactions as well. Some performance issues was noted during full-indexing testing related\nto the time of the initial sync. This could be improved by handling more than one block at the time during initial sync. Also a more efficient mongodb querying strategy\nfor batches would decrease initial sync time.\n\n* **More API implementations** To make this data available for everyone to use and contribute to another API layer built in another language like JavaScript would\nbe a great addition to the project. This would give anyone regardless or preferred language the ability to contribute and build on top of the MongoDb slp-data that we\nhave from the indexer.\n\n* **Sharding MongoDb** As the blockchain grows there might be a need to use sharding for MongoDb. This will help with scaling the queries and the MongoDb to support large datasets. Currently this is not needed.\n\n* **Zeromq connection to the node** Add support for zeromq connection to the node. Currently the app uses BitcoinJ for connections to the nodes. These connections sometimes disconnect after running for a while mostly when the connected nodes goes offline for a long time. Zeromq would be add an extra way for transactions to be picked up by the indexer and it is likely it is faster than using BitcoinJ.\n\n* **Caching** To improve the querying speed a distributed cache can be added. The writer will then write to both mongodb and to the cache\nand the API will try to read data from the cache if it is there or read from mongodb if the data is not available. Doing so\nwould increase the throughput of the API and allow for bigger loads without the need to scale mongodb.\n\n##  5. \u003ca name='setup'\u003e\u003c/a\u003eSetup\n\n### What do I need to run the services?\n* 1. Java 11\n* 2. MongoDb/Amazon DocumentDb\n* 3. Bitcoin Cash Node. Either connect to a public node or your own\n\n### What do I need to build it?\n* 1. Java 11\n* 2. Maven\n\n### How do i build it?\n`cd slp-indexer`\n\n`cd slp-indexer-service`  \n`mvn clean install`\n\n`cd slp-indexer-api-service`  \n`mvn clean install`\n\n### How do i run it?\nThese steps below is how we run it on a Ubuntu machine    \nCreate this folder on your machine\n`mkdir /var/blockchain/indexer/`\n`chown user:user -R /var/blockchain/indexer`\n\nChange these settings in the `slp-indexer-service-app/src/main/resources/application-prod.properties`\nto what is applicable for you\n\n`bitcoinj.max.connection=MAX_AVAILABLE_NODES` This settings is for how many nodes you want to connect to  \n`bitcoinj.awaiting.peer=PEERS_NEEDED_FOR_STARTUP` How many connected peers is needed for the service to start \n`bitcoinj.peers.bch=127.0.0.1:8333,127.0.0.2:8333` Point to the location of your nodes if you don't want to use dns.discovery\n`dns.discovery=DNS_TO_YOUR_NODES` If you want to use DNS seeding then specify your address here  \n`spring.data.mongodb.uri=CONNECTIONSTRING` MongoDb connectionsString  \n`spring.data.mongodb.database=slpblockchain` MongoDb databasename\n\n\nThere is several ways of running the jar file. You can run it in a a docker container or using systemd. Whatever works\nbest for your case.\n\nTo run the Writer follow the build instructions and when you have built the project find the jar.  \n`slp-indexer-service-app/target/slp-indexer-service-app-0.0.1-SNAPSHOT-exec.jar`\n\nTo run the jar use the command  \n`java -jar -Dspring.profiles.active={prod/localhost/test} slp-indexer-service-app-0.0.1-SNAPSHOT-exec.jar`  \n\nExample : `java -jar -Dspring.profiles.active=localhost slp-indexer-service-app-0.0.1-SNAPSHOT-exec.jar`\n\nTo run the Reader follow the build instructions and find the jar.\n`slp-indexer-service-api-app/target/slp-indexer-service-app-0.0.1-SNAPSHOT-exec.jar`\n\nTo run the jar use the command  \n`java -jar -Dspring.profiles.active={prod/localhost/test} slp-indexer-api-service-app-0.0.1-SNAPSHOT-exec.jar`  \n\nExample : `java -jar -Dspring.profiles.active=localhost slp-indexer-api-service-app-0.0.1-SNAPSHOT-exec.jar`\n\n##  6. \u003ca name='query'\u003e\u003c/a\u003eHow do you query for data?\nThe reader has a REST api which connects to the MongoDb cluster.\n\nThese are the endpoints which are currently supported. You can find more information about the endpoints in the swagger\ndocs http://localhost:1337/api/swagger-ui.html#/ or here https://rest.bitcoin.com/\n\n\nGET validateTxid/{txId}\n\nPOST validateTxid\n\nGET list/{tokenId}\n\nPOST list/\n\nGET tokenStats/{tokenId}\n\nPOST tokenStats\n\nGET balancesForAddress/{address}\n\nPOST balancesForAddress\n\nGET balancesForToken/{tokenId}\n\nGET balance/{address}/{tokenId}\n\nGET burnTotal/{transactionId}\n\nGET recentTxForTokenId/{tokenId}/{page}\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBitcoin-com%2Fslp-indexer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FBitcoin-com%2Fslp-indexer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBitcoin-com%2Fslp-indexer/lists"}