Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/emeraldpay/dshackle

Fault Tolerant Load Balancer for Ethereum and Bitcoin APIs
https://github.com/emeraldpay/dshackle

bitcoin ethereum java kotlin l7 load-balancer

Last synced: 23 days ago
JSON representation

Fault Tolerant Load Balancer for Ethereum and Bitcoin APIs

Awesome Lists containing this project

README 

        
= Emerald Dshackle

:imagesdir: docs/assets

ifdef::env-github[]

:imagesdir: https://raw.githubusercontent.com/emeraldpay/dshackle/master/docs/assets

endif::[]

:version: 0.15.0

:version-short: 0.15

:link-docs: https://github.com/emeraldpay/dshackle/tree/release/v{version-short}/docs



image:https://github.com/emeraldpay/dshackle/workflows/Tests/badge.svg["Unit Tests"]

image:https://codecov.io/gh/emeraldpay/dshackle/branch/master/graph/badge.svg["Coverage",link="https://codecov.io/gh/emeraldpay/dshackle"]

image:https://img.shields.io/docker/pulls/emeraldpay/dshackle?style=flat-square["Docker",link="https://hub.docker.com/r/emeraldpay/dshackle"]

image:https://img.shields.io/github/license/emeraldpay/dshackle.svg?style=flat-square&maxAge=2592000["License",link="https://github.com/emeraldpay/dshackle/blob/master/LICENSE"]

image:https://img.shields.io/discord/1107840420240707704?style=flat-square[Discord,link="https://discord.gg/k9HpF9Jqee"]



[.lead]

_Emerald Dshackle is a Fault Tolerant Load Balancer for Blockchain API._



Its primary goal is to ensure reliable routing to multiple nodes, executing each request on a suitable provider.

This is achieved by evaluating various node characteristics including location, state, current height, and the RPC methods it can offer.



It tries to recover from connection errors, faulty nodes, invalid responses, etc.

If an upstream node lags behind, loses peers below the minimum requirement, initiates a resync, or goes offline, Dshackle temporarily removes it from the request pool.

Dshackle reinstates the connection to the node once the upstream issue is resolved.



The upstreams may be blockchain nodes such as Bitcoind, Geth, Parity, or public providers like Infura, QuickNode, etc.

It continuously checks their availability and the network's current status, executing commands while ensuring the response is consistent and data is successfully broadcast to the network.



Provides:



- Standard Bitcoin and Ethereum JSON RPC API over both HTTP and WebSocket

- An advanced gRPC-based API, featuring upstream selection, asynchronous execution, and more.

- **Secure** TLS with optional client authentication

- Blockchain-aware edge **caching**, in memory and Redis

- Intelligent routing based on **data availability** (peers, height, sync status)

- **Data consistency**, always providing the most up-to-date state

- Automatic **failover** and retry

- Separate public blockchain nodes from your internal servers



Blockchains support:



- Ethereum and Ethereum Classic

- Various Ethereum testnets

- Bitcoin

- Bitcoin testnet



image::dshackle-intro.png[alt="",width=80%,align="center"]



WARNING: The project is still under development, please use with caution.



== Quick Start



=== Configuration



Create file `dshackle.yaml` with the following content:



[source,yaml]

----

version: v1

port: 2449

tls:

  enabled: false



proxy:

  host: 0.0.0.0

  port: 8545

  routes:

    - id: eth

      blockchain: ethereum

    - id: kovan

      blockchain: kovan

    - id: btc

      blockchain: bitcoin



cluster:

  upstreams:

    - id: infura-eth

      blockchain: ethereum

      connection:

        ethereum:

          rpc:

            url: "https://mainnet.infura.io/v3/${INFURA_USER}"

          ws:

            url: "wss://mainnet.infura.io/ws/v3/${INFURA_USER}"

    - id: infura-kovan

      blockchain: kovan

      connection:

        ethereum:

          rpc:

            url: "https://kovan.infura.io/v3/${INFURA_USER}"

          ws:

            url: "wss://kovan.infura.io/ws/v3/${INFURA_USER}"

    - id: bitcoin-main

      blockchain: bitcoin

      connection:

        bitcoin:

          rpc:

            url: "http://localhost:8332"

            basic-auth:

              username: bitcoin

              password: mypassword

----



Which sets the following:



- gRPC access through 0.0.0.0:2449

** TLS security is disabled (_please don't use in production!_)

- JSON RPC access through 0.0.0.0:8545 (both HTTP and WebsScket)

** proxy requests to Ethereum and Kovan upstreams

** request path for Ethereum Mainnet is `/eth`, `/kovan` for Kovan Testnet, and `/btc` for bitcoin

** i.e. call Ethereum Mainnet by `POST http://127.0.0.0:8545/eth` with JSON RPC payload

- two upstreams, one for Ethereum Mainnet and another for Kovan Testnet (both upstreams are configured to use Infura endpoint)

- for Ethereum Mainnet it connects using JSON RPC and WebSocket connections,

- for Bitcoin Mainet only JSON RPC is used

- `${INFURA_USER}` will be provided through environment variable



Please note that you can configure many upstreams for a single blockchains.

If there is more than one upstream, then Dshackle routes requests to them as Round Robin.

If one of them becomes unavailable, Dshackle continues to use only active nodes.



I.e., you can set up a node in the local network, plus Infura with `role: fallback`.

If anything happened to your local node, you still have access to a consistent state of the Ethereum blockchain via Infura.



{link-docs}[See full documentations].



==== Run docker image



Official Docker image you can find at: https://hub.docker.com/r/emeraldpay/dshackle[emeraldpay/dshackle]



.Setup Infura username

[source,bash]

----

export INFURA_USER=...

----



.Run Dshackle

[source,bash,subs="attributes"]

----

docker run -p 2449:2449 -p 8545:8545 -v $(pwd):/etc/dshackle -e "INFURA_USER=$INFURA_USER" emeraldpay/dshackle:{version-short}

----



Now it listens on port 2449 at the localhost and can be connected from any gRPC compatible client.

Tools such as https://github.com/fullstorydev/grpcurl[gRPCurl] can automatically parse protobuf definitions and connect to it (actual Protobuf sources are located in a separate repository which you can find at https://github.com/emeraldpay/proto)



Alternatively you can connect to port 8545 with traditional JSON RPC requests



==== Access using JSON RPC over HTTP



Dshackle implements standard JSON RPC interface, providing additional caching layer, upstream readiness/liveness checks, retry and other features for building Fault Tolerant services.



.Request using Curl

[source,bash]

----

curl --request POST \

  --url http://localhost:8545/eth \

  --header 'content-type: application/json' \

  --data '{"jsonrpc":"2.0", "method":"eth_getBalance", "id":1, "params":["0x690b2bdf41f33f9f251ae0459e5898b856ed96be", "latest"]}'

----



.Output

[source,bash]

----

{"jsonrpc":"2.0","id":1,"result":"0x72fa5e0181"}

----



==== Access using JSON RPC over WebSocket



Or the same Proxy URL can be accessed through WebSocket



[source,bash]

----

websocat ws://localhost:8545/eth

----



Then make RPC calls or subscriptions:



----

> | {"jsonrpc":"2.0", "id": 1, "method": "eth_subscribe", "params": ["newHeads"]}



< | {"jsonrpc":"2.0","id":1,"result":"1f8"}

< | {"jsonrpc":"2.0","method":"eth_subscription","params":{"result":{....},"subscription":"1f8"}}

----



==== Access using gRPC



NOTE: It's not necessary to use gRPC, as Dshackle can provide standard JSON RPC proxy, but Dshackle gRPC interface improves performance and provides additional features.



Dshackle provides a custom gRPC based API, which provides additional methods and other features such as streaming responses.

Please refer to the documentation: {link-docs}/07-methods.adoc[gRPC Methods]

The Protobuf definitions could be found in link:proto/[./proto].



.Connect and listen for new blocks on Ethereum Mainnet

[source,bash]

----

grpcurl -import-path ./proto/ -proto blockchain.proto -d '{\"type\": 100}' -plaintext 127.0.0.1:2449 emerald.Blockchain/SubscribeHead

----



`type: 100` specifies the blockchain id, and 100 means Ethereum Mainnet. `1` is for Bitcoin Mainnet.

There we use Ethereum because it creates new blocks every 14 seconds, which works better for demo purposes, but the same request applied to Bitcoin as well.



.Output would be like

----

{

  "chain": "CHAIN_ETHEREUM",

  "height": 8396159,

  "blockId": "fc58a258adccc94466ae967b1178eea721349b0667f59d5fe1b0b436460bce75",

  "timestamp": 1566423564000,

  "weight": "AnMcf2VJB5kOSQ=="

}

{

  "chain": "CHAIN_ETHEREUM",

  "height": 8396160,

  "blockId": "787899711b862b77df8d2faa69de664048598265a9f96abf178d341076e200e0",

  "timestamp": 1566423574000,

  "weight": "AnMch35tO6hSGg=="

}

...

...

----



The output above is for a _streaming subscription_ to all new blocks on the Ethereum Mainnet.



It's one of the services provided by Dshackle, in addition to standard methods provided by RPC JSON of underlying nodes.



.You can also subscribe to balances changes of the balance on an address:

[source,bash]

----

grpcurl -import-path ./proto/ -proto blockchain.proto -d '{\"asset\": {\"chain\": \"100\", \"code\": \"ether\"}, \"address\": {\"address_single\": {\"address\": \"0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2\"}}}' -plaintext 127.0.0.1:2449 emerald.Blockchain/SubscribeBalance

----



.and see how balance of the contract responsible for Wrapped Ether is changing:

----

{

  "asset": {

    "chain": "CHAIN_ETHEREUM",

    "code": "ETHER"

  },

  "address": {

    "address": "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2"

  },

  "balance": "2410941696896999943701015"

}

{

  "asset": {

    "chain": "CHAIN_ETHEREUM",

    "code": "ETHER"

  },

  "address": {

    "address": "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2"

  },

  "balance": "2410930748488073834320430"

}

...

----



The balance subscription works with main coin (_ether_, _bitcoin_), or with tokens like ERC-20 if configured additionally.

See link:{link-docs}/reference-configuration.adoc[Configuration Reference].



See other enhanced methods in the link:{link-docs}/07-methods.adoc[Documentation for Enhanced Methods].



== Documentation



For detailed documentation see:



- Release {version}: {link-docs}[{link-docs}]

- Development version: link:docs/[] directory.



== Client Libraries



=== JSON RPC



Dshackle should be compatible with all standard libraries that use Ethereum JSON RPC over HTTP.



=== Java gRPC Client



https://github.com/emeraldpay/emerald-java-client



[source,groovy]

----

repositories {

    maven { url  "https://maven.emrld.io" }

}



dependencies {

    implementation 'io.emeraldpay:emerald-api:0.11.0'

}

----



=== Javascript gRPC Client



image:https://img.shields.io/npm/v/@emeraldpay/api-node.svg["npm (scoped)",link="https://www.npmjs.com/package/@emeraldpay/api-node"]



https://github.com/emeraldpay/emerald-api-js



[source,json]

----

"dependencies": {

    "@emeraldpay/api-node": "0.3.2",

}

----



See more in the documentation for {link-docs}/11-client-libraries.adoc[Client Libraries].



== Development



WARNING: The code in `master` branch is considered a development version, which may lack proper testing and should not be used in production.



=== Setting up environment



Dshackle is JVM based project written in Kotlin.

To build and run it from sources you'll need to install https://openjdk.java.net/projects/jdk/13/[Java JDK] and https://gradle.org/[Gradle]



=== Build Dshackle



==== Build everything



[source,bash]

----

gradle build

----



==== Make a Zip distribution



[source,bash]

----

gradle distZip

----



You can find a redistributable zip in `build/distributions`



==== Make a Docker distribution



[source, bash]

----

gradle jib -Pdocker=gcr.io/myproject

----



Gradle will prepare a Docker image and upload it to your custom Docker Registry at `gcr.io/myproject` (please change to address of your actual registry)



==== Architecture



Dshackle is built using:



- Kotlin

- Spring Framework + Spring Boot

- Spring Reactor

- Netty

- Etherjar

- gRPC and HTTP2 protocol

- Groovy and Spock for testing



== Community



=== Development Chat



Join our Discord chat to discuss development and ask questions:



image:https://img.shields.io/discord/1107840420240707704?style=flat-square[Discord,link="https://discord.gg/k9HpF9Jqee"]



== Commercial Support



Want to support the project, prioritize a specific feature, or get commercial help with using Dshackle in your project?

Please contact [email protected] to discuss the possibility.



== License



Copyright 2023 EmeraldPay, Inc



Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.

You may obtain a copy of the License at



http://www.apache.org/licenses/LICENSE-2.0



Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and limitations under the License.