Emerald Dshackle is a Fault Tolerant Load Balancer for Blockchain API.
Its primary goal is to ensure reliable routing to multiple nodes, executing each request on a suitable provider. This is achieved by evaluating various node characteristics including location, state, current height, and the RPC methods it can offer.
It tries to recover from connection errors, faulty nodes, invalid responses, etc. If an upstream node lags behind, loses peers below the minimum requirement, initiates a resync, or goes offline, Dshackle temporarily removes it from the request pool. Dshackle reinstates the connection to the node once the upstream issue is resolved.
The upstreams may be blockchain nodes such as Bitcoind, Geth, Parity, or public providers like Infura, QuickNode, etc. It continuously checks their availability and the network’s current status, executing commands while ensuring the response is consistent and data is successfully broadcast to the network.
Provides:
-
Standard Bitcoin and Ethereum JSON RPC API over both HTTP and WebSocket
-
An advanced gRPC-based API, featuring upstream selection, asynchronous execution, and more.
-
Secure TLS with optional client authentication
-
Blockchain-aware edge caching, in memory and Redis
-
Intelligent routing based on data availability (peers, height, sync status)
-
Data consistency, always providing the most up-to-date state
-
Automatic failover and retry
-
Separate public blockchain nodes from your internal servers
Blockchains support:
-
Ethereum and Ethereum Classic
-
Various Ethereum testnets
-
Bitcoin
-
Bitcoin testnet
|
Warning
|
The project is still under development, please use with caution. |
Create file dshackle.yaml with the following content:
version: v1
port: 2449
tls:
enabled: false
proxy:
host: 0.0.0.0
port: 8545
routes:
- id: eth
blockchain: ethereum
- id: kovan
blockchain: kovan
- id: btc
blockchain: bitcoin
cluster:
upstreams:
- id: infura-eth
blockchain: ethereum
connection:
ethereum:
rpc:
url: "https://mainnet.infura.io/v3/${INFURA_USER}"
ws:
url: "wss://mainnet.infura.io/ws/v3/${INFURA_USER}"
- id: infura-kovan
blockchain: kovan
connection:
ethereum:
rpc:
url: "https://kovan.infura.io/v3/${INFURA_USER}"
ws:
url: "wss://kovan.infura.io/ws/v3/${INFURA_USER}"
- id: bitcoin-main
blockchain: bitcoin
connection:
bitcoin:
rpc:
url: "http://localhost:8332"
basic-auth:
username: bitcoin
password: mypasswordWhich sets the following:
-
gRPC access through 0.0.0.0:2449
-
TLS security is disabled (please don’t use in production!)
-
-
JSON RPC access through 0.0.0.0:8545 (both HTTP and WebsScket)
-
proxy requests to Ethereum and Kovan upstreams
-
request path for Ethereum Mainnet is
/eth,/kovanfor Kovan Testnet, and/btcfor bitcoin -
i.e. call Ethereum Mainnet by
POST http://127.0.0.0:8545/ethwith JSON RPC payload
-
-
two upstreams, one for Ethereum Mainnet and another for Kovan Testnet (both upstreams are configured to use Infura endpoint)
-
for Ethereum Mainnet it connects using JSON RPC and WebSocket connections,
-
for Bitcoin Mainet only JSON RPC is used
-
${INFURA_USER}will be provided through environment variable
Please note that you can configure many upstreams for a single blockchains. If there is more than one upstream, then Dshackle routes requests to them as Round Robin. If one of them becomes unavailable, Dshackle continues to use only active nodes.
I.e., you can set up a node in the local network, plus Infura with role: fallback.
If anything happened to your local node, you still have access to a consistent state of the Ethereum blockchain via Infura.
Official Docker image you can find at: emeraldpay/dshackle
export INFURA_USER=...docker run -p 2449:2449 -p 8545:8545 -v $(pwd):/etc/dshackle -e "INFURA_USER=$INFURA_USER" emeraldpay/dshackle:0.15Now it listens on port 2449 at the localhost and can be connected from any gRPC compatible client. Tools such as gRPCurl can automatically parse protobuf definitions and connect to it (actual Protobuf sources are located in a separate repository which you can find at https://github.com/emeraldpay/proto)
Alternatively you can connect to port 8545 with traditional JSON RPC requests
Dshackle implements standard JSON RPC interface, providing additional caching layer, upstream readiness/liveness checks, retry and other features for building Fault Tolerant services.
curl --request POST \
--url http://localhost:8545/eth \
--header 'content-type: application/json' \
--data '{"jsonrpc":"2.0", "method":"eth_getBalance", "id":1, "params":["0x690b2bdf41f33f9f251ae0459e5898b856ed96be", "latest"]}'{"jsonrpc":"2.0","id":1,"result":"0x72fa5e0181"}Or the same Proxy URL can be accessed through WebSocket
websocat ws://localhost:8545/ethThen make RPC calls or subscriptions:
> | {"jsonrpc":"2.0", "id": 1, "method": "eth_subscribe", "params": ["newHeads"]}
< | {"jsonrpc":"2.0","id":1,"result":"1f8"}
< | {"jsonrpc":"2.0","method":"eth_subscription","params":{"result":{....},"subscription":"1f8"}}
|
Note
|
It’s not necessary to use gRPC, as Dshackle can provide standard JSON RPC proxy, but Dshackle gRPC interface improves performance and provides additional features. |
Dshackle provides a custom gRPC based API, which provides additional methods and other features such as streaming responses. Please refer to the documentation: gRPC Methods The Protobuf definitions could be found in ./proto.
grpcurl -import-path ./proto/ -proto blockchain.proto -d '{\"type\": 100}' -plaintext 127.0.0.1:2449 emerald.Blockchain/SubscribeHeadtype: 100 specifies the blockchain id, and 100 means Ethereum Mainnet. 1 is for Bitcoin Mainnet.
There we use Ethereum because it creates new blocks every 14 seconds, which works better for demo purposes, but the same request applied to Bitcoin as well.
{
"chain": "CHAIN_ETHEREUM",
"height": 8396159,
"blockId": "fc58a258adccc94466ae967b1178eea721349b0667f59d5fe1b0b436460bce75",
"timestamp": 1566423564000,
"weight": "AnMcf2VJB5kOSQ=="
}
{
"chain": "CHAIN_ETHEREUM",
"height": 8396160,
"blockId": "787899711b862b77df8d2faa69de664048598265a9f96abf178d341076e200e0",
"timestamp": 1566423574000,
"weight": "AnMch35tO6hSGg=="
}
...
...
The output above is for a streaming subscription to all new blocks on the Ethereum Mainnet.
It’s one of the services provided by Dshackle, in addition to standard methods provided by RPC JSON of underlying nodes.
grpcurl -import-path ./proto/ -proto blockchain.proto -d '{\"asset\": {\"chain\": \"100\", \"code\": \"ether\"}, \"address\": {\"address_single\": {\"address\": \"0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2\"}}}' -plaintext 127.0.0.1:2449 emerald.Blockchain/SubscribeBalance{
"asset": {
"chain": "CHAIN_ETHEREUM",
"code": "ETHER"
},
"address": {
"address": "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2"
},
"balance": "2410941696896999943701015"
}
{
"asset": {
"chain": "CHAIN_ETHEREUM",
"code": "ETHER"
},
"address": {
"address": "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2"
},
"balance": "2410930748488073834320430"
}
...
The balance subscription works with main coin (ether, bitcoin), or with tokens like ERC-20 if configured additionally. See Configuration Reference.
See other enhanced methods in the Documentation for Enhanced Methods.
For detailed documentation see:
-
Release 0.15.0: https://github.com/emeraldpay/dshackle/tree/release/v0.15/docs
-
Development version: docs/ directory.
Dshackle should be compatible with all standard libraries that use Ethereum JSON RPC over HTTP.
repositories {
maven { url "https://maven.emrld.io" }
}
dependencies {
implementation 'io.emeraldpay:emerald-api:0.11.0'
}
"dependencies": {
"@emeraldpay/api-node": "0.3.2",
}See more in the documentation for Client Libraries.
|
Warning
|
The code in master branch is considered a development version, which may lack proper testing and should not be used in production.
|
gradle jib -Pdocker=gcr.io/myprojectGradle will prepare a Docker image and upload it to your custom Docker Registry at gcr.io/myproject (please change to address of your actual registry)
Want to support the project, prioritize a specific feature, or get commercial help with using Dshackle in your project? Please contact [email protected] to discuss the possibility.
Copyright 2023 EmeraldPay, Inc
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent