Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/BitcoinExchangeFH/BitcoinExchangeFH

Cryptocurrency exchange market data feed handler
https://github.com/BitcoinExchangeFH/BitcoinExchangeFH

bitcoin bitfinex bitmex bitstamp btcc exchange-data gatecoin gdax huobi kraken market-data okcoin quoine

Last synced: 22 days ago
JSON representation

Cryptocurrency exchange market data feed handler

Awesome Lists containing this project

README 

        


  




# BitcoinExchangeFH - Cryptocurrency exchange market data feed handler



BitcoinExchangeFH is a slim application to record the price depth and trades in various exchanges. You can set it up quickly and record the all the exchange data in a few minutes!



Users can



1. Streaming market data to a target application (via ZeroMQ)

2. Recording market data for backtesting and analysis.

3. Recording market data to a in-memory database and other applications can quickly access to it.

4. Customize the project for trading use.



### MySQL





  




### Kdb+





  




## Supported exchanges



All exchanges supported by [ccxt](https://github.com/ccxt/ccxt). Currently more than 130 exchanges are supported.



Websocket feeds of the following exchanges are supported by [cryptofeed](https://github.com/bmoscon/cryptofeed)



- Bitfinex



- Coinbase



- Poloniex



- Gemini



- HitBTC



- Bitstamp



- BitMEX



- Kraken



- Binance



- EXX



- Huobi



- OKCoin



- OKEx



If the exchange is not supported with websocket API feed, it will automatically fall into using its REST API feed.



## Supported database/channel



- RDMBS (e.g. sqlite, MySQL, PostgreSQL)



- ZeroMQ



- Kdb+ (Coming soon)



## Getting started



```

pip install bitcoinexchangefh

bitcoinexchangefh --configuration example/configuration.yaml

```



## Configuration



The configuration follows [YAML](https://pyyaml.org/wiki/PyYAMLDocumentation) syntax and contains two sections



- subscriptions



- handlers



### Subscriptions



Subscription section specifies the exchange and instruments to subscribe. 



The first key is the exchange name and then follows the exchange details, 

    

- instruments 



- number of depth (default is 5 if not specified)



For example, 



```

subscription:

    Binance:

        instruments:

            - XRP/BTC

            - BCH/BTC

        depth

    Poloniex:

        instruments:

            - ETH/BTC

        depth: 10

```



### Handlers



After receiving the order book or trade update, each handler is updated. For example, for SQL database handler, it is updated with the corresponding SQl statements.



For example,



```

handlers:

    sql: 

        connection: "sqlite://"

    

```



#### SQL handler



The following settings can be customized



|Parameter|Description|

|---|---|

|connection|Database connection string required by [SQLAlchemy](https://docs.sqlalchemy.org/en/latest/core/engines.html)|

|is_rotate|Boolean indicating whether to rotate to record the table.|

|rotate_frequency|String in [format](https://docs.python.org/2/library/datetime.html#strftime-strptime-behavior) same as `strftime` and `strptime`|



#### ZeroMQ handler



The feed handler acts as a [publisher](https://learning-0mq-with-pyzmq.readthedocs.io/en/latest/pyzmq/patterns/pubsub.html) in ZeroMQ. To receive the feed, please follow ZeroMQ instructions to start a [subscriber](tests/zmq/zmq_subscriber.py).



The following settings can be customized



|Parameter|Description|

|---|---|

|connection|Connection [format](http://api.zeromq.org/3-2:zmq-connect) in ZeroMQ. For example, "tcp://127.0.0.1:3456"|



## Examples



You can first create a directory `.data` and run the command



```

$ bitcoinexchangefh --configuration example/configuration.yaml 

2020-08-07 23:33:32,110 INFO Loading runner

2020-08-07 23:33:32,110 INFO Creating handler sql

2020-08-07 23:33:32,205 INFO Loading handler SqlHandler

2020-08-07 23:33:32,710 INFO Loading exchange Binance

2020-08-07 23:33:34,062 INFO Loading exchange Bitmex

2020-08-07 23:33:40,203 INFO Start running the feed handler

2020-08-07 23:33:40,203 INFO Running handler sql

2020-08-07 23:33:40,210 INFO Running exchange Binance

2020-08-07 23:33:40,215 INFO Running SqlHandler

2020-08-07 23:33:40,219 INFO Running exchange Bitmex

2020-08-07 23:33:40,231 INFO Joining all the processes

2020-08-07 23:33:40,232 INFO Creating table binance_ethbtc_order

2020-08-07 23:33:40,263 INFO Created table binance_ethbtc_order

2020-08-07 23:33:40,265 INFO Creating table binance_xrpbtc_order

2020-08-07 23:33:40,294 INFO Created table binance_xrpbtc_order

2020-08-07 23:33:40,297 INFO Creating table bitmex_ethusd_order

2020-08-07 23:33:40,322 INFO Created table bitmex_ethusd_order

```



After a while, the data is streamed into the file `.data/order_book.db`. You can terminate the process and the database contains the following tables.



```

$ python -c "import sqlite3; print(sqlite3.connect('.data/order_book.db').execute('SELECT name from sqlite_master where type= \"table\"').fetchall())"

[('binance_ethbtc_order',), ('binance_xrpbtc_order',), ('bitmex_ethusd_order',)]



```



## Inquiries



You can first look up to the page [FAQ](https://github.com/gavincyi/BitcoinExchangeFH/wiki/FAQ). For more inquiries, you can either leave it in issues or drop me an email. I will get you back as soon as possible.



## Compatibility

The application is compatible with version higher or equal to python 3.0.



## Contributions

Always welcome for any contribution. Please fork the project, make the changes, and submit the merge request. :)



For any questions and comment, please feel free to contact me through email (gavincyi at gmail)



Your comment will be a huge contribution to the project!



## Continuity

If you are not satisified with python performance, you can contact me to discuss migrating the project into other languages, e.g. C++.