https://github.com/dzakarias/orderbook-replayer
Replay/analyse historical orderbook data
https://github.com/dzakarias/orderbook-replayer
cryptocurrency docker fastapi orderbook poetry python vuejs
Last synced: about 1 month ago
JSON representation
Replay/analyse historical orderbook data
- Host: GitHub
- URL: https://github.com/dzakarias/orderbook-replayer
- Owner: dzakarias
- License: mit
- Created: 2025-01-01T13:29:58.000Z (10 months ago)
- Default Branch: master
- Last Pushed: 2025-01-14T21:30:44.000Z (9 months ago)
- Last Synced: 2025-06-20T03:06:26.723Z (4 months ago)
- Topics: cryptocurrency, docker, fastapi, orderbook, poetry, python, vuejs
- Language: Python
- Homepage: https://orderbook-replayer.onrender.com/
- Size: 5.06 MB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Orderbook Replayer
A small fullstack app: a Python FastAPI backend maintaining state of an archive central limit order book (CLOB), and a Vue.js frontend visualizing said state, and providing controls for replay.
## Screenshots
## Orderbooks
The app works on historical orderbook data. Bybit (a major cryptocurrency exchange) provides 500-depth orderbook history for free download. The problem with this is size: a typical day of BTCUSDT trading is well over a gigabyte in the format they provide. So the backend includes a utility class (OrderBookProcessor) that takes such a Bybit history file as input, and outputs a new file with configurable depth. For an example, the repo includes a 20-depth book for February 3, 2024 for BNBUSDT. 20 levels are enough for most practical purposes and they require 3-10x less space than a full, depth-500 one.
## More technical details on orderbooks
Bybit's files include a snapshot at start, i.e. all the 500 price levels and quantities for both the Bid and the Ask side. After this snapshot, a set of deltas are provided: they represent changed quantities for a given price level: 0 means no more offers on that price.
Translating a depth 500 file to a depth 20 one means the deltas have to be processed, and for each set of deltas (one set contains all the deltas carrying the same timestamp) internal state is updated, and a - potentially different - set of deltas is computed. For example deltas more than 20 levels beneath best price need not be propagated at this point: but it may become necessary to include them later if they enter top-20 (if no other delta for its price level appears in the meantime).
Some (most) deltas never become relevant, so will never make it to the output file - that's how space can be saved.
The internal orderbook implementation uses sorted lists of tuples and the [bisect](https://docs.python.org/3/library/bisect.html) module for quick (O(log n)) lookups essential for processing performance: enabling conversion speed of around 10K delta-sets per second on big pairs (like BTCUSDT) for a 500->20 conversion.
## Run Locally
Clone the project:
```bash
git clone https://github.com/dzakarias/orderbook-replayer.git
```
Go to the project directory:
```bash
cd orderbook-replayer
```
The backend targets Python 3.11 and uses Poetry for dependencies. When you have these installed, you install dependencies with:
```bash
poetry install
```
Start the backend:
```bash
poetry run uvicorn src.backend.ob_replayer_backend:app --host 0.0.0.0 --port 8000 --reload
```
Run the frontend by opening http://localhost:8000 in your browser. It isn't necessary to build the frontend, so you don't need Node.js, Webpack, etc. at all.
Still a package.json is provided in the frontend directory, so you can run `npm install` and `npm run build` if you insist.
## Deploy online
The project includes a Dockerfile you can use to deploy it in a containerized manner. Here's it deployed to Render: https://orderbook-replayer.onrender.com/
It might take a minute to load as it's on a free account: so uses a Spot instance on AWS which take some time to fire up.
## Usage
When the app loads, select a date. A list of markets for which orderbook histories are available at the backend will be populated. Choose one to load the initial snapshot of that orderbook.
The repo includes only one orderbook file, so you'll need to select 02-03-2024 as date, and the BNBUSDT book history will be loaded.
## How can I create my own orderbook files?
Goto [Bybit's historical data download page](https://www.bybit.com/derivatives/en/history-data), download the orderbook(s) you need, then extract them. Do not modify the filename. Run orderbook processor (/src/backend/orderbook_processor.py) like this:
```bash
python -m orderbook_processor --file [--depth ]
```
The output file will be created in the same directory, and the same filename-structure will be maintained: the "ob500" part will be modified only to reflect the output depth.
## License
[MIT](https://choosealicense.com/licenses/mit/)
## Author
[@dzakarias](https://www.github.com/dzakarias) David Zakarias. Contact me at [LinkedIn](https://www.linkedin.com/in/david-zakarias-9720a24/) or email me at david (dot) zakarias (at) gmail (dot) com.