https://github.com/srgchrksv/stream-crypto

Crypto trades streaming with azure services
https://github.com/srgchrksv/stream-crypto

azure binance crypto data databricks dataengineering pyspark python streaming websocket

Last synced: about 2 months ago
JSON representation

Crypto trades streaming with azure services

• Host: GitHub
• URL: https://github.com/srgchrksv/stream-crypto
• Owner: srgchrksv
• Created: 2024-03-05T20:32:51.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2024-03-06T21:53:10.000Z (over 2 years ago)
• Last Synced: 2025-02-26T08:43:55.577Z (over 1 year ago)
• Topics: azure, binance, crypto, data, databricks, dataengineering, pyspark, python, streaming, websocket
• Language: Jupyter Notebook
• Homepage:
• Size: 311 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: readme.md

Awesome Lists containing this project

README

          
## Streaming crypto trades data

Got a python script that consumes trades data from binance and produces messages to azure eventhubs.

Databricks notebook processes eventhubs with pyspark and writes delta tables.

Power BI using partner connect to Databricks streams and visualize data

### Streaming system

!["Stream"](/docs/stream.png)

## Silver layer table

!["Silver Layer Table"](/docs/silver-layer.png)

The gold layer of this stream system holds separate table for each symbol

## PowerBI  databricks streaming visualizations

!["Power BI Of Silver Layer Table"](/docs/powerbi.png)

Useful links: 

- [https://github.com/malvik01/Real-Time-Streaming-with-Azure-Databricks](https://github.com/malvik01/Real-Time-Streaming-with-Azure-Databricks)

- [https://www.youtube.com/watch?v=pwWIegHgNRw](https://www.youtube.com/watch?v=pwWIegHgNRw)

- [https://github.com/binance/binance-spot-api-docs/blob/master/web-socket-streams.md](https://github.com/binance/binance-spot-api-docs/blob/master/web-socket-streams.md)