https://github.com/mneedham/pinot-cryptowatch
https://github.com/mneedham/pinot-cryptowatch
Last synced: 8 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/mneedham/pinot-cryptowatch
- Owner: mneedham
- Created: 2022-04-06T13:20:31.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2022-04-13T14:25:56.000Z (about 4 years ago)
- Last Synced: 2025-04-11T14:57:01.387Z (about 1 year ago)
- Language: Python
- Size: 3.67 MB
- Stars: 3
- Watchers: 2
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.adoc
Awesome Lists containing this project
README
# Cryptowatch on Pinot
https://docs.cryptowat.ch/websocket-api/data-subscriptions/trades
## trades table
[source,bash]
----
docker exec -it pinot-controller-crypto bin/pinot-admin.sh AddTable \
-tableConfigFile /config/trades_table.json \
-schemaFile /config/trades_schema.json \
-exec
----
[source,bash]
----
docker exec -it pinot-controller-crypto bin/pinot-admin.sh AddTable \
-tableConfigFile /config/trades_offline_table.json \
-exec
----
[source,bash]
----
docker exec -it pinot-controller-crypto bin/pinot-admin.sh LaunchDataIngestionJob \
-jobSpecFile /config/trades-job-spec.yml
----
## markets table
[source,bash]
----
docker exec -it pinot-controller-crypto bin/pinot-admin.sh AddTable \
-tableConfigFile /config/markets_table.json \
-schemaFile /config/markets_schema.json \
-exec
----
[source, bash]
----
export tableName="markets"
curl -X DELETE "http://localhost:9000/segments/${tableName}?type=OFFLINE" -H "accept: application/json" &&
docker exec -it pinot-controller-crypto bin/pinot-admin.sh LaunchDataIngestionJob \
-jobSpecFile /config/${tableName}-refresh-job-spec.yml
----
[source,bash]
----
docker exec -it pinot-controller-crypto bin/pinot-admin.sh LaunchDataIngestionJob \
-jobSpecFile /config/markets-job-spec.yml
----
## exchanges table
[source,bash]
----
docker exec -it pinot-controller-crypto bin/pinot-admin.sh AddTable \
-tableConfigFile /config/exchanges_table.json \
-schemaFile /config/exchanges_schema.json \
-exec
----
[source,bash]
----
docker exec -it pinot-controller-crypto bin/pinot-admin.sh LaunchDataIngestionJob \
-jobSpecFile /config/exchanges-job-spec.yml
----
[source, bash]
----
export tableName="exchanges"
curl -X DELETE "http://localhost:9000/segments/${tableName}?type=OFFLINE" -H "accept: application/json" &&
docker exec -it pinot-controller-crypto bin/pinot-admin.sh LaunchDataIngestionJob \
-jobSpecFile /config/${tableName}-refresh-job-spec.yml
----
## pairs table
[source,bash]
----
docker exec -it pinot-controller-crypto bin/pinot-admin.sh AddTable \
-tableConfigFile /config/pairs_table.json \
-schemaFile /config/pairs_schema.json \
-exec
----
[source,bash]
----
docker exec -it pinot-controller-crypto bin/pinot-admin.sh LaunchDataIngestionJob \
-jobSpecFile /config/pairs-job-spec.yml
----
[source, bash]
----
export tableName="pairs"
curl -X DELETE "http://localhost:9000/segments/${tableName}?type=OFFLINE" -H "accept: application/json" &&
docker exec -it pinot-controller-crypto bin/pinot-admin.sh LaunchDataIngestionJob \
-jobSpecFile /config/${tableName}-refresh-job-spec.yml
----
[source, bash]
----
docker exec -i kafka-crypto /opt/kafka/bin/kafka-topics.sh \
--bootstrap-server localhost:9092 \
--topic trades \
--create \
--partitions 10
----
[source, bash]
----
docker exec -i kafka-crypto /opt/kafka/bin/kafka-console-consumer.sh \
--bootstrap-server localhost:9092 \
--topic trades \
--from-beginning \
--max-messages 5
----
## Ideas
* Calculate the total market capitalisation
* Candle light chart showing the moving average of prices for a specific currency pair
* Number of trades per for each pair
* Drill down to see what's happening in e.g. Bitcoin
* What's happening right now vs historical analysis
* Biggest gainers (in terms of coins/prices)
## Generate schema outline
[source,bash]
----
docker run \
-v ${PWD}/trades.json:/tmp/trades.json \
-v ${PWD}/output:/output \
apachepinot/pinot:0.9.3 JsonToPinotSchema \
-jsonFile /tmp/trades.json \
-pinotSchemaName="trades" \
-outputDir="/output" \
-metrics "marketUpdate.tradesUpdate.trades.priceStr,marketUpdate.tradesUpdate.trades.amountStr" \
-timeColumnName marketUpdate.tradesUpdate.trades.timestampNano \
-fieldsToUnnest=marketUpdate.tradesUpdate.trades,marketUpdate.market
----