https://github.com/blockchain-etl/blockchain-streaming-analytics
Blockchain streaming analytics
https://github.com/blockchain-etl/blockchain-streaming-analytics
Last synced: 1 day ago
JSON representation
Blockchain streaming analytics
- Host: GitHub
- URL: https://github.com/blockchain-etl/blockchain-streaming-analytics
- Owner: blockchain-etl
- License: mit
- Created: 2020-03-27T09:19:41.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2022-11-16T03:20:10.000Z (over 2 years ago)
- Last Synced: 2025-06-03T18:28:01.549Z (23 days ago)
- Language: Java
- Size: 105 KB
- Stars: 9
- Watchers: 3
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Blockchain Streaming Analytics
Pipeline in this repo aggregates Ethereum transactions to candlestick records over defined windows
and publishes results to PubSub topics.
## How to run
### Create output topics manually
Where:
- "ethereum_candles_" - topic prefix, default value of the `--outputTopicsPrefix` pipeline option;
- "60", "180", "300", "900" - aggregation windows in seconds, default value of the `--aggregationWindowsInSeconds` pipeline options.
```shell script
gcloud pubsub topics create ethereum_candles_60
gcloud pubsub topics create ethereum_candles_180
gcloud pubsub topics create ethereum_candles_300
gcloud pubsub topics create ethereum_candles_900
```
Note: we might refactor later to durations in ISO-8601 notation e.g. - PT1M, PT3M etc.
### Build pipeline
```shell script
mvn clean package
```
### Start pipeline locally
```shell script
export PROJECT=$(gcloud config get-value project 2> /dev/null)
java -cp target/blockchain-streaming-analytics-bundled-0.1.jar io.blockchainetl.streaming.candlestick.CandlestickPipeline \
--runner=DirectRunner \
--project=$PROJECT \
--inputDataTopicOrSubscription=projects/$PROJECT/subscriptions/crypto_ethereum.transactions
```
### Or start pipeline in GCP with Dataflow runner
```shell script
export PROJECT=$(gcloud config get-value project 2> /dev/null)
java -cp target/blockchain-streaming-analytics-bundled-0.1.jar io.blockchainetl.streaming.candlestick.CandlestickPipeline \
--runner=org.apache.beam.runners.dataflow.DataflowRunner \
--project=$PROJECT \
--inputDataTopicOrSubscription=projects/$PROJECT/subscriptions/crypto_ethereum.transactions
```