https://github.com/seilylook/cryptocurrency-data-pipeline
This project involved the integration of several cutting-edge technologies to ensure efficient data processing and management for Cryptocurrency analytics.
https://github.com/seilylook/cryptocurrency-data-pipeline
airflow crpyto hdfs hive hue livy postgresql spark
Last synced: 8 months ago
JSON representation
This project involved the integration of several cutting-edge technologies to ensure efficient data processing and management for Cryptocurrency analytics.
- Host: GitHub
- URL: https://github.com/seilylook/cryptocurrency-data-pipeline
- Owner: seilylook
- License: apache-2.0
- Created: 2024-06-06T03:22:21.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-06-07T06:15:47.000Z (over 1 year ago)
- Last Synced: 2024-12-28T03:15:39.749Z (10 months ago)
- Topics: airflow, crpyto, hdfs, hive, hue, livy, postgresql, spark
- Language: Shell
- Homepage:
- Size: 59.6 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Flow chart of the Cryptocurrency Data Pipeline
1. Check availability of Cryptocurrency cost in CoinMarketCap
2. Download top 10th Cryptocurrency(ex. BTC, ETH) cost with python
4. Save the Cryptocurrency cost in HDFS
5. Create a hive table to store Cryptocurrency cost from the HDFS
6. Process Cryptocurrency cost with Spark
7. Send an email notificatoin