Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/samiksha-khare/crypto-real-time-analysis-using-kafka
Real-time analysis, visualization, and storage of cryptocurrency data with Kafka, Apache Spark, and MongoDB
https://github.com/samiksha-khare/crypto-real-time-analysis-using-kafka
api cryptocurrency kafka kafka-consumer kafka-producer kafka-topic matplotlib mongodb nosql python real-time spark visualization zookeeper
Last synced: about 1 month ago
JSON representation
Real-time analysis, visualization, and storage of cryptocurrency data with Kafka, Apache Spark, and MongoDB
- Host: GitHub
- URL: https://github.com/samiksha-khare/crypto-real-time-analysis-using-kafka
- Owner: samiksha-khare
- Created: 2024-11-11T07:51:12.000Z (about 2 months ago)
- Default Branch: main
- Last Pushed: 2024-11-11T09:24:10.000Z (about 2 months ago)
- Last Synced: 2024-11-11T10:26:20.262Z (about 2 months ago)
- Topics: api, cryptocurrency, kafka, kafka-consumer, kafka-producer, kafka-topic, matplotlib, mongodb, nosql, python, real-time, spark, visualization, zookeeper
- Language: Jupyter Notebook
- Homepage:
- Size: 878 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Real-Time Cryptocurrency Data Analysis
## 📄 Project Overview
This project focuses on real-time cryptocurrency data analysis using **Apache Kafka**, **MongoDB**, and **Python** for data visualization. The system fetches live cryptocurrency data from the CoinCap public [API](https://api.coincap.io/v2/assets), processes it using Kafka, stores it in MongoDB, and visualizes the data to observe price fluctuations over time.
## 🛠️ Architecture
The project follows a **Producer-Consumer** pattern:
- **Producer**: Continuously fetches cryptocurrency data from the API and publishes it to a Kafka topic.
- **Consumer**: Consumes data from the Kafka topic, processes it using Apache Spark, and stores the processed data in MongoDB for further visualization.
---
## 🛠️ Technologies Used
- **Python**: Data fetching, processing, and programming.
- **Apache Kafka**: Real-time data streaming.
- **MongoDB**: NoSQL database for storing historical data.
- **CoinCap API**: Source for live cryptocurrency data.
- **Pandas**: Data analysis and manipulation.
- **Matplotlib**: Data visualization and plotting.
---
## ⚙️ Initial Setup
Follow the steps below to set up the project on your local system:
### Step 1: Download and Extract Kafka
```
wget https://downloads.apache.org/kafka/3.8.1/kafka_2.12-3.8.1.tgz
tar -xvf kafka_2.12-3.8.1.tgz
cd kafka_2.12-3.8.1
```
### Step 2: Start Zookeeper Server
```
bin/zookeeper-server-start.sh config/zookeeper.properties
```
### Step 3: Start Kafka Server
```
bin/kafka-server-start.sh config/server.properties
```
### Step 4: Create a Kafka Topic
```
bin/kafka-topics.sh --create --topic crypto-currency --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
```
### Step 5: Start Producer
```
bin/kafka-console-producer.sh --topic crypto-currency --bootstrap-server localhost:9092
```
### Step 6: Start Consumer
```
bin/kafka-console-consumer.sh --topic crypto-currency --bootstrap-server localhost:9092 --from-beginning
```
### Step 7: Install MongoDB
```
brew tap mongodb/brew
brew install mongodb-community
brew services start mongodb/brew/mongodb-community
```