https://github.com/lixx21/kafka-ibm-stocking
Move data to JSON file real time using Kafka for IBM Stocking Data
https://github.com/lixx21/kafka-ibm-stocking
data-engineer kafka kafka-streams
Last synced: over 1 year ago
JSON representation
Move data to JSON file real time using Kafka for IBM Stocking Data
- Host: GitHub
- URL: https://github.com/lixx21/kafka-ibm-stocking
- Owner: lixx21
- Created: 2023-11-18T16:36:16.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-11-18T16:39:09.000Z (over 2 years ago)
- Last Synced: 2025-02-09T11:41:44.106Z (over 1 year ago)
- Topics: data-engineer, kafka, kafka-streams
- Language: Jupyter Notebook
- Homepage:
- Size: 3.91 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.MD
Awesome Lists containing this project
README
# Kafka IBM Real Time Stocking
## Dataset
In this project, we will use this dataset:
[https://www.kaggle.com/datasets/bhanuprasanna527/stock-market-prediction](https://www.kaggle.com/datasets/bhanuprasanna527/stock-market-prediction)
## Install Environment
- install kafka ```wget https://downloads.apache.org/kafka/3.3.1/kafka_2.12-3.3.1.tgz```
- extract kafka ```tar -xvf kafka_2.12-3.3.1.tgz```
- install java jdk 1.8.0 (set environment for JAVA_HOME)
## Start Kafka Server
- move directory to kafka: ```cd kafka_2.12-3.3.1```
- Start Zookeeper: ```bin/zookeeper-server-start.sh config/zookeeper.properties```
- Start Broker: ```bin/kafka-server-start.sh config/server.properties```
- Create Topic:```bin/kafka-topics.sh --create --topic demo_kafka --bootstrap-server localhost:9092```
- To see Topic's lists: ```bin/kafka-topics.sh --bootstrap-server localhost:9092 --list```
- Start Producer: ```bin/kafka-console-producer.sh --topic demo_kafka --bootstrap-server localhost:9092```
- Start Consumer: ```bin/kafka-console-consumer.sh --topic demo_kafka --bootstrap-server localhost:9092```
## How to use
- clone this github ```git clone https://github.com/lixx21/kafka-IBM-Stocking.git```
- install all libraries ```pip install -r requirements.txt```
- run all shell in **kafka_consumer.ipynb** and **kafka_producer.ipynb**