Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/priyanka387/ml-data-pipeline
The project aims to read the data from sensors attached in heavy vehicles and push it to kafka. Apache Kafka is a distributed event store and stream-processing platform. We are finally consuming the data from kafka to Mongodb.
https://github.com/priyanka387/ml-data-pipeline
kafka kafka-cluster kafka-consumer kafka-producer kafkatopic mongodb-atlas mongodb-database
Last synced: 13 days ago
JSON representation
The project aims to read the data from sensors attached in heavy vehicles and push it to kafka. Apache Kafka is a distributed event store and stream-processing platform. We are finally consuming the data from kafka to Mongodb.
- Host: GitHub
- URL: https://github.com/priyanka387/ml-data-pipeline
- Owner: priyanka387
- Created: 2022-12-16T10:19:07.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2022-12-16T10:36:54.000Z (about 2 years ago)
- Last Synced: 2024-11-16T08:28:13.611Z (2 months ago)
- Topics: kafka, kafka-cluster, kafka-consumer, kafka-producer, kafkatopic, mongodb-atlas, mongodb-database
- Language: Jupyter Notebook
- Homepage:
- Size: 10 MB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# confluent-kafka-python
This repo help us to know how to publish and consume data to and from kafka confluent in json format.
Step 1: Create a conda environment
```
conda --version
```
Step2: Create a conda environment
```
conda create -p venv python==3.8 -y
```
Step3:
```
conda activate venv/
```
Step4:
```
pip install -r requirements.txt
```
Cluster Environment Variable
```
API_KEY
API_SECRET_KEY
BOOTSTRAP_SERVER
```
Schema related Environment Variable
```
SCHEMA_REGISTRY_API_KEY
SCHEMA_REGISTRY_API_SECRET
ENDPOINT_SCHEMA_URL
```
Data base related Environment Variable
```
MONGO_DB_URL
```
## Update the credential in .env file and run below command to run your application in docker container
Create .env file in root dir of your project if it is not available
paste the below content and update the credentials
```
API_KEY=asgdakhlsa
API_SECRET_KEY=dsdfsdf
BOOTSTRAP_SERVER=sdfasd
SCHEMA_REGISTRY_API_KEY=sdfsaf
SCHEMA_REGISTRY_API_SECRET=sdfasdf
ENDPOINT_SCHEMA_URL=sdafasf
MONGO_DB_URL=sdfasdfas
```
Build docker image
```
docker build -t data-pipeline:lts .
```
For linux or mac
Run docker image
```
docker run -it -v $(pwd)/logs:/logs --env-file=$(pwd)/.env data-pipeline:lts
```