https://github.com/shixi99/data-streaming-kafka

Data streaming showcase using Kafka, Python , PostgreSQL, Streamlit and Docker
https://github.com/shixi99/data-streaming-kafka

docker kafka postgresql python streamlit

Last synced: about 1 month ago
JSON representation

Data streaming showcase using Kafka, Python , PostgreSQL, Streamlit and Docker

• Host: GitHub
• URL: https://github.com/shixi99/data-streaming-kafka
• Owner: Shixi99
• Created: 2024-09-14T13:42:15.000Z (2 months ago)
• Default Branch: main
• Last Pushed: 2024-09-14T17:04:09.000Z (2 months ago)
• Last Synced: 2024-10-12T23:41:43.486Z (about 1 month ago)
• Topics: docker, kafka, postgresql, python, streamlit
• Language: Shell
• Homepage:
• Size: 583 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
## Data streaming showcase using Kafka (Python API), PostgreSQL, Streamlit

In this simple data streaming project, 

 - In producer service the fake data (a senetence) is generated and sent to the kafka topic every 5 seconds.

 - In consumer service kafka consumer consume messages from a Kafka topic, analyze their sentiment using the VADER Sentiment Analysis tool from NLTK, and then store the results in a PostgreSQL database.

 - In the UI, there is a streamlit dashboard that fetches data from a PostgreSQL database and displays the results on a live dashboard, updating the information in real time.

### Architecture of the project

![pipeline.img](pipeline.png)

![stream.gif](stream.gif)

To implement the project in your machine follow the steps below. First, make sure you have Docker installed on your machine.

```

mkdir my_project

cd my_project

```

```

git clone https://github.com/Shixi99/data-streaming-kafka.git

cd data-streaming-kafka

```

Add .env file to **ui** folder

```

DB_HOST=your_postgres_hostname

DB_NAME=your_postgres_db_name

DB_USER=your_postgres_username

DB_PASS=your_postgres_password

DB_PORT=5432

```

Add .env file to project folder (data-streaming-kafka)

```

POSTGRES_USER=your_postgres_username

POSTGRES_PASSWORD=your_postgres_password

POSTGRES_DB=your_postgres_db_name

DB_NAME=your_postgres_db_name

DB_USER=your_postgres_username

DB_PASSWORD=your_postgres_password

```

Start docker containers

```

docker-compose up -d

```

Stop docker containers

```

docker-compose stop

```

Or stop and remove containers

```

docker-compose down

```