https://github.com/city-of-helsinki/mittaridatapumppu

Realtime data processing and management
https://github.com/city-of-helsinki/mittaridatapumppu

Last synced: about 2 months ago
JSON representation

Realtime data processing and management

• Host: GitHub
• URL: https://github.com/city-of-helsinki/mittaridatapumppu
• Owner: City-of-Helsinki
• License: apache-2.0
• Created: 2023-05-09T09:28:02.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2024-06-26T10:21:03.000Z (7 months ago)
• Last Synced: 2024-06-27T11:14:15.866Z (7 months ago)
• Language: Python
• Size: 1.89 MB
• Stars: 1
• Watchers: 7
• Forks: 2
• Open Issues: 5
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# mittaridatapumppu

Realtime data processing and management

![plot](architecture.png)

# Introduction

The following repositories contain the services for the data pipeline shown in diagram above.

* [mittaridatapumppu](https://github.com/City-of-Helsinki/mittaridatapumppu): django app for storing device metadata into postgres db

* [mittaridatapumppu-endpoint](https://github.com/City-of-Helsinki/mittaridatapumppu-endpoint): a fastapi app that receives sensor data in POST requests and produces them to kafka

* [mittaridatapumppu-parser](https://github.com/City-of-Helsinki/mittaridatapumppu-parser): python based microservice that consumer streaming data and converts them from raw format to something else as defined in corresponding parser modules obtained from device metadata

* [mittaridatapumppu-persister](https://github.com/City-of-Helsinki/mittaridatapumppu-persister/):

# Build and Test

Go To [test set up](tests/README.md)

```

pip install pre-commit

pre-commit install

pip-compile

pip-compile --extra dev --output-file requirements-dev.txt

pip-sync

pre-commit run --all-files

```