{"id":22743939,"url":"https://github.com/j-hoplin/iot-data-streaming-using-message-queue","last_synced_at":"2026-04-10T00:35:56.537Z","repository":{"id":171697658,"uuid":"647269843","full_name":"J-Hoplin/IoT-data-streaming-using-Message-Queue","owner":"J-Hoplin","description":"🖥️ 4학년 1학기 모바일 시스템 융합 학기 개인 프로젝트 (1st)","archived":false,"fork":false,"pushed_at":"2023-06-06T11:52:51.000Z","size":6224,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-05T15:48:21.497Z","etag":null,"topics":["amqp","docker","docker-compose","express","grafana","influxdb","nestjs","nodejs","rabbitmq","tkinter","typescript"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/J-Hoplin.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-05-30T12:24:08.000Z","updated_at":"2024-10-29T14:20:49.000Z","dependencies_parsed_at":"2024-01-15T21:58:54.369Z","dependency_job_id":"6bcb6a6d-e5d9-4334-8e2d-e572c417f55e","html_url":"https://github.com/J-Hoplin/IoT-data-streaming-using-Message-Queue","commit_stats":null,"previous_names":["j-hoplin1/mobile-system-convergence","j-hoplin/iot-data-streaming-using-message-queue"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/J-Hoplin%2FIoT-data-streaming-using-Message-Queue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/J-Hoplin%2FIoT-data-streaming-using-Message-Queue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/J-Hoplin%2FIoT-data-streaming-using-Message-Queue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/J-Hoplin%2FIoT-data-streaming-using-Message-Queue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/J-Hoplin","download_url":"https://codeload.github.com/J-Hoplin/IoT-data-streaming-using-Message-Queue/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246277350,"owners_count":20751548,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amqp","docker","docker-compose","express","grafana","influxdb","nestjs","nodejs","rabbitmq","tkinter","typescript"],"created_at":"2024-12-11T01:36:52.856Z","updated_at":"2025-12-30T20:30:16.916Z","avatar_url":"https://github.com/J-Hoplin.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Mobile System Convergence final project\n\n---\n\n**This document do not provide a package installation method such as tkinter or etc. Please install it on your own. Make an issue if something is wrong or not working on process**\n\n---\n\n**[Demonstration Video](https://www.youtube.com/watch?v=CnLUcU8CaCY)**\n\n---\n\n## System Block Diagram\n\n![img](./img/blockdiagram.png)\n\n## Message Queue Design\n\n![img](./img/message-queue-design-dht11.png)\n\n![img](./img/message-queue-design-metrics.png)\n\n**Require 3 computers to make experiment including Raspberry Pi4**\n\n1. Raspberry Pi 4B\n\n- 4GB of RAM\n\n- Role of `Publisher`\n\n- Need to install\n  - Node.js v18+\n  - Docker(Optional)\n\n2. Synology NAS DS 720+\n\n- Intel Celeron, with 10GB of RAM\n\n- Role of `Rabbit MQ Server`\n\n- Need to install\n  - Docker\n\n3. Mac mini\n\n- Apple Silicon, 16GB of RAM\n- Role of `Consumer`\n- Need to install\n  - Python\n  - Node.js v18+\n  - Docker\n  - Docker-Compose\n\n## Skills used in the design\n\n### Programming Language\n\n- Node.js\n  - Pakcage manager : yarn\n- TypeScript\n- Python3\n  - Runtime : Anaconda\n\n### Framework\n\n- TypeScript + Express.js\n- Nest.js\n\n### Infrastructure\n\n- Platform : Raspberry Pi4\n- Docker\n- Docker-Compose\n- Message Queue : Rabbit MQ\n- Influx DB\n- Grafana\n- Telegraf\n\n## Order of execution\n\n### 1. Rabbit MQ Setting\n\nExecute `Rabbit MQ` docker in Rabbit MQ Server. I recently made Rabbit MQ tutorial contents with Node.js. So if you need those information, please follow this [link].(https://github.com/J-hoplin1/RabbitMQ-R-n-D)\n\n```\nsudo docker run -d --name rabbitmq -p 5672:5672 -p 15672:15672 -p 1883:1883 -e RABBITMQ_DEFAULT_USER=user -e RABBITMQ_DEFAULT_PASS=password rabbitmq:3-management\n```\n\nAfter execute docker container of rabbit mq you need to edit `.env` file of both client-side and server-side\n\n- [Client's envfile : client-producer/env/.env](./client-producer/env)\n\n  ```\n  # Connection information\n  MQ_NAME=(Rabbit MQ username)\n  MQ_PASSWORD=(Rabbit MQ user password)\n  MQ_HOST=(Rabbit MQ Server Host)\n  MQ_PORT=(Rabbit MQ Port of server)\n\n  ...\n  ```\n\n- [Server's envfile : server-dotenv/.env](./server-dotenv/)\n\n  ```\n  # Connection information\n  MQ_NAME=(Rabbit MQ username)\n  MQ_PASSWORD=(Rabbit MQ user password)\n  MQ_HOST=(Rabbit MQ Server Host)\n  MQ_PORT=(Rabbit MQ Port of server)\n\n  ...\n  ```\n\n### 2. Server setting\n\nFirst you need to start `InfluxDB - Grafana` docker compose located in [here](./server-influx-grafana-compose/). If you want to use telegraf also, please follow this [document](./docs/telegraf-settings.md).\n\n```\ncd ./server-influx-grafana-compose\n\ndocker compose up -d\n```\n\nPlease refer to this [page](./docs/influx-grafana-setting.md) to make setting of `InfluxDB - Grafana`\n\nAfter setting `InfluxDB - Grafana` start `Consumer` and `API Server` and `Monitoring GUI` in order.\n\n- [Server Consumer](./server-consumer/)\n\n  ```\n  cd server-consumer/\n\n  yarn install\n\n  yarn tsc\n\n  node dist/\n  ```\n\n- [API Server](./server-api-server/)\n\n  ```\n  cd server-api-server/\n\n  yarn install\n\n  yarn run start\n  ```\n\n  Connect to `localhost:3000/docs` in server machine browser for API endpoints documet\n\n  ```\n  localhost:3000/docs\n  ```\n\n- [Monitoring GUI](./monitoring-gui/)\n\n  ```\n  cd monitoring-gui/\n\n  python3 app.py\n  ```\n\n  It's normal that no data appears\n  ![img](./img/gui-plain.png)\n\nIf it's normally executed, Monitoring GUI will request to API server per second\n\n![img](./img/api-console.png)\n\n### 3. Client setting\n\n**Client should be Raspberry Pi4.** Make an DHT 11 sensor connection. Follow picture under below\n\n![img](https://hackster.imgix.net/uploads/attachments/625477/setup-dht11-raspi_VVgjXEkmlT.png?auto=compress\u0026w=1600\u0026h=1200\u0026fit=min\u0026fm=jpg)\n\nand install Adafruit Python DHT. Follow the instructions on this [github](https://github.com/adafruit/Adafruit_Python_DHT)\n\nCheck sensor connected to gpio by executing [`client-DHT11/app.py`](./client-DHT11/app.py)\n\n```\npython3 app.py\n```\n\nIf it's normal temperature, humidity sensor should be estimate. Don't care about `Connection error occured. Please check API Server` message.\n\nNext we need to execute REST API based producer.\n\n```\ncd client-producer/\n\nyarn install\n\nyarn tsc\n\nnode dist/www.js\n```\n\nConnect to `localhost:3000/docs` in client machine browser for API endpoints documet\n\n```\nlocalhost:3000/docs\n```\n\nIf producer start normally it will publish message to rabbit mq\n![img](./img/publisher.png)\n\nand rabbit mq will receive message and save to queue by routing key.\n\n![img](./img/rbmq.png)\n\nServer consumer will consume message\n\n![img](./img/consuming.png)\n\nand save it as time-series data to influx db\n\n![img](./img/influx-res.png)\n\nfinally you can view datas in realtime via monitoring GUI and grafana dashboard\n\n![img](./img/gui-res.png)\n![img](./img/grafana-example.png)\n\n### 4. Reason why I made producer as API type\n\nReason why I made producer as API type, is to reduce user dependence on programming languages, packages, and frameworks and module coupling when developing applications. Instead, using a protocol called http allows users to focus more on business logic by simply sending data to endpoints, and the source code they write does not need to be combined with source code that relies on message queue modules such as Rabbit mq.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fj-hoplin%2Fiot-data-streaming-using-message-queue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fj-hoplin%2Fiot-data-streaming-using-message-queue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fj-hoplin%2Fiot-data-streaming-using-message-queue/lists"}