{"id":15416857,"url":"https://github.com/ussserrr/raspberrypi-uart-logger","last_synced_at":"2025-04-26T11:32:05.813Z","repository":{"id":113442617,"uuid":"146029604","full_name":"ussserrr/raspberrypi-uart-logger","owner":"ussserrr","description":"Autonomous Python app that logging UART messages to USB drive","archived":false,"fork":false,"pushed_at":"2019-05-02T19:18:07.000Z","size":133,"stargazers_count":3,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-10-25T20:07:57.371Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ussserrr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-08-24T19:01:39.000Z","updated_at":"2023-05-26T15:25:24.000Z","dependencies_parsed_at":null,"dependency_job_id":"60badca3-a770-40ba-ad15-96b40f07858b","html_url":"https://github.com/ussserrr/raspberrypi-uart-logger","commit_stats":{"total_commits":16,"total_committers":1,"mean_commits":16.0,"dds":0.0,"last_synced_commit":"dde656d1172ff56e9e02d616be0cf0050ff70bc8"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ussserrr%2Fraspberrypi-uart-logger","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ussserrr%2Fraspberrypi-uart-logger/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ussserrr%2Fraspberrypi-uart-logger/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ussserrr%2Fraspberrypi-uart-logger/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ussserrr","download_url":"https://codeload.github.com/ussserrr/raspberrypi-uart-logger/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224032179,"owners_count":17244373,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-01T17:14:02.320Z","updated_at":"2024-11-11T01:15:50.638Z","avatar_url":"https://github.com/ussserrr.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Raspberry Pi serial UART logger\r\n\r\n## Overview\r\nThe application takes log data (with the special formatting, see \"Log data format\") from the serial port, parses it and writes to the USB flash drive. By default, the program uses the built-in UART port of the Raspberry Pi as the input but you can swap it to another device (external UART-USB converter for example (CH340, CP2102, etc.)) for sure. It was designed to be fully autonomous in terms of the external control of any kind and to deliver following functionality. The device as a whole sits as a plugin beside some primary device and plays the role of a \"black box\" recorder that logs all useful information from this main device. From time to time some *support man* takes out the flash drive and swaps it with another one (new empty drive). During any mode, Black Box is staying powered and worked, can diagnose itself and reboot if some error or drive replacement has occurred. Keep these reboots in mind if you plan to run another stuff in parallel.\r\n\r\nThe current version had been developed and tested only for Raspberry Pi 1 and 3 with official Raspbian system (more stability with Pi 3).\r\n\r\n## Dependencies and requirements\r\n - Raspbian OS\r\n - Python3 and its standard library\r\n - `pyserial`\r\n - `termcolor` – for colorful printing of some useful debugging information to `stdout`. Can be safely removed from the app entirely\r\n - `RPi.GPIO` – used in Raspberry Pi systems to control the indication LED\r\n - Root permissions / administrative privileges\r\n\r\n## Raspberry Pi UART\r\nThe Logger uses primary, full-functional `/dev/ttyAMA0` UART module so make sure that RX line GPIO15 (pin 10) is free. Further setup will be done by `manage.py` script during installation. Basically, it detaches console and Bluetooth from UART module via `raspi-config` utility and patches `/boot/config.txt` file with necessary parameters (reboot is required). You can preliminarily test your UART connection using any serial monitor program.\r\n\r\nSolid electrical contact for both GND and RX is an important thing because otherwise there are real risk of bad data transmissions (some sort of garbage non-UTF8 symbols, etc.).\r\n\r\n## Usage\r\n  0. `$ sudo apt update \u0026\u0026 sudo apt upgrade \u0026\u0026 sudo apt install git`\r\n  1. Clone the repo (you will also be needed in Internet connection during installation to satisfy dependencies)\r\n  2. Edit settings in `manage.py`, `raspberrypi-uart-logger/miscs.py` according to your system:\r\n  \r\n    - UART parameters\r\n    - log string format\r\n    - timeouts (some default values are relatively small for debug purposes)\r\n    - drive and log file names\r\n    - working (installation) directory\r\n    - optional LED, that indicates Logger ON/OFF status - if it doesn't light then logger app is shutdown'ed. So you can judge about the status of the system by briefly taking the look on the LED.\r\n  3. Run\r\n     ```bash\r\n     $ cd raspberrypi-uart-logger/\r\n     $ sudo python3 manage.py install\r\n     ```\r\n  4. You doesn't really have to prepare your USB drives because the app is able to remount/format improper drives (so do not store any important documents on drives intended for usage)\r\n  5. Reboot the system. It will apply new preferences and also start the logger.\r\n\r\nAs it is was designed as a fully autonomous service, some intermediate layer is used to start up and turn off the program. **systemd** is used to run the application at every system start. Once systemd did his work, Bash script `startup_script.sh` will be going on. His job is basically to run the Python with the main `.py`-file and to detect incidents of some kind of unpredicted behavior. If the Python is crashed, the script will trigger the system reboot, so it works as an additional safety wrapper around the main app.\r\n\r\nAlternative, you can test the app in \"interactive\" mode first. Disable the daemon (`sudo systemctl disable logger.service`) and place `/bin/bash /opt/raspberrypi-uart-logger/startup_script.sh` in the end of your `~/.bashrc`. So now, every time you log in into shell (during startup or manually) you will see entire logger output including service and log messages.\r\n\r\n## Log data format\r\nThere are 5 levels of warning messages available. You can, of course, specify any format of the log string but default considerations are:\r\n - **D** – debug\r\n - **I** – info\r\n - **W** – warning\r\n - **E** – error\r\n - **C** – critical (error)\r\n\r\n`[PREFIX LETTER] [MESSAGE] \\r`\r\n\r\nThese prefix letters you should put at the start of your message and separate with a space from an actual payload. Output string will contain a type of the message, time \u0026 date and then actual payload:\r\n```plain\r\nDEBUG    [2018-08-24 03:50:00,844] Debug message from UART received\r\nINFO     [2018-08-24 03:50:00,844] It's OK!\r\nWARNING  [2018-08-24 03:50:00,845] Something happened\r\nERROR    [2018-08-24 03:50:00,845] You really need to repair this\r\nCRITICAL [2018-08-24 03:50:00,845] PAN!C\r\n```\r\n\r\nThere are also 2 service messages:\r\n - `is_present`: send this every `serial.Serial.timeout` seconds to notify the logger that your UART device is alive if there are no other messages to deliver. If it is not present for some time (see `no_ping_counter`) the logger will reboot itself. You, of course, can send any data over UART to reset the timeout but such message would not be recognized and be written to the USB drive with the `WARNING` prefix\r\n - `end`: this message terminates the logger program in a normal way (LED is turning off)\r\n\r\nAlways end every message with the CR `\\r` symbol to notify the system about it.\r\n\r\n## USB drive replacement\r\nAssume the system is working in normal mode and some man unplug the flash drive. During the closest logging event the app will detect that no drive is present (that why the custom `logging.FileHandler` class is used for) and will go in the search mode to wait for a new drive. After successful detection the Raspberry will reboot itself and the program starts over. Such approach is a little bit clumsy but in the same time simpler. You can implement a non-reboot mode as well to improve the logger. Also, some sort of periodical checks can be performed to detect disconnections earlier.\r\n\r\nPlease avoid situations when more than one drive is plugged to the Raspberry simultaneously as it leads to an undefined behavior.\r\n\r\nRaspbian by default is set to auto-mount some drives and create `DRIVE_NAMEn` mountpoints (`n` is a number) in case their names are equal. Also, different drives in system are represented by `/dev/sdX1` devices (`X` is a letter). Based on all of the above, we should be prepared for any of these occasions in any combinations even if they will not happened ever. So some code blocks may have additional safety wrappers and checks.\r\n\r\n## Example UART device usage\r\nFind STM32-F0 example (in C) of how to use this logger in your embedded app in `client-usage-example` folder. Sample library is also available.\r\n\r\n## Project structure\r\nMain files:\r\n - `/raspberrypi-uart-logger` - core app, will be copied in your system\r\n - `/manage.py` - installation/deinstallation utility. Keep it to manage your set-up in the future\r\n - `/logger.service` - description of systemd service, is used for autostart/start/stop functionality\r\n - `/client-usage-example` - sample C library and usage example. Runs on STM32 and continuously sends log messages via UART (115200, 8N1)\r\n\r\n## Notes\r\nDue to the specific purpose of the app, `logging` module is used as the main feature and not as an accessory one. So for actual indication of any debug information to `stdout` `print()`s and `cprint()`s statements are used. So in production final version, you can remove them from the app entirely to optimize performance. Of course, another instance of `logging.Logger` class with easy activation/deactivation method can be applied for this task but this wasn't implemented yet.\r\n\r\nThe program also creates and manages `workdir/reboots_cnt_filename` file that stores a number of reboots.\r\n\r\nYou can find the rough app scheme in `uml` folder (made with draw.io).\r\n\r\nSee TODOs for more information about current weaknesses and what would be great to do.\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fussserrr%2Fraspberrypi-uart-logger","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fussserrr%2Fraspberrypi-uart-logger","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fussserrr%2Fraspberrypi-uart-logger/lists"}