{"id":13801922,"url":"https://github.com/tve/mqboard","last_synced_at":"2025-10-25T19:30:48.969Z","repository":{"id":40444014,"uuid":"249892150","full_name":"tve/mqboard","owner":"tve","description":"Micro Framework for MicroPython Boards Managed via MQTT","archived":false,"fork":false,"pushed_at":"2022-05-08T14:47:45.000Z","size":361,"stargazers_count":118,"open_issues_count":20,"forks_count":19,"subscribers_count":17,"default_branch":"master","last_synced_at":"2024-10-01T05:09:51.722Z","etag":null,"topics":["micropython","micropython-mqtt","mqtt","mqtt-libraries"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tve.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-03-25T05:11:09.000Z","updated_at":"2024-08-12T20:22:02.000Z","dependencies_parsed_at":"2022-08-09T20:31:15.406Z","dependency_job_id":null,"html_url":"https://github.com/tve/mqboard","commit_stats":null,"previous_names":["tve/micropython-mqtt"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tve%2Fmqboard","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tve%2Fmqboard/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tve%2Fmqboard/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tve%2Fmqboard/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tve","download_url":"https://codeload.github.com/tve/mqboard/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219865145,"owners_count":16555931,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["micropython","micropython-mqtt","mqtt","mqtt-libraries"],"created_at":"2024-08-04T00:01:30.555Z","updated_at":"2025-10-25T19:30:48.546Z","avatar_url":"https://github.com/tve.png","language":"Python","funding_links":[],"categories":["Libraries"],"sub_categories":["Communications"],"readme":"# MQBoard - MicroPython MQTT Micro-Framework\n\nThis repository contains a micro-framework for using MQTT with asyncio on MicroPython boards,\nprimarily on the ESP32. The `mqtt_async` library can be used stand-alone as a robust\nMQTT client library designed for asyncio. The rest of this repo forms a micro-framework\ndesigned to operate MQTT-connected nodes remotely with great flexibility and robustness.\n\n## Goals and features\n\nEverything in this repo is engineered for systems that:\n- run at some remote location and need to be managed from a distance, i.e., without plugging\n  them into the USB port of some computer, and\n- use a single encrypted connection, specifically, an MQTT connection to a broker.\n\nBy using a single connection only one set of TLS buffers is needed (20KB per connection, ouch!),\nand there are no per-board certificates to maintain. There are also no open ports an attacker\ncould target.\n\nThe above goals make the following features desirable:\n- Everything must be designed to stay up 24x7 and recover from failures.\n- REPL access: being able to access the repl on each board via MQTT in order to run diagnostics,\n  perform upgrades, or otherwise troubleshoot.\n- OTA update: being able to send a MicroPython firmware update via MQTT.\n- Modules: being able to add or update functional modules easily and with minimal impact\n  on existing modules.\n- Remote logging of ideally everything normally printed to the serial console via MQTT.\n- Crash logging to some local persistent store so crashes can be analyzed remotely after the board\n  resets.\n- Watchdog timer and safe mode after successive crashes to ensure the board always\n  comes up and can be accessed via MQTT even if the application misbehaves.\n\nCurrently all these features except for the crash log to local storage are implemented.\n\n### Open issues\n\nThe main open (high-level) issue is low power operation. Right now power is not taken into\nconsideration and, in particular, the start-up time is not optimized to enable periodic wake-up\noperation on batteries.\n\nOTA firmware upgrades and safe mode are currently not integrated, which means that an OTA upgrade\nrequires more care and thought than it should.\n\nThe `mqtt_async` implementation tries to be compatible with Peter Hinch's `mqtt_as` and it's time to\ndepart from that so the functionality in `board/mqtt.py` can be integrated directly and so the\nmanagement of Wifi can be decoupled.\n\n## Contents\n\nThe contents of this repo is:\n- `mqtt_async` contains an MQTT client library that uses asyncio, is optimized for streaming files,\n  and forms the backbone of most other libraries and tools here. It is used by other parts of the\n  repo but can easily be used stand-alone.\n- `mqrepl` contains a library to access the REPL via MQTT, basically to be able to send filesystem\n  and interactive commands via MQTT.\n- `mqboard` contains a python commandline tool to be run on a developer's machine to send commands\n  to `mqrepl`. The directory also contains a watchdog and support for a safe mode.\n- `board` contains `boot.py` and `main.py` that make it easy to add modules to a board.\n  The directory also contain a `logging.py` implementation that functions via MQTT.\n\nAdditional modules of interest can be found in https://github.com/tve/mpy-lib, specifically:\n- `sntp` to synchronize time\n- `sysinfo` to send periodic MQTT messages with memory and other system info\n\nFinally, some of the functionality depends on enhancements to MicroPython that are stuck\nin pull requests on mainline. These can be found integrated into the `tve` branch of\nhttps://github.com/tve/micropython.\n\n## Testing\n\nEverything in this repository is subject to CI testing. Some of the tests are run on Linux\nusing CPython but the majority are actually executed on an ESP32 using gohci.\n\n## Getting started\n\nThe getting-started consists of three steps:\n1. Set-up some prerequisites\n2. Load the board with the safemode software, from there on it can be managed over MQTT\n3. Try out the sample blinky app (blinks the LED on the board at a frequency that can be controlled\n   over MQTT.\n\n### Prerequisites\n\n- MQTT broker, preferably local, preferably supporting TLS (MQTTS), preferably using a public\n  certificate, e.g. from Let's Encrypt.\n- ESP32 pre-loaded with TvE's version of MicroPython supporting the \"new asyncio\", i.e. post-v1.12, \n  as well as a functioning `RTC.memory()`, asyncio with TLS, and a bunch of other bug fixes: \n  https://github.com/tve/micropython/releases/tag/v1.12-tve2.\n- The micropython repository (https://github.com/micropython/micropython) or at least the\n  `pyboard.py` tool from its `tools` directory.\n- The Python click and paho-mqtt packages: `pip install click paho-mqtt`.\n\n### Loading up safemode\n\nLoading up the files that make up safe-mode is initially done over USB. Later on they can be updated\nover MQTT, but we need to bootstrap first.\n\n1. Wipe the esp32's filesystem clean (you don't need this if you just did a flash erase and\n   loaded the firmware):\n```\n    $ /home/src/esp32/micropython/tools/pyboard.py board/rm_rf.py\n    rmdir contents: /\n    rm //boot.py\n    rm //main.py\n```\n2. Copy the config file template to `board/board_config.py`, and modify it to suit your\nenvironment, e.g.:\n```\n    cp board/board_config_tmpl.py board/board_config.py\n    vim board/board_config.py\n```\n   The lines you need to edit are clearly marked.\n\n3. Load the files (adjust the path to the tools dir and the device name):\n```\n    $ PATH=/home/src/esp32/micropython/tools:$PATH PYBOARD_DEVICE=/dev/ttyUSB0 ./load.sh\n    device: /dev/ttyUSB0\n    cp ./board/boot.py :boot.py\n    mkdir :/safemode\n    cp ./board/main.py :/safemode/main.py\n    cp ./board/board.py :/safemode/board.py\n    cp ./board/logging.py :/safemode/logging.py\n    cp ./board/mqtt.py :/safemode/mqtt.py\n    cp ./mqrepl/mqrepl.py :/safemode/mqrepl.py\n    cp ./mqrepl/watchdog.py :/safemode/watchdog.py\n    cp ./mqtt_async/mqtt_async.py :/safemode/mqtt_async.py\n    cp ./board/board_config.py :/safemode/board_config.py\n```\n4. Open a new terminal window/tab/pane you can keep open to watch what the board is doing and\n   connect, ideally with a pgm that supports ANSI color escape sequences (sadly, colors don't show\n   here). Then either press the reset button on the board or toggle DTR (in minicom that's ctrl-t\n   ctrl-d two times, in picocom that's ctrl-a ctrl-p):\n```\n    $ miniterm2.py --filter direct /dev/ttyUSB0 115200\n    --- Miniterm on /dev/ttyUSB0  115200,8,N,1 ---\n    --- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---\n    --- DTR inactive ---\n    --- DTR active ---\n    ets Jun  8 2016 00:22:57\n\n    rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)\n    ...\n    I (579) cpu_start: Starting scheduler on PRO CPU.\n    I (0) cpu_start: Starting scheduler on APP CPU.\n    Traceback (most recent call last):\n      File \"boot.py\", line 51, in \u003cmodule\u003e\n    ImportError: no module named 'logging'\n    Switching to SAFE MODE\n    W 1394     main:\n\n    W 1401     main: MicroPython 1.12.0 (v1.12-weather-1-8-g251c8f5a3 on 2020-05-23)\n    W 1408     main: 4MB/OTA NO-BT module with ESP32\n    W 1414     main: esp32/test mqtest starting at (1970, 1, 1, 0, 0, 1, 3, 1, 0)\n\n    W 1423     main: Boot partition: ota_0\n    W 1431     main: SAFE MODE boot (normal mode failed)\n    W 1460     main: Reset cause: PWRON_RESET\n    I 1578     main: MEM free=101424 contig=94752\n    ...\n    I (9198) network: CONNECTED\n    I (9234) wifi: AP's beacon interval = 102400 us, DTIM period = 1\n    I (10053) tcpip_adapter: sta ip: 192.168.0.106, mask: 255.255.255.0, gw: 192.168.0.1\n    I (10055) network: GOT_IP\n    I 7080 mqtt_async: Connecting to ('192.168.0.14', 4883) id=esp32/test-mqtest clean=1\n    I 9244 mqtt_async: Connecting to ('192.168.0.14', 4883) id=esp32/test-mqtest clean=0\n    I 11166     mqtt: Initial MQTT connection (-\u003e3)\n    I 11175     main: Logging to esp32/test/log\n    I 11182     main: Log buf: 1350/10240 bytes\n    I 11214     mqtt: MQTT connected (-\u003e0)\n    I 11474     main: Log buf: 754/1024 bytes\n    I 11569   mqrepl: Subscribed to esp32/test/mqb/cmd/#\n    I 11607 watchdog: esp32/test/mqb/cmd/eval/0F00D/\n    I 11664   mqrepl: Dispatch eval, msglen=15 seq=0 last=True id=0F00D dup=0\n    ...\n```\n   What you see is are the initial boot messages from ESP-IDF followed by a python exception,\n   which is from `boot.py` trying to load the logger from the application. But we've only loaded\n   safe-mode files so far, hence the exception. As a result, `boot.py` switches to safe-mode and\n   proceeds. Then main prints some hello-world info and starts loading the modules comprising\n   the safe-mode application. These start wifi, connect via MQTT, and at the end you see the\n   watchdog sending a round-trip message to the MQTT Repl to feed the WDT timer. Your board is\n   now up!\n\n5. Try something:\n```\n    $ ./mqboard/mqboard --prefix esp32/test eval '45+876'\n    921\n    $ ./mqboard/mqboard --prefix esp32/blinky eval 'import sys; print(sys.implementation)'\n    (name='micropython', version=(1, 12, 0), mpy=10757)\n```\n   The prefix corresponds to the part before the \"/mqb\" in the log message `I 11569   mqrepl:\n   Subscribed to esp32/test/mqb/cmd/#` (it's the `mqtt_prefix` variable in `board_config.py`).\n\nYou can now proceed to the blinky demo app.\n\n\nFor help, please post on https://forum.micropython.org \n\nFor license info see the LICENSE file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftve%2Fmqboard","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftve%2Fmqboard","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftve%2Fmqboard/lists"}