{"id":13801756,"url":"https://github.com/peterhinch/micropython-iot","last_synced_at":"2025-07-23T08:33:58.307Z","repository":{"id":54249066,"uuid":"159850382","full_name":"peterhinch/micropython-iot","owner":"peterhinch","description":"An approach to designing IOT applications using ESP8266, ESP32 or Pyboard D endpoints","archived":false,"fork":false,"pushed_at":"2020-09-20T15:17:55.000Z","size":1997,"stargazers_count":94,"open_issues_count":1,"forks_count":16,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-30T15:33:24.611Z","etag":null,"topics":["esp32","esp8266","iot","pyboard"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/peterhinch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-11-30T16:31:34.000Z","updated_at":"2025-01-25T22:58:52.000Z","dependencies_parsed_at":"2022-08-13T10:00:43.967Z","dependency_job_id":null,"html_url":"https://github.com/peterhinch/micropython-iot","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/peterhinch/micropython-iot","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peterhinch%2Fmicropython-iot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peterhinch%2Fmicropython-iot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peterhinch%2Fmicropython-iot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peterhinch%2Fmicropython-iot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/peterhinch","download_url":"https://codeload.github.com/peterhinch/micropython-iot/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peterhinch%2Fmicropython-iot/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266645150,"owners_count":23961658,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-23T02:00:09.312Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["esp32","esp8266","iot","pyboard"],"created_at":"2024-08-04T00:01:26.944Z","updated_at":"2025-07-23T08:33:58.268Z","avatar_url":"https://github.com/peterhinch.png","language":"Python","readme":"# Introduction\n\nThis library provides a resilient full duplex communication link between a WiFi\nconnected board and a server on the wired LAN. The board may be an ESP8266,\nESP32 or other target including the Pyboard D. The design is such that the code\ncan run for indefinite periods. Temporary WiFi or server outages are tolerated\nwithout message loss.\n\nThe API is simple and consistent between client and server applications,\ncomprising `write` and `readline` methods. The `ujson` library enables various\nPython objects to be exchanged. Guaranteed message delivery is available.\n\nThis project is a collaboration between Peter Hinch and Kevin Köck.\n\nAs of July 2020 it has been updated to use (and require) `uasyncio` V3. See\n[section 3.1.1](./README.md#311-existing-users) for details of consequent API\nchanges.\n\n# 0. MicroPython IOT application design\n\nIOT (Internet of Things) systems commonly comprise a set of endpoints on a WiFi\nnetwork. Internet access is provided by an access point (AP) linked to a\nrouter. Endpoints run an internet protocol such as MQTT or HTTP and normally\nrun continuously. They may be located in places which are hard to access:\nreliability is therefore paramount. Security is also a factor for endpoints\nexposed to the internet.\n\nUnder MicroPython the available hardware for endpoints is limited. Testing has\nbeen done on the ESP8266, ESP32 and the Pyboard D.\n\nThe ESP8266 remains as a readily available inexpensive device which, with care,\nis capable of long term reliable operation. It does suffer from limited\nresources, in particular RAM. Achieving resilient operation in the face of WiFi\nor server outages is not straightforward: see\n[this document](https://github.com/peterhinch/micropython-samples/tree/master/resilient).\nThe approach advocated here simplifies writing robust ESP8266 IOT applications\nby providing a communications channel with inherent resilience.\n\nThe usual arrangement for MicroPython internet access is as below.\n![Image](./images/block_diagram_orig.png)\n\nRunning internet protocols on ESP8266 nodes has the following drawbacks:\n 1. It can be difficult to ensure resilience in the face of outages of WiFi and\n of the remote endpoint.\n 2. Running TLS on the ESP8266 is demanding in terms of resources: establishing\n a connection can take 30s.\n 3. There are potential security issues for internet-facing nodes.\n 4. The security issue creates a requirement periodically to install patches to\n firmware or to libraries. This raises the issue of physical access.\n 5. Internet applications can be demanding of RAM.\n\nThis document proposes an approach where multiple remote nodes communicate with\na local server. This runs CPython or MicroPython code and supports the internet\nprotocol required by the application. The server and the remote nodes\ncommunicate using a simple protocol based on the exchange of lines of text. The\nserver can run on a Linux box such as a Raspberry Pi; this can run 24/7 at\nminimal running cost.\n\n![Image](./images/block_diagram.png)\n\nBenefits are:\n 1. Security is handled on a device with an OS. Updates are easily accomplished.\n 2. The text-based protocol minimises the attack surface presented by nodes.\n 3. The protocol is resilient in the face of outages of WiFi and of the server:\n barring errors in the application design, crash-free 24/7 operation is a\n realistic prospect.\n 4. The amount of code running on the remote is smaller than that required to\n run a resilient internet protocol such as [this MQTT version](https://github.com/peterhinch/micropython-mqtt.git).\n 5. The server side application runs on a relatively powerful machine. Even\n minimal hardware such as a Raspberry Pi has the horsepower easily to support\n TLS and to maintain concurrent links to multiple client nodes. Use of\n threading is feasible.\n 6. The option to use CPython on the server side enables access to the full\n suite of Python libraries including internet modules.\n\nThe principal drawback is that in addition to application code on the ESP8266\nnode, application code is also required on the PC to provide the \"glue\" linking\nthe internet protocol with each of the client nodes. In many applications this\ncode may be minimal.\n\nThere are use-cases where conectivity is entirely local, for example logging\nlocally acquired data or using some nodes to control and monitor others. In\nsuch cases no internet protocol is required and the server side application\nmerely passes data between nodes and/or logs data to disk.\n\nThis architecture can be extended to non-networked clients such as the Pyboard\nV1.x. This is described and diagrammed [here](./README.md#9-extension-to-the-pyboard).\n\n# 1. Contents\n\nThis repo comprises code for resilent full-duplex connections between a server\napplication and multiple clients. Each connection is like a simplified socket,\nbut one which persists through outages and offers guaranteed message delivery.\n\n 0. [MicroPython IOT application design](./README.md#0-microPython-iot-application-design)  \n 1. [Contents](./README.md#1-contents)  \n 2. [Design](./README.md#2-design)  \n  2.1 [Protocol](./README.md#21-protocol)  \n 3. [Files and packages](./README.md#3-files-and-packages)  \n  3.1 [Installation](./README.md#31-installation)  \n  \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;3.1.1 [Existing users](./README.md#311-existing-users)  \n  \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;3.1.2 [Firmware and dependency](./README.md#312-firmware-and-dependency)  \n  \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;3.1.3 [Preconditions for demos](./README.md#313-preconditions-for-demos)  \n  3.2 [Usage](./README.md#32-usage)  \n  \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;3.2.1 [The main demo](./README.md#321-the-main-demo)  \n  \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;3.2.2 [The remote control demo](./README.md#322-the-remote-control-demo)  \n  \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;3.2.3 [Quality of Service demo](./README.md#323-quality-of-service-demo)  \n  \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;3.2.4 [The fast qos demo](./README.md#324-the-fast-qos-demo)  \n  \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;3.2.5 [Troubleshooting the demos](./README.md#325-troubleshooting-the-demos)  \n 4. [Client side applications](./README.md#4-client-side-applications)  \n  4.1 [The Client class](./README.md#41-the-client-class)  \n   4.1.1 [Initial Behaviour](./README.md#411-initial-behaviour)  \n   4.1.2 [Watchdog Timer](./README.md#412-watchdog-timer)  \n 5. [Server side applications](./README.md#5-server-side-applications)  \n  5.1 [The server module](./README.md#51-the-server-module)  \n 6. [Ensuring resilience](./README.md#6-ensuring-resilience) Guidelines for application design.   \n 7. [Quality of service](./README.md#7-quality-of-service) Guaranteeing message delivery.  \n  7.1 [The qos argument](./README.md#71-the-qos-argument)  \n  7.2 [The wait argument](./README.md#71-the-wait-argument) Concurrent writes of qos messages.  \n 8. [Performance](./README.md#8-performance)  \n  8.1 [Latency and throughput](./README.md#81-latency-and-throughput)  \n  8.2 [Client RAM utilisation](./README.md#82-client-ram-utilisation)  \n  8.3 [Platform reliability](./README.md#83-platform-reliability)  \n 9. [Extension to the Pyboard](./README.md#9-extension-to-the-pyboard)  \n 10. [How it works](./README.md#10-how-it-works)  \n  10.1 [Interface and client module](./README.md#101-interface-and-client-module)  \n  10.2 [Server module](./README.md#102-server-module)  \n\n# 2. Design\n\nThe code is asynchronous and based on `asyncio`. Client applications on the\nremote import `client.py` which provides the interface to the link. The server\nside application uses `server.py`.\n\nMessages are required to be complete lines of text. They typically comprise an\narbitrary Python object encoded using JSON. The newline character ('\\n') is not\nallowed within a message but is optional as the final character.\n\nGuaranteed message delivery is supported. This is described in\n[section 7](./README.md#7-quality-of-service). Performance limitations are\ndiscussed in [section 8](./README.md#8-performance).\n\n## 2.1 Protocol\n\nClient and server applications use `readline` and `write` methods to\ncommunicate: in the case of an outage of WiFi or the connected endpoint, the\nmethod will pause until the outage ends. While the system is tolerant of\nruntime server and WiFi outages, this does not apply on initialisation. The\nserver must accessible before clients are started.\n\nThe link status is determined by periodic exchanges of keepalive messages. This\nis transparent to the application. If a keepalive is not received within a user\nspecified timeout an outage is declared. On the client the WiFi is disconnected\nand a reconnection procedure is initiated. On the server the connection is\nclosed and it awaits a new connection.\n\nEach client has a unique ID which is an arbitrary string. In the demo programs\nthis is stored in `local.py`. The ID enables the server application to\ndetermine which physical client is associated with an incoming connection.\n\n###### [Contents](./README.md#1-contents)\n\n# 3. Files and packages\n\nThis repo has been updated for `uasyncio` V3. This is incorporated in daily\nbuilds of firmware and will be available in release builds later than V1.12.\nServer code may be run under CPython V3.8 or above. It may be run under\nMicroPython (Unix build), but at the time of writing this requires\n[this fix](https://github.com/micropython/micropython/issues/6109#issuecomment-639376529)\nto incorporate `uasyncio`.\n\nDirectory `iot`:\n 1. `client.py` / `client.mpy` Client module. The ESP8266 has insufficient RAM\n to compile `client.py` so the precompiled `client.mpy` should be used. See\n note below.\n 2. `server.py` Server module. (runs under CPython 3.5+ or MicroPython 1.10+).\nDirectory `iot/primitives`:\n 1. `__init__.py` Functions common to `Client` and `Server`.\n 2. `switch.py` Debounced switch interface. Used by `remote` demo.\nOptional directories containing Python packages:\n 1. `iot/examples` A simple example. Up to four clients communicate with a\n single server instance.\n 2. `iot/remote` Demo uses the library to enable one client to control another.\n This may need adapting for your hardware.\n 3. `iot/qos` Demonstrates and tests the qos (quality of service) feature, see\n [Quality of service](./README.md#7-quality-of-service).\n 4. `iot/pb1` Contians packages enabling a Pyboard V1.x to communicate with the\n server via an ESP8266 connected by I2C. See [documentation](./pb_link/README.md).\n\nNOTE: The file `client.mpy` works with daily builds at the time of writing. The\nbytecode format changes occasionally. If an application throws a bytecode error\nit is necessary to cross-compile `client.py` with the associated version of\n`mpy-cross`. Or raise an issue and I will post an update.\n\n## 3.1 Installation\n\nThis section describes the installation of the library and the demos. The\nESP8266 has limited RAM: there are specific recommendations for installation on\nthat platform.\n\n### 3.1.1 Existing users\n\nIt is recommended to remove the old version and re-install as below.\n\nThere have been API changes to accommodate the new `uasyncio` version: the\nevent loop argument is no longer required or accepted in `Client` and `Server`\nconstructors. The directory structure has changed, requiring minor changes to\n`import` statements.\n\n### 3.1.2 Firmware and dependency\n\nOn ESP8266, RAM can be saved by building firmware from source, freezing\n`client.py` as bytecode. If this is not done, it is necessary to\n[cross compile](https://github.com/micropython/micropython/tree/master/mpy-cross)\n`client.py`. The file `client.mpy` is provided for those unable to do this. If\nfreezing, create an `iot` directory in your modules directory and copy\n`iot/client.py` and the directory `iot/primitives` and contents there.\n\nPre-requisites: firmware must be a current daily build or a release build after\nV1.12. If upgrading, particularly on an ESP8266, it is wise to erase flash\nprior to installtion. In particular this will ensure the use of littlefs with\nits associated RAM saving.\n\nThis repository is a python package, consequently on the client the directory\nstructure must be retained. The following installs all demos on the target.\n\nOn your PC move to a directory of your choice and clone the repository there:\n```\ngit clone https://github.com/peterhinch/micropython-iot\n```\nInstallation consists of copying the `iot` directory and contents to an `iot` \ndirectory on the boot device. On ESP8266 or ESP32 the boot device is`/pyboard`.\nOn the Pyboard D it will be `/flash` or `/sd` depending on whether an SD card\nis fitted.\n\nCopying may be done using any tool but I recommend\n[rshell](https://github.com/dhylands/rshell). If this is used start in the\ndirectory on your PC containing the clone, start `rshell` and issue (adapting\nthe boot device for your platform):\n```\nrsync iot /pyboard/iot\n```\nOn ESP8266, unless frozen, it is necessary to delete `client.py` to force the\nuse of `client.mpy`:\n```\nrm /pyboard/iot/client.py\n```\n\n### 3.1.3 Preconditions for demos\n\nThe demo programs store client configuration data in a file `local.py`. Each\ndemo has its own `local.py` located in the directory of the demo code. This\ncontains the following constants which should be edited to match local\nconditions. Remove the `use_my_local` hack designed for my WiFi privacy.:\n\n```python\nMY_ID = '1'  # Client-unique string.\nSERVER = '192.168.0.10'  # Server IP address.\nSSID = 'use_my_local'  # Insert your WiFi credentials\nPW = 'PASSWORD'\nPORT = 8123\nTIMEOUT = 2000\n# The following may be deleted\nif SSID == 'use_my_local':\n    from iot.examples.my_local import *\n```\n\nThe ESP8266 can store WiFi credentials in flash memory. If desired, ESP8266\nclients can be initialised to connect to the local network prior to running\nthe demos. In this case the SSID and PW variables may optionally be empty\nstrings (`SSID = ''`).\n\nNote that the server-side examples below specify `python3` in the run command.\nIn every case `micropython` may be substituted to run under the Unix build of\nMicroPython.\n\n## 3.2 Usage\n\n### 3.2.1 The main demo\n\nThis illustrates up to four clients communicating with the server. The demo\nexpects the clients to have ID's in the range 1 to 4: if using multiple clients\nedit each one's `local.py` accordingly.\n\nOn the server navigate to the parent directory of `iot` and run:\n```\npython3 -m iot.examples.s_app_cp\n```\nor\n```\nmicropython -m iot.examples.s_app_cp\n```\nOn each client run:\n```\nimport iot.examples.c_app\n```\n\n### 3.2.2 The remote control demo\n\nThis shows one ESP8266 controlling another. The transmitter should have a\npushbutton between GPIO 0 and gnd, both should have an LED on GPIO 2.\n\nOn the server navigate to the parent directory of `iot` and run:\n```\npython3 -m iot.remote.s_comms_cp\n```\nor\n```\nmicropython -m iot.remote.s_comms_cp\n```\n\nOn the esp8266 run (on transmitter and receiver respectively):\n\n```\nimport iot.remote.c_comms_tx\nimport iot.remote.c_comms_rx\n```\n\n### 3.2.3 Quality of Service demo\n\nThis test program verifies that each message (in each direction) is received\nexactly once. On the server navigate to the parent directory of `iot` and run:\n```\npython3 -m iot.qos.s_qos_cp\n```\nor\n```\nmicropython -m iot.qos.s_qos_cp\n```\nOn the client, after editing `/pyboard/qos/local.py`, run:\n```\nimport iot.qos.c_qos\n```\n\n### 3.2.4 The fast qos demo\n\nThis tests the option of concurrent `qos` writes. This is an advanced feature\ndiscussed in [section 7.1](./README.md#71-the-wait-argument). To run the demo,\non the server navigate to the parent directory of `iot` and run:\n```\npython3 -m iot.qos.s_qos_fast\n```\nor\n```\nmicropython -m iot.qos.s_qos_fast\n```\nOn the client, after editing `/pyboard/qos/local.py`, run:\n```\nimport iot.qos.c_qos_fast\n```\n\n### 3.2.5 Troubleshooting the demos\n\nIf `local.py` specifies an SSID, on startup the demo programs will pause\nindefinitely if unable to connect to the WiFi. If `SSID` is an empty string the\nassumption is an ESP8266 with stored credentials; if this fails to connect an\n`OSError` will be thrown. An `OSError` will also be thrown if initial\nconnectivity with the server cannot be established.\n\n###### [Contents](./README.md#1-contents)\n\n# 4. Client side applications\n\nA client-side application instantiates a `Client` and launches a coroutine\nwhich awaits it. After the pause the `Client` has connected to the server and\ncommunication can begin. This is done using `Client.write` and\n`Client.readline` methods.\n\nEvery client ha a unique ID (`MY_ID`) typically stored in `local.py`. The ID\ncomprises a string subject to the same constraint as messages:\n\nMessages comprise a single line of text; if the line is not terminated with a\nnewline ('\\n') the client library will append it. Newlines are only allowed as\nthe last character. Blank lines will be ignored.\n\nA basic client-side application has this form:\n```python\nimport uasyncio as asyncio\nimport ujson\nfrom iot import client\nimport local  # or however you configure your project\n\n\nclass App:\n    def __init__(self, verbose):\n        self.cl = client.Client(local.MY_ID, local.SERVER,\n                                local.PORT, local.SSID, local.PW,\n                                local.TIMEOUT, conn_cb=self.state, \n                                verbose=verbose)\n        asyncio.create_task(self.start())\n\n    async def start(self):\n        await self.cl  # Wait until client has connected to server\n        asyncio.create_task(self.reader())\n        await self.writer()  # Wait forever\n\n    def state(self, state):  # Callback for change in connection status\n        print(\"Connection state:\", state)\n\n    async def reader(self):\n        while True:\n            line = await self.cl.readline()  # Wait until data received\n            data = ujson.loads(line)\n            print('Got', data, 'from server app')\n\n    async def writer(self):\n        data = [0, 0]\n        count = 0\n        while True:\n            data[0] = count\n            count += 1\n            print('Sent', data, 'to server app\\n')\n            await self.cl.write(ujson.dumps(data))\n            await asyncio.sleep(5)\n\n    def close(self):\n        self.cl.close()\n\napp = None\nasync def main():\n    global app  # For closure by finally clause\n    app = App(True)\n    await app.start()  # Wait forever\n\ntry:\n    asyncio.run(main())\nfinally:\n    app.close()  # Ensure proper shutdown e.g. on ctrl-C\n    asyncio.new_event_loop()\n```\nIf an outage of server or WiFi occurs, the `write` and `readline` methods will\npause until connectivity has been restored. The server side API is similar.\n\n###### [Contents](./README.md#1-contents)\n\n## 4.1 The Client class\n\nThe constructor has a substantial number of configuration options but in many\ncases defaults may be accepted for all but the first five.\n\nConstructor args:\n 1. `my_id` The client id.\n 2. `server` The server IP-Adress to connect to.\n 3. `port=8123` The port the server listens on.\n 4. `ssid=''` WiFi SSID. May be blank for ESP82666 with credentials in flash.\n 5. `pw=''` WiFi password. \n 6. `timeout=2000` Connection timeout in ms. If a connection is unresponsive\n for longer than this period an outage is assumed.\n 7. `conn_cb=None` Callback or coroutine that is called whenever the connection\n changes.\n 8. `conn_cb_args=None` Arguments that will be passed to the *connected_cb*\n callback. The callback will get these args preceeded by a `bool` indicating\n the new connection state.\n 9. `verbose=False` Provides optional debug output.\n 10. `led=None` If a `Pin` instance is passed it will be toggled each time a\n keepalive message is received. Can provide a heartbeat LED if connectivity is\n present. On Pyboard D a `Pin` or `LED` instance may be passed.\n 10. `wdog=False` If `True` a watchdog timer is created with a timeout of 20s.\n This will reboot the board if it crashes - the assumption is that the\n application will be restarted via `main.py`.\n\nMethods (asynchronous):\n 1. `readline` No args. Pauses until data received. Returns a line.\n 2. `write` Args: `buf`, `qos=True`, `wait=True`. `buf` holds a line of text.  \n If `qos` is set, the system guarantees delivery. If it is clear messages may\n (rarely) be lost in the event of an outage.  \n The `wait` arg determines the behaviour when multiple concurrent writes are\n launched with `qos` set. See [Quality of service](./README.md#7-quality-of-service).\n\nThe following asynchronous methods are described in Initial Behaviour below. In\nmost cases they can be ignored.\n 3. `bad_wifi`\n 4. `bad_server`\n\nMethods (synchronous):\n 1. `status` Returns `True` if connectivity is present. May also be read using\n function call syntax (via `__call__`).\n 2. `close` Closes the socket. Should be called in the event of an exception\n such as a `ctrl-c` interrupt. Also cancels the WDT in the case of a software\n WDT.\n\nBound variable:\n 1. `connects` The number of times the `Client` instance has connected to WiFi.\n This is maintained for information only and provides some feedback on the\n reliability of the WiFi radio link.\n\nThe `Client` class is awaitable. If\n```python\nawait client_instance\n```\nis issued, the coroutine will pause until connectivity is (re)established.\n\nApplications which always `await` the `write` method do not need to check or\nawait the client status: `write` will pause until it can complete. If `write`\nis launched using `create_task` it is essential to check status otherwise\nduring an outage unlimited numbers of coroutines will be created.\n\nThe client buffers up to 20 incoming messages. To avoid excessive queue growth\napplications should have a single coroutine which spends most of its time\nawaiting incoming data.\n\n###### [Contents](./README.md#1-contents)\n\n### 4.1.1 Initial Behaviour\n\nWhen an application instantiates a `Client` it attemps to connect to WiFi and\nthen to the server. Initial connection is handled by the following `Client`\nasynchronous bound methods (which may be modified by subclassing):\n\n 1. `bad_wifi` No args.\n 2. `bad_server` No args. Awaited if server refuses an initial connection.\n\nNote that, once a server link has been initially established, these methods\nwill not be called: reconnection after outages of WiFi or server are automatic.\n\nThe `bad_wifi` coro attempts to connect using the WiFi credentials passed to\nthe constructor. This will pause until a connection has been achieved. The\n`bad_server` coro raises an `OSError`. Behaviour of either of these may be\nmodified by subclassing.\n\nPlatforms other than ESP8266 launch `bad_wifi` unconditionally on startup. In\nthe case of an ESP8266 which has WiFi credentials stored in flash it will first\nattempt to connect using that data, only launching `bad_wifi` if this fails in\na timeout period. This is to minimise flash wear.\n\n### 4.1.2 Watchdog Timer\n\nThis option provides a last-ditch protection mechanism to keep a client running\nin the event of a crash. The ESP8266 can (rarely) crash, usually as a result of\nexternal electrical disturbance. The WDT detects that the `Client` code is no\nlonger running and issues a hard reset. Note that this implies a loss of\nprogram state. It also assumes that `main.py` contains a line of code which\nwill restart the application.\n\nDebugging code with a WDT can be difficult because bugs or software interrupts\nwill trigger unexpected resets. It is recommended not to enable this option\nuntil the code is stable.\n\nOn the ESP8266 the WDT uses a sofware timer: it can be cancelled which\nsimplifies debugging. See `examples/c_app.py` for the use of the `close` method\nin a `finally` clause.\n\nThe WDT on the Pyboard D is a hardware implementation: it cannot be cancelled.\nIt may be necessary to use safe boot to bypass `main.py` to access the code.\n\n###### [Contents](./README.md#1-contents)\n\n# 5. Server side applications\n\nA typical example has an `App` class with one instance per physical client\ndevice. This enables instances to share data via class variables. Each instance\nlaunches a coroutine which acquires a `Connection` instance for its individual\nclient (specified by its client_id). This process will pause until the client\nhas connected with the server. Communication is then done using the `readline`\nand `write` methods of the `Connection` instance.\n\nMessages comprise a single line of text; if the line is not terminated with a\nnewline (`\\n`) the server library will append it. Newlines are only allowed as\nthe last character. Blank lines will be ignored.\n\nA basic server-side application has this form:\n```python\nimport asyncio\nimport json\nfrom iot import server\nimport local  # or however you want to configure your project\n\nclass App:\n    def __init__(self, client_id):\n        self.client_id = client_id  # This instance talks to this client\n        self.conn = None  # Will be Connection instance\n        self.data = [0, 0, 0]  # Exchange a 3-list with remote\n        asyncio.create_task(self.start())\n\n    async def start(self):\n        # await connection from the specific EP8266 client\n        self.conn = await server.client_conn(self.client_id)\n        asyncio.create_task(self.reader())\n        asyncio.create_task(self.writer())\n\n    async def reader(self):\n        while True:\n            # Next line will pause for client to send a message. In event of an\n            # outage it will pause for its duration.\n            line = await self.conn.readline()\n            self.data = json.loads(line)\n            print('Got', self.data, 'from remote', self.client_id)\n\n    async def writer(self):\n        count = 0\n        while True:\n            self.data[0] = count\n            count += 1\n            print('Sent', self.data, 'to remote', self.client_id, '\\n')\n            await self.conn.write(json.dumps(self.data))  # May pause in event of outage\n            await asyncio.sleep(5)\n\nasync def main():\n    clients = {1, 2, 3, 4}\n    apps = [App(n) for n in clients]  # Accept 4 clients with ID's 1-4\n    await server.run(clients, True, local.PORT, local.TIMEOUT)  # Verbose\n\ndef run():\n    try:\n        asyncio.run(main())\n    except KeyboardInterrupt:  # Delete this if you want a traceback\n        print('Interrupted')\n    finally:\n        server.Connection.close_all()\n        asyncio.new_event_loop()\n\nif __name__ == \"__main__\":\n    run()\n```\n\n## 5.1 The server module\n\nServer-side applications should create and run a `server.run` task. This runs\nforever and takes the following args:\n 1. `expected` A set of expected client ID strings.\n 2. `verbose=False` If `True` output diagnostic messages.\n 3. `port=8123` TCP/IP port for connection. Must match clients.\n 4. `timeout=2000` Timeout for outage detection in ms. Must match the timeout\n of all `Client` instances.\n\nThe `expected` arg causes the server to produce a warning message if an\nunexpected client connects, or if multiple clients have the same ID (this will\ncause tears before bedtime).\n\nThe module is based on the `Connection` class. A `Connection` instance provides\na communication channel to a specific client. The `Connection` instance for a\ngiven client is a singleton and is acquired by issuing\n```python\nconn = await server.client_conn(client_id)\n```\nThis will pause until connectivity has been established. It can be issued at\nany time: if the `Connection` has already been instantiated, that instance will\nbe returned. The `Connection` constructor should not be called by applications.\n\n#### The `Connection` instance\n\nMethods (asynchronous):\n 1. `readline` No args. Pauses until data received. Returns a line.\n 2. `write` Args: `buf`, `qos=True`, `wait=True`. `buf` holds a line of text.  \n If `qos` is set, the system guarantees delivery. If it is clear messages may\n (rarely) be lost in the event of an outage.__\n The `wait` arg determines the behaviour when multiple concurrent writes are\n launched with `qos` set. See [Quality of service](./README.md#7-quality-of-service).\n\nMethods (synchronous):\n 1. `status` Returns `True` if connectivity is present. The connection state\n may also be retrieved using function call syntax (via `.__call__`).\n 2. `__getitem__` Enables the `Connection` of another client to be retrieved\n using list element access syntax. Will throw a `KeyError` if the client is\n unknown (has never connected).\n\nClass Method (synchronous):\n 1. `close_all` No args. Closes all sockets: call on exception (e.g. ctrl-c).\n\nBound variable:\n 1. `nconns` Maintains a count of (re)connections for information or monitoring\n of outages.\n\nThe `Connection` class is awaitable. If\n```python\nawait connection_instance\n```\nis issued, the coroutine will pause until connectivity is (re)established.\n\nApplications which always `await` the `write` method do not need to check or\nawait the server status: `write` will pause until it can complete. If `write`\nis launched using `create_task` it is essential to check status otherwise\nduring an outage unlimited numbers of coroutines will be created.\n\nThe server buffers incoming messages but it is good practice to have a coro\nwhich spends most of its time waiting for incoming data.\n\nServer module coroutines:\n\n 1. `run` Args: `expected` `verbose=False` `port=8123` `timeout=2000`\n This is the main coro and starts the system. \n `expected` is a set containing the ID's of all clients.  \n `verbose` causes debug messages to be printed.  \n `port` is the port to listen to.  \n `timeout` is the number of ms that can pass without a keepalive until the \n  connection is considered dead.\n 2. `client_conn` Arg: `client_id`. Pauses until the sepcified client has\n connected. Returns the `Connection` instance for that client.\n 3. `wait_all` Args: `client_id=None` `peers=None`. See below.\n\nThe `wait_all` coroutine is intended for applications where clients communicate\nwith each other. Typical user code cannot proceed until a given set of clients\nhave established initial connectivity.\n\n`wait_all`, where a `client_id` is specified, behaves as `client_conn` except\nthat it pauses until further clients have also connected. If a `client_id` is\npassed it will returns that client's `Connection` instance. If `None` is passed\nthe  assumption is that the current client is already connected and the coro\nreturns `None`.\n\nThe `peers` argument defines which clients it must await: it must either be\n`None` or a set of client ID's. If a set of `client_id` values is passed, it\npauses until all clients in the set have connected. If `None` is passed, it\npauses until all clients specified in `run`'s `expected` set have connected.\n\nIt is perhaps worth noting that the user application can impose a timeout on\nthis by means of `asyncio.wait_for`.\n\n###### [Contents](./README.md#1-contents)\n\n# 6. Ensuring resilience\n\nThere are two principal ways of provoking `LmacRxBlk` errors and crashes.\n 1. Failing to close sockets when connectivity is lost.\n 2. Feeding excessive amounts of data to a socket after connectivity is lost:\n this causes an overflow to an internal ESP8266 buffer.\n\nThese modules aim to address these issues transparently to application code,\nhowever it is possible to write applications which violate 2.\n\nThere is a global `TIMEOUT` value defined in `local.py` which should be the\nsame for the server and all clients. Each end of the link sends a `keepalive`\n(KA) packet (an empty line) at a rate guaranteed to ensure that at least one KA\nwill be received in every `TIMEOUT` period. If it is not, connectivity is\npresumed lost and both ends of the interface adopt a recovery procedure.\n\nIf an application always `await`s a write with `qos==True` there is no risk of\nFeeding excess data to a socket: this is because the coroutine does not return\nuntil the remote endpoint has acknowledged reception.\n\nOn the other hand if multiple messages are sent within a timeout period with\n`qos==False` there is a risk of buffer overflow in the event of an outage.\n\n###### [Contents](./README.md#1-contents)\n\n# 7. Quality of service\n\nIn the presence of a stable WiFi link TCP/IP should ensure that packets sent\nare received intact. In the course of extensive testing with the ESP8266 we\nfound that (very rarely) packets were lost. It is not known whether this\nbehavior is specific to the ESP8266. Another mechanism for message loss is the\ncase where a message is sent in the interval between an outage occurring and it\nbeing detected. This is likely to occur on all platforms.\n\nThe client and server modules avoid message loss by the use of acknowledge\npackets: if a message is not acknowledged within a timeout period it is\nretransmitted. This implies duplication where the acknowledge packet is lost.\nReceive message de-duplication is employed to provide a guarantee that the\nmessage will be delivered exactly once. While delivery is guaranteed,\ntimeliness is not. Messages are inevitably delayed for the duration of a WiFi\nor server outage where the `write` coroutine will pause for the duration.\n\nGuaranteed delivery involves a tradeoff against throughput and latency. This is\nmanaged by optional arguments to `.write`, namely `qos=True` and `wait=True`.\n\n## 7.1 The qos argument\n\nMessage integrity is determined by the `qos` argument. If `False` message\ndelivery is not guaranteed. A use-case for disabling `qos` is in applications\nsuch as remote control. If the user presses a button and nothing happens they\nwould simply repeat the action. Such messages are always sent immediately: the\napplication should limit the rate at which they can be sent, particularly on\nESP8266 clients, to avoid risk of buffer overflow.\n\nWith `qos` set, the message will be delivered exactly once.\n\nWhere successive `qos` messages are sent there may be a latency issue. By\ndefault the transmission of a `qos` message will be delayed until reception\nof its predecessor's acknowledge. Consequently the `write` coroutine will\npause, introducing latency. This serves two purposes. Firstly it ensures that\nmessages are received in the order in which they were sent (see below).\n\nSecondly consider the case where an outage has occurred but has not yet been\ndetected. The first message is written, but no acknowledge is received.\nSubsequent messages are delayed, precluding the risk of ESP8266 buffer\noverflows. The interface resumes operation after the outage has cleared.\n\n## 7.2 The wait argument\n\nThis default can be changed with the `wait` argument to `write`. If `False` a\n`qos` message will be sent immediately, even if acknowledge packets from\nprevious messages are pending. Applications should be designed to limit the\nnumber of such `qos` messages sent in quick succession: on ESP8266 clients\nbuffer overflows can occur.\n\nIn testing in 2019 the ESP32 was not resilient under these circumstances; this\nappears to have been fixed in current firmware builds. Nevertheless setting\n`wait=False` potentially risks resilience. If used, applications should be\ntested to verify quality of service in the presence of WiFi outages.\n\nIf messages are sent with `wait=False` there is a chance that they may not be\nreceived in the order in which they were sent. As described above, in the event\nof `qos` message loss, retransmission occurs after a timeout period has\nelapsed. During that timeout period the application may have successfully sent\nanother non-waiting `qos` message resulting in out of order reception.\n\nThe demo programs `qos/c_qos_fast.py` (client) and `qos/s_qos_fast.py` issue\nfour `write` operations with `wait=False` in quick succession. This number is\nprobably near the maximum on an ESP8266. Note the need explicitly to check for\nconnectivity before issuing the `write`: this is to avoid spawning large\nnumbers of coroutines during an outage.\n\nIn summary specifying `wait=False` should be considered an \"advanced\" option\nrequiring testing to prove that resilence is maintained.\n\n###### [Contents](./README.md#1-contents)\n\n# 8. Performance\n\n## 8.1 Latency and throughput\n\nThe interface is intended to provide low latency: if a switch on one node\ncontrols a pin on another, a reasonably quick response can be expected. The\nlink is not designed for high throughput because of the buffer overflow issue\ndiscussed in [section 6](./README.md#6-ensuring-resilence). This is essentially\na limitation of the ESP8266 device: more agressive use of the `wait` arg may be\npossible on platforms such as the Pyboard D.\n\nIn practice latency on the order of 100-200ms is normal; if an outage occurs\nlatency will inevitably persist for the duration.\n\n**TIMEOUT**\n\nThis defaults to 2s. On `Client` it is a constructor argument, on the server\nit is an arg to `server.run`. Its value should be common to all clients and\nthe sever. It determines the time taken to detect an outage and the frequency\nof `keepalive` packets. This time was chosen on the basis of measured latency\nperiods on WiFi networks. It may be increased at the expense of slower outage\ndetection. Reducing it may result in spurious timeouts with unnecessary WiFi\nreconnections.\n\n## 8.2 Client RAM utilisation\n\nOn ESP8266 with a current (June 2020) daily build the demo reports over 20KB\nfree. Free RAM of 25.9KB was achieved with compiled firmware with frozen \nbytecode as per [Installation](./README.md#31-installation).\n\n## 8.3 Platform reliability\n\nIn extensive testing the Pyboard D performed impeccably: no failures of any\nkind were observed in weeks of testing through over 1000 outages.\n\nESP32 was prone to occasional spontaneous reboots. It would typically run for a\nfew days through multiple WiFi outages before rebooting.\n\nESP8266 still occasionally crashes and it is recommended to use the watchdog\nfeature to reboot it should this occur.\n\nIt would take a very long time to achieve more than a subjective impression of\nthe effectof usage options on failure rate. The precautionary principle\nsuggests maximising free ram with frozen bytecode on ESP8266 and avoiding\nconcurrent `qos==1` writes on ESPx platforms.\n\n###### [Contents](./README.md#1-contents)\n\n# 9. Extension to the Pyboard\n\nThis extends the resilient link to MicroPython targets lacking a network\ninterface; for example the Pyboard V1.x. Connectivity is provided by an ESP8266\nrunning a fixed firmware build: this needs no user code.\n\nThe interface between the Pyboard and the ESP8266 uses I2C and is based on the\n[existing I2C module](https://github.com/peterhinch/micropython-async/tree/master/v3/as_drivers/i2c).\n\n![Image](./images/block_diagram_pyboard.png)\n\nResilient behaviour includes automatic recovery from WiFi and server outages;\nalso from ESP8266 crashes.\n\nSee [documentation](./docs/PB_LINK.md).\n\n# 10. How it works\n\n## 10.1 Interface and client module\n\nThe `client` module was designed on the expectation that client applications\nwill usually be simple: acquiring data from sensors and periodically sending it\nto the server and/or receiving data from the server and using it to control\ndevices. Developers of such applications probably don't need to be concerned\nwith the operation of the module.\n\nThere are ways in which applications can interfere with the interface's\noperation either by blocking or by attempting to operate at excessive data\nrates. Such designs can produce an erroneous appearance of poor WiFi\nconnectivity.\n\nOutages are detected by a timeout of the receive tasks at either end. Each peer\nsends periodic `keepalive` messages consisting of a single newline character,\nand each peer has a continuously running read task. If no message is received\nin the timeout period (2s by default) an outage is declared.\n\nFrom the client's perspective an outage may be of the WiFi or the server. In\npractice WiFi outages are more common: server outages on a LAN are typically\ncaused by the developer testing new code. The client assumes a WiFi outage. It\ndisconnects from the network for long enough to ensure that the server detects\nthe outage. It then attempts repeatedly to reconnect. When it does so, it\nchecks that the connection is stable for a period (it might be near the limit\nof WiFi range).\n\nIf this condition is met it attempts to reconnect to the server. If this\nsucceeds the client runs. Its status becomes `True` when it first receives data\nfrom the server.\n\nA client or server side application which blocks or hogs processor time can\nprevent the timely transmission of `keepalive` messages. This will cause the\nserver to declare an outage: the consequence is a sequence of disconnect\nand reconnect events even in the presence of a strong WiFi signal.\n\n## 10.2 Server module\n\nServer-side applications communicate via a `Connection` instance. This is\nunique to a client. It is instantiated when a specified client first connects\nand exists forever. During an outage its status becomes `False` for the\nduration. The `Connection` instance is retrieved as follows, with the\n`client_conn` method pausing until initial connectivity has been achieved:\n```python\nimport server\n# Class details omitted\n    self.conn = await server.client_conn(self.client_id)\n```\nEach client must have a unique ID. When the server detects an incoming\nconnection on the port it reads the client ID from the client. If a\n`Connection` instance exists for that ID its status is updated, otherwise a\n`Connection` is instantiated.\n\nThe `Connection` has a continuously running coroutine `._read` which reads data\nfrom the client. If an outage occurs it calls the `._close` method which closes\nthe socket, setting the bound variable `._sock` to `None`. This corresponds to\na `False` status. The `._read` method pauses until a new connection occurs. The\naim here is to read data from ESP8266 clients as soon as possible to minimise\nrisk of buffer overflows.\n\nThe `Connection` detects an outage by means of a timeout in the `._read`\nmethod: if no data or `keepalive` is received in that period an outage is\ndeclared, the socket is closed, and the `Connection` status becomes `False`.\n\nThe `Connection` has a `._keepalive` method. This regularly sends `keepalive`\nmessages to the client. Application code which blocks the scheduler can cause\nthis not to be scheduled in a timely fashion with the result that the client\ndeclares an outage and disconnects. The consequence is a sequence of disconnect\nand reconnect events even in the presence of a strong WiFi signal.\n","funding_links":[],"categories":["Libraries"],"sub_categories":["Communications"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpeterhinch%2Fmicropython-iot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpeterhinch%2Fmicropython-iot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpeterhinch%2Fmicropython-iot/lists"}