{"id":20912520,"url":"https://github.com/cpq/embedded-network-programming-guide","last_synced_at":"2025-03-12T22:26:36.217Z","repository":{"id":214785408,"uuid":"737353221","full_name":"cpq/embedded-network-programming-guide","owner":"cpq","description":"A complete guide for network programming on microcontroller devices","archived":false,"fork":false,"pushed_at":"2024-08-24T20:26:36.000Z","size":153,"stargazers_count":193,"open_issues_count":1,"forks_count":19,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-01-19T15:46:31.763Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cpq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-30T18:12:46.000Z","updated_at":"2025-01-14T06:58:19.000Z","dependencies_parsed_at":"2024-01-02T10:54:03.484Z","dependency_job_id":"7b075510-0385-45ba-a6e4-c04f392ed776","html_url":"https://github.com/cpq/embedded-network-programming-guide","commit_stats":null,"previous_names":["cpq/embedded-network-programming-guide"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpq%2Fembedded-network-programming-guide","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpq%2Fembedded-network-programming-guide/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpq%2Fembedded-network-programming-guide/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpq%2Fembedded-network-programming-guide/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cpq","download_url":"https://codeload.github.com/cpq/embedded-network-programming-guide/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243303135,"owners_count":20269630,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-18T14:28:14.463Z","updated_at":"2025-03-12T22:26:36.175Z","avatar_url":"https://github.com/cpq.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Embedded network programming guide\n\n[![License: MIT](https://img.shields.io/badge/license-MIT-blue)](https://opensource.org/licenses/MIT)\n\nThis guide is written for embedded developers who work on connected products.\nThat includes microcontroller based systems that operate in bare metal or\nRTOS mode, as well as microprocessor based systems which run embedded Linux.\n\n## Prerequisites\n\nAll fundamental concepts are explained in details, so the guide should be\nunderstood by a reader with no prior networking knowledge.\n\nA reader is expected to be familiar with microcontroller C programming - for\nthat matter, I recommend reading my [bare metal programming\nguide](https://github.com/cpq/bare-metal-programming-guide).  I will be using\nEthernet-enabled Nucleo-H743ZI board throughout this guide.  Examples for other\narchitectures are summarized in a table below - this list will expand with\ntime. Regardless, for the best experience I recommend Nucleo-H743ZI to get the\nmost from this guide: [buy it on\nMouser](https://www.mouser.ie/ProductDetail/STMicroelectronics/NUCLEO-H743ZI2?qs=lYGu3FyN48cfUB5JhJTnlw%3D%3D).\n\n## Network stack explained\n\n### Network frame structure\n\nWhen any two devices communicate, they exchange discrete pieces of data called\nframes. Frames can be sent over the wire (like Ethernet) or over the air (like\nWiFi or Cellular).  Frames differ in size, and typically range from couple of\ndozen bytes to a 1.5Kb.  Each frame consists of a sequence of protocol headers\nfollowed by user data:\n\n![Network Frame](media/frame.svg)\n\nThe purpose of the headers is as follows:\n\n**MAC (Media Access Control) header** is only 3 fields: destination MAC\naddress, source MAC addresses, and an upper level protocol. MAC addresses are\n6-byte unique addresses of the network cards, e.g. `42:ef:15:c8:29:a1`.\nProtocol is usually 0x800, which means that the next header is IP. MAC header\nhandles addressing in the local network (LAN).\n\n\n**IP (Internet Protocol) header** has many fields, but the most important are:\ndestination IP address, source IP address, and upper level protocol. IP\naddresses are 4-bytes, e.g. `209.85.202.102`, and they identify a machine\non the Internet, so their purpose is similar to phone numbers. The upper level\nprotocol is usually 6 (TCP) or 17 (UDP). IP header handles global addressing.\n\n**TCP or UDP header** has many fields, but the most important are destination\nport and source ports. On one device, there can be many network applications,\nfor example, many open tabs in a browser. Port number identifies\nan application. \n\n**Application protocol** depends on the target application. For example, there\nare servers on the Internet that can tell an accurate current time. If you want\nto send data to those servers, the application protocol must SNTP (Simple\nNetwork Time Protocol). If you want to talk to a web server, the protocol must\nbe HTTP. There are other protocols, like DNS, MQTT, etc, each having their own\nheaders, followed by the application data.\n\nInstall [Wireshark](https://www.wireshark.org/) tool to observe network\nframes. It helps to identify issues quickly, and looks like this:\n\n![Wireshark](https://www.wireshark.org/docs/wsug_html_chunked/images/ws-main.png)\n\nThe structure of a frame described above, makes it possible to accurately\ndeliver a frame to the correct device and application over the Internet. When a\nframe arrives to a device, a software that handles that frame (a network\nstack), is organised in four layers.\n\n### Network stack architecture\n\n![Network Stack](media/stack.svg)\n\nLayer 1: **Driver layer**, only reads and writes frames from/to network hardware  \nLayer 2: **TCP/IP stack**, parses protocol headers and handles IP and TCP/UDP  \nLayer 3: **Network Library**, parses application protocols like DNS, MQTT, HTTP  \nLayer 4: **Application** - Web dashboard, smart sensor, etc\n\n\n### DNS request example\n\nLet's provide an example. In order to show your this guide on the Github, your\nbrowser first needs to find out the IP address of the Github's machine. For\nthat, it should make a DNS (Domain Name System) request to one of the DNS\nservers. Here's how your browser and the network stack work for that case:\n\n![DNS request](media/dns.svg)\n\n**1.** Your browser (an application) asks from the lower layer (library), \"what\nIP address `github.com` has?\". The lower layer (layer 3, a library layer - in\nthis case, it is a C library) provides an API function `gethostbyname()`\nthat returns an IP address for a given host name. So everything said below,\nessentially describes how `gethostbyname()` works.\n\n**2.** The library layer gets the name `github.com` and creates a properly\nformatted, binary DNS request: `struct dns_request`. Then it calls an API\nfunction `sendto()` provided by the TCP/IP stack layer (layer 2), to send that\nrequest over UDP to the DNS server. The IP of the DNS server is known to the library\nfrom the workstation settings. The UDP port is also known - port 53, a standard\nport for DNS.\n\n**3.** The TCP/IP stack's `sendto()` function receives a chunk of data to send.\nit contains DNS request, but `sendto()` does not know that and does not care\nabout that. All it knows is that this is the piece of user data that needs to\nbe delievered over UDP to a certain IP address (IP address of a DNS server) on\nport 53. Hence TCP/IP stack prepends UDP, IP, and MAC headers\nto the user data to form a frame. Then it calls API function `send_frame()`\nprovided by the driver layer, layer 1.\n\n**4.** A driver's `send_frame()` function transmits a frame over the wire or\nover the air, the frame travels to the destination DNS server. A chain of\nInternet routers pass that frame from one to another, until a frame finally hits\nDNS server's network card.\n\n**5.** A network card on the DNS server gets a frame and generates a hardware\ninterrupt, invoking interrupt handler. It is part of a driver - layer 1. It\ncalls a function `recv_frame()` that reads a frame from the card, and passes\nit up by calling `ethernet_input()` function provided by the TCP/IP stack\n\n**6.** TCP/IP stack parses the frame, and finds out that it is for the UDP port\n53, which is a DNS port number. TCP/IP stack finds an application that listens\non UDP port 53, which is a DNS server application, and wakes up its `recv()`\ncall. So, DNS server application that is blocked on a `recv()` call, receives\na chunk of data - which is a DNS request. A library routine parses that request\nby extracting a host name, and passes that parsed DNS request to the application.\n\n\n**7.** A DNS server application receives DNS request: \"someone wants an\nIP address for `github.com`\". Then the application layer looks at its\nconfiguration, figures out \"Oh, it's me who is responsible for the github.com\ndomain, and this is the IP address I should respond with\". The application\nextracts an IP address from the configuration, and calls a library function\n\"get this IP, wrap into a DNS response, and send back\". And the response\ntravels all the way back in the reverse order.\n\n### BSD socket API\n\nThe communication between layers are done via a function calls. So, each\nlayer has its own API, which upper and lower levels can call. They are not\nstandardized, so each implementation provides their own set of functions.\nHowever, on OSes like Windows/Mac/Linux/UNIX, a driver and TCP/IP layers are\nimplemented in kernel, and TCP/IP layer provides a standard API to the\nuserland which is called a \"BSD socket API\":\n\n![BSD socket API](media/bsd.svg)\n\nThis is done becase kernel code does not implement application level protocols\nlike MQTT, HTTP, etc, - so it let's user application to implement them in\nuserland. So, a library layer and an application layer reside in userland.\nSome library level routines are provided in C standard library, like\nDNS resolution function `gethostbyname()`, but that DNS library functions\nare probably the only ones that are provided by OS. For other protocols,\nmany libraries exist that provide HTTP, MQTT, Websocket, SSH, API. Some\napplications don't use any external libraries: they use BSD socket API\ndirectly and implement library layer manually. Usually that is done when\napplication decides to use some custom protocol.\n\nEmbedded systems very often use TCP/IP stacks that provide the same BSD API as\nmainstream OSes do. For example, lwIP (LightWeight IP) TCP/IP stack, Keil's MDK\nTCP/IP stack, Zephyr RTOS TCP/IP stack - all provide BSD socket API. Thus let's\nreview the most important BSD API stack functions:\n\n- `socket(protocol)` - creates a connection descriptor and assigns an integer ID for it, a \"socket\"\n- `bind(sock, addr)` - assigns a local IP:PORT for a listening socket\n- `accept(sock, addr)` - creates a new socket, assigns local IP:PORT\n   and remote IP:PORT (incoming)\n- `connect(sock, addr)` - assigns local IP:PORT and remote IP:PORT for\n   a socket (outgoing)\n- `send(sock, buf, len)` - sends data\n- `recv(sock, buf, len)` - receives data\n- `close(sock)` - closes a socket\n\nSome implementations do not implement BSD socket API, and there are perfectly\ngood reasons for that. Examples for such implementation is lwIP raw API,\nand Mongoose Library.\n\n### TCP echo server implemented with socket API\n\nLet me demonstrate the two approaches (using socket and non-socket API) on a simple\nTCP echo server example. TCP echo server is a simple application that\nlistens on a TCP port, receives data from clients that connect to that port,\nand writes (echoes) that data back to the client. That means, this application\ndoes not use any application protocol on top of TCP, thus it does not need\na library layer. Let's see how this application would look like written\nwith a BSD socket API. First, a TCP listener should bind to a port, and\nfor every connected client, spawn a new thread that would handle it. A thread\nfunction that sends/receives data, looks something like this:\n\n```c\nvoid handle_new_connection(int sock) {\n  char buf[20];\n  for (;;) {\n    ssize_t len = recv(sock, buf, sizeof(buf), 0);  // Receive data from remote\n    if (len \u003c= 0) break;                            // Error! Exit the loop\n    send(sock, buf, len);                           // Success. Echo data back\n  }\n  close(sock);  // Close socket, stop thread\n}\n```\n\nNote that `recv()` function blocks until it receives some data from the client.\nThen, `send()` also blocks until is sends requested data back to the client.\nThat means that this code cannot run in a bare metal implementation, because\n`recv()` would block the whole firmware. For this to work, an RTOS is required.\nA TCP/IP stack should run in a separate RTOS task, and both `send()` and\n`recv()` functions are implemented using an RTOS queue API, providing a\nblocking way to pass data from one task to another. Overall, this is how an\nembedded receive path looks like with socket API:\n\n![BSD socket API](media/socket.svg)\n\nThe `send()` part would work in the reverse direction. Note that this approach\nrequires TCP/IP stack implement data buffering for each socket, because\nan application consumes received data not immediately, but after some time,\nwhen RTOS queue delivers data. Note that using non-blocking sockets and\n`select()/poll()` changes things that instead of many application tasks,\nthere is only one application task, but the mechanism stays the same.\n\nTherefore this approach with socket API has\nthe following major characteristics:\n\n1. It uses queues for exchanging data between TCP/IP stack and\n   application tasks, which consumes both RAM and time\n2. TCP/IP stack buffers received and sent data for each socket. Note that\n   the app/library layer may also buffer data - for example, buffering a full\n   HTTP request before it can be processed. So the same data goes through\n   two buffering \"zones\" - TCP/IP stack, and library/app\n\nThat means, socket API implementation takes extra time for data to be processed,\nand takes extra RAM for double-buffering in the TCP/IP stack.\n\n### TCP echo server with non-socket (callback) API\n\nNow let's see how the same approach works without BSD socket API. Several\nimplementations, including lwIP and Mongoose Library, provide callback API to\nthe TCP/IP stack. Here is how TCP echo server would look like written using\nMongoose API:\n\n```c\n// This callback function is called for various network events, MG_EV_*\nvoid event_handler(struct mg_connection *c, int ev, void *ev_data, void *fn_data) {\n  if (ev == MG_EV_READ) {\n    // MG_EV_READ means that new data got buffered in the c-\u003erecv buffer\n    mg_send(c, c-\u003erecv.buf, c-\u003erecv.len);  // Send back the data we received\n    c-\u003erecv.len = 0;                       // Discard received data\n  }\n}\n```\n\nIn this case, all functions are non-blocking, that means that data exchange\nbetween TCP/IP stack and an app can be implemented via direct function calls.\nThis is how receive path looks like:\n\n![Raw callback API](media/raw.svg)\n\nAs you can see, in this case TCP/IP stack provides a callback API which\na library or application layer can use to receive data directly. No need\nto send it over a queue. A library/app layer can buffer data, and that's\nthe only place where buffering takes place. This approach wins for\nmemory usage and performance. A firmware developer should use\na proprietary callback API instead of BSD socket API.\n\nlwIP TCP/IP stack, for example, provides both socket and non-socket (raw) API,\nand raw API is more efficient in terms of RAM and performance. However\ndevelopers rarely use raw API, because it is not trivial to understand and use\ncompared to the socket API. The API of the Mongoose Library shown above\nis designed to be simple and easy to understand. API design can make things\nvery easy or very difficult, so it is important to have a good API.\n\n## Implementing layers 1,2,3 - making ping work\n\n### Development environment and tools\n\nNow let's make our hands dirty and implement a working network stack on\na microcontroller board. I will be using\n[Mongoose Library](https://github.com/cesanta/mongoose) for all examples\nfurther on, for the following reasons:\n\n- Mongoose is very easy to integrate: just by copying two files,\n  [mongoose.c]() and [mongoose.h]()\n- Mongoose has a built-in drivers, TCP/IP stack, HTTP/MQTT/Websocket library,\n  and TLS 1.3 all in one, so it does not need any other software to create\n  a network-enabled application\n- Mongoose provides a simple, polished callback API designed specifically\n  for embedded developers\n\nThe diagram below shows Mongoose architecture. As you can see, Mongoose can\nuse external TCP/IP stack and TLS libraries, as well as built-in ones. In the\nfollowing example, we are going to use only a built-in functionality, so we\nwon't need any other software.\n\n![Mongoose architecture](media/mongoose.svg)\n\nAll source code in this guide is MIT licensed, however Mongoose\nis licensed under a dual GPLv2/commercial license.\nI will be using a Nucleo board from ST Microelectronics, and there are several choices for the\ndevelopment environment:\n- Use Cube IDE provided by ST: [install Cube](https://www.st.com/en/development-tools/stm32cubeide.html)\n- Use Keil from ARM: [install Keil](https://www.keil.com/)\n- Use make + GCC compiler, no IDE: follow [this guide](https://mongoose.ws/documentation/tutorials/tools/)\n\nHere, I am going to use Cube IDE. In the templates, however, both Keil and\nmake examples are provided, too. So, in order to proceed, install Cube IDE\non your workstation, and plug in Nucleo board to your workstation.\n\n### Skeleton firmware\n\nNote: this and the following sections has a Youtube helper video recorded:\nhttps://www.youtube.com/watch?v=lKYM4b8TZts\n\nThe first step would be to create a minimal, skeleton firmware that does\nnothing but logs messages to the serial console. Once we've done that, we'll\nadd networking functionality on top of it. The table below summarises\nperipherals for various boards:\n\n\n| Board            | UART, TX, RX    | Ethernet                              |    LED         |\n| ---------------- | --------------- | ------------------------------------- | -------------- |\n| STM32H747I-DISCO | USART1, A9, A10 | A1, A2, A7, C1, C4, C5, G12, G11, G13 | I12, I13, I14  |\n| STM32H573I-DK    | USART1, A9, A10 | A1, A2, A7, C1, C4, C5, G12, G11, G13 | I8, I9, F1     |\n| Nucleo-H743ZI    | USART3, D8, D9  | A1, A2, A7, C1, C4, C5, B13, G11, G13 | B0, E1, B14    |\n| Nucleo-H723ZG    | USART3, D8, D9  | A1, A2, A7, C1, C4, C5, B13, G11, G13 | B0, E1, B14    |\n| Nucleo-H563ZI    | USART3, D8, D9  | A1, A2, A7, C1, C4, C5, B15, G11, G13 | B0, F4, G4     |\n| Nucleo-F746ZG    | USART3, D8, D9  | A1, A2, A7, C1, C4, C5, B13, G11, G13 | B0, B7, B14    |\n| Nucleo-F756ZG    | USART3, D8, D9  | A1, A2, A7, C1, C4, C5, B13, G11, G13 | B0, B7, B14    |\n| Nucleo-F429ZI    | USART3, D8, D9  | A1, A2, A7, C1, C4, C5, B13, G11, G13 | B0, B7, B14    |\n\n**Step 1.** Start Cube IDE. Choose File / New / STM32 project  \n**Step 2.** In the \"part number\" field, type the microcontroller name,\nfor example \"H743ZI\". That should narrow down\nthe MCU/MPU list selection in the bottom right corner to a single row.\nClick on the row at the bottom right, then click on the Next button  \n**Step 3.** In the project name field, type any name, click Finish.\nAnswer \"yes\" if a pop-up dialog appears  \n**Step 4.** A configuration window appears. Click on Clock configuration tab.\nFind a field with a system clock value. Type the maximum value, hit enter,\nanswer \"yes\" on auto-configuration question, wait until configured  \n**Step 5.** Switch to the Pinout tab, Connectivity, then enable the UART controller\nand pins (see table above), choose \"Asynchronous mode\"  \n**Step 6.** Click on Connectivity / ETH, Choose Mode / RMII, verify that the\nconfigured pins are like in the table above - if not, change pins  \n**Step 7.** Lookup the LED GPIO from the peripherals table, and configure it\nfor output. Click on the corresponding pin, select \"GPIO output\"  \n**Step 8.** Click Ctrl+S to save the configuration. This generates the code\nand opens main.c file  \n**Step 9.** Navigate to the `main()` function and add some logging to the\n`while` loop. Make sure to insert your code between the \"USER CODE\" comments,\nbecause CubeIDE will preserve it during code regeneration:\n```c\n  /* USER CODE BEGIN WHILE */\n  while (1)\n  {\n  printf(\"Tick: %lu\\r\\n\", HAL_GetTick());\n  HAL_Delay(500);\n```\n**Step 10.** Redirect `printf()` to the UART. Note the UART global variable\ngenerated by Cube at the beginning of `main.c` - typically it is\n`UART_HandleTypeDef huart3;`. Copy it, open `syscalls.c`, find function\n`_write()` and modify it the following way :\n```c\n#include \"main.h\"\n\n__attribute__((weak)) int _write(int file, char *ptr, int len) {\n    if (file == 1 || file == 2) {\n      extern UART_HandleTypeDef huart3;\n      HAL_UART_Transmit(\u0026huart3, (unsigned char *) ptr, len, 999);\n    }\n    return len;\n  }\n```\n**Step 11.** Click on \"Run\" button to flash this firmware to the board.  \n**Step 12.** Attach a serial monitor tool (e.g. putty on Windows, or\n`cu -l COMPORT -s 115200` on Mac/Linux) and observe UART logs:\n```\nTick: 90358\nTick: 90860\n...\n```\nOur skeleton firmware is ready!\n\n### Integrate Mongoose\n\nNow it's time to implement a functional TCP/IP stack. We'll use Mongoose\nLibrary for that. To integrate it, we need to copy two files into our source tree.\n\n**Step 1**. Open https://github.com/cesanta/mongoose in your browser, click on \"mongoose.h\". Click on \"Raw\" button, and copy file contents into clipboard.\nIn the CubeIDE, right click on Core/Inc, choose New/File in the menu, type\n\"mongoose.h\", paste the file content and save.  \n**Step 2**. Repeat for \"mongoose.c\". On Github, copy `mongoose.c` contents\nto the clipboard. In the CubeIDE, right click on Core/Src, choose New/File\nin the menu, type \"mongoose.c\", paste the file content and save.  \n**Step 3**. Right click on Core/Inc, choose New/File in the menu, type\n\"mongoose_custom.h\", and paste the following contents:\n```c\n#pragma once\n\n// See https://mongoose.ws/documentation/#build-options\n#define MG_ARCH MG_ARCH_NEWLIB\n\n#define MG_ENABLE_TCPIP 1          // Enables built-in TCP/IP stack\n#define MG_ENABLE_CUSTOM_MILLIS 1  // We must implement mg_millis()\n#define MG_ENABLE_DRIVER_STM32H 1  // On STM32Fxx series, use MG_ENABLE_DRIVER_STM32F\n```\n**Step 4**. Implement Layer 1 (driver), 2 (TCP/IP stack) and 3 (library) in\nour code. Open `main.c`. Add `#include \"mongoose.h\"` at the top:\n```c\n/* USER CODE BEGIN Includes */\n#include \"mongoose.h\"\n/* USER CODE END Includes */\n```\n**Step 5**. Before `main()`, define function `mg_millis()` that returns\nan uptime in milliseconds. It will be used by Mongoose Library for the time\nkeeping:\n```c\n/* USER CODE BEGIN 0 */\nuint64_t mg_millis(void) {\n  return HAL_GetTick();\n}\n/* USER CODE END 0 */\n```\n**Step 6**. Navigate to `main()` function and change the code around `while`\nloop this way:\n```c\n    /* USER CODE BEGIN WHILE */\n    struct mg_mgr mgr;\n    mg_mgr_init(\u0026mgr);\n    mg_log_set(MG_LL_DEBUG);\n\n    // On STM32Fxx, use _stm32f suffix instead of _stm32h\n    struct mg_tcpip_driver_stm32h_data driver_data = {.mdc_cr = 4};\n    struct mg_tcpip_if mif = {.mac = {2, 3, 4, 5, 6, 7},\n                              // Uncomment below for static configuration:\n                              // .ip = mg_htonl(MG_U32(192, 168, 0, 223)),\n                              // .mask = mg_htonl(MG_U32(255, 255, 255, 0)),\n                              // .gw = mg_htonl(MG_U32(192, 168, 0, 1)),\n                              .driver = \u0026mg_tcpip_driver_stm32h,\n                              .driver_data = \u0026driver_data};\n    NVIC_EnableIRQ(ETH_IRQn);\n    mg_tcpip_init(\u0026mgr, \u0026mif);\n\n    while (1) {\n      mg_mgr_poll(\u0026mgr, 0);\n  /* USER CODE END WHILE */\n```\n\n**Step 7**. Connect your board to the Ethernet. Flash firmware. In the serial\nlog, you should see something like this:\n```\nbb8    3 mongoose.c:14914:mg_tcpip_driv Link is 100M full-duplex\nbbd    1 mongoose.c:4676:onstatechange  Link up\nbc2    3 mongoose.c:4776:tx_dhcp_discov DHCP discover sent. Our MAC: 02:03:04:05:06:07\nc0e    3 mongoose.c:4755:tx_dhcp_reques DHCP req sent\nc13    2 mongoose.c:4882:rx_dhcp_client Lease: 86400 sec (86403)\nc19    2 mongoose.c:4671:onstatechange  READY, IP: 192.168.2.76\nc1e    2 mongoose.c:4672:onstatechange         GW: 192.168.2.1\nc24    2 mongoose.c:4673:onstatechange        MAC: 02:03:04:05:06:07\n```\nIf you don't, and see DHCP requests message like this:\n```\n130b0  3 mongoose.c:4776:tx_dhcp_discov DHCP discover sent. Our MAC: 02:03:04:05:06:07\n13498  3 mongoose.c:4776:tx_dhcp_discov DHCP discover sent. Our MAC: 02:03:04:05:06:07\n...\n```\nThe most common cause for this is you have your Ethernet pins wrong. Click\non the `.ioc` file, go to the Ethernet configuration, and double-check the\nEthernet pins against the table above.\n\n**Step 8**. Open terminal/command prompt, and run a `ping` command against\nthe IP address of your board:\n```sh\n$ ping 192.168.2.76\nPING 192.168.2.76 (192.168.2.76): 56 data bytes\n64 bytes from 192.168.2.76: icmp_seq=0 ttl=64 time=9.515 ms\n64 bytes from 192.168.2.76: icmp_seq=1 ttl=64 time=1.012 ms\n```\n\nNow, we have a functional network stack running on our board. Layers 1,2,3\nare implemented. It's time to create an application - a simple web server,\nhence implement layer 4.\n\n## Implementing layer 4 - a simple web server\n\nLet's add a very simple web server that responds \"ok\" to any HTTP request.\n\n**Step 1**. After the `mg_tcpip_init()` call, add this line that creates HTTP listener\nwith `fn` event handler function:\n```c\n  mg_http_listen(\u0026mgr, \"http://0.0.0.0:80\", fn, NULL);\n```\n**Step 2**. Before the `mg_millis()` function, add the `fn` event handler function:\n```c\nstatic void fn(struct mg_connection *c, int ev, void *ev_data) {\n  if (ev == MG_EV_HTTP_MSG) {\n    struct mg_http_message *hm = ev_data;  // Parsed HTTP request\n    mg_http_reply(c, 200, \"\", \"ok\\r\\n\");\n  }\n}\n```\nThat's it! Flash the firmware. Open your browser, type board's IP address and\nsee the \"ok\" message.\n\nNote that the [mg_http_reply()](https://mongoose.ws/documentation/#mg_http_reply)\nfunction is very versatile: it cat create formatted output, like printf\non steroids. See [mg_snprintf()](https://mongoose.ws/documentation/#mg_snprintf-mg_vsnprintf)\nfor the supported format specifiers: most of them are standard printf, but\nthere are two non-standard: `%m` and `%M` that accept custom formatting\nfunction - and this way, Mongoose's printf can print virtually anything.\nFor example, JSON strings. That said, with the aid of `mg_http_reply()`,\nwe can generate HTTP responses of arbitrary complexity.\n\nSo, how the whole flow works? Here is how. When a browser connects,\nan Ethernet IRQ handler (layer 1) kicks in. It is defined by Mongoose, and activated by\nthe `#define MG_ENABLE_DRIVER_STM32H 1` line in the `mongoose_custom.h`: [ETH_IRQHandler](https://github.com/cesanta/mongoose/blob/68e2cd9b296733c9aea8b3401ab946dd25de9c0e/src/drivers/stm32h.c#L252). Other environments, like CubeIDE, implement `ETH_IRQHandler`\nand activate it when you select \"Enable Ethernet interrupt\" in the Ethernet\nconfiguration. To avoid clash with Cube, we did not activate Ethernet interrupt.\n\nIRQ handler reads frame from the DMA, copies that frame to the Mongoose's\n[receive queue](https://github.com/cesanta/mongoose/blob/68e2cd9b296733c9aea8b3401ab946dd25de9c0e/src/net_builtin.h#L30), and exits.\nThat receive queue is special, it is a thread-safe\nsingle-producer-single-consumer non-blocking queue, so an IRQ handler, being\nexecuted in any context, can safely write to it.\n\nThe `mg_poll()` function in the infinite `while()` loop constantly\nverifies, whether we receive any data in the receive queue. When it detects\na frame in the receive queue, it extracts that frame, passes it on to the\n[mg_tcp_rx()](https://github.com/cesanta/mongoose/blob/68e2cd9b296733c9aea8b3401ab946dd25de9c0e/src/net_builtin.c#L800) function - which is an etry point to the layer 2 TCP/IP stack.\n\n\nThat `mg_tcp_rx()` function parses headers, starting from Ethernet header,\nand when it detects that a received frame belongs to one of the Mongoose\nTCP or UDP connections, it copies frame payload to the connection's `c-\u003erecv`\nbuffer and [calls `MG_EV_READ` event](https://github.com/cesanta/mongoose/blob/68e2cd9b296733c9aea8b3401ab946dd25de9c0e/src/net_builtin.c#L687).\n\n\nAt this point, processing leaves layer 2 and enters layer 3 - a library layer.\nMongoose's HTTP event handlers catches `MG_EV_READ`, parses received data,\nand when it detects that the full HTTP message is buffered, it [sends the\n`MG_EV_HTTP_MSG` with parsed HTTP message](https://github.com/cesanta/mongoose/blob/68e2cd9b296733c9aea8b3401ab946dd25de9c0e/src/http.c#L1033) to the application - layer 4.\n\nAnd this is where our event handler function `fn()` gets called. Our code is\nsimple - we catch `MG_EV_HTTP_MSG` event, and use Mongoose's API function\n`mg_http_reply()` to craft a simple HTTP response:\n```\nHTTP/1.1 200 OK\nContent-Length: 4\n\nok\n```\n\nThis response goes to Mongoose's `c-\u003esend` output buffer, and `mg_mgr_poll()`\ndrains that data to the browser, [splitting the response by frames](https://github.com/cesanta/mongoose/blob/68e2cd9b296733c9aea8b3401ab946dd25de9c0e/src/net_builtin.c#L587-L588)\nin layer 2, then passing to the layer 1. An Ethernet driver's output function [mg_tcpip_driver_stm32h_tx()](https://github.com/cesanta/mongoose/blob/68e2cd9b296733c9aea8b3401ab946dd25de9c0e/src/drivers/stm32h.c#L208) sends those frames back to the browser.\n\nThis is how Mongoose Library works.\n\nOther implementations, like Zephyr, Amazon FreeRTOS-TCP, Azure, lwIP, work in\na similar way. They implement BSD socket layer so it is a bit more complicated\ncause it includes an extra socket layer, but the principle is the same.\n\n## Implementing Web UI\n\nUsing `mg_http_reply()` function is nice, but it's very good for creating\ncustom responses. It is not suitable for serving files. And the standard way\nto build a web UI is to split it into two parts:\n- a static part, which consists of directory with `index.html`, CSS,\n  JavaScript and image files,\n- a dynamic part, which serves REST API\n\nSo instead of using `mg_http_reply()` and responding with \"ok\" to any request,\nlet's create a directory with `index.html` file and serve that directory.\nMongoose has API function `mg_http_serve_dir()` for that. Let's change the\nevent handler code to use that function:\n\n```c\nstatic void fn(struct mg_connection *c, int ev, void *ev_data) {\n  if (ev == MG_EV_HTTP_MSG) {\n    struct mg_http_message *hm = ev_data;  // Parsed HTTP request\n    struct mg_http_serve_opts opts = {.root_dir = \"/web_root\"};\n    mg_http_serve_dir(c, hm, \u0026opts);\n  }\n}\n```\n\nBuild it and get build error \"undefined reference to 'mkdir'\". This is because\n`mg_http_serve_dir()` function tries to use a default POSIX filesystem to\nread files from directory `/web_root`, and our firmware does not have support\nfor the POSIX filesystem.\n\nWhat are the possibilities here? First, we can implement POSIX filesystem,\nby using an internal or external flash memory. Then we can copy our `web_root`\ndirectory there, and our code will start to work. This is the hard way.\n\nThe easy way is to use a so-called embedded filesystem, by\ntransforming all files in the web directory into C arrays, and compiling them\ninto the firmware binary. This way, all UI files are simply hardcoded into the\nfirmware binary, and there is no need to implement a \"real\" filesystem:\n\n**Step 1**. Tell `mg_http_serve_dir()` to use packed filesystem:\n  ```c\n  struct mg_http_serve_opts opts = {.root_dir = \"/web_root\", .fs = \u0026mg_fs_packed};\n  ```\n**Step 2**. Enable packed filesystem, and disable POSIX filesystem in `mongoose_custom.h`:\n  ```c\n  #define MG_ENABLE_PACKED_FS 1\n  #define MG_ENABLE_POSIX_FS 0\n  ```\n**Step 3**. Create a new file `Core/Src/packed_fs.c`. Go to https://mongoose.ws/ui-pack/,\n  review UI files. Copy/paste the contents of generated `packed_fs.c`, save.\n\nBuild the firmware - and now it should build with no errors.\n\nLet's review what that UI packer does. As you can see, it has 3 files, which\nimplement a very simple Web UI with LED control. The `index.html` file\nloads `main.js` file, which defines a button click handler. When a button\ngets clicked, it makes a request to the `api/led/toggle` URL, and when\nthan request completes, it makes another request to `api/led/get` URL,\nand sets the status span element to the result of the request.\n\nThe tool has a preview window, and if any of the files are changed,\nit automatically refreshes preview and regenerates packed_fs.c. The packed_fs.c\nis a simple C file, which contains three C arrays, representing three files\nwe have, and two helper functions `mg_unlist()` and `mg_unpack()`, used by\nMongoose:\n- the `mg_unlist()` function allows to scan the whole \"filesystem\" and get names of every file,\n- the `mg_unpack()` function returns file contents, size, and modification time for a given file.\n\nMongoose provides a command line utility [pack.c](https://github.com/cesanta/mongoose/blob/master/test/pack.c)\nto generate `packed_fs.c` automatically during the build. The example of that\nis a [Makefile](https://github.com/cesanta/mongoose/blob/f883504d2d44d24cae1ca6c9f88ce780ab36f59b/examples/device-dashboard/Makefile#L38-L43)\nfor device dashboard example in Mongoose repository, which not only packs,\nbut also compresses files to minimise their size. But here, we'll use the\nweb tool because it is visual and makes it easy to understand the flow.\n\nThe static HTML is extremely simple. There 3 files: `index.html`, `style.css`\nand `main.js`. The `index.html` references, or loads, the other two:\n\n**index.html:**\n```html\n\u003c!DOCTYPE html\u003e\n\u003chtml\u003e\n\u003chead\u003e\n  \u003clink href=\"style.css\" rel=\"stylesheet\" /\u003e\n\u003c/head\u003e\n\u003cbody\u003e\n  \u003cdiv class=\"main\"\u003e\n    \u003ch1\u003eMy Device\u003c/h1\u003e\n    \u003cspan\u003eLED status:\u003c/span\u003e\n    \u003cspan id=\"status\"\u003e0\u003c/span\u003e\n    \u003cbutton id=\"btn\"\u003eToggle LED\u003c/button\u003e\n  \u003c/div\u003e\n  \u003cscript src=\"main.js\"\u003e\u003c/script\u003e\n\u003c/body\u003e\n\u003c/html\u003e\n```\n\n**style.css:**\n```css\n.main  { margin: 1em; }\n#status { display: inline-block; width: 2em; }\n```\n\nThe Javascript code in the `main.js` file installs an event handler on button\nclick, so when a user clicks on a button, JS code makes HTTP requests - \nI'll comment down below how it all works together:\n\n**main.js:**\n```javascript\nvar getStatus = ev =\u003e fetch('api/led/get')\n  .then(r =\u003e r.json())\n  .then(r =\u003e { document.getElementById('status').innerHTML = r; });\n\nvar toggle = ev =\u003e fetch('api/led/toggle')\n  .then(r =\u003e getStatus());\n\ndocument.getElementById('btn').onclick = toggle;\n```\n\nNow, let's flash the firmware. Go to the IP address in the browser - and\nnow we see the Web UI with a button! Click on the button, and see that nothing\nhappens! The LED does not turn on and off. Open developer tools and see that\non every click, a browser makes \"toggle\" and \"get\" requests which return 404\nerror - not found. Let's implement those API calls.\n\nChange the event handler in the following way:\n```c\nstatic void fn(struct mg_connection *c, int ev, void *ev_data) {\n  if (ev == MG_EV_HTTP_MSG) {\n    struct mg_http_message *hm = (struct mg_http_message *) ev_data;\n    if (mg_http_match_uri(hm, \"/api/led/get\")) {\n      mg_http_reply(c, 200, \"\", \"%d\\n\", HAL_GPIO_ReadPin(GPIOB, GPIO_PIN_0));\n    } else if (mg_http_match_uri(hm, \"/api/led/toggle\")) {\n      HAL_GPIO_TogglePin(GPIOB, GPIO_PIN_0); // Can be different on your board\n      mg_http_reply(c, 200, \"\", \"true\\n\");\n    } else {\n      struct mg_http_serve_opts opts = {.root_dir = \"/web_root\", .fs = \u0026mg_fs_packed};\n      mg_http_serve_dir(c, hm, \u0026opts);\n    }\n  }\n}\n```\n\nNote the `mg_http_match_uri()` checks. There, we are making different responses\nto different URLs. On `/api/led/get` URL request, we're responding with\nLED status, and on `/api/led/toggle` request, we're toggling the pin and\nresponding with `true`.\n\nBuild and flash this firmware. Refresh the page in the browser. Click on the\nbutton - and now, LED toggle works! If we open developer tools in the browser,\nwe can see the sequence of the network requests made by the browser.\n\n![Simple Web UI screenshot](media/simple_ui.webp)\n\nBelow is the diagram of the interaction between the browser and the device,\nwith explanations of every step:\n\n![Web UI sequence flow](media/web_ui_flow.svg)\n\nThis is the flow for the Web UI of any complexity. Now, it is just a matter of\ncreating a professional UI interface using any suitable JS/CSS framework, and\nextending the event handler function with the API calls that that UI invokes.\nThat's all it takes.\n\n## Implementing Device Dashboard\n\nLet me show you how to repeat everything we did in Cube - in the make + GCC\nenvironment in one minute. Navigate to https://mongoose.ws/demo/?clear=1\nThis simple web tool creates a make project completely in your browser.\nChoose the board, the \"simple project\". You can download the project to your\nworkstation and build manually. But we'll build in a browser - click on Build\nbutton. That zips the projects and sends it to mongoose.ws site, which has\nARM GCC pre-installed. It simply runs `make`, creates firmware binary, and\nsends that binary back to your browser. Now you can download that binary,\nor flash it directly from your browser.\n\nThe \"simple\" project repeats what we've already done in Cube, with one\nimportant difference - it also implements TLS. In other words, it can serve\nboth HTTP and HTTPS. Note that the binary size is less than 60 Kb! We will\ncover TLS later, as it needs a separate discussion.\n\nNow, let's click on \"Start Over\" button and build \"Web UI Dashboard\" project.\nIt follows absolutely the same flow as \"simple\" project, just the Web UI is\nsignificantly more versatile, built with Preact JS framework and Tailwind CSS\nframework. The event handler function moved into a separate file, `net.c`,\nand supports many API calls required by Web UI - to show dashboard stats,\nsettings, and firmware update. By the way, the firmware update is completely\nfunctional - but I won't cover it here, as it is a big topic on itself.\nI won't cover the process of static UI creation in React, as there are tons\nof very good tutorials on that. But if you want me to cover that, join our\nDiscord server and let me know.\n\nWhat I'll do is to move that UI into the Cube project of ours.\n\n**Step 1.** Copy net.c, net.h, packed_fs.c into the Cube project  \n**Step 2.** Add the following `include \"net.h\"` at the top of the main.c file  \n**Step 3.** Comment out `mg_http_listen(...)` call, add `web_init()` call  \n**Step 4.** Open net.h, modify HTTP_URL port 8000 to port 80\n\nRebuild, reflash, refresh your browser. We have a functional versatile\nWeb UI device dashboard reference running!\n\n## Device management using MQTT protocol\n\n\n## Enabling TLS\n\n## Talking to AWS IoT and Microsoft Azure services\n\n## About me\n\nI am Sergey Lyubka, an engineer and entrepreneur. I hold a MSc in Physics from\nKyiv State University, Ukraine. I am a director and a co-founder at Cesanta - a\ntechnology company based in Dublin, Ireland. Cesanta develops embedded\nsolutions:\n\n- https://mongoose.ws - an open source HTTP/MQTT/Websocket network library\n- https://vcon.io - a remote firmware update / serial monitor framework\n\nYou are welcome to register for\n[my free webinar on embedded network programming](https://mongoose.ws/webinars/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcpq%2Fembedded-network-programming-guide","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcpq%2Fembedded-network-programming-guide","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcpq%2Fembedded-network-programming-guide/lists"}