{"id":23872118,"url":"https://github.com/sb-im/ncp","last_synced_at":"2026-05-13T13:46:07.044Z","repository":{"id":103057191,"uuid":"376246929","full_name":"sb-im/ncp","owner":"sb-im","description":"Edge gateway core communication service ","archived":false,"fork":false,"pushed_at":"2023-04-06T16:20:39.000Z","size":386,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-21T15:52:44.875Z","etag":null,"topics":["gateway","gateway-services","iot","mqtt"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sb-im.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-12T09:09:54.000Z","updated_at":"2022-03-31T14:48:44.000Z","dependencies_parsed_at":"2023-04-16T17:31:05.939Z","dependency_job_id":null,"html_url":"https://github.com/sb-im/ncp","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sb-im%2Fncp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sb-im%2Fncp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sb-im%2Fncp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sb-im%2Fncp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sb-im","download_url":"https://codeload.github.com/sb-im/ncp/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240222493,"owners_count":19767458,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gateway","gateway-services","iot","mqtt"],"created_at":"2025-01-03T15:20:15.606Z","updated_at":"2026-05-13T13:46:06.960Z","avatar_url":"https://github.com/sb-im.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Node control protocol\n\nEdge gateway core communication service\n\n[![Build Status](https://github.com/sb-im/ncp/workflows/ci/badge.svg)](https://github.com/sb-im/ncp/actions?query=workflow%3Aci)\n[![codecov](https://codecov.io/gh/sb-im/ncp/branch/master/graph/badge.svg)](https://codecov.io/gh/sb-im/ncp)\n[![Go Report Card](https://goreportcard.com/badge/github.com/sb-im/ncp)](https://goreportcard.com/report/github.com/sb-im/ncp)\n[![Go Reference](https://pkg.go.dev/badge/github.com/sb-im/ncp.svg)](https://pkg.go.dev/github.com/sb-im/ncp)\n[![GitHub release](https://img.shields.io/github/tag/sb-im/ncp.svg?label=release)](https://github.com/sb-im/ncp/releases)\n[![license](https://img.shields.io/github/license/sb-im/ncp.svg?maxAge=2592000)](https://github.com/sb-im/ncp/blob/master/LICENSE)\n\n[README](README.md) | [中文文档](README_zh.md)\n\n## What is ncp\n\nThis is a communication module, and there is no specific use specified\n\n## Table of Contents\n\n\u003c!-- vim-markdown-toc GFM --\u003e\n\n* [Compile \u0026\u0026 Run](#compile--run)\n* [Architecture design](#architecture-design)\n* [NcpIO](#ncpio)\n  * [Rules](#rules)\n  * [Debug](#debug)\n  * [Name](#name)\n* [IOs](#ios)\n  * [API](#api)\n  * [tcpc](#tcpc)\n  * [tcps](#tcps)\n  * [exec](#exec)\n  * [jsonrpc2](#jsonrpc2)\n  * [logger](#logger)\n  * [mqtt](#mqtt)\n* [mqttd](#mqttd)\n  * [mqttd-rpc](#mqttd-rpc)\n  * [mqttd-status](#mqttd-status)\n  * [mqttd-network](#mqttd-network)\n  * [mqttd-trans](#mqttd-trans)\n  * [mqttd-buildin](#mqttd-buildin)\n* [Example](#example)\n* [TODO](#todo)\n\n\u003c!-- vim-markdown-toc --\u003e\n\n## Compile \u0026\u0026 Run\n\n* make\n* go \u003e= 1.13\n\n```sh\nmake\n\n# run\ncp conf/config-dist.yml config.yml\nncp -c config.yml\n```\n\n## Architecture design\n\n![architecture](docs/ncp-architecture.svg)\n\n## NcpIO\n\nThis is the core of the entire application and can also be used as a library\n\nThe model used here is an `IO` module and a message center, where each `IO` module receives a message and broadcasts it to the other `IO` modules\n\nThe `rules` are used to determine whether the message is acceptable or not, and it is negative when no rule is matched (the message is not accepted)\n\nThis module can be used as a library.\nMessages can be sent and received directly using the `api` interface\n\n```yaml\nncpio:\n  - type: tcpc\n    name: airctl\n    params: \"localhost:8980\"\n    i_rules:\n      - regexp: '.*\"method\": ?\"(webrtc|history|ncp)\".*'\n        invert: true\n      - regexp: '.*\"method\".*'\n    o_rules:\n      - regexp: '.*'\n  - type: api\n    name: airctl\n```\n\n### Rules\n\n![ncpio-rules](docs/ncp-ncpio.svg)\n\n* `o_rules` output the io rule (after recv)\n* `i_rules` input the io rule (before send)\n\n1. `i_rules` \u0026 `o_rules` use regexp matched, because message should plain\n2. There can be more than one rule, match them accordingly, and as long as one of them is met, the rule will be allows\n3. The `invert` field controls whether the regular match result should be inverted\n4. When no rule is matched, the message does disallows\n\n### Debug\n\nThe complexity of using the regular expression part is hard to debug. That's why debug mode was added\n\n```bash\n./ncp -debug\n```\n\nEvery messages by `IO` need two rules group\n\nThree log: (RECV, BROADCAST, SEND)\n\n1. `RECV`: `IO`  When received the message\n2. by `o_rules`\n3. `BROADCAST`: When broadcasting a message to all `IO`\n4. by `i_rules`\n5. `SEND`: `IO` When can send a message\n\n### Name\n\nThere can be more than one `IO` module of each type. You can give each one a name\n\nBy default, a unique name is automatically generated, incrementing from `0`, `1`, `2`.\n\nIt is recommended not to set the `name` field manually, if you set the same `name` field,\nit will be placed in the broadcast message and sent to you by `name`.\n\nThe same `name` will not receive the broadcast message with the same name.\n**Please check if you intentionally do not receive the broadcast, or if you accidentally set it to be the same**\n\n## IOs\n\nType                 | Description\n-------------------- | -----------\n[API](#api)          | build-in interface\n[tcpc](#tcpc)        | TCP socket client\n[tcps](#tcps)        | TCP socket server\n[exec](#exec)        | Execute system command\n[jsonrpc2](#jsonrpc2)| JSONRPC 2.0 simulation\n[logger](#logger)    | Record message log\n[mqtt](#mqtt)        | Connect Mqtt Broker\n\n\nAt the core are six `IO` modules, with different combinations defined by configuration files to achieve different functions\n\nWhy is there no `udp server` and `udp client`?\n\nThere was no need for this at the time of development, some messages need to be verified for reliability. `udp` is not suitable\n\n### API\n\nIs a special interface\n\nIt is used when ncpio is used as a library, and through this API is an IO module\n\n```go\nimport(\n\tncpio \"sb.im/ncp/ncpio\"\n)\n\nfunc main()\n\tncpios := ncpio.NewNcpIOs([]ncpio.Config{\n\t\t{\n\t\t\tType: \"api\",\n\t\t\tIRules: []ncpio.Rule{\n\t\t\t\t{\n\t\t\t\t\tRegexp: `.*`,\n\t\t\t\t\tInvert: false,\n\t\t\t\t},\n\t\t\t},\n\t\t\tORules: []ncpio.Rule{\n\t\t\t\t{\n\t\t\t\t\tRegexp: `.*`,\n\t\t\t\t\tInvert: false,\n\t\t\t\t},\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tType:   \"jsonrpc2\",\n\t\t\tParams: `{\"result\":\"ok\"}`,\n\t\t\tIRules: []ncpio.Rule{\n\t\t\t\t{\n\t\t\t\t\tRegexp: `.*`,\n\t\t\t\t\tInvert: false,\n\t\t\t\t},\n\t\t\t},\n\t\t\tORules: []ncpio.Rule{\n\t\t\t\t{\n\t\t\t\t\tRegexp: `.*`,\n\t\t\t\t\tInvert: false,\n\t\t\t\t},\n\t\t\t},\n\t\t},\n\t})\n\tgo ncpios.Run(ctx)\n\n\tncpio.I \u003c- []byte(`{\"jsonrpc\":\"2.0\",\"method\":\"test\",\"id\":\"test.0\"}`)\n\tfmt.Println(string(\u003c-ncpio.O))\n}\n```\n\nNote: Unit tests are tested using this interface\n\nThere can be at most one API interface\n\n### tcpc\n\nAs a tcp client to connect to the server, the most common working mode\n\n`tcpc`, `tcps` message splitting using `\\n` as separator, will automatically `chomp` `\\r\\n` the case\n\n### tcps\n\nAs a tcp server to wait for client connections, reverse tcp mode of operation\n\nNote: Only one client can be connected at a time.\nmultiple clients connected at the same time will wait until the previous close before handling new connections\n\n### exec\n\nexec system command. **NOTE: This is very dangerous!**\n\n```yaml\n  - type: exec\n    params: \"echo\"\n    i_rules:\n      - regexp: '.*\"method\": ?\"exec\".*'\n    o_rules:\n      - regexp: '.*'\n```\n\nThis `params` is command, example: `params: \"/bin/sh\"`\n\nThis Recvice JSONRPC, ignore `jsonrpc.method`. **jsonrpc.method is use of only match**\n\nFor example:\n\n`params: \"echo\"`. jsonrpc2 `{\"jsonrpc\":\"2.0\",\"method\":\"exec\",\"params\":[\"-n\", \"xxx\"],\"id\":\"x\"}`\n\nFinally exec: `echo -n xxx`\n\n### jsonrpc2\n\nIs a simulator that can simulate jsonrpc correct and error messages, which is often used for development and debugging\n\nThe `params` field of this module is a bit more complicated: the\n\nWhen it is json, this is inserted directly (only `result` and `error` are legal)\n\nWhen it's not json, the result is assigned directly to `result`.\n\nSimulation success: `params: 233` is the same as this.\n\n```json\n{\"result\": 233}\n```\n\nSimulation error:\n\n```json\n{\"error\": {\"code\": 0, \"message\": \"xxxxx\"}}\n```\n\n### logger\n\n```yaml\nparams: \"file:///tmp/ncp/test.log?size=128M\u0026count=8\u0026prefix=SB\"\n```\n\nWhy `//`, this is the standard way to write `//` when the url is part of `/tmp` is the root path\n\n* size The maximum size of each log file, beyond which it will rotate\n* count Keeps the last few files\n* prefix log prefix\n\n### mqtt\n\nThis module is a bit complicated so break it down to ncpio's mqtt and mqttd\n\nOf course, you can also ignore this function and treat it as a normal io\n\nAll params are defined as strings, because yaml doesn't support save parsing, so it's not possible to do the same as `json.RawMessage`\n\nThe reason is: [go-yaml/issues#13](https://github.com/go-yaml/yaml/issues/13)\n\nyaml is contextual, so it doesn't have the same context as json\n\nSo mqtt's params are specified as a path\n\n## mqttd\n\n![mqttd](docs/ncp-mqttd.svg)\n\nMessages received through the IO module will arrive here\n\nThe config file [mqtt.yaml](conf/mqtt.yml)\n\n### mqttd-rpc\n\nconfig params:\n\n```yaml\nmqttd:\n  rpc:\n    qos: 0\n    lru: 128\n    i: \"nodes/%s/rpc/recv\"\n    o: \"nodes/%s/rpc/send\"\n```\n\nThere are two mqtt topics corresponding to `i` and `o`, sending and receiving. topic is full duplex, why do we need to split into two topics?\n\nA simple use can be to set both values to be the same and use one topic for sending and receiving\n\n* * *\n\nMessage reliability control, two modes. Let's review `tcp` and `kcp` first\n\n`tcp` returns an acknowledgement message when it receives a message, and then sends a new message, which is acknowledged by a sliding window\n\nLater, in the pursuit of low latency, `udp` is wrapped with reliability, and packets are sent violently to improve response time. This is what `kcp` does\n\nThe `kcp` approach has lower latency than `tcp`, but consumes more traffic than `tcp`.\n\nMessage reliability control, although `mqtt` but `qos` can be reliability control, set `qos 1` or `qos 2`\n\nIn the actual test product, `mqtt` itself `qos` does not perform well when the network conditions are bad, the real-time requirements are high, and the data volume is large\n\nmqtt's `qos` will repeatedly confirm the arrival of packet messages (`qos` is doubled for broker, two clients), so use `qos 0` for reliability control at the upper layer itself\n\n* * *\n\nThis introduces two parameters: `qos` and `lru`\n\nUse mqtt's own `qos` (similar to `tcp` mode) to set `qos` to `2`. lru is set to `0`.\n\nown reliability processing (`kcp`-like mode) to set `qos` to `0`. lru to a number other than `0` (default is `128`)\n\nWhy is there a `LRU` here?\n\nSo use a new mode, brute force data sending, using LRU to filter duplicate messages sent\n\nIt's more complicated to write your own message validation part to control reliability.\n\nIt is recommended that the demo use mqtt's own qos. product environment uses `LRU` and `qos 0` for this working mode\n\n### mqttd-status\n\n`status` \u0026 `network` is send self status\n\n`status` send self status: mqtt last will message, network error\n\n```json\n{\n  \"msg\":\"neterror\",\n  \"timestamp\":\"1619164695\",\n  \"status\":{\n    \"lat\":\"22.6876423001\",\n    \"lng\":\"114.2248673001\",\n    \"alt\":\"10088.0001\"\n  }\n}\n```\n\n`status` config is `mqttd.static` . An extended field that can hold some meta information\n\n`msg`: `online`, `offline`, `neterror`\n\n### mqttd-network\n\n```json\n{\"loss\":0,\"delay\":5}\n```\n\n`network` send self and mqtt broker network status\n\n* `loss` (1~100)%\n* `delay` unit `ms`\n\n### mqttd-trans\n\nThe received message will be judged as non-jsonrpc, if it is jsonrpc message will connect to mqtt broker in rpc way\n\nIf it is not jsonrpc, it will go through this module, which is a one-way data\n\ntran socket recv\n\n```json\n{\"weather\":{\"WD\":0,\"WS\":0,\"T\":66115,\"RH\":426,\"Pa\":99780}}\n```\n\nmqtt send topic: `nodes/:id/msg/weather`\n\n```json\n{\"WD\":0,\"WS\":0,\"T\":66115,\"RH\":426,\"Pa\":99780}\n```\n\nAutomatically maps the outermost key of the json as part of the topic.\n(Multiple outer keys will be split into multiple messages)\n\n`gtran.prefix` config mqtt prefix\n\n`trans` config mqtt topic `qos` and `retain`\n\nFor example:\n\n```json\n{\n  \"weather\":{\"WD\":0,\"WS\":0,\"T\":66115,\"RH\":426,\"Pa\":99780},\n  \"battery\":{\"status\":\"ok\"}\n}\n```\n\nmqtt send topic: `nodes/:id/msg/weather`\n\n```json\n{\"WD\":0,\"WS\":0,\"T\":66115,\"RH\":426,\"Pa\":99780}\n```\n\nmqtt send topic: `nodes/:id/msg/battery`\n\n```json\n{\"status\":\"ok\"}\n```\n\n### mqttd-buildin\n\nbuild-in `history` method\n\n`history` get `trans` send history data\n\nparams:\n\nField | Type | Description\n----- | ---- | -----------\ntopic | string | Topic\ntime  | string | such as \"300ms\", \"-1.5h\" or \"2h45m\". Valid time units are \"ns\", \"us\" (or \"µs\"), \"ms\", \"s\", \"m\", \"h\" [golang time#ParseDuration](https://golang.org/pkg/time/#ParseDuration)\n\nRequest:\n\n```json\n{\n  \"jsonrpc\":\"2.0\",\n  \"id\":\"sdwc.1-1553321035000\",\n  \"method\":\"history\",\n  \"params\":{\n    \"topic\":\"msg/weather\",\n    \"time\": \"10s\"\n  }\n}\n```\n\nResponse:\n\n```json\n{\n  \"jsonrpc\": \"2.0\",\n  \"id\":\"sdwc.1-1553321035000\",\n  \"result\":[\n    {\"1553321035\": {\"WD\": 20, \"WS\": 5}},\n    {\"1553321034\": {\"WD\": 10, \"WS\": 1}},\n    {\"1553321000\": {\"WD\": 1, \"WS\": 1}}\n  ]\n}\n```\n\nThe result[key] , key is timestamp\n\n`ncp_online` and `ncp_offline` control online status\n\nIs jsonrpc notification. not result, not params\n\n## Example\n\nIf use: `tcpc`, `tcps`, `mqtt`, `logger`\n\nUse this config.yaml :\n\n```yaml\n# type: tcps / tcpc / mqtt / logger / jsonrpc2 / api\nncpio:\n  - type: tcpc\n    params: \"localhost:8980\"\n    i_rules:\n      - regexp: '.*\"method\": ?\"(webrtc|history|ncp)\".*'\n        invert: true\n      - regexp: '.*\"method\".*'\n    o_rules:\n      - regexp: '.*'\n  - type: tcps\n    params: \"localhost:1208\"\n    i_rules:\n      - regexp: '.*\"method\": ?\"webrtc\".*'\n    o_rules:\n      - regexp: '.*\"method\".*'\n        invert: true\n      - regexp: '.*'\n  - type: mqtt\n    params: \"config.yml\"\n    i_rules:\n      - regexp: '.*'\n    o_rules:\n      - regexp: '.*'\n  - type: logger\n    params: \"file:///tmp/ncp/test.log?size=128M\u0026count=8\u0026prefix=SB\"\n    i_rules:\n      - regexp: '.*'\n```\n\nbroker to mqttd send:\n\n```json\n{\"jsonrpc\":\"2.0\",\"id\":\"test.0-1553321035000\",\"method\":\"webrtc\",\"params\":[]}\n```\n\n1. `mqtt` will first go through its own `o_rules` and find that it is `. *` (matching any message), so it broadcasts this message to `tcpc` `tcps` `logger`\n2. `logger` is a logger: `i_rules` is usually `. *` writes to the log after receiving this message\n3. `tcpc`'s `i_rules` matches `. *\"method\": ?\" (webrtc|history|ncp)\". *`, but `invert` reverses the match. This message is rejected\n4. `i_rules` of `tcps` just matches this command. This command is sent to `tcps`\n5. `tcps` receives the message filtered by `o_rules` and sends it to the message center, which broadcasts the message to every enabled `IO` other than itself (excluding the sender of the message).\n6. `mqtt` also receives the message, and after receiving the broadcast message, it uses its own `i_rules` to determine if the message is ready to be sent.\n7. `logger` will also receive this message, and after receiving the broadcast message, it will use its own `i_rules` to determine if the message needs to be logged to the log file\n\n## TODO\n\n* [ ] io udp's io maybe needs to be added\n* [ ] io mqtt or need a configuration item to enable to use json and jsonrpc\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsb-im%2Fncp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsb-im%2Fncp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsb-im%2Fncp/lists"}