{"id":23235945,"url":"https://github.com/mariotoffia/godeviceshadow","last_synced_at":"2026-04-26T12:31:59.040Z","repository":{"id":268143153,"uuid":"903441962","full_name":"mariotoffia/godeviceshadow","owner":"mariotoffia","description":"Model persister that can emulate a device shadow. Pluggable storage, diff/merge, loggers, advanced filtering/notification mechanism with implementations in memory/DynamoDB/streams.","archived":false,"fork":false,"pushed_at":"2025-08-08T12:28:11.000Z","size":596,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-11-03T00:02:13.624Z","etag":null,"topics":["antlr4","aws","device-shadow","dsl","dynamodb","dynamodbstreams","iot","notification-service","pluggable"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mariotoffia.png","metadata":{"files":{"readme":"README.adoc","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-12-14T16:04:49.000Z","updated_at":"2025-08-08T12:28:14.000Z","dependencies_parsed_at":"2025-07-07T07:27:27.573Z","dependency_job_id":"ff762aee-6076-4cdc-83c9-c08cd1de9f58","html_url":"https://github.com/mariotoffia/godeviceshadow","commit_stats":null,"previous_names":["mariotoffia/godeviceshadow"],"tags_count":24,"template":false,"template_full_name":null,"purl":"pkg:github/mariotoffia/godeviceshadow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mariotoffia%2Fgodeviceshadow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mariotoffia%2Fgodeviceshadow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mariotoffia%2Fgodeviceshadow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mariotoffia%2Fgodeviceshadow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mariotoffia","download_url":"https://codeload.github.com/mariotoffia/godeviceshadow/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mariotoffia%2Fgodeviceshadow/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32297893,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-26T09:34:17.070Z","status":"ssl_error","status_checked_at":"2026-04-26T09:34:00.993Z","response_time":129,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["antlr4","aws","device-shadow","dsl","dynamodb","dynamodbstreams","iot","notification-service","pluggable"],"created_at":"2024-12-19T03:30:13.218Z","updated_at":"2026-04-26T12:31:59.033Z","avatar_url":"https://github.com/mariotoffia.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":":!example-caption:\n\n= Go Device Shadow (godeviceshadow)\n\nimage:https://github.com/mariotoffia/godeviceshadow/actions/workflows/go-test.yml/badge.svg[Go Tests,link=https://github.com/mariotoffia/godeviceshadow/actions/workflows/go-test.yml]\nimage:https://img.shields.io/github/go-mod/go-version/mariotoffia/godeviceshadow[Go Version]\nimage:https://img.shields.io/github/license/mariotoffia/godeviceshadow[License]\nimage:https://img.shields.io/github/v/release/mariotoffia/godeviceshadow[Release]\nimage:https://sonarcloud.io/api/project_badges/measure?project=mariotoffia_godeviceshadow\u0026metric=alert_status[Quality Gate Status,link=https://sonarcloud.io/summary/new_code?id=mariotoffia_godeviceshadow]\n\n== Introduction\n[.lead]\nGoDeviceShadow provides a pluggable device shadow implementation with persistence and notification capabilities.\n\n* 🧩 *Modular Architecture* - Use components together or separately with a pluggable design\n* 🔄 *Shadow State Management* - Handles both reported and desired states with configurable merging strategies - but can be handled in any custom way!\n* 🗄️ *Pluggable Persistence* - Built-in support for in-memory storage.\n* 📢 *Notification System* - Configurable event notification with filtering capabilities for in-memory notifications is built-in.\n* 🔍 *Change Tracking* - Detailed logging of changes through merge loggers\n* 🧰 *Type-Safe Models* - Works with Go structs instead of plain JSON for type safety\n* 🚀 *No Dependencies* - Core runtime has zero external dependencies (only plugins have dependencies)\n* 🔌 *Extensible* - Easy to implement custom plugins for storage, notification, and logging\n\n=== Module Overview\n\n[cols=\"1,4\"]\n|===\n|Module |Description\n\n|📦 *godeviceshadow*\n|Core module with the main runtime, merge logic, and interfaces\n\n|💾 https://github.com/mariotoffia/godeviceshadow/tree/main/persistence/dynamodbpersistence[dynamodbpersistence]\n|Persistence implementation for Amazon DynamoDB.\n\n|📡 https://github.com/mariotoffia/godeviceshadow/tree/main/notify/dynamodbnotifier[dynamodbnotifier]\n|DynamoDB Streams based event notification system.\n\n|🔤 https://github.com/mariotoffia/godeviceshadow/tree/main/notify/selectlang[selectlang]\n|DSL for creating notification selection filters.\n|===\n\n=== Quick Start\nThis system is in it's essence a in memory diff-merge tool that allows code to track changes in real-time changes in model. It has the ability to do the `Desired` function that will acknowledge reported values. It allows to use `ServerIsMaster` or `ClientIsMaster` to control how merging of reported and desired are performed.\n\nIt separates Reported and Desired states and thus can be stored separately or combined (see DynamoDB persistence) for performance and flexibility. Persistence, notification is pluggable.\n\n.Example Report, Desire \u0026 Loggers\n[source,go]\n----\nctx := context.Background()\nnow := time.Now()\nmgr := // \u003c1\u003e\n\nid := persistencemodel.ID{ID: \"device123\", Name: \"homeHub\"}\n\nres := mgr.Report(ctx, managermodel.ReportOperation{ // \u003c2\u003e\n  ClientID: \"myClient\",\n  Version:  0, // \u003c3\u003e\n  ID: id,\n  Model: TestModel{\n    TimeZone: tz, Sensors: map[string]Sensor{ \"temp\": {Value: 23.4, TimeStamp: now} },\n  },\n})\n\nchl := changelogger.Find(res[0].MergeLoggers) // \u003c4\u003e\nsns, err := chl.ManagedFromPath(`Sensors\\..*`)\nsensors := sns.All()\n\nfmt.Printf(\"%s: %s\", sensors[0].Path, sensors[0].NewValue.GetTimestamp().Format(time.RFC3339)) // \u003c5\u003e\n\nres2 := mgr.Desire(ctx, managermodel.DesireOperation{ // \u003c6\u003e\n  ClientID: \"myClient\",\n  ID: id,\n  Model: TestModel{\n    TimeZone: tz, Sensors: map[string]Sensor{ \"sp\": {Value: 99.2, TimeStamp: now} } },\n})\n\nres = mgr.Report(ctx, managermodel.ReportOperation{\n  ClientID: \"myClient\",\n  Version:  0, // \u003c7\u003e\n  ID: id,\n  Model: TestModel{\n    TimeZone: tz, Sensors: map[string]Sensor{ \"sp\": {Value: 99.2, TimeStamp: now} }, // \u003c7\u003e\n  },  \n})\n----\n\u003c1\u003e Manager created elsewhere (_see below example_).\n\u003c2\u003e Report the model and thereby merge with model in the persistence and ensure any desired acknowledgements are done. In this case nothing is persisted so it will create a new model and merge it.\n\u003c3\u003e The version is 0 and will be incremented by the system. If using zero it will always use the latest version to merge with. If explicit version is use, it will only merge if the version is the same (and then increment the version).\n\u003c4\u003e Find the change logger and extract the managed values from the path `Sensors\\..*` (_Regular Expression_).\n\u003c5\u003e Outputs e.q. _Sensors.temp: 2025-01-22T13:22:26+01:00_\n\u003c6\u003e Will add _sp_ to desired state (it is possible to have merge loggers here as well to listen for desire merge).\n\u003c7\u003e Will acknowledge and therefore remove or zero it from desired state (can be listen to, just supply desire loggers).\n\nThe above sample shows how to report and desired a certain value including how to access a merge logger. It relies on a manager to be configured and built.\n\n.Example Create a In-Memory Persistence Manager\n[source,go]\n----\nmgr := stdmgr.New(). // \u003c1\u003e\n  WithPersistence(mempersistence.New()). // \u003c2\u003e\n  WithSeparation(persistencemodel.SeparateModels). // \u003c3\u003e\n  WithReportedLoggers(changelogger.New()). // \u003c4\u003e\n  WithTypeRegistryResolver( // \u003c5\u003e\n    types.NewRegistry().RegisterResolver(\n      model.NewResolveFunc(func(id, name string) (model.TypeEntry, bool) {\n        if name == \"homeHub\" { // \u003c5\u003e\n          return model.TypeEntry{\n            Name: \"homeHub\", Model: reflect.TypeOf(TestModel{}),\n          }, true\n        }\n\n        return model.TypeEntry{}, false\n      }),\n    ),\n  ).\n  Build()\n----\n\u003c1\u003e Create a builder to create a new manager.\n\u003c2\u003e Use in-memory persistence. Swap this e.g. for _DynamoDB_ persistence via `dynamodbpersistence.New(...)`.\n\u003c3\u003e Separate the model persistence by default - can be overridden on each write operation. Default is to combine desired and reported in persistence. However, it is up to persistence how adhere to this.\n\u003c4\u003e Use the change logger to log changes both managed and plain values for post examination. This registers the `New` function so a new logger is always created on each report. It is possible to create your own or use existing merge loggers to participate in the merge.* \n\u003c5\u003e There are a few ways of resolving what type (used in read operation) the model is in. This registers a on-the-fly resolver.\n\nIt is then possible to notify using a notification manager. Then it is possible to define selection that will resolve to a target. Thus where to notify may be heavily customized. There is a *experimental* _DSL_ that can render the selectors and hence not needed to code those (even though they are super simple - just one function).\n\nTIP: See the link:notify/selectlang/README.adoc[Selection Language Documentation]\n\n.Example Notification Selection DSL\n[source,sql]\n----\nSELECT * FROM Notification WHERE\n    (\n        obj.ID ~= 'myDevice-\\\\d+' AND # \u003c1\u003e\n        obj.Name == 'homeHub' AND \n        obj.Operation IN 'report','desired'\n    )\n    AND\n    (\n        log.Operation IN 'add','update' AND\n        log.Path ~= '^Sensors.indoor-\\\\d+$' AND # \u003c2\u003e\n        log.Name == 'temp' AND\n        (   # \u003c3\u003e\n            log.Value \u003e 20 OR (log.Value ~= '^re-\\\\d+' AND log.Value != 'apa' OR \n            (log.Value \u003e 99 AND log.Value ~!= '^bubben-\\\\d+$'))\n        )\n    )\n    OR\n    (log.Operation == 'acknowledge') # \u003c4\u003e\n----\n\u003c1\u003e One or more primary expressions that matches the ID and which operation.\n\u003c2\u003e Zero or more log expressions that interacts with the values being handled\n\u003c3\u003e Log expressions may have as many constraints as needed. It is possible to mix _value_ expressions and it will capture\nonly values that it may do with the expression. For example float values will be converted to string when regex etc.\n\u003c4\u003e It is also possible to select all acknowledged values\n\nWhen a `Selection` returns `true`, the target may be invoked.\n\nNOTE: 🚨 *The DSL is experimental and may change in the future.*\n\nThe `Selection` may be used to capture a set of values. Just submit `true` on the _value_ parameter when processing. Thus, they may be used outside the notification mechanism.\n\n== TIP 💡: View All Examples\nTo view all examples, visit the https://github.com/mariotoffia/godeviceshadow/tree/main/examples[Examples] directory.\n\n== Core Concepts\n\nThis is a model runtime and not a plain _JSON_ runtime, thus it handles golang models. The main interface is the `model.ValueAndTimestamp` of which it uses to discover variables and handle them.\n\nValueAndTimestamp Interface\n[source,go]\n----\n// ValueAndTimestamp is the interface that fields must implement if they\n// support timestamp-based merging.\ntype ValueAndTimestamp interface {\n  // GetTimestamp will return the timestamp associated with the value. This is\n  // used to determine which value is newer when a merge is commenced.\n  GetTimestamp() time.Time\n  // GetValue will return the value that the timestamp is associated with.\n  //\n  // If multiple values, the instance itself is the value and this method\n  // will return the _\"default\"_ value. If the value is a map[string]any\n  // it will return all values where the key is the name of the value.\n  //\n  // The latter gives the caller a way of knowing what values are relevant\n  // to e.g. log instead of iterate the whole struct.\n  GetValue() any\n}\n----\n\nThose may be anywhere in a structs, maps etc. The system will iterate all and handle all such elements.\n\n.Example \"DeviceShadow\" Model\n[source,go]\n----\ntype HomeTemperatureHub struct {\n  *MetaInfo      `json:\"meta,omitempty\"`\n  ClimateSensors *ClimateSensors            `json:\"climate,omitempty\"`\n  IndoorTempSP   *IndoorTemperatureSetPoint `json:\"indoor_temp_sp,omitempty\"` // Important omitempty when used in desired\n}\n\ntype MetaInfo struct {\n  TimeZone string `json:\"tz,omitempty\"`\n  Owner    string `json:\"owner,omitempty\"`\n}\n\ntype Direction string\n\nconst (\n  DirectionNorth Direction = \"north\"\n  DirectionSouth Direction = \"south\"\n  DirectionEast  Direction = \"east\"\n  DirectionWest  Direction = \"west\"\n)\n\ntype IndoorTemperatureSensor struct {\n  Floor       int       `json:\"floor\"`\n  Direction   Direction `json:\"direction\"`\n  Temperature float64   `json:\"t\"`\n  Humidity    float64   `json:\"h\"`\n  UpdatedAt   time.Time `json:\"ts\"`\n}\n\nfunc (idt *IndoorTemperatureSensor) GetTimestamp() time.Time {\n  return idt.UpdatedAt\n}\n\nfunc (idt *IndoorTemperatureSensor) GetValue() any {\n  return map[string]any{ // \u003c1\u003e\n    \"floor\":       idt.Floor,\n    \"direction\":   idt.Direction,\n    \"temperature\": idt.Temperature,\n    \"humidity\":    idt.Humidity,\n  }\n}\n\ntype OutdoorTemperatureSensor struct {\n  Direction   Direction `json:\"direction\"`\n  Temperature float64   `json:\"t\"`\n  Humidity    float64   `json:\"h\"`\n  UpdatedAt   time.Time `json:\"ts\"`\n}\n\nfunc (ots *OutdoorTemperatureSensor) GetTimestamp() time.Time {\n  return ots.UpdatedAt // \u003c2\u003e\n}\n\nfunc (ots *OutdoorTemperatureSensor) GetValue() any {\n  return map[string]any{\n    \"direction\":   ots.Direction,\n    \"temperature\": ots.Temperature,\n    \"humidity\":    ots.Humidity,\n  }\n}\n\ntype IndoorTemperatureSetPoint struct {\n  SetPoint  float64   `json:\"sp\"`\n  UpdatedAt time.Time `json:\"ts\"`\n}\n\nfunc (sp *IndoorTemperatureSetPoint) GetTimestamp() time.Time {\n  return sp.UpdatedAt\n}\n\nfunc (sp *IndoorTemperatureSetPoint) GetValue() any {\n  return sp.SetPoint\n}\n\ntype ClimateSensors struct {\n  Outdoor map[string]OutdoorTemperatureSensor `json:\"outdoor,omitempty\"`\n  Indoor  map[string]IndoorTemperatureSensor  `json:\"indoor,omitempty\"`\n}\n----\n\u003c1\u003e When map, it will check all values to determine if any value change has occurred, otherwise just return a plain value.\n\u003c2\u003e This is the timestamp it will use to determine if the value is newer or older (or same).\n\n== Device Shadow Layout\n\nThe device shadow is rather alike the IoT Core Device Shadow but with a few differences. It can split the _Reported_ and _Desired_ states into two different sort keys to allow for more data and better querying and possibly performance.\n\n=== Loggers\n\nThere is a pluggable logger architecture to allow for multiple loggers to participate in report diff or desired acknowledges/diffs. This allows for e.g. output the changes or to store added/changed values in _Amazon Aurora DSQL_, _Time-Stream_ or similar storage. Loggers may interact with \"plain\" elements such as simple string or the \"managed\" (those who implements the `model.ValueAndTimestamp` interface).\n\nLoggers ar very easy to create since they rely on two functions only to allow for add, remove, changed, and not changed. Thus it is possible to check what has not changed as well!\n\n.Logger Interface\n[source,go]\n----\ntype MergeLogger interface {\n  Managed(\n    path string,\n    operation MergeOperation, // \u003c1\u003e\n    oldValue, newValue ValueAndTimestamp,\n    oldTimeStamp, newTimeStamp time.Time)\n\n  Plain(path string, operation MergeOperation, oldValue, newValue any) // \u003c2\u003e\n}\n----\n\u003c1\u003e The `MergeOperation` specifies if it is an add, remove, change or not changed operation.\n\u003c2\u003e The `Plain` method is used for plain values that does not implement the `ValueAndTimestamp` interface such as a `string`.\n\n=== Notifications\n\nWhen a shadow is updated, a notification can be sent to listeners. This is done by the notification implementation. \n\nEach target registration specifies what type of plugin (e.g. _SQS_), attributes such as the queue name, topic name, etc.\n\nIn addition the attributes specifies what type of events to listen for:\n* Report, Desired or Both\n* Regexp for PK and SK combined with a'#' separator.\n* Old, New, Diff (or any combination of these)\n\nThe registrations are stored as _JSON_ with the event lambda itself (for dynamodb stream). \n\n\n== Client SDK\n\n=== Deviations\n\nThere are many deviations from the IoT Core Device Shadow. One of the most prominent is the notion of the device shadow _MODEL_ in go struct instead of plain _JSON_. This allows for a more type-safe way of handling the device shadow.\n\nIn this implementation, it is possible to control how the merge is done i.e. if server is master or client is master where the latter allows for client to delete entries that are not present in the client model. The former do not allow for deletion of entries, instead it only supports addition, updates and no changes.\n\n=== Timestamps\n\nThe timestamps on the items in the device shadow is completely different than for the IoT Core Device Shadow. The timestamps a _RFC3339_ timestamp (but since it uses the interface, they may be anything). The _RFC3339_ timestamp may be used when the tz may differ between the different items.\n\nThe value and timestamp is clumped together and is accessed via `ValueAndTimestamp` _interface_. The underlying struct may be anything. Each item that you want to make the client handle timestamps for must implement this interface.\n\n.Example Model\n[source,go]\n----\ntype SensorValue struct {\n  ValueAndTimestamp\n  Timestamp time.Time `json:\"timestamp\"` // \u003c1\u003e\n  Value any `json:\"value\"` // \u003c2\u003e\n}\n\ntype Building struct {\n  Controller Controller `json:\"controller\"`\n}\n\ntype Controller struct {\n  ID string `json:\"id\"`\n  Serial string `json:\"serial\"`\n  Brand string `json:\"brand,omitempty\"`\n  Circuits map[int]Circuit `json:\"circuits,omitempty\"`\n}\n\ntype Circuit struct {\n   Senors map[string]SensorValue `json:\"sensors,omitempty\"` // \u003c3\u003e\n}\n----\n\u003c1\u003e This is the timestamp that the sensor value was read for this example, it is possible to have many different types as long as it implements the `ValueAndTimestamp` interface.\n\u003c2\u003e The value may be anything. If it is a map[string]any, it will compare each entry in the map to determine if it has changed or not. In that way it is possible to present a set of values that this sensor value represents.\n\u003c3\u003e Here all sensor values are stored as a map with the sensor name as the key and the value as the value. The value is a struct that implements the `ValueAndTimestamp` interface.\n\n=== Creating or Updating the Device Shadow\n\nWhen writing to the device shadow, for example _Report_, the _SDK_ will read the whole document and marshal it to the registered model. For example `Building` it will iterate all the fields and check if they implement the `ValueAndTimestamp` interface. If they do, it will use it to check if the client model is newer than the device shadow model. If it is, the client model value will be kept, if older, the device shadow model value will be copied to the client model.\n\nIf any field is missing in the client model but present in the shadow model, it will be added to the client model. If any field is present in the client model but not in the shadow model, it will be kept (se _Deleting an Element_ for the options).\n\nWhen done it will write the loaded it back conditionally on version and increment the version (atomically). This is done with an updated timestamp of `time.Now.UTC().UnixNano()`. If the client supplied a `ClientToken` string, it will be added to the shadow as well.\n\nOn conflict, the client will read the shadow again and redo the merge and write it back again. After _n_ times it will give up and return an conflict error.\n\n=== Deleting an Element\n\nWhen iterating merging the structures there are two modes:  _ClientIsMaster_ and  _ServerIsMaster_.\n\nWhen _ClientIsMaster_ it will just check elements that are timestamped and exists on both models. If the server model value is newer, the value will be copied to the client model. Otherwise the client model will be kept as is.\n\nIf the _ServerIsMaster_ mode it will not allow the client to delete any property only, add, update or keep values are possible.\n\nIn both modes, all values that do not implement `ValueAndTimestamp` are just used as is on the client model to write the device shadow (i.e. always overwritten without any timestamp handling).\n\nWhen _ServerIsMaster_ it is not possible to delete elements only add and updates are possible from the client model.\n\n=== Desired State\n\nThis is to denote the desired state and when the client wants to report a state it may also include that the _SDK_ shall load the desired state and clear it when the desired state value are the same as reported.\n\nWhen a value in the desired state is acknowledged (matches a value in the reported state), it may be either completely removed from the desired model or set to zero values, depending on the implementation. This behavior ensures that desired values that have been acknowledged are effectively cleared, making it easy to identify values that still need to be addressed.\n\nIn this case it will need to do this in a transaction since it is two different sort keys. For example in DynamoDB this is done using the transaction _API_.\n\n== Development\n\n=== Submodules\n\nWhen a plugin needs to have a external dependency it is *REQUIRED* that it will be it's own module in order to have the core framework free from other dependencies that the go framework and the test framework.\n\nAdd the `Makefile` to do versioning see https://github.com/mariotoffia/godeviceshadow/blob/main/examples/Makefile[Makefile] and copy the _version_ target to allow for versioning of the plugin. Add the module in this readme under the \u003c\u003cModules\u003e\u003e section so it is clear that this is a submodule that may be referenced in a external project (or this).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmariotoffia%2Fgodeviceshadow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmariotoffia%2Fgodeviceshadow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmariotoffia%2Fgodeviceshadow/lists"}