{"id":20713218,"url":"https://github.com/linaro/step","last_synced_at":"2025-04-23T08:04:02.111Z","repository":{"id":37102560,"uuid":"376259511","full_name":"Linaro/step","owner":"Linaro","description":"Secure data pipeline proof of concept module for Zephyr RTOS","archived":false,"fork":false,"pushed_at":"2023-09-19T13:50:59.000Z","size":7768,"stargazers_count":11,"open_issues_count":2,"forks_count":4,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-23T08:03:43.925Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Linaro.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-06-12T10:17:34.000Z","updated_at":"2023-12-05T23:13:49.000Z","dependencies_parsed_at":"2024-02-03T20:49:30.089Z","dependency_job_id":null,"html_url":"https://github.com/Linaro/step","commit_stats":null,"previous_names":["microbuilder/linaro_sensor_pipeline"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Linaro%2Fstep","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Linaro%2Fstep/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Linaro%2Fstep/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Linaro%2Fstep/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Linaro","download_url":"https://codeload.github.com/Linaro/step/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250395279,"owners_count":21423400,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T02:24:00.565Z","updated_at":"2025-04-23T08:04:02.091Z","avatar_url":"https://github.com/Linaro.png","language":"C","readme":".. _secure_telemetry_pipeline:\n\nSecure Telemetry Pipeline (STeP) for Zephyr\n###########################################\n\nOverview\n********\n\nSTeP is an experimental asychronous data processing pipeline that enables\ntelemetry data to be represented in a generic manner (`measurements`), and\nprocessed via one or more **processor nodes** in pipelines called\n**node chains**.\n\nProcessor nodes can be thought of as mini telemetry applications that can be\nchained together to represent complex data processing workflows.\n\nProcessor nodes and node-chains are added to a runtime-updateable\n**node registry**. Incoming measurements will be evaluated againt registered\nand active processor nodes to determine if there is a **filter match** between\nthe measurement and the first record in the node chain. If a match occurs, the\nmeasurement will be processed by the matching node or node chain.\n\n**NOTE**: STeP is NOT intended to replace Zephyr's sensor API or sensor\ndrivers, but to act as an endpoint for that raw telemetry data!\n\nPresentation\n************\n\nSTeP was presented as a proof of concept at Linaro Connect Fall 2021\nin LVC21F-303. This slidedeck provides more detailed information on STeP as of\nthe date that presentation was given.\n\nClick the image below to see a PDF of the slidedeck:\n\n.. image:: doc/LVC21F-303.png\n   :target: doc/LVC21F-303%20Secure%20Sensor%20Data%20Pipeline.pdf\n\nStatus\n******\n\nThis project is a proof of concept and a work in progress. It is intended to\ntest the concept of a flexible telemetry pipeline that can handle common\nsecurity requirements such as sanitising, hashing, signing, encrypting and\nencoding telemetry data, reducing individual operations to reusable\nbuilding blocks or mini applications.\n\nEventually, some of these nodes, or the entire system, should be allocated to\na trusted execution environment (TF-M, etc.), and linked to an inferrence\nengine for demonstration purposes. As a first step, however, implementing this\nas a Zephyr module allows for a flexible CI workflow, easy emulation in QEMU,\nand rapid development and testing.\n\nAdding STeP to your project via ``west``\n****************************************\nThe recommended way is to use ``west`` to initialize this repository directly and\nall its dependencies:\n\n.. code-block:: console\n\n   $ west init -m git@github.com:Linaro/step.git \n   $ west update\n\nAlternatively you can add a local copy of this module by adding the following sections\nto ``zephyr/west.yml``:\n\n1. In the ``manifest/remotes`` section add:\n\n.. code-block::\n\n   remotes:\n     - name: linaro\n       url-base: https://github.com/Linaro\n\n2. In the ``manifest/projects`` section add:\n\n.. code-block::\n\n   - name: step\n     remote: linaro\n     path: modules/lib/step\n     revision: main\n\n3. Save the file, and run ``west update`` from the project root to retrieve the\nlatest version of the library from Github, or whatever ``revision`` was\nspecified above.\n\nBuilding and Running\n********************\n\nThis application can be built and executed on QEMU as follows:\n\n.. code-block:: console\n\n   $ west build -p auto -b qemu_cortex_m3 modules/lib/step/samples/shell/ -t run\n\nTo build for another board, change ``qemu_cortex_m3`` above to that board's\nname, and remove the ``-t run`` appendix.\n\nSample Output\n=============\n\n.. code-block:: console\n\n   *** Booting Zephyr OS build zephyr-v2.6.0-536-g89212a7fbf5f  ***\n   Type 'step help' for command options.\n   \n   1.) Populate the processor registry: step add\n   2.) Publish measurement(s):          step pub\n   3.) Check results:                   step stats\n   \n   uart:~$ step add\n   [00:01:42.089,072] \u003cdbg\u003e proc_mgr: step_pm_register: Registering node/chain (handle 0, pri 4)\n   [00:01:42.089,125] \u003cdbg\u003e step_shell: step_shell_cmd_add: Took 53750 ns\n   uart:~$ step pub\n   [00:05:38.661,550] \u003cdbg\u003e step_shell: step_shell_cmd_pub: Published 1 measurement\n   [00:05:38.661,580] \u003cdbg\u003e step_shell: step_shell_cmd_pub: Took 60916 ns, excluding polling thread processing time (run 'step list').\n   [00:05:38.771,241] \u003cinf\u003e step_shell: [7496940] Received die temp: 32.00 C (handle 0:0)\n   [00:05:38.771,266] \u003cinf\u003e step_shell: cfg: mult by 10.00 (handle 0:1)\n   [00:05:38.771,289] \u003cinf\u003e step_shell: [7496940] Received die temp: 320.00 C (handle 0:1)\n   [00:05:38.771,316] \u003cinf\u003e step_shell: [7496940] Received die temp: 320.00 C (handle 0:2)\n   uart:~$ step stats\n   init:     3\n   evaluate: 0\n   matched:  1\n   start:    1\n   run:      3\n   stop:     1\n   error:    0\n\nExit QEMU by pressing :kbd:`CTRL+A` :kbd:`x`.\n\nRunning Unit Tests\n==================\n\nTo run the unit tests for this module, you can run ``twister`` via:\n\n.. code-block:: console\n\n   $ cd $ZEPHYR_BASE\n   $ twister --inline-logs \\\n     -p qemu_cortex_m3 \\\n     -T ../modules/lib/step/tests\n\nBasic Architecture\n******************\n\nThe Secure Telemetry Pipeline (STeP) aims to implement an extensible workflow\nto process generic sensor data (**measurements**) in a content-agnostic manner.\n\nIn theory, any type of measurement, using any standard SI unit and scale, and\nrepresented in any standard C type should be expressable in a relatively\nlight-weight manner, keeping in mind the memory constraints of small embedded\nsystems.\n\nProcessing of measurements happens based on one or more **processor nodes**,\nwhich can be chained together for more complex operations.\n\nProcessor nodes have an optional filter mechanism to indicate which types of\nmeasurements they process, allowing for processing workflows to be defined on\na per-measurement-type basis.\n\nThe **Secure** in STeP comes from the goal to provide basic secure processor\nnodes out of the box, implementing common operations like: hash, sign,\ncompress, encrypt, etc.\n\nA high-level overview of the system is shown here:\n\n.. raw:: html\n\n   \u003cp align=\"center\"\u003e\n     \u003cimg src=\"doc/Overview.png\" align=\"center\" alt=\"Basic Workflow\"\u003e\n   \u003c/p\u003e\n\nMeasurement Values\n==================\n\nMeasurements are the main component in STeP, and traverse the system starting\nas inputs from a data source, are processed, and output to an appropriate\ndata endpoint.\n\nSTeP attempts to compromise between optimising for memory in small embedded\nsystems, and trying to describe exactly what this measurement represents in as\nexpressive a manner as possible. It aims to balance the ability to precisely\nrepresent the exact meaning of the measurement, without wasting precious memory\non that representation.\n\nMeasurements make use of the following header, with a 12-byte overhead:\n\n::\n\n      3                   2                   1\n    1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0\n   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n   |              Flags            |  Ext. M Type  |  Base M Type  | \u003c- Filter\n   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n   |     C Type    | Scale Factor  |         SI Unit Type          | \u003c- Unit\n   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n   |   Source ID   | S Cnt | V | F |        Payload Length         | \u003c- SrcLen\n   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n   |                      Timestamp (optional)                     |\n   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n   |                                                               |\n   |                            Payload                            |\n   |                                                               |\n   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n   \n              1\n    5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0\n   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n   | Res | TSt | CMP | Encod |  DF | \u003c- Flags\n   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n       |    |     |      |      |\n       |    |     |      |      +-------- Data Format (CBOR, etc.)\n       |    |     |      +--------------- Encoding (BASE64, BASE45, etc.)\n       |    |     +---------------------- Compression (LZ4, etc.)\n       |    +---------------------------- Timestamp\n       +--------------------------------- Reserved (version flag?)\n\nFor futher technical details, see ``ìnclude/step/measurement.h``, but a\nhigh-level summary of these three key words is shown below:\n\nFilter\n------\n\nThe **Filter** word allows processor nodes to determine if this measurement\ninterests them or not.\n\nIt consists of an 8-bit **Base Measurement Type**, and an optional 8-bit\n**Extended Measurement Type**, which can be used to specialise the meaning of\nthe base type.\n\nEXAMPLE: ``STEP_MES_TYPE_LIGHT`` is a base type, which uses a default\nSI unit of ``STEP_MES_UNIT_SI_LUX``. If we wish to represent a different\nmeasurement in the same measurement family (base type), we could indicate\n``STEP_MES_EXT_TYPE_LIGHT_RADIO_RADIANCE`` as the extended type, which\nrepresents a radiometric measurement based on W/(sr m^2).\n\nThe **Flags** field indicates other important data about this measurement\npacket, such as how the data has been formatted, encoded, what compression\nalgorithm has been used (if any), and if a timestamp is present.\n\nUnit\n----\n\nThe **Unit** word describes the SI unit and optional scale factor this\nmeasurement uses, as well as how that unit is represented in memory. A 32-bit\nfloating-point value may use less memory in most cases, but we may require the\nadditional range and precision a 64-bit float provides. The ``unit`` word\nallows for a flexible expression of this information on a per-measurement basis,\nwithout an excessive amount of overhead.\n\nStandard SI units, scale factors and C types are all represented via enums in\nSTeP in the ``include/step/measurement`` folder.\n\nSrcLen\n------\n\nThe **Source/Len** word describes the size of the payload, with an option to\nspread larger payloads over multiple packets.\n\nThe vector size field can be set to indicate that individual samples are\nvectors (versus scalars), consisting of 2, 3 or 4 components. This covers\nthe most common vector representations: XY coordinates (2), XYZ vectors (3),\nquaternions (4), etc.\n\nIt also indicates the number of samples present in this measurement payload,\nin steps of power of two (2, 4, 8, 16, 32, etc., samples). This allows for\nbetter use of system resources by hashing, signing and encrypting larger sets\nof data, with only one 12-byte header as additional memory overhead. The 4-bits\nreserved to indicate that multiple samples are present allows for between 2 and\n16384 samples to be stored in the payload (2^n), or an arbitrary value:\n\n::\n\n   0 = 1 sample (default)     8 = 256 samples\n   1 = 2 samples              9 = 512 samples\n   2 = 4 samples              10 = 1024 samples\n   3 = 8 sammples             11 = 2048 samples\n   4 = 16 samples             12 = 4096 samples\n   5 = 32 samples             13 = 8192 samples\n   6 = 64 samples             14 = 16384 samples\n   7 = 128 samples            15 = Arbitrary (see below)\n\nIf the sample count is set to 15 (0xF), the number of samples should be\nindicated via an unsigned 32-bit integer in little-endian format at the start\nof the payload, but AFTER the optional timestamp (if present).\n\nThis word also contains an 8-bit **Source ID** field, which allows the\nmeasurement value's source to be identified to retrieve further information\nabout the source device out of band, such as it's min/max values, sample rate,\ngain setting, etc.\n\nMeasurement Memory Management\n=============================\n\nIn order to minimize endless memcpy operations, and deal with variable length\nmeasurements, all ``step_measurement`` records are allocated from a central\nheap memory block managed by the **sample pool manager**.\n\nAllocating and freeing memory imposes a certain amount of rigor on behalf of\nthe developper, and heap memory fragmentation may be an issue over time, but\nat present this seems like the best tradoff for an initial proof of concept.\n\nThe allocation, population, consumption and release of the measurement packet\nis describe in the sequence diagram below:\n\n.. raw:: html\n\n   \u003cp align=\"center\"\u003e\n     \u003cimg src=\"doc/SamplePool.png\" align=\"center\" alt=\"Sample Pool Memory Management\"\u003e\n   \u003c/p\u003e\n\nFilter Engine\n=============\n\nThe **processor manager** makes uses of the ``.filter`` word in measurements to\noptionally determine if registered filter nodes should or shouldn't process\nthe incoming measurement value(s). \n\nIf the processor node's filter chain is set to ``NULL`` (default), it will\naccept all incoming measurements. If one or more filters are indicated for the\nprocessor node, the filter engine will evaluate the measurement's filter fields\nagainst the processor node's filter value(s), to determine if there is a match.\n\nThis evaluation process introduces some overhead, which can be addressed by\nenabling **filter caching**, which works as follows:\n\n.. raw:: html\n\n   \u003cp align=\"center\"\u003e\n     \u003cimg src=\"doc/FilterEngineCache.png\" align=\"center\" alt=\"Filter Engine Caching\"\u003e\n   \u003c/p\u003e\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinaro%2Fstep","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flinaro%2Fstep","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinaro%2Fstep/lists"}