{"id":20841454,"url":"https://github.com/zephyrproject-rtos/hal_ethos_u","last_synced_at":"2025-05-08T22:10:43.968Z","repository":{"id":44536036,"uuid":"429145091","full_name":"zephyrproject-rtos/hal_ethos_u","owner":"zephyrproject-rtos","description":"HAL for Arm Ethos-U NPUs","archived":false,"fork":false,"pushed_at":"2024-11-20T17:09:32.000Z","size":611,"stargazers_count":5,"open_issues_count":0,"forks_count":7,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-05-08T08:44:04.292Z","etag":null,"topics":["hal","module"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zephyrproject-rtos.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null}},"created_at":"2021-11-17T17:47:08.000Z","updated_at":"2025-04-21T03:10:15.000Z","dependencies_parsed_at":"2023-01-19T19:00:46.835Z","dependency_job_id":null,"html_url":"https://github.com/zephyrproject-rtos/hal_ethos_u","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zephyrproject-rtos%2Fhal_ethos_u","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zephyrproject-rtos%2Fhal_ethos_u/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zephyrproject-rtos%2Fhal_ethos_u/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zephyrproject-rtos%2Fhal_ethos_u/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zephyrproject-rtos","download_url":"https://codeload.github.com/zephyrproject-rtos/hal_ethos_u/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253154975,"owners_count":21862622,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hal","module"],"created_at":"2024-11-18T01:20:09.386Z","updated_at":"2025-05-08T22:10:43.962Z","avatar_url":"https://github.com/zephyrproject-rtos.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Arm(R) Ethos(TM)-U core driver\n\nThis repository contains a device driver for the Arm(R) Ethos(TM)-U NPU.\n\n## Building\n\nThe source code comes with a CMake based build system. The driver is expected to\nbe cross compiled for any of the supported Arm Cortex(R)-M CPUs, which requires\nthe user to configure the build to match their system configuration.\n\n\nOne such requirement is to define the target CPU, normally by setting\n`CMAKE_SYSTEM_PROCESSOR`. **Note** that when using the toolchain files provided\nin [core_platform](https://git.mlplatform.org/ml/ethos-u/ethos-u-core-platform.git),\nthe variable `TARGET_CPU` must be used instead of `CMAKE_SYSTEM_PROCESSOR`.\n\nTarget CPU is specified on the form \"cortex-m\u003cnr\u003e\u003cfeatures\u003e\", for example:\n\"cortex-m55+nodsp+nofp\".\n\nSimilarly the target NPU configuration is\ncontrolled by setting `ETHOSU_TARGET_NPU_CONFIG`, for example \"ethos-u55-128\".\n\nThe build configuration can be defined either in the toolchain file or\nby passing options on the command line.\n\n```[bash]\n$ cmake -B build  \\\n    -DCMAKE_TOOLCHAIN_FILE=\u003ctoolchain\u003e \\\n    -DCMAKE_SYSTEM_PROCESSOR=cortex-m\u003cnr\u003e\u003cfeatures\u003e \\\n    -DETHOSU_TARGET_NPU_CONFIG=ethos-u\u003cnr\u003e-\u003cmacs\u003e\n$ cmake --build build\n```\n\nor when using toolchain files from [core_platform](https://git.mlplatform.org/ml/ethos-u/ethos-u-core-platform.git)\n\n```[bash]\n$ cmake -B build  \\\n    -DCMAKE_TOOLCHAIN_FILE=\u003ccore_platform_toolchain\u003e \\\n    -DTARGET_CPU=cortex-m\u003cnr\u003e\u003cfeatures\u003e \\\n    -DETHOSU_TARGET_NPU_CONFIG=ethos-u\u003cnr\u003e-\u003cmacs\u003e\n$ cmake --build build\n```\n\n## Driver APIs\n\nThe driver APIs are defined in `include/ethosu_driver.h` and the related types\nin `include/ethosu_types.h`. Inferences can be invoked in two manners:\nsynchronously or asynchronously. The two types of invocation can be freely mixed\nin a single application.\n\n### Synchronous invocation\n\nA typical usage of the driver can be the following:\n\n```[C]\n// reserve a driver to be used (this call could block until a driver is available)\nstruct ethosu_driver *drv = ethosu_reserve_driver();\n...\n// run one or more inferences\nint result = ethosu_invoke(drv,\n                           custom_data_ptr,\n                           custom_data_size,\n                           base_addr,\n                           base_addr_size,\n                           num_base_addr);\n...\n// release the driver for others to use\nethosu_release_driver(drv);\n```\n\n### Asynchronous invocation\n\nA typical usage of the driver can be the following:\n\n```[C]\n// reserve a driver to be used (this call could block until a driver is available)\nstruct ethosu_driver *drv = ethosu_reserve_driver();\n...\n// run one or more inferences\nint result = ethosu_invoke_async(drv,\n                                 custom_data_ptr,\n                                 custom_data_size,\n                                 base_addr,\n                                 base_addr_size,\n                                 num_base_addr,\n                                 user_arg);\n...\n// do some other work\n...\nint ret;\ndo {\n    // true = blocking, false = non-blocking\n    // ret \u003e 0 means inference not completed (only for non-blocking mode)\n    ret = ethosu_wait(drv, \u003ctrue|false\u003e);\n} while(ret \u003e 0);\n...\n// release the driver for others to use\nethosu_release_driver(drv);\n```\n\nNote that if `ethosu_wait` is invoked from a different thread and concurrently\nwith `ethosu_invoke_async`, the user is responsible to guarantee that\n`ethosu_wait` is called after a successful completion of `ethosu_invoke_async`.\nOtherwise `ethosu_wait` might fail and not actually wait for the inference\ncompletion.\n\n### Driver initialization\n\nIn order to use a driver it first needs to be initialized by calling the `init`\nfunction, which will also register the handle in the list of available drivers.\nA driver can be torn down by using the `deinit` function, which also removes the\ndriver from the list.\n\nThe correct mapping is one driver per NPU device. Note that the NPUs must have\nthe same configuration, indeed the NPU configuration can be only one, which is\ndefined at compile time.\n\n## Implementation design\n\nThe driver is structured in two main parts: the driver, which is responsible to\nprovide an unified API to the user; and the device part, which deals with the\ndetails at the hardware level.\n\nIn order to do its task the driver needs a device implementation. There could be\nmultiple device implementation for different hardware model and/or\nconfigurations. Note that the driver can be compiled to target only one NPU\nconfiguration by specializing the device part at compile time.\n\n## Data caching\n\nFor running the driver on Arm CPUs which are configured with data cache, certain\ncaution must be taken to ensure cache coherency. The driver expects that cache\nclean/flush has been done by the user application before being invoked. The\ndriver does provide a deprecated weakly linked function `ethosu_flush_dcache`\nthat could be overriden, causing the driver to cache flush/clean base pointers\nmarked in the flush mask before each inference. By default the flush mask is set\nto only clean the scratch base pointer containing RW data (IFM in particular).\nIt is recommended to not implement this function but have the user application\nmake sure that IFM data has been written to memory before invoking an inference\non the NPU.\n\nThe driver also exposes a weakly linked symbol for cache invalidation called\n`ethosu_invalidate_dcache`, that must be overriden when the data cache is used.\nAfter starting an inference on the NPU, the driver will call this function to\ninvalidate the base pointers marked in the invalidation mask. By defaults it\ninvalidates the scratch base pointer keeping RW data, to ensure cache coherency\nafter the inference is done. The invalidation call is done before waiting for\nthe NPU to finish the inference so that depending on the network, the cycles\nfor invalidating the cache may be completely hidden (the CPU performs cache\ninvalidation before yielding while waiting for the NPU to finish).\n\nMake sure that any base pointers marked for flush/invalidation is aligned to the\ncache line size of your CPU, typically 32 bytes. While not implemented, to the\nreally advanced user aiming for maximum performance, it is theoretically\npossible to tell the network compiler to align the IFM/OFM to cache line size,\nand modify the driver so that only OFM data is invalidated (and if left to the\ndriver, only IFM data is cache cleaned/flushed). Due to the uncertainty of\ntensor alignment, the driver only flushes/invalidates on base pointer level.\n\nBy default the cache flush- and invalidation mask is set to only mark the\ndefault scratch base pointer (base pointer 1). For maximum flexibility, the\ndriver provides a function to modify the cache flush/invalidate masks called\n`ethosu_set_basep_cache_mask`. This function sets the two 8 bit masks, one for\nflush and one for invalidate, where bit 0 corresponds to base pointer 0, bit 1\ncorresponds to base pointer 1 etc. See `include/ethosu_driver.h` for more\ninformation.\n\nAn example implementation for the weak functions, using CMSIS primitives could\nlook like below:\n\n```[C++]\nextern \"C\" {\n// Deprecated - recommended to flush/clean in application code\n// p must be 32 byte aligned\nvoid ethosu_flush_dcache(uint32_t *p, size_t bytes) {\n    SCB_CleanDCache_by_Addr(p, bytes);\n}\n\n// p must be 32 byte aligned\nvoid ethosu_invalidate_dcache(uint32_t *p, size_t bytes) {\n    SCB_InvalidateDCache_by_Addr(p, bytes);\n}\n}\n```\nThe NPU contain memory attributes that should be set to match the settings used\nin the MPU configuration for the memories used. See `NPU_MEM_ATTR_[0-3]` for\nEthos-U85 and the `AXI_LIMIT[0-3]_MEM_TYPE` for Ethos-U55/Ethos-U65 in\ncorresponding `src/ethosu_config_uX5.h` files.\n\n## Mutex and semaphores\n\nTo ensure the correct functionality of the driver mutexes and semaphores are\nused internally. The default implementations of mutexes and semaphores are\ndesigned for a single-threaded baremetal environment. Hence for integration in\nenvironemnts where multi-threading is possible, e.g., RTOS, the user is\nresponsible to provide implementation for mutexes and semaphores to be used by\nthe driver.\n\nThe mutex and semaphores are used as synchronisation mechanisms and unless\nspecified, the timeout is required to be 'forever'.\n\nThe driver allows for an RTOS to set a timeout for the NPU interrupt semaphore.\nThe timeout can be set with the CMake variable `ETHOSU_INFERENCE_TIMEOUT`, which\nis then used as `timeout` argument for the interrupt semaphore take call. Note\nthat the unit is implementation defined, the value is shipped as is to the\n`ethosu_semaphore_take()` function and an override implementation should cast it\nto the appropriate type and/or convert it to the unit desired.\n\nA macro `ETHOSU_SEMAPHORE_WAIT_FOREVER` is defined in the driver header file,\nand should be made sure to map to the RTOS' equivalent of\n'no timeout/wait forever'. Inference timeout value defaults to this if left\nunset. The macro is used internally in the driver for the available NPU's, thus\nthe driver does NOT support setting a timeout other than forever when waiting\nfor an NPU to become available (global ethosu_semaphore).\n\nThe mutex and semaphore APIs are defined as weak linked functions that can be\noverridden by the user. The APIs are the usual ones and described below:\n\n```[C]\n// create a mutex by returning back a handle\nvoid *ethosu_mutex_create(void);\n// lock the given mutex\nint ethosu_mutex_lock(void *mutex);\n// unlock the given mutex\nint ethosu_mutex_unlock(void *mutex);\n\n// create a (binary) semaphore by returning back a handle\nvoid *ethosu_semaphore_create(void);\n// take from the given semaphore, accepting a timeout (unit impl. defined)\nint ethosu_semaphore_take(void *sem, uint64_t timeout);\n// give from the given semaphore\nint ethosu_semaphore_give(void *sem);\n```\n\n## Begin/End inference callbacks\n\nThe driver provide weak linked functions as hooks to receive callbacks whenever\nan inference begins and ends. The user can override such functions when needed.\nTo avoid memory leaks, any allocations done in the ethosu_inference_begin() must\nbe balanced by a corresponding free of the memory in the ethosu_inference_end()\ncallback.\n\nThe end callback will always be called if the begin callback has been called,\nincluding in the event of an interrupt semaphore take timeout.\n\n```[C]\nvoid ethosu_inference_begin(struct ethosu_driver *drv, void *user_arg);\nvoid ethosu_inference_end(struct ethosu_driver *drv, void *user_arg);\n```\n\nNote that the `void *user_arg` pointer passed to invoke() function is the same\npointer passed to the begin() and end() callbacks. For example:\n\n```[C]\nvoid my_function() {\n    ...\n    struct my_data data = {...};\n    int result = int ethosu_invoke_v3(drv,\n                                  custom_data_ptr,\n                                  custom_data_size,\n                                  base_addr,\n                                  base_addr_size,\n                                  num_base_addr,\n                                  (void *)\u0026data);\n    ....\n}\n\nvoid ethosu_inference_begin(struct ethosu_driver *drv, void *user_arg) {\n        struct my_data *data = (struct my_data*) user_arg;\n        // use drv and data here\n}\n\nvoid ethosu_inference_end(struct ethosu_driver *drv, void *user_arg) {\n        struct my_data *data = (struct my_data*) user_arg;\n        // use drv and data here\n}\n```\n\n## License\n\nThe Arm Ethos-U core driver is provided under an Apache-2.0 license. Please see\n[LICENSE.txt](LICENSE.txt) for more information.\n\n## Contributions\n\nThe Arm Ethos-U project welcomes contributions under the Apache-2.0 license.\n\nBefore we can accept your contribution, you need to certify its origin and give\nus your permission. For this process we use the Developer Certificate of Origin\n(DCO) V1.1 (https://developercertificate.org).\n\nTo indicate that you agree to the terms of the DCO, you \"sign off\" your\ncontribution by adding a line with your name and e-mail address to every git\ncommit message. You must use your real name, no pseudonyms or anonymous\ncontributions are accepted. If there are more than one contributor, everyone\nadds their name and e-mail to the commit message.\n\n```[]\nAuthor: John Doe \\\u003cjohn.doe@example.org\\\u003e\nDate:   Mon Feb 29 12:12:12 2016 +0000\n\nTitle of the commit\n\nShort description of the change.\n\nSigned-off-by: John Doe john.doe@example.org\nSigned-off-by: Foo Bar foo.bar@example.org\n```\n\nThe contributions will be code reviewed by Arm before they can be accepted into\nthe repository.\n\nIn order to submit a contribution push your patch to\n`ssh://\u003cGITHUB_USER_ID\u003e@review.mlplatform.org:29418/ml/ethos-u/ethos-u-core-driver`.\nTo do this you will need to sign-in to\n[review.mlplatform.org](https://review.mlplatform.org) using a GitHub account\nand add your SSH key under your settings. If there is a problem adding the SSH\nkey make sure there is a valid email address in the Email Addresses field.\n\n## Security\n\nPlease see [Security](SECURITY.md).\n\n## Trademark notice\n\nArm, Cortex and Ethos are registered trademarks of Arm Limited (or its\nsubsidiaries) in the US and/or elsewhere.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzephyrproject-rtos%2Fhal_ethos_u","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzephyrproject-rtos%2Fhal_ethos_u","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzephyrproject-rtos%2Fhal_ethos_u/lists"}