{"id":21251224,"url":"https://github.com/gvaliente/pocket-tensor","last_synced_at":"2025-07-11T01:31:59.412Z","repository":{"id":201436187,"uuid":"132698170","full_name":"GValiente/pocket-tensor","owner":"GValiente","description":"Run Keras models from a C++ application on embedded devices","archived":false,"fork":false,"pushed_at":"2020-08-17T13:38:11.000Z","size":24220,"stargazers_count":66,"open_issues_count":2,"forks_count":10,"subscribers_count":11,"default_branch":"master","last_synced_at":"2023-10-19T20:15:12.889Z","etag":null,"topics":["cpp11","deep-learning","embedded","keras","machine-learning","simd"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GValiente.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-05-09T03:42:26.000Z","updated_at":"2023-10-19T21:17:25.130Z","dependencies_parsed_at":null,"dependency_job_id":"7e1d477c-4197-4f63-b781-a05597beba37","html_url":"https://github.com/GValiente/pocket-tensor","commit_stats":null,"previous_names":["gvaliente/pocket-tensor"],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GValiente%2Fpocket-tensor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GValiente%2Fpocket-tensor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GValiente%2Fpocket-tensor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GValiente%2Fpocket-tensor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GValiente","download_url":"https://codeload.github.com/GValiente/pocket-tensor/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225669674,"owners_count":17505363,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cpp11","deep-learning","embedded","keras","machine-learning","simd"],"created_at":"2024-11-21T03:41:06.011Z","updated_at":"2024-11-21T03:41:06.657Z","avatar_url":"https://github.com/GValiente.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pocket-tensor\n\npocket-tensor is an [arquolo's](https://github.com/arquolo) [Kerasify](https://github.com/moof2k/kerasify) fork designed for running trained Keras models from a C++ application on embedded devices.\n\n## Discontinued\n\nThis project is not in development anymore.\n\nIf you want to run trained Keras models from a C++ application, try [TensorFlow Lite](https://www.tensorflow.org/lite) and [frugally-deep](https://github.com/Dobiasd/frugally-deep).\n\n## Design goals\n\n* Compatibility with sequential networks generated by Keras 2.x using Tensorflow backend.\n* Multithread CPU support.\n* Low RAM usage.\n* Easy to build and run (no external dependencies).\n* Fast build times.\n\n## Improvements over Kerasify\n\n* Thanks to the awesome [libsimdpp library](https://github.com/p12tic/libsimdpp), tensor operations have been rewritten using SIMD instructions to improve prediction performance.\n* Predictions run across multiple CPU cores.\n* Memory (re)usage has been improved in order to reduce memory allocations.\n* Apart from `float`, `double` precision tensors are supported (see `pt_tweakme.h` file).\n* Tensor dimensions are rigorously validated on each layer to avoid wrong models usage.\n* Besides GCC and Clang, Visual Studio compiler is properly supported.\n\n## Hardware requirements\n\nSince there's no GPU support, by default pocket-tensor requires the following CPU SIMD instruction sets:\n\n* ARM: NEON with floating point support.\n* x86: AVX.\n\nRequired SIMD instruction sets are specified in the `pt_tweakme.h` file, so they can be modified with ease.\n\n## Software requirements\n\nSince a copy of libsimdpp comes bundled with this library, there's no external dependencies required, so the only software requirements are a C++11-compatible compiler and CMake \u003e= 3.4.  \n\npocket-tensor has been tested with these compilers: \n\n* GCC 4.9.\n* MSVC 2017.\n* Whatever Clang comes with Apple LLVM 9.1.0.\n* Whatever Clang comes with Android Studio 3.1.3 (see Android section).\n\n## How to build\n\nA CMakeLists.txt is provided with this library, so in order to use it you only need to include this file in your CMake project.  \n\nTo build and run the unit tests, you need to generate them first:\n\n```\npython make_tests.py\nmkdir tests_build\ncd tests_build\ncmake -DPT_BUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Release ..\nmake\n./tests/pocket-tensor-tests\n```\n\n## Usage\n\n1) Use Keras to build (`model.compile(...)`) and train (`model.fit(...)`) your model as usual.\n\n2) Now convert it to the pocket-tensor file format with `pt.export_model(model, 'example.model')`.\n\n3) Finally load it in C++ (`pt::create(\"example.model\")`) and use `model-\u003epredict(...)` to perform a prediction with your data.\n\nThe following example shows the full workflow:\n\n```python\n# make_model.py:\n\nimport numpy as np\nfrom keras.models import Sequential\nfrom keras.layers import Dense\nfrom pt import export_model\n\ntest_x = np.random.rand(10, 10).astype('f')\ntest_y = np.random.rand(10).astype('f')\n\nmodel = Sequential()\nmodel.add(Dense(1, input_dim=10))\n\nmodel.compile(loss='mean_squared_error', optimizer='adamax')\nmodel.fit(test_x, test_y, epochs=1)\n\nprint model.predict(np.array([[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]]))\n\nexport_model(model, 'example.model')\n```\n\n```cpp\n// main.cpp:\n\n#include \u003ciostream\u003e\n#include \"pt_model.h\"\n#include \"pt_tensor.h\"\n\nint main()\n{\n    // Initialize model:\n    auto model = pt::Model::create(\"example.model\");\n    // REQUIRE(model);\n\n    // Create input tensor:\n    pt::Tensor in(10);\n    in.setData({0, 1, 2, 3, 4, 5, 6, 7, 8, 9});\n\n    // Run prediction:\n    pt::Tensor out;\n    bool success = model-\u003epredict(std::move(in), out);\n    // REQUIRE(success);\n\t\n    // Print output:\n    std::cout \u003c\u003c out \u003c\u003c std::endl;\n    return 0;\n}\n```\n\n## Supported layer types\n\nThe most common layer types used in image recognition and sequences prediction are supported, making many popular model architectures possible:\n\n* Core: `Input`, `Dense`, `Flatten`, `RepeatVector`.\n* Convolutional: `Conv1D`, `Conv2D`.\n* Pooling: `MaxPooling2D`, `GlobalMaxPooling2D`.\n* Locally-connected: `LocallyConnected1D`.\n* Recurrent: `LSTM`.\n* Embedding: `Embedding`.\n* Normalization: `BatchNormalization`.\n* Activations: `Linear`, `ReLU`, `ELU`, `SeLU`, `Softplus`, `Softsign`, `Tanh`, `Sigmoid`, `HardSigmoid`, `Softmax`.\n* Advanced activations: `LeakyReLU`, `ELU`.\n\n\n## Performance\n\nA benchmark application is included with this library. To build and run it:\n\n```\nmkdir benchmark_build\ncd benchmark_build\ncmake -DPT_BUILD_BENCHMARK=ON -DCMAKE_BUILD_TYPE=Release ..\nmake\n./benchmark/pocket-tensor-benchmark\n```\n\nThe prediction time of the following models has been measured on a PC with a Intel Core i7-6500U CPU @ 2.50GHz and on a Raspberry Pi 3:\n\n### MNIST CNN\n\n```python\nmodel = Sequential()\nmodel.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=(28, 28, 1)))\nmodel.add(Conv2D(64, (3, 3), activation='relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\nmodel.add(Dropout(0.25))\nmodel.add(Flatten())\nmodel.add(Dense(128, activation='relu'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(10, activation='sigmoid'))\n```\n\n| Library            | PC elapsed time (μs) | RPi3 elapsed time (μs) |\n| ------------------ | -------------------: | ---------------------: |\n| Keras              |                 1470 |                  23363 |\n| arquolo's Kerasify |                 3502 |                  64238 |\n| frugally-deep      |                 1402 |                  29298 |\n| pocket-tensor      |                 1049 |                  27329 |\n\n### IMDB LSTM\n\n```python\nmodel = Sequential()\nmodel.add(Embedding(20000, 128))\nmodel.add(LSTM(128, return_sequences=True, dropout=0.2, recurrent_dropout=0.2))\nmodel.add(LSTM(128, return_sequences=False, dropout=0.2, recurrent_dropout=0.2))\nmodel.add(Dense(1, activation='sigmoid'))\n```\n\n| Library            | PC elapsed time (μs) | RPi3 elapsed time (μs) |\n| ------------------ | -------------------: | ---------------------: |\n| Keras              |                10160 |                  89344 |\n| arquolo's Kerasify |                 5378 |                  79060 |\n| frugally-deep      |        Not supported |          Not supported |\n| pocket-tensor      |                 3314 |                  67115 |\n\n## Android\n\npocket-tensor supports Android apps (armeabi-v7a ABI only).\n\nTo add pocket-tensor to an Android project with C++ support, you must:\n\n1) Enable ARM NEON instructions on the build.gradle project file (https://developer.android.com/ndk/guides/cmake):\n\n```\nandroid {\n    ...\n    defaultConfig {\n        ...\n        externalNativeBuild {\n            cmake {\n                arguments \"-DANDROID_ARM_NEON=TRUE\"\n            }\n        }\n    }\n}\n```\n\n2) Disable all ABIs except armeabi-v7a on the build.gradle project file (https://developer.android.com/studio/build/configure-apk-splits):\n\n```\nandroid {\n    ...\n    splits {\n        abi {\n            enable true\n            reset()\n            include \"armeabi-v7a\"\n        }\n    }\n}\n```\n\n3) Include pocket-tensor on the CMakeLists.txt file of your native library:\n\n```\nadd_subdirectory(/path/to/pocket-tensor pocket-tensor)\ntarget_link_libraries(native-lib pocket-tensor)\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgvaliente%2Fpocket-tensor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgvaliente%2Fpocket-tensor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgvaliente%2Fpocket-tensor/lists"}