Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/uber/neuropod
A uniform interface to run deep learning models from multiple frameworks
https://github.com/uber/neuropod
deep-learning deeplearning incubation inference keras machine-learning machinelearning pytorch tensorflow
Last synced: 3 months ago
JSON representation
A uniform interface to run deep learning models from multiple frameworks
- Host: GitHub
- URL: https://github.com/uber/neuropod
- Owner: uber
- License: apache-2.0
- Created: 2019-01-23T19:22:22.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2024-01-03T06:09:01.000Z (10 months ago)
- Last Synced: 2024-06-20T05:49:17.618Z (5 months ago)
- Topics: deep-learning, deeplearning, incubation, inference, keras, machine-learning, machinelearning, pytorch, tensorflow
- Language: C++
- Homepage: https://neuropod.ai
- Size: 3.19 MB
- Stars: 932
- Watchers: 26
- Forks: 77
- Open Issues: 54
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
- awesome-robotic-tooling - neuropod - Neuropod is a library that provides a uniform interface to run deep learning models from multiple frameworks in C++ and Python. (Sensor Processing / Machine Learning)
README
# Neuropod
## What is Neuropod?
[Neuropod](https://github.com/uber/neuropod) is a library that provides a uniform interface to run deep learning models from multiple frameworks in C++ and Python. Neuropod makes it easy for researchers to build models in a framework of their choosing while also simplifying productionization of these models.
It currently supports TensorFlow, PyTorch, TorchScript, Keras and [Ludwig](http://ludwig.ai).
For more information:
- [Uber Engineering blog post introducing Neuropod](https://eng.uber.com/introducing-neuropod/)
- [Talk at NVIDIA GTC Spring 2021](https://www.nvidia.com/en-us/on-demand/session/gtcspring21-s31643/)
## Why use Neuropod?
#### Run models from any supported framework using one API
Running a TensorFlow model looks exactly like running a PyTorch model.
```py
x = np.array([1, 2, 3, 4])
y = np.array([5, 6, 7, 8])
for model_path in [TF_ADDITION_MODEL_PATH, PYTORCH_ADDITION_MODEL_PATH]:
# Load the model
neuropod = load_neuropod(model_path)
# Run inference
results = neuropod.infer({"x": x, "y": y})
# array([6, 8, 10, 12])
print results["out"]
```
See the [tutorial](https://neuropod.ai/tutorial/), [Python guide](https://neuropod.ai/pyguide/), or [C++ guide](https://neuropod.ai/cppguide/) for more examples.
Some benefits of this include:
- All of your inference code is framework agnostic.
- You can easily switch between deep learning frameworks if necessary without changing runtime code.
- Avoid the learning curve of using the C++ libtorch API and the C/C++ TF API
Any Neuropod model can be run from both C++ and Python (even PyTorch models that have not been converted to TorchScript).
#### Define a Problem API
This lets you focus more on the problem you're solving rather than the framework you're using to solve it.
For example, if you define a problem API for 2d object detection, any model that implements it can reuse all the existing inference code and infrastructure for that problem.
```py
INPUT_SPEC = [
# BGR image
{"name": "image", "dtype": "uint8", "shape": (1200, 1920, 3)},
]
OUTPUT_SPEC = [
# shape: (num_detections, 4): (xmin, ymin, xmax, ymax)
# These values are in units of pixels. The origin is the top left corner
# with positive X to the right and positive Y towards the bottom of the image
{"name": "boxes", "dtype": "float32", "shape": ("num_detections", 4)},
# The list of classes that the network can output
# This must be some subset of ['vehicle', 'person', 'motorcycle', 'bicycle']
{"name": "supported_object_classes", "dtype": "string", "shape": ("num_classes",)},
# The probability of each class for each detection
# These should all be floats between 0 and 1
{"name": "object_class_probability", "dtype": "float32", "shape": ("num_detections", "num_classes")},
]
```
This lets you
- Build a single metrics pipeline for a problem
- Easily compare models solving the same problem (even if they're in different frameworks)
- Build optimized inference code that can run any model that solves a particular problem
- Swap out models that solve the same problem at runtime with no code change (even if the models are from different frameworks)
- Run fast experiments
See the [tutorial](https://neuropod.ai/tutorial/) for more details.
#### Build generic tools and pipelines
If you have several models that take in a similar set of inputs, you can build and optimize one framework-agnostic input generation pipeline and share it across models.
#### Other benefits
- Fully self-contained models (including custom ops)
- [Efficient zero-copy operations](https://neuropod.ai/advanced/efficient_tensor_creation/)
- [Tested on](https://neuropod.ai/developing/#build-matrix) platforms including
- Mac, Linux, Linux (GPU)
- Four or five versions of each supported framework
- Five versions of Python
- Model isolation with [out-of-process execution](https://neuropod.ai/advanced/ope/)
- Use multiple different versions of frameworks in the same application
- Ex: Experimental models using Torch nightly along with models using Torch 1.1.0
- Switch from running in-process to running out-of-process with [one line of code](https://neuropod.ai/advanced/ope/)
## Getting started
See the [basic introduction tutorial](https://neuropod.ai/tutorial/) for an overview of how to get started with Neuropod.
The [Python guide](https://neuropod.ai/pyguide/) and [C++ guide](https://neuropod.ai/cppguide/) go into more detail on running Neuropod models.