https://github.com/ibob/pytorch-ggml-plugin
A demo a ggml plugin for pytorch
https://github.com/ibob/pytorch-ggml-plugin
Last synced: 11 months ago
JSON representation
A demo a ggml plugin for pytorch
- Host: GitHub
- URL: https://github.com/ibob/pytorch-ggml-plugin
- Owner: iboB
- License: mit
- Created: 2023-10-09T13:17:29.000Z (almost 3 years ago)
- Default Branch: master
- Last Pushed: 2023-10-12T09:52:23.000Z (almost 3 years ago)
- Last Synced: 2025-02-14T14:55:59.422Z (over 1 year ago)
- Language: C++
- Size: 31.3 KB
- Stars: 3
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# PyTorch GGML Plugin
This is an example of how to create a [ggml](https://github.com/ggerganov/ggml) plugin for PyTorch.
Note that it relies on ggml features which are not in the main repo (yet). Most noitably:
* Instantiating a ggml cuda backend where the cuda device, stream handle and CUBLAS handle are provided externally
* Setting an external pointer to a ggml tensor, one that is not allocated and managed from a ggml buffer
These changes are required to use the ggml cuda backend and the data pointers from torch cuda tensors directly. Using the ggml cpu backend or copying the actual data between ggml and torch tensors will work with vanilla ggml.
[This is the PR](https://github.com/ggerganov/ggml/pull/570) which tracks the proposed changes from this repo.
## Structure
* `model/` - a static library which has a (trivial) ggml model
* `pytorch-plugin/` - a PyTorch plugin which exposes the module to a PyTorch app
* `pytorch-example.py` - an example of funning the model from the plugin
* `cpp-example.cpp` - an example of running the model as a standalone C++ executable using the static library
## License
[](https://opensource.org/licenses/MIT)
This software is distributed under the MIT Software License.
See accompanying file LICENSE or copy [here](https://opensource.org/licenses/MIT).
Copyright © 2023 [Borislav Stanimirov](http://github.com/iboB)