Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/nod-ai/shark-platform
SHARK Inference Modeling and Serving
https://github.com/nod-ai/shark-platform
Last synced: 7 days ago
JSON representation
SHARK Inference Modeling and Serving
- Host: GitHub
- URL: https://github.com/nod-ai/shark-platform
- Owner: nod-ai
- License: apache-2.0
- Created: 2024-04-20T22:16:54.000Z (7 months ago)
- Default Branch: main
- Last Pushed: 2024-10-31T23:39:09.000Z (12 days ago)
- Last Synced: 2024-11-01T00:18:40.012Z (12 days ago)
- Language: Python
- Size: 2.36 MB
- Stars: 11
- Watchers: 14
- Forks: 22
- Open Issues: 73
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# SHARK Modeling and Serving Libraries
**WARNING: This is an early preview that is in progress. It is not ready for
general use.**
[](https://github.com/pre-commit/pre-commit)
## Sub-projects
### [`sharktank/`](./sharktank/)
[](https://badge.fury.io/py/sharktank) [](https://github.com/nod-ai/SHARK-Platform/actions/workflows/ci-sharktank.yml?query=event%3Apush)
The SHARK Tank sub-project contains a collection of model recipes and
conversion tools to produce inference-optimized programs.
* See the [SHARK Tank Programming Guide](./docs/programming_guide.md) for
information about core concepts, the development model, dataset management,
and more.
* See [Direct Quantization with SHARK Tank](./docs/quantization.md)
for information about quantization support.
### [`shortfin/`](./shortfin/)
[](https://badge.fury.io/py/shortfin) [](https://github.com/nod-ai/SHARK-Platform/actions/workflows/ci_linux_x64-libshortfin.yml?query=event%3Apush)
The shortfin sub-project is SHARK's high performance inference library and
serving engine.
* API documentation for shortfin is available on
[readthedocs](https://shortfin.readthedocs.io/en/latest/).
### [`tuner/`](./tuner/)
[](https://github.com/nod-ai/SHARK-Platform/actions/workflows/ci-tuner.yml?query=event%3Apush)
The Tuner sub-project assists with tuning program performance by searching for
optimal parameter configurations to use during model compilation.
## Support matrix
### Models
Model name | Model recipes | Serving apps
---------- | ------------- | ------------
SDXL | [`sharktank/sharktank/models/punet/`](https://github.com/nod-ai/SHARK-Platform/tree/main/sharktank/sharktank/models/punet) | [`shortfin/python/shortfin_apps/sd/`](https://github.com/nod-ai/SHARK-Platform/tree/main/shortfin/python/shortfin_apps/sd)
llama | [`sharktank/sharktank/models/llama/`](https://github.com/nod-ai/SHARK-Platform/tree/main/sharktank/sharktank/models/llama) | [`shortfin/python/shortfin_apps/llm/`](https://github.com/nod-ai/SHARK-Platform/tree/main/shortfin/python/shortfin_apps/llm)
## Development getting started
Use this as a guide to get started developing the project using pinned,
pre-release dependencies. You are welcome to deviate as you see fit, but
these canonical directions mirror what the CI does.
### Setup a venv
We recommend setting up a virtual environment (venv). The project is configured
to ignore `.venv` directories, and editors like VSCode pick them up by default.
```
python -m venv --prompt sharktank .venv
source .venv/bin/activate
```
### Install PyTorch for your system
If no explicit action is taken, the default PyTorch version will be installed.
This will give you a current CUDA-based version. Install a different variant
by doing so explicitly first:
*CPU:*
```
pip install -r pytorch-cpu-requirements.txt
```
*ROCM:*
```
pip install -r pytorch-rocm-requirements.txt
```
### Install development packages
```
# Clone and install editable iree-turbine dep in deps/
pip install -f https://iree.dev/pip-release-links.html --src deps \
-e "git+https://github.com/iree-org/iree-turbine.git#egg=iree-turbine"
# Install editable local projects.
pip install -r requirements.txt -e sharktank/ shortfin/
```
### Running tests
```
pytest sharktank
pytest shortfin
```
### Optional: pre-commits and developer settings
This project is set up to use the `pre-commit` tooling. To install it in
your local repo, run: `pre-commit install`. After this point, when making
commits locally, hooks will run. See https://pre-commit.com/