https://github.com/neuralmagic/optimum-deepsparse

https://github.com/neuralmagic/optimum-deepsparse

Last synced: about 1 year ago
JSON representation

• Host: GitHub
• URL: https://github.com/neuralmagic/optimum-deepsparse
• Owner: neuralmagic
• License: apache-2.0
• Created: 2023-06-28T03:01:00.000Z (almost 3 years ago)
• Default Branch: main
• Last Pushed: 2024-01-15T19:23:25.000Z (over 2 years ago)
• Last Synced: 2025-04-23T09:12:02.872Z (about 1 year ago)
• Language: Python
• Size: 375 KB
• Stars: 5
• Watchers: 11
• Forks: 0
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# optimum-deepsparse

Accelerated inference of 🤗 models on CPUs using the [DeepSparse Inference Runtime](https://github.com/neuralmagic/deepsparse).

[![DeepSparse Modeling / Python - Test](https://github.com/neuralmagic/optimum-deepsparse/actions/workflows/test_check.yaml/badge.svg)](https://github.com/neuralmagic/optimum-deepsparse/actions/workflows/test_check.yaml)

[![DeepSparse Modeling Nightly](https://github.com/neuralmagic/optimum-deepsparse/actions/workflows/test_nightly.yaml/badge.svg)](https://github.com/neuralmagic/optimum-deepsparse/actions/workflows/test_nightly.yaml)

## Install

There is a PyPi package published at [optimum-deepsparse](https://pypi.org/project/optimum-deepsparse/) so you can simply:

`pip install optimum-deepsparse`

Optimum DeepSparse is a fast-moving project, and you may want to install from source.

`pip install git+https://github.com/neuralmagic/optimum-deepsparse.git`

### Installing in developer mode

If you are working on the `optimum-deepsparse` code then you should use an editable install by cloning and installing `optimum` and `optimum-deepsparse`:

```

git clone https://github.com/huggingface/optimum

git clone https://github.com/neuralmagic/optimum-deepsparse

pip install -e optimum -e optimum-deepsparse

```

Now whenever you change the code, you'll be able to run with those changes instantly.

## How to use it?

To load a model and run inference with DeepSparse, you can just replace your `AutoModelForXxx` class with the corresponding `DeepSparseModelForXxx` class. 

```diff

import requests

from PIL import Image

- from transformers import AutoModelForImageClassification

+ from optimum.deepsparse import DeepSparseModelForImageClassification

from transformers import AutoFeatureExtractor, pipeline

url = "http://images.cocodataset.org/val2017/000000039769.jpg"

image = Image.open(requests.get(url, stream=True).raw)

model_id = "microsoft/resnet-50"

- model = AutoModelForImageClassification.from_pretrained(model_id)

+ model = DeepSparseModelForImageClassification.from_pretrained(model_id, export=True, input_shapes="[1,3,224,224]")

feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)

cls_pipe = pipeline("image-classification", model=model, feature_extractor=feature_extractor)

outputs = cls_pipe(image)

```

| Supported Task                              | Model Class |

| ------------------------------------------- | ------------- |

| "image-classification"                      | DeepSparseModelForImageClassification  |

| "text-classification"/"sentiment-analysis"  | DeepSparseModelForSequenceClassification  |

| "audio-classification"                      | DeepSparseModelForAudioClassification  |

| "question-answering"                        | DeepSparseModelForQuestionAnswering  |

| "image-segmentation"                        | DeepSparseModelForSemanticSegmentation  |

If you find any issue while using those, please open an issue or a pull request.