{"id":22695639,"url":"https://github.com/lakeraai/onnx_clip","last_synced_at":"2025-09-08T10:32:47.722Z","repository":{"id":92277304,"uuid":"570078750","full_name":"lakeraai/onnx_clip","owner":"lakeraai","description":"An ONNX-based implementation of the CLIP model that doesn't depend on torch or torchvision.","archived":false,"fork":false,"pushed_at":"2024-07-03T16:27:59.000Z","size":1971,"stargazers_count":60,"open_issues_count":0,"forks_count":7,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-12-23T12:42:10.404Z","etag":null,"topics":["clip","deep-learning","onnx","onnxruntime","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lakeraai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-11-24T09:39:09.000Z","updated_at":"2024-12-19T10:57:06.000Z","dependencies_parsed_at":null,"dependency_job_id":"504dd6b5-160e-4100-8dc8-c5402ec5db7c","html_url":"https://github.com/lakeraai/onnx_clip","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lakeraai%2Fonnx_clip","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lakeraai%2Fonnx_clip/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lakeraai%2Fonnx_clip/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lakeraai%2Fonnx_clip/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lakeraai","download_url":"https://codeload.github.com/lakeraai/onnx_clip/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":232302277,"owners_count":18502114,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clip","deep-learning","onnx","onnxruntime","pytorch"],"created_at":"2024-12-10T04:11:22.023Z","updated_at":"2025-01-03T06:47:55.332Z","avatar_url":"https://github.com/lakeraai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# onnx_clip\n\nAn [ONNX](https://onnx.ai/)-based implementation of [CLIP](https://github.com/openai/CLIP) that doesn't\ndepend on `torch` or `torchvision`.\nIt also has a friendlier API than the original implementation. \n\nThis works by\n- running the text and vision encoders (the ViT-B/32 variant) in [ONNX Runtime](https://onnxruntime.ai/)\n- using a pure NumPy version of the tokenizer\n- using a pure NumPy+PIL version of the [preprocess function](https://github.com/openai/CLIP/blob/3702849800aa56e2223035bccd1c6ef91c704ca8/clip/clip.py#L79).\n  The PIL dependency could also be removed with minimal code changes - see `preprocessor.py`.\n\n## Installation\nTo install, run the following in the root of the repository:\n```bash\npip install .\n```\n\n## Usage\n\nAll you need to do is call the `OnnxClip` model class. An example:\n\n```python\nfrom onnx_clip import OnnxClip, softmax, get_similarity_scores\nfrom PIL import Image\n\nimages = [Image.open(\"onnx_clip/data/franz-kafka.jpg\").convert(\"RGB\")]\ntexts = [\"a photo of a man\", \"a photo of a woman\"]\n\n# Your images/texts will get split into batches of this size before being\n# passed to CLIP, to limit memory usage\nonnx_model = OnnxClip(batch_size=16)\n\n# Unlike the original CLIP, there is no need to run tokenization/preprocessing\n# separately - simply run get_image_embeddings directly on PIL images/NumPy\n# arrays, and run get_text_embeddings directly on strings.\nimage_embeddings = onnx_model.get_image_embeddings(images)\ntext_embeddings = onnx_model.get_text_embeddings(texts)\n\n# To use the embeddings for zero-shot classification, you can use these two\n# functions. Here we run on a single image, but any number is supported.\nlogits = get_similarity_scores(image_embeddings, text_embeddings)\nprobabilities = softmax(logits)\n\nprint(\"Logits:\", logits)\n\nfor text, p in zip(texts, probabilities[0]):\n    print(f\"Probability that the image is '{text}': {p:.3f}\")\n```\n\n## Building \u0026 developing from source\n\n**Note**: The following may give timeout errors due to the filesizes. If so, this can be fixed with poetry version 1.1.13 - see [this related issue.](https://github.com/python-poetry/poetry/issues/6009)\n\n### Install, run, build and publish with Poetry\n\nInstall [Poetry](https://python-poetry.org/docs/)\n```\ncurl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -\n```\n\nTo setup the project and create a virtual environment run the following command from the project's root directory.\n```\npoetry install\n```\n\nTo build a source and wheel distribution of the library run the following command from the project's root directory.\n```\npoetry build\n```\n\n#### Publishing a new version to PyPI (for project maintainers)\n\nFirst, remove/move the downloaded LFS files, so that they're not packaged with the code.\nOtherwise, this creates a huge `.whl` file that PyPI refuses and it causes confusing errors.\n\nThen, follow [this guide](https://towardsdatascience.com/how-to-publish-a-python-package-to-pypi-using-poetry-aa804533fc6f).\ntl;dr: go to the [PyPI account page](https://pypi.org/manage/account/), generate an API token\nand put it into the `$PYPI_PASSWORD` environment variable. Then run\n```shell\npoetry publish --build --username lakera --password $PYPI_PASSWORD\n```\n\n## Help\n\nPlease let us know how we can support you: [earlyaccess@lakera.ai](mailto:earlyaccess@lakera.ai).\n\n## LICENSE\nSee the [LICENSE](./LICENSE) file in this repository.\n\nThe `franz-kafka.jpg` is taken from [here](https://www.knesebeck-verlag.de/franz_kafka/p-1/270).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flakeraai%2Fonnx_clip","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flakeraai%2Fonnx_clip","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flakeraai%2Fonnx_clip/lists"}