{"id":19986438,"url":"https://github.com/bentoml/clip-api-service","last_synced_at":"2025-05-04T07:31:21.273Z","repository":{"id":183673947,"uuid":"629677902","full_name":"bentoml/CLIP-API-service","owner":"bentoml","description":"CLIP as a service - Embed image and sentences, object recognition, visual reasoning, image classification and reverse image search","archived":false,"fork":false,"pushed_at":"2024-01-15T16:26:41.000Z","size":968,"stargazers_count":52,"open_issues_count":1,"forks_count":4,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-10-29T01:28:01.645Z","etag":null,"topics":["ai-applications","clip","cloud-native","mlops","model-inference","model-inference-service","model-serving","openai-clip"],"latest_commit_sha":null,"homepage":"https://bentoml.com","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bentoml.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-04-18T20:00:38.000Z","updated_at":"2024-10-24T03:50:29.000Z","dependencies_parsed_at":"2023-10-11T23:30:22.065Z","dependency_job_id":"08958b37-5fdb-4f2d-a437-40023ff99a12","html_url":"https://github.com/bentoml/CLIP-API-service","commit_stats":{"total_commits":26,"total_committers":4,"mean_commits":6.5,"dds":"0.42307692307692313","last_synced_commit":"fa999cbf2df422c83e087bc00f228106183845e0"},"previous_names":["bentoml/clip-api-service"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bentoml%2FCLIP-API-service","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bentoml%2FCLIP-API-service/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bentoml%2FCLIP-API-service/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bentoml%2FCLIP-API-service/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bentoml","download_url":"https://codeload.github.com/bentoml/CLIP-API-service/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224390053,"owners_count":17303449,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-applications","clip","cloud-native","mlops","model-inference","model-inference-service","model-serving","openai-clip"],"created_at":"2024-11-13T04:29:09.399Z","updated_at":"2024-11-13T04:29:10.084Z","avatar_url":"https://github.com/bentoml.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n    \u003ch1 align=\"center\"\u003eCLIP API Service\u003c/h1\u003e\n    \u003cbr\u003e\n    \u003cstrong\u003eDiscover the effortless integration of OpenAI's innovative CLIP model with our streamlined API service. \u003cbr\u003e\u003c/strong\u003e\n    \u003ci\u003ePowered by BentoML 🍱\u003c/i\u003e\n    \u003cbr\u003e\n\u003c/div\u003e\n\u003cbr\u003e\n\n## 📖 Introduction 📖\n[CLIP](https://openai.com/research/clip), or Contrastive Language-Image Pretraining, is a cutting-edge AI model that comprehends and connects text and images, revolutionizing how we interpret online data.\n\nThis library provides you with an instant, easy-to-use interface for CLIP, allowing you to harness its capabilities without any setup hassles. BentoML takes care of all the complexity of serving the model!\n\n## 🔧 Installation 🔧\nEnsure that you have Python 3.8 or newer and `pip` installed on your system. We highly recommend using a Virtual Environment to avoid any potential package conflicts.\n\nTo install the service, enter the following command:\n```bash\npip install clip-api-service\n```\n\n## 🏃 Quick start 🏃\n\nOnce the installation process is complete, you can start the service by running:\n```bash\nclip-api-service serve --model-name=ViT-B-32:openai\n```\nYour service is now running! Interact with it via the Swagger UI at `localhost:3000`\n![SwaggerUI](images/swagger-ui.png)\n\nOr try this tutorial in Google Colab: [CLIP demo](https://colab.research.google.com/github/bentoml/CLIP-API-service/blob/main/example/clip_demo.ipynb).\n\n## 🎯 Use cases 🎯\nHarness the capabilities of the CLIP API service across a range of applications:\n\n### Encode\n1. Text and Image Embedding\n    - Use `encode` to transform text or images into meaningful embeddings. This makes it possible to perform tasks such as:\n        1. **Neural Search**: Utilize encoded embeddings to power a search engine capable of understanding and indexing images based on their textual descriptions, and vice versa.\n        2. **Custom Ranking**: Design a ranking system based on embeddings, providing unique ways to sort and categorize data according to your context.\n\n### Rank\n2. Zero-Shot Image Classification\n    - Use `rank` to perform image classification without any training. For example:\n        1. Given a set of images, classify an image as being \"a picture of a dog\" or \"a picture of a cat\".\n        2. More complex classifications such as recognizing different breeds of dogs can also be performed, illustrating the versatility of the CLIP API service.\n\n3. Visual Reasoning\n    - The `rank` function can also be used to provide reasoning about visual scenarios. For instance:\n\n| Visual Scenario | Query Image | Candidates | Output |\n|-----------------|-------|---------------|--------|\n| Counting Objects | ![Three Dog](images/three-dog.jpg) | This is a picture of 1 dog\u003cbr\u003eThis is a picture of 2 dogs\u003cbr\u003eThis is a picture of 3 dogs | Image matched with \"3 dogs\" |\n| Identifying Colors | ![Blue Car](images/bluecar.jpeg)  | The car is red\u003cbr\u003eThe car is blue\u003cbr\u003eThe car is green | Image matched with \"blue car\" |\n| Understanding Motion | ![Parked Car](images/parkedcar.jpeg)  | The car is parked\u003cbr\u003eThe car is moving\u003cbr\u003eThe car is turning| Image matched with \"parked car\" |\n| Recognizing Location | ![Suburb Car](images/car-suburb.jpeg)  | The car is in the suburb\u003cbr\u003eThe car is on the highway\u003cbr\u003eThe car is in the street| Image matched with \"car in the street\" |\n| Relative Positioning | ![Big Small car](images/big-small-car.jpg) | The big car is on the left, the small car is on the right\u003cbr\u003eThe small car is on the left, the big car is on the right| Image matched with the provided description |\n \n\n## 🚀 Deploying to Production 🚀\nEffortlessly transition your project into a production-ready application using [BentoCloud](https://www.bentoml.com/bento-cloud/), the production-ready platform for managing and deploying machine learning models.\n\nStart by creating a BentoCloud account. Once you've signed up, log in to your BentoCloud account using the command:\n\n```bash\nbentoml cloud login --api-token \u003cyour-api-token\u003e --endpoint \u003cbento-cloud-endpoint\u003e\n```\n\u003e Note: Replace `\u003cyour-api-token\u003e` and `\u003cbento-cloud-endpoint\u003e` with your specific API token and the BentoCloud endpoint respectively.\n\nNext, build your BentoML service using the `build` command:\n\n```bash\nclip-api-service build --model-name=ViT-B-32:openai\n```\n\nThen, push your freshly-built Bento service to BentoCloud using the `push` command:\n\n```bash\nbentoml push \u003cname:version\u003e\n```\n\nLastly, deploy this application to BentoCloud with a single `bentoml deployment create` command following the [deployment instructions](https://docs.bentoml.org/en/latest/reference/cli.html#bentoml-deployment-create).\n\nBentoML offers a number of options for deploying and hosting online ML services into production, learn more at [Deploying a Bento](https://docs.bentoml.org/en/latest/concepts/deploy.html).\n\n## 📚 Reference 📚\n### API reference\n#### `/encode`\nAccepts either:\n* `img_uri` : An Image URI, i.e `https://hips.hearstapps.com/hmg-prod/images/dog-puppy-on-garden-royalty-free-image-1586966191.jpg`\n* `text` : A string\n* `img_blob` : Base64 encoded string\n\nReturns a vector of embeddings of length 768.\n\n**Example:**\n```\ncurl -X 'POST' \\\n  'http://localhost:3000/encode' \\\n  -H 'accept: application/json' \\\n  -H 'Content-Type: application/json' \\\n  -d '[\n  {\n    \"img_uri\": \"https://hips.hearstapps.com/hmg-prod/images/dog-puppy-on-garden-royalty-free-image-1586966191.jpg\"\n  },\n  {\n    \"text\": \"picture of a dog\"\n  },\n  {\n    \"img_blob\": \"iVBORw0KGgoAAAANSUhEUgAAABIAAAAPCAYAAADphp8SAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AAAApdEVYdENyZWF0aW9uIFRpbWUARnJpIDI2IE1heSAyMDIzIDA0OjE2OjIxIEFNCXIaIQAAAnhJREFUOI1Fk82OY0UMhT+7qu6dJPfmZ3p6EhYR3QSGnk0P6QV/4rlYsOIBeKrhIQAJtZCgNQsmEWiQSG5V2SwSeixZtiydYx9blpsJPpv2fPP1V3zx5ee8vHnJbDFlNl/Q9GMQBwLUAM7JcXADN9wN3Im1wOaja7afbfnk4xcsVytS25DaFsyptSDiaIyAnMn8kVTOMc6mI+62d2w2G5bLFZOuBxUkJsyNYoIGRVXfTyMCLmByIseJm+trbl58ynL5nK7v0dTgOGgENyQqsWlxdxxH4FEW7og7mBHvtq+4vHxG381IqQF3qjvBnOrgKBIiOR/fA89gMUPdoRpxe/uKbjym6zton4AZ6oZoRNxQUQiJOhzO3R1xeyTiXItXV1csLi4gRjADOIHdEEBEoGSCCOaOmUGtKI4CQQMIxL7vmIwmcBygAdoWAKuV6o5GwUwQB/HzjkTOufC/xZRaSi6IOjHFU1UVrRkrmTwcERViiohDQFANiAmYYbUgBnE8miAiDDlT/j0gQ0aj4qoklZMkHBsKIShBFJXTJLVWai64GbGUQnrSEori7ljO5Gy4gMZIahMpRA7DABZAwXGsVnIulOGIuxFfv/6RUdtwcfmM2XTKfDEnNYlcCjkP1BJIqYUChuNU3J1aM+Us382Jv97f8/d+x3w+YzbtWa5WLD94ztOnCyZdR0QRM8TkdACr1FqoJeNuuFUAwg/fffu9m7Hf73j4/Q9+/uUn3jw8YG6MRyNijFgtJG0wM8owMBwPlJKppZ6+RiA2TWK9XvPhes27f96x3+04Hg/s/vyLe/2N29ueyWxCzQe8Gvu3b0GUXCu7/Y5ijgblP3zyX4rqQyp1AAAAAElFTkSuQmCC\"\n  }\n]'\n```\n\n#### `/rank`\nAccepts a list of `queries` and a list of `candidates`. Similar to above, `queries` and `candidates` are either:\n* `img_uri` : An Image URI, i.e `https://hips.hearstapps.com/hmg-prod/images/dog-puppy-on-garden-royalty-free-image-1586966191.jpg`\n* `text` : A string\n* `img_blob` : Base64 encoded string\n\nReturns a list of probabilies and cosine similarities of each candidate with respect to the query.\n\n**Example:**\n```\ncurl -X 'POST' \\\n  'http://localhost:3000/rank' \\\n  -H 'accept: application/json' \\\n  -H 'Content-Type: application/json' \\\n  -d '{\n  \"queries\": [\n    {\n      \"img_uri\": \"https://hips.hearstapps.com/hmg-prod/images/dog-puppy-on-garden-royalty-free-image-1586966191.jpg\"\n    }\n  ],\n  \"candidates\": [\n    {\n      \"text\": \"picture of a dog\"\n    },\n    {\n      \"text\": \"picture of a cat\"\n    },\n    {\n      \"text\": \"picture of a bird\"\n    },\n    {\n      \"text\": \"picture of a car\"\n    },\n    {\n      \"text\": \"picture of a plane\"\n    },\n    {\n      \"text\": \"picture of a boat\"\n    }\n  ]\n}'\n```\nAnd the response looks like:\n```\n{\n  \"probabilities\": [\n    [\n      0.9958375692367554,\n      0.0022114247549325228,\n      0.001514736912213266,\n      0.00011969256593147293,\n      0.00019143625104334205,\n      0.0001251235808013007\n    ]\n  ],\n  \"cosine_similarities\": [\n    [\n      0.2297772467136383,\n      0.16867777705192566,\n      0.16489382088184357,\n      0.13951312005519867,\n      0.14420939981937408,\n      0.13995687663555145\n    ]\n  ]\n}\n```\n\n### CLI reference\n#### `serve`\nSpins up a HTTP Server with the model of your choice.\n\nArguments:\n* `--model-name` : Name of the CLIP model. Use `list_models` to see the list of available model. Default: `openai/clip-vit-large-patch14`\n\n#### `build`\nBuilds a Bento with the model of your choice\n\nArguments:\n* `--model-name` : Name of the CLIP model. Use `list_models` to see the list of available model. Default: `openai/clip-vit-large-patch14`\n\n#### `list_models`\nList all available CLIP models.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbentoml%2Fclip-api-service","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbentoml%2Fclip-api-service","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbentoml%2Fclip-api-service/lists"}