{"id":13572116,"url":"https://github.com/roboflow/awesome-openai-vision-api-experiments","last_synced_at":"2025-05-14T13:00:27.640Z","repository":{"id":206079775,"uuid":"715680093","full_name":"roboflow/awesome-openai-vision-api-experiments","owner":"roboflow","description":"Must-have resource for anyone who wants to experiment with and build on the OpenAI vision API 🔥","archived":false,"fork":false,"pushed_at":"2025-01-14T16:38:43.000Z","size":42430,"stargazers_count":1681,"open_issues_count":2,"forks_count":132,"subscribers_count":26,"default_branch":"main","last_synced_at":"2025-05-07T02:02:03.225Z","etag":null,"topics":["chatgpt","classification","clip","computer-vision","grounding-dino","open-vocabulary-detection","open-vocabulary-segmentation","openai","segment-anything","zero-shot"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/roboflow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-07T16:08:33.000Z","updated_at":"2025-05-05T08:37:01.000Z","dependencies_parsed_at":"2024-01-11T20:42:38.459Z","dependency_job_id":"45628e38-aa3d-4338-8f8a-66d0b5eabe3c","html_url":"https://github.com/roboflow/awesome-openai-vision-api-experiments","commit_stats":{"total_commits":41,"total_committers":7,"mean_commits":5.857142857142857,"dds":"0.36585365853658536","last_synced_commit":"7adff72c2de03e104f1029dfab3988cf4dca83be"},"previous_names":["roboflow/awesome-openai-vision-api-experiments"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roboflow%2Fawesome-openai-vision-api-experiments","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roboflow%2Fawesome-openai-vision-api-experiments/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roboflow%2Fawesome-openai-vision-api-experiments/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roboflow%2Fawesome-openai-vision-api-experiments/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/roboflow","download_url":"https://codeload.github.com/roboflow/awesome-openai-vision-api-experiments/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253948411,"owners_count":21988953,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatgpt","classification","clip","computer-vision","grounding-dino","open-vocabulary-detection","open-vocabulary-segmentation","openai","segment-anything","zero-shot"],"created_at":"2024-08-01T14:01:13.846Z","updated_at":"2025-05-14T13:00:27.477Z","avatar_url":"https://github.com/roboflow.png","language":"Python","funding_links":[],"categories":["Python","Awesome-lists","Inbox: Speech-to-text (STT) and spoken content analysis","API´s","Documentation and examples","📦 Legacy \u0026 Inactive Projects"],"sub_categories":["Creative Uses of Generative AI Image Synthesis Tools","Documentation, lists, guides, or examples"],"readme":"\u003ch1 align=\"center\"\u003eopenai vision api experiments 🧪\u003c/h1\u003e\n\n## 👋 Hello\n\nThe must-have resource for anyone who wants to experiment with and build on the [OpenAI\nVision API](https://platform.openai.com/docs/guides/vision). This repository serves as\na hub for innovative experiments, showcasing a variety of applications ranging from\nsimple image classifications to advanced zero-shot learning models. It's a space for\nboth beginners and experts to explore the capabilities of the Vision API, share their\nfindings, and collaborate on pushing the boundaries of visual AI.\n\nExperimenting with the OpenAI API requires an API 🔑. You can get one\n[here](https://platform.openai.com/api-keys).\n\n## ⚠️ Limitations\n\n- 100 API requests per single API key per day.\n- Can't be used for object detection or image segmentation. We can solve this problem by combining GPT-4V with foundational models like GroundingDINO or Segment Anything (SAM). Please take a look at the [example](https://github.com/roboflow/awesome-openai-vision-api-experiments/tree/main/experiments/gpt4v-grounding-dino-detection) and read our [blog post](https://blog.roboflow.com/dino-gpt-4v).\n\n## 🧪 Experiments\n\n\u003c!--- AUTOGENERATED_EXPERIMENTS_LIST --\u003e\n\u003c!---\n   WARNING: DO NOT EDIT THIS LIST MANUALLY. IT IS AUTOMATICALLY GENERATED.\n   HEAD OVER TO CONTRIBUTING.MD FOR MORE DETAILS ON HOW TO MAKE CHANGES PROPERLY.\n--\u003e\n| **experiment** | **complementary materials** | **authors** |\n|:--------------:|:---------------------------:|:-----------:|\n| WebcamGPT - chat with video stream | [![GitHub](https://badges.aleen42.com/src/github.svg)](https://github.com/roboflow/awesome-openai-vision-api-experiments/blob/main/experiments/webcam-gpt) [![Gradio](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/Roboflow/webcamGPT)  | @SkalskiP |\n| HotDogGPT - simple image classification application | [![GitHub](https://badges.aleen42.com/src/github.svg)](https://github.com/roboflow/awesome-openai-vision-api-experiments/blob/main/experiments/hot-dog-not-hot-dog) [![Gradio](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/Roboflow/HotDogGPT)  | @SkalskiP |\n| zero-shot image classifier with GPT-4V | [![GitHub](https://badges.aleen42.com/src/github.svg)](https://github.com/roboflow/awesome-openai-vision-api-experiments/tree/main/experiments/gpt4v-classification)   | @capjamesg |\n| zero-shot object detection with GroundingDINO + GPT-4V | [![GitHub](https://badges.aleen42.com/src/github.svg)](https://github.com/roboflow/awesome-openai-vision-api-experiments/tree/main/experiments/gpt4v-grounding-dino-detection) [![Gradio](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/Roboflow/DINO-GPT4V)  | @capjamesg |\n| GPT-4V vs. CLIP | [![GitHub](https://badges.aleen42.com/src/github.svg)](https://github.com/roboflow/awesome-openai-vision-api-experiments/tree/main/experiments/gpt4v-vs-clip)   | @capjamesg |\n| GPT-4V with Set-of-Mark (SoM) | [![GitHub](https://badges.aleen42.com/src/github.svg)](https://github.com/microsoft/SoM)   | Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, Jianfeng Gao |\n| GPT-4V on Web | [![GitHub](https://badges.aleen42.com/src/github.svg)](https://github.com/Jiayi-Pan/GPT-V-on-Web)   | @Jiayi-Pan |\n| automated voiceover of NBA game | [![GitHub](https://badges.aleen42.com/src/github.svg)](https://github.com/roboflow/awesome-openai-vision-api-experiments/tree/main/experiments/automated-voiceover-of-nba-game)  [![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/roboflow/awesome-openai-vision-api-experiments/blob/main/experiments/automated-voiceover-of-nba-game/notebook.ipynb) | @SkalskiP |\n| screenshot-to-code | [![GitHub](https://badges.aleen42.com/src/github.svg)](https://github.com/abi/screenshot-to-code)   | @abi |\n| GPT with Vision Checkup | [![GitHub](https://badges.aleen42.com/src/github.svg)]( https://github.com/roboflow/gpt-checkup)   |  Roboflow team |\n\nhttps://github.com/roboflow/awesome-openai-vision-api-experiments/assets/26109316/c63fa3c0-4564-49ee-8982-a9e6a23dae9b\n\n## 🗞️ Must Read Papers\n\n- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V](https://arxiv.org/abs/2310.11441)\nby Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, Jianfeng Gao\n- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://arxiv.org/abs/2309.17421)\nby Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, Lijuan Wang\n- [GPT-4 System Card](https://cdn.openai.com/papers/gpt-4-system-card.pdf) by OpenAI\n\n## 🖊️ Blogs\n\n- [How CLIP and GPT-4V Compare for Classification](https://blog.roboflow.com/clip-vs-gpt-4v/)\n- [Experiments with GPT-4V for Object Detection](https://blog.roboflow.com/gpt-4v-object-detection/)\n- [Distilling GPT-4 for Classification with an API](https://blog.roboflow.com/gpt-4-image-classification/)\n- [DINO-GPT4-V: Use GPT-4V in a Two-Stage Detection Model](https://blog.roboflow.com/dino-gpt-4v/)\n- [First Impressions with GPT-4V(ision)](https://blog.roboflow.com/gpt-4-vision/)\n\n## 🦸 Contribution\n\nWe would love your help in making this repository even better! Whether you want to\nadd a new experiment or have any suggestions for improvement,\nfeel free to open an [issue](https://github.com/roboflow/awesome-openai-vision-api-experiments/issues)\nor [pull request](https://github.com/roboflow/awesome-openai-vision-api-experiments/pulls).\n\nIf you are up to the task and want to add a new experiment, please look at our [contribution guide](https://github.com/roboflow/awesome-openai-vision-api-experiments/blob/main/CONTRIBUTING.md). There you can find all the information you need.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Froboflow%2Fawesome-openai-vision-api-experiments","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Froboflow%2Fawesome-openai-vision-api-experiments","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Froboflow%2Fawesome-openai-vision-api-experiments/lists"}