{"id":13584171,"url":"https://github.com/roboflow/foundation-vision-benchmark","last_synced_at":"2026-01-28T23:33:46.549Z","repository":{"id":207062647,"uuid":"718332109","full_name":"roboflow/foundation-vision-benchmark","owner":"roboflow","description":"A qualitative set of tests for use in evaluating the capabilities of foundation vision models.","archived":false,"fork":false,"pushed_at":"2023-11-13T22:05:06.000Z","size":9,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-09-29T09:42:59.258Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/roboflow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-11-13T21:34:52.000Z","updated_at":"2024-01-29T18:13:35.000Z","dependencies_parsed_at":"2023-11-13T23:22:33.299Z","dependency_job_id":"b9ca8123-3817-45a8-8320-14af026ce27d","html_url":"https://github.com/roboflow/foundation-vision-benchmark","commit_stats":null,"previous_names":["roboflow/foundation-vision-benchmark"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/roboflow/foundation-vision-benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roboflow%2Ffoundation-vision-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roboflow%2Ffoundation-vision-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roboflow%2Ffoundation-vision-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roboflow%2Ffoundation-vision-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/roboflow","download_url":"https://codeload.github.com/roboflow/foundation-vision-benchmark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roboflow%2Ffoundation-vision-benchmark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28855171,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-28T22:56:21.783Z","status":"ssl_error","status_checked_at":"2026-01-28T22:56:00.861Z","response_time":57,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T15:04:03.708Z","updated_at":"2026-01-28T23:33:46.533Z","avatar_url":"https://github.com/roboflow.png","language":null,"funding_links":[],"categories":["Others"],"sub_categories":[],"readme":"# foundation-vision-benchmark\n\nA qualitative set of tests for use in evaluating the capabilities of foundation vision models.\n\n## 📝 Abstract\n\nIn 2023, multimodal vision models have become substantially more powerful; the year has been coined as the \"year of multimodality.\" Roboflow actively explores new vision models and their capabilities, conducting qualitative tests to learn more about how different models perform.\n\nOur tests help us answer questions like \"will this model help with labeling data?\" and \"could this model be used in a two-stage detection system (i.e. OCR)?\"\n\nThis repository contains several tests we have run on foundation vision models to understand their performance. If you are curious about exploring new vision models, this repository may give you direction, ideas, and result in novel learnings about the model with which you are working.\n\n## ✅ Tests\n\n| Task Type | Prompt        | Image               |\n|-----------|---------------|---------------------|\n| Classification   | What is in the image? Return the class of the object in the image. Here are the classes: Tesla Model 3, Toyota Camry. You can only return one class from that list. | ![Image Example](https://lh7-us.googleusercontent.com/-sp5y_4HnZrQeqbC4Ufb-CnHDBkG1uZT5E-A6LDEItuFzk4rxkoC-RFyL7xuzGKJZ029Zwbcyvip-dHrFX8lCbW7da1MRK97bORbaVrZdiUzRFhoSM-bwp3TgoNJKUxQHuMcC9fJpHAyLnaO1ig_MlQ)  |\n| Classification   | What is in the image? Return the class of the object in the image. Here are the classes: pizza, deep dish pizza. You can only return one class from that list. | ![Image Example](https://lh7-us.googleusercontent.com/dxD4vgo4fMWSF32Uxh6eB8Bjj85Gfrm9lgSgCuH0KWJYhv4khZ9SfC9OMpOB0ResXi8npF5gEL3A0GZC9sqq-V-NlM19WBn1GIop59IdPqUbO29FhUvCbVZJZkZmjsAWM6BCEbTCNoK22hezBSD_sKo)  |\n| Optical Character Recognition (Documents)   | Read the text in the image. | ![Image Example](https://media.roboflow.com/swift.png)  |\n| Object Detection | Detect a dog on the image. Provide me with x_min_y_min, x_max and y_max coordinates. | ![Image Example](https://media.roboflow.com/dog.jpeg) |\n\n## 🧪 Test Surface Area\n\nWe test:\n\n1. Object detection\n2. Classification\n4. Document OCR\n5. Handwriting OCR\n6. Math OCR\n\nSee the sections below for reference images and prompts.\n\n### General Recommendations\n\nWe also recommend testing visual question answering (asking general questions about an image), exploring areas such as:\n\n- Spatial awareness: Can the model answer questions about how objects relate?\n- Veracity: When asked a question about an image, is the model correct?\n- Comprehensiveness: Does a model provide a comprehensive answer to your question? Are details missed?\n- Consistently: Does a model consistently answer questions it can answer? Does a model say it cannot help with a question with which it has previously helped?\n\n## ⚠️ Limitations\n\nGiven the vast array of possibilities with vision models, no set of tests, this one included, can comprehensively evaluate what a model can do; this repository is a starting point for exploration.\n\n## 🦸 Contribute\n\nWe would love your help in making this repository even better! Whether you want to\nadd a new experiment or have any suggestions for improvement,\nfeel free to open an [issue](https://github.com/roboflow/awesome-openai-vision-api-experiments/issues)\nor [pull request](https://github.com/roboflow/awesome-openai-vision-api-experiments/pulls).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Froboflow%2Ffoundation-vision-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Froboflow%2Ffoundation-vision-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Froboflow%2Ffoundation-vision-benchmark/lists"}