{"id":20058642,"url":"https://github.com/roboflow/gpt-checkup","last_synced_at":"2025-03-02T09:42:37.235Z","repository":{"id":207229962,"uuid":"718655119","full_name":"roboflow/gpt-checkup","owner":"roboflow","description":"Monitor the performance of OpenAI's GPT O3 Mini model over time.","archived":false,"fork":false,"pushed_at":"2025-02-26T11:32:43.000Z","size":23053,"stargazers_count":33,"open_issues_count":1,"forks_count":5,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-02T02:56:13.597Z","etag":null,"topics":["computer-vision","gpt-o1","gpt4v","model-analysis","o1"],"latest_commit_sha":null,"homepage":"https://www.gptcheckup.com","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/roboflow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-14T14:31:54.000Z","updated_at":"2025-02-26T11:32:46.000Z","dependencies_parsed_at":"2024-01-06T15:29:21.926Z","dependency_job_id":"89ce9364-b8ed-4ccd-bacd-f631417f2ecf","html_url":"https://github.com/roboflow/gpt-checkup","commit_stats":{"total_commits":459,"total_committers":5,"mean_commits":91.8,"dds":0.1568627450980392,"last_synced_commit":"c116aa2aebdce4453e61507ab08495ebaf9e05fa"},"previous_names":["roboflow/gpt-checkup"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roboflow%2Fgpt-checkup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roboflow%2Fgpt-checkup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roboflow%2Fgpt-checkup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roboflow%2Fgpt-checkup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/roboflow","download_url":"https://codeload.github.com/roboflow/gpt-checkup/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241488154,"owners_count":19970826,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","gpt-o1","gpt4v","model-analysis","o1"],"created_at":"2024-11-13T13:02:41.852Z","updated_at":"2025-03-02T09:42:37.215Z","avatar_url":"https://github.com/roboflow.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GPT-4V Checkup\n\nOpen source monitor that tests how [GPT-4V](https://openai.com/research/gpt-4v-system-card) performs on image prompts over time.\n\nThis project is not affiliated with OpenAI.\n\nSee the companion [foundation-vision-benchmark](https://github.com/roboflow/foundation-vision-benchmark) repository for more information about qualitatively evaluating foundation vision models like we do on this website.\n\n![GPT-4V Monitor](screenshot.png)\n\n## ⚠️ Limitations\n\nGiven the vast array of possibilities with vision models, no set of tests, this one included, can comprehensively evaluate what a model can do; this repository is a starting point for exploration.\n\n## 🦸 Contribute\n\nWe would love your help in making this repository even better! Whether you want to\nadd a new experiment or have any suggestions for improvement,\nfeel free to open an [issue](https://github.com/roboflow/awesome-openai-vision-api-experiments/issues)\nor [pull request](https://github.com/roboflow/awesome-openai-vision-api-experiments/pulls).\n\nWe welcome additions to the list of tests. If you contribute a test, we will run it daily.\n\nTo contribute a test, first fork this repository. Then, clone your fork locally:\n\n```bash\ngit clone \u003cyour fork\u003e\ncd gpt-checkup\n```\n\nHere is the structure of the project:\n\n- `tests`: Contains the tests that are run daily.\n- `images`: Contains the images used in the tests.\n- `results`: Contains the results of the tests saved from each day.\n- `web.py`: Contains the code used to run the tests and create the website.\n- `template.html`: The template for the website. Used to generate the `index.html` file.\n\nThen, create a new file in the `tests` directory with the name of the test you want to add. Use the `mathocr.py` file as an example.\n\nIn your test, you will need to specify:\n\n1. A test name\n2. A test ID\n3. The question you are answering (for display on the website only)\n4. The prompt to send to the model\n5. The image to send to the model\n6. A description of the method used in the test (for display on the website only)\n\nAdd the image(s) you want to use in your test(s) in the `images` directory.\n\nAdd the name of the test class you created to the `test_list` list in the `web.py` file\nand to the imports in the `tests/__init__.py` file.\n\nBefore you run the tests locally, you will need to set up an OpenAI API key. Refer to the official OpenAI documentation for instructions on [how to retrieve your OpenAI API key](https://help.openai.com/en/articles/4936850-where-do-i-find-my-api-key).\n\nExport your key into a variable called `OPENAI_API_KEY` in your environment:\n\n```bash\nexport OPENAI_API_KEY=\u003cyour key\u003e\n```\n\nThen, run the tests:\n\n```bash\npython3 web.py\n```\n\nYou will see messages printed in your console as each test is run:\n\n```text\nRunning Document OCR test...\nRunning Handwriting OCR test...\nRunning Structured Data OCR test...\nRunning Math OCR test...\nRunning Object Detection test...\nRunning Graph Understanding test...\n...\n```\n\nWhen the script has run, the results of all tests will show up in the `index.html` file. A new file will be added to the `results` directory which contains a JSON representation of the test results.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Froboflow%2Fgpt-checkup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Froboflow%2Fgpt-checkup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Froboflow%2Fgpt-checkup/lists"}