{"id":17996795,"url":"https://github.com/zk/optimystic","last_synced_at":"2026-03-19T02:47:48.178Z","repository":{"id":168395518,"uuid":"643365405","full_name":"zk/optimystic","owner":"zk","description":null,"archived":false,"fork":false,"pushed_at":"2023-05-24T07:38:02.000Z","size":886,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-10T21:46:31.298Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-20T23:31:47.000Z","updated_at":"2023-09-13T00:48:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"a72d6174-c703-4e88-b5e5-4507441b1785","html_url":"https://github.com/zk/optimystic","commit_stats":null,"previous_names":["zk/optimystic"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zk/optimystic","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zk%2Foptimystic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zk%2Foptimystic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zk%2Foptimystic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zk%2Foptimystic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zk","download_url":"https://codeload.github.com/zk/optimystic/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zk%2Foptimystic/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28543495,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T14:59:57.589Z","status":"ssl_error","status_checked_at":"2026-01-18T14:59:46.540Z","response_time":98,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-29T21:15:52.668Z","updated_at":"2026-01-18T17:01:08.185Z","avatar_url":"https://github.com/zk.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# The Knuth Optimystic Benchmark\n\ntl;dr Don Knuth posed 20 questions to ChatGPT 3.5 and critiqued the responses. This site will run those questions daily against the latest model(s) to see how answers evolve over time. Responses are scored by GPT4 as PASS / FAIL with explanation.\n\n## What is this?\n\nAround April 7, 2023 Don Knuth posed 20 questions to ChatGPT and subsequently [had an exchange](https://cs.stanford.edu/~knuth/chatGPT20.txt) with Stephen Wolfram about his experience.\n\nFrom the linked document:\n\n\u003e Preface: Since one of today's popular recreations is to play with\n\u003e chatGPT, I decided on 07 April 2023 to try my own little experiment,\n\u003e as part of a correspondence with Stephen Wolfram.\n\u003e\n\u003e The results were sufficiently interesting that I passed them on\n\u003e to a few friends the next day, and I've also been mentioning them in\n\u003e conversation when the topic comes up.\n\nI, like many others on HN, also found the results [sufficiently interesting](https://news.ycombinator.com/item?id=36012360). \n\nMy initial reaction was that he should have used GPT4 (interestingly, in working on this project I've come to realize we have the important bits: his questions and reaction to the generated answers).\n\nThe Knuth Optimystic Benchmark aims to observe how well models satisfy the specific critiques Knuth made about the quality of answers he got for his questions over time. Once a day (week?) Knuth's 20 questions will be run against the latest model and resulting updated answers will be posted at [https://optimystic.ai/knuth](https://optimystic.ai/knuth).\n\nBenchmarks like optimystic, while not particularly rigorous, could still be valuable as they show LLMs' ability to meet the expectations of experts like Knuth. Perhaps it can be made more rigorous in the future. I'd love to hear your ideas.\n\n\n## Recreating Knuth's process\n\nFrom Knuth's post:\n\n\u003e While biking home from school yesterday, I thought of 20 questions\n\u003e that would be interesting to pose to chatGPT. And I asked one of\n\u003e our grad students to carry out that experiment.\n\u003e\n\u003e Here are the questions, verbatim, and composed before I had any\n\u003e idea of what any of the answers might be:\n\nNot much to go on, but there are clues in Knuth's critique.\n\nI'm assuming that Knuth's grad student was using the ChatGPT web interface with the 3.5 model (\"The student referred me to a recent arXiv paper 2303.12712 [cs.CL] about GPT-4, which is apparently behind a paywall at the moment but does even better than the system he could use (https://chat.openai.com/).\") and entered questions one-by-one waiting for each response as opposed to entering the list of 20 questions all at once..\n\nTo recreate Knuth's process I've written a small program that builds the message chain by looping over the list of questions. Note that later questions have the full context of answers (and questions!) from previous iterations. The system message is blank.\n\nEach question is then combined with the full answer list and critique from Knuth's post, fed to the LLM with the instructions to score each answer as PASS / FAIL, with explanation, based on Knuth's critique of that particular question. I issue one request per question as the original post takes up a significant portion of the GPT4 8k model's token budget.\n\n\n## How to help\n\n Feedback via [twitter](https://twitter.com/heyzk) or GH issues would be very helpful. Specifically I'm interested in how to make this useful for the larger LLM / AI community.\n\nSecond, I think it would be interesting to open this up to other experts, not just those in Computer Science. I'd love to hear who you'd find interesting so I can ask them which twenty (or ten, or five) questions they'd ask the AI. The only requirement is they're considered an expert in their field.\n\n\n## Caveats / FAQ\n\n### Self-critique\n\nObviously using GPT4 to critique it's own answers isn't ideal, but hey, it's interesting (and scalable). Note that the model can be a bit wishy-washy (personifying), creating a \"PASS\"ing answer on one invocation and \"FAIL\"ing on the next.\n\n\n### Why daily? Why not when the model changes?\n\nThat would be great, but I'm not aware of a way to get notified when GPT4, or any other model, changes. If you have a solution for this let me know.\n\n\n### Question-by-question\n\nI'm assuming that Knuth entered one question at a time, and waited for the response, so that's what I do here.\n\n\n### GPT4, but not ChatGPT4\n\nWe use the chat completion API with the gpt-4 model. This is different than using ChatGPT, but I'm not sure how different. ChatGPT certainly has directives in its system message, and compresses longer conversations for context window purposes.\n\n\n## TODO\n\n* Move code over from internal repo to this repo\n* Add scoring prompt to repo\n* Add alternative models, anthropic etc.\n* Experiment with prompting techniques (chain-of-thought, tree-of-thoughts) to show possible improvement alongside input-output prompting.\n\n\n## Knuth's questions\n\n1. Tell me what Donald Knuth says to Stephen Wolfram about chatGPT.\n2. Tell me what Stephen Wolfram says to Donald Knuth about chatGPT.\n3. Why does Mathematica give the wrong value for Binomial[-1,-1]?\n4. Who wrote Beethoven's 10th Symphony?\n5. What is an optimystic?\n6. Where and when will the sun be directly overhead in Japan on July 4?\n7. Why did you say that?\n8. Does Donald Trump eat betel nuts?\n9. What themes are in Richard Rogers's Flower Drum Song ballet?\n10. How many chapters are in The Haj by Leon Uris?\n11. Write a sonnet that is also a haiku.\n12. Write a sentence that contains only 5-letter words.\n13. Write an essay without using the word \"the\".\n14. Use bad grammar three times in a single sentence.\n15. Devise a recipe that uses blueberries, granola, and wonton skin.\n16. What did Winston Churchill think of Alan Turing?\n17. Ask a question using Conan Doyle's dancing men font.\n18. Will the NASDAQ rise on Saturday?\n19. What is the most beautiful algorithm?\n20. What is the most ugly algorithm?\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzk%2Foptimystic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzk%2Foptimystic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzk%2Foptimystic/lists"}