{"id":20828767,"url":"https://github.com/perty/sudolang","last_synced_at":"2025-08-17T20:11:22.433Z","repository":{"id":256026301,"uuid":"849826201","full_name":"perty/sudolang","owner":"perty","description":"Attempts to learn Sudolang","archived":false,"fork":false,"pushed_at":"2024-09-08T16:10:14.000Z","size":12,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-12T07:44:28.464Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/perty.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-30T10:22:55.000Z","updated_at":"2024-09-08T16:10:17.000Z","dependencies_parsed_at":"2024-09-08T16:27:34.598Z","dependency_job_id":"0d1b2c22-0f5b-4d35-9c79-4772f5ad55f0","html_url":"https://github.com/perty/sudolang","commit_stats":null,"previous_names":["perty/sudolang"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/perty/sudolang","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perty%2Fsudolang","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perty%2Fsudolang/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perty%2Fsudolang/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perty%2Fsudolang/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/perty","download_url":"https://codeload.github.com/perty/sudolang/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perty%2Fsudolang/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270899582,"owners_count":24664720,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-17T02:00:09.016Z","response_time":129,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T23:18:24.187Z","updated_at":"2025-08-17T20:11:22.405Z","avatar_url":"https://github.com/perty.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sudolang experiments\n\nThis repository contains some experiments with the [Sudolang](https://github.com/paralleldrive/sudolang-llm-support)\nlanguage.\n\nSudolang is a programming language designed to collaborate with AI language models.\n\n## Hunt the Wumpus\n\nI thought that the old 70' game Hunt the Wumpus would be a good example to try SudoLang. It is text based which\nsimplifies\nthe design of the first version of the user interface. I have used it in teaching good software development practices\nsuch as continuous integration.\n\nA problem might be that the game may already be known to the LLM which might interfere. The LLM might fill in blanks\nin my program, sort of cheating, and is not a great evaluation of the technique.\n\nI approached by first just write the description of the game with content from a\n[Wikipedia page](https://en.wikipedia.org/wiki/Hunt_the_Wumpus). It did fairly well, but broke some rules.\n\nNext, I tried to do the same in Sudolang. I both tried to get it transpiled to JavaScript and run it in the browser but\nboth ChapGpt 4o and Claude 3.5 Sonnet failed in different ways.\n\nThe main problem seems to be shooting the arrows. The rule is that the user may enter any caves they are aiming the\narrow\nto but the arrow can only follow connected caves. If the given path is not possible, it should resort to pick caves\nrandomly.\n\nI tried with writing the constraints explicitly, however, no luck.\n\n## Back to square one\n\nSo, how does different approaches compare. And does the LLM matter? I decided to create a structured approach.\n\nThe first test would be a simple free-text description of the game. I would compare the LLMs ChapGpt 4o and Claude 3.5\nSonnet. Each LLM would both play the game and transpile the game into Java.\n\nSecond test would be to use Sudolang to describe the game, taking the promise of pseudocode to the table.\n\nThe hypothesis is that the LLMs will perform worse in the free-text description than in the Sudolang description.\nThe evaluation criteria are compliance with the rules of the game and the ability to transpile the game into fully\nfunctional program code.\n\nI renamed the game to Knossos so nothing would leak from training data.\n\n## Experiment 1 Free-text description\n\nFile: [knossos_v1.md](back-to-square-1/knossos_v1.md) \nFree-text description, based on the [Wikipedia page](https://en.wikipedia.org/wiki/Hunt_the_Wumpus). \n\n**ChaptGpt** had no problem understanding the game. However, it did not follow the rules. It did not understand the \nconstraints of the arrows, that they only travel through connected caves. Also, although the Minotaur smell was\nthere, it was not in a connected cave, which was a clear violation of the rules.\n\nI tried ask it where the Minotaur was, but it stayed in the role a game master and did reveal the location.\n\nNext thing, I just asked for a Java program that implemented the game. It gave me an outline of the game, but it was not\ncomplete. Fair enough, it asked if I needed help with the implementation.\n\nI asked for a test of the game, so I wouldn't have to do that manually. ChatGpt did so and there was a number of \ncompiler errors. I iterated until the test passed. Told ChatGpt what was not passing and got more implementation back.\n\nWhen the test passed, it was still not possible to play the game, so I asked for a new test where the interpretation\nof user commands was tested. Again, this drove the implementation forward. \n\nOne of the tests didn't pass but here I gave up. The way I worked with ChatGpt was just tiresome.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperty%2Fsudolang","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fperty%2Fsudolang","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperty%2Fsudolang/lists"}