{"id":25725548,"url":"https://github.com/umar221b/autograder_llm_test_generator","last_synced_at":"2026-06-15T15:32:29.700Z","repository":{"id":227815787,"uuid":"738298432","full_name":"umar221b/autograder_llm_test_generator","owner":"umar221b","description":"A small app that talks to OpenAI to generate tests for CS1 programming problems","archived":false,"fork":false,"pushed_at":"2024-03-25T21:59:13.000Z","size":3549,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-25T22:38:10.857Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/umar221b.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-02T22:52:10.000Z","updated_at":"2024-03-15T07:36:28.000Z","dependencies_parsed_at":"2024-03-25T22:52:02.245Z","dependency_job_id":null,"html_url":"https://github.com/umar221b/autograder_llm_test_generator","commit_stats":null,"previous_names":["umar221b/autograder_llm_test_generator"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/umar221b/autograder_llm_test_generator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umar221b%2Fautograder_llm_test_generator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umar221b%2Fautograder_llm_test_generator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umar221b%2Fautograder_llm_test_generator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umar221b%2Fautograder_llm_test_generator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/umar221b","download_url":"https://codeload.github.com/umar221b/autograder_llm_test_generator/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umar221b%2Fautograder_llm_test_generator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34369842,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-15T02:00:07.085Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-02-25T22:30:40.313Z","updated_at":"2026-06-15T15:32:29.669Z","avatar_url":"https://github.com/umar221b.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Autograding Using LLMs\n\n## Workflow\n\n1. Instructor provides problem statement, reference solution, and any extra code\n2. Reference solution is required\n    1. Provided by instructor\n    2. Generated by AI, but approved/edited by instructor (not implemented)\n3. Find problem type: Test case inputs (full-program) / Unit Tests (function-implementation), or have the instructor select it (currently the instructor selects it). \"Unit Tests\" is no longer the right way to describe this type of grading.\n4. Find assignment's programming language, or have the instructor select it (currently the instructor selects it)\n5. LLM: Generate detailed problem description from the provided problem description, reference solution, and extra code\n6. LLM: Generate test cases based on the detailed problem description and the reference solution:\n    1. Find edge cases\n    2. Reflect on their correctness and coverage \n    3. Generate tests (edge + random)\n       - For full-programs: Tests are only inputs.\n       - For unit tests:\n         - Python (not fully implemented): tests compare the result of calling the student's function with the result of calling the reference solution on the same input\n         - C: tests call the tested function and print its result to standard output\n       - Note: The LLM was not able to run the reference solution to generate the outputs. It expects wrong outputs for the tests, even when it says it did run the solution to generate the expected outputs\n7. Run tests and drop any test that crashes the reference solution\n\n## Security Concerns\n\nInstructor and LLM generated code must be run inside a sandbox to avoid any unauthorized access to the host machine.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fumar221b%2Fautograder_llm_test_generator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fumar221b%2Fautograder_llm_test_generator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fumar221b%2Fautograder_llm_test_generator/lists"}