{"id":23685824,"url":"https://github.com/dodona-edu/universal-judge","last_synced_at":"2025-09-02T12:32:49.899Z","repository":{"id":42436144,"uuid":"211055282","full_name":"dodona-edu/universal-judge","owner":"dodona-edu","description":"Universal judge for educational software testing","archived":false,"fork":false,"pushed_at":"2024-10-29T12:35:23.000Z","size":93501,"stargazers_count":9,"open_issues_count":42,"forks_count":5,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-10-29T15:01:52.485Z","etag":null,"topics":["dodona","educational-software","judge"],"latest_commit_sha":null,"homepage":"https://docs.dodona.be/en/tested","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dodona-edu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-26T09:47:31.000Z","updated_at":"2024-10-22T09:39:54.000Z","dependencies_parsed_at":"2023-11-28T12:24:14.015Z","dependency_job_id":"f859dde6-28b5-417c-9447-dbf9c4bbbc86","html_url":"https://github.com/dodona-edu/universal-judge","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dodona-edu%2Funiversal-judge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dodona-edu%2Funiversal-judge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dodona-edu%2Funiversal-judge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dodona-edu%2Funiversal-judge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dodona-edu","download_url":"https://codeload.github.com/dodona-edu/universal-judge/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":231785230,"owners_count":18426289,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dodona","educational-software","judge"],"created_at":"2024-12-29T21:15:49.733Z","updated_at":"2024-12-29T21:15:50.163Z","avatar_url":"https://github.com/dodona-edu.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TESTed: universal judge for educational software testing\n\nTESTed is a software test framework to evaluate submissions for programming exercises across multiple programming languages, using a single test suite per exercise.\n\nTESTed is developed by [Team Dodona](https://dodona.ugent.be/en/about/) at Ghent University.\nIf you use this software in research, please cite:\n\n- Strijbol, N., Van Petegem, C., Maertens, R., Sels, B., Scholliers, C., Dawyndt, P., \u0026 Mesuere, B. (2023). TESTed—An educational testing framework with language-agnostic test suites for programming exercises. SoftwareX, 22, 101404. [doi:10.1016/j.softx.2023.101404](https://doi.org/10.1016/j.softx.2023.101404)\n\n\u003e [!IMPORTANT]\n\u003e The documentation below is intended for running TESTed as a standalone tool.\n\u003e If you are looking to create exercises for Dodona, we have [more suitable documentation available](https://docs.dodona.be/nl/guides/exercises/).\n\n\n## Installing TESTed\n\nTESTed is implemented in Python, but has various dependencies for its language-specific modules.\n\nTo be able to work with all these dependencies on different platforms we make use of [devcontainers](https://containers.dev/).\nThis means that you can use the provided `.devcontainer/devcontainer.json` to open a container with all dependencies installed.\n\nModern IDEs like [Visual Studio Code](https://code.visualstudio.com/) and [PyCharm](https://www.jetbrains.com/pycharm/) support devcontainers out of the box.\n\nIf you prefer installing all dependencies on your local machine, you can find the installed dependencies in the [dockerfile](./.devcontainer/dodona-tested.dockerfile).\nThe extra development dependencies are listed in the [dev-dependencies.sh](./devcontainer/dev-dependencies.sh) file.\n\n## Running TESTed\n\nTESTed evaluates a submission for a programming exercise based on a test suite that specifies some test cases for the exercise.\nIn what follows, we guide you through the configuration of a simple programming exercise and running TESTed to evaluate a submission using the test suite of the exercise.\nThe directory `./exercise/` in the root directory of TESTed contains some more examples of programming exercises with test suites for TESTed.\n\n### 1. Create an exercise\n\nLet's configure a simple programming exercise that asks to implement a function `echo`.\nThe function takes a single argument and returns its argument.\n\nStart creating a directory for the configuration of the exercise.\nTo keep things simple, we add the exercise to the `exercise` subdirectory in the root directory of TESTed.\n\n```bash\nmkdir exercise/simple-example\n```\n\nNote that you would normally not store your exercises in the TESTed repository.\nWe recommend creating a new repository for your exercises.\n\n### 2. Create a test suite\n\nThe next step is to design a test suite that will be used to evaluate submission for the exercise.\nAgain, to keep things simple, we will only include a single test case in the test suite.\n\n```yaml\n- tab: Echo\n  testcases:\n    - expression: \"echo('input-1')\"\n      return: \"input-1\"\n```\n\nThis test suite describes the following tests: we have one tab, which is named `Echo`.\nInside this tab, there is one test case, in which we call the function `echo` with the string argument `\"input-1\"`.\nThe expected output is a return value (again a string) of `\"input-1\"`.\nAll other tests use the defaults: for example, no output is allowed on stderr, while stdout is ignored. \n\nPut the file containing the test suite in the following location:\n\n```bash\n# Create the file\n$ touch exercise/simple-example/suite.yaml\n# Now you should put the content from above in the file.\n```\n\n### 3. Create some submissions\n\nNow create two Python submissions for the programming exercise.\nThe first one contains a correct solution, and the second one returns the wrong result.\n\n```bash\n$ cat exercise/simple-example/correct.py\ndef echo(argument):\n  return argument\n$ cat exercise/simple-example/wrong.py\ndef echo(argument):\n  # Oops, this is wrong.\n  return argument * 2\n```\n\n### 4. Evaluate the submissions\n\nTo evaluate a submission with TESTed, you need to provide a test suite and configuration information.\nThis information can be piped to TESTed via stdin, but to make things easier, we will add the information to a configuration file in the directory of the exercise.\nIn practice, this configuration file would be created by the learning environment in which TESTed is integrated.\n\n```bash\n$ cat exercise/simple-example/config.json\n{\n  \"programming_language\": \"python\",\n  \"natural_language\": \"en\",\n  \"resources\": \"exercise/simple-example/\",\n  \"source\": \"exercise/simple-example/correct.py\",\n  \"judge\": \".\",\n  \"workdir\": \"workdir/\",\n  \"test_suite\": \"suite.yaml\",\n  \"memory_limit\": 536870912,\n  \"time_limit\": 60\n}\n```\n\nThese attributes are used by TESTed:\n\n- `programming_language`: programming language of the submission\n- `resources`: path of a directory with resources TESTed can use\n- `source`: path of the submission that must be evaluated\n- `judge`: path of the root directory of TESTEd\n- `workdir`: path of a temporary directory (see below)\n- `test_suite`: path of the test suite, relative to the resources directory (as defined above)\n\nBefore evaluating a submission, TESTed generates test code in the workdir.\nCreate that directory:\n\n```bash\n$ mkdir workdir/\n```\n\nThe content in this directory stays in place after TESTed finishes its evaluation, so you can inspect the generated test code.\nBefore running TESTed again, you'll need to clear this directory.\n\nWith this command, TESTed will evaluate the submission and generate feedback on stdout.\n\n```bash\n$ python -m tested -c exercise/simple-example/config.json\n{\"command\": \"start-judgement\"}\n{\"title\": \"Echo\", \"command\": \"start-tab\"}\n{\"command\": \"start-context\"}\n{\"description\": {\"description\": \"echo('input-1')\", \"format\": \"python\"}, \"command\": \"start-testcase\"}\n{\"expected\": \"input-1\", \"channel\": \"return (String)\", \"command\": \"start-test\"}\n{\"generated\": \"input-1\", \"status\": {\"enum\": \"correct\"}, \"command\": \"close-test\"}\n{\"command\": \"close-testcase\"}\n{\"command\": \"close-context\"}\n{\"command\": \"close-tab\"}\n{\"command\": \"close-judgement\"}\n```\nBy default, TESTed generates its feedback on stdout. The feedback is formatted in the [JSON Lines](https://jsonlines.org/) text format, meaning that each line contains a JSON object. Here's how you get an overview of all options supported by TESTed:\n\n```bash\n$ python -m tested --help\nusage: __main__.py [-h] [-c CONFIG] [-o OUTPUT] [-v]\n\nThe programming-language-agnostic educational test framework.\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -c CONFIG, --config CONFIG\n                        Where to read the config from\n  -o OUTPUT, --output OUTPUT\n                        Where the judge output should be written to.\n  -v, --verbose         Include verbose logs. It is recommended to also use -o in this case.\n```\n\nAdjust the configuration file if you want to evaluate the wrong submission.\n\nFor reference, the file `tested/dsl/schema.json` contains the JSON Schema of the test suite format.\n\n## Running TESTed locally\n\nThe `python -m tested` command is intended for production use.\nHowever, it is not always convenient to create a `config.json` file for each exercise to run.\n\nTESTed supports two ways of running TESTed without a config file.\nThe first way is:\n\n```bash\n# Run a hard-coded exercise with logs enabled, useful for debugging\n$ python -m tested.manual\n```\n\nThis command is useful when debugging TESTed itself or a particularly challenging exercise.\nIt will execute a hardcoded config, which is set in `tested/manual.py`.\n\nThe second way is:\n\n```bash\n# Run an exercise with CLI paramaters\n$ python -m tested.cli --help\nusage: cli.py [-h] -e EXERCISE [-s SUBMISSION] [-t TESTSUITE] [-f] [-v] [-d] [-p PROGRAMMING_LANGUAGE]\n\nSimple CLI for TESTed\n\noptions:\n  -h, --help            show this help message and exit\n  -e EXERCISE, --exercise EXERCISE\n                        Path to a directory containing an exercise\n  -s SUBMISSION, --submission SUBMISSION\n                        Path to a submission to evaluate\n  -t TESTSUITE, --testsuite TESTSUITE\n                        Path to a test suite\n  -f, --full            If the output should be shown in full (default: false)\n  -v, --verbose         If the judge should be verbose in its output (default: false)\n  -d, --debug           If the judge should be outputing the debug messages (default: false)\n  -p PROGRAMMING_LANGUAGE, --programming_language PROGRAMMING_LANGUAGE\n                        The programming language to use\n\nadditional information: The CLI only looks at a config.json file in the exercise directory. It does not look in folders above the exercise directory.\n```\n\nThis is the \"CLI mode\": here you can pass various options as command line parameters.\nFor example, for exercises following a standardized directory structure, the path to the exercise folder is often enough.\n\n## TESTed repository\n\nThe repository of TESTed is organized as follows:\n\n- `tested`: Python code of the actual judge (run by Dodona)\n- `tests`: unit tests for TESTed\n\n\n## Useful commands\n\nYou can run the basic unit tests with:\n\n```bash\npytest tests/test_functionality.py\n```\n\nYou can run the full test suite with:\n\n```bash\npytest -n auto tests/\n```\n\nWe use `black` and `isort` for code formatting. `pyright` is used for type checking.\nYou can run them with:\n\n```bash\nblack ./tested ./tests\nisort ./tested ./tests\npyright ./tested ./tests\n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdodona-edu%2Funiversal-judge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdodona-edu%2Funiversal-judge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdodona-edu%2Funiversal-judge/lists"}