https://github.com/johnhany97/grader
Automatic Programming Assignments Grader in Go
https://github.com/johnhany97/grader
assignments go grader job-queue programming
Last synced: 10 months ago
JSON representation
Automatic Programming Assignments Grader in Go
- Host: GitHub
- URL: https://github.com/johnhany97/grader
- Owner: johnhany97
- Created: 2019-03-17T21:11:22.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2019-04-25T00:31:36.000Z (over 6 years ago)
- Last Synced: 2025-01-13T21:34:21.013Z (12 months ago)
- Topics: assignments, go, grader, job-queue, programming
- Language: Go
- Homepage:
- Size: 35.5 MB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Automatic Programming Assignments Grader
```
Usage of main.go:
-schema string
Marking scheme to follow when grading the assignment (required)
```
## Testing & Benchmarking
To run the test suite, simply execute the following command:
```
go test -v -cover ./...
```
To run the benchmarks, simply execute the following command:
```
go test -v -run="none" -bench=. -benchmem ./... -benchtime="5s"
```
## Installation
First off, you'll need to [Install Go](https://golang.org/doc/install) and make sure enviornment variables are set up properly. Following that, you'll need to [Install Docker](https://docs.docker.com/v17.09/engine/installation/). To be able to run Java Style checking test tasks, you'll need to install [Checkstyle](http://checkstyle.sourceforge.net/index.html) as well.
Running the following two commands install two Go-based projects that run the grading.
```
go get -u github.com/docker-exec/dexec
go get -u github.com/johnhany97/grader
```
Note: We use a modified version of `dexec` which supports timing out executions. Changes should've been merged, however, ensure you have version `1.0.9-SNAPSHOT` of `dexec` installed before continuing.
That customized version could also be found at [https://github.com/johnhany97/dexec](https://github.com/johnhany97/dexec)
## Supported languages
Following is a list of supported languages with the required docker image.
- `C`: `dexec/lang-c`
- `Clojure`: `dexec/lang-clojure`
- `CoffeeScript`: `dexec/lang-coffee`
- `C++`: `dexec/lang-cpp`
- `C#`: `dexec/lang-csharp`
- `D`: `dexec/lang-d`
- `Erlang`: `dexec/lang-erlang`
- `F#`: `dexec/lang-fsharp`
- `Go`: `dexec/lang-go`
- `Groovy`: `dexec/lang-groovy`
- `Haskell`: `dexec/lang-haskell`
- `Java`: `dexec/lang-java` AND `johnhany97/grader-junit`
- `Lisp`: `dexec/lang-lisp`
- `Lua`: `dexec/lang-lua`
- `JavaScript`: `dexec/lang-node`
- `Nim`: `dexec/lang-nim`
- `Objective C`: `dexec/lang-objc`
- `OCaml`: `dexec/lang-ocaml`
- `Perl 6`: `dexec/lang-perl6`
- `Perl`: `dexec/lang-perl`
- `PHP`: `dexec/lang-php`
- `Python`: `dexec/lang-python`
- `R`: `dexec/lang-r`
- `Racket`: `dexec/lang-racket`
- `Ruby`: `dexec/lang-ruby`
- `Rust`: `dexec/lang-rust`
- `Scala`: `dexec/lang-scala`
- `Bash`: `dexec/lang-bash`
## Example usage
The following commands are to be ran from the project's root directory.
```
grader -schema="examples/inputOutput/javaSchema.json"
```
```
grader -schema="examples/python/pySchema.json"
```
```
grader -schema="examples/inputOutput/goSchema.json"
```
## Schema Properties
- `file` - Name of the file being graded, should contain it's extension - required
- `language` - Name of the programming language in which the file being graded is written - optional
- `className` - Class name of the file provided, usually is the name without the extension - required by junit and pyunit tests
- `folder` - Folder within which the file provided is located - required
- `tests` - Array of test tasks - required (even if just empty array)
- `outfile` - Whether to store results in a file or just output it in the terminal - default = ''
### Test Tasks
Currently supported are 4 different types of test tasks. They should confirm to the following format:
```
{
"type": "TYPE_OF_TEST", // io - junit - pyunit - output
// io --- start
"input": [ " " ], // Array of strings where each string is a line in the input. They are joined by \n
"expectedOutput": [ " " ], // Array of strings where each string is a line in the expected output. They are joined by \n
// io --- end
// junit/pyunit --- start
"unitTest": "", // The test to be injected in the shell file, after being escaped
// junit/pyunit --- end
// output --- start
"expectedOutput": [ " " ], // Array of strings where each string is a line in the expected output. They are joined by \n
// output --- end
}
```
## Example Schema
```JSON
{
"file": "Solution.java",
"language": "java",
"className": "Solution",
"folder": "examples/inputOutput/",
"tests": [
{
"type": "javaStyle"
},
{
"type": "io",
"description": "Testing normal boundaries",
"input": [
"3",
"1",
"2",
"3"
],
"expectedOutput": [
"1",
"2",
"3"
]
},
{
"type": "io",
"input": [
"2",
"2",
"5"
],
"expectedOutput": [
"2",
"5"
]
},
{
"type": "junit",
"unitTest": "@Test\n public void adderWorksWithZero() {\n Solution s = new Solution();\n int actual = s.adder(0, 3);\n assertEquals(3, actual);\n }"
}
],
"outfile": ""
}
```
Output: (Order may vary as tasks are finished by the job worker)
```JSON
[
{
"test":{
"type":"javaStyle",
"description":"",
"input":null,
"expectedOutput":null,
"unitTest":""
},
"description":"Results generated by Checkstyle. A static analyzer built for analyzing Java Programs.",
"stdOut":"Starting audit...\n[WARN] /Users/john/projects/dissertation/grader/examples/inputOutput/Solution.java:1: Using the '.*' form of import should be avoided - java.util.*. [AvoidStarImport]\n[WARN] /Users/john/projects/dissertation/grader/examples/inputOutput/Solution.java:4:3: Missing a Javadoc comment. [JavadocMethod]\n[WARN] /Users/john/projects/dissertation/grader/examples/inputOutput/Solution.java:16: Comment has incorrect indentation level 0, expected is 2, indentation should be the same level as line 18. [CommentsIndentation]\n[WARN] /Users/john/projects/dissertation/grader/examples/inputOutput/Solution.java:16: Line is longer than 100 characters (found 211). [LineLength]\nAudit done.",
"stdErr":"",
"successful":false,
"similarity":0,
"timedOut":false
},
{
"test":{
"type":"io",
"description":"Testing normal boundaries",
"input":[
"3",
"1",
"2",
"3"
],
"expectedOutput":[
"1",
"2",
"3"
],
"unitTest":""
},
"description":"Expected:\n1\n2\n3\nGot:\n1\n2\n3\nTest passed: true",
"stdOut":"1\n2\n3",
"stdErr":"",
"successful":true,
"similarity":1,
"timedOut":false
},
{
"test":{
"type":"io",
"description":"",
"input":[
"2",
"2",
"5"
],
"expectedOutput":[
"2",
"5"
],
"unitTest":""
},
"description":"Expected:\n2\n5\nGot:\n2\n5\nTest passed: true",
"stdOut":"2\n5",
"stdErr":"",
"successful":true,
"similarity":1,
"timedOut":false
},
{
"test":{
"type":"junit",
"description":"",
"input":null,
"expectedOutput":null,
"unitTest":"@Test\n public void adderWorksWithZero() {\n Solution s = new Solution();\n int actual = s.adder(0, 3);\n assertEquals(3, actual);\n }"
},
"description":"",
"stdOut":"JUnit version 4.10\r\n.\r\nTime: 0.009\r\n\r\nOK (1 test)",
"stdErr":"",
"successful":true,
"similarity":0,
"timedOut":false
}
]
```
## Test and Benchmark runs
Last Test run:
```
=== RUN TestExecute
--- PASS: TestExecute (8.65s)
=== RUN TestExecuteWithInput
--- PASS: TestExecuteWithInput (8.89s)
=== RUN TestExecuteJUnitTests
--- PASS: TestExecuteJUnitTests (4.23s)
=== RUN TestExecutePyUnitTests
--- PASS: TestExecutePyUnitTests (2.38s)
=== RUN TestExecuteJavaStyle
--- PASS: TestExecuteJavaStyle (0.66s)
PASS
coverage: 77.9% of statements
ok github.com/johnhany97/grader/processors 24.823s coverage: 77.9% of statements
=== RUN TestRunTestInputOutputTestHandler
--- PASS: TestRunTestInputOutputTestHandler (28.10s)
input_output_test.go:98: Successfully graded the solution for .java language
input_output_test.go:98: Successfully graded the solution for .py language
input_output_test.go:98: Successfully graded the solution for .cpp language
input_output_test.go:98: Successfully graded the solution for .cs language
input_output_test.go:98: Successfully graded the solution for .java language
input_output_test.go:98: Successfully graded the solution for .py language
input_output_test.go:98: Successfully graded the solution for .cpp language
input_output_test.go:98: Successfully graded the solution for .cs language
input_output_test.go:98: Successfully graded the solution for .java language
input_output_test.go:98: Successfully graded the solution for .py language
input_output_test.go:98: Successfully graded the solution for .cpp language
input_output_test.go:98: Successfully graded the solution for .cs language
=== RUN TestNewResultInputOutputTestHandler
--- PASS: TestNewResultInputOutputTestHandler (0.00s)
input_output_test.go:187: Successfully obtained the expected result
input_output_test.go:187: Successfully obtained the expected result
=== RUN TestHandleErrInputOutputTestHandler
--- PASS: TestHandleErrInputOutputTestHandler (0.00s)
=== RUN TestRunTestJavaStyleTestHandler
--- PASS: TestRunTestJavaStyleTestHandler (0.62s)
java_style_test.go:49: Successfully style checked the solution for .java language
=== RUN TestNewResultJavaStyleTestHandler
--- PASS: TestNewResultJavaStyleTestHandler (0.00s)
java_style_test.go:119: Successfully obtained the expected result
java_style_test.go:119: Successfully obtained the expected result
=== RUN TestHandleErrJavaStyleTestHandler
--- PASS: TestHandleErrJavaStyleTestHandler (0.00s)
=== RUN TestRunTestJUnitTestHandler
--- PASS: TestRunTestJUnitTestHandler (8.23s)
junit_test.go:65: Successfully graded the solution for .java language
junit_test.go:65: Successfully graded the solution for .java language
=== RUN TestNewResultJUnitTestHandler
--- PASS: TestNewResultJUnitTestHandler (0.00s)
junit_test.go:138: Successfully obtained the expected result
junit_test.go:138: Successfully obtained the expected result
=== RUN TestHandleErrJUnitTestHandler
--- PASS: TestHandleErrJUnitTestHandler (0.00s)
=== RUN TestRunTestOutputTestHandler
--- PASS: TestRunTestOutputTestHandler (8.67s)
output_test.go:66: Successfully graded the solution for .java language
output_test.go:66: Successfully graded the solution for .py language
output_test.go:66: Successfully graded the solution for .cpp language
output_test.go:66: Successfully graded the solution for .cs language
=== RUN TestNewResultOutputTestHandler
--- PASS: TestNewResultOutputTestHandler (0.00s)
output_test.go:145: Successfully obtained the expected result
output_test.go:145: Successfully obtained the expected result
=== RUN TestHandleErrOutputTestHandler
--- PASS: TestHandleErrOutputTestHandler (0.00s)
=== RUN TestRunTestPyUnitTestHandler
--- PASS: TestRunTestPyUnitTestHandler (3.36s)
pyunit_test.go:54: Successfully graded the solution for .py language
pyunit_test.go:54: Successfully graded the solution for .py language
=== RUN TestNewResultPyUnitTestHandler
--- PASS: TestNewResultPyUnitTestHandler (0.00s)
pyunit_test.go:109: Successfully obtained the expected result
pyunit_test.go:109: Successfully obtained the expected result
=== RUN TestHandleErrPyUnitTestHandler
--- PASS: TestHandleErrPyUnitTestHandler (0.00s)
PASS
coverage: 93.3% of statements
ok github.com/johnhany97/grader/test 48.986s coverage: 93.3% of statements
```
Last Benchmark Run:
```
goos: darwin
goarch: amd64
pkg: github.com/johnhany97/grader/processors
BenchmarkExecute-12 2 2687419688 ns/op 18880 B/op 177 allocs/op
BenchmarkExecuteWithInput-12 2 2746329358 ns/op 19224 B/op 183 allocs/op
BenchmarkExecuteJUnitTests-12 2 4290938234 ns/op 12016 B/op 100 allocs/op
BenchmarkExecutePyUnitTests-12 5 1649888741 ns/op 18478 B/op 182 allocs/op
BenchmarkExecuteJavaStyle-12 10 622639507 ns/op 13876 B/op 115 allocs/op
PASS
ok github.com/johnhany97/grader/processors 53.897s
goos: darwin
goarch: amd64
pkg: github.com/johnhany97/grader/test
BenchmarkRunTestInputOutputTestHandler-12 2 2855917014 ns/op 20500 B/op 197 allocs/op
BenchmarkNewResultInputOutputTestHandler-12 20000000 430 ns/op 96 B/op 7 allocs/op
BenchmarkHandleErrInputOutputTestHandler-12 200000000 45.9 ns/op 0 B/op 0 allocs/op
BenchmarkRunTestJavaStyleTestHandler-12 10 664239069 ns/op 14096 B/op 117 allocs/op
BenchmarkNewResultJavaStyleTestHandler-12 200000000 44.0 ns/op 0 B/op 0 allocs/op
BenchmarkHandleErrJavaStyleTestHandler-12 200000000 46.9 ns/op 0 B/op 0 allocs/op
BenchmarkRunTestJUnitTestHandler-12 2 4260429266 ns/op 13772 B/op 105 allocs/op
BenchmarkNewResultJUnitTestHandler-12 100000000 57.2 ns/op 0 B/op 0 allocs/op
BenchmarkHandleErrJUnitTestHandler-12 200000000 37.6 ns/op 0 B/op 0 allocs/op
BenchmarkRunTestOutputTestHandler-12 2 2676039517 ns/op 20284 B/op 188 allocs/op
BenchmarkNewResultOutputTestHandler-12 20000000 399 ns/op 92 B/op 6 allocs/op
BenchmarkHandleErrOutputTestHandler-12 200000000 45.6 ns/op 0 B/op 0 allocs/op
BenchmarkNewResultPyUnitTestHandler-12 100000000 56.4 ns/op 0 B/op 0 allocs/op
BenchmarkHandleErrPyUnitTestHandler-12 200000000 36.7 ns/op 0 B/op 0 allocs/op
PASS
ok github.com/johnhany97/grader/test 141.407s
```