{"id":22281082,"url":"https://github.com/ls1intum/ares","last_synced_at":"2025-09-05T10:35:14.011Z","repository":{"id":37941036,"uuid":"233112933","full_name":"ls1intum/Ares","owner":"ls1intum","description":"The Artemis Java Test Sandbox. A JUnit 5 Extension for Easy and Secure Artemis Java Testing","archived":false,"fork":false,"pushed_at":"2025-08-23T07:31:18.000Z","size":1639,"stargazers_count":24,"open_issues_count":18,"forks_count":7,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-08-31T00:36:10.910Z","etag":null,"topics":["ajts","ares","artemis","deadline","education","exercise","feedback","java","jqwik","junit","junit5","sandbox","students","teaching","test","test-framework","testing","thread","tum"],"latest_commit_sha":null,"homepage":"https://ls1intum.github.io/Ares/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ls1intum.png","metadata":{"files":{"readme":"README.adoc","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-01-10T19:04:44.000Z","updated_at":"2025-08-29T14:14:20.000Z","dependencies_parsed_at":"2023-02-17T18:16:06.072Z","dependency_job_id":"c4f87672-4269-4026-9df3-9163b62d3295","html_url":"https://github.com/ls1intum/Ares","commit_stats":null,"previous_names":["ls1intum/artemis-java-test-sandbox"],"tags_count":84,"template":false,"template_full_name":null,"purl":"pkg:github/ls1intum/Ares","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ls1intum%2FAres","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ls1intum%2FAres/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ls1intum%2FAres/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ls1intum%2FAres/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ls1intum","download_url":"https://codeload.github.com/ls1intum/Ares/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ls1intum%2FAres/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273746939,"owners_count":25160647,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-05T02:00:09.113Z","response_time":402,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ajts","ares","artemis","deadline","education","exercise","feedback","java","jqwik","junit","junit5","sandbox","students","teaching","test","test-framework","testing","thread","tum"],"created_at":"2024-12-03T16:13:46.419Z","updated_at":"2025-09-05T10:35:13.988Z","avatar_url":"https://github.com/ls1intum.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":":title: Ares\r\n:description: The Artemis Java Test Sandbox\r\n:keywords: java, testing, students, deadline, education, tum, test, feedback, sandbox, thread, exercise, teaching, junit, test-framework, ares, junit5, artemis, jqwik, ajts\r\n:author: Christian Femers\r\n:showtitle:\r\n:sectnums:\r\n:toc: preamble\r\n:toclevels: 3\r\n:icons: font\r\n\r\n= Ares\r\n\r\nThe Artemis Java Test Sandbox\r\n\r\nimage:https://github.com/ls1intum/Ares/workflows/Java%20CI/badge.svg?branch=main[Java CI]\r\nhttps://maven-badges.herokuapp.com/maven-central/de.tum.in.ase/artemis-java-test-sandbox[image:https://img.shields.io/maven-central/v/de.tum.in.ase/artemis-java-test-sandbox[Maven Central]]\r\nimage:https://img.shields.io/nexus/s/de.tum.in.ase/artemis-java-test-sandbox?label=latest%20snapshot\u0026server=https%3A%2F%2Foss.sonatype.org[Sonatype Nexus (Snapshots)]\r\n\r\n// ----------------------------------------------------------\r\n// Display the following for standard GitHub AsciiDoc display\r\n// ----------------------------------------------------------\r\nifdef::env-github[]\r\n\r\nhttps://ls1intum.github.io/Ares/[*View this documentation on GitHub Pages!*]\r\n\r\nendif::env-github[]\r\n\r\nAres, also known as Artemis Java Test Sandbox, is a JUnit\r\n5 extension for easy and secure Java testing on the interactive learning\r\nplatform https://github.com/ls1intum/Artemis[Artemis].\r\n\r\nIts main features are:\r\n\r\n* a security manager to prevent students crashing the tests or cheating\r\n* more robust tests and builds due to limits on time, threads and io\r\n* support for public and hidden Artemis tests, where hidden ones obey a custom deadline\r\n* utilities for improved feedback in Artemis like processing multiline error\r\n  messages or pointing to a possible location that caused an Exception\r\n* utilities to test exercises using System.out and System.in comfortably\r\n\r\n*Project Status*\r\n\r\nhttps://sonarcloud.io/dashboard?id=ls1intum_Ares[image:https://sonarcloud.io/api/project_badges/measure?project=ls1intum_Ares\u0026metric=alert_status[Quality Gate Status]]\r\nhttps://sonarcloud.io/dashboard?id=ls1intum_Ares[image:https://sonarcloud.io/api/project_badges/measure?project=ls1intum_Ares\u0026metric=sqale_rating[Maintainability Rating]]\r\nhttps://sonarcloud.io/dashboard?id=ls1intum_Ares[image:https://sonarcloud.io/api/project_badges/measure?project=ls1intum_Ares\u0026metric=security_rating[Security Rating]]\r\nhttps://sonarcloud.io/dashboard?id=ls1intum_Ares[image:https://sonarcloud.io/api/project_badges/measure?project=ls1intum_Ares\u0026metric=reliability_rating[Reliability Rating]]\r\n\r\nhttps://sonarcloud.io/dashboard?id=ls1intum_Ares[image:https://sonarcloud.io/api/project_badges/measure?project=ls1intum_Ares\u0026metric=ncloc[Lines of Code]]\r\nhttps://sonarcloud.io/dashboard?id=ls1intum_Ares[image:https://sonarcloud.io/api/project_badges/measure?project=ls1intum_Ares\u0026metric=coverage[Coverage]]\r\nhttps://sonarcloud.io/dashboard?id=ls1intum_Ares[image:https://sonarcloud.io/api/project_badges/measure?project=ls1intum_Ares\u0026metric=sqale_index[Technical Debt]]\r\nhttps://sonarcloud.io/dashboard?id=ls1intum_Ares[image:https://sonarcloud.io/api/project_badges/measure?project=ls1intum_Ares\u0026metric=vulnerabilities[Vulnerabilities]]\r\nhttps://sonarcloud.io/dashboard?id=ls1intum_Ares[image:https://sonarcloud.io/api/project_badges/measure?project=ls1intum_Ares\u0026metric=bugs[Bugs]]\r\nhttps://sonarcloud.io/dashboard?id=ls1intum_Ares[image:https://sonarcloud.io/api/project_badges/measure?project=ls1intum_Ares\u0026metric=duplicated_lines_density[Duplicated Lines (%)]]\r\n\r\n== Installation\r\n\r\n_Note: Ares requires at least Java 17._\r\n\r\nAres is provided as Maven/Gradle dependency. To use Ares in the test\r\nenvironment of a Maven project, include\r\n\r\n[source,xml]\r\n----\r\n\u003cdependency\u003e\r\n    \u003cgroupId\u003ede.tum.in.ase\u003c/groupId\u003e\r\n    \u003cartifactId\u003els1intum_Ares\u003c/artifactId\u003e\r\n    \u003cversion\u003e1.14.0\u003c/version\u003e\r\n    \u003cscope\u003etest\u003c/scope\u003e\r\n\u003c/dependency\u003e\r\n----\r\n\r\nin the `dependencies` section.\r\n\r\nFor Gradle projects, include\r\n[source,groovy]\r\n----\r\ntestImplementation 'de.tum.in.ase:ls1intum_Ares:1.14.0'\r\n----\r\n\r\nin the `dependencies` section.\r\n\r\nYou can now remove dependencies to JUnit 5, AssertJ and Hamcrest if\r\npresent because Ares already includes them. If you want to use jqwik (\u003e=\r\n1.2.4) or JUnit 4 (JUnit 5 vintage), simply include them in the\r\ndependencies section.\r\n\r\n== Basic Usage\r\n\r\n_Ares provides a high level of security, which comes at the cost of\r\nusability. Several steps need to be taken in order to make tests work\r\nproperly, and it might require some time to understand what Ares does.\r\nPlease study at least this complete basic usage guide before using Ares\r\nin production._\r\n\r\n=== Setup\r\n\r\nAssume you have a Java 17 Maven project, and the inside of `pom.xml`\r\nlooks like this:\r\n\r\n[source,xml]\r\n----\r\n\u003cproperties\u003e\r\n    \u003cmaven.compiler.source\u003e17\u003c/maven.compiler.source\u003e\r\n    \u003cmaven.compiler.target\u003e17\u003c/maven.compiler.target\u003e\r\n\u003c/properties\u003e\r\n\r\n\u003cdependencies\u003e\r\n    \u003cdependency\u003e\r\n        \u003cgroupId\u003ede.tum.in.ase\u003c/groupId\u003e\r\n        \u003cartifactId\u003els1intum_Ares\u003c/artifactId\u003e\r\n        \u003cversion\u003e1.14.0\u003c/version\u003e\r\n        \u003cscope\u003etest\u003c/scope\u003e\r\n    \u003c/dependency\u003e\r\n\u003c/dependencies\u003e\r\n----\r\n\r\nConsider the following student class that needs to be tested:\r\n\r\n[source,java]\r\n----\r\nimport java.util.Objects;\r\n\r\npublic final class Penguin {\r\n\r\n    private final String name;\r\n\r\n    public Penguin(String name) {\r\n        this.name = Objects.requireNonNull(name, \"name must not be null\");\r\n    }\r\n\r\n    public String getName() {\r\n        return name;\r\n    }\r\n}\r\n----\r\n\r\nAnd you have already written the following simple JUnit 5 test class:\r\n\r\n[source,java]\r\n----\r\nimport static org.junit.jupiter.api.Assertions.*;\r\nimport org.junit.jupiter.api.Test;\r\n\r\npublic class PenguinTest {\r\n\r\n    @Test\r\n    void testPenguinPublic() {\r\n        Penguin pingu = new Penguin(\"Julian\");\r\n        assertEquals(\"Julian\", pingu.getName(), \"getName() does not return the name supplied to the contructor\");\r\n    }\r\n\r\n    @Test\r\n    void testPenguinHidden() {\r\n        assertThrows(NullPointerException.class, () -\u003e new Penguin(null));\r\n    }\r\n}\r\n----\r\n\r\nIn this example,\r\n\r\n- `testPenguinPublic()` is supposed to be executed\r\n  after each push and directly give the students their feedback, while\r\n- `testPenguinHidden()` should be executed only after the exercise\r\n  deadline, and the results should not be visible before the deadline.\r\n\r\nWhile Artemis has a feature to mark test cases as hidden, this will not\r\nprevent the contents of the test case leaking through static variables,\r\nfiles and similar, be it accidentally or on purpose. To prevent that,\r\n*the hidden test case must not be executed before the deadline at all.*\r\n\r\nThe public test case does not need to be hidden, as its purpose is to\r\ngive direct feedback. However, there are still multiple possible\r\nproblems like crashing the Maven build by `System.exit(0)` or containing\r\nan endless loop. Both can have a negative impact on the interactive\r\nlearning experience because the students get confronted with an\r\nincomprehensible log of a failed build. Such errors can be explained,\r\nbut that takes a lot of time, especially if it happens a lot (and it\r\nwill, if the number of students is sufficiently large).\r\n\r\nIt is also a security concern again, students could try to read the\r\n`.java` files containing the test classes.\r\n\r\n=== Integrating Ares\r\n\r\nTherefore, we will use Ares to secure the tests and avoid unintelligible\r\nfeedback. The most basic way to do this is by using the `@Public` and\r\n`@Hidden` annotations:\r\n\r\n[source,java]\r\n----\r\nimport static org.junit.jupiter.api.Assertions.*;\r\nimport org.junit.jupiter.api.Test;\r\n\r\n// IMPORTANT: make sure to use the \"jupiter\" ones (if you are not using jqwik)\r\nimport de.tum.in.test.api.jupiter.Hidden;\r\nimport de.tum.in.test.api.jupiter.Public;\r\n\r\n// This example won't work just like that, see below why\r\npublic class PenguinTest {\r\n\r\n    @Public\r\n    @Test\r\n    void testPenguinPublic() {\r\n        Penguin pingu = new Penguin(\"Julian\");\r\n        assertEquals(\"Julian\", pingu.getName(), \"getName() does not return the name supplied to the contructor\");\r\n    }\r\n\r\n    @Hidden\r\n    @Test\r\n    void testPenguinHidden() {\r\n        assertThrows(NullPointerException.class, () -\u003e new Penguin(null));\r\n    }\r\n}\r\n----\r\n\r\nThe code above won’t work just like that, if you try to run it as is,\r\nyou will get the following reported by JUnit:\r\n`java.lang.annotation.AnnotationFormatError: cannot find a deadline for hidden test testPenguinHidden()`\r\n\r\nAres needs to know what the deadline is. We tell Ares with another\r\nannotation:\r\n\r\n[source,java]\r\n----\r\n// Format must be ISO_LOCAL_DATE(T| )ISO_LOCAL_TIME( ZONE_ID)?\r\n@Deadline(\"2020-06-09 03:14 Europe/Berlin\")\r\npublic class PenguinTest {\r\n    // ...\r\n}\r\n----\r\n\r\nThat annotation (like most of the Ares annotations) can also be placed\r\non the test method (and nested classes), if multiple are present, the\r\none that is closest to the test case is used.\r\n\r\nNow, it already works! Try to play around with the deadline in the\r\nannotation. If the given `LocalDateTime` lies in the past, the test case\r\nis executed and - together with the student code presented earlier -\r\npasses. If the deadline hasn’t passed, the test case won’t pass either.\r\nIt fails with\r\n`org.opentest4j.AssertionFailedError: hidden tests will be executed after the deadline.`\r\nand the test was not executed, as the deadline is always checked before\r\nany hidden test case is executed.\r\n\r\nYou might have noticed that we specify the time zone as well. Although\r\nthe annotation parser permits leaving it unspecified, this bears the\r\nrisk of (not) executing the tests at the correct time if the build\r\nagent's time zone is different from the one on your machine or what you\r\nwould expect it to be. If you run tests where the time zone is/was not\r\nset, Ares will warn you about that in the logs.\r\n\r\n=== What about Security?\r\n\r\nThe hidden test case was not executed and static variables cannot leak\r\nits contents. If you change `getName()` to\r\n\r\n[source,java]\r\n----\r\npublic String getName() {\r\n    System.exit(0);\r\n    return name;\r\n}\r\n----\r\n\r\nYou will now with Ares get the following error message:\r\n\r\n....\r\njava.lang.SecurityException: do not use System.exit(int)\r\n/// potential problem location: Penguin.getName(Penguin.java:12) ///\r\n....\r\n\r\nAs you might be able to see, Ares threw a SecurityException. But it also\r\nadded\r\n`/// potential problem location: Penguin.getName(Penguin.java:12) ///`.\r\nThis is the line from the stack trace which Ares thinks is most relevant\r\nfor the student, essentially, it searches for the uppermost stack frame\r\nthat is located in the student's code. Student code is basically\r\neverything that is not whitelisted.\r\n\r\nBut what is whitelisted?\r\n\r\n* The test class itself (in case of nested classes, the outermost class\r\n  is whitelisted) and therefore, _all_ its nested classes and methods,\r\n  too.\r\n* A predefined set of packages, like everything that starts with\r\n  `java.`, `sun.`, `com.sun.`, `org.junit`, `org.apache.`, …\r\n  Therefore, *never use such package names for student assignments!*\r\n* Single classes whitelisted using `@WhitelistClass` and\r\n  all classes matching `@AddTrustedPackage`\r\n* Additional package prefixes provided with by system property\r\n  `ares.security.trustedpackages` separated by a comma `,`.\r\n\r\nAres also grants permissions that are requested by certain actions\r\n(`System.exit`, File IO, Networking, Threads, …) based on whitelisted\r\nstack frames. *Ares granting a permission requires all stack frames to\r\nbe whitelisted.*\r\n\r\nAnother test: +\r\nAdding one of the following lines to `testPenguinPublic()`\r\nitself, and it will still pass using the correct student code:\r\n\r\n[source,java]\r\n----\r\nFiles.readString(Path.of(\"pom.xml\"));\r\n// or\r\nFiles.readString(Path.of(\"src/test/java/PenguinTest.java\")); // assuming default maven structure\r\n----\r\n\r\nIf you instead add one of the lines to the `getName()` method again, you\r\nwill get something like: +\r\n`java.lang.SecurityException: access to path src\\test\\java\\PenguinTest.java denied in line 16 in Penguin.java`. +\r\nWhich is exactly what you want, students should not be able to read the\r\ncode of the test classes. By default, student code has no access to any\r\npath, not even read access.\r\n\r\nBy the way, adding `@WhitelistClass(Penguin.class)` to the test class or\r\nmethod will make the test run fine again because `Penguin` is now\r\nwhitelisted and can therefore access all files without problems. *So\r\nnever whitelist classes that students can edit.*\r\n\r\n[#what-you-need-to-do-outside-ares]\r\n==== What you need to do outside Ares\r\n\r\nSadly, due to the way classes are loaded and the class path works when\r\ntesting student code with maven, there are still vulnerabilities if\r\nstudents manage to load classes that would be in trusted packages. This\r\nis especially problematic if they shadow library classes, such as\r\nJUnit’s `Assertions`.\r\n\r\nTo prevent that, you have to use the Maven Enforcer Plugin to make sure\r\nno student content lands in trusted packages:\r\n\r\nMaven:\r\n[source,xml]\r\n----\r\n\u003cplugin\u003e\r\n    \u003cgroupId\u003eorg.apache.maven.plugins\u003c/groupId\u003e\r\n    \u003cartifactId\u003emaven-enforcer-plugin\u003c/artifactId\u003e\r\n    \u003cversion\u003e3.1.0\u003c/version\u003e\r\n    \u003cexecutions\u003e\r\n        \u003cexecution\u003e\r\n            \u003cid\u003eenforce-no-student-code-in-trusted-packages\u003c/id\u003e\r\n            \u003cphase\u003eprocess-classes\u003c/phase\u003e \u003c!--1--\u003e\r\n            \u003cgoals\u003e\r\n                \u003cgoal\u003eenforce\u003c/goal\u003e\r\n            \u003c/goals\u003e\r\n        \u003c/execution\u003e\r\n    \u003c/executions\u003e\r\n    \u003cconfiguration\u003e\r\n        \u003crules\u003e\r\n            \u003crequireFilesDontExist\u003e\r\n                \u003cfiles\u003e\r\n                    \u003c!--2--\u003e\r\n                    \u003cfile\u003e${project.build.outputDirectory}/ch/qos/logback/\u003c/file\u003e\r\n                    \u003cfile\u003e${project.build.outputDirectory}/com/github/javaparser/\u003c/file\u003e\r\n                    \u003cfile\u003e${project.build.outputDirectory}/com/intellij/\u003c/file\u003e\r\n                    \u003cfile\u003e${project.build.outputDirectory}/com/sun/\u003c/file\u003e\r\n                    \u003cfile\u003e${project.build.outputDirectory}/de/tum/in/test/api/\u003c/file\u003e\r\n                    \u003cfile\u003e${project.build.outputDirectory}/java/\u003c/file\u003e\r\n                    \u003cfile\u003e${project.build.outputDirectory}/javax/\u003c/file\u003e\r\n                    \u003cfile\u003e${project.build.outputDirectory}/jdk/\u003c/file\u003e\r\n                    \u003cfile\u003e${project.build.outputDirectory}/net/jqwik/\u003c/file\u003e\r\n                    \u003cfile\u003e${project.build.outputDirectory}/org/apache/\u003c/file\u003e\r\n                    \u003cfile\u003e${project.build.outputDirectory}/org/assertj/\u003c/file\u003e\r\n                    \u003cfile\u003e${project.build.outputDirectory}/org/eclipse/\u003c/file\u003e\r\n                    \u003cfile\u003e${project.build.outputDirectory}/org/jacoco/\u003c/file\u003e\r\n                    \u003cfile\u003e${project.build.outputDirectory}/org/json/\u003c/file\u003e\r\n                    \u003cfile\u003e${project.build.outputDirectory}/org/junit/\u003c/file\u003e\r\n                    \u003cfile\u003e${project.build.outputDirectory}/org/opentest4j/\u003c/file\u003e\r\n                    \u003cfile\u003e${project.build.outputDirectory}/sun/\u003c/file\u003e\r\n                    \u003cfile\u003e${project.build.outputDirectory}/org/gradle/\u003c/file\u003e\r\n                    \u003cfile\u003e${project.build.outputDirectory}/worker/org/gradle/\u003c/file\u003e\r\n                \u003c/files\u003e\r\n            \u003c/requireFilesDontExist\u003e\r\n        \u003c/rules\u003e\r\n    \u003c/configuration\u003e\r\n\u003c/plugin\u003e\r\n----\r\n\r\nGradle:\r\n[source,groovy]\r\n----\r\n\r\ndef forbiddenPackageFolders = [ //\u003c2\u003e\r\n    \"$studentOutputDir/ch/qos/logback/\",\r\n    \"$studentOutputDir/com/github/javaparser/\",\r\n    \"$studentOutputDir/com/intellij/\",\r\n    \"$studentOutputDir/com/sun/\",\r\n    \"$studentOutputDir/de/tum/in/test/api/\",\r\n    \"$studentOutputDir/java/\",\r\n    \"$studentOutputDir/javax/\",\r\n    \"$studentOutputDir/jdk/\",\r\n    \"$studentOutputDir/net/jqwik/\",\r\n    \"$studentOutputDir/org/assertj/\",\r\n    \"$studentOutputDir/org/apache/\",\r\n    \"$studentOutputDir/org/eclipse/\",\r\n    \"$studentOutputDir/org/gradle/\",\r\n    \"$studentOutputDir/org/jacoco/\",\r\n    \"$studentOutputDir/org/json/\",\r\n    \"$studentOutputDir/org/junit/\",\r\n    \"$studentOutputDir/org/opentest4j/\",\r\n    \"$studentOutputDir/sun/\",\r\n    \"$studentOutputDir/worker/org/gradle/\"\r\n]\r\ntest {\r\n    doFirst { //\u003c1\u003e\r\n        for (String packageFolder in forbiddenPackageFolders) {\r\n            assert !file(packageFolder).exists(): \"$packageFolder must not exist within the submission.\"\r\n        }\r\n    }\r\n    // ...\r\n}\r\n----\r\n\u003c1\u003e Important: you want to enforce the non-existence of classes after\r\n    their generation but before testing.\r\n\u003c2\u003e This is where all folders/packages go that we don't want to exist\r\n    in student code. You will always find the most recent recommendation\r\n    for Ares here. If you use additional third-party libraries that need\r\n    to be configured using `@AddTrustedPackage`, you should add those\r\n    packages here as well. Ares will check that all entries are present. +\r\n    If you don't want Ares to do so, set the `ares.maven.ignore` or\r\n    `ares.gradle.ignore` system property to `true`. In case you want Ares\r\n    to look into a different file, you can set the `ares.maven.pom` or\r\n    `ares.gradle.build` to a path other than the default `pom.xml` or\r\n    `build.gradle`.\r\n\r\n=== Further Important Options\r\n\r\nAre we done now? With the most fundamental parts yes, but there is a bit\r\nmore you need to know about testing with Ares, as this was just a very\r\nbasic example with a single class and not much testing. Without further\r\nknowledge, you might not get Ares to work and consequently get rather\r\nannoyed or even enraged. To prevent that, please read on.\r\n\r\n==== Path Access and Class Loading\r\n\r\nYou can use `@WhitelistPath` and `@BlacklistPath` to control access to\r\npaths. By default, no access is granted, and so you need to use\r\n`@WhitelistPath` to give student code the permission to read and write\r\nfiles explicitly. You can specify exceptions using `@BlacklistPath`\r\nwhich will overpower the whitelisted paths.\r\n\r\n_The following examples will make use of `course1920xyz` as placeholder\r\nvalue for the real Artemis exercise name/id. Replace it with the real\r\none when borrowing code snippets, or nothing will work as expected._\r\n\r\nMost importantly, this does not only apply to explicit file IO, but also\r\nto the `.class` files that the class loader reads, as needed. This\r\nalready happens if one student class requires another one, that has not\r\nbeen loaded after that. You can recognize that in the standard error\r\noutput:\r\n\r\n....\r\n[WARN] [main] BAD PATH ACCESS: K:\\repo\\course1920xyz-solution\\bin\\some\\Thing.class (BL:false, WL:false)\r\n....\r\n\r\nThis usually means the class loader could not load the class. The\r\nparentheses show, that the problem is the missing whitelisting.\r\n*Therefore, all test setups should have some whitelisting.*\r\n\r\nA number of examples how you can whitelist paths in Ares:\r\n\r\n* `@WhitelistPath(\"\")` will grant read access to the paths in the\r\n  directory of execution, which is usually where the `pom.xml` is.\r\n* `@WhitelistPath(\"pom.xml\")` will allow students to read the `pom.xml`.\r\n* `@WhitelistPath(\"..\")` will allow read access to the level above the\r\n  maven project. In Eclipse, that is the level of your workspace.\r\n* `@WhitelistPath(value = \"../course1920xyz**\", type = PathType.GLOB)`\r\n  grants read access to projects beginning with the exercise \"id\" used\r\n  by Artemis. Should you use the Eclipse feature \"Referenced Projects\"\r\n  (or the analog to that in your IDE) to link the student/solution project\r\n  to the tests, you will need a setting like this.\r\n* `@WhitelistPath(value = \"data\", level = PathActionLevel.DELETE)` will\r\n  allow students to read, write and delete files in the `data` directory\r\n  and subdirectories.\r\n* `@WhitelistPath(\"target\")` allows reading files in target (Maven output folder)\r\n* `@BlacklistPath(value = \"**Test*.{java,class}\", type = PathType.GLOB)`\r\n  prevents access to classes in source code or compiled form that contain\r\n  `Test`. If you leave away the `*` after `Test`, nested classes are not\r\n  blacklisted. Student classes should not be called something with\r\n  \"Test\" then.\r\n\r\nThat was not everything but already quite a lot. Take a look at the\r\nJavadoc of the annotations and enums used, if you want to know more.\r\nBefore you give up, here is my recommendation how to start:\r\n\r\n[source,java]\r\n----\r\n@WhitelistPath(value = \"../course1920xyz**\", type = PathType.GLOB) // for manual assessment and development\r\n@WhitelistPath(\"target\") // mainly for Artemis\r\n@BlacklistPath(\"target/test-classes\") // prevent access to test-related classes and resources\r\n----\r\n\r\nAdd a `@BlacklistPath` for other important classes, like your reference\r\nimplementations of the solution to test against should you use that\r\napproach.\r\n\r\n_Note: the Artemis project starts with `course1920xyz`, but the build in\r\nBamboo (by Artemis) will happen in a directory named after the build\r\nplan, which is in upper case and therefore, begins with `COURSE1920XYZ`.\r\nMake sure that you do not build multiple student solutions in the same\r\ndirectory on the same machine using the git clone (lower case) approach.\r\nOtherwise, adjust the whitelisting to your needs._\r\n\r\n==== Timeouts\r\n\r\nJUnit already provides means of applying timeouts to tests. However,\r\nthose are _not strict_ in the sense of \"enforced in the strongest\r\npossible way\". What is meant by that?\r\n\r\nThere are three different ways how the timeouts can work:\r\n\r\n* like `org.junit.jupiter.api.Timeout` +\r\n  This timeout is not preemptive, and the test itself runs in the same\r\n  thread executing the tests. It will only try to stop the test via an\r\n  interrupt. If that fails like it does for an endless loop, the test\r\n  will definitively fail. After it is finished. Which might never\r\n  happen and the main reason not to use this when it comes to testing\r\n  unknown code.\r\n* like `org.junit.jupiter.api.Assertions.assertTimeoutPreemptively` +\r\n  This will fail the test preemptively by executing the `Executable`\r\n  argument itself in a different thread than the thread executing all\r\n  tests. It will only try to stop the test via an interrupt, but if\r\n  that fails it will simply carry on. The test thread might still run,\r\n  though.\r\n* like `de.tum.in.test.api.StrictTimeout` +\r\n  This uses a mechanism similar to `assertTimeoutPreemptively`, but\r\n  will resort to harder means if necessary.\r\n  It will in the following order:\r\n  1. wait the given duration\r\n  2. interrupt the thread executing the test and wait no longer (like\r\n     `assertTimeoutPreemptively`)\r\n  3. block the creation of new threads\r\n  4. interrupt all threads created during the test and try to join the\r\n     threads\r\n  5. if that fails, use `Thread.stop()` on all remaining threads\r\n     and try to join again\r\n  6. repeat step 5 multiple times, if required\r\n  7. Should that fail, report a special SecurityException that not all\r\n     threads could be stopped. (see the standard error output for a detailed\r\n     report then) _If that happens, no more tests can be properly executed\r\n     because the security cannot be guaranteed and the test cases cannot be\r\n     executed \"in isolation\". All following tests will fail._\r\n\r\n*Rule 1: When testing with Ares, always use `@StrictTimeout` for\r\ntimeouts, the others will not work reliably, especially in conjunction\r\nwith the Ares security.*\r\n\r\n*Rule 2: When writing tests for Artemis, always use `@StrictTimeout`.*\r\nThere is no reason to omit the timeout, since you do not know the code\r\nstudents will write. (And they will write code spawning millions of\r\nthreads in endless loops, which in turn will do the same recursively.)\r\n\r\n[#showing-standard-output]\r\n==== Showing Standard Output\r\n\r\nBy default, Ares will record standard and error output of each test\r\ninternally and not print it to the console. The recorded output can then\r\nbe obtained and tested, see\r\n\u003c\u003ctesting-console-interaction,`IOTester`\u003e\u003e The reason for this is on\r\nthe one hand to keep the console and logs short and clean and on the\r\nother hand prevent students from accidentally messing up the logs with\r\nmillions of lines. Ares also has a hard limit on the total number of\r\nprinted chars at around 10 million.\r\n\r\nTo mirror the output recorded by Ares to the console, use the\r\n`@MirrorOutput` annotation on the test class or method.\r\n\r\nIt is also worth noting that Ares enforces valid UTF-8\r\nbeing printed and throws an appropriate exception otherwise.\r\n\r\n==== Testing the Exercise before Release\r\n\r\nHidden tests will be executed by Ares only after the deadline. This\r\nposes the problem, how the exercise creators should work on the tasks,\r\ntests and the sample solution. One possible solution would be to use an\r\nalternative deadline annotation or change the deadline temporarily. The\r\nproblem is that it is quite likely one might forget to change it back\r\nagain, and protecting the hidden tests would fail.\r\n\r\nUse `@ActivateHiddenBefore` just like `@Deadline` to state the\r\nLocalDateTime before which hidden tests should be executed. This date\r\nshould, of course, be before the release of the exercise on Artemis.\r\n\r\n==== Extending a Deadline and Disability Compensation\r\n\r\nYou can use `@ExtendedDeadline` together with a duration like `1d` or\r\n`2d 12h 30m` to extend the deadline by the given amount.\r\n`@ExtendedDeadline(\"1d\")`, for example, extends the deadline by one day.\r\nIf you use the annotation on different levels (e.g. class and method)\r\nwithout stating a new deadline (e.g. deadline only on class level), the\r\nextensions will be added together.\r\n\r\n==== Threads and Concurrency\r\n\r\nBy default, Ares will not allow non-whitelisted code to use threads at\r\nall. That includes thread pools, but excludes the common pool and its\r\nusers, like parallel streams. To allow the use of Threads, use the\r\nannotation `@AllowThreads`. The number of active threads is also\r\nlimited, the default value of that is 1000, but can be changed in the\r\nannotation. Please keep in mind that this limit should not be larger\r\nthan 1000 to prevent performance and timeout chaos.\r\n\r\nNew threads are for security reasons not directly whitelisted by Ares\r\nand will not be allowed to do anything security critical. If you trust a\r\nthread (at least its entry point), you can explicitly request the thread\r\nto be whitelisted using\r\n`ArtemisSecurityManager.requestThreadWhitelisting(Thread)`. The thread\r\ncalling the method and its stack must be whitelisted, of course.\r\n\r\n[#testing-console-interaction]\r\n==== Testing Console Interaction\r\n\r\nOne example showing some possibilities here:\r\n\r\n[source,java]\r\n----\r\nvoid testSquareCorrect(IOTester tester) { //\u003c1\u003e\r\n    tester.provideInputLines(\"5\"); //\u003c2\u003e\r\n\r\n    InputOutputPenguin.calculateSquare(); //\u003c3\u003e\r\n\r\n    tester.err().assertThat().isEmpty(); //\u003c4\u003e\r\n    tester.out().assertThat().isEqualTo(\"\"\"\r\n                Enter Number:\r\n                Answer:\r\n                25\"\"\"); //\u003c5\u003e\r\n}\r\n----\r\n\u003c1\u003e Declare `IOTester` as parameter.\r\n\u003c2\u003e Provide input lines before calling the student code.\r\n    This content will be used for reading lines from `System.in`.\r\n\u003c3\u003e Call the student code to process the input and produce output.\r\n\u003c4\u003e Assert that nothing was printed to `System.err`.\r\n\u003c5\u003e Assert that the standard output (in this case excluding the final\r\n    line break) is equal to the given text block (if you use text\r\n    blocks, be aware of their newline handling).\r\n\r\nNote that Ares normalizes the line breaks to `\\n`, and\r\nhttps://github.com/ls1intum/Ares/blob/main/src/main/java/de/tum/in/test/api/io/OutputTester.java[`OutputTester`]\r\noffers many different approaches to checking output (e.g. single string, list of strings, ...).\r\n\r\nIf students read more lines than provided, they get the following feedback:\r\n...\r\njava.lang.IllegalStateException: no further console input request after the last(number 1: \"5\") expected.\r\n...\r\n\r\nSee also `IOTester` and for more examples, the\r\nhttps://github.com/ls1intum/Ares/blob/main/src/test/java/de/tum/in/test/integration/testuser/InputOutputUser.java[`InputOutputUser`]\r\ntest.\r\n\r\n\u003c\u003cshowing-standard-output\u003e\u003e covers how the student output is\r\nmanaged and shown in the test logs.\r\n\r\n[TIP]\r\n====\r\nIn case the default `IOTester` from Ares does not meet your requirements,\r\nyou can provide a custom implementation by applying `@WithIOManager(MyCustomOne.class)`\r\nto e.g. the test class or individual methods. This also allows you to register\r\na custom parameter to control IO testing with ease inside the test method.\r\nHave a look into the test class linked above to learn more or read the documentation of\r\nhttps://github.com/ls1intum/Ares/blob/main/src/main/java/de/tum/in/test/api/io/IOManager.java[`IOManager`].\r\n====\r\n\r\n==== Networking\r\n\r\nAres allows for local network connections by using the `@AllowLocalPort`\r\nannotation.\r\n\r\nThere are plenty of configuration options, and the code can get complicated\r\nquickly due to the threads required to test network connections.\r\nOne issue can be that waiting network connections block threads in such\r\na way that they cannot be stopped (waiting in native code), so we\r\nrecommend using timeouts for connections at least on one end consistently.\r\n\r\nFor examples, have a look at the test\r\nhttps://github.com/ls1intum/Ares/blob/main/src/test/java/de/tum/in/test/integration/testuser/NetworkUser.java[`NetworkUser`].\r\n\r\n==== Locale\r\n\r\nYou can set a locale for Ares (and the rest of Java) by adding the\r\n`@UseLocale` JUnit extension to classes/methods, which will set the Java\r\nhttps://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/Locale.html#setDefault(java.util.Locale)[default locale]\r\nthat is also used by Ares. The locale is changed only for the scope\r\nwhere the annotation is applied.\r\n\r\nAres is currently localized in German (`de_DE`) and English (`en_US`),\r\nwhere `en_US` is the fallback for any other locale.\r\n\r\nSee also the https://github.com/ls1intum/Ares/blob/main/src/test/java/de/tum/in/test/integration/testuser/LocaleUser.java[`LocaleUser`]\r\ntest for more examples.\r\n\r\n== Additional Notes\r\n\r\n=== Older Versions\r\n\r\nFor versions prior to `1.0.0`, a repository block had to be added to\r\n`\u003crepositories\u003e` section of the `pom.xml` that referenced the Maven\r\nrepository URL `https://gitlab.com/ajts-mvn/repo/raw/main/`.\r\n\r\n*Using older Ares versions is highly discouraged, remove these\r\nrepository declarations and update to the newest Ares version if they\r\nappear in your projects.*\r\n\r\n=== GitHub Packages\r\n\r\nGitHub Packages does currently not allow unregistered, public access to\r\nthe packages. Therefore, you will need to authenticate to GitHub if you\r\nuse\r\n\r\n[source,xml]\r\n----\r\n\u003crepositories\u003e\r\n    \u003crepository\u003e\r\n        \u003cid\u003eares\u003c/id\u003e\r\n        \u003cname\u003eAres Maven Packages\u003c/name\u003e\r\n        \u003curl\u003ehttps://maven.pkg.github.com/ls1intum/Ares\u003c/url\u003e\r\n    \u003c/repository\u003e\r\n\u003c/repositories\u003e\r\n----\r\n\r\n== License\r\n\r\nAres was initially created by Christian Femers and is maintained by Applied Education Technologies\r\nIt is licensed under the https://github.com/ls1intum/Ares/blob/main/LICENSE[MIT License, see `LICENSE.md`].\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fls1intum%2Fares","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fls1intum%2Fares","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fls1intum%2Fares/lists"}