{"id":23459582,"url":"https://github.com/d-led/java-testing-the-tests","last_synced_at":"2025-06-22T00:05:45.393Z","repository":{"id":219761755,"uuid":"748642016","full_name":"d-led/java-testing-the-tests","owner":"d-led","description":"Programming is not easy. Coverage numbers don't tell you the whole story. The build is broken intentionally in this repo. See the README for motivation \u0026 a walk-through","archived":false,"fork":false,"pushed_at":"2024-01-30T09:39:09.000Z","size":241,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-11T23:40:38.633Z","etag":null,"topics":["100-percent-coverage-not-enough","hypothesis-testing","java","mutation-testing","property-based-testing"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/d-led.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-01-26T12:51:21.000Z","updated_at":"2024-06-04T21:57:52.000Z","dependencies_parsed_at":"2025-04-11T21:37:57.868Z","dependency_job_id":"437657bb-6c48-4f01-98d9-4fa30c689437","html_url":"https://github.com/d-led/java-testing-the-tests","commit_stats":null,"previous_names":["d-led/java-testing-the-tests"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/d-led/java-testing-the-tests","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d-led%2Fjava-testing-the-tests","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d-led%2Fjava-testing-the-tests/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d-led%2Fjava-testing-the-tests/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d-led%2Fjava-testing-the-tests/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/d-led","download_url":"https://codeload.github.com/d-led/java-testing-the-tests/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d-led%2Fjava-testing-the-tests/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261212483,"owners_count":23125583,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["100-percent-coverage-not-enough","hypothesis-testing","java","mutation-testing","property-based-testing"],"created_at":"2024-12-24T06:16:01.470Z","updated_at":"2025-06-22T00:05:40.381Z","avatar_url":"https://github.com/d-led.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# java-testing-the-tests\n\n\u003e The build is broken by design in this repo\n\nChasing high line or even branch coverage is not the whole picture.\n\nThe quality lies in the expectations and assumptions of the people involved.\n\nThe assumptions and the explicit expectations are further verified via Property-Based Testing \u0026 Mutation Testing.\n\nIt's about **testing the tests**.\n\nThe system under test is a simple function:\n\n```java\nint sumOf(int a, int b) {\n    return a + b;\n}\n```\n\nand the test that would give a 100% coverage:\n\n```java\n@Test void summation() {\n    // how hard can 2+2 be?\n    assertThat(maths.sumOf(2, 2))\n        .isEqualTo(4)\n    ;\n}\n```\n\nYet in this simple example, with a 100% coverage *easily* attainable, the expected qualities may be insufficient.\n\n![hypothesis](doc/img/test_vs_hypothesis.png)\n\n[Source](app/src/test/java/com/siemens/code/dled/MathsProperties.java)\n\n## Mutation Testing\n\n\u003e How hard is it to test 2+2 (addition)?\n\nTesting the quality of test via mutation testing: https://pitest.org\n\n```bash\ngradle pitest\n```\n\nor\n\n```bash\n./gradlew pitest\n```\n\n\u0026darr;\n\ntest `2 + 2 == 4`:\n\nopen [`app/build/reports/pitest/com.siemens.code.dled/index.html`](app/build/reports/pitest/com.siemens.code.dled/index.html)\n\n![mutation coverage by class](doc/img/coverage_1.png)\n\n![the uncovered line](doc/img/maths_1.png))\n\nthe line coverage is perfect, yet mutations hint at inputs not verified: `2*2==4`\n\nAdding one more test: `2+4=6` fills the coverage gap:\n\n\u0026darr;\n\n![multiplication substituted](doc/img/coverage_2.png)\n\n![coverage 100%](doc/img/maths_2.png))\n\ncomment out the second test in [MathsTest.java](app/src/test/java/com/siemens/code/dled/MathsTest.java) to see the mutation coverage drop.\n\n## Property-Based Testing\n\n\u003e Are you sure, programming and eliminating bugs is that simple?\n\nTesting the assumptions about code: https://jqwik.net\n\n```java\nboolean hypothesis_sum_of_two_positive_numbers_is_positive(\n  @ForAll(\"positive\") int a,\n  @ForAll(\"positive\") int b\n) {\n  return maths.sumOf(a, b) \u003e 0;\n}\n```\n\n(such assumptions [can be fatal](https://en.wikipedia.org/wiki/Integer_overflow#Examples))\n\n\u0026darr;\n\n```bash\ngradle test --rerun-tasks --info\n```\n\n\u0026darr;\n\n```text\nMathsProperties \u003e hypothesis sum of two positive numbers is positive STANDARD_OUT\n    timestamp = 2021-07-07T16:36:48.104006, MathsProperties:hypothesis sum of two positive numbers is positive =\n      org.opentest4j.AssertionFailedError:\n        Property [MathsProperties:hypothesis sum of two positive numbers is positive] failed with sample {0=1, 1=2147483647}\n\n                                  |-------------------jqwik-------------------\n    tries = 12                    | # of calls to property\n    checks = 12                   | # of not rejected calls\n    generation = RANDOMIZED       | parameters are randomly generated\n    after-failure = PREVIOUS_SEED | use the previous seed\n    when-fixed-seed = ALLOW       | fixing the random seed is allowed\n    edge-cases#mode = MIXIN       | edge cases are mixed in\n    edge-cases#total = 16         | # of all combined edge cases\n    edge-cases#tried = 2          | # of edge cases tried in current run\n    seed = -5436190997333692374   | random seed to reproduce generated values\n\n    Shrunk Sample (1 steps)\n    -----------------------\n      a: 1\n      b: 2147483647\n\n    Original Sample\n    ---------------\n      a: 19\n      b: 2147483647\n\n\nTest hypothesis sum of two positive numbers is positive(com.siemens.code.dled.MathsProperties) took 421 ms.\n\nMathsProperties \u003e hypothesis sum of two positive numbers is positive FAILED\n```\n\n\u003c!--\nSPDX-FileCopyrightText: © 2024 Siemens AG\nSPDX-License-Identifier: Unlicense\n--\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fd-led%2Fjava-testing-the-tests","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fd-led%2Fjava-testing-the-tests","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fd-led%2Fjava-testing-the-tests/lists"}