{"id":15044179,"url":"https://github.com/gradle/exemplar","last_synced_at":"2025-04-10T01:15:12.443Z","repository":{"id":37426666,"uuid":"135485050","full_name":"gradle/exemplar","owner":"gradle","description":"Discover and verify code samples and services","archived":false,"fork":false,"pushed_at":"2025-03-25T12:24:42.000Z","size":736,"stargazers_count":35,"open_issues_count":17,"forks_count":14,"subscribers_count":34,"default_branch":"main","last_synced_at":"2025-04-10T01:15:02.742Z","etag":null,"topics":["documentation","gradle-bt","gradle-bt-build-infrastructure","samples","testing"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gradle.png","metadata":{"files":{"readme":"docs/README.adoc","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-05-30T18:54:32.000Z","updated_at":"2025-03-25T12:24:07.000Z","dependencies_parsed_at":"2023-09-28T17:06:59.686Z","dependency_job_id":"af815d55-8f6d-4df8-a071-dd877c0e72c3","html_url":"https://github.com/gradle/exemplar","commit_stats":{"total_commits":178,"total_committers":15,"mean_commits":"11.866666666666667","dds":0.702247191011236,"last_synced_commit":"ea22ebe45f8df0c69660c8feb5f4dc6e99c1b47c"},"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gradle%2Fexemplar","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gradle%2Fexemplar/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gradle%2Fexemplar/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gradle%2Fexemplar/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gradle","download_url":"https://codeload.github.com/gradle/exemplar/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248137891,"owners_count":21053775,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["documentation","gradle-bt","gradle-bt-build-infrastructure","samples","testing"],"created_at":"2024-09-24T20:50:10.612Z","updated_at":"2025-04-10T01:15:12.424Z","avatar_url":"https://github.com/gradle.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"= Exemplar - check and present samples for CLI tools\n:toc:\n:toc-placement!:\n\nGiven a collection of sample projects, this library allows you to verify the samples' output.\n\nIt does this by discovering, executing, then verifying output of sample projects in a separate directory or embedded within asciidoctor. In fact, the samples embedded here are verified by `ReadmeTest`.\n\ntoc::[]\n\n== Use cases\n\nThe intent of the `samples-check` module is to ensure that users of your command-line tool see what you expect them to see.\nIt handles sample discovery, normalization (semantically equivalent output in different environments), and flexible output verification.\nIt allows any command-line executable on the `PATH` to be invoked. You are not limited to Gradle or Java.\n\nWhile this has an element of integration testing, it is not meant to replace your integration tests unless the logging output is the only result of executing your tool.\nOne cannot verify other side effects of invoking the samples, such as files created or their contents, unless that is explicitly configured.\n\nThis library is used to verify the functionality of samples in https://docs.gradle.org[Gradle documentation].\n\n== Installation\n\nFirst things first, you can pull this library down from Gradle's artifactory repository. This https://github.com/gradle/kotlin-dsl[Gradle Kotlin DSL] script shows one way to do just that.\n\n.Installing with Gradle\n[.testable-sample]\n====\n\n.build.gradle.kts\n[source,kotlin]\n----\nplugins {\n    id(\"java-library\")\n}\n\nrepositories {\n    mavenCentral()\n}\n\ndependencies {\n    testImplementation(\"org.gradle.exemplar:samples-check:1.0.0\")\n}\n----\n\n[.sample-command,allow-additional-output=true]\n----\n$ gradle check\n----\n\n====\n\n== Usage\n\n=== Configuring external samples\n\nAn external sample consists of a directory containing all the files required to run that sample.\nIt may include link:https://asciidoctor.org/docs/user-manual/#include-partial[tagged content regions] that can be extracted into documentation.\n\nYou can configure a sample to be tested by creating a file ending with `.sample.conf` (e.g. `hello-world.sample.conf`) in a sample project dir.\nThis is a file in https://github.com/lightbend/config/blob/master/HOCON.md[HOCON format] that might look something like this:\n\n.quickstart.sample.conf\n[source,hocon]\n----\nexecutable: bash\nargs: sample.sh\nexpected-output-file: quickstart.sample.out\n----\n\nor maybe a more complex, multi-step sample:\n\n.incrementalTaskRemovedOutput.sample.conf\n[source,hocon]\n----\ncommands: [{\n  executable: gradle\n  args: originalInputs incrementalReverse\n  expected-output-file: originalInputs.out\n  allow-additional-output: true\n}, {\n  executable: gradle\n  args: removeOutput incrementalReverse\n  flags: --quiet\n  expected-output-file: incrementalTaskRemovedOutput.out\n  allow-disordered-output: true\n}]\n----\n\nWhen there are multiple steps specified for a sample, they are run one after the other, in the order specified. The 'executable' can be either a command on the `$PATH`, or `gradle` (to run Gradle), or `cd` (to change the working directory for subsequent steps).\n\n_See \u003c\u003csample-conf-fields,Sample Conf fields\u003e\u003e for a detailed description of all the possible options._\n\n*NOTE:* There are a bunch of (tested) samples under `samples-check/src/test/samples` of this repository you can use to understand ways to configure samples.\n\n=== Configuring embedded samples\n\nAn embedded sample is one in which the source for the sample is written directly within an link:https://asciidoctor.org/[Asciidoctor] source file.\n\nUse this syntax to allow samples-discovery to extract your sources from the doc, execute the `sample-command`, and verify the output matches what is declared in the doc.\n\n[source,adoc]\n----\n.Sample title\n[.testable-sample]       // \u003c1\u003e\n====\n\n.hello.rb                // \u003c2\u003e\n[source,ruby]            // \u003c3\u003e\n-----\nputs \"hello, #{ARGV[0]}\" // \u003c4\u003e\n-----\n\n[.sample-command]        // \u003c5\u003e\n-----\n$ ruby hello.rb world    // \u003c6\u003e\nhello, world             // \u003c7\u003e\n-----\n\n====\n----\n\u003c1\u003e Mark blocks containing your source files with the role `testable-sample`\n\u003c2\u003e The title of each source block should be the name of the source file\n\u003c3\u003e All source blocks with a title are extracted to a temporary directory\n\u003c4\u003e Source code. This can be `include::`d\n\u003c5\u003e Exemplar will execute the commands in a block with role `sample-command`. There can be multiple blocks.\n\u003c6\u003e Terminal commands should start with \"$ \". Everything after the \"$ \" is treated as a command to run. There can be multiple commands in a block.\n\u003c7\u003e One or more lines of expected output\n\n[NOTE] All sources have to be under the same block, and you must set the title of source blocks to a valid file name.\n\n=== Verify samples\n\nYou can verify samples either through one of the \u003c\u003cverifying-using-a-junit-runner,JUnit Test Runners\u003e\u003e or use the API.\n\n==== Verifying using a JUnit Runner\n\nThis library provides 2 JUnit runners link:../samples-check/src/main/java/org/gradle/exemplar/test/runner/SamplesRunner.java[`SamplesRunner`] (executes via CLI) and link:../samples-check/src/main/java/org/gradle/exemplar/test/runner/GradleSamplesRunner.java[`GradleSamplesRunner`] (executes samples using https://docs.gradle.org/current/userguide/test_kit.html[Gradle TestKit]). If you are using `GradleSamplesRunner`, you will need to add `gradleTestKit()` and SLF4J binding dependencies as well:\n\n[source,kotlin]\n----\ndependencies {\n    testImplementation(gradleTestKit())\n    testRuntimeOnly(\"org.slf4j:slf4j-simple:1.7.16\")\n}\n----\n\n*NOTE:* `GradleSamplesRunner` supports Java 8 and above and ignores tests when running on Java 7 or lower.\n\nTo use them, just create a JUnit test class in your test sources (maybe something like `src/integTest/com/example/SamplesIntegrationTest.java`, https://docs.gradle.org/current/userguide/java_testing.html#sec:configuring_java_integration_tests[keeping these slow tests separate] from your fast unit tests.) and annotate it with which JUnit runner implementation you'd like and where to find samples.\nLike this:\n\n// NOTE: inception bites us if we try to turn this into a testable sample.\n.SamplesRunnerIntegrationTest.java\n[source,java]\n----\npackage com.example;\n\nimport org.junit.runner.RunWith;\nimport org.gradle.exemplar.test.runner.GradleSamplesRunner;\nimport org.gradle.exemplar.test.runner.SamplesRoot;\n\n@RunWith(GradleSamplesRunner.class)\n@SamplesRoot(\"src/docs/samples\")\npublic class SamplesIntegrationTest {\n}\n----\n\nWhen you run this test, it will search recursively under the samples root directory (`src/docs/samples` in this example) for any file with a `*.sample.conf` suffix.\nAny directory found to have one of these will be treated as a sample project dir (nesting sample projects is allowed).\nThe test runner will copy each sample project to a temporary location, invoke the configured commands, and capture and verify logging output.\n\n==== Verifying using the API\n\nUse of the JUnit runners is preferred, as discovery, output normalization, and reporting are handled for you. If you want to write custom samples verification or you're using a different test framework, by all means go ahead :) -- please contribute back runners or normalizers you find useful!\n\nYou can get some inspiration for API use from link:https://github.com/gradle/exemplar/blob/main/samples-check/src/main/java/org/gradle/exemplar/test/runner/SamplesRunner.java[SamplesRunner] and link:https://github.com/gradle/exemplar/blob/main/samples-check/src/main/java/org/gradle/exemplar/test/runner/GradleSamplesRunner.java[GradleSamplesRunner].\n\nCommand execution is handled in the `org.gradle.exemplar.executor.*` classes, some output normalizers are provided in the `org.gradle.exemplar.test.normalizer` package, and output verification is handled by classes in the `org.gradle.exemplar.test.verifier` package.\n\n=== Using Samples for other integration tests\n\nYou might want to verify more than just log output, so this library includes link:https://github.com/junit-team/junit4/wiki/rules[JUnit rules] that allow you to easily copy sample projects to a temporarily location for other verification. Here is an example of a test that demonstrates use of the `@Sample` and `@UsesSample` rules.\n\n.BasicSampleTest.java\n[source,java]\n----\npackage com.example;\n\nimport org.junit.Rule;\nimport org.junit.Test;\nimport org.junit.rules.TemporaryFolder;\nimport org.gradle.exemplar.test.rule.Sample;\nimport org.gradle.exemplar.test.rule.UsesSample;\n\npublic class BasicSampleTest {\n    public TemporaryFolder temporaryFolder = new TemporaryFolder()\n    public Sample sample = Sample.from(\"src/test/samples/gradle\")\n            .into(temporaryFolder)\n            .withDefaultSample(\"basic-sample\")\n\n    @Rule\n    public TestRule ruleChain = RuleChain.outerRule(temporaryFolder).around(sample)\n\n    @Test\n    void verifyDefaultSample() {\n        assert sample.getDir() == new File(temporaryFolder.getRoot(), \"samples/basic-sample\");\n        assert sample.getDir().isDirectory();\n        assert new File(sample.getDir(), \"build.gradle\").isFile();\n\n        // TODO(You): Execute what you wish in the sample project\n        // TODO(You): Verify file contents or whatever you want\n    }\n\n    @Test\n    @UsesSample(\"composite-sample/basic\")\n    void verifyOtherSample() {\n        // TODO(You): Utilize sample project under samples/composite-sample/basic\n    }\n}\n----\n\n=== External sample.conf reference\n\nOne of `executable` or `commands` are required at the root.\nIf `executable` is found, the sample will be considered a single-command sample.\nOtherwise, `commands` is expected to be an Array of link:https://github.com/gradle/exemplar/blob/main/samples-discovery/src/main/java/org/gradle/exemplar/model/Command.java[Commands]:\n\n* repeated Command `commands` -- An array of commands to run, in order.\n\nA link:https://github.com/gradle/exemplar/blob/main/samples-discovery/src/main/java/org/gradle/exemplar/model/Command.java[Command] is specified with these fields.\n\n* required string `executable` -- Executable to invoke.\n* optional string `execution-subdirectory` -- Working directory in which to invoke the executable. _If not specified, the API assumes `./` (the directory the sample config file is in)._\n* optional string `args` -- Arguments for executable. Default is `\"\"`.\n* optional string `flags` -- CLI flags (separated for tools that require these be provided in a certain order). Default is `\"\"`.\n* optional string `expected-output-file` -- Relative path from sample config file to a readable file to compare actual output to. Default is `null`. _If not specified, output verification is not performed._\n* optional boolean `expect-failure` -- Invoking this command is expected to produce a non-zero exit code. Default: `false`.\n* optional boolean `allow-additional-output` -- Allow extra lines in actual output. Default: `false`.\n* optional boolean `allow-disordered-output` -- Allow output lines to be in any sequence. Default: `false`.\n\n=== Output normalization\n\nsamples-check allows actual output to be normalized in cases where output is semantically equivalent.\nYou can use normalizers by annotating your JUnit test class with `@SamplesOutputNormalizers` and specifying which normalizers (in order) you'd like to use.\n\n[source,java]\n----\n@SamplesOutputNormalizers({JavaObjectSerializationOutputNormalizer.class, FileSeparatorOutputNormalizer.class, GradleOutputNormalizer.class})\n----\n\nCustom normalizers must implement the link:https://github.com/gradle/exemplar/blob/main/samples-check/src/main/java/org/gradle/exemplar/test/normalizer/OutputNormalizer.java[`OutputNormalizer`] interface. The two above are included in check.\n\n=== Common sample modification\n\nsamples-check supports modifying all samples before they are executed by implementing the link:https://github.com/gradle/exemplar/blob/main/samples-check/src/main/java/org/gradle/exemplar/test/runner/SampleModifier.java[`SampleModifier`] interface and declaring link:https://github.com/gradle/exemplar/blob/main/samples-check/src/main/java/org/gradle/exemplar/test/runner/SampleModifiers.java[`SampleModifiers`].\nThis allows you to do things like set environment properties, change the executable or arguments, and even conditionally change verification based on some logic.\nFor example, you might prepend a `Command` that sets up some environment before other commands are run or change `expect-failure` to `true` if you know verification conditionally won't work on Windows.\n\n[source,java]\n----\n@SampleModifiers({SetupEnvironmentSampleModifier.class, ExtraCommandArgumentsSampleModifier.class})\n----\n\n=== Custom Gradle installation\n\nTo allow Gradle itself to run using test versions of Gradle, the `GradleSamplesRunner` allows a custom installation to be injected using the system property \"integTest.gradleHomeDir\".\n\n== Contributing\n\n[link=https://builds.gradle.org/viewType.html?buildTypeId=Build_Tool_Services_Exemplar]\nimage::https://builds.gradle.org/guestAuth/app/rest/builds/buildType:(id:Build_Tool_Services_Exemplar)/statusIcon.svg[Build status]\n\n[link=https://gradle.org/conduct/]\nimage::https://img.shields.io/badge/code%20of-conduct-lightgrey.svg?style=flat\u0026colorB=ff69b4[code of conduct]\n\n== Releasing\n\n1. Change the version number in the root build script to the version you want to release.\n1. Add missing changes to the \u003c\u003cChanges\u003e\u003e section below.\n1. Push the changes to `main` branch.\n1. Run the https://builds.gradle.org/buildConfiguration/Build_Tool_Services_Exemplar_Verify[Verify Exemplar] job on the commit you\nwant to release.\n1. Run the https://builds.gradle.org/buildConfiguration/Build_Tool_Services_Exemplar_Publish[Publish Exemplar] job on the commit you\nwant to release. This job publishes everything to the Maven Central staging repository.\n1. https://s01.oss.sonatype.org/#stagingRepositories[Login to Sonatype], close the staging repository after reviewing\nits contents.\n1. Release the staging repository.\n1. Tag the commit you just release with the version number `git tag -s VERSION -m \"Tag VERSION release\" \u0026\u0026 git push --tags`\n1. Go to the https://github.com/gradle/exemplar/releases[Releases section on GitHub] and create a new release using the tag you just pushed. Copy the release notes from the \u003c\u003cChanges\u003e\u003e section into the release description.\n1. Bump the version in the root build script to the next snapshot version. Push the change to `main` branch.\n\n== Changes\n\n=== 1.0.2\n\n- Published with new PGP key (no functional changes since 1.0.0)\n\n=== 1.0.1\n\n- Published with new PGP key (no functional changes since 1.0.0)\n\n=== 1.0.0\n\n- Publish all artifacts to the Maven Central repository under `org.gradle.exemplar` group\n- Renamed modules from `sample-check` and `sample-discovery` to `samples-check` and `samples-discovery`\n- Changed root package name from `org.gradle.samples` to `org.gradle.exemplar`\n\n=== 0.12.6\n\n- `AsciidoctorCommandsDiscovery` now support `user-inputs` for providing inputs to a command\n- Introduce a output normalizer removing leading new lines\n\n=== 0.12.5\n\n- `GradleOutputNormalizer` can normalize incubating feature message\n\n=== 0.12.4\n\n- Ensure output normalizer aren't removing trailing new lines (except for the `TrailingNewLineOutputNormalizer`\n\n=== 0.12.3\n\n- `GradleOutputNormalizer` can normalize snapshot documentation urls\n\n=== 0.12.2\n\n- `GradleOutputNormalizer` can normalize wrapper download message\n- `TrailingNewLineOutputNormalizer` can normalize empty output\n\n=== 0.12.1\n\n- `GradleOutputNormalizer` can normalize build scan urls\n\n=== 0.12\n\n- Rename `AsciidoctorAnnotationNormalizer` to `AsciidoctorAnnotationOutputNormalizer`\n- Introduce `WorkingDirectoryOutputNormalizer` to normalize paths in the output\n\n=== 0.11.1\n\n- Introduce `AsciidoctorCommandsDiscovery` to only discover commands\n\n=== 0.11\n\n- Downgraded AsciidoctorJ to 1.5.8.1 to play nice with Asciidoctor extension used by Gradle documentation.\n\nNote: The upgrade of AsciidoctorJ will need to be cross-cutting.\n\n=== 0.10\n\n- Fixes to the `AsciidoctorSamplesDiscovery` classes\n- Allow configuring the underlying `Asciidoctor` instance used by `AsciidoctorSamplesDiscovery`\n- A bunch of out-of-the-box `OutputNormalizer`\n\n=== 0.8\n\n- Handle `cd \u003cdir\u003e` commands, to keep track of the user's working directory and apply it to later commands in the same sample.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgradle%2Fexemplar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgradle%2Fexemplar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgradle%2Fexemplar/lists"}