Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/lefou/mill-integrationtest
Integration testing for mill plugins
https://github.com/lefou/mill-integrationtest
build-tool integration-testing mill mill-plugin scala
Last synced: 9 days ago
JSON representation
Integration testing for mill plugins
- Host: GitHub
- URL: https://github.com/lefou/mill-integrationtest
- Owner: lefou
- License: apache-2.0
- Created: 2019-02-21T09:54:24.000Z (over 5 years ago)
- Default Branch: main
- Last Pushed: 2024-10-01T20:09:29.000Z (about 1 month ago)
- Last Synced: 2024-10-16T09:22:16.420Z (22 days ago)
- Topics: build-tool, integration-testing, mill, mill-plugin, scala
- Language: Scala
- Homepage:
- Size: 253 KB
- Stars: 12
- Watchers: 3
- Forks: 6
- Open Issues: 5
-
Metadata Files:
- Readme: README.adoc
- Funding: .github/FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
README
= mill-integrationtest - Integration test plugin for mill plugins
:version: 0.7.1
:mill-platform: 0.11
:scala-platform: 2.13
:min-mill-version: 0.9.3
:example-mill-version: 0.11.0
:project-home: https://github.com/lefou/mill-integrationtest
:scoverage-version: 1.4.11
:munit-version: 0.7.7
:toc:
:toc-placement: preamble
ifdef::env-github[]
image:https://github.com/lefou/mill-integrationtest/workflows/.github/workflows/build.yml/badge.svg[Build Status (GitHub Actions), link="https://github.com/lefou/mill-integrationtest/actions"]
image:https://codecov.io/gh/lefou/mill-integrationtest/branch/main/graph/badge.svg[Test Coverage (Codecov.io), link="https://codecov.io/gh/lefou/mill-integrationtest"]
image:https://javadoc.io/badge2/de.tototec/de.tobiasroeser.mill.integrationtest_mill{mill-platform}_{scala-platform}/scaladoc.svg["ScalaDoc", link = "https://javadoc.io/doc/de.tototec/de.tobiasroeser.mill.integrationtest_mill{mill-platform}_{scala-platform}"]
endif::[]
Integration testing for https://github.com/com-lihaoyi/mill[Mill] plugins.
== Quickstart
Here we assume, you use mill {example-mill-version} and develop a mill plugin named `demo`
[source,scala]
----
// build.sc
import mill._, mill.scalalib._
object demo extends ScalaModule with PublishModule {
// ...
}
----
First you need to add a new test module, e.g. `itest`.
[source,scala,subs="verbatim,attributes"]
----
// build.sc
import $ivy.`de.tototec::de.tobiasroeser.mill.integrationtest::{version}`
import de.tobiasroeser.mill.integrationtest._
object demo extends ScalaModule with PublishModule {
// ...
}
object itest extends MillIntegrationTestModule {
def millTestVersion = "{example-mill-version}"
def pluginsUnderTest = Seq(demo)
}
----
Your test cases will be located in the source directory of the newly added `itest` project.
The idea is that each sub-directory represents a separate mill project representing a test case.
Your project should now look similar to this:
----
.
+-- demo/
| +-- src/
|
+-- itest/
+-- src/
+-- 01-first-test/
| +-- build.sc
| +-- src/
|
+-- 02-second-test/
+-- build.sc
----
As the buildfiles `build.sc` in your test cases typically want to access the locally built plugin(s),
the plugins publishes all modules referenced under `pluginsUnderTest` and `temporaryIvyModule` to a temporary ivy repository,
just before the test is executed.
The mill version used in the integration test then uses that temporary ivy repository.
Because you are using your locally developed plugin,
instead of referring to your plugin with `import $ivy.'your::plugin:version'`,
you should use the following line, which ensures that you use the correct locally build plugins.
[source,scala]
----
// build.sc
import $file.plugins
----
Effectively, at execution time, Mill is also loading file `plugins.sc`,
a file which was generated just before the test started to execute.
It will `$ivy` import all dependencies you listed in `pluginsUnderTest`.
(If you want to find out how this works, read https://ammonite.io/#import$file[this])
.Example for a generated `plugins.sc`
[source,scala]
----
// Import a locally published version of the plugin under test
import $ivy.`org.example:mill-demo_2.12:0.1.0-SNAPSHOT`
----
== Configuration and Targets
Mill {min-mill-version} or newer is required. See also <>.
The `MillIntegrationTestModule` trait provides the following targets:
.Mandatory configuration
* `def millTestVersion: T[String]`
The mill version used for executing the test cases.
Used by `downloadMillTestVersion` to automatically download.
* `def pluginsUnderTest: Seq[PublishModule]` -
The plugins used in the integration test.
You should at least add your plugin under test here.
You can also add additional libraries, e.g. those that assist you in the test result validation (e.g. a local test support project).
The defined modules will be published into a temporary ivy repository before the tests are executed.
In your test `build.sc` file, instead of the typical `import $ivy.` line,
you should use `import $file.plugins` to include all plugins that are defined here.
.Optional configuration
* `def temporaryIvyModules: Seq[PublishModule]` -
Additional modules you need in the temporary ivy repository, but not in the resulting mill build classpath.
The defined modules will be published into a temporary ivy repository before the tests are executed.
This is almost the same as `pluginsUnderTest`, but does not end up in the generated `plugins.sc`.
* `def sources: Sources` -
Locations where integration tests are located.
Each integration test is a sub-directory, containing a complete test mill project.
* `perTestResources: Sources` -
Shared test resources, will be copied as-is into each test case working directory before the test is run.
You can also generate these, making some test setups easier (e.g. including additional classpath resources).
* `def testCases: Target[Seq[PathRef]]` -
The directories each representing a mill test case.
Derived from `sources`.
* `def testInvocations: Target[Seq[(PathRef, Seq[TestInvocation.Targets])]]` -
The test invocations to test the project.
Defaults to run `TestInvokation.Targets` with the targets from [[testTargets]] and expecting successful execution.
For each test case, you can define a seq of invocations.
* `def testTargets: Target[Seq[String]]` -
_Deprecated: Please use `testInvocations` instead_
The targets which are called to test the project.
Defaults to `verify`, which should implement test result validation.
* `def downloadMillTestVersion: T[PathRef]` -
Download the mill version as defined by `millTestVersion`.
Override this, if you need to use a custom built mill version.
Returns the `PathRef` to the mill executable (must have the executable flag).
* `def useCachedMillDownload: T[Boolean]` -
If `true`, the downloaded mill version used for tests will be cached to the system cache dir (e.g. `~/.cache`).
Default: `true`.
* `def showFailedRuns: T[Boolean]` -
If `true`, The run log of a failed test case will be shown.
Default: `true`.
* `def prefetchIvyDeps: T[Agg[Dep]]` -
Add dependencies here, which you want to prefetch into your local coursier cache before acually running the tests.
Each dependency is resolved and fetched independently, so it is possible to fetch multiple versions of the same artifact.
Use this target to prepare integration test which should run offline.
.Commands / Action Targets
* `def test(args: String*): Command[Seq[TestCase]]` -
Run the integration tests. The args here are the actual test cases that will
run. By default this will run them all, but it's also possible to just pass
the single test name in to run that single test.
* `def testCached: Target[Seq[TestCase]]` -
Run the integration tests (same as `test`), but only if any input has changed since the last run.
* `def prepareOffline: Command[Unit]` -
Prepares going offline by pre-fetching all known dependencies.
== How can I ...
=== Run multiple targets in one go
Use `testInvocations` to configure the targets to execute.
[source,scala]
----
def testInvocations = T{
Seq(
pathRefToTest1 -> Seq(
TestInvocation.Targets(Seq("target1", "target2"))
)
)
}
----
=== Run multiple mill invocations with different or even the same targets
Use `testInvocations` to configure the targets to execute.
[source,scala]
----
def testInvocations = T{
Seq(
PathRef()-> Seq(
// first mill run
TestInvocation.Targets(Seq("target1", "target2")),
// second mill run
TestInvocation.Targets(Seq("target3", "target4")),
// third mill run with same targets
TestInvocation.Targets(Seq("target3", "target4"))
)
)
}
----
=== Run a single defined integration test
Given a setup like this:
[source,scala]
----
def testInvocations = T{
Seq(
PathRef(testBaseDir / "exampleTestDir") -> Seq(
TestInvocation.Targets(Seq("target1")),
)
)
}
----
You can run an individual target by passing in the name to `itest`:
[source]
---
mill itest exampleTestDir
---
=== Test failing mill targets
Use `testInvocations` to configure the targets to execute and fail.
[source,scala]
----
def testInvocations = T{
Seq(
pathRefToTest1 -> Seq(
// first 2 targets that should succeed
TestInvocation.Targets(Seq("target1", "target2")),
// third target should fail with exit code 1
TestInvocation.Targets(Seq("target3"), expectedExitCode = 1)
)
)
}
----
=== Better prints context of failed targets
Many test libraries provide nice asserting APIs which produce helpful error messages.
For example, use `munit`'s Assertions when defining your test targets
[source,scala,subs="attributes"]
----
// itest/src/project1/build.sc
import $ivy.`org.scalameta::munit:{munit-version}`, munit.Assertions._
def verify() = T.command {
assert(None.isDefined)
val fixedScala = read(os.pwd / "foo" / "src" / "Fix.scala")
val expected = """object Fix {
| def procedure(): Unit = {} xxx
|}
|""".stripMargin
assertEquals(fixedScala, expected)
}
----
=== Properly test a mill plugin that uses a worker implementation
You probably want to load the worker in a separated classloader,
hence it should not end up in mills classpath.
Define the plugin module with `pluginsUnderTest` and the worker module with `temporaryIvyModules`.
This will ensure that all modules will be build and published to the test ivy repository,
but only those listed in `pluginsUnderTest` will end up in the generated `plugins.sc`.
[source,scala]
----
def itest extends MillIntegrationTestModule {
def pluginsUnderTest = Seq(plugin)
def temporaryIvyModules = Seq(api, worker)
// ...
}
----
=== Test with multiple mill versions, e.g. on a CI server
Mill hasn't a stable API (yet) and there are no binary compatibility guarantees.
So, it is a good idea to add all supported mill version to your CI setup.
The recommended way of supporting multiple mill versions is via mill's built-in support for cross building (`mill.define.Cross`).
[source,scala]
----
val millItestVersions = Seq("0.7.3", "0.7.2", "0.7.1", "0.7.0")
object itest extends Cross[ItestCross](millItestVersions: _*)
class ItestCross(millItestVersion: String) extends MillIntegrationTestModule {
def millTestVersion = millItestVersion
// correct the source path (remove the extra level for the mill version)
override def millSourcePath = super.millSourcePath / os.up
..
}
----
Now you can run a single integration test with
[source,sh]
mill itest[0.7.3].test
Or you can all integration test in parallel with
[source,sh]
mill -j 0 itest[_].test
=== Cross-Testing a cross built mill-plugin
In case you cross build your mill plugin to support multiple API versions,
you need to parametrize your plugins under test.
[source,scala]
----
trait Deps {
def millVersion = "0.7.0"
def scalaVersion = "2.13.2"
val millMain = ivy"com.lihaoyi::mill-main:${millVersion}"
val millScalalib = ivy"com.lihaoyi::mill-scalalib:${millVersion}"
}
object Deps_0_7 extends Deps
object Deps_0_6 extends Deps {
override def millVersion = "0.6.0"
override def scalaVersion = "2.12.10"
}
// The Mill API versions you want to support
val millApiVersions: Map[String, Deps] = ListMap(
"0.7" -> Deps_0_7,
"0.6" -> Deps_0_6
)
// The Released Mill versions you want to use in your integration tests
val millItestVersions = Seq(
"0.7.3", "0.7.2", "0.7.1", "0.7.0",
"0.6.3", "0.6.2", "0.6.1", "0.6.0"
)
// Your mill plugin
object core extends Cross[CoreCross](millApiVersions.keysIterator.toSeq: _*)
class CoreCross(val millApiVersion: String) extends CrossScalaModule with PublishModule {
def deps: Deps = millApiVersions(millApiVersion)
override def crossScalaVersion = deps.scalaVersion
override def compileIvyDeps = Seq(
deps.millMain,
deps.millScalalib
)
..
}
// Your integration test for your mill plugin
object itest extends Cross[ItestCross](millItestVersions: _*)
class ItestCross(millItestVersion: String) extends MillIntegrationTestModule {
val millApiVersion = millItestVersion.split("[.]").take(2).mkString(".")
override def millSourcePath: Path = super.millSourcePath / os.up
override def millTestVersion = millItestVersion
override def pluginsUnderTest = Seq(core(millApiVersion))
..
}
----
Have a look at the `build.sc` of this mill plugin to see how this is done.
Here are also link to two other mill plugins that uses this technique (at the time of writing this):
* https://github.com/lefou/mill-vcs-version
* https://github.com/lefou/mill-kotlin
=== Collecting integration test coverage data with Scoverage
Mill already provides the `mill.contrib.scoverage.ScoverageModule` as part of its contrib plugin collection.
To ensure you're using the scoverage-enhanced class files (which are configured to write coverage data into a directrory)
in your integration tests, you need to make sure to use the right JAR with the enhanced class files `.scoverage.jar` instead of the `.jar`.
To accomplish this, you need to override the protected `pluginsUnderTestDetails` target and swap the binary JAR with it's `ScoverageModule` version.
This trick has the effect that we install the scoverage-enhanced JAR file into the test ivy repository.
If you also use `temporaryIvyModules`, you need to do the same for `temporaryIvyModulesDetails`.
[IMPORTANT]
--
It's important to only use the scoverage-enhanced classes in tests! Do not distribute them.
If you would use them outside of your test case, loading them or executing their code would fail in almost all cases.
--
[source,scala,subs="attributes"]
----
class core extends ScalaModule with PublishModule with ScoverageModule {
override def scoverageVersion = "{scoverage-version}"
..
}
object itest extends MillIntegrationTestModule {
override def pluginsUnderTest = Seq(core)
override def pluginUnderTestDetails: Task.Sequence[(PathRef, (PathRef, (PathRef, (PathRef, (PathRef, Artifact)))))] =
T.traverse(pluginsUnderTest) { p =>
val jar = p match {
case p: ScoverageModule => p.scoverage.jar
case p => p.jar
}
jar zip (p.sourceJar zip (p.docJar zip (p.pom zip (p.ivy zip p.artifactMetadata))))
}
..
}
----
Also, you need to make sure, that you load the required scoverage runtime library into your mill under test.
You can do this by adding the following `$ivy` import to your `build.sc` in each test case.
[source,scala,subs="attributes"]
import $ivy.`org.scoverage::scalac-scoverage-runtime:{scoverage-version}`
Now, when you run the integration tests coverage data will be gathered and can be used to generate reports.
[source,sh]
mill -j 0 itest.test
mill core.scoverage.htmlReport
== How is `mill-integrationtest` tesing itself?
Glad you asked!
`mill-integrationtest` is using a previously released version of itself to test itself.
This means we have three levels of `mill-integrationtest`:
1. The project itself, configured in `build.sc`
2. A previously released version of `mill-integrationtest` to run the integration tests, configured in the cross module `itest` (in top-level `build.sc`).
The cross parameter denotes the Mill version to run the tests against.
3. And finally the freshly built `mill-integrationtest` plugin under test, used in the test cases located under `itest/src`.
This makes understanding the test setup and the build/test output rather hard to read, even for me.
== Download
You can download binary releases from
https://search.maven.org/artifact/de.tototec/de.tobiasroeser.mill.integrationtest_mill{mill-platform}_{scala-platform}[Maven Central].
Newer versions of this plugin (after version 0.3.3) have a _mill platform suffix_ in the artifact name.
If you use Mill 0.9.10 or above, you can use the double colon (`::`) between artifact name and version to always use the right Mill binary platform.
On older Mill versions, you need to add the platform suffix manually.
.Mill Platform suffix
[options="header"]
|===
| mill version | mill platform | suffix | example
| 0.10.x | 0.10 | `_mill0.10` | ```import $ivy.`de.tototec::de.tobiasroeser.mill.integrationtest::{version}````
| 0.9.3 - 0.9.x | 0.9 | `_mill0.9` | ```import $ivy.`de.tototec::de.tobiasroeser.mill.integrationtest_mill0.9:{version}````
|===
== License
This project is published under the https://www.apache.org/licenses/LICENSE-2.0[Apache License, Version 2.0].
== Version Compatibility Matrix
Mill is still in active development, and has no stable API yet.
Hence, not all mill-integrationtest versions work with every mill version.
The following table shows a matrix of compatible mill and mill-integrationtest versions.
Newer version of mill may or may not work. (Feel free to update this page via a pull request, thanks.)
.Version Compatibility Matrix
[options="header"]
|===
| mill-integrationtest | mill
| 0.7.0 | 0.9.x, 0.10.x, 0.11.0-M8
| 0.6.1 | 0.9.3 - 0.9.x, 0.10.x
| 0.6.0 | 0.9.3 - 0.9.x, 0.10.x
| 0.5.1 | 0.9.3 - 0.9.x, 0.10.x
| 0.5.0 | 0.9.3 - 0.9.12, 0.10.0 - 0.10.1
| 0.4.2 | 0.9.3 - 0.9.12, 0.10.0 - 0.10.1
| 0.4.1 | 0.6.2 - 0.9.12
| 0.4.0 | 0.6.2 - 0.9.6, (not 0.9.7), 0.9.8 - 0.9.9
| 0.3.3 | 0.6.2 - 0.8.0
| 0.3.2 | 0.6.2 - 0.8.0
| 0.3.1 | 0.6.2 - 0.8.0
| 0.3.0 | 0.6.2 - 0.8.0
| 0.2.1 | 0.6.0 - 0.6.3
| 0.2.0 | 0.5.7
| 0.1.2 | 0.5.7
| 0.1.1 | 0.5.7
| 0.1.0 | 0.3.6 - 0.5.3
|===
== About
mill::
https://github.com/lihaoyi/mill[Mill] is a Scala-based open source build tool.
In my opinion the best build tool for the JVM.
It is fast, reliable and easy to understand.
me::
+
--
https://github.com/lefou/[I'm] a professional software developer and love to write and use open source software.
I'm actively developing and maintaining mill as well as https://github.com/lefou?utf8=%E2%9C%93&tab=repositories&q=topic%3Amill&type=&language=[several mill plugins].
If you like my work, please star it on GitHub. You can also support me via https://github.com/sponsors/lefou[GitHub Sponsors].
--
Contributing::
If you found a bug or have a feature request, please open a {project-home}/issues[new issue on GitHub].
I also accept {project-home}/pulls[pull requests on GitHub].
== Changelog
=== 0.7.1 - 2023-06-07
* Support for Mill 0.11 API
* Update Scala to 2.13.11
=== 0.7.0 - 2023-04-27
* Support for Mill 0.11.0-M8
* Slight API changes to accommodate to Mill 0.11 (only return values of `pluginUnderTestDetails` and `temoraryIvyModuleDetails`)
* Minor documentation and internal improvements
=== 0.6.1 - 2022-07-11
* Fixed default value for `TestInvocation.Targets.noServer`
=== 0.6.0 - 2022-04-13
* Support running Mill under test in server mode
* mill-integrationtest is now also CI tested on Windows
=== 0.5.1 - 2022-04-12
* Fixed non-functional tests under Windows
=== 0.5.0 - 2022-03-09
* Support to specify environment variables for test runs
* Support `moduleDeps` of tests plugins (to publish them transitively into the test repository)
* Updated toolchain to use Mill 0.10.1 and newer plugins
=== 0.4.2 - 2022-03-08
* Added support for Mill 0.10
* Added support for mill milestone versions
* Added `prefetcIvyDeps` and offline support
* Dependency updates
* Dropped support for older Mill versions
=== 0.4.1 - 2021-06-09
* Improved output, esp. in error case
* Added new `perTestResources` target
* Work around binary compatibility issues with mill 0.9.7
=== 0.4.0 - 2020-11-30
* Added support for mill 0.9.3 while maintaining backward-compatible versions down to mill 0.6.2
* Introduce a new artifact name suffix (`_mill0.9` for mil 0.9.3) to support multiple mill API versions.
* Various version bumps: scalatest 3.2.3, scalafmt 2.7.5, scoverage 1.4.2
=== 0.3.3 - 2020-07-03
* New option `showFailedRuns` to always show output of failed runs
=== 0.3.2 - 2020-07-03
* Re-use mill download cache under `~/.cache`
* Added integration tests
* Improved output and error reporting
* Integration test runs now will be written to a dedicated log file
* When mill it run in debug mode (`-d`), the complete log of a failed run will be printed after the test summary
* More documentation
=== 0.3.1 - 2020-05-19
* Fixed issues on Windows when setting script permissions
=== 0.3.0 - 2020-05-15
* Cross-publishing for Mill API 0.6.2 (Scala 2.12) and mill API 0.7.0 (Scala 2.13)
* Use newer mill 0.6.2 API to publish to custom ivy repositories
* Fixes Windows support
* Only scan existing source dirs for test cases
=== 0.2.1 - 2020-02-27
* Bumped Mill API to 0.6.0
=== 0.2.0 - 2020-02-27
* Added support to run selective tests
* Targets `test` and `testCached` no return the test result
* new target `testCachedArgs` to control args feeded to testCachedArgs
* Test executor now generated a mill script which allows you to manually invoke mill in
a test destination directory
* New target `testInvocations` providing much finer control over executed targets and their
expected exit value
=== 0.1.2 - 2020-02-18
* New target `temporaryIvyModulesDetails`
* New target `testCached`
=== 0.1.1 - 2020-01-08
* Version bump mill API to 0.5.7
=== 0.1.0 - 2019-02-21
* Initial public release