https://github.com/pennylaneai/plugin-test-matrix
Scheduled integration tests for the official plugins
https://github.com/pennylaneai/plugin-test-matrix
Last synced: about 1 year ago
JSON representation
Scheduled integration tests for the official plugins
- Host: GitHub
- URL: https://github.com/pennylaneai/plugin-test-matrix
- Owner: PennyLaneAI
- Created: 2020-08-02T13:24:01.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2025-05-14T19:12:06.000Z (about 1 year ago)
- Last Synced: 2025-05-14T20:28:33.756Z (about 1 year ago)
- Language: Python
- Homepage:
- Size: 505 KB
- Stars: 13
- Watchers: 13
- Forks: 4
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Plugin Test Matrix
This repository contains regularly scheduled continuous-integration for the official PennyLane plugins.
The continuous integration is performed by GitHub actions. Tests are executed on every push to `master`,
and are also run on a schedule of once per day.
Finally, tests can be triggered manually via the 'Actions' tab.
## Testing matrix
All entries in the matrix are tested against PennyLane latest (GitHub master).
| | PyPI version | Stable plugin/stable PennyLane | Stable plugin/latest PennyLane | Latest plugin/ stable PennyLane | Latest plugin/latest PennyLane |
| :-------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Qiskit](https://github.com/PennyLaneAI/pennylane-qiskit) |  | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/qiskit-stable-stable.yml) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/qiskit-stable-latest.yml) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/qiskit-latest-stable.yml) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/qiskit-latest-latest.yml) |
| [Cirq](https://github.com/PennyLaneAI/pennylane-cirq) |  | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/cirq-stable-stable.yml) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/cirq-stable-latest.yml) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/cirq-latest-stable.yml) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/cirq-latest-latest.yml) |
| [Qulacs](https://github.com/PennyLaneAI/pennylane-qulacs) |  | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/qulacs-stable-stable.yml) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/qulacs-stable-latest.yml) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/qulacs-latest-stable.yml) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/qulacs-latest-latest.yml) |
| [AQT](https://github.com/PennyLaneAI/pennylane-aqt) |  | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/aqt-stable-stable.yml) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/aqt-stable-latest.yml) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/aqt-latest-stable.yml) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/aqt-latest-latest.yml) |
| [IonQ](https://github.com/PennyLaneAI/pennylane-ionq) |  | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/ionq-stable-stable.yml) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/ionq-stable-latest.yml) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/ionq-latest-stable.yml) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/ionq-latest-latest.yml) |
| [Lightning](https://github.com/PennyLaneAI/pennylane-lightning) |  | [](https://github.com/PennyLaneAI/pennylane-lightning/actions/workflows/compat-check-stable-stable.yml) | [](https://github.com/PennyLaneAI/pennylane-lightning/actions/workflows/compat-check-stable-latest.yml) | [](https://github.com/PennyLaneAI/pennylane-lightning/actions/workflows/compat-check-latest-stable.yml) | [](https://github.com/PennyLaneAI/pennylane-lightning/actions/workflows/compat-check-latest-latest.yml) |
| [Braket](https://github.com/aws/amazon-braket-pennylane-plugin-python) |  | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/braket-stable-stable.yml) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/braket-stable-latest.yml) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/braket-latest-stable.yml) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/braket-latest-latest.yml) |
*Notes:*
* The device integration tests currently do not support devices with a fixed
number of wires.
* The Braket plugin device integration tests are run with `-k “not Sample and not no_0_shots”`,
see #6
* The Qiskit tests are run using local simulators. There are no tests that access the IBM Quantum backends.
## QML repo
| | Status |
| :---------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [`build-branch-dev`](https://github.com/PennyLaneAI/qml/blob/master/.github/workflows/build-branch-dev.yml) | [](https://github.com/PennyLaneAI/qml/actions/workflows/build-branch-dev.yml) |
| [`build-branch-master`](https://github.com/PennyLaneAI/qml/blob/master/.github/workflows/build-branch-master.yml) | [](https://github.com/PennyLaneAI/qml/actions/workflows/build-branch-master.yml) |
| [`update-dev`](https://github.com/PennyLaneAI/qml/blob/master/.github/workflows/update-dev.yml) | [](https://github.com/PennyLaneAI/qml/actions/workflows/update-dev.yml) |
## Lightning Docker builds
| | Status |
| :--------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Docker build `stable`](https://github.com/PennyLaneAI/pennylane-lightning/blob/master/.github/workflows/compat-docker-stable.yml) | [](https://github.com/PennyLaneAI/pennylane-lightning/actions/workflows/compat-docker-stable.yml) |
| [Docker build `latest`](https://github.com/PennyLaneAI/pennylane-lightning/blob/master/.github/workflows/compat-docker-latest.yml) | [](https://github.com/PennyLaneAI/pennylane-lightning/actions/workflows/compat-docker-latest.yml) |
## Catalyst compiler
The following table shows the current Catalyst compatibility with the Lightning and PennyLane packages, where each of the 3 packages is tested as "latest" and "stable". Latest versions are built directly from the repository's main or master branch, while stable is built from the repository at the most recent release tag. The reason this is needed rather than installing PyPI packages is that the Lightning dependency in Catalyst needs to be built from source and cannot be obtained from the PyPI package. PennyLane stable is installed from PyPI.
PennyLane stable
PennyLane latest
Lightning stable
Lightning latest
Lightning stable
Lightning latest
Catalyst stable
Catalyst latest
## Latest reported warnings
### PennyLane
UserWarning
DeprecationWarning
pytest.PytestRemovedIn9Warning
PendingDeprecationWarning
pytest.PytestUnraisableExceptionWarning
numpy.exceptions.ComplexWarning
RuntimeWarning
pytest.PytestCollectionWarning
FutureWarning
pennylane.PennyLaneDeprecationWarning
## Interpreting the test matrix
* **Stable/stable**: This reflects a scenario where PennyLane and the plugin are both installed
from PyPI. This column should **always** pass. If this test fails, it indicates that either (a) a
bug exists, or (b) an upstream dependency (such as Qiskit or Qulacs)
were released with breaking changes. In both cases, a bugfix release of the plugin should be made.
* **Stable/latest passing**: A new plugin release is not essential; the current
PyPI release will continue to work once the new PennyLane version is released.
- **Latest/latest passing**: Nothing to be done! However, it is still worth checking the
plugin repository to see if new unreleased changes have been added that would
be worth releasing.
- **Latest/latest failing**: Indicates that recent changes have been made to the plugin repository
post-release; further updates must be made to the plugin to ensure GitHub master passes.
* **Stable/latest failing**: A new plugin release is required for compatibility
with the upcoming PennyLane release.
- **Latest/latest failing**: Updates must be made to the plugin to ensure GitHub
master passes.
- **Latest/stable failing**: The requirement of PennyLane in the plugin will
have to be increased.
- **Latest/latest passing**: Nothing to be done! The plugin is ready to be released
with the upcoming PennyLane release.
## Feature freeze testing matrix
The latest of all plugins are tested against the PennyLane release candidate branch (GitHub `vX.XX.X-rc0` branch).
| | Status |
| :-------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Qiskit](https://github.com/PennyLaneAI/pennylane-qiskit) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/qiskit-latest-rc.yml) |
| [Cirq](https://github.com/PennyLaneAI/pennylane-cirq) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/cirq-latest-rc.yml) |
| [Qulacs](https://github.com/PennyLaneAI/pennylane-qulacs) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/qulacs-latest-rc.yml) |
| [AQT](https://github.com/PennyLaneAI/pennylane-aqt) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/aqt-latest-rc.yml) |
| [IonQ](https://github.com/PennyLaneAI/pennylane-ionq) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/ionq-latest-rc.yml) |
| [Braket](https://github.com/aws/amazon-braket-pennylane-plugin-python) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/braket-latest-rc.yml) |
The latest and release candidate of Lightning and Catalyst are tested with the release candidate of PennyLane:
| | Latest | Release candidate |
| :------------------------------------------------------------------------------------ | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Lightning](https://github.com/PennyLaneAI/pennylane-lightning) | [](https://github.com/PennyLaneAI/plugin-test-matrix/actions/workflows/lightning-latest-rc.yml) | [](https://github.com/PennyLaneAI/pennylane-lightning/actions/workflows/compat-check-release-release.yml) |
| [Catalyst](https://github.com/PennyLaneAI/catalyst) | | [](https://github.com/PennyLaneAI/catalyst/actions/workflows/CPL_rc-rc-rc.yaml) |
| [Docker build](https://github.com/PennyLaneAI/pennylane-lightning/tree/master/docker) | | [](https://github.com/PennyLaneAI/pennylane-lightning/actions/workflows/compat-docker-release.yml) |
## Adding a plugin
Two Jinja2 workflow templates are provided, that makes it easier to add a new plugin to the test matrix:
* [`workflow-template-latest.yml`](workflow-template-latest.yml): for testing plugins against PennyLane latest
* [`workflow-template-stable.yml`](workflow-template-stable.yml): for testing plugins against PennyLane stable
Simply add a new plugin to the `workflows` list in [`compile.py`](compile.py), with the following dictionary keys:
* `plugin` (required): the name of the plugin **not including the PennyLane prefix**. The plugin
repository and PyPI project are inferred from the plugin name. E.g., `'plugin': 'qiskit'` will
correspond to the GitHub repo PennyLaneAI/pennylane-qiskit and the PyPI project
`pennylane-qiskit`.
* `gh_user` (required): the GitHub user or organization which hosts the plugin repository.
* `which` (required): a set specifying whether to generate a workflow to test the `{"latest"}`
(GitHub master) version of the plugin, `{"stable"}` (PyPI) version of the plugin, or both `{"latest", "stable"}`.
* `requirements` (optional): a list of Python packages that should be installed prior to plugin
installation. You may use `pip` syntax, e.g., `pyscf==1.7.2`.
* `requirements_latest` (optional): *additional* list of Python packages that should be installed prior to
latest/GitHub master plugin installation. For example, if the latest version of the plugin depends on a
development version of a particular framework.
* `device_tests` (optional): a list of command line arguments to pass to the PennyLane device
integration tests. Each list element corresponds to a single test run, e.g.,
```python
"device_tests":
[
"--device=cirq.simulator --tb=short --skip-ops --analytic=True",
"--device=cirq.simulator --tb=short --skip-ops --analytic=False --shots=8000"
]
```
* `additional_setup` (optional): multiline string containing additional GitHub actions for execution
before the installment of plugins.
* `token` (optional): sets an environment variable with the same name as the token. The value is loaded from the
repository secrets. The token is required by some plugins for backend authentication.
* `additional_env_vars` (optional): string containing additional environment variables you'd like to be set for
the CI action. Double-check the whitespace when using this - variables should be separated with "\n "
* `runs_on` (optional): string to override the workflow's default `runs-on` attribute of `ubuntu-latest`.
* `test_kwargs` (optional): additional arguments to pass to pytest for the given plugin.
* `no_deprecation_error` (optional): set to True to not raise PL deprecation warnings as errors when testing
the latest version of the plugin. By default, PL deprecation warnings are raised as errors.
Once you have added your plugin, run
```console
$ python compile.py
```
This will autogenerate up to five workflow files, depending on the variable `which`:
* `.github/workflows/plugin-stable-stable.yml`
* `.github/workflows/plugin-stable-latest.yml`
* `.github/workflows/plugin-latest-stable.yml`
* `.github/workflows/plugin-latest-latest.yml`
* `.github/workflows/plugin-latest-rc.yml`
Finally, make sure to add a row to the testing matrix above!