Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/lightning-ai/ecosystem-ci
Automate issue discovery for your projects against Lightning nightly and releases.
https://github.com/lightning-ai/ecosystem-ci
ci-cd compatibility-testing deep-learning integration-testing lightweight python
Last synced: 2 months ago
JSON representation
Automate issue discovery for your projects against Lightning nightly and releases.
- Host: GitHub
- URL: https://github.com/lightning-ai/ecosystem-ci
- Owner: Lightning-AI
- License: apache-2.0
- Created: 2021-10-30T23:12:41.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2024-05-02T00:56:58.000Z (9 months ago)
- Last Synced: 2024-05-02T06:03:47.988Z (9 months ago)
- Topics: ci-cd, compatibility-testing, deep-learning, integration-testing, lightweight, python
- Language: Python
- Homepage: https://devblog.pytorchlightning.ai/stay-ahead-of-breaking-changes-with-the-new-lightning-ecosystem-ci-b7e1cf78a6c7
- Size: 723 KB
- Stars: 45
- Watchers: 9
- Forks: 14
- Open Issues: 15
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
**Automated Testing for Lightning EcoSystem Projects**
[](https://lightning.ai)
[](https://github.com/Lightning-AI/ecosystem-ci/actions/workflows/ci_test-acts.yml)
[](https://codecov.io/gh/Lightning-AI/ecosystem-ci)
[](https://results.pre-commit.ci/latest/github/Lightning-AI/ecosystem-ci/main)
______________________________________________________________________
Automate issue discovery for your projects against Lightning nightly and releases.
You get CPUs, Multi-GPUs testing for free, and Slack notification alerts if issues arise!
## How do I add my own Project?
### Pre-requisites
Here are pre-requisites for your project before adding to the Lightning EcoSystem CI:
- Your project already includes some **Python tests with PyTorch Lightning** as a dependency
- You'll be a **contact/responsible** person to resolve any issues that the CI finds in the future for your project
### Adding your own project config
1. First, fork this project (with [CLI](https://cli.github.com/) or in browser) to be able to create a new Pull Request, and work within a specific branch.
```bash
gh repo fork Lightning-AI/ecosystem-ci
cd ecosystem-ci/
```
1. Copy the [template file](_actions/_config.yaml) in `configs` folder and call it `.yaml`.
```
cp configs/template.yaml configs/.yaml
```
1. At the minimum, modify the `HTTPS` variable to point to your repository. See [Configuring my project](https://github.com/Lightning-AI/ecosystem-ci/tree/main#configuring-my-project) for more options.
```yaml
source_repository:
HTTPS: https://github.com/MyUsername/MyProject.git
...
```
If your project tests multiple configurations or you'd like to test against multiple Lightning versions such as master and release branches, create a config file for each one of them.
As an example, have a look at [metrics master](configs/Lightning-AI/metrics_pl-develop.yaml) and [metrics release](configs/Lightning-AI/metrics_pl-release.yaml) CI files.
1. Define your `runtimes` (OS and Python version) in your config file to be executed on CPU and/or add the config filename in the [Azure GPU CI file](.azure/ci-testig-parameterized.yml).
- For CPU integration, specify the OS and Python version combinations inside your config file:
```yaml
runtimes:
- {os: "ubuntu-20.04", python: "3.9"}
- {os: "macOS-12", python: "3.7"}
- {os: "windows-2019", python: "3.8"}
...
```
- For GPU integration, add your config filename in the [Azure GPU CI file](.azure/ci-testig-parameterized.yml) file:
```yaml
...
jobs:
- template: testing-template.yml
parameters:
configs:
- "Lightning-AI/metrics_pl-develop.yaml"
- "Lightning-AI/metrics_pl-release.yaml"
- "MyUsername/my_project-master.yaml"
```
1. Add the responsible person(s) to [CODEOWNERS](.github/CODEOWNERS) for your organization folder or just the project.
```
# MyProject
/configs/Myusername/MyProject* @Myusername
```
1. Finally, create a draft PR to the repo!
### Additional suggestions and engagement rules
- To qualify for GPU machines we require your project to have 100+ GitHub stars (please note that this is for capacity reasons and may change in the future)
- (**Optional**) Join our [Slack](https://www.lightning.ai/community) channel `#alerts-ecosystem-ci` to be notified if your project is breaking
- (**Kind request**) include Lightning badge in your readme:
```md
[](https://lightning.ai)
```
## Configuring my project
The config include a few different sections:
- `source_repository` include your project
- `env` (optional) define any environment variables required when running tests
- `dependencies` listing all dependencies which are taken outside pip
- `testing` defines specific pytest arguments and what folders shall be tested
All dependencies as well as the target repository is sharing the same template with the only required field `HTTPS` and all others are optional:
```yaml
source_repository:
HTTPS: https://github.com/Lightning-AI/metrics.git
username: my-nick # Optional, used when checking out private/protected repo
password: dont-tell-anyone # Optional, used when checking out private/protected repo
token: authentication-token # Optional, overrides the user/pass when checking out private/protected repo
checkout: master # Optional, checkout a particular branch or a tag
install_extras: all # Refers to standard pip option to install some additional dependencies defined with setuptools, typically used as `[]`.
# Optional, if any installation/tests require some env variables
env:
MY_ENV_VARIABLE: "VAR"
copy_tests:
- integrations # copied folder from the original repo into the running test directory
# this is copied as we use the helpers inside integrations as regular python package
- tests/__init__.py
- tests/helpers
# Optional, additional pytest arguments and control which directory to test on
testing:
dirs:
- integrations
pytest_args: --strict
```
Note: If you define some files as done above, and they are using internal-cross imports, you need to copy the `__init__.py` files from each particular package level.
The `testing` section provides access to the pytest run args and command.
```yaml
testing:
# by default pytest is called on all copied items/tests
dirs:
- integrations
# OPTIONAL, additional pytest arguments
pytest_args: --strict
```