https://github.com/nickofthyme/playwright-docker-arch-matrix
A simple repo to test playwright screenshots across multiple platforms
https://github.com/nickofthyme/playwright-docker-arch-matrix
Last synced: 17 days ago
JSON representation
A simple repo to test playwright screenshots across multiple platforms
- Host: GitHub
- URL: https://github.com/nickofthyme/playwright-docker-arch-matrix
- Owner: nickofthyme
- Created: 2025-04-11T14:57:16.000Z (about 1 month ago)
- Default Branch: main
- Last Pushed: 2025-04-19T23:33:33.000Z (about 1 month ago)
- Last Synced: 2025-04-20T00:28:11.510Z (about 1 month ago)
- Language: TypeScript
- Homepage: https://nickofthyme.github.io/playwright-docker-arch-matrix/
- Size: 408 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 8
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## Playwright docker architecture testing matrix
This repo is meant to test playwright screenshot testing on a multitude of UIs against a multitude of architectures. If uses GitHub actions to run all tests against their baselines.
UI cases
- Text - Variant in typography.
- Canvas - A simple scatter plot using `Chart.js` rendered to [`canvas`]().
- Icons - A wide range of SVG icons using heroicons.
- Image - A high resolution image ([sample image]()).
GitHub Runners
We are testing across multiple architecture types and operating systems. See [docs](https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories) for complete list of GitHub runners. The ones used for testing are detailed below.
Workflow label | Virtual Machine | Processor (CPU) | Memory (RAM) | Storage (SSD) | Architecture
-- | -- | -- | -- | -- | --
[`ubuntu-24.04`](https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2404-Readme.md) | Linux | 4 | 16 GB | 14 GB | x64
[`windows-2025`](https://github.com/actions/runner-images/blob/main/images/windows/Windows2025-Readme.md) | Windows | 4 | 16 GB | 14 GB | x64
[`ubuntu-24.04-arm`](https://github.com/actions/partner-runner-images/blob/main/images/arm-ubuntu-24-image.md) | Linux [Public preview] | 4 | 16 GB | 14 GB | arm64
[`macos-13`](https://github.com/actions/runner-images/blob/main/images/macos/macos-13-Readme.md) | macOS | 4 | 14 GB | 14 GB | Intel
[`macos-15`](https://github.com/actions/runner-images/blob/main/images/macos/macos-15-Readme.md) | macOS | 3 (M1) | 7 GB | 14 GB | arm64
## Playwright system requirements
- Latest version of Node.js 18, 20 or 22.
- Windows 10+, Windows Server 2016+ or Windows Subsystem for Linux (WSL).
- macOS 13 Ventura, or later.
- Debian 12, Ubuntu 22.04, Ubuntu 24.04, on x86-64 and arm64 architecture.
## Demo apps
- Playwright.dev - A copy of home page for testing - [link](https://nickofthyme.github.io/playwright-docker-arch-matrix/playwright-dev).
- React demo app - React starter app with added examples - [link](https://nickofthyme.github.io/playwright-docker-arch-matrix/simple-react-app).
- Text - Variety of typography via tailwind - [link](https://nickofthyme.github.io/playwright-docker-arch-matrix/simple-react-app/examples/text)
- Canvas - `Chart.js` example scatter plot - [link](https://nickofthyme.github.io/playwright-docker-arch-matrix/simple-react-app/examples/canvas)
- Chart - Elastic charts examples from original issue - [link](https://nickofthyme.github.io/playwright-docker-arch-matrix/simple-react-app/examples/chart)
- Icons - Set of svg icons - [link](https://nickofthyme.github.io/playwright-docker-arch-matrix/simple-react-app/examples/icons)
- Image - A high-res image - [link](https://nickofthyme.github.io/playwright-docker-arch-matrix/simple-react-app/examples/image)
## Screenshot naming
By default, Playwright names the screenshots by joining the sanitized test title, project name and platform, like so...
```
{snapshotDir}/{testFilePath}/{testName}-{projectName}-{platform}{ext}
// ./e2e/screenshots/playwright-dev.test.ts/should-render-full-page-chromium-darwin.png
```
To compare screenshot agnostically across all runs, we need to alter this. We do so using the [`expect.toHaveScreenshot.pathTemplate`](https://playwright.dev/docs/api/class-testconfig#test-config-snapshot-path-template) option by setting it to...
```
{snapshotDir}/{testFilePath}/{testName}-{projectName}{ext}
// ./e2e/screenshots/playwright-dev.test.ts/should-render-full-page-chromium.png
```
## Testing cases
### Playwright.dev
file: `e2e/tests/playwright-dev.test.ts`
These tests are meant to simulate a full page screenshot testing and how variability affects the screenshot testing.
### `@elastic/charts`
file: `e2e/tests/elastic-charts.test.ts`
These tests are meant to test the original case described in the issue using `@elastic/charts` and how hover, patterns and canvas affects the screenshot testing.
### Starter react app home page
file: `e2e/tests/react-app/home.test.ts`
These tests are meant to test how hover, icons and drop shadows affects the screenshot testing.
### Example cases
file: `e2e/tests/react-app/examples.test.ts`
These are the examples described above and are meant to test the basic static simple use case of screenshot testing.
## Test Results
| Baselines
created from | CI run on `ubuntu-24.04` (docker) | CI run on `ubuntu-24.04-arm` (docker) |
|--------|:------:|:------:|
| Linux - x64 [PR](https://github.com/nickofthyme/playwright-docker-arch-matrix/pull/11) | ✅ 72 of 72 pass ([report](https://nickofthyme.github.io/playwright-docker-arch-matrix/pr/11/index.html#[email protected])) | ✅ 72 of 72 pass ([report](https://nickofthyme.github.io/playwright-docker-arch-matrix/pr/11/index.html#[email protected])) |
| Linux - x64 [PR](https://github.com/nickofthyme/playwright-docker-arch-matrix/pull/12)
(docker) | ✅ 72 of 72 pass ([report](https://nickofthyme.github.io/playwright-docker-arch-matrix/pr/12/index.html#[email protected])) | ✅ 72 of 72 pass ([report](https://nickofthyme.github.io/playwright-docker-arch-matrix/pr/12/index.html#[email protected])) |
| Linux - arm64 [PR](https://github.com/nickofthyme/playwright-docker-arch-matrix/pull/10) | ✅ 72 of 72 pass ([report](https://nickofthyme.github.io/playwright-docker-arch-matrix/pr/10/index.html#[email protected])) | ✅ 72 of 72 pass ([report](https://nickofthyme.github.io/playwright-docker-arch-matrix/pr/10/index.html#[email protected])) |
| Linux - arm64 [PR](https://github.com/nickofthyme/playwright-docker-arch-matrix/pull/9)
(docker) | ✅ 72 of 72 pass ([report](https://nickofthyme.github.io/playwright-docker-arch-matrix/pr/9/index.html#[email protected])) | ✅ 72 of 72 pass ([report](https://nickofthyme.github.io/playwright-docker-arch-matrix/pr/9/index.html#[email protected])) |
| Mac - intel [PR](https://github.com/nickofthyme/playwright-docker-arch-matrix/pull/8) | ❌ 36 of 72 pass ([report](https://nickofthyme.github.io/playwright-docker-arch-matrix/pr/8/index.html#[email protected])) | ❌ 36 of 72 pass ([report](https://nickofthyme.github.io/playwright-docker-arch-matrix/pr/8/index.html#[email protected])) |
| Mac - intel [PR](https://github.com/nickofthyme/playwright-docker-arch-matrix/pull/7)
(docker) | ❌ 36 of 72 pass ([report](https://nickofthyme.github.io/playwright-docker-arch-matrix/pr/7/index.html#[email protected])) | ❌ 36 of 72 pass ([report](https://nickofthyme.github.io/playwright-docker-arch-matrix/pr/7/index.html#[email protected])) |
| Mac - M1 [PR](https://github.com/nickofthyme/playwright-docker-arch-matrix/pull/4)
| ❌ 36 of 72 pass ([report](https://nickofthyme.github.io/playwright-docker-arch-matrix/pr/4/index.html#[email protected])) | ❌ 36 of 72 pass ([report](https://nickofthyme.github.io/playwright-docker-arch-matrix/pr/4/index.html#[email protected])) |
| Mac - M1 [PR](https://github.com/nickofthyme/playwright-docker-arch-matrix/pull/6)
(docker) | ❌ 36 of 72 pass ([report](https://nickofthyme.github.io/playwright-docker-arch-matrix/pr/6/index.html#[email protected])) | ❌ 36 of 72 pass ([report](https://nickofthyme.github.io/playwright-docker-arch-matrix/pr/6/index.html#[email protected])) |
> All tests are tagged with the following:
> - Browser - `chromium`, `firefox`, `webkit`
> - Platform - `linux`, `darwin`
> - Architecture - `x64`, `arm64`, `intel`
> - Runner/Machine - `mac`, `ubuntu-24.04`, `ubuntu-24.04-arm`
> - Docker - `docker` (all tests are run in docker)
> - Comparator - `comparator:default` or `comparator:ssim-cie94`
Observation from these tests:
- It seems linux machines are very consistent regardless of if they are run on host or docker or run from `x64` or `arm64`.
- The comparator does not seem to have any impact on the results using `[email protected]`.
- `webkit` seems to be the most stable browser, most tests pass when the same tests fail in other browsers (i.e. `chromium` and `firefox`).
- Most flaky UI that often fails
- Hover or focus affects
- Drop shadows
- Full page context - lots of elements captured