{"id":13998467,"url":"https://github.com/AnimaApp/scooby","last_synced_at":"2025-07-23T06:31:49.283Z","repository":{"id":150740253,"uuid":"546177063","full_name":"AnimaApp/scooby","owner":"AnimaApp","description":"UI regression and fidelity testing framework","archived":false,"fork":false,"pushed_at":"2024-02-14T07:48:03.000Z","size":19556,"stargazers_count":130,"open_issues_count":13,"forks_count":2,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-10-23T02:10:06.788Z","etag":null,"topics":["fidelity","regression-testing","testing-framework"],"latest_commit_sha":null,"homepage":"https://animaapp.github.io/scooby","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AnimaApp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-05T16:56:39.000Z","updated_at":"2024-10-16T08:14:14.000Z","dependencies_parsed_at":"2024-01-15T19:45:01.542Z","dependency_job_id":"22b15d11-1245-4352-86d1-d41e0f97a5bd","html_url":"https://github.com/AnimaApp/scooby","commit_stats":{"total_commits":113,"total_committers":6,"mean_commits":"18.833333333333332","dds":0.504424778761062,"last_synced_commit":"87c321602d769c53e58667a2ab17767f1b828352"},"previous_names":[],"tags_count":190,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AnimaApp%2Fscooby","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AnimaApp%2Fscooby/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AnimaApp%2Fscooby/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AnimaApp%2Fscooby/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AnimaApp","download_url":"https://codeload.github.com/AnimaApp/scooby/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227245179,"owners_count":17753239,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fidelity","regression-testing","testing-framework"],"created_at":"2024-08-09T19:01:41.886Z","updated_at":"2024-11-30T00:31:38.178Z","avatar_url":"https://github.com/AnimaApp.png","language":"TypeScript","readme":"![Scooby](images/scoobyLogo.png)\n\n_Scooby_ is an optimized regression and fidelity testing framework.\nIt can be used to test for regressions (both visual and textual), and provides all\ntools and integrations necessary to streamline the experience (a CLI, a review UI, web service, and more).\n\nFor a quick introduction and demo, see this video:\n\n[![Demo Video](https://cdn.loom.com/sessions/thumbnails/e05ded71bde947f7a1d7fc270b0db0cf-with-play.gif)](https://www.loom.com/share/e05ded71bde947f7a1d7fc270b0db0cf)\n\nExample use-cases:\n\n- **Prevent visual regressions in your website UI**\n  - After a PR is opened, you can use Scooby to verify that the changes didn't\n    cause any unwanted regression in your website UI, flagging them for review and approval if necessary.\n- **Test for regressions in generated configuration files**\n  - You might use Scooby to do Snapshot testing of your UI structure (JSON) or other configuration files. With Scooby, you can avoid committing the snapshots into your repository, and instead generate and compare them on the fly inside a CI pipeline.\n- **Support migrations by comparing the output of two systems**\n  - If you are migrating a system, you might want to verify that the output of the new system is equal to the legacy one. Scooby can be used to compare and visualize the differences between the outputs, flagging them up for review and approval if necessary\n- **Test for Storybook components regressions (soon)**\n  - Verify that a code change doesn't cause any visual regression in your Storybook components\n\nScooby was built with these goals in mind:\n\n- Make it easy to add new regression and fidelity tests, both visual and textual\n- Support multiple test formats out of the box (such as HTML, PNG, code, etc) to minimize the amount of boilerplate glue code.\n- Being cost-efficient\n- Be highly extensible to support new kinds of tests and reports\n- Integrate well with our GitHub-based workflows\n\nScooby has been extensively battle-tested at [Anima](https://www.animaapp.com/), where it's powering most of our production pipelines.\n\n# Overview\n\nScooby is composed of three main components:\n\n- A **CLI** used to run reports, either locally or inside a CI pipeline\n- A **Web UI** used to view reports, and leave reviews (approvals, rejections, etc)\n- (Optional) An **API** service to send reviews, and other actions. Scooby can also work without this component, but approvals won't work in the Web UI without it.\n\nWhile Scooby has been built to be storage-agnostic, as of now, only **AWS S3 buckets** are supported as destination for reports and artifacts. If you are interested in other storage options, please open an issue!\n\nThe typical flow of a Scooby-powered report looks as follows:\n\n1. The Scooby CLI is invoked to run a given report, pointing it to the relevant test dataset\n2. After all report tasks have been run, the Scooby CLI is called to update the GitHub Status\n3. Reports can be then viewed on the Scooby Web UI, and reviewers can approve/reject them\n\nThe following diagram illustrates an high-level overview of the system:\n\n![Overview](images/overview.png)\n\n# Getting started\n\nThis section covers the basics to use Scooby in your projects.\nWe'll start from the initial repository setup, gradually\ncovering all the steps to get a simple working pipeline.\n\n\u003e **Note for Scooby developers**: this section does not cover how to contribute to the Scooby project.\n\u003e If your goal is to extend or modify Scooby itself, please\n\u003e head to the [Developer guide](#developer-guide) section below.\n\n## First-time repository setup\n\nIf this is the first time Scooby is used inside your project,\nyou'll need to set up a couple of things beforehand.\n\n### Environment variables\n\nScooby requires several environment variables to work.\nThese variables provide the information and access tokens necessary\nto generate reports, upload artifacts, update GitHub statuses and more.\n\nThese are the environment variables you need to specify in the project CI:\n\n\u003e If you are using CircleCI in your project, you'll need to specify them\n\u003e inside the Project Settings \u003e Environment Variables section.\n\n| Name                           | Description                                                                                                 |\n| ------------------------------ | ----------------------------------------------------------------------------------------------------------- |\n| `SCOOBY_AWS_S3_BUCKET`         | The name of the AWS S3 bucket that will host the reports and artifacts.                                     |\n| `SCOOBY_AWS_S3_REGION`         | The AWS region for the S3 bucket, such as `us-west-2`                                                       |\n| `SCOOBY_AWS_ACCESS_KEY_ID`     | An AWS Access Key ID with write access to the S3 bucket.                                                    |\n| `SCOOBY_AWS_SECRET_ACCESS_KEY` | The AWS secret access key associated with the above account.                                                |\n| `SCOOBY_GITHUB_ACCESS_TOKEN`   | A Github Access Token with write access to the repository you are planning to use.                          |\n| `SCOOBY_WEB_BASE_URL`          | Set this to `https://animaapp.github.io/scooby` unless you're planning to host the frontend somewhere else. |\n\nAfter the necessary environment variables have been set, you are ready to generate a Scooby report.\n\n## Generating your first Scooby report on CircleCI\n\nIn this section, we'll discuss the process to generate a visual regression report.\nIn particular, the goal is to have an automated check that fails whenever some HTML pages change visually.\n\n\u003e While this example focuses on visual regression reports, other report types can be generated\n\u003e following a similar process. Please refer to the Reference section below for more information.\n\nAssuming that the test HTML files are located in the `path/to/tests` folder,\nyou can generate a Scooby regression report by running this command:\n\n```\nnpx @animaapp/scooby-cli regression --name \"my-html-regression\" --tests path/to/tests --file-type=html\n```\n\nThis command tells Scooby to generate a regression report named `my-html-regression` based\non all HTML files located in the `path/to/tests` folder.\nScooby will automatically take care of comparing these HTML files with the known reference\n(usually the one on `main` branch).\n\nNow that we generated the report, we are ready to trigger the GitHub status update, which will show\nthe status for each report in our PRs and commits:\n\n![GitHub Commit Status](images/status.png)\n\nTo update the GitHub status, you'll need to run:\n\n```\nnpx @animaapp/scooby-cli update-status\n```\n\nFor efficiency reasons, it's better to run the `update-status` step only after all Scooby\nreport tasks have been completed.\nIf you are running multiple reports in parallel on CircleCI, a common pattern\nis to use job dependencies:\n\n```yaml\n- run-scooby-fidelity-tests\n- run-scooby-regression-tests\n- scooby-update-status:\n    requires:\n      - run-scooby-regression-tests\n      - run-scooby-fidelity-tests\n```\n\nIn this example, the `scooby-update-status` job is only run after the two other report tasks\nhave been completed, thus invoking `update-status` only once.\n\n# FAQ\n\nThis section discusses some of the commonly asked questions related\nto Scooby or testing in general.\n\n## What's the difference between Fidelity and Regression tests?\n\nWhile Fidelity and Regression tests share some common aspects,\nthey are designed to answer two very different questions:\n\n- **Regression tests compare _two different versions of the same dataset_**.\n  For example, a regression test might verify that the output of a\n  code generator inside a feature branch is not different from a\n  known-good state on the `main` branch.\n- **Fidelity tests compare _the same version of two different datasets_**.\n  For example, a fidelity test might compare how visually similar a given web page is to its Figma design.\n\nFrom these definitions, we can draw several conclusions:\n\n- Regression tests usually produce a binary good/bad result for each\n  test entry, while fidelity tests produce a continuous similarity ratio.\n- Regression tests aim at having zero differences between the two\n  datasets, as we are testing the same system at two different points in time.\n- Fidelity tests _could_ have differences and still be successful, as\n  having a perfect match between datasets produced by two different systems might not be achieavable.\n\n# Reference\n\nThis section acts as a reference for the different tests that can be performed with Scooby, along with other useful commands.\n\n## Visual Regression tests\n\nVisual regression tests can be used to verify if changes introduced\nin a commit (PR) are causing visual regressions.\n\nIn a nutshell, Scooby compares the dataset generated in a given commit (or PR) with the dataset of the _base commit_ (main branch).\nThis allow us to determine whether the changes in a given PR are\nchanging the behavior of our production systems (the ones on main) in a visual way.\n\n```mermaid\ngraph LR;\n  prod(Production - main branch)\n  pr(PR commit - feature branch)\n  pr--\u003eScooby\n  prod--\u003eScooby\n  Scooby--\u003eRegression[Regression Report]\n```\n\n### Basic usage\n\n```\nnpx @animaapp/scooby-cli regression --name \"regression-name\" --tests /path/to/tests --file-type=html\n```\n\n## Visual Fidelity tests\n\n```\nnpx @animaapp/scooby-cli fidelity --name \"fidelity-name\" --expected /path/to/expected/tests --actual /path/to/actual/tests --file-type=png\n```\n\n# Developer guide\n\n## Running the CLI in Dev Mode\n\nWhen working on Scooby, you might need to invoke the development version of the CLI with your changes.\nFrom the repository root, you can run the following command:\n\n```\nyarn cli:run:dev --help\n```\n\nThis command will build the CLI from source and invoke it with the `--help` flag. It is equivalent to running `npx @animaapp/scooby-cli --help`.\n\nBy changing the argument, you can then test the various CLI commands, for example:\n\n```\nyarn cli:run:dev fidelity --name \"fidelity-name\" --expected /path/to/expected/tests --actual /path/to/actual/tests --file-type=png\n```\n\n# Troubleshooting\n\n## Error: Could not find expected browser (chrome) locally. Run `npm install` to download the correct Chromium revision (1045629).\n\nIf you are on macOS you might experience the following error when running Scooby:\n\n```\nError: Could not find expected browser (chrome) locally. Run `npm install` to download the correct Chromium revision (1045629).\n```\n\nTo solve it, you can force a chromium installation with:\n\n```\nnode node_modules/puppeteer/install.js\n```\n","funding_links":[],"categories":["TypeScript"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAnimaApp%2Fscooby","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FAnimaApp%2Fscooby","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAnimaApp%2Fscooby/lists"}