{"id":15372074,"url":"https://github.com/dfandrich/testclutch","last_synced_at":"2026-03-01T05:02:29.500Z","repository":{"id":221790421,"uuid":"755352765","full_name":"dfandrich/testclutch","owner":"dfandrich","description":"Test Clutch is a system for tracking and analyzing automated regression test results over multiple continuous integration services.","archived":false,"fork":false,"pushed_at":"2025-12-27T00:10:17.000Z","size":1221,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-12-28T15:22:06.755Z","etag":null,"topics":["developer-tools","statistics","testing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dfandrich.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-02-10T00:18:01.000Z","updated_at":"2025-12-27T00:10:21.000Z","dependencies_parsed_at":"2024-02-10T05:28:14.317Z","dependency_job_id":"869b4c63-1f4e-483e-9e83-79f017c6d737","html_url":"https://github.com/dfandrich/testclutch","commit_stats":{"total_commits":265,"total_committers":2,"mean_commits":132.5,"dds":0.04528301886792452,"last_synced_commit":"40fcf0c0e3ec980b90c08d27fe02532056c48611"},"previous_names":["dfandrich/testclutch"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dfandrich/testclutch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dfandrich%2Ftestclutch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dfandrich%2Ftestclutch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dfandrich%2Ftestclutch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dfandrich%2Ftestclutch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dfandrich","download_url":"https://codeload.github.com/dfandrich/testclutch/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dfandrich%2Ftestclutch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29960261,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-01T01:47:18.291Z","status":"online","status_checked_at":"2026-03-01T02:00:07.437Z","response_time":124,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["developer-tools","statistics","testing"],"created_at":"2024-10-01T13:49:32.221Z","updated_at":"2026-03-01T05:02:29.481Z","avatar_url":"https://github.com/dfandrich.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Test Clutch logo](test-clutch-logo.svg)\n\nTest Clutch is a system for tracking and analyzing automated regression test\nresults over multiple continuous integration services. It was born from the\nneed in the curl project to make sense of the more than 300,000 tests run every\nday over its six CI services.  It does not run tests itself, but rather\ncollects statistics from one or more test runners.\n\n## Introduction\n\nTest Clutch was born out of the curl project which was spreading its CI load\nover the free tier of five different CI providers (plus user-submitted builds\nrun on even more build farms), so detailed results could not be viewed in a\nsingle location. The summary provided by GitHub for PRs showed only failed\nbuilds without much detail, and it ignored entirely automated builds run by\nusers on the daily tarballs provided by the project. There was also no test\nsummary of the master branch more than a simple binary failed/succeeded in\nstatus badges.  Test Clutch brings into one place information about millions of\ntest runs and attempts to give the user a useful view of them.\n\n## Architecture\n\nTest Clutch is built around a database of test results. Test results enter the\ndatabase in the ingestion phase where they can then be summarized or queried.\nThe database holds information down to the level of which test numbers were run\nin a given run and their success or failure. Unlimited metadata can be attached\nto each run to help in later analysis, ranging from the git commit of the test\nsource and the CI service being used, down to the hostname and OS version of\nthe test runner. sqlite3 is used as the database, which means the program does\nnot need an external database running.\n\n### Ingestion\n\nA periodic job polls the relevant CI services for new test results that are\nstored in the database. The polling interval should be set depending on how\nfresh you want the data to be versus how much load you want or are allowed to\nput on the CI system.\n\n### Augmentation\n\nSome useful data may not be available from the CI service. Test data\naugmentation can add data to the test runs in the database to fill in the gaps.\nFor example, a git hash might only be available in its short form; an\naugmentation job can be run to expand the hash to its full value by looking it\nup in a relevant git repository, making queries by hash behave consistently.\n\n### Analysis\n\nOnce ingestion is complete, analysis of the runs can be performed. The main\nanalysis currently available is a summary of the most recent test runs showing\ntheir overall success or failure, along with information on which tests have\nbeen flaky recently.  Values collected from tests can also be exported in\nOpenMetrics format to be analyzed and graphed by tools such as Prometheus and\nGrafana.\n\nA tool is also available to query the database to search for matching tests or\njobs manually.\n\n### Comment\n\nTest Clutch can periodically look at test results generated by PRs and add\ncomments to them regarding any test failures they caused. Specifically, it\nmentions when failed tests are known to be flaky (based on their history),\nwhether failures of the same test occurred in multiple test jobs, and in each\ncase provides a link to the test logs for quick access.\n\n## Installation\n\nThe latest source code can be obtained from\nhttps://github.com/dfandrich/testclutch/\n\nThe code is written entirely in Python.  Build and install the latest code\nfrom Github with:\n\n  `python -m pip install https://github.com/dfandrich/testclutch/archive/refs/heads/master.tar.gz`\n\nThe regression test suite can be run with the command:\n\n  `pytest`\n\nor\n\n  `python -m unittest`\n\nTest Clutch does not require git for its basic functions, but those directly\ninvolved in querying git (like `tcgitcommitinfo`) require the git command-line\ntool be installed (version 1.8.3 and newer are known to work).\n\n## Configuration\n\n### Configuration File\n\nCreate a file `~/.config/testclutchrc` in the same format as `configdef.py`\n(you can use the file `examples/testclutchrc` as a template).\nMost items have sane defaults, but `check_repo` must be set to a URL for the\nrepository of your source code. This is used as an identifier in the database\nand can be used by augmentation jobs.  Configuration options can be overridden\non the command line by using `--set NAME=VALUE`. Note that values must be\nspecified as in the configuration file, which, in particular, means that\nstrings must be surrounded by quotation marks which will normally need to be\nescaped to get them through the shell.\n\n### Ingestion Jobs\n\nYou must configure the ingestion jobs for the CI services you are using. Test\nClutch supports these CI services:\n\n* Appveyor\n* Azure\n* Circle CI\n* Cirrus\n* GitHub Actions\n* curl autobuild (specific to the curl project)\n\nSome of these may require credentials to access the log files. These are\nconfigured on the command-line according to each service's needs using the\n`--account` and `--authfile` command-line options. Currently, only GitHub\nActions needs an authentication token, stored in the file in the `--authfile`\nargument. Create one by going to https://github.com/settings/tokens and\nchoosing *fine-grained tokens*. Create a new token with these characteristics:\n\n  - Only select repositories (selecting the source repository or repositories)\n  - \"Metadata\" repository permissions (read)\n  - \"Pull requests\" repository permissions (read and write)\n\nAlternately, a classic token works as well, with the `public_repo` scope\nenabled.  Copy the token contents from the web browser and store it in a file\nin a protected location on your local machine and with permissions that do not\nallow other users to access it. Tokens set up this way will work with public\nrepositories.\n\nTest Clutch has built-in support for these test log formats:\n\n* curl runtests (specific to the curl project)\n* Gnu automake\n* Pytest (short, verbose, color, and with xdist and pytest-astropy-header)\n* Python unittest (with -v -b flags)\n\nThe logs parsed are those that are displayed as the tests are being run.  If\nyour test runner uses a different format for reporting on test results, you\nwill need to create a parser to ingest them.\n\n### Reports Jobs\n\nThe most interesting report currently available is obtained by running\n`tcanalysissum`. Another program `tcanalyzepr` is also available to download\ntest results relating to a GitHub PR to summarize the run tests and indicate\nwhich failing tests have recently been flaky.\n\n### Sample Update Script\n\nThe file `examples/daily-update` is an example that you can use to create a\ncustom periodic update script for your own use case. It can be installed as a\ncron job to generate up-to-date test reports.\n\n### Prometheus\n\nTest Clutch can export test times in OpenMetrics format which can be imported\ninto Prometheus for analysis and graphing.  The file\n`prometheus/data/user/testclutch.html` is a static page containing links to\nsome useful graphs that can be hosted on a Prometheus instance by providing\nthe `--web.user-assets=user` option (when `prometheus/data/` is the Prometheus\nbase directory). That page will be available at\nhttp://localhost:9090/user/testclutch.html (or whatever origin your instance\nis using).\n\nThe default Prometheus retention time is quite short relative to the rate of\nTest Clutch data generated, which is generally most interesting when viewed\nover weeks rather than hours or even days.  To avoid data from being\nautomatically deleted too early, supply an option like\n`--storage.tsdb.retention.time=26w` to keep half a year of data.\n\nYou can import about one month's worth of data into Prometheus by running:\n```\ntcquerytests --since 720 --format openmetrics \u003emetrics.om\npromtool tsdb create-blocks-from openmetrics metrics.om\n```\n\n## Programs\n\nThese are the main entry points to Test Clutch. Most of them access `--help` to\nshow some information on using them.\n\n### tcanalysissum\n\nAnalyze ingested data looking for patterns of failure in the tests and generate\na report in text or HTML formats.\n\n### tcanalyzepr\n\nAnalyze logs from a GitHub PR for patterns of failure in the tests and generate\na report in text or HTML formats. Or, check if the CI jobs associated with a PR\nhave run to completion. Note that --only-failed-prs may not be reliable if use\nwith commands other than --ci-status.\n\n### tcaugmentcurldaily\n\nAdds git commit hashes \u0026 summary to autobuilds built from curl's daily\ntarballs.\n\nOBSOLETE since curl started storing the commit hash in daily tarballs on\n2024-08-07.\n\n### tcaugmentgithash\n\nAdds full-length git commit hashes \u0026 summary to runs that only have a short\nhash. This is currently only applicable to curl autobuild logs.\n\n### tcfindtestruns\n\nFinds test job runs in which a particular test failed or succeeded.\n\n### tcgitcommitinfo\n\nReads in information about git commits into the database.\n\n### tcingestlog\n\nReads test results from log files from CI services and ingests them into the\ndatabase.\n\n### tcmetadatastats\n\nCreate reports summarizing the [metadata](metadata.md) and statistics about\nrecent test logs.\n\n\n## Debugging Programs\n\nThese are additional entry points that can be useful for debugging.\n\n### tcquerytests\n\nShow information about a specific ingested job based on specific\n[metadata](metadata.md). This can output in text or OpenMetrics formats.\nA single query term may be given, which restricts output to those runs that\nmatch on the given metadata. A query term is of the format `field\u003cop\u003evalue`\nwhere `\u003cop\u003e` is one of `=` `\u003c\u003e` `!=` `\u003c=` `\u003e=` `\u003c` `\u003e` `%` or `!%`. The\noperators have their expected meanings, except `%` and `!%` mean the SQL `LIKE`\nand `NOT LIKE` operators, respectively.\n\n\n### tcdbutil\n\nPerform low-level manipulations of the database such as deleting a run or\nchecking a commit chain.\n\n### tclogparse\n\nParse a single log file on disk from stdin and view the parsed data on the\nstdout using the configured log parsers.\n\n### testclutch.augment.curldailyinfo\n\nRead the specified curl daily tarball and dump its metadata.\n\n### testclutch.prdef\n\nDump the raw form of the analyzepr PR status cache.\n\n## Code Structure\n\n### logparser\n\nModules that parse different test log formats.\n\n### ingest\n\nModules that ingest test logs from various CI services.\n\n### augment\n\nModules that augment test metadata with additional data potentially from\nelsewhere.\n\n### tests\n\nModules for regression testing the rest of the code.\n\n### examples\n\nSome example configuration and script files are supplied for reference.\n\n## Extending\n\nYou can create plug-ins for ingesting new log formats by writing a Python\nmodule similar to the existing ingestion ones and by referencing it by name in\nthe `log_parsers` configuration entry. Try to use metadata in the same format\nas existing parsers, if possible and relevant, to make future analysis tasks\nsimpler.  See [metadata](metadata.md) for a list of standard mandatory and some\noptional metadata types.\n\n## Compatibility\n\nTest Clutch is in rapid development and no guarantees of compatibility with\nfuture or previous versions is currently being made. Contact the developers if\nyou would like to propose an API be stabilized. The first one is likely to be\nthe test log parsing API.\n\n## Future Work\n\nHaving a database filled with test run information opens up a range of\npossibilities for its use. Here are some:\n\n* determining which tests are flaky to prioritize work to fix them\n\n* keeping track of the current success/failure status of tests on the master\n  branch\n\n* notifying developers who are responsible for submitting a change that caused\n  tests to start failing\n\n* notifying PR developers through GitHub PR comments about failing tests that\n  are likely to be flaky and not the fault of the PR\n\n* finding commonalities between failing tests, e.g. tests that fail only on ARM\n  processors, or tests that fail only on a Linux 6.1 kernel, or tests that fail\n  only with clang 11.\n\n* identifying when a commit causes test coverage to suddenly drop\n\n* identifying which CI builds have the most/least test coverage\n\n* determine which builds are running a specific test number\n\n* finding builds that match specific build criteria (e.g. compiler version, OS,\n  curl features, curl dependencies) and looking at their test results\n\n* seeing if specific tests started running faster (or slower) after a specific\n  commit\n\n* alerting somebody on specific conditions, such as if no tests have been run\n  in 2 days, or the overall test success rate drops below 95%\n\n## License\n\nCopyright (C) 2023–2025  Daniel Fandrich dan@telarity.com\n\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU Affero General Public License as published by\nthe Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\nGNU Affero General Public License for more details.\n\nYou should have received a copy of the GNU Affero General Public License\nalong with this program.  If not, see \u003chttps://www.gnu.org/licenses/\u003e.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdfandrich%2Ftestclutch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdfandrich%2Ftestclutch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdfandrich%2Ftestclutch/lists"}