{"id":15707889,"url":"https://github.com/kostmo/circleci-failure-tracker","last_synced_at":"2025-05-12T19:54:30.349Z","repository":{"id":86949809,"uuid":"181066897","full_name":"kostmo/circleci-failure-tracker","owner":"kostmo","description":"A log analyzer for CircleCI.  Note that this project is now hosted at pytorch/dr-ci","archived":false,"fork":false,"pushed_at":"2020-06-04T21:15:45.000Z","size":1855,"stargazers_count":5,"open_issues_count":25,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-01T02:52:57.390Z","etag":null,"topics":["ci","circleci","log-analysis"],"latest_commit_sha":null,"homepage":"https://github.com/pytorch/dr-ci","language":"Haskell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kostmo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-12T18:53:47.000Z","updated_at":"2020-09-20T18:03:01.000Z","dependencies_parsed_at":"2023-05-29T23:00:28.053Z","dependency_job_id":null,"html_url":"https://github.com/kostmo/circleci-failure-tracker","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kostmo%2Fcircleci-failure-tracker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kostmo%2Fcircleci-failure-tracker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kostmo%2Fcircleci-failure-tracker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kostmo%2Fcircleci-failure-tracker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kostmo","download_url":"https://codeload.github.com/kostmo/circleci-failure-tracker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253813137,"owners_count":21968428,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ci","circleci","log-analysis"],"created_at":"2024-10-03T20:41:38.888Z","updated_at":"2025-05-12T19:54:30.328Z","avatar_url":"https://github.com/kostmo.png","language":"Haskell","funding_links":[],"categories":[],"sub_categories":[],"readme":"- [![TravisCI Build Status](https://travis-ci.com/kostmo/circleci-failure-tracker.svg?branch=master)](https://travis-ci.com/kostmo/circleci-failure-tracker)\n\n# A log analyzer for CircleCI\n\n\n## Intro\n\nAn organization would like to determine what the most common causes of intermittent build\nfailures/flaky tests are in the a repository so that effort can be prioritized\nto fix them.\n\n## Outputs\n\nThe Dr. CI project entails two distinct user-facing outputs:\n\n* [Automatically-posted GitHub PR comments](application-logic/pull-request-comments)\n* The Dr. CI website\n\nThe latter has several distinct utilities:\n* Annotation interface for determinisitic `master` failures\n* Flakiness review tool\n* Stats dashboards\n\n# Codebase\n\nSee [docs/CODEBASE-OVERVIEW.md](docs/CODEBASE-OVERVIEW.md).\n\n# Repository assumptions\n\nDr. CI assumes a linear history of the `master` branch.\nThis can be enforced on GitHub via the following setting under the \"Branches\" -\u003e \"Branch protection rule\" section for `master`:\n\n![GitHub setting](docs/images/github-linear-history-option.png)\n\n## Functionality\nThis tool obtains a list of CircleCI builds run against a GitHub repository for\na master branch, downloads their logs (stripped of ANSI escape codes) from AWS, and scans the logs for a\npredefined list of labeled patterns (regular expressions).\n\nThese patterns are curated by an operator.  The frequency of occurrence of each\npattern are tracked and presented in a web UI.\n\nThe database tracks which builds have been already scanned for a given pattern,\nso that scanning may be performed incrementally or resumed after abort.\n\n### Tool workflow\n\n* A webhook listens for build status changes on a GitHub PR\n* For each failed build, that build's log will be scanned for any of the patterns in the database tagged as \"flaky\"\n* If all of the failures were flaky, the indicator will be green.  There will be a link in the status box to dive into the details.\n* likewise for failures marked with my tool as \"known problems\"\n\n### Known Problem reporting\n\nRequiring that failures in the master branch be annotated will facilitate tracking of the frequency of \"brokenness\" of master over time, and allow measurement of whether this metric is improving.\n\nIt is possible for only specific jobs of a commit to be marked as \"known broken\", e.g. the Travis CI Lint job.\n\n## Log scanning data flow diagram\n\n![flow diagram](docs/data-flow.svg)\n\n\nDeployment\n-------------\n\n### Development Environment Setup\n\nSee: [docs/development-environment](docs/development-environment)\n\n\n### AWS dependencies and deployment\n\nSee: [docs/aws](docs/aws)\n\n\n### Ingestion overview\n\n1. A small webservice (named `gh-notification-ingest-env` in Elastic Beanstalk, and hosted at domain `github-notifications-ingest.pytorch.org`) receives GitHub webhook notifications and stores them (synchronously) in a database.\n2. A periodic (3-minute interval) AWS Lambda task `EnqueSQSBuildScansFunction` queries for unprocessed notifications in the database, and enqueues an SQS message for each of them.\n3. Finally, an Elastic Beanstalk Worker-tier server named `log-scanning-worker` process the SQS messages as capacity allows.\n\nWe want a cool-off period during which multiple builds for a given commit can be aggregated into one task for that commit.\nThis is accomplished via an SQS deduplicating queue, where multiple instances of the same commit are consolidated while in the queue.\n\n\nOptimizations\n-------------\n\n* We can skip inspecting *all* of the \"previously-visited\" builds if the master \"scan\" record points to the newest pattern ID.\n    * Better yet, use a single DB query to get the list of out-of-date \"already-visited\" builds, instead of a separate query per build to obtain the unscanned pattern list.\n\n\n## Other Features\n\n* Periodically fetches builds directly from CircleCI API to catch up on GitHub notifications that may have been dropped\n\n\n## Source attribution\n\nAho-Corasick implementation is from here: https://github.com/channable/alfred-margaret\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkostmo%2Fcircleci-failure-tracker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkostmo%2Fcircleci-failure-tracker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkostmo%2Fcircleci-failure-tracker/lists"}