{"id":30219138,"url":"https://github.com/mastert/flaky_test_tracker","last_synced_at":"2025-10-07T13:26:25.549Z","repository":{"id":42984174,"uuid":"391608054","full_name":"masterT/flaky_test_tracker","owner":"masterT","description":"Automatic flaky test tracking system.","archived":false,"fork":false,"pushed_at":"2022-03-28T19:21:17.000Z","size":209,"stargazers_count":2,"open_issues_count":2,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2023-08-05T16:03:32.975Z","etag":null,"topics":["flaky","flaky-tests","ruby","test","tracking"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/masterT.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-08-01T11:23:42.000Z","updated_at":"2023-08-05T16:03:32.976Z","dependencies_parsed_at":"2022-09-19T17:00:42.263Z","dependency_job_id":null,"html_url":"https://github.com/masterT/flaky_test_tracker","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"purl":"pkg:github/masterT/flaky_test_tracker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/masterT%2Fflaky_test_tracker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/masterT%2Fflaky_test_tracker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/masterT%2Fflaky_test_tracker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/masterT%2Fflaky_test_tracker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/masterT","download_url":"https://codeload.github.com/masterT/flaky_test_tracker/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/masterT%2Fflaky_test_tracker/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270385318,"owners_count":24574544,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-14T02:00:10.309Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["flaky","flaky-tests","ruby","test","tracking"],"created_at":"2025-08-14T07:47:58.340Z","updated_at":"2025-10-07T13:26:20.504Z","avatar_url":"https://github.com/masterT.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FlakyTestTracker\n\n\u003e **:warning: Work in progress**\n\n[![Build](https://github.com/masterT/flaky_test_tracker/actions/workflows/build.yml/badge.svg)](https://github.com/masterT/flaky_test_tracker/actions/workflows/build.yml)\n\nFlakyTestTracker is an automatic flaky test tracking system. It is testing framework agnostic and very customizable.\n\nFeatures:\n- automatically [track](#track) failing tests\n- automatically [resolve](#resolve) tracked failing tests\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'flaky_test_tracker'\n```\n\nAnd then execute:\n\n```\n$ bundle install\n```\n\nOr install it yourself as:\n\n```\n$ gem install flaky_test_tracker\n```\n\n## Documentation\n\nThe documentation is available online on [rubydoc.info](https://rubydoc.info/github/masterT/flaky_test_tracker).\n\n## Usage\n\n### Track\n\nYou can automatically track failing tests on the configured [storage](#storage) for any testing framework, as long as it can provide a unique _reference_ to identify the test.\n\nAdd failing test attributes to the internal queue:\n\n```ruby\nFlakyTestTracker.tracker.add(\n  reference: \"./spec/foo_spec.rb[1:2]\",\n  description: \"returns true when foo\",\n  exception: %{\n    expected: true\n     got: false\n    (compared using ==)\n    Diff:\n    @@ -1 +1 @@\n    -true\n    +false\n  },\n  file_path: \"./spec/foo_spec.rb\",\n  line_number: 8\n)\n```\n\nPersists the test on the configured storage.\n\nWhen test attributes has a `reference` matching a test persisted in the configured storage, the test's `number_occurrences` attribute will be incremented and the test will be updated with the test attributes. Otherwise, a test will be created with the test attributes and `number_occurrences` equals to `1`.\n\n```ruby\nFlakyTestTracker.tracker.track\n```\n\nReset the test attributes internal queue:\n\n```ruby\nFlakyTestTracker.tracker.clear\n```\n\n#### Examples\n\nExample with [RSpec](https://rubygems.org/gems/rspec) testing framework.\n\n```ruby\nRSpec.configure do |config|\n  config.before(:suite) do\n    FlakyTestTracker.tracker.clear\n  end\n\n  config.after do |example|\n    if example.exception\n      FlakyTestTracker.tracker.add(\n        reference: example.id,\n        description: example.full_description,\n        exception: example.exception.gsub(/\\x1b\\[[0-9;]*[a-zA-Z]/, \"\"), # Remove ANSI formatting.\n        file_path: example.metadata[:file_path],\n        line_number: example.metadata[:line_number]\n      )\n    end\n  end\n\n  config.after(:suite) do\n    FlakyTestTracker.tracker.track\n  rescue StandardError\n    # ...\n  end\nend\n```\n\n### Resolve\n\nYou can automatically resolve previously tracked failing tests on the configured [storage](#storage).\n\nThe `resolve` method calls the block with each tracked test and resolves those for which the block returns a truthy value.\n\nThe resolved test are not deleted from the configured [storage](#storage), they will have the attribute `resolved_at` set to the current time and will respond `true` to `resolved?`.\n\n```ruby\nFlakyTestTracker.resolver.resolve do |test|\n  # return truthy value to resolve the tracked test passed to block.\nend\n```\n\n#### Examples\n\nResolve tests with the last failing occurrence that occurred 14 days ago.\n\n```ruby\nDAY_IN_SECOND = 86_400\nFlakyTestTracker.resolver.resolve do |test|\n  test.finished_at \u003c Time.now - 14 * DAY_IN_SECOND\nend\n```\n\nIt is useful to periodically resolve tests, this can be done using a [Rake](https://rubygems.org/gems/rake) task.\n\n```ruby\n# frozen_string_literal: true\nnamesapce :falky_test_tracker do\n  desc 'Resolve tests with the last failing occurence tracked 14 days ago'\n  task :resolve do\n    DAY_IN_SECOND = 86_400\n    FlakyTestTracker.resolver.resolve do |test|\n      test.finished_at \u003c Time.now - 14 * DAY_IN_SECOND\n    end\n  end\nend\n```\n\n### Configuration\n\nThe configuration is kept in the `FlakyTestTracker` module and is used to initialize the module [tracked](#track) and [resolver](#resolve) instances.\n\n```ruby\nFlakyTestTracker.configure do |config|\n  config.storage_type = :github_issue\n  config.storage_options = {\n    client: {\n      access_token: ENV['GITHUB_ACCESS_TOKEN']\n    },\n    repository: 'foo/bar',\n    labels: ['flaky test']\n  }\n  config.source_type = :github\n  config.source_options = {\n    repository: \"foo/bar\",\n    commit: ENV[\"CI_COMMIT\"],\n    branch: ENV[\"CI_BRANCH\"]\n  }\n  config.reporter = []\n  config.context = {\n    ci_build_id: ENV['CI_BUILD_ID'],\n    ci_build_url: ENV['CI_BUILD_URL']\n  }\n  config.verbose = true\n  config.pretend = false\nend\n```\n\nYou can access the configuration through the module `::configuration` method.\n\n```ruby\nFlakyTestTracker.configuration\n# =\u003e #\u003cFlakyTestTracker::Configuration:0x00 ...\u003e\n```\n\n#### Storage\n\nThe storage is responsible to persist tests.\n\nYou can use configure the storage by specifying the type, class, or class name of the store along with the options. The `storage_options` will be passed the storage class method `::build` as keyword arguments using `**`.\n\n```ruby\nFlakyTestTracker.configure do |config|\n  config.storage_type = :github_issue\n  # or `config.storage_class = FlakyTestTracker::Storage::GitHubIssueStorage`\n  # or `config.storage_class_name = \"FlakyTestTracker::Storage::GitHubIssueStorage\"`\n  config.storage_options = {}\nend\n```\n\nAlternatively, you can also specify the storage instance directly.\n\n```ruby\nFlakyTestTracker.configure do |config|\n  config.storage = FlakyTestTracker::Storage::GitHubIssueStorage.build(**options)\nend\n```\n\nYou can access the configured storage through the  method `#storage` of the `configuration` instance.\n\n```ruby\nFlakyTestTracker.configuration.storage\n# =\u003e #\u003cFlakyTestTracker::Storage::GitHubIssueStorage:0x00 ...\u003e\n```\n\n##### Storages\n\n|Storage Type|Storage Class|Documentation|\n|---|---|---|\n|`:github_issue`|`FlakyTestTracker::Storage::GitHubIssueStorage`|...|\n\n##### Custom Storage\n\nYou can use a custom storage, the storage class will need to have to the following interface:\n\n```ruby\nclass MyStorage\n  def self.build(**options)\n    new(**options)\n  end\n\n  def initialize(**options)\n    # ...\n  end\n\n  # @return [Array\u003cTest\u003e]\n  def all\n    # ...\n  end\n\n  # @param [TestInput] test_input\n  # @return [Test]\n  def create(test_input)\n    # ...\n  end\n\n  # @param [String] id\n  # @param [TestInput] test_input\n  # @return [Test]\n  def update(id, test_input)\n    # ...\n  end\n\n  # @return [Test]\n  def delete(id)\n    # ...\n  end\nend\n```\n\n#### Source\n\nThe source represents the current source code version on which tests are executed. It is used to resolves a test file location.\n\nYou can use configure the source by specifying the type, class, or class name, along with the options. The `source_options` will be passed the source class method `::build` as keyword arguments using `**`.\n\n```ruby\nFlakyTestTracker.configure do |config|\n  config.source_type = :github\n  # or `config.source_class = FlakyTestTracker::Source::GitHubSource`\n  # or `config.source_class_name = \"FlakyTestTracker::Source::GitHubSource\"`\n  config.source_options = {}\nend\n```\n\nAlternatively, you can also specify the source instance directly.\n\n```ruby\nFlakyTestTracker.configure do |config|\n  config.source = FlakyTestTracker::Source::GitHubSource.build(**options)\nend\n```\n\nYou can access the configured source through the  method `#source` of the `configuration` instance.\n\n```ruby\nFlakyTestTracker.configuration.source\n# =\u003e #\u003cFlakyTestTracker::Source::GitHubSource:0x00 ...\u003e\n```\n\n##### Sources\n\n|Source Type|Source Class|Documentation|\n|---|---|---|\n|`:github`|`FlakyTestTracker::Source::GitHubSource`|...|\n\n##### Custom Source\n\nYou can use a custom source, the source class will need to have to the following interface:\n\n```ruby\nclass MySource\n  def self.build(**options)\n    new(**options)\n  end\n\n  def initialize(**options)\n    # ...\n  end\n\n  # @param file_path [String]\n  # @param line_number [Integer]\n  # @return [URI]\n  def file_source_location_uri(file_path:, line_number:)\n    # ...\n  end\n\n  # @return [URI]\n  def source_uri\n    # ...\n  end\nend\n```\n\n#### Reporter\n\nThe reporter can be used to alert your team on your preferred communication channel (email, Slack, Discord, etc.).\n\nThe [tracker](#tracker) will call the reporter `#tracked_tests` method with all the tests tracked, the [source](#source), and the [context](#context).\n\nThe [resolver](#resolver) will calls the reporter `#resolved_tests` method with all the tests resolved.\n\nYou can configure the reporter by specifying the class, or class name, along with the options. The `reporter_options` will be passed the reporter class method `::build` as keyword arguments using `**`.\n\n```ruby\nFlakyTestTracker.configure do |config|\n  config.reporter_class = MyReporter\n  # or `config.reporter_class = \"MyReporter\"`\n  config.reporter_options = {}\nend\n```\n\nAlternatively, you can also specify the reporter instance directly.\n\n```ruby\nFlakyTestTracker.configure do |config|\n  config.reporter = MyReporter.build(**options)\nend\n```\n\nYou can access the configured reporter through the  method `#reporter` of the `configuration` instance.\n\n```ruby\nFlakyTestTracker.configuration.reporter\n# =\u003e #\u003cMyReporter:0x00 ...\u003e\n```\n\n##### Reporter\n\n|Reporter Class|Documentation|\n|---|---|\n|`FlakyTestTracker::Reporter::CollectionReporter`|...|\n\n##### Custom Reporter\n\nYou can use a custom reporter. The reporter class will need to respect the following interface:\n\n```ruby\nclass MyReporter\n  def self.build(**options)\n    new(**options)\n  end\n\n  def initialize(**options)\n    # ...\n  end\n\n  # @param tests [Array\u003cTest\u003e]\n  # @param source [#file_source_location_uri, #source_uri]\n  # @param context [Hash]\n  def tracked_tests(tests:, source:, context:)\n    # ...\n  end\n\n  # @param tests [Array\u003cTest\u003e]\n  def resolved_tests(tests:)\n    # ...\n  end\nend\n```\n\nIf you inherit from `FlakyTestTracker::Reporter::BaseReporter`, you will only need to create the `::build` and override the methods you need.\n\n```ruby\nclass MyReporter \u003c FlakyTestTracker::Reporter::BaseReporter\n  def self.build(**options)\n    new(**options)\n  end\n\n  def initialize(**options)\n    # ...\n  end\nend\n```\n\n#### Context\n\nYou can set a custom context that will be passed to your reporter `#tracked_tests` method. This is useful to identify the CI build in which the tests failed.\n\n```ruby\nFlakyTestTracker.configure do |config|\n  config.context = {}\nend\n```\n\nYou can access the configured context through the  method `#context` of the `configuration` instance.\n\n```ruby\nFlakyTestTracker.configuration.context\n# =\u003e {}\n```\n\n#### Verbose\n\nWhen set to `true`, verbose messages will be print in the `STDOUT`.\n\n```ruby\nFlakyTestTracker.configure do |config|\n  config.verbose = true\nend\n```\n\nYou can access the configured verbose option through the  method `#verbose` of the `configuration` instance.\n\n```ruby\nFlakyTestTracker.configuration.verbose\n# =\u003e true\n```\n\n#### Pretend\n\nWhen set to `true` the [tracker](#tracker) and [resolver](#resolver) will not make changes on the configured [storage](#storage). This is useful to test your configuration setup.\n\n```ruby\nFlakyTestTracker.configure do |config|\n  config.pretend = true\nend\n```\n\nYou can access the configured pretend option through the  method `#pretend` of the `configuration` instance.\n\n```ruby\nFlakyTestTracker.configuration.pretend\n# =\u003e true\n```\n\n## Development\n\nAfter checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.\n\nTo install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).\n\n### Documentation\n\nThe documentation is generated using [yard](https://yardoc.org/).\n\nBuild documentation\n\n```shell\nbundle exec yard doc\n```\n\n## Contributing\n\nBug reporter and pull requests are welcome on GitHub at https://github.com/masterT/flaky_test_tracker. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/masterT/flaky_test_tracker/blob/main/CODE_OF_CONDUCT.md).\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).\n\n## Code of Conduct\n\nEveryone interacting in the FlakyTestTracker project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the [code of conduct](https://github.com/masterT/flaky_test_tracker/blob/main/CODE_OF_CONDUCT.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmastert%2Fflaky_test_tracker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmastert%2Fflaky_test_tracker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmastert%2Fflaky_test_tracker/lists"}