{"id":15714613,"url":"https://github.com/eskerda/busted-flaky","last_synced_at":"2025-03-30T19:47:15.814Z","repository":{"id":137874685,"uuid":"339502822","full_name":"eskerda/busted-flaky","owner":"eskerda","description":"busted-flaky is a busted extension for handling flaky specs, heavily inspired on RSpec::Retry. Using busted-flaky any block can be retried a specified number of times until it succeeds.","archived":false,"fork":false,"pushed_at":"2023-10-17T13:19:46.000Z","size":8,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-05T22:39:11.818Z","etag":null,"topics":["busted","flaky","flaky-tests","lua"],"latest_commit_sha":null,"homepage":"","language":"Lua","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eskerda.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-02-16T19:10:09.000Z","updated_at":"2024-10-13T20:57:35.000Z","dependencies_parsed_at":"2023-12-19T04:23:00.447Z","dependency_job_id":null,"html_url":"https://github.com/eskerda/busted-flaky","commit_stats":{"total_commits":4,"total_committers":1,"mean_commits":4.0,"dds":0.0,"last_synced_commit":"898f278c0b090c68adfda22c43e6c93448fb8dcf"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eskerda%2Fbusted-flaky","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eskerda%2Fbusted-flaky/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eskerda%2Fbusted-flaky/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eskerda%2Fbusted-flaky/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eskerda","download_url":"https://codeload.github.com/eskerda/busted-flaky/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246372503,"owners_count":20766626,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["busted","flaky","flaky-tests","lua"],"created_at":"2024-10-03T21:38:55.643Z","updated_at":"2025-03-30T19:47:15.789Z","avatar_url":"https://github.com/eskerda.png","language":"Lua","funding_links":[],"categories":[],"sub_categories":[],"readme":"# busted-flaky\n\n`busted-flaky` is a [busted](https://olivinelabs.com/busted/) extension for\nhandling flaky specs, heavily inspired on RSpec::Retry. Using `busted-flaky`\nany block can be retried a specified number of times until it succeeds.\n\n## What is test flakiness?\n\n\u003e A flaky test is a test that both passes and fails periodically without any\ncode changes. Flaky tests are definitely annoying but they can also be quite\n\u003e costly since they often require engineers to retrigger entire builds on CI and\n\u003e often waste a lot of time waiting for new builds to complete successfully.\n\u003e\n\u003e --- [Test Flakiness – Methods for identifying and dealing with flaky tests](https://engineering.atspotify.com/2019/11/18/test-flakiness-methods-for-identifying-and-dealing-with-flaky-tests/)\n\n## Installation\n\n```bash\n$ luarocks install busted-flaky\n```\n\n## Usage\n\n`busted-flaky` adds a new block called `flaky`. Use it around `it` and\n`describe` blocks for these to be retried an specific number of times until\nthey succeed. All setup / teardown / finally blocks enclosed will also get\nexecuted.\n\n\n```lua\ndescribe(\"some thing\", function()\n\n  it(\"does stuff\", function()\n\n  end)\n\n  flaky(function()\n\n    it(\"does sometimes fail stuff\", function()\n\n    end)\n  end)\n\n  flaky(\"accepts a description too\", function()\n\n    describe(\"has a feature\", function()\n      lazy_setup(function()\n\n      end)\n\n      teardown(function()\n\n      end)\n\n      it(\"does stuff\", function()\n\n      end)\n    end)\n  end)\nend)\n```\n\nA name level shortcut is available to wrap blocks on runtime, by tagging them\nwith `#flaky`.\n\n```lua\ndescribe(\"some thing\", function()\n\n  it(\"does stuff\", function()\n\n  end)\n\n  it(\"does sometimes fail stuff #flaky\", function()\n\n  end)\n\n  describe(\"has a #flaky feature\", function()\n\n    lazy_setup(function()\n\n    end)\n\n    teardown(function()\n\n    end)\n\n    it(\"does stuff\", function()\n\n    end)\n  end)\nend)\n```\n\nThe extension needs to be loaded by busted on runtime using the `--helper`\nargument for the runner.\n\n```bash\n$ busted -o gtest --helper=flaky path/to/some_spec.lua\n```\n\n## Configuration\n\n### attempts\n\n\u003e default: `5`\n\nSet the number of attempts the block will be retried until it succeeds.\n\n### format\n\n\u003e default `[#{tag} ({attempt}/{attempts}) {name}]`\n\u003e like: [#flaky (3/5) some comment]\n\nFormat string for flaky attempts.\n\n### wait\n\n\u003e default `0`\n\nNumber of seconds to wait between flaky attempts.\n\n### callback\n\nA callback that will be run between attempts. Can only configured through\nblock attributes.\n\n\n```lua\n\nflaky(\"some flaky block\", function()\n  it(...)\nend, {\n  callback = function(element, status, attempts, ctx)\n    -- gets run on every attempt\n  end,\n})\n```\n\n### tag\n\n\u003e default: `flaky`\n\nThe tag used for name level wrapping shortcut. Can only be configured through\ncli arguments.\n\n### cli arguments\n\n```\n  --attempts=NUM    number of attempts for flaky blocks (default: 5)\n  --tag=TAG         tag for marking flaky blocks (default: 'flaky')\n  --format=FORMAT   format string for flaky blocks\n  --wait=TIME       seconds to wait between retries (default: 0)\n```\n\n```bash\n$ busted --helper=flaky -Xhelper --attempts=10 -Xhelper --tag=failure ...\n```\n\n### block attributes\n\nflaky blocks accept block level attributes to customize them.\n\n```lua\ndescribe(\"something\", function()\n  it(\"does stuff\", function() end)\n\n  flaky(function()\n    -- something\n  end, {\n    -- block level attributes\n    wait = 5,\n    retry = 10,\n    callback = function(element, status, attempts, ctx)\n      -- ...\n    end\n  })\n```\n\n## Example\n\n\n```lua\n-- spec/flaky_spec.lua\n\nlocal NOOP = function() end\n\ndescribe(\"busted-flaky:\", function()\n\n  describe(\"when unused\", function()\n    describe(\"does leave describe blocks alone\", function()\n      it(\"and it blocks too\", NOOP)\n    end)\n  end)\n\n  describe(\"flaky block\", function()\n\n    describe(\"description\", function()\n\n      flaky(\"some description of the issue\", function()\n        it(\"can use an (optional) description\", NOOP)\n      end)\n\n      flaky(function()\n        it(\"does work without a description too\", NOOP)\n      end)\n\n      flaky(function()\n        it(\"does accept a different description format\", NOOP)\n      end, { fmt = \"[ #flaky run {attempt} of {attempts} ]\" })\n    end)\n\n    describe(\"contained blocks get retried until they succeed\", function()\n\n      local n = 1\n      local p = 1\n\n      after_each(function()\n        p = p * 2\n      end)\n\n      flaky(\"only works the 3rd time\", function()\n        describe(\"assert\", function()\n\n          after_each(function()\n            n = n + 1\n          end)\n\n          it(tostring(n) .. \" == 3\", function()\n            assert.is_equal(3, n)\n          end)\n        end)\n      end)\n\n      describe(\"more attempts can be specified\", function()\n        flaky(\"only works the 7th time\", function()\n          it(tostring(p) .. \" == 512\", function()\n            assert.is_equal(512, p)\n          end)\n        end, { attempts = 10 })\n      end)\n\n    end)\n\n    describe(\"a tag shortcut can be used too\", function()\n\n      local n = 1\n      local p = 1\n\n      after_each(function()\n        p = p * 2\n      end)\n\n      describe(\"#flaky assert\", function()\n\n        after_each(function()\n          n = n + 1\n        end)\n\n        it(tostring(n) .. \" == 3\", function()\n          assert.is_equal(3, n)\n        end)\n      end)\n\n    end)\n\n  end)\n\nend)\n```\n\n```bash\n$ busted -o gtest --helper=flaky spec/flaky_spec.lua\n[==========] Running tests from scanned files.\n[----------] Global test environment setup.\n[----------] Running tests from example_spec.lua\n[ RUN      ] example_spec.lua @ 7: busted-flaky: when unused does leave describe blocks alone and it blocks too\n[       OK ] example_spec.lua @ 7: busted-flaky: when unused does leave describe blocks alone and it blocks too (0.58 ms)\n[ RUN      ] example_spec.lua @ 16: busted-flaky: flaky block description [#flaky (1/5) some description of the issue] can use an (optional) description\n[       OK ] example_spec.lua @ 16: busted-flaky: flaky block description [#flaky (1/5) some description of the issue] can use an (optional) description (0.50 ms)\n[ RUN      ] example_spec.lua @ 20: busted-flaky: flaky block description [#flaky (1/5)] does work without a description too\n[       OK ] example_spec.lua @ 20: busted-flaky: flaky block description [#flaky (1/5)] does work without a description too (0.54 ms)\n[ RUN      ] example_spec.lua @ 24: busted-flaky: flaky block description [ #flaky run 1 of 5 ] does accept a different description format\n[       OK ] example_spec.lua @ 24: busted-flaky: flaky block description [ #flaky run 1 of 5 ] does accept a different description format (0.54 ms)\n[ RUN      ] example_spec.lua @ 44: busted-flaky: flaky block contained blocks get retried until they succeed [#flaky (1/5) only works the 3rd time] assert 1 == 3\nexample_spec.lua:45: Expected objects to be equal.\nPassed in:\n(number) 1\nExpected:\n(number) 3\n[ RUN      ] example_spec.lua @ 44: busted-flaky: flaky block contained blocks get retried until they succeed [#flaky (2/5) only works the 3rd time] assert 2 == 3\nexample_spec.lua:45: Expected objects to be equal.\nPassed in:\n(number) 2\nExpected:\n(number) 3\n[ RUN      ] example_spec.lua @ 44: busted-flaky: flaky block contained blocks get retried until they succeed [#flaky (3/5) only works the 3rd time] assert 3 == 3\n[       OK ] example_spec.lua @ 44: busted-flaky: flaky block contained blocks get retried until they succeed [#flaky (3/5) only works the 3rd time] assert 3 == 3 (0.62 ms)\n[ RUN      ] example_spec.lua @ 52: busted-flaky: flaky block contained blocks get retried until they succeed more attempts can be specified [#flaky (1/10) only works the 7th time] 8 == 512\nexample_spec.lua:53: Expected objects to be equal.\nPassed in:\n(number) 8\nExpected:\n(number) 512\n[ RUN      ] example_spec.lua @ 52: busted-flaky: flaky block contained blocks get retried until they succeed more attempts can be specified [#flaky (2/10) only works the 7th time] 16 == 512\nexample_spec.lua:53: Expected objects to be equal.\nPassed in:\n(number) 16\nExpected:\n(number) 512\n[ RUN      ] example_spec.lua @ 52: busted-flaky: flaky block contained blocks get retried until they succeed more attempts can be specified [#flaky (3/10) only works the 7th time] 32 == 512\nexample_spec.lua:53: Expected objects to be equal.\nPassed in:\n(number) 32\nExpected:\n(number) 512\n[ RUN      ] example_spec.lua @ 52: busted-flaky: flaky block contained blocks get retried until they succeed more attempts can be specified [#flaky (4/10) only works the 7th time] 64 == 512\nexample_spec.lua:53: Expected objects to be equal.\nPassed in:\n(number) 64\nExpected:\n(number) 512\n[ RUN      ] example_spec.lua @ 52: busted-flaky: flaky block contained blocks get retried until they succeed more attempts can be specified [#flaky (5/10) only works the 7th time] 128 == 512\nexample_spec.lua:53: Expected objects to be equal.\nPassed in:\n(number) 128\nExpected:\n(number) 512\n[ RUN      ] example_spec.lua @ 52: busted-flaky: flaky block contained blocks get retried until they succeed more attempts can be specified [#flaky (6/10) only works the 7th time] 256 == 512\nexample_spec.lua:53: Expected objects to be equal.\nPassed in:\n(number) 256\nExpected:\n(number) 512\n[ RUN      ] example_spec.lua @ 52: busted-flaky: flaky block contained blocks get retried until they succeed more attempts can be specified [#flaky (7/10) only works the 7th time] 512 == 512\n[       OK ] example_spec.lua @ 52: busted-flaky: flaky block contained blocks get retried until they succeed more attempts can be specified [#flaky (7/10) only works the 7th time] 512 == 512 (0.79 ms)\n[ RUN      ] example_spec.lua @ 75: busted-flaky: flaky block a tag shortcut can be used too [#flaky (1/5) flaky]  assert 1 == 3\nexample_spec.lua:76: Expected objects to be equal.\nPassed in:\n(number) 1\nExpected:\n(number) 3\n[ RUN      ] example_spec.lua @ 75: busted-flaky: flaky block a tag shortcut can be used too [#flaky (2/5) flaky]  assert 2 == 3\nexample_spec.lua:76: Expected objects to be equal.\nPassed in:\n(number) 2\nExpected:\n(number) 3\n[ RUN      ] example_spec.lua @ 75: busted-flaky: flaky block a tag shortcut can be used too [#flaky (3/5) flaky]  assert 3 == 3\n[       OK ] example_spec.lua @ 75: busted-flaky: flaky block a tag shortcut can be used too [#flaky (3/5) flaky]  assert 3 == 3 (0.61 ms)\n[----------] 7 tests from example_spec.lua (27.96 ms total)\n\n[----------] Global test environment teardown.\n[==========] 7 tests from 1 test file ran. (29.37 ms total)\n[  PASSED  ] 7 tests.\n```\n\n## Credits\n\n* rspec:retry\n* busted\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feskerda%2Fbusted-flaky","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feskerda%2Fbusted-flaky","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feskerda%2Fbusted-flaky/lists"}