{"id":18510702,"url":"https://github.com/operator-framework/flake-analyzer","last_synced_at":"2025-10-31T19:30:25.490Z","repository":{"id":61626975,"uuid":"278380646","full_name":"operator-framework/flake-analyzer","owner":"operator-framework","description":null,"archived":false,"fork":false,"pushed_at":"2020-08-08T15:31:58.000Z","size":2336,"stargazers_count":1,"open_issues_count":1,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-12-25T20:26:16.862Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/operator-framework.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-07-09T14:02:33.000Z","updated_at":"2023-07-25T14:37:18.000Z","dependencies_parsed_at":"2022-10-19T19:00:11.187Z","dependency_job_id":null,"html_url":"https://github.com/operator-framework/flake-analyzer","commit_stats":null,"previous_names":["operator-framework/flak-analyzer"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/operator-framework%2Fflake-analyzer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/operator-framework%2Fflake-analyzer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/operator-framework%2Fflake-analyzer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/operator-framework%2Fflake-analyzer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/operator-framework","download_url":"https://codeload.github.com/operator-framework/flake-analyzer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239225766,"owners_count":19603162,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T15:24:37.487Z","updated_at":"2025-10-31T19:30:25.434Z","avatar_url":"https://github.com/operator-framework.png","language":"Go","readme":"# Flake-Analyzer\n\nThis Flake Analyzer is a project to summarize JUNIT test reports generated from GitHub workflow where the results are\n uploaded to the repository at the end of every (failed) run. The analyzer generates reports by aggregating all the\n  results available and sort the failed tests by their occurrences from high to low. This analyzer is a\n   standalone project that can be used as a part of the workflow to download test results from GitHub as artifacts and\n    aggregate them.\n\nThe commenter is a feature to post flake report to a pull request based on the failed test runs from that PR. The analyzer performs a periodic check against the repo\n and post reports if a new artifact that has the new run ID and has the commit id associated with the PR exists.\n\nThe flake analyzer supports periodic reports for different time windows, errors from specific PRs, and commits with respect to different tests.\nThe flake analyzer has the following modes:\n  - Periodically generate report artifacts for a specific repository with respect to individual tests. (requires GITHUB_TOKEN)\n  - Comment on PRs failed for specific tests with aggregated error report. (requires Personal Access Token with access to download REPO artifact and post comment)\n    \nTo use the periodic flake analysis reporting feature, you need to set up the following:\n1. Upload your test results to your repository via workflow using this [template](#github-upload-junit-test-result-as-artifacts)).\n2. Include this project in your Repo's workflow and supply filters accordingly including Test Suite Name supplied as\n part of your uploaded artifact (eg. for [example](#report-daily/weekly-flake-failures)).\n3. Enjoy the analysis report as artifacts. \n\nTo use the commenting report feature, you need to set up the following:\n1. Upload your test results to your repository via workflow using this [template\n](#github-upload-junit-test-result-as-artifacts)).\n2. Create a new workflow on this repo to monitor your repositories using this [template](#enable-commenter).\n3. [Create a personal access token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) and add to flake-analyzer Repo\n to feed into the workflow from the second step for access your repository. (You need the token for downloading the artifacts)\n4. Test out commenting feature by opening a PR from non-fork repository.(forked repository can not access token)\n\n## Github Upload JUNIT Test Result as Artifacts\n\nTo analyize test results, you need to upload your test results onto Github. The following is an example of the\n worflow configuration `.github/workflows/\u003cYour Test Config\u003e.yml`. See https://github.com/actions/upload-artifact for\n  more information.\n  \n  The important thing is to upload your test artifacts in the format \u003cTest Suite Name\\\u003e-\u003cCommit\\\u003e-\u003cRun ID\\\u003e, which is\n   used in the example below.\n```yaml\n\u003cYour Reo\u003e/.github/workflows/\u003cyour test\u003e.yml\nname: \u003cTest Name\u003e\non:\n  pull_request:\njobs:\n  \u003cJob Name\u003e:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v2\n      - run: \u003cYour Test\u003e\n      - name: Archive production artifacts \n        if: ${{ always() }}\n        uses: actions/upload-artifact@v2\n        with:\n          name: \u003cTest Suite Name\u003e-${{(github.event.pull_request.head.sha||github.sha)}}-${{ github.run_id }}\n          path: \u003cPath/To/Your/Artifacts\u003e\n```\n\n## Report Daily/Weekly Flake Failures\n\n```yaml\n\u003cYour Reo\u003e/.github/workflows/\u003cyour periodics\u003e.yml\nname: flake-analyzer-periodics\non:\n  schedule:\n    - cron: '0 1 * * *' # daily\njobs:\n  generate-flake-analysis-report:\n    runs-on: ubuntu-latest\n    steps:\n      - name: Periodic Flake Report\n        env:\n          OWNER: \u003cyour repo owner\u003e\n          REPO: \u003cyour repo\u003e\n          TEST_SUITE: \u003cTest Suite Name\u003e\n          TOKEN: ${{ secrets.GITHUB_TOKEN }}\n        run: |\n          git clone -b v0.1.1 https://github.com/operator-framework/flake-analyzer.git\n          cd ./flake-analyzer\n          make report-today  OUTPUT_FILE=./report/artifacts/flake-report-today-$(date +\"%m-%d-%Y\").yaml\n          make report-last-7-days OUTPUT_FILE=./report/artifacts/flake-report-last-7-days-$(date +\"%m-%d-%Y\").yaml\n          make report-prev-7-days OUTPUT_FILE=./report/artifacts/flake-report-prev-7-days-$(date +\"%m-%d-%Y\").yaml\n      - name: Archive Reoport artifacts \n        uses: actions/upload-artifact@v2\n        with:\n          name: flake-report-${{ github.run_id }}\n          path: ${{ github.workspace }}/flake-analyzer/report/artifacts/*\n```\n\n## Enable Commenter\n\n```yaml\n\u003cflake-analyzer\u003e/.github/workflows/\u003cyour commenter\u003e.yml\nname: Post Report As Comment On PR\non:\n  pull_request:\n  schedule:\n    - cron: '*/15 * * * *'\njobs:\n  post-report-as-pr-comment:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v2\n      - name: Comment On PR\n        env:\n          OWNER: \u003cyour repo owner\u003e\n          REPO: \u003cyour repo\u003e\n          LOWNER: \u003cOwner of flake-analyzer, e.g. operator-framework\u003e\n          LREPO: flake-analyzer\n          TEST_SUITE: \u003cyour test suite\u003e\n          PROGRESS_FILE: ./artifacts/commenter-progress-\u003cYour Bot Name\u003e.yaml\n          ARTIFACT: flake-bot-\u003cyour owner\u003e-artifact\n          TOKEN: ${{secrets.\u003cPersonal Access Token Name\u003e}}\n        run: make commenter\n      - uses: actions/upload-artifact@v2 # upload your commenter progress\n        with:\n          name: flake-bot-\u003cyour owner\u003e-artifact\n          path: ./artifacts/commenter-progress-\u003cYour Bot Name\u003e.yaml\n```\n\n## Analysis Report Example\n```yaml\ntotaltestcount: 30\nflaketestcount: 43\nskippedtestcount: 4\nflaketests:\n- classname: End-to-end\n  name: Installing bundles with new object types when a bundle with a pdb, priorityclass,\n    and VPA object is installed should create the additional bundle objects\n  counts: 28\n  details:\n  - count: 28\n    error:\n      type: Failure\n      body: |-\n        /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/bundle_e2e_test.go:78\n        Timed out after 60.000s.\n        expected no error getting pdb object associated with CSV\n        Expected success, but got an error:\n            \u003c*errors.StatusError | 0xc00113eaa0\u003e: {\n                ErrStatus: {\n                    TypeMeta: {Kind: \"\", APIVersion: \"\"},\n                    ListMeta: {\n                        SelfLink: \"\",\n                        ResourceVersion: \"\",\n                        Continue: \"\",\n                        RemainingItemCount: nil,\n                    },\n                    Status: \"Failure\",\n                    Message: \"poddisruptionbudgets.policy \\\"busybox-pdb\\\" not found\",\n                    Reason: \"NotFound\",\n                    Details: {\n                        Name: \"busybox-pdb\",\n                        Group: \"policy\",\n                        Kind: \"poddisruptionbudgets\",\n                        UID: \"\",\n                        Causes: nil,\n                        RetryAfterSeconds: 0,\n                    },\n                    Code: 404,\n                },\n            }\n            poddisruptionbudgets.policy \"busybox-pdb\" not found\n        /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/bundle_e2e_test.go:98\n    systemout: |\n      15:13:31.0741: UpgradePending (busybox.v2.0.0): \u0026ObjectReference{Kind:InstallPlan,Namespace:operators,Name:install-cgghb,UID:580aae23-1784-4f6b-9486-ce9479f563c6,APIVersion:operators.coreos.com/v1alpha1,ResourceVersion:5826,FieldPath:,}\n      15:13:31.731: UpgradePending (busybox.v2.0.0): \u0026ObjectReference{Kind:InstallPlan,Namespace:operators,Name:install-cgghb,UID:580aae23-1784-4f6b-9486-ce9479f563c6,APIVersion:operators.coreos.com/v1alpha1,ResourceVersion:5826,FieldPath:,}\n      15:13:32.7313: UpgradePending (busybox.v2.0.0): \u0026ObjectReference{Kind:InstallPlan,Namespace:operators,Name:install-cgghb,UID:580aae23-1784-4f6b-9486-ce9479f563c6,APIVersion:operators.coreos.com/v1alpha1,ResourceVersion:5826,FieldPath:,}\n      15:13:33.7321: UpgradePending (busybox.v2.0.0): \u0026ObjectReference{Kind:InstallPlan,Namespace:operators,Name:install-cgghb,UID:580aae23-1784-4f6b-9486-ce9479f563c6,APIVersion:operators.coreos.com/v1alpha1,ResourceVersion:5826,FieldPath:,}\n      15:13:34.7309: UpgradePending (busybox.v2.0.0): \u0026ObjectReference{Kind:InstallPlan,Namespace:operators,Name:install-cgghb,UID:580aae23-1784-4f6b-9486-ce9479f563c6,APIVersion:operators.coreos.com/v1alpha1,ResourceVersion:5826,FieldPath:,}\n      15:13:35.7534: UpgradePending (busybox.v2.0.0): \u0026ObjectReference{Kind:InstallPlan,Namespace:operators,Name:install-cgghb,UID:580aae23-1784-4f6b-9486-ce9479f563c6,APIVersion:operators.coreos.com/v1alpha1,ResourceVersion:5826,FieldPath:,}\n      15:13:36.7354: AtLatestKnown (busybox.v2.0.0): \u0026ObjectReference{Kind:InstallPlan,Namespace:operators,Name:install-cgghb,UID:580aae23-1784-4f6b-9486-ce9479f563c6,APIVersion:operators.coreos.com/v1alpha1,ResourceVersion:5826,FieldPath:,}\n      skipping cleanup\n    systemerr: \"\"\n  commits:\n  - 0b8233d0c2eefb9c3b7402f3709525c7ec6752a7\n  - 15f0d9741dd33e2672b552540fa4ed564cec92ec\n  - ...\n  meandurationsec: 103.58520275000001\n...\n\nskippedtests:\n- classname: End-to-end\n  name: Subscriptions create required objects from Catalogs Given a Namespace when\n    a CatalogSource is created with a bundle that contains prometheus objects creating\n    a subscription using the CatalogSource should have created the expected prometheus\n    objects\n  counts: 30\n  details: []\n  commits:\n  - 0b8233d0c2eefb9c3b7402f3709525c7ec6752a7\n  - 15f0d9741dd33e2672b552540fa4ed564cec92ec\n  - ...\n  meandurationsec: 5.392025833333333\n...\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foperator-framework%2Fflake-analyzer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foperator-framework%2Fflake-analyzer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foperator-framework%2Fflake-analyzer/lists"}