{"id":18510705,"url":"https://github.com/operator-framework/audit","last_synced_at":"2025-08-13T01:18:54.205Z","repository":{"id":37423470,"uuid":"359530937","full_name":"operator-framework/audit","owner":"operator-framework","description":"audit operator bundles and catalogs, producing a report.","archived":false,"fork":false,"pushed_at":"2024-11-12T21:57:03.000Z","size":44525,"stargazers_count":10,"open_issues_count":15,"forks_count":15,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-05-30T00:41:42.312Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/operator-framework.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-19T16:44:11.000Z","updated_at":"2025-01-20T08:59:04.000Z","dependencies_parsed_at":"2023-11-13T13:46:41.755Z","dependency_job_id":"af669ed0-fac9-4a3c-b903-dc6eb4a0f96a","html_url":"https://github.com/operator-framework/audit","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/operator-framework/audit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/operator-framework%2Faudit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/operator-framework%2Faudit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/operator-framework%2Faudit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/operator-framework%2Faudit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/operator-framework","download_url":"https://codeload.github.com/operator-framework/audit/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/operator-framework%2Faudit/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270162224,"owners_count":24537783,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-12T02:00:09.011Z","response_time":80,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T15:24:37.687Z","updated_at":"2025-08-13T01:18:54.183Z","avatar_url":"https://github.com/operator-framework.png","language":"Go","readme":"~~[![Go Report Card](https://goreportcard.com/badge/github.com/camilamacedo86/audit)](https://goreportcard.com/report/github.com/camilamacedo86/audit)\n[![Coverage Status](https://coveralls.io/repos/github/github.com/operator-framework/audit/badge.svg?branch=main)](https://coveralls.io/github/camilamacedo86/audit?branch=main)\n\n---\n# Audit\n\n## Overview\n\nThe audit is an **experimental** analytic tool which uses the Operator Framework solutions. Its purpose is to obtain and report and aggregate data provided by checks and analyses done in the operator bundles, packages and channels from an index catalog image.\n\nNote that the latest version of the reports generated for all images can be checked in [testdata/report](testdata/reports). The file names are create by using the kind/type of the report, image name and date. (E.g. `testdata/report/bundles_quay.io_operatorhubio_catalog_latest_2021-04-22.xlsx`).\n\nFor further information about its motivation see the [EP Audit command operation][audit-ep]. \n\n## Pre-requirements\n\n- go 1.19 \n- docker or podman\n- access to the registry where the index catalog and operator bundle images are distributed\n- access to a Kubernetes cluster\n- [operator-sdk][operator-sdk] installed \u003e= `1.5.0\n\n**NOTE** that you can run the reports without SDK and the cluster running with by using the flag `--disable-scorecard`. That is only required for the scorecard results.  \n\n## Install binary:\n\nCheck the release binaries provided in the [release page](https://github.com/operator-framework/audit/releases).\n\n## Install from the source\n\nTo get the project and install the binary:\n\n```sh\n$ git clone git@github.com:operator-framework/audit.git\n$ cd audit\n$ make install\n```\n\nNow, you can run `$ audit-tool --help` to check it out.\n\n## Usage\n\n### Ensure that you have access to pull the images\n\nYou may first need to run `docker login` or `podman login` to have access to the images.\n\n#### With Podman\n\nPer default the audit commands use docker for dealing with container images. If you wish to use podman instead\n\n- either set the environment variable `export CONTAINER_ENGINE=podman` beforehand.\n- or add `--container-engine=podman` to each command\n\n```sh\nexport CONTAINER_ENGINE=podman\n```\n\n### Generating the reports\n\nNow, you can audit all operator bundles of an image catalog with: \n\n```sh\naudit-tool index bundles --index-image=registry.redhat.io/redhat/redhat-operator-index:v4.7\n```\n\n### Scanning for NetworkPolicy Resources\n\nTo identify any `NetworkPolicy` resources included in bundle manifests across catalogs, use the `np` sub-command:\n\n```sh\n$ audit-tool index np --indexes=registry.redhat.io/redhat/redhat-operator-index:v4.16,registry.redhat.io/redhat/redhat-operator-index:v4.17\n```\n\nYou can also filter to a specific package:\n\n```sh\n$ audit-tool index np --indexes=registry.redhat.io/redhat/redhat-operator-index:v4.17 --package=bakery-operator\n```\n\nSpecify `podman` if needed:\n\n```sh\n$ audit-tool index np --indexes=registry.redhat.io/redhat/redhat-operator-index:v4.17 --container-engine=podman\n```\n\nThen, this report will result in a JSON file with all data exctract from the index and the bundles. Note that audit\nwill download each bundle and extracted the info from them. Therefore, the reports available in the page [https://operator-framework.github.io/audit/](https://operator-framework.github.io/audit/)\nare done using the sub-command `dashboard`. All custom reports requires the bundles report in jSON format\nso that they can are able to gathering the data and process it accordingly.\n\n### Options\n\nUse the `--help` flag to check the options and the further information about its commands. Following an example:\n\n```sh\n$ audit-tool --help\nThe audit is an analytic tool which uses the Operator Framework solutions. Its purpose is to obtain and report and aggregate data provided by checks and analyses done in the operator bundles, packages and channels from an index catalog image.\n\nUsage:\n  audit-tool [command]\n\nAvailable Commands:\n  completion  Generate the autocompletion script for the specified shell\n  dashboard   generate specific custom reports based on the audit JSONs output\n  help        Help about any command\n  index       audit index catalog image\n\nFlags:\n  -h, --help   help for audit-tool\n\nUse \"audit-tool [command] --help\" for more information about a command.\n...\n```\n\n### To have a faster result, you can filter using the package name\n\nSee that you can use the `--filter` --flag to filter the results by the package name:\n\n```sh\naudit-tool index [bundles] --index-image=registry.redhat.io/redhat/redhat-operator-index:v4.5 --filter=\"mypackagename\"\n```\n\n### To run in dedicated environments\n\nUse the flag `--server-mode` to generate the reports in dedicated environments. By using this flag option the images\nwhich are downloaded will not be removed, allowing the reports to be generated faster after the first execution.\n\nAlso, ensure that you have enough space to store all images. Note that the default behavior is to remove them, when this option is not used.  \n\n## Reports\n\n### Base for all reports (audit index bundle)\n\nThe command `audit index bundle --index-image [OPTIONS]` will audit the image and bundles shipped on the index to extract all data.\n\n### HTML reports \n\nTo generate the reports such as you can find in [https://operator-framework.github.io/audit/](https://operator-framework.github.io/audit/) you\nwill need to have the bundles report (JSON one build with `audit index bundle --index-image` ) and you will use the `dashboard` commands, see:\n\n```\n$ audit-tool dashboard --help\ngenerate specific custom reports based on the audit JSONs output\n\nUsage:\naudit-tool dashboard [command]\n\nAvailable Commands:\ndeprecate-apis generates a custom report to check packages impact by k8s apis removal.\nmultiarch      generates a custom report based on defined criteria over Multiple Architectures\nqa             it is an custom dashboard which generates a custom report based on defined criteria over some specific defined criteria over the quality of the packages\nvalidator      generates a custom report based on the results filter by this validation informed\n\nFlags:\n-h, --help   help for dashboard\n```\n\nExample:\n\n```sh\naudit-tool dashboard deprecate-apis --file=testdata/report/bundles_quay.io_operatorhubio_catalog_latest_2021-04-22.json \n```\n\n#### deprecate-apis:  \n\n* By default, it only checks the bundles which are using APIs that were removed on OCP 4.9, and K8s 1.22\n* You can use to check the potential impact on the catalog for APIs that were removed in 1.25 and 1.26 (in this case, we can only \nverify the Operator bundles which are asking permissions for those APIs. However, RBAC configurations does \nnot require the versions of the APIs so that, we cannot know if the project is using the removed version or not)\n\n#### multiarch:\n\nThis one will check the Operator bundles against multiple architecture configurations.\nTo know more see the [Operator Framework/API validator](https://github.com/operator-framework/api/blob/v0.17.1/pkg/validation/internal/multiarch.go)\n\n**Note**: Check [here](https://operator-framework.github.io/audit/testdata/reports/redhat_redhat_operator_index/dashboards/multiarch_registry.redhat.io_redhat_redhat_operator_index_v4.11.html) example.\n\n#### qa:\n\nThis option will create a report to check the projects against some quality aspects. The results of the \nchecks done checked against the [validators][validator] in and [SDK scorcard][scorecard]  and are used to build this reports.\n\n**Note**: Check [here](https://operator-framework.github.io/audit/testdata/reports/redhat_redhat_operator_index/dashboards/qa_registry.redhat.io_redhat_redhat_operator_index_v4.11.html) example.\n\n#### validator\n\nThis option is useful if you are looking for to generate a report with all Operator bundles that fails\nunder some [validator][validator] or [SDK scorcard][scorecard] check.\n\n## How the reports in the page are generated\n\nSee that you will find a directory `testdata`. Therefore, you can: \n\n- run `make generate-samples` just for test purpose and to generate `testdata/samples` \n- run `make generate-testdata` to re-generate all reports in the testdata\n- run `make generate-all` which will run all reports and dashboards(html ones) as the index.html\n\n### Index page\n\nThe `index.html` page is generated via `make generate-index`. \nIt will aggregate in its results all dashboards found per image which are available in the testdata. \nTo check it, see https://operator-framework.github.io/audit/ . \n\n## FAQ\n\n### How Audit works?\n\nFollowing the steps performed by Audit. \n\n- Extract the database from the image informed\n- Perform SQL queries to obtain the data from the index db\n- Download and extract all bundles files by using the operator bundle path which is stored in the index db  \n- Get the required data for the report from the operator bundle manifest files \n- Use the [operator-framework/api][of-api] to execute the bundle validator checks\n- Use SDK tool to execute the Scorecard bundle checks\n- Output a JSON report providing the information obtained and processed. \n\nFor some detailed information about its implementation check [here](docs/steps.md).\n\nThen, the based JSON can be used to generated the other custom dashbaords.\n\n**Example: (Multi-arch reports)**\n\nThey are generated by the command:\n\n`audit-tool dashboard multiarch --file=\"bundle report - json file with all datat\"`\n\nThe command will do:\n- get the data from the JSON, which has all bundle info extracted from the index\n- get all packages and head of channels\n- run docker manifest inspect for each image used/defined in the CSV\n- grab the info and the logic criteria as we do in the validator \n- Then, with all results, aduit build the report in HTML\n\nSee that we have the makefile targets that generate all reports: https://github.com/operator-framework/audit/blob/v0.2.0/Makefile#L105-L107\n\n### What are the images used to generate the full reports?\n\n- OCP images: See [Understanding Operator catalogs](https://github.com/openshift/openshift-docs/blob/master/modules/olm-understanding-operator-catalog-images.adoc#understanding-operator-catalogs)\n- Community operator image (`quay.io/operatorhubio/catalog:latest`): Its source is from [upstream-community-operators](https://github.com/operator-framework/community-operators/tree/master/upstream-community-operators)\n\n### What is in the hack/special-needs \n\nIn this directory we have been storing some scripts that help us to generate specific \nspecial needs reports that would not fit under the sub-command or that could one day be\nimproved to become a sub-command. \n\n### What is in the hack directory?\n\nAll scripts to automate the reports generated in the page for example are in the hack\ndirectory. \n\n### What are the steps to generate the pages?\n\n- Check pre-requirements\n\n```\n$ operator-sdk version (see if you have SDK locally it will be required for the scorecard checks)\n$ kind create cluster (ensure that you have a cluster up and running it will also required for scarecard checks)\n```\n\n- Login in the registry and run `make generate-all`\n\n```shell\ndocker login https://registry.redhat.io\nmake generate-all\n```\n\n**NOTE** If something fails you can check what failed and just call directory\nthe scripts for what is missing to acomplished the goal. You can look at\nthe Makefile to know how to do manually the calls. \n\n### Release Process\n\nOnly creates and push a new tag then, the github actions will build and \nadd the artefacts in the release page. \n\n[of-api]: https://github.com/operator-framework/api\n[scorecard-config]: https://github.com/operator-framework/operator-sdk/blob/v1.5.0/testdata/go/v3/memcached-operator/bundle/tests/scorecard/config.yaml\n[operator-sdk]: https://github.com/operator-framework/operator-sdk\n[audit-ep]: https://github.com/operator-framework/enhancements/blob/master/enhancements/audit-command.md\n[validator]: https://github.com/operator-framework/api/blob/v0.17.1/pkg/validation/validation.go#L66-L85\n[scorecard]: https://sdk.operatorframework.io/docs/testing-operators/scorecard/\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foperator-framework%2Faudit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foperator-framework%2Faudit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foperator-framework%2Faudit/lists"}