{"id":20528083,"url":"https://github.com/redhatinsights/compliance-backend","last_synced_at":"2025-04-14T04:44:32.527Z","repository":{"id":37733069,"uuid":"148551035","full_name":"RedHatInsights/compliance-backend","owner":"RedHatInsights","description":":construction: Insights Compliance backend gets reports, and offers everything to know about them through an API ","archived":false,"fork":false,"pushed_at":"2025-04-12T09:15:49.000Z","size":12947,"stargazers_count":16,"open_issues_count":5,"forks_count":35,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-12T10:26:31.063Z","etag":null,"topics":["compliance","openscap","policy","security-audit"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RedHatInsights.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"security-scan.sh","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-09-12T22:44:02.000Z","updated_at":"2025-04-09T10:13:22.000Z","dependencies_parsed_at":"2024-01-26T12:43:31.566Z","dependency_job_id":"1485bcd7-ff25-469b-9c7d-2cc739b5578d","html_url":"https://github.com/RedHatInsights/compliance-backend","commit_stats":null,"previous_names":[],"tags_count":89,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedHatInsights%2Fcompliance-backend","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedHatInsights%2Fcompliance-backend/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedHatInsights%2Fcompliance-backend/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedHatInsights%2Fcompliance-backend/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RedHatInsights","download_url":"https://codeload.github.com/RedHatInsights/compliance-backend/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248824665,"owners_count":21167343,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compliance","openscap","policy","security-audit"],"created_at":"2024-11-15T23:23:00.213Z","updated_at":"2025-04-14T04:44:32.521Z","avatar_url":"https://github.com/RedHatInsights.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![codecov](https://codecov.io/gh/RedHatInsights/compliance-backend/branch/master/graph/badge.svg)](https://codecov.io/gh/RedHatInsights/compliance-backend)\n[![CI](https://github.com/RedHatInsights/compliance-backend/actions/workflows/ci.yml/badge.svg)](https://github.com/RedHatInsights/compliance-backend/actions/workflows/ci.yml)\n\n\n# Cloud Services for RHEL: Compliance Backend\n\ncompliance-backend is a project meant to parse OpenSCAP reports into a database,\nand perform all kind of actions that will make your systems more compliant with\na policy. For example, you should be able to generate reports of all kinds for\nyour auditors, get alerts, and create playbooks to fix your hosts.\n\n\n## Architecture\n\nThis project does two main things:\n\n1. Serve as the API backend for the web UI\n   [compliance-frontend](https://github.com/RedHatInsights/compliance-frontend)\n   and for other consumers,\n2. Connect to a Kafka message queue provided by the Insights Platform.\n\n### Components\n\nThe Insights Compliance backend comprises of these components/services:\n\n* Rails web server — serving REST API (port 3000)\n* Sidekiq — job runner connected through Redis (see [app/jobs](app/jobs))\n* Inventory Consumer (racecar) — processor of Kafka messages,\n  mainly to process and parse reports\n\n### Dependent Services\n\nBefore running the project, these services must be running and acessible:\n\n* Kafka — message broker (default port 29092)\n  - set by `SETTINGS__KAFKA__BROKERS` environment variable\n* Redis — Job queue and cache\n* PostgreSQL compatible database\n  - `DATABASE_SERVICE_NAME=postgres`\n  - conrolled by environment variables `POSTGRES_SERVICE_HOST`,\n    `POSTGRESQL_DATABASE`, `POSTGRESQL_USER`, `POSTGRESQL_PASSWORD`\n* [Insights Ingress](https://github.com/RedHatInsights/insights-ingress-go)\n  (also requires S3/minio)\n* [Insights PUPTOO](https://github.com/RedHatInsights/insights-puptoo)\n  — Platform Upload Processor\n* [Insights Host Inventory](https://github.com/RedHatInsights/insights-host-inventory) (MQ service and web service)\n* [Insights RBAC](https://github.com/RedHatInsights/insights-rbac) (Role-based access control)\n\n\n## Getting started\n\nLet's examine how to run the project:\n\n### Option 1: Ephemeral Environment with [Bonfire](https://github.com/RedHatInsights/bonfire/)\n\n#### Prerequisites\n\n* [`oc`](https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable/openshift-client-linux.tar.gz)\n  ([docs](https://docs.openshift.com/dedicated/4/cli_reference/openshift_cli/getting-started-cli.html))\n* [`bonfire`](https://github.com/RedHatInsights/bonfire/)\n\n#### Setup\n\n1. Log in into the ephemeral cluster\n2. Set up bonfire configuration (optional)\n\n    To use local deployement confguration update the `~/.config/bonfire/config.yaml` as follows:\n    ```\n    apps:\n    - name: compliance\n      components:\n        - name: compliance\n          host: local\n          repo: ~/path/to/local/compliance-backend\n          path: deploy/clowdapp.yaml\n          parameters:\n            REDIS_SSL: 'false'\n    ```\n\n#### Deployment\n\n```shell\nbonfire deploy compliance --optional-deps-method hybrid --frontends true --source=appsre --ref-env insights-stage --timeout 900\n```\n\nThis will set up the environment with all service dependencies, 3scale gateway, frontend\nand platform [mocks](https://github.com/RedHatInsights/mocks/) (authentication \u0026 authorization).\n\nA custom (local) clowdapp can be used for deployment if the step 2 of the setup has been not skipped.\nA custom image can be used by overwriting parameters of the clowder template. Note that the container\nimage needs to be pushed to an accessible location.\n```shell\nbonfire deploy compliance ... -p compliance/IMAGE=quay.io/me/myimage -p compliance/IMAGE_TAG=mytag\n```\n\n#### Access\n\nThe frontend route and credentials can be retrieved by calling the following command:\n```shell\nbonfire namespace describe \u003cephemeral-######\u003e\n```\n\n### Option 2: Docker/Podman Compose Development setup\n\nThe first step is to copy over the .env file from .env.example and modify the\nvalues as needed:\n\n```shell\ncp .env.example .env\n```\n\nEither podman-compose or docker-compose should work. Note that *1.0.6* and newer\nversions of podman-compose [handle multi-level dependencies poorly](https://github.com/containers/podman-compose/issues/683)\nwhich might cause problems with startup. Furthermore podman-compose does not\nsupport exec, so podman commands must be run manually against the running\ncontainer, as demonstrated:\n\n```shell\n# docker\ndocker-compose exec rails bash\n\n# podman\npodman exec compliance-backend_rails_1 bash\n```\n\nBring up the everything, including inventory, ingress, etc.:\n\n```shell\ndocker-compose up\n```\n\nAccess the rails console:\n\n```shell\ndocker-compose exec rails bundle exec rails console\n```\n\nAttach the container to the current terminal:\n```shell\ndocker attach compliance-backend_rails_1\n```\n\nDebug with pry-remote:\n\n```shell\ndocker-compose exec rails pry-remote -w\n```\n\nRun the tests:\n\n```shell\n# run all tests (same thing run on PRs to master)\ndocker-compose exec rails bundle exec rake test:validate\n\n# run a single test file\ndocker-compose exec -e TEST=$TEST rails bundle exec rake test TEST=test/consumers/inventory_events_consumer_test.rb\n\n# run a specific Rspec test\npodman-compose exec -e SPEC_OPTS=\"-e 'V2::ReportsController /reports GET index' --color --tty --format documentation\" rails bundle exec rake spec\n\n# run a single test case\ndocker-compose exec rails bundle exec rake test TEST=test/path/to_test.rb TESTOPTS=\"-n '/matching test description/'\"\n```\n\nAccess logs:\nnote: podman-compose does not support the logs command, so similar to exec,\nit must be run against the container itself, as shown\n\n```shell\ndocker-compose logs -f sidekiq\n\n# podman\npodman logs -f compliance-backend_sidekiq_1\n```\n\n### Building the image\n\nThis project uses `build_deploy.sh` to build and push this image to a remote registry.\nYou must set the required variables to provide authentication to the required registries.\nYou must set at least the Red Hat Registry credentials to be able to pull the base image.\n\nExample:\n```\nLOCAL_BUILD=true RH_REGISTRY_USER=guybrush RH_REGISTRY_TOKEN=M0nk3y ./build_deploy.sh\n```\n\nOptionally, if you want to push to the remote registry you must set the remote registry credentials as well (LOCAL_BUILD defaults to `false`)\n\nExample:\n```\nLOCAL_BUILD=true RH_REGISTRY_USER=guybrush RH_REGISTRY_TOKEN=monkey QUAY_USER=lechuck QUAY_TOKEN=Ela1ne ./build_deploy.sh\n```\n\n## Development notes\n\n### Seeding data\n\nTo seed accounts, policies, results and hosts use `dev:db:seed` rake task.\nIt is required to wait for the SSG content to be synchronized. Watch `compliance-backend_import-ssg_1` container to see the synchronization progress.\nIt is recommeneded to run the command after a first log in was initiated, as it would generate data for that account.\n\n```\nbundle exec rake dev:db:seed\n```\n\n### Creating hosts in the inventory\n\nTo create hosts in the inventory the `kafka_producer.py` script can be used from the `inventory` container:\n\n```\ndocker-compose run -e NUM_HOSTS=1000 -e INVENTORY_HOST_ACCOUNT=00001 inventory-web bash -c 'pipenv install --system --dev; python3 ./utils/kafka_producer.py;'\n```\n\n### Tagging\n\nIf there is a `tags` column defined in any model, it always should be a `jsonb` column and follow the structured representation of tags described in Insights, i.e. an array of hashes. If this convention is not kept, the controllers might break when a user tries to pass the `tags` attribute to a GET request.\n\n## API documentation\n\nThe API documentation can be found at `/api/compliance` and you may access the raw OpenAPI\ndefinition [here](https://github.com/RedHatInsights/compliance-backend/blob/master/swagger/v2/openapi.json).\nDocs can be updated by running:\n```shell\nrake rswag:specs:swaggerize\n```\n\n## Contributing\n\nIf you'd like to contribute, please fork the repository and use a feature\nbranch. Pull requests are warmly welcome.\n\nThis project ensures code style guidelines are followed on every pull request\nusing [Rubocop](https://github.com/rubocop-hq/rubocop).\nFormat for git commit message is described in `.commitlint.yml`.\n\n## Licensing\n\nThe code in this project is licensed under GPL v3 license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredhatinsights%2Fcompliance-backend","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fredhatinsights%2Fcompliance-backend","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredhatinsights%2Fcompliance-backend/lists"}