{"id":26793005,"url":"https://github.com/osbuild/manifest-db","last_synced_at":"2025-03-29T16:18:33.424Z","repository":{"id":37084852,"uuid":"500869414","full_name":"osbuild/manifest-db","owner":"osbuild","description":"Stores all the manifests and corresponding image-infos to test osbuild.","archived":false,"fork":false,"pushed_at":"2024-12-03T17:21:59.000Z","size":82223,"stargazers_count":0,"open_issues_count":3,"forks_count":9,"subscribers_count":11,"default_branch":"main","last_synced_at":"2024-12-03T18:27:25.808Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/osbuild.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-07T14:13:17.000Z","updated_at":"2024-12-03T17:22:12.000Z","dependencies_parsed_at":"2023-02-19T10:45:31.791Z","dependency_job_id":"c27623ee-a3f5-4d9c-8eb3-eb211b7dda91","html_url":"https://github.com/osbuild/manifest-db","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osbuild%2Fmanifest-db","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osbuild%2Fmanifest-db/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osbuild%2Fmanifest-db/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osbuild%2Fmanifest-db/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/osbuild","download_url":"https://codeload.github.com/osbuild/manifest-db/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246207503,"owners_count":20740723,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-29T16:18:32.896Z","updated_at":"2025-03-29T16:18:33.402Z","avatar_url":"https://github.com/osbuild.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003e [!CAUTION]\n\u003e # This repository is no longer maintained\n\nManifest-db\n=======\n\nThis repo serves as a data base to store the test cases for OSBuild. It contains\nall the tooling to automatically update the DB by scanning regularly\nOSBuild-composer.\n\n## The test code and its cases are stored on Manifest-DB\n\n### test code\n\nIn\n[`tools/osbuild-image-test`](https://github.com/osbuild/manifest-db/tree/main/tools/osbuild-image-test)\nthis serves as the main entry point to both:\n\n- generate an update to the database (with `--generator-mode` flag)\n- test a version of OSBuild against the database\n\nThe default mode (test mode) will compare that the resulting `image-info` JSON\nextracted from the build is the same as the one in the DB.\n\nNote that a\n[filter](https://github.com/osbuild/manifest-db/blob/main/tools/osbuild-image-test#L281)\nfunction will mask some of the fields during the comparison time.\n\n[Learn more why here](https://github.com/osbuild/manifest-db/commit/f89978d957fd3085063ae44692d326d11223d92f)\n\n### DB\n\nStored in the\n[`manifest-db`](https://github.com/osbuild/manifest-db/tree/main/manifest-db)\ndirectory.\n\n# Update workflow\n\nTo update the data base, an automated workflow is triggered every 15 days, on\n1st and 15th of each month. This process allows to sync with the latest\ncomposer.\n\n#### 1/ on GitHub: trigger the Gitlab CI\n\nUsing a GitLab `trigger token` the GitHub action starts the CI with a\ndifferent workflow than the one used to run the regression testing.\n\n#### 2/ on CI: Manifest generation\n\nThe first job downloads the latest `osbuild/images` sources from upstream,\n[generates all the test\nmanifests](https://github.com/osbuild/manifest-db/blob/main/.gitlab-ci.yml#L68)\nusing the gen-manifest tool and stores them as an\nartifact.\n\nThis step is easy to run, consume few\n[resources](https://github.com/osbuild/manifest-db/blob/main/.gitlab-ci.yml#L58)\n\n#### 3/ on CI: Image-info generation\n\nA job for each architecture and distribution listed in\n[.gitlab-ci.yml](https://github.com/osbuild/manifest-db/blob/main/.gitlab-ci.yml)\nwill run in parallel. Each job will:\n\n- [Run OSBuild on\nthem](https://github.com/osbuild/manifest-db/blob/main/test/cases/manifest_tests#L27)\n- [Generate the image\ninfo](https://github.com/osbuild/manifest-db/blob/main/tools/osbuild-image-test#L274)\n\nThis step is the longest, it can take up to 2h to finish some jobs.\n\n#### 4/ on CI: Image-info generation\n\nFinally the last job imports the generated manifests and image-info,\n[updates the DB with these information](https://github.com/osbuild/manifest-db/blob/main/.gitlab-ci.yml#L117),\ncreates a branch, a commit, pushes it on Github and finally [opens](https://github.com/osbuild/manifest-db/blob/main/schutzbot/include_image_info.sh#L33) a\n[PR](https://github.com/osbuild/manifest-db/pull/55) using the\n[Schutzbot](https://github.com/schutzbot) user.\n\nThe PR message contains a check list of changed manifests and image-info. The\nlist is generated with the command `tool/update_tool report --github-markdown`.\n\n#### 5/ on Github: reviewing the PR.\n\nThen a reviewer needs to look that the update PR is in good shape.\nTooling to help [`tools/update_tool`](https://github.com/osbuild/manifest-db/pull/42)\n\nNote: you can add yourself to the [list of\nreviewers](https://github.com/osbuild/manifest-db/blob/main/schutzbot/include_image_info.sh#L37)\nif you want to get notified\n\n#### Resuming in a diagram\n\n```mermaid\nflowchart LR\n    subgraph Gitlab\n        GH0[(Repo)]\n        GH01[Pipeline]\n        GH01 --\u003e GH1\n        GH1 -.- GH0\n        GH2 -.- GH0\n        GH3 -.- GH0\n        GH1{3. Generate manifests}\n        GH01 --\u003e GH2{4. Build all image-info}\n        GH01 --\u003e GH3{5. Create $BRANCH\\n create update commit\\npush $BRANCH}\n    end\n    subgraph Github/Manifest-db\n        RS0[ 2. Action\\n\\nTrigger pipeline\\non branch main ]\n        RSO1[ 7. Action\\n\\n Propagate main's\\nSHA on OSBuild]\n        GH4[(Repo)]\n    end\n    subgraph Github/OSBuild\n        OS0[(Repo)]\n        OS1[ 9. Action\\n\\n Propagate main's\\nSHA on Manifest-db]\n        OS0 ---|upon\\nupdate\\non main| OS1\n\n    end\n    RS0 --\u003e GH01    \n    i((Scheduler)) -- 1. start github action--\u003e RS0\n    GH3 == open PR ===\u003e GH4\n    j{{Maintainer}} -.6. review to\\nmerge/discard\\nPR .-\u003e GH4\n    j -..8. review to\\nmerge/discard\\nPR ..-\u003e OS0\n\n    RSO1 == open PR ==\u003e OS0\n    GH4 ---|upon\\nupdate\\non main| RSO1\n\n    OS1 == open PR ==\u003e GH4\n    j -..10. review to\\nmerge/discard\\nPR ..-\u003e GH4\n```\n\n# Keeping toolchain up to date\n\nTwo Github actions will propagate the up-to-date DB to OSBuild and the\nup-to-date OSbuild to manifest-db.\n\n- [GH action](https://github.com/osbuild/manifest-db/blob/main/.github/workflows/propagate_to_osbuild.yml) to update OSBuild's reference commit\n- [GH action](https://github.com/osbuild/osbuild/blob/main/.github/workflows/propagate_to_manifestdb.yml) to update Manifest-DB's reference commit\n\n\n# Why this repository\n\nBefore Manifest-DB the test-case generation was manual and required:\n\n- in depth knowledge of our toolchain is required\n- access to internal network for rhel distributions\n- access to specialized hardware (ppc)\n\nAs a result, it was not accessible especially for external developers\n(preventing them from contributing to image definitions)\n\n## The tests and their cases were stored on Composer\n\nAs their main purpose is to test the non regression of OSBuild (using\n[image_test](https://github.com/osbuild/osbuild-composer/blob/main/test/cases/image_tests.sh))\nthey should not belong in Composer\n\n# Terminology\n\n## Image definitions\n\nImage definitions are the way we describe an image and there is one for every image\nwe support. Definitions can be found in\n[images](https://github.com/osbuild/images) repository.\n\n### Image definitions must be tested\n\n- We need to run OSBuild against each one of them\n- Check whether the build is *correct*\n    - Absence of accidental change\n    - Validate changes on a rare occasion\n\nOSBuild can't run the image definition directly.\nIt needs an intermediate format, the\n[manifest](https://github.com/osbuild/manifest-db/blob/main/manifest-db/centos_8-aarch64-ami-boot.json#L8)\n\n## Manifests\n\nThe Manifest is a JSON representation of an image definition. It tells OSBuild\nwhat to do to make an Image.\n\n## Image info\n\nThe [Image\ninfo](https://github.com/osbuild/manifest-db/blob/main/manifest-db/centos_8-aarch64-ami-boot.json#L6868)\nis a JSON representation of an image.\n\nGenerated by the tool\n[`image-info`](https://github.com/osbuild/manifest-db/blob/main/tools/image-info)\nit contains a list of discoverable pieces of information such as:\n- Package list\n- Network configuration\n- ...\n\n## Test cases\n\nCombination of manifest and image-info. They are the reference point of what is\n`correct` for a given input. They must be generated and stored for later use.\n\n![](https://mermaid.ink/svg/pako:eNo9j70OwjAMhF_F8kxfoAMTDJXoAiPpYBqnWGoclDpCCPHuhN_NJ393p7vjmDxji2FO1_FM2WC3d9odu0gTg-cgKiZJB2iaiRUiqQRerGnW0B_7rxqc9hWAtJyKzB5e3y5-Q0RDGnCFkXMk8bXs7hTAoZ05ssO2nrWIymwOnT4qWi6ejLdeLGVsA80Lr5CKpcNNR2wtF_5BG6EpU_xT_Db1n1XvcY8nBzNOUA)\n\n## The data base\n\nHosted under the `manifest-db` directory, each file is an entry to a test case.\n\n\n### Contributing\n\nPlease refer to the [developer guide](https://www.osbuild.org/guides/developer-guide/developer-guide.html) to learn about our workflow, code style and more.\n\n### License:\n\n - **Apache-2.0**\n - See LICENSE file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosbuild%2Fmanifest-db","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fosbuild%2Fmanifest-db","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosbuild%2Fmanifest-db/lists"}