{"id":20102975,"url":"https://github.com/stackrox/kernel-packer","last_synced_at":"2025-05-06T08:31:04.981Z","repository":{"id":37268687,"uuid":"169502167","full_name":"stackrox/kernel-packer","owner":"stackrox","description":"📦 Crawl and repackage kernel headers for collector","archived":false,"fork":false,"pushed_at":"2024-04-14T08:02:45.000Z","size":17464,"stargazers_count":5,"open_issues_count":14,"forks_count":1,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-04-14T12:22:00.971Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stackrox.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2019-02-07T01:13:15.000Z","updated_at":"2024-04-15T14:07:54.983Z","dependencies_parsed_at":"2023-09-22T21:14:45.842Z","dependency_job_id":"f1ffee61-4a63-42f4-84e5-46a5a2c21a4e","html_url":"https://github.com/stackrox/kernel-packer","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackrox%2Fkernel-packer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackrox%2Fkernel-packer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackrox%2Fkernel-packer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackrox%2Fkernel-packer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stackrox","download_url":"https://codeload.github.com/stackrox/kernel-packer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224496023,"owners_count":17321036,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T17:33:51.516Z","updated_at":"2024-11-13T17:33:52.143Z","avatar_url":"https://github.com/stackrox.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/stackrox/kernel-packer/main.yml?style=for-the-badge)\n[![GCS Packages][gcs-packages-badge]][gcs-packages-link]\n[![GCS Bundles][gcs-bundles-badge]][gcs-bundles-link]\n\n# Kernel Packer\n\n📦 Crawl and repackage kernel headers for collector\n\n## Motivations and Goals\n\nKernel modules, and additionally eBPF modules, are the basis for how StackRox does runtime monitoring.\n\nThe production of kernel modules has historically been difficult, due to differences and inconsistencies in how various \nLinux distributions build their kernel modules.\n\nThis repository aims to define and abstract these processes away, so that downstream products can consume simplified and\nhomogeneous artifacts that can then be built upon. Additionally, this repository aims to fully automate the discovery \nand packaging of newly available kernel bundles. No human intervention should be necessary when upstream distros release\nnew kernel versions.\n\n## Concepts\n\n### Upstream\n\nLinux distributions such as CoreOS, Debian, RedHat. \u0026 Ubuntu.\n\n### Downstream\n\nThe [stackrox/collector](https://github.com/stackrox/collector) repository, specifically.\n\n### Kernel Packages\n\nA package file, typically a `.rpm`, or `.deb`, that is discovered from an upstream package repository by crawling. One \nor several different kernel packages are used in the production of one kernel bundle.\n\n### Kernel Bundles\n\nAn artifact file produced from kernel packages. A bundle is a gzipped tarball with a `.tgz` extention. Consumed by \ndownstream products.\n\n### Crawling\n\nUpstream kernel modules are distributed via a distribution's package repository. These package repositories are \norganized in a semi-standardized fashion, and can be programmatically scraped in order to discover the existence of new \npackages. Crawling is performed by the [`kernel-crawler`](kernel-crawler), and produces files inside of \n[`kernel-package-lists`](kernel-package-lists).\n\nCrawling can be done by running `make crawl`. This is [done automatically](.github/workflows/main.yml), and shouldn't have\nto be run manually.\n\n### Manifest\n\nAfter crawling, the set of discovered kernel packages are not in a very machine-consumable format. The generated \n[`manifest.yml`](kernel-package-lists/manifest.yml) YAML file is the source of truth for which sets of kernel packages \nto use for building a kernel bundle.\n\nGenerating the manifest can be done by running `make manifest`. This is done automatically, and shouldn't have to be run\nmanually.\n\n## Kernel Bundles\n\nBundles are gzipped tarballs and around ~12MB each. They contain a file tree derived from a given distro's kernel header\npackages. This file tree is usually a subset of the original packages, but is sufficient to compile modules against.\n\n### Bundle Meta Files\n\nBundles contain a number of additional \"meta\" files that can be leveraged by bundle consumers. These files all exist at \nthe root level of the tarball, and start with the `BUNDLE_` prefix.\n\n| Filename             | Example          | Purpose                         |\n| -------------------- | ---------------- | ------------------------------- |\n| `./BUNDLE_BUILD_DIR` | `./build`        | Directory to run `make` from.   |\n| `./BUNDLE_CHECKSUM`  | `02f...cd8`      | Build cache checksum.           |\n| `./BUNDLE_DISTRO`    | `coreos`         | The type of Linux distribution. |\n| `./BUNDLE_UNAME`     | `4.12.10-coreos` | The full kernel uname.          |\n| `./BUNDLE_VERSION`   | `4`              | The kernel \"version\" component. |\n| `./BUNDLE_MAJOR`     | `12`             | The kernel \"major\" component.   |\n| `./BUNDLE_MINOR`     | `10`             | The kernel \"minor\" component.   |\n\nAll meta files contain a single value and are meant to be read like so:\n\n```bash\nuname=\"$(cat ./BUNDLE_UNAME)\"\n```\n\n## Development\n\n### Kernel Bundles\nKernel packages and kernel bundles are cached in `${source_root}/.build-data/`.  To generate all bundles locally, execute \n`make bundles` to build all bundles or `./script/local-bundle \u003ckernel-version-regex\u003e` to only build a subset of kernel bundles.\nBuilding all bundles will take a long time and require downloading of several gigabytes of archived source packages. \nTo test modifications to kernel bundle builder for a subset of kernel packages, create a manifest yaml file\ncontaining only the subset and execute `MANIFEST_FILE={path to manifest.yml} make bundles`\n\n### PR Automation\n- The `crawl` job will not commit the new kernel versions.\n- The `repackage` job will not commit the new kernel header packages. Those will be available as task artefacts.\n\n[gcs-bundles-badge]:   https://img.shields.io/badge/gcs-kernel%20bundles-blue.svg?style=flat\u0026logo=google\n[gcs-bundles-link]:    https://console.cloud.google.com/storage/browser/stackrox-kernel-bundles?project=stackrox-collector\n[gcs-packages-badge]:  https://img.shields.io/badge/gcs-kernel%20packages-blue.svg?style=flat\u0026logo=google\n[gcs-packages-link]:   https://console.cloud.google.com/storage/browser/stackrox-kernel-packages?project=stackrox-collector\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstackrox%2Fkernel-packer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstackrox%2Fkernel-packer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstackrox%2Fkernel-packer/lists"}