{"id":17928681,"url":"https://github.com/hartwork/binary-gentoo","last_synced_at":"2025-09-18T01:32:20.202Z","repository":{"id":37976112,"uuid":"353422356","full_name":"hartwork/binary-gentoo","owner":"hartwork","description":":cow: Collection of simple CLI tools to help build Gentoo packages on a non-Gentoo Linux host","archived":false,"fork":false,"pushed_at":"2024-05-13T18:32:10.000Z","size":485,"stargazers_count":17,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-05-19T00:03:18.851Z","etag":null,"topics":["automation","cli","compilation","docker","ebuild","gentoo","python","python3"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/binary-gentoo/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hartwork.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-31T16:31:33.000Z","updated_at":"2024-05-20T12:50:01.848Z","dependencies_parsed_at":"2023-12-22T17:24:55.142Z","dependency_job_id":"9b65ff6f-6f59-4fd1-9755-1412f94adbed","html_url":"https://github.com/hartwork/binary-gentoo","commit_stats":{"total_commits":267,"total_committers":4,"mean_commits":66.75,"dds":0.4269662921348315,"last_synced_commit":"fcabe0ddf2abd052264ead7fc2be62c73b82944d"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hartwork%2Fbinary-gentoo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hartwork%2Fbinary-gentoo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hartwork%2Fbinary-gentoo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hartwork%2Fbinary-gentoo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hartwork","download_url":"https://codeload.github.com/hartwork/binary-gentoo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233437386,"owners_count":18676144,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","cli","compilation","docker","ebuild","gentoo","python","python3"],"created_at":"2024-10-28T21:04:42.048Z","updated_at":"2025-09-18T01:32:20.156Z","avatar_url":"https://github.com/hartwork.png","language":"Python","readme":"[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit)](https://github.com/pre-commit/pre-commit)\n[![Build and smoke test](https://github.com/hartwork/binary-gentoo/actions/workflows/smoke_test.yml/badge.svg)](https://github.com/hartwork/binary-gentoo/actions/workflows/smoke_test.yml)\n\n# binary-gentoo\n\n## About\n\n**binary-gentoo**\nis a collection of\nsimple\nCLI tools\nto help **build Gentoo packages on a non-Gentoo Linux host**, primarily.\nA typical scenario is operation of an\n**_active_ [Gentoo binary package host](https://wiki.gentoo.org/wiki/Binary_package_guide#Setting_up_a_binary_package_host)**\n — an active \"binhost\".\n\n*Secondarily*, **binary-gentoo** can also build Gentoo packages on a Gentoo host\nwith Docker isolation and a full `emerge` perspective\n(while not affecting your host system).\n\nThere currently four CLI tools\nthat follow the [Unix philosophy](https://en.wikipedia.org/wiki/Unix_philosophy)\nand are meant to be combined using a glue language like Bash:\n\n- `gentoo-build` – Builds a Gentoo package with Docker isolation\n- `gentoo-clean` — Clean Gentoo pkgdir/distdir files using `eclean` of `app-portage/gentoolkit` with Docker isolation\n- `gentoo-local-queue` – Manages simple file-based push/pop build task queues\n- `gentoo-packages` — Do operations on pkgdir (other than `emaint --fix binhost`)\n- `gentoo-tree-diff` – Lists packages/versions/revisions that one portdir has over another\n- `gentoo-tree-sync` – Brings a given portdir directory (and its backup) up to date\n\n**binary-gentoo**\nis software libre licensed under the `GNU Affero GPL version 3 or later` license.\n\n\n## Installation\n\nInstall Docker, Python \u003e=3.9, pip and rsync and then:\n\n```console\n# pip install binary-gentoo\n```\n\n\n## Active Binhost Operation\n\nLet's start with a picture:\n\n[![How do binary-gentoo's tools play together?](binary-gentoo.svg)](binary-gentoo.svg)\n(Graphviz source: [`binary-gentoo.dot`](binary-gentoo.dot))\n\nTo set-up an active binhost using **binary-gentoo** you will likely need\na virtual machine with…\n\n- Enough resources, i.e. probably\n  - []()\u003e=2 virtual cores\n  - []()\u003e=8 GB RAM\n  - []()\u003e=100 GB disk space\n- Some distribution of Linux\n- Auto-updates enabled to stay reasonably secure over time\n- Runtime dependencies installed:\n  - Docker\n  - Python \u003e=3.9\n  - rsync\n- An SSH or an HTTP daemon\n  (to serve the produced binaries to your consuming Gentoo boxes,\n  unless that is done from another machine)\n- An up-to-date copy of the `/etc/portage` folder to use\n- An up-to-date text file with the names of all packages to build,\n  e.g. generated by `EIX_LIMIT=0 eix -I --only-names`\n- A set of compile flags (`CFLAGS`, `CXXFLAGS`, `LDFLAGS`, `CPU_FLAGS_X86`)\n  that works for *both* the producing machine and the consuming machine(s)\n  and ideally [resolve-march-native](https://github.com/hartwork/resolve-march-native)\n  installed — more on finding the ideal flags below\n- Two portdir copies — one synced with `gentoo-tree-sync` (or some form of `git pull`),\n  the other brought back in sync using `rsync` after a call to `gentoo-tree-diff`\n- A glue script that…\n  - respects your inclusion list\n    (e.g. `grep -q -f installed.txt -F \u003c\u003c\u003c\"${atom}\" || continue`\n    if a few false positives are okay)\n  - respects your exclusion list\n    (e.g. something like `*sys-kernel/*|*-bin-*|*-9999*|*acct-*/*`)\n  - pushes `gentoo-tree-diff` news onto the queue\n  - pops off the queue,\n    extracts the atom using e.g. `jq -r .atom`,\n    and then let's `gentoo-build` do a build\n  - loops forever\n  - sleeps every now and then\n  - runs `gentoo-clean` every now and then\n\n\n## Determining Ideal Build Flags\n\nLet me give a concrete example.\n\n\n### Consumer Machine\n\nOn the consumer machine, we have this hardware situation:\n\n```console\n# lscpu | fgrep 'Model name' | sed 's,^.\\{33\\},,'\nIntel(R) Core(TM) i5-2520M CPU @ 2.50GHz\n\n# resolve-march-native\n-march=sandybridge -maes --param l1-cache-line-size=64 --param l1-cache-size=32 --param l2-cache-size=3072 -O2 -pipe\n\n# cpuid2cpuflags\nCPU_FLAGS_X86: aes avx mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3\n```\n\n\n### Producer Machine\n\nOn the [producer machine](https://www.netcup.de/bestellen/produkt.php?produkt=2554),\nwe have this hardware situation:\n\n```console\n# lscpu | fgrep 'Model name' | sed 's,^.\\{33\\},,'\nQEMU Virtual CPU version 2.5+\n\n# resolve-march-native\n-march=k8-sse3 -maes -mcx16 -mno-3dnow -mno-3dnowa -mpclmul -mpopcnt -mrdrnd -msahf -msse4 -msse4.1 -msse4.2 -mssse3 -mtune=k8 --param=l1-cache-line-size=64 --param=l1-cache-size=64 --param=l2-cache-size=512 -O2 -pipe\n\n# cpuid2cpuflags\nCPU_FLAGS_X86: aes mmx mmxext pclmul popcnt rdrand sse sse2 sse3 sse4_1 sse4_2 ssse3\n```\n\n\n### Resulting Producer Build Flags\n\n\nSimplified, we are looking for flags\nthat are **as close to the consumer machines as possible**\nwhile still resulting in binaries,\nthat **both the consumer machines and the producer machine can execute**\n(so that the producer machine is be able to install and run the binaries of *dependencies* of the package of interest).\n\nFor the example above, I ended up with these values for the producer machine:\n\n- `CFLAGS`: `-march=x86-64 -mtune=sandybridge -maes --param l1-cache-line-size=64 --param l1-cache-size=32 --param l2-cache-size=3072 -mpclmul -mpopcnt -msse4.1 -msse4.2 -mssse3  -O2 -pipe` (note `-march` and `-mtune` in particular)\n- `CXXFLAGS`: same as `CFLAGS`\n- `LDFLAGS`: whatever `portageq envvar LDFLAGS` on the consumer machine returns\n- `CPU_FLAGS_X86`: `aes mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3` (i.e. without `avx` and without `rdrand`)\n\n\n## Design Decisions\n\n- All code in **binary-gentoo** must work on a non-Gentoo Linux machine,\n  provided that it has Docker installed and working internet access.\n\n- Big packages like Chromium need a pile of RAM and CPU time.\n  Therefore, the build defaults to `MAKETOPTS=-j1`\n  the package of interest is emerged separate from it dependencies.\n  That allows to build e.g. a package of Chromium in a VM with only 8 GB of RAM.\n\n- With regard to dependency constraints,\n  some packages can be *built* without conflicts but not be *installed* without conflicts.\n  Hence the default is to only install dependencies, but not the package of interest.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhartwork%2Fbinary-gentoo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhartwork%2Fbinary-gentoo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhartwork%2Fbinary-gentoo/lists"}