{"id":13560077,"url":"https://github.com/dbnicholson/deb-ostree-builder","last_synced_at":"2025-04-14T16:56:30.757Z","repository":{"id":42862586,"uuid":"61309972","full_name":"dbnicholson/deb-ostree-builder","owner":"dbnicholson","description":"Stripped down Endless ostree builder for debian","archived":false,"fork":false,"pushed_at":"2022-03-26T14:48:11.000Z","size":41,"stargazers_count":65,"open_issues_count":5,"forks_count":25,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-03-28T05:31:45.110Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dbnicholson.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-06-16T16:40:21.000Z","updated_at":"2025-01-31T07:59:59.000Z","dependencies_parsed_at":"2022-08-27T13:34:08.741Z","dependency_job_id":null,"html_url":"https://github.com/dbnicholson/deb-ostree-builder","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dbnicholson%2Fdeb-ostree-builder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dbnicholson%2Fdeb-ostree-builder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dbnicholson%2Fdeb-ostree-builder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dbnicholson%2Fdeb-ostree-builder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dbnicholson","download_url":"https://codeload.github.com/dbnicholson/deb-ostree-builder/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248923663,"owners_count":21183949,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T13:00:37.251Z","updated_at":"2025-04-14T16:56:30.732Z","avatar_url":"https://github.com/dbnicholson.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Endless OSTree Builder (EOB)\n============================\n\nThis program assembles the Endless OS (EOS) from prebuilt packages and\ncontent. Its main functions are:\n 1. Assemble packages into ostree\n 2. Publish the ostree repo to a remote server\n\nDesign\n======\n\nEOB is designed to be simple. It is written in bash script and has just\nenough flexibility to meet our needs. The simplicity allows us to have a\ncomplete in-house understanding of the build system, enabling smooth\norganic growth as our requirements evolve. The build master(s) who\nmaintain this system are not afraid of encoding our requirements in\nsimple bash script.\n\nWhen added complexity is minimal, we prefer calling into lower level\ntools directly rather than utilizing abstraction layers (e.g. we call\ndebootstrap instead of using live-tools). This means we have a thorough\nunderstanding of the build system and helps to achieve our secondary\ngoals of speed and flexibility.\n\nThe build process is divided into several stages, detailed below. An\ninvocation can run some or all of these stages.\n\ncheck_update stage\n------------------\n\nThis stage does not perform ostree building, but is used to determine if\nan ostree build is required. If it exits successfully, no ostree build\nis needed.\n\nOS stage\n--------\n\nThis stage creates the OS in a clean directory tree, populating it with\napt packages.\n\nostree stage\n------------\n\nThis stage makes appropriate modifications to the output of the previous\nstage and commits it to a locally stored ostree repository. The ostree\nrepository is created if it does not already exist.\n\npublish stage\n-------------\n\nThis stage publishes the local ostree repository to the remote ostree\nserver.\n\nerror stage\n-----------\n\nThis stage is only run in the event of an error and simply cleans up for\na subsequent build.\n\nSetup\n=====\n\nKnown to work on Debian Wheezy, Ubuntu 13.04 and Ubuntu 13.10.\nRequired packages:\n * rsync\n * ostree\n * python3-apt\n * attr\n * x86: grub2\n * arm: mkimage, device-tree-compiler\n\nostree signing\n--------------\n\nEOB signs the ostree commits it makes with GPG. A private keyring must\nbe installed in /etc/deb-ostree-builder/gnupg and the key ID must be set\nin the configuration.\n\nConfiguration\n=============\n\nThe ostree builder configuration is built up from a series of INI files.\nThe configuration files are stored in the `config` directory of the\nbuilder. The order of configuration files read in is:\n\n  * Default settings - `config/defaults.ini`\n  * Product settings - `config/product/$product.ini`\n  * Branch settings - `config/branch/$branch.ini`\n  * Architecture settings - `config/arch/$arch.ini`\n  * Platform settings - `config/platform/$platform.ini`\n  * System config settings - `/etc/deb-ostree-builder/config.ini`\n  * Local build settings - `config/local.ini`\n\nNone of these files are required to be present, but the `defaults.ini`\nfile contains many settings that are expected throughout the core of the\nbuild.\n\nNew configuration options should be added and documented in\n`defaults.ini`. See the existing file for options that are available to\ncustomize. Settings in the default `build` section are usually set in\nthe `OSTreeBuilder` class as they're static across all builds.\n\nFormat\n------\n\nThe format of the configuration files is INI as mentioned above.\nHowever, a form of interpolation is used to allow referring to other\noptions. For instance, an option `foo` can use the value from an option\n`bar` by using `${bar}` in its value. If `bar` was in a different\nsection, it can be referred to by prepending the other section in the\nform of `${other:bar}`. The `build` section is the default section. Any\ninterpolation without an explicit section can fallback to a value in the\n`build` section. For example, if `bar` doesn't exist in the current\nsection, it will also be looked for in the `build` section.\n\nThe INI file parsing is done using the `configparser` `python` module.\nThe interpolation feature is provided by its `ExtendedInterpolation`\nclass. See the `python`\n[documentation](https://docs.python.org/3/library/configparser.html#configparser.ExtendedInterpolation)\nfor a more detailed discussion of this feature.\n\nThe system and local configuration files are not typically used. They\ncan allow for a permanent or temporary override for a particular host or\nbuild.\n\nMerged options\n--------------\n\nIn some cases, an option needs to represent a set of values rather than\na single setting. Adding or removing items from the list is not possible\nwith the features in the configuration parser.\n\nTo allow some method of building these lists, the builder will take\nmultiple options of the form `$prefix_add_*` and `$prefix_del_*` and\nmerge them together into one option named `$prefix`. Values in the\nvarious `$prefix_add_*` options are added to a set, and then values in\nthe various `$prefix_del_*` options are removed from the set. If the\noption `$prefix` already exists, it is not changed. This allows a\nconfiguration file to override all of the various `add` and `del`\noptions from other files to provide the list exactly in the form it\nwants.\n\nThe currently supported merged options are:\n\n* `cache:hooks`\n* `seed:hooks`\n* `os:hooks`\n* `os:packages`\n* `ostree:extra_refs`\n* `publish:hooks`\n* `error:hooks`\n\nSee the `defaults.ini` file for a description of these options.\n\nAccessing options\n-----------------\n\nThe build core accesses these settings via environment variables. The\nvariables take the form of `EOB_$SECTION_$OPTION`. The `build` section\nis special and these settings are exported in the form `EOB_$OPTION`\nwithout the section in the variable name.\n\nExecution\n=========\n\nTo run EOB, use the deb-ostree-builder script, optionally with a branch name:\n # ./deb-ostree-builder [options] master\n\nIf no branch name is specified, master is used. If you want to only run\ncertain stages, modify the `buildscript` file accordingly before\nstarting the program.\n\nOptions available:\n  --product : specify product to build (debian, debianplatform, debiansdk)\n  --arch : specify architecture to build (i386, armhf)\n  --platform : specify a sub-architecture to build\n  --force : perform a build even if the update check says it's not needed\n  --dry-run : perform a build, but do not publish the results\n\nCustomization\n=============\n\nThe core of EOB is just a wrapper. The real content of the output is\ndefined by customization scripts found under hooks/. These scripts have\naccess to environment variables and library functions allowing them to\nintegrate correctly with the core.\n\nThe scripts to run are organized under `hooks/GROUP` where `GROUP` is a\ngroup of hooks run by a particular stage. The hooks to run are managed\nin the configuration with merged `hooks` keys under each group. For\ninstance, the `os` group hooks to run are defined in the `os:hooks`\nconfiguration key. This allows easy customization for different OS\nvariants. These are merged options as described above, so they can be\nadded to or pruned by specific products.\n\nIf a script has an executable bit, it is executed directly. Otherwise it\nis executed through bash and will have access to the library functions.\n\nIf a script filename finishes with \".chroot\" then it is executed within\nthe chroot environment, as if it is running on the final system.\nOtherwise, the script is executed under the regular host environment. It\nis preferred to avoid chrooted scripts when it is easy to run the\noperation outside of the chroot environment.\n\nScripts are executed in lexical order and the convention is to prefix\nthem with a two-digit number to make the order explicit. Each script\nshould be succinct - we prefer to have a decent number of small-ish\nscripts, rather than having a small number of huge bash rambles.\n\ncheck_update customization\n--------------------------\n\nThe check_update stage calls the `cache` customization hooks. The\nintention is to determine facts about the current build and compare them\nto cached facts from the previous build. Facts are stored in the build\nspecific cache directory, determined from the function `eob_cachedir`.\nCache files should be named using the eob_cachefile function.\n\nThe check_update stage determines if an update is needed by seeing if\nthe modification times for any files in the cache directory have been\nupdated. Therefore, the hook should only update its cache file if\nthere's a difference from the previous build.\n\nos customization\n----------------\n\nAt the start of the os stage, the customization hooks under `seed` are\nrun. At this stage the `${EOB_ROOTDIR}` is totally empty. Place\nanything here that you want to be present at the time of initial\nbootstrap, which follows.\n\nAfter the initial bootstrap, the customization hooks under `os` are run.\nThese scripts are responsible for making any configuration changes to\nthe system (discouraged), installing packages, etc. The OS packages are\ninstalled by scripts at index 50.\n\nostree customization\n--------------------\n\nThe ostree stage currently has no customization hooks.\n\npublish customization\n---------------------\n\nKeeping with the design that the core is simple and the meat is kept\nunder customization, the publish stage does nothing more than call into\ncustomization hooks kept in `publish`. This stage should publish the\nlocal ostree repo to the remote server.\n\nerror customization\n---------------------\n\nLike the publish stage, the error stage simply calls the customization\nhooks kept in `error`. These hooks should clean up for subsequent\nbuilds.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdbnicholson%2Fdeb-ostree-builder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdbnicholson%2Fdeb-ostree-builder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdbnicholson%2Fdeb-ostree-builder/lists"}