{"id":24443623,"url":"https://github.com/endlessm/eos-image-builder","last_synced_at":"2025-07-31T19:03:52.897Z","repository":{"id":37031624,"uuid":"348432599","full_name":"endlessm/eos-image-builder","owner":"endlessm","description":"Endless image builder","archived":false,"fork":false,"pushed_at":"2025-02-05T13:47:35.000Z","size":32882,"stargazers_count":19,"open_issues_count":1,"forks_count":11,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-03-26T15:42:55.286Z","etag":null,"topics":["flatpak","linux","ostree"],"latest_commit_sha":null,"homepage":"https://support.endlessos.org/en/deployment/image-builder","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/endlessm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-16T17:19:40.000Z","updated_at":"2025-02-28T15:25:07.000Z","dependencies_parsed_at":"2024-01-18T17:29:32.396Z","dependency_job_id":"1355e659-e631-48d0-8011-96b242456be2","html_url":"https://github.com/endlessm/eos-image-builder","commit_stats":null,"previous_names":[],"tags_count":33,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/endlessm%2Feos-image-builder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/endlessm%2Feos-image-builder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/endlessm%2Feos-image-builder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/endlessm%2Feos-image-builder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/endlessm","download_url":"https://codeload.github.com/endlessm/eos-image-builder/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248633560,"owners_count":21136890,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["flatpak","linux","ostree"],"created_at":"2025-01-20T22:16:55.386Z","updated_at":"2025-04-12T21:26:58.282Z","avatar_url":"https://github.com/endlessm.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Endless Image Builder (EIB)\n===========================\n\nThis program assembles the disk images for the Endless OS (EOS) from\nOSTree filesystem trees. Its main functions are:\n\n 1. Pull the latest OSTree filesystem tree\n 2. Create images from ostree with content added\n\nDesign\n======\n\nEIB is designed to be simple but flexible. The core is written in bash\nscript and has just enough flexibility to our needs. The simplicity\nallows us to have a complete in-house understanding of the build system,\nenabling smooth organic growth as our requirements evolve. The build\nmaster(s) who maintain this system are not afraid of encoding our\nrequirements in simple bash script.\n\nThe top level entry point is written in python. This is done to provide\na rich configuration environment that allows Endless to quickly adjust\nto different product needs. See the [Configuration](#Configuration)\nsection below for a detailed description of the image builder\nconfiguration.\n\nWhen added complexity is minimal, we prefer calling into lower level\ntools directly rather than utilizing abstraction layers (e.g. we call\nmke2fs instead of using live-tools). This means we have a thorough\nunderstanding of the build system and helps to achieve our secondary\ngoals of speed and flexibility.\n\nThe build process is divided into several stages, detailed below. An\ninvocation can run some or all of these stages.\n\nostree stage\n------------\n\nThis stage pulls the latest tree from the remote ostree server to use as\nthe image base.\n\nimage stage\n-----------\n\nThis stage checks out the latest ostree into a new directory, configures\nthe bootloader, adds content, and creates output images. After\ncompleting the image, a 2nd set of images for use in a 2 disk setup is\ncreated.\n\nmanifest stage\n-------------\n\nThis stage assembles facts about the build and generates a merged JSON\nfile in the output directory for publishing.\n\npublish stage\n-------------\n\nThis stage does a final publishing of the output directory to the remote\nimage server.\n\nerror stage\n-----------\n\nThis stage is only run in the event of an error and simply publishes the\nbuild log to the image server.\n\nSetup\n=====\n\nKnown to work on Debian Buster (10) and newer. Required packages:\n\n * ostree\n * python3\n\nImage signing\n-------------\n\nEIB signs the completed images with GPG. A private keyring must be\ninstalled in /etc/eos-image-builder/gnupg and the key ID must be set in\nthe configuration.\n\nSSH authentication\n------------------\n\nEIB uses a private key at /etc/eos-image-builder/ssh-key.pem as the\nidentity file whenever ssh is used. SSH may be used for git fetching,\ncontent downloading, or image publishing.\n\nAWS authentication\n------------------\n\nEIB uses a shared AWS credentials file at\n`/etc/eos-image-builder/aws-credentials` to authenticate to AWS services\nsuch as S3. See the\n[AWS documentation](http://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html)\nfor details on this file.\n\nNetrc authentication\n--------------------\n\nAuthentication credentials for various remote servers are stored in a\nnetrc(5) file at `/etc/eos-image-builder/netrc`. This can be passed to\n`curl` or parsed with the `netrc` `python` module.\n\nConfiguration\n=============\n\nThe image builder configuration is built up from a series of INI files.\nThe configuration files are stored in the `config` directory of the\nbuilder. The order of configuration files read in is:\n\n  * Default settings - `config/defaults.ini`\n  * Product settings - `config/product/$product.ini`\n  * Branch settings - `config/branch/$branch.ini`\n  * Architecture settings - `config/arch/$arch.ini`\n  * Platform settings - `config/platform/$platform.ini`\n  * Personality settings - `config/personality/$personality.ini`\n\nThe image builder will then search for combinations of the build\nattributes. For instance, a product and personality combination would be\nread from `config/product-personality/$product-$personality.ini`. The\norder and format of these files follows the order of the attributes\nshown above. For instance, a branch and arch combination will always be\nspecified in the form `branch-arch` rather than `arch-branch`. Likewise,\nthis combination will come before a branch and platform combination.\n\nSystem specific configuration is then loaded from\n`/etc/eos-image-builder/config.ini`.\n\nNext, a local tree of configuration can be loaded from\n`$localdir/config` if a local settings directory is specified with the\n`--localdir` option. The configuration loaded here is the same as in the\n`config` directory in the source tree. For example,\n`$localdir/config/defaults.ini` and\n`$localdir/config/branch/$branch.ini` will be loaded.\n\nFor build specific settings, `config/local.ini` is then loaded from the\nsource tree.\n\nFinally, there are 3 configuration files whose settings will be used\nduring the build but not displayed in the saved configuration file:\n\n  * System private settings - `/etc/eos-image-builder/private.ini`\n  * Local private settings = `$localdir/config/private.ini`\n  * Checkout private settings - `config/private.ini`\n\nNone of these files are required to be present, but the `defaults.ini`\nfile contains many settings that are expected throughout the core of the\nbuild.\n\nNew configuration options should be added and documented in\n`defaults.ini`. See the existing file for options that are available to\ncustomize. Settings in the `build` section are usually set in the\n`ImageBuilder` class as they're static across all builds.\n\nFormat\n------\n\nThe format of the configuration files is INI as mentioned above.\nHowever, a form of interpolation is used to allow referring to other\noptions. For instance, an option `foo` can use the value from an option\n`bar` by using `${bar}` in its value. If `bar` was in a different\nsection, it can be referred to by prepending the other section in the\nform of `${other:bar}`.\n\nThe INI file parsing is done using the `configparser` `python` module.\nThe interpolation feature is provided by its `ExtendedInterpolation`\nclass. See the `python`\n[documentation](https://docs.python.org/3/library/configparser.html#configparser.ExtendedInterpolation)\nfor a more detailed discussion of this feature.\n\nThe system and local configuration files are not typically used. They\ncan allow for a permanent or temporary override for a particular host or\nbuild.\n\nSchema\n------\n\nThe image builder configuration scheme is intentionally very flexible to\nallow building images with various combinations of attributes. That\nflexibility means that it's easy to initiate builds with unsupported\nsettings. A configuration schema can be defined to help ensure specific\nkeys are set or to limit their values to a supported set.\n\nThe configuration schema is defined in `config/schema.ini` and\n`$localdir/config/schema.ini`. Schema files are INI formatted like the\nconfiguration itself. Sections and key names match the config files,\nwith key suffixes as follows:\n\n * `_required`: `true` means that the key must be set\n * `_values`: the value, if set, must be within the space-separated list\n   of values here\n * `_type`: `path` means the value, if set, must be the path to a file which\n   exists\n * `_type`: `paths` means the value, if set, must be a space-separated list of\n   path to files which exist\n\nMerged options\n--------------\n\nIn some cases, an option needs to represent a set of values rather than\na single setting. Adding or removing items from the list is not possible\nwith the features in the configuration parser.\n\nTo allow some method of building these lists, the builder will take\noptions of the form `\u003coption\u003e_add*` and `\u003coption\u003e_del*` and merge them\ntogether into one option named `\u003coption\u003e`. Each whitespace separated\nvalue in the `add` and `del` variants is counted to determine whether it\nwill remain in the merged option value. A value found in `add` will have\nits count incremented while a value found in `del` will have its count\ndecremented. If the final count is less than or equal to 0, it is\nremoved from the merged value.\n\nNormally options loaded later in the configuration will override\nidentically named options from earlier in the configuration. If an\nunmerged variant ends in `_add` or `_del`, a suffix based on the\nfilesystem path will automatically be appended to make it unique. For\ninstance, the option `packages_add` in `defaults.ini` will be converted\nto `packages_add_defaults`, and the option `apps_add` in\n`product/eos.ini` will be converted to `apps_add_product_eos`. These\noptions can be interpolated in other parts of the configuration using\nthe converted names.\n\nConfiguration files in the system directory will additionally include\n`system` in the merged option. For example, the options `apps_del` in\n`/etc/eos-image-builder/config.ini` will be converted to\n`apps_del_system_config`. Alternatively, any unmerged option that\ncontains a suffix after `add` or `del` will be left as is such as\n`apps_add_mandatory` in `defaults.ini`.\n\nIf the option `\u003coption\u003e` already exists, it is not changed. This allows\na configuration file to override all of the various `add` and `del`\noptions from other files to provide the list exactly in the form it\nwants.\n\nThe current merged options are defined in the `ImageConfigParser` class\nattribute `MERGED_OPTIONS` in the [eib](lib/eib.py) module. See the\n`defaults.ini` file for a description of these options.\n\nAccessing options\n-----------------\n\nThe build core accesses these settings via environment variables. The\nvariables take the form of `EIB_$SECTION_$OPTION`. The `build` section\nis special and these settings are exported in the form `EIB_$OPTION`\nwithout the section in the variable name.\n\nSeeing the full configuration\n-----------------------------\n\nIn order to see what the full configuration will look like after merging\nall configuration files and keys, run `./eos-image-builder\n--show-config` with other `--product` type options for selecting the\nappropriate image variant. This will print the merged configuration in\nINI format. The merged configuration is also saved during the build into\nthe output directory.\n\nSeeing the apps and runtimes\n----------------------------\n\nSometimes you may want to see which apps and runtimes will be included in an\nimage, without actually building the image. To do this, run\n`./eos-image-builder --show-apps` with other `--product` type options for\nselecting the appropriate image variant. This will print tables of apps,\ngrouped by their runtime, along with compressed and uncompressed size estimates\nfor each app and runtime.\n\n```\n# ./eos-image-builder --show-apps --product eos --personality pt_BR eos3.4\n```\n\nIf you want to group by regional-personality-specific vs generic vs runtime\ninstead, use `--group-by nature`:\n\n```\n# ./eos-image-builder --show-apps --group-by nature --product eos --personality pt_BR eos3.4\n```\n\nIf you are trying to reduce the compressed image size by, say, 300 MB, you can\npass `--trim BYTES`, and see crude suggestions for which\napps to remove. (Hint: for images with a size limit, the number to use is in\nthe image build log.)\n\n```\n# ./eos-image-builder --show-apps --trim 300000000 --product eos --personality pt_BR eos3.4\n```\n\nExecution\n=========\n\nTo run EIB, use the `eos-image-builder` script, optionally with a branch name:\n\n```\n# ./eos-image-builder [options] master\n```\n\nIf no branch name is specified, master is used. If you want to only run\ncertain stages, modify the `buildscript` file accordingly before\nstarting the program.\n\nOptions available:\n\n* `--product`: specify product to build (eos, eosnonfree, eosdev)\n* `--platform`: specify a sub-architecture to build (e.g. pinebookpro)\n* `--personality`: specify image personality to build (base, en)\n* `--dry-run`: perform a build, but do not publish the results\n\nCustomization\n=============\n\nThe core of EIB is just a wrapper. The real content of the output is\ndefined by customization scripts found under hooks/. These scripts have\naccess to environment variables and library functions allowing them to\nintegrate correctly with the core. If a local settings directory is\nprovided with the `--localdir` option, hooks in the `$localdir/hooks`\ndirectory are preferred to those in the checkout's `hooks` directory.\nThis allows providing both custom hooks as well as overriding existing\nhooks.\n\nThe scripts to run are organized under `hooks/GROUP` where `GROUP` is a\ngroup of hooks run by a particular stage. The hooks to run are managed\nin the configuration with merged `hooks` keys under each group. For\ninstance, the `image` group hooks to run are defined in the\n`image:hooks` configuration key. This allows easy customization for\ndifferent OS variants. These are merged options as described above, so\nthey can be added to or pruned by specific products.\n\nIf a script has an executable bit, it is executed directly. Otherwise it\nis executed through bash and will have access to the library functions.\n\nIf a script filename finishes with \".chroot\" then it is executed within\nthe chroot environment, as if it is running on the final system.\nOtherwise, the script is executed under the regular host environment. It\nis preferred to avoid chrooted scripts when it is easy to run the\noperation outside of the chroot environment.\n\nScripts are executed in lexical order and the convention is to prefix\nthem with a two-digit number to make the order explicit. Each script\nshould be succinct - we prefer to have a decent number of small-ish\nscripts, rather than having a small number of huge bash rambles.\n\nostree customization\n--------------------\n\nThe ostree stage currently has no customization hooks.\n\nimage customization\n-------------------\n\nAt the start of the image stage, the customization hooks under `content`\nare run. These hooks are intended to ensure that all content for the\ncurrent personality is available on the host disk, to be used later.\n`${EIB_CONTENTDIR}` should be used for storing this, and\n`${EIB_PERSONALITY}` states which personality is being built.\n\nOnce the ostree has been checked out (onto the host disk), customization\nhooks under `image` are run. `${OSTREE_DEPLOYMENT}` contains the path to\nthe checkout, and `${EIB_PERSONALITY}` states which personality is being\nbuilt.\n\nThe entire cache (`${EIB_CACHEDIR}`) and source (`${EIB_SRCDIR}`)\ndirectories are made available to both the `image` chroot hooks. This\nincludes other directories and files derived from the cache directory\nsuch as `${EIB_CONTENTDIR}` and `${EIB_OUTDIR}` or source directory such\nas `${EIB_DATADIR}` or `${EIB_HELPERSDIR}`.\n\nInstallation of Flatpaks is handled in the image stage. This uses\nconfiguration in `flatpak-remote-*` sections. See the comments in\n[default configuration file](config/defaults.ini) for details.\n\nmanifest customization\n----------------------\n\nThe manifest stage takes fragment JSON files found in the\n`${EIB_MANIFESTDIR}` directory and assembles them into a merged JSON\nfile in `${EIB_OUTDIR}`. Some basic facts about the build are provided,\nbut it's expected that other stages generate fragment JSON files as they\nproduce content. A `manifest` hook group is provided for customization\nthat is best done when the build is completed.\n\npublish customization\n---------------------\n\nKeeping with the design that the core is simple and the meat is kept\nunder customization, the publish stage does nothing more than call into\ncustomization hooks kept in `publish`. These hooks should take the\noutput of `${EIB_OUTDIR}` and push it to the final destination.\n\nerror customization\n-------------------\n\nLike the publish stage, the error stage simply calls the customization\nhooks kept in `error`. These hooks should take the `build.txt` file from\n`${EIB_OUTDIR}` and push it to the final destination. This stage should\nalso clean up for subsequent builds.\n\nTesting\n=======\n\nSome parts of the image builder can be tested with [pytest][pytest-url].\nInstall the testing dependencies with `pip3 install -r\nrequirements-test.txt`. After installing pytest, run `pytest` (or\n`pytest-3` if `pytest` is for python 2) from the root of the checkout.\n\nVarious options can be passed to `pytest` to control how the tests are\nrun. See the pytest [usage][pytest-usage] documentation for details.\nWhen debugging test failures, the `--basetemp` option allows specifying\nthe directory where each test's temporary directory will be stored. This\ncan be helpful to examine the files generated during tests. Another\nuseful option is `--log-cli-level=DEBUG`. Normally log messages are\nsuppressed unless there's a test failure. This option prints the\nmessages in the test output as they happen.\n\n[pytest-url]: https://docs.pytest.org/en/stable/\n[pytest-usage]: https://docs.pytest.org/en/stable/usage.html\n\nThe default image builder configuration and execution options are setup\nfor building production images on the Endless builders with access to\nall needed assets. However, when making changes on the image builder,\nit's important to test them locally. There are a few options available\nfor running the image builder locally in a mostly unprivileged\nenvironment.\n\nFirst, `eos-image-builder` provides options that are more appropriate\nfor testing. The `-n` or `--dry-run` option will skip publishing of the\ncompleted image. This not only keeps the test image from being\npublished, but it avoids likely authentication errors with other Endless\nservices.\n\nNext, `config/local.ini` can be used to change the image configuration\nin various ways that make a local build more likely to succeed. Since\n`local.ini` is parsed last, it can be used to override any other\nsettings. The file `config/local.ini.example` has examples of these\ntypes of settings. Copying it to `config/local.ini` and making any\nfurther local adjustments is recommended. See the comments in the\nexample file.\n\nFinally, `eos-image-builder` uses private keys in\n`/etc/eos-image-builder` to manage authentication and image signing. The\n`local.ini.example` file sets various options so that authentication to\nexternal services is generally not required. See the [Setup](#setup)\nsection above if you want to test authentication or GPG signing.\n\nNow you should be able to run `sudo ./eos-image-builder` with the\noptions mentioned above as well as any `--product` type options to\nselect the appropriate image variant for the base configuration.\n\nLicensing and redistribution\n============================\n\nImages built with this tool include Endless OS (more precisely: the Endless\nOS ostree filesystem tree), a copyrighted collective work of the Endless OS\nFoundation, and hence any redistribution of such images is subject to the\n[Endless OS Redistribution Policy](https://endlessos.com/redistribution-policy/).\n\nThis eos-image-builder tool in itself is Open Source software licensed\nunder the GNU GPL v2.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fendlessm%2Feos-image-builder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fendlessm%2Feos-image-builder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fendlessm%2Feos-image-builder/lists"}