{"id":21638085,"url":"https://github.com/purecloudlabs/gprovision","last_synced_at":"2025-04-11T16:42:29.157Z","repository":{"id":48506633,"uuid":"241436222","full_name":"purecloudlabs/gprovision","owner":"purecloudlabs","description":"Code used to provision (image)/factory restore appliances and to build the binaries used for those tasks.","archived":false,"fork":false,"pushed_at":"2022-12-15T22:12:43.000Z","size":3126,"stargazers_count":7,"open_issues_count":1,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-25T12:51:21.601Z","etag":null,"topics":["buildroot","embedded","golang","initramfs","iot","linux","provisioning","software-update","system"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/purecloudlabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-18T18:27:13.000Z","updated_at":"2024-04-10T19:21:08.000Z","dependencies_parsed_at":"2023-01-29T04:45:18.823Z","dependency_job_id":null,"html_url":"https://github.com/purecloudlabs/gprovision","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/purecloudlabs%2Fgprovision","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/purecloudlabs%2Fgprovision/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/purecloudlabs%2Fgprovision/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/purecloudlabs%2Fgprovision/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/purecloudlabs","download_url":"https://codeload.github.com/purecloudlabs/gprovision/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248442317,"owners_count":21104156,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["buildroot","embedded","golang","initramfs","iot","linux","provisioning","software-update","system"],"created_at":"2024-11-25T04:08:08.479Z","updated_at":"2025-04-11T16:42:29.132Z","avatar_url":"https://github.com/purecloudlabs.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# gprovision\n[![GoDoc](https://godoc.org/github.com/purecloudlabs/gprovision?status.svg)](https://godoc.org/github.com/purecloudlabs/gprovision)\n\nSource code related to factory restore and provisioning (aka manufacture or imaging). Also includes buildroot files.\n\nWritten in [go](https://golang.org).\n\n## target architecture\n\nx86 linux\n\nThis has only been tested on x86, but there shouldn't be much that is x86-specific. Buildroot is used to get a repeatable build of AMD64 utilities, as well as x86-32 libraries (one tool we use is only available in 32-bit versions).\n\n### known architecture-specific bits\n\nA few ioctl's are used by their raw number, rather than by symbol. These numbers are unlikely to be portable. All uses should be in packages under `hw/`.\n\nThe type of the device (aka platform, variant) is currently determined from dmi/smbios values.\n\nPCI is ubiquitous on x86 but not as common elsewhere. There is some pci-specific logic but probably not in any critical paths.\n\n\n## packages that may be of particular interest\n\n* provisioning, aka manufacture `gprovision/pkg/mfg`\n* factory restore `gprovision/pkg/recovery`\n* data erase `gprovision/pkg/erase`\n* linux /init binary `gprovision/cmd/init`, `gprovision/pkg/init`\n  * determines whether to erase, factory restore, or do normal boot\n  * or, with the appropriate build tag, provisions the unit\n* integ tests for the above\n* crystalfontz lcd code, including menus `gprovision/pkg/hw/cfa`\n* partitioning/formatting/bootability code for uefi and legacy `gprovision/pkg/recovery/disk`\n* flexible logging to multiple sinks (lcd, file, console) `gprovision/pkg/log`\n* appliance package identifies the hardware variant it's running on `gprovision/pkg/appliance`\n* config of ipmi account, password `gprovision/pkg/hw/ipmi`\n\n## proprietary/oss\n\nSome packages couldn't be open sourced, and a few provided functionality that is absolutely required; for those, replacements were written and live under `gprovision/pkg/oss`. These packages are related to remote logging, record keeping, and factory restore data persistence.\n\nTo facilitate open source release while still using proprietary packages within genesys, `gprovision/pkg/common` (and subpackages) was added. This uses interfaces to abstract away exactly what package is providing the functionality. In a few dirs, you will find an `oss.go` file; internally, we use a `proprietary.go` in place of `oss.go`, to similar effect. These files have the effect of initializing interface vars in the aforementioned common package.\n\nIf you wish to customize gprovision's behavior, start by looking at the interfaces in `common` - re-implementing one or more of these may provide all you need.\n\n## not included\n\n### security\n\nThe code we could open source does *not* include a mechanism for secure storage of passwords. Password storage is required for the integ tests, so some sort of implementation was required. To discourage use of the simplistic and weak mechanism included in this code, it deliberately uses annoying passwords and emits warnings.\n\n### image creation\n\nThis code does not include anything to build the images (`*.upd`).\n\n### OTA\n\nA mechanism for distribution of over-the-air updates is not included, nor is any sort of fleet status/control dashboard. This code does not (currently) integrate with mender.io but that's probably the most logical choice for those functionalities.\n\n## image\n\n### name\n\nThe image name must match a specific format: `WIDGET.LNX.SHINY.YYYY-MM-DD.NNNN.upd`\n* extension: `.upd`\n* prefix must match the string returned by common/strs.ImgPrefix()\n  * default is `WIDGET.LNX.SHINY.`\n    * product: widget\n    * os: lnx\n    * hardware platform: shiny\n* remaining fields are date and a build number\n  * see recovery/archive package for additional info including sort algo\n\n### format\n\n* xz-compressed tarball\n  * xz _must_ be invoked with the sha256 checksum option\n  * treated as invalid if:\n    * xz signature is missing\n    * xz checksum type is not sha256\n    * xz checksum validation fails\n\n### content\n\n* a tarball of the root filesystem that will be laid down\n  * assumptions\n    * systemd is the init system\n    * systemd-networkd manages networking\n    * `/boot/norm_boot` is the kernel built by mage in this repo, with embedded initramfs\n* on install, the kernel located under /boot in the image is compared to the previously installed kernel\n  * comparison is by _build number_, not kernel version\n    * example: `4.19.16 (user@host) #300 SMP \u003ctimestamp\u003e` =\u003e build number is 300\n    * use the kernel built here by mage\n      * build number is set from an env var\n    * kernel with higher number will overwrite the other\n  * do not include any boot-related files except the kernel; they will be ignored.\n    * grub\n    * grub menus\n    * separate initramfs\n    * etc\n* accounts\n  * admin must exist\n  * no accounts in the image (other than admin) should allow login\n  * factory restore will set password for admin account\n\n### file location\n\nFactory restore looks for update files on the recovery volume, under Image/. The first image is written during provisioning. Others are the responsibility of your code to download/place. Note that factory restore verifies the checksum to avoid corruption, but this does not provide any guarantees that the image came from the right place.\n\nYour download code should have additional safeguards, such as\n\n* signature verification\n* only allow downloads from a host under your control\n* only allow https downloads\n* pin the certificate for the download host\n* etc\n\n## building\n\nModules are not supported yet; at least one of our dependencies needs changes before it'll work with modules. This repo may also need changes for modules. Thus, you will need to have GOPATH set up.\n\nMany packages can be built with `go build`, but the initramfs's, kernels, and integ tests require mage.\n\n### generated code\n\nGenerated code is used in a number of packages and (with the exception of protocol buffer code) isn't included in the repo. If go-bindata is present, `go generate ./...` will suffice; otherwise, `mage bins:generate` will fetch the correct version and generate code.\n\nIf you desire to re-generate pb code (for pkg/oss/pblog), remove `disabled ` from the go:generate line in gen.go first. You'll need protoc installed, as well as the plugins for grpc and go.\n\n### PATH (for mage)\nWith bash on linux, run\n    . ./bash_env.sh\nto set up your env. This adds `bin/` to your PATH, runs `dep ensure` if it looks like you haven't, and lists mage targets.\n\nFor other shells, you'll need to update PATH to include the repo's `/bin` so that our `mage` is found before any version you may have installed.\n\n### TEMPDIR\n\nFor integ tests, your /tmp must have plenty of free space. If it has \u003c 8GB or so, set the TEMPDIR env var to point to a dir that does have that much free space.\n\nPassing tests delete their temp dirs, but failing tests generally do not. Re-running a failed integ test will delete temp dirs from previous runs of that test. This delete is done with a wildcard like `test-gprov-erase*`, so there is a slight but non-zero chance it would delete something it didn't create.\n\n### dep\ndep is needed. Once installed, `cd` to the repo root and run `dep ensure` to download all dependencies.\n\n### mage\nUsing mage is not necessary for simple packages like the crystalfontz code, but it is needed for things like the kernels with embedded initramfs and for integ tests. These integ tests depend on env vars mage sets, and are skipped if the vars are not set.\n\nDue to required dependencies, mage will not run until after you run `dep` - see its section above.\n\nMage targets (listed with `mage -l`) are case insensitive, and each corresponds to a public function in `build/`. For example, `mage bins:embedded` runs the function with signature `func (Bins) Embedded(ctx context.Context)`.\n\n### qemu\nqemu is also needed for integ tests. See `build/qemu/` for a Dockerfile that'll build qemu and copy the necessary files out. A pre-built version is available on github under releases, and is automatically downloaded by mage.\n\n### protoc\nOptional: protoc, the protocol buffer compiler, is only needed to regenerate code for `pkg/oss/pblog`.\n\n### kernel\nAll tools required to build the kernel are needed - gcc, flex, bison, make, binutils, etc. See kernel documentation for details.\n\n#### kernel version\nThe kernel source to download is determined from the name of the `linux-*.config` file.\n\n### buildroot\nFiles in `build/brx/` are inputs to build a cpio of non-go binaries. A pre-built version is available on github under releases, and is automatically downloaded by mage.\n\n### Known to work with...\nKnown to work with `go v1.12.5` or higher, the latest `dep`, and `protoc 3.x`.\n\n### go packages\nTo pull in dependencies:\n\n    cd $GOPATH/src/gprovision\n    dep ensure\n\n### verify\n\nOnce that is done, running `mage -l` from any dir should print a list of targets. (This is automatically done when you `source bash-env.sh`, so generally this can be skipped.)\n\nTargets are run like so:\n* `mage tests:unit`\n  * runs unit tests (but not the much more lengthy integ tests)\n\n## Known issues\n\nThe integ tests are occasionally flaky, as are the crystalfontz tests.\n\nShells out too much. u-root provides many packages we could use to avoid shelling out, but don't (yet).\n\nOld images are never removed from the recovery volume. This means you will eventually run out of space, depending on volume size, image size, and frequency of pushing out images. Though this could be seen as the responsibility of the code downloading images...\n\n## Sequence\n\n### Provision, AKA manufacture\n\n    +---------+     +---------+\n    | Legacy  |     | UEFI    |\n    | PXEboot |     | PXEboot |\n    +----+----+     +----+----+\n         |               |\n         +-------+-------+\n                 |\n                 v\n         +-------+-------+\n         | provision.pxe |\n         +-------+-------+\n                 |\n                 v\n       +---------+----------+\n       | load json from url |\n       |  passed in kernel  |\n       | parameter 'mfgurl' |\n       +---------+----------+\n                 |\n                 v\n     +-----------+-------------+\n     | verify hardware specs,  |\n     | download image \u0026 write  |\n     |   to recovery volume,   |\n     | write credentials, etc  |\n     +-----------+-------------+\n                 |\n                 v\n        +--------+---------+\n        |  configure boot  |\n        | (legacy or uefi) |\n        +--------+---------+\n                 |\n                 v\n            +----+-----+\n            |  reboot  |\n            +----------+\n\nSee Infrastructure below for off-device services that are required.\n\n### All boots after manufacture\n\n      +--------+   +------+\n      | Legacy |   | UEFI |\n      |  boot  |   | boot |\n      +----+---+   +---+--+\n           |           |\n           +-----+-----+\n                 |\n                 v\n        +--------+--------+\n        | \"normal\" kernel |\n        +--------+--------+\n                 |\n                 v\n         +-------+-------+\n         | erase needed? |\n         +--+----------+-+\n            |          |\n            v          v\n          +-+--+    +--+--+\n          | no |    | yes |\n          +-+--+    +--+--+\n            |          |\n            |          v\n            |      +---+---+     +--------+\n            |      | erase +----\u003e+ reboot |\n            |      +-------+     +--------+\n            |\n            v\n     +------+---------------+\n     | locate primary drive |\n     +----------+-----------+\n                |\n                v\n    +-----------+-----------+\n    | does flag file exist? |\n    +--+--------------+-----+\n       |              |\n       v              v\n    +--+-+         +--+--+\n    | no |         | yes |\n    +--+-+         +--+--+\n       |              |\n       |              v\n       |       +------+--------+\n       |       | all good,     |     +----------+\n       |       | start systemd +----\u003e+ eventual |\n       |       | from primary  |     |  reboot  |\n       |       +---------------+     +----------+\n       v\n     +-+---------------------+\n     | start factory restore |\n     +----------+------------+\n                |\n                v\n     +----------+-------------------------+\n     |                                    |       +--------+\n     | find latest good image on recovery +------\u003e+ reboot |\n     | format primary \u0026 write image       |       +--------+\n     | write network config               |\n     | write flag file                    |\n     | etc                                |\n     |                                    |\n     +------------------------------------+\n\n\n## Infrastructure for provisioning/manufacture\n\nThe integ tests (especially lifecycle and mfg) set up minimal infrastructure, which could be used as examples for some of the infrastructure.\n\nFor production, you will need the following:\n\n* DHCP and TFTP set up for PXEboot\n  * Setting up a network that will PXEboot both legacy and uefi can be painful. If you can get away with uefi-only, great!\n  * iPXE supports http transfers. They say this is far faster than tftp, so serve only the bare minimum (iPXE) from TFTP and everything else over http (assuming you use iPXE)\n* web server\n  * serve files needed by iPXE (if used) - menus, provisioning kernel\n  * serve file passed to kernel as mfgurl, and files referenced by that file\n    * when booting provisioning.pxe, pass in an arg like `mfgurl=http://1.2.3.4/blah/manufData.json`\n    * content: see below\n* log server\n  * collects logs produced by devices being imaged\n  * included implementation (cmd/util/pbLogServer, pkg/oss/pblog/server) bundles additional functionality\n    * password generation \u0026 storage\n    * qa doc printing\n    * other record keeping\n    * web server for viewing logs\n\n### mfgurl / manufData.json\n\n* format: json\n* lists additional files needed, common to all hardware variants\n  * image\n  * kernel\n  * (optional) other files to write to recovery volume\n* specific to a particular variant:\n  * hardware characteristics for validation (# cpus, pci devices present, etc)\n  * additial configuration steps\n\nAs an example, see [doc/manufDataSample.json](doc/manufDataSample.json).\n\nExactly what variant a device is, is determined by the [appliance](pkg/appliance) package. That package primarily uses dmi/smbios fields.\n\n## If this repo interests you...\n\nThird party repos that cover similar topics:\n\n* github.com/u-root/u-root\n* github.com/gokrazy/gokrazy\n* github.com/mendersoftware/mender\n* github.com/sbabic/swupdate\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpurecloudlabs%2Fgprovision","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpurecloudlabs%2Fgprovision","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpurecloudlabs%2Fgprovision/lists"}