{"id":21375335,"url":"https://github.com/lsst/lsst_build","last_synced_at":"2025-07-13T09:32:49.219Z","repository":{"id":38429272,"uuid":"23013298","full_name":"lsst/lsst_build","owner":"lsst","description":null,"archived":false,"fork":false,"pushed_at":"2024-11-19T12:11:37.000Z","size":599,"stargazers_count":1,"open_issues_count":0,"forks_count":10,"subscribers_count":63,"default_branch":"main","last_synced_at":"2024-11-19T12:43:54.979Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lsst.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-08-16T07:42:56.000Z","updated_at":"2024-07-12T03:05:57.000Z","dependencies_parsed_at":"2024-11-12T12:37:09.222Z","dependency_job_id":null,"html_url":"https://github.com/lsst/lsst_build","commit_stats":null,"previous_names":[],"tags_count":443,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lsst%2Flsst_build","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lsst%2Flsst_build/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lsst%2Flsst_build/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lsst%2Flsst_build/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lsst","download_url":"https://codeload.github.com/lsst/lsst_build/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225871094,"owners_count":17537173,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-22T09:09:50.034Z","updated_at":"2024-11-22T09:09:50.516Z","avatar_url":"https://github.com/lsst.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"lsst-build, a builder and continuous integration tool for LSST\n==============================================================\n\n[![Build Status](https://travis-ci.org/lsst/lsst_build.svg?branch=master)](https://travis-ci.org/lsst/lsst_build)\n\nProvides the following capabilities:\n\n* Given one or more top-level packages, intelligently clone their git\n  repositories and check out the requested branches into a build directory:\n\n  ```bash\n  lsst-build prepare\n     [--repository-pattern=format_pattern_for_repo_URLs]\n     [--exclusion-map=exclusions.txt]\n     [--version-git-repo=versiondbdir]\n     [--ref=branch1 [--ref=branch2 [...]]]\n     \u003cbuilddir\u003e \u003cproduct1\u003e [product2 [product3 [...]]]\n  ```\n\n  Run `lsst-build prepare -h` to see the full list of options.\n\n* Given that build directory, and an EUPS stack, intelligently build and\n  install those packages:\n\n  ```bash\n  lsst-build build \u003cbuilddir\u003e\n  ```\n\n  Run `lsst-build build -h` to see the full list of options.\n\nExample:\n\n```bash\ngit clone git@github.com:lsst/versiondb.git\nexport REPOSITORY_PATTERN=\"git://git.lsstcorp.org/LSST/DMS/%(product)s.git|git://git.lsstcorp.org/LSST/DMS/devenv/%(product)s.git|git://git.lsstcorp.org/LSST/DMS/testdata/%(product)s.git|git://git.lsstcorp.org/LSST/external/%(product)s.git\"\n\ncat \u003e exclusions.txt \u003c\u003c-EOF\n    # Exclusion map. Format:\n    # dependency regex      product regex\n    cuda_toolkit            .*\n    cuda_sdk                .*\n    scipy                   .*\n    healpy                  .*\n    condor                  .*\nEOF\n\nmkdir build\n\nlsst-build prepare \\\n    --exclusion-map=exclusions.txt\n    --version-git-repo=versiondb\n    build lsst_distrib\n\nlsst-build build build\n```\n\n`lsst-build prepare`\n------------------\n\n`lsst-build prepare` clones the given top level product(s) into `\u003cbuilddir\u003e`,\ndiscovers its dependencies by reading its `ups/$PRODUCT.table` file, and\nrecursively clones them until all dependencies are exhausted.  Following the\nclone, it writes out `\u003cbuilddir\u003e/manifest.txt` file, which lists all cloned\nproducts, their versions and dependency relationships.\n\nIt maps product names to git repository URLs via a repository pattern, a\n`|`-separated string of python string format patterns passed by\n`--repository-pattern` option, or existing in a `$REPOSITORY_PATTERN`\nenvironment variable. Each format pattern is evaluated in order with\n`%(product)` replaced by the product name to construct the URL, and a git-clone\nis attempted until a clone is successful.\n\n`lsst-build` skips any dependencies matching a rule in an exclusion map given\n`--via exclusion-map` option.  The exclusion map is a text file with two\nentries per line: the dependency regex and the product regex. Any product that\nmatches the dependency regex, while being considered for cloning as a\ndependency of a product that matches product regex, will be skipped.\n\nOnce cloned, a branch/tag/commit given by `--ref` is checked out. If multiple\n`--ref` options are given, each ref is tested for existence until one that\nexists is found and checked out. If no requested refs exist, `master` is\nchecked out. This is commonly used to prepare and test the build of a ticket\nbranch, falling back to master if that branch doesn't exist in some\nrepositories (i.e., `lsst-build prepare --ref ticket/1234 --ref master ...`).\n\nUpon completing the clone and ref checkouts of all packages in the product\ntree, `lsst-build prepare` writes out a \"build manifest\" in\n`\u003cbuilddir\u003e/manifest.txt`.  This is a topologically sorted whitespace-separated\nlist of (product, sha1, version, dependencies), with one line per cloned\nproduct.  Given the topological sorting, it is possible to build the packages\nin order of appearance in the file. The 'dependencies' entry is a\ncomma-separated list of dependency product names (first level only).\n\nIn addition to these tuples, the manifest may also contain variable assignments\nof the form `VARNAME=VARVALUE`.  Current implementation of `lsst-build` defines\nonly one, `BUILD`, which is a locally unique identifier identifying this\nparticular set of packages (i.e., a \"build number\").\n\nThe construction of version string in the build manifest depends on whether\n`--version-git-repo` option has been used. If it is not, the version string is\nconstructed as `\u003cpkgautoversion\u003e+\u003cdeps_sha1\u003e`, where pkgautoversion is the\noutput of EUPS' pkgautoversion tool when run in the product's git repository,\nand `\u003cdeps_sha1\u003e` is the abbreviated SHA1 of a sorted, space-separated list of\n(product, version) tuples of its dependencies. This guarantees that two\ndisconnected runs of `lsst-build` with exactly the same input repositories will\ncompute the same versions.\n\nIf `--version-git-repo=\u003cversiondb\u003e` is given, then the versions are of the form\n`\u003cpkgautoversion\u003e+\u003cN\u003e`, where N is a monotonically increasing integer.\nNevertheless, N is guaranteed to be the same for the same set of dependencies.\nTo achieve this, the files in `\u003cversiondb\u003e` directory record the mapping from\ndeps_sha1, computed as described in the previous paragraph, and the integer N.\nGiven the same source repositories, and the same versiondb repository, two\ndisconnected `lsst-build prepare` runs are guaranteed to generate the same\nversions.  Note that `\u003cversiondb\u003e` must be a specially formatted git repository\n(see the 'VersionDB Repository' section for more).\n\nThe construction of the build ID (`BUILD=xxx` entry in the manifest) also\ndepends on whether `--version-git-repo` option is in effect. If no, then this\nID is computed by looking for the highest EUPS tag matching `b*` and\nincrementing it by one (e.g., if EUPS tags `b1`, `b2`, and `b3` already exist,\n`BUILD=b4` would be written into the manifest). If `--version-git-repo` is\nused, the same algorithm is applied but on *git* tags of the versiondb\nrepository.  Also, if `--version-git-repo` is used the manifest will be stored\nin versiondb repository, the repository will be git-committed and tagged with\nthe build ID.\n\nIf `--version-git-repo` is used, it is advisable to git-push the repository\nupstream upon a successful `lsst-build prepare` (esp. before `lsst-build build` is\nrun). This ensures that no versions can exist in the installed stack that are\nnot present in the versiondb repo on the central server.\n\n`lsst-build build`\n----------------\n\n`lsst-build build` builds, installs and EUPS-declares, each product found in\nthe list in `\u003cbuilddir\u003e/manifest.txt` unless the product with the same name and\nversion is already EUPS-declared in the stack. The build is done using the\neupspkg mechanism; products not using eupspkg cannot be built with `lsst-build\nbuild`. Each successfully installed product is EUPS-tagged with the build\nidentifier (the `BUILD=xxxx` entry in `manifest.txt`).\n\nFor each product found in `\u003cbuilddir\u003e/manifest.txt`, `lsst-build` build first\nchecks if it already exists on the stack in EUPS_PATH. If not, it enters its\ndirectory, sets up its dependencies (roughly, using `setup -r .`\"), and runs\neupspkg {prep, config, build, install} sequence. If the build is successful, it\ndeclares the product to EUPS. Either way, the product is tagged with the `BUILD`\nvalue given in manifest.txt.\n\nAt the end of a successful run, all products listed in `\u003cbuilddir\u003e/manfest.txt`\nwill have been build, declared and installed into the active EUPS stack (the\nfirst entry on `$EUPS_PATH`), and tagged with the value of `BUILD` (the \"build\nnumber\").\n\nEnvironment Variables\n---------------------\n\n`lsst-build` may use the following environment variables:\n\n* `REPOSITORY_PATTERN`\n\n  A `|`-separated string of python string format patterns used to map product\n  names to URLs of their git repositories.  Each format pattern is evaluated\n  with `%(product)` replaced by the product name to construct the URL.  A\n  git-clone is attempted on each one (in order), until a clone is successful.\n  Setting this variable is equivalent to specifying the patter via\n  `--repository-pattern` option.\n\n* `EUPS_PATH`\n\n  The colon-separated path to EUPS-managed software stacks. `lsst-build` will\n  use these (via EUPS) to determine whether products need to be built or have\n  already been built, and will install new products into this location (using\n  eupspkg).\n\nVersionDB Repository\n--------------------\n\nVersionDB is a specially-formatted git-managed directory that serves as a `+N`\ndependency number and build ID number database for `lsst-build` with\n`--version-git-repo` option (see above).\n\nIt consists of three subdirectories:\n\n* `ver_db/`\n\n  A directory of text files, one per product, with space-separated\n  (product_version, N, deps_sha1) pairs.  Used by `lsst-build` to always assign\n  the same `+N` suffix to same set of dependencies (both products and their\n  versions).\n\n* `dep_db/`\n\n  A directory of text files, one per product, with space-separated list of\n  (product_version, N, dep_name, dep_version) tuples.  May be used by the user\n  to quickly look up a set of dependencies corresponding to some `+N` suffix.\n  Note that this information is also available in the product's expanded table\n  file.\n\n* `manifests/`\n\n  A directory where build manifests are stored. Each manifest is named\n  `$BUILD.txt`, where `$BUILD` is the unique build ID.  Every time `lsst-build`\n  is run with `--version-git-repo`, it will add the resulting manifest to this\n  directory, git-commit it, and tag it with the build ID, unless a manifest\n  with matching content already exists. In the latter case, that build ID will\n  be reused and no new commits will be added to VersionDB.\n\nNote that it is not necessary to understand this internal format to use this\nrepository; in fact, one should *not* depend on its internal format, as it may\nchange as `lsst-build` itself is improved.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flsst%2Flsst_build","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flsst%2Flsst_build","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flsst%2Flsst_build/lists"}