{"id":19624001,"url":"https://github.com/bioconductor/bbs","last_synced_at":"2025-04-28T05:31:01.604Z","repository":{"id":33573956,"uuid":"37220301","full_name":"Bioconductor/BBS","owner":"Bioconductor","description":"The Bioconductor Build System","archived":false,"fork":false,"pushed_at":"2024-10-26T16:17:57.000Z","size":3834,"stargazers_count":9,"open_issues_count":49,"forks_count":11,"subscribers_count":16,"default_branch":"devel","last_synced_at":"2024-10-27T08:56:21.574Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Bioconductor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-06-10T20:17:29.000Z","updated_at":"2024-10-26T16:18:01.000Z","dependencies_parsed_at":"2023-09-22T03:14:49.116Z","dependency_job_id":"4f602658-8324-41de-a824-6f58310b4d2e","html_url":"https://github.com/Bioconductor/BBS","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bioconductor%2FBBS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bioconductor%2FBBS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bioconductor%2FBBS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bioconductor%2FBBS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Bioconductor","download_url":"https://codeload.github.com/Bioconductor/BBS/tar.gz/refs/heads/devel","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224096675,"owners_count":17255135,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T11:36:29.375Z","updated_at":"2024-11-11T11:36:30.621Z","avatar_url":"https://github.com/Bioconductor.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"Table of Contents\n=================\n\n  * [Bioconductor Build System Overview](#bioconductor-build-system-overview)\n    * [What is BBS?](#what-is-bbs)\n    * [What is BBS \u003cstrong\u003enot\u003c/strong\u003e?](#what-is-bbs-not)\n    * [Where is the code?](#where-is-the-code)\n    * [Human resources](#human-resources)\n    * [General overview of BBS](#general-overview-of-bbs)\n    * [What builds where](#what-builds-where)\n      * [About the build machines\\.](#about-the-build-machines)\n      * [How the build machines are organized\\.](#how-the-build-machines-are-organized)\n      * [DNS resolution and https specifics](#dns-resolution-and-https-specifics)\n        * [Address and canonical DNS records](#address-and-cononical-dns-records)\n        * [Traffic routing within the DFCI DMZ](#traffic-routing-within-the-dfci-dmz)\n        * [Mac builders and the DFCI DMZ](#mac-builders-and-the-dfci-dmz)\n      * [What machines are used in which builds?](#what-machines-are-used-in-which-builds)\n        * [A note about time zones\\.](#a-note-about-time-zones)\n    * [How the build system works](#how-the-build-system-works)\n      * [Builds](#builds)\n        * [prerun](#prerun)\n        * [run](#run)\n        * [postrun](#postrun)\n      * [Propagation Pipeline](#propagation-pipeline)\n        * [update](#update)\n        * [prepare](#prepare)\n        * [push](#push)\n\n\nBioconductor Build System Overview\n==================================\n\nThis is the main README for the Bioconductor Build System (BBS).\n\nFurther documentation on specific tasks is in the [Doc](Doc/) directory.\n\n## What is BBS?\n\n* A **nightly** build system, not incremental or continuous integration. Maybe\n  it can be replaced by those things in the future.\n* Home-grown. The system was written originally by Herv\u0026eacute; Pag\u0026egrave;s\n  and is now maintained Herv\u0026eacute;, Lori, and Jen.\n* Written in a mix of shell scripting (bash shell, Windows batch files),\n  Python, and R.\n\n## What is BBS **not**?\n\nBBS is different from the Single Package Builder, which is triggered when a\ntarball is submitted to the new package tracker. Though there is some common\ncode.\n\n## Where is the code?\n\nThe canonical location of the code is in GitHub:\n\n[https://github.com/Bioconductor/BBS](https://github.com/Bioconductor/BBS)\n\n## Human resources\n\nIf you have a question not covered here:\n\n* Ask Herv\u0026eacute; Pag\u0026egrave;s, Lori Shepherd, or Jen Wokaty.\n\n## General overview of BBS\n\n| Branch  | Build             | Builders                                     | Schedule              |\n|---------|-------------------|----------------------------------------------|-----------------------|\n| Release | Software (\"bioc\") | Linux (x86_64, aarch64[^1]), Mac x86_64, Win | Mon-Sat               |\n| Release | Software (\"bioc\") | Mac ARM64                                    | Start Sun, Finish Fri |\n| Release | Data Annotation   | Linux x86_64                                 | Wed                   |\n| Release | Data Experiment   | Linux x86_64                                 | Tue, Thu              |\n| Release | Workflows         | Linux x86_64, Mac x86_64, Win                | Tue, Fri              |\n| Release | Book              | Linux x86_64                                 | Mon, Wed, Fri         |\n| Release | Long Tests        | Linux x86_64                                 | Sat                   |\n| Devel   | Software (\"bioc\") | Linux (x86_64, aarch64), Mac x86_64, Win     | Mon-Sat               |\n| Devel   | Software (\"bioc\") | Mac ARM64                                    | Mon, Wed, Fri         |\n| Devel   | Data Annotation   | Linux x86_64                                 | Wed                   |\n| Devel   | Data Experiment   | Linux x86_64                                 | Tue, Thu              |\n| Devel   | Workflows         | Linux x86_64, Mac x86_64, Win                | Tue, Fri              |\n| Devel   | Book              | Linux x86_64                                 | Mon, Wed, Fri         |\n| Devel   | Long Tests        | Linux x86_64                                 | Sat                   |\n\n[^1]: As of 2023, there is a third-party guest builder running Linux aarch64 named kunpeng1.\n\n## What builds where\n\nAs of April 2023, the Linux x86_64 builders and the Mac x86_64 builder named\nlconway are in the DFCI DMZ, the Windows builders are in Azure, and the other\nMac builders are in MacStadium.\n\n### About the build machines.\n\nBioconductor maintains eight build machines, four each for release and devel.\n\n| Machine              | Arch         | OS                             |\n|----------------------|--------------|--------------------------------|\n| Nebbiolo1, Nebbiolo2 | x86_64       | Ubuntu 22.04 LTS               |\n| Palomino3, Palomino4 | x64          | Windows Server 2022 Datacenter |\n| Lconway, Merida1     | x86_64       | MacOS 12.x Monterey            |\n| Kjohnson2            | arm64        | MacOS 12.x Monterey            |\n| Kjohnson1            | arm64        | MacOS 13.x Ventura             |\n\n### How the build machines are organized.\n\nEach build has a *primary builder* which is the Linux build machine.\n\nThe *primary builder* is where all build machines send their build products (via\nrsync and ssh). Build products are not just package archives (.tar.gz, .tgz,\nand .zip files for source packages, Mac packages, and Windows packages\nrespectively) but also the output of each build phase and other information\nabout the build, enough to construct the build report.\n\n### DNS resolution and https specifics\n\nIn Stage 2, the Windows and Mac builders get packages to build from the *primary\nbuilder*. Historically this was done via http then we transitioned to https.\n\n#### Address and canonical DNS records\n\nSome machines are available via a pubic IP.  In AWS Route 53 we have CNAME\n(canonical) record DNS entries that point to the .bioconductor.org extension.\n\nhttps://console.aws.amazon.com/route53/home?region=us-east-1#resource-record-sets:Z2LMJH3A2CQNZZ\n\n#### Mac builders and the DFCI DMZ\n\nThe Mac builders are located outside the DFCI DMZ. When they https to the\n*primary builder* they are directed to the public IP which redirects to the\nprivate IP. The outgoing and return routes are the same.\n\n### What machines are used in which builds?\n\nThis changes with every release. The *active_devel_builders* and\n*active_release_builders* sections of\n[config.yaml](http://bioconductor.org/config.yaml) list the current builders.\n\n#### A note about time zones.\n\nThe builds are on Eastern Standard Time.\n\n## How the build system works\n\nThe build system has 2 distinct parts: building and propagation. The first\nis managed by the *biocbuild* user and the second by the *biocpush* user.\n\n### Builds\n\nThe BBS code is checked out on all build machines. Each builder has a cron job\n(or Scheduled Task on Windows) that kicks off the builds. On all build\nmachines, the build system runs as *biocbuild*.\n\nThe crontab for the *biocbuild* user on one of the Linux build machines\n(a/k/a primary build nodes) lists all tasks involved in the builds.\n\n#### prerun\n\nThe first line in the crontab on the primary Linux builder is the start of\nthe prerun script:\n\n    # prerun\n    00 17 * * * /bin/bash --login -c 'cd /home/biocbuild/BBS/3.17/bioc/`hostname` \u0026\u0026 ./prerun.sh \u003e\u003e/home/biocbuild/bbs-3.17-bioc/log/`hostname`-`date +\\%Y\\%m\\%d`-prerun.log 2\u003e\u00261'\n\nThe *prerun* step happens only on the primary build node. `prerun.sh`\nsources `config.sh` and then calls python script `BBS-prerun.py`.\n\n##### config.sh\n\nThe sourcing of `config.sh` sets up environment variables used during the\nbuild. First, variables specific to this build machine are set up. Then, inside\n`config.sh`, another `config.sh` script one level up is sourced. This sets up all\nenvironment variables specific to all Unix (Linux and Mac) nodes involved in\nthis *software* build. Inside this `config.sh`, the `config.sh` one level up is\nalso sourced. That script sets up more environment variables common to all\nbuilds (software and experiment data) for this version of Bioconductor.\n\nIt's important to understand this pattern because it occurs in several places\nin BBS. Shell scripts (or batch files on windows) are essentially used to\nensure that configuration is correct, but most of the actual build work is done\nby python scripts.\n\nAfter `prerun.sh` sets up all the environment variables, it runs python\nscript `BBS-prerun.py`.\n\n`BBS-prerun.py` runs the following stages:\n\n* STAGE1: [on Linux only] Make a local copy of all packages to be built from\n          the version control location (i.e., git, svn etc.).\n\nThe start time of this script is the deadline for changes for the day. Any\nchanges made after that time won't be picked up until the following day's\nbuild.\n\n#### run\n\nThe next line in the crontab starts the `run.sh` script:\n\n    # run:\n    55 17 * * * /bin/bash --login -c 'cd /home/biocbuild/BBS/3.17/bioc/`hostname` \u0026\u0026 ./run.sh \u003e\u003e/home/biocbuild/bbs-3.17-bioc/log/`hostname`-`date +\\%Y\\%m\\%d`-run.log 2\u003e\u00261'\n\nAt the time of this writing, prerun takes about 55 min so the run script\nmust start after that time.\n\nThis script sources config files in the same way as `prerun.sh`. It also\nsets up Xvfb (the virtual frame buffer for X11; this makes sure\nthat packages which need access to X11 can have it).\n\nAfter loading environment variables, the main python build script, `BBS-run.py`, is run.\n\nThis script runs the following stages:\n\n* STAGE2: Preinstall all package dependencies\n          (INSTALL column on the build report)\n\n* STAGE3: Run `R CMD build` on all BioC packages\n          (BUILD column on the build report)\n\n* STAGE4: Run `R CMD check` on all package source tarballs produced by STAGE2\n          (CHECK column on the build report)\n\n* STAGE5: [Windows and Mac only] Make binary packages\n          (BUILD BIN column on the build report)\n\nEach stage is run in parallel. The system does not move from one stage to the\nnext until all jobs in the current stage are completed.\n\n\n#### postrun\n\nAt this point the builds should have finished on all nodes. The next line\nin the crontab starts the posrun script. This must not start until the\n`run.sh` job has finished on all nodes.\n\n    # postrun:\n    45 13 * * * /bin/bash --login -c 'cd /home/biocbuild/BBS/3.17/bioc/`hostname` \u0026\u0026 ./postrun.sh \u003e\u003e/home/biocbuild/bbs-3.17-bioc/log/`hostname`-`date +\\%Y\\%m\\%d`-postrun.log 2\u003e\u00261'\n\nThe prerun build script started at 17:00 and now it is 13:45 the following\nafternoon. We hope that all builders have finished by now, otherwise there will\nbe (as there often is) some manual steps to do at this point.\n\nThe build system will now run `postrun.sh` which initializes environment\nvariables as described above and then runs the following 3 python scripts:\n\n##### BBS-make-BUILD_STATUS_DB.py\n\nThis script performs stage6a:\n\n* stage6a: [Linux only] Create `BUILD_STATUS_DB.txt` file which records the status of\n           STAGES 2-5 on all platforms.\n\n    biocbuild@malbec1:~/public_html/BBS/3.17/bioc$ head BUILD_STATUS_DB.txt\n    a4#malbec1#install: NotNeeded\n    a4#malbec1#buildsrc: OK\n    a4#malbec1#checksrc: OK\n    a4#tokay1#install: NotNeeded\n\n##### BBS-make-OUTGOING.py\n\nThis script performs stage6b:\n\n* stage6b: [Linux only] Copy build products to OUTGOING folder for later\n           transfer to the website by *biocpush*.\n\n##### BBS-make-PROPAGATION_STATUS_DB.py\n\nThis script calls `BBS/utils/makePropagationStatusDb.R` which\ncreates the `PROPAGATION_STATUS_DB.txt` file.\nThis file identifies which packages and what format, e.g., source or binary,\nwill be pushed to the website.\n\n    biocbuild@malbec1:~/public_html/BBS/3.17/bioc$ head PROPAGATION_STATUS_DB.txt\n    a4#source#propagate: UNNEEDED, same version is already published\n    a4#win.binary#propagate: UNNEEDED, same version is already published\n    a4#mac.binary.el-capitan#propagate: UNNEEDED, same version is already published\n    a4Base#source#propagate: UNNEEDED, same version is already published\n\n##### BBS-report.py\n\nThis script performs stage6d:\n\n* stage6d: [Linux only] Generate and publish HTML report to the website.\n\n\nThe crontab contains essentially the same entries for the\nexperiment data builds though they run at different times.\n\n### Propagation Pipeline\n\nThe steps discussed so far complete the `Run` portion of the builds. All\nnodes have finished building and build products have been deposited on the\nLinux primary builder. The build report was created and posted on the website.\n\nThe second part of this process is called the \"propagation pipe\" and involves\nmoving build products from the primary builder to the website. The products are\nthe package tarballs and binaries that will become available via\n`BiocManager::install()`\nas well as information used to build the landing pages. These steps are\nperformed by the *biocpush* user and involve the primary builder only.\n\nLooking at *biocpush*'s crontab, we see:\n\n    35 14 * * * cd /home/biocpush/propagation/3.17 \u0026\u0026 (./updateReposPkgs-bioc.sh \u0026\u0026 ./prepareRepos-bioc.sh \u0026\u0026 ./pushRepos-bioc.sh) \u003e\u003e/home/biocpush/cron.log/3.17/updateRepos-bioc-`date +\\%Y\\%m\\%d`.log 2\u003e\u00261\n\nNotice the job starts at 14:35. This is hopefully enough time for\nthe `postrun.sh` script (above) to have finished; otherwise we'll have to\nre-run some things manually.\n\nThe `cron` job above runs three scripts, to *update*, *prepare*, and *push*.\n\n#### update\n\nThe *update* script moves the build products that can be propagated from\n`/home/biobuild/public_html/BBS/X.Y/REPO/OUTGOING/` into\n`/home/biocpush/PACKAGES/X.Y/REPO/` where `X.Y` is the version of\nBioconductor and `REPO` is the type of package, e.g., bioc or data.\n\nIf a package has been updated, with an appropriate version bump, the older\nversion is either moved to the 'Archive' folder (in release) or removed from\nthe repository (in devel).\n\n#### prepare\n\nThe *prepare* script does not move files around but just populates other parts\nof our internal repository which will later be moved to the web site. Most\nimportantly this includes the package indexes (`PACKAGES` and `PACKAGES.gz`)\nwhich tell install.packages() and BiocManager::install() and friends which packages can be\ninstalled.  There's also a `VIEWS` file which is used to build parts of our web\nsite (especially the package landing pages). From each built package we also\nextract vignettes (built documents, source documents, and Stangled R source),\n`README`s, `INSTALL` and `LICENSE` files, reference manuals, and other material\nthat we want to link to on the package landing page.\n\n#### push\n\nFinally the *push* script uses *rsync* to copy the internal repository to our\nweb site, which is where users go when they install a package via\n`BiocManager::install()`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbioconductor%2Fbbs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbioconductor%2Fbbs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbioconductor%2Fbbs/lists"}