{"id":19623914,"url":"https://github.com/bioconductor/packagebuilder","last_synced_at":"2025-04-28T05:30:50.332Z","repository":{"id":36376206,"uuid":"40681062","full_name":"Bioconductor/packagebuilder","owner":"Bioconductor","description":null,"archived":false,"fork":false,"pushed_at":"2024-11-08T19:40:08.000Z","size":2500,"stargazers_count":5,"open_issues_count":1,"forks_count":7,"subscribers_count":6,"default_branch":"devel","last_synced_at":"2024-11-08T20:29:56.851Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Bioconductor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-08-13T20:48:43.000Z","updated_at":"2024-11-08T19:40:13.000Z","dependencies_parsed_at":"2024-11-08T20:35:27.208Z","dependency_job_id":null,"html_url":"https://github.com/Bioconductor/packagebuilder","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bioconductor%2Fpackagebuilder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bioconductor%2Fpackagebuilder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bioconductor%2Fpackagebuilder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bioconductor%2Fpackagebuilder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Bioconductor","download_url":"https://codeload.github.com/Bioconductor/packagebuilder/tar.gz/refs/heads/devel","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224096604,"owners_count":17255123,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T11:36:04.318Z","updated_at":"2024-11-11T11:36:05.274Z","avatar_url":"https://github.com/Bioconductor.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"Bioconductor Single Package Builder (SPB)\n=========================================\n[![Build\nStatus](https://travis-ci.org/Bioconductor/packagebuilder.svg)](https://travis-ci.org/Bioconductor/packagebuilder)\n\nOverview\n========\n\nThe single package builder runs R CMD build and R CMD check on packages and provides the output of these processes. The system consists of the EC2 instances staging.bioconductor.org, issues.bioconductor.org and rabbitmq as well as the current devel build machines. The flow of information starts at issues.bioconductor.org triggered by a new package submission or update (version bump). A message is sent to rabbitmq which sends messages to the build nodes. staging.bioconductor.org tracks build progress and records logs.\n\nCode\n====\n\nCode for the SPB is stored in [GitHub](https://github.com/Bioconductor/packagebuilder).\n\nCurrently, the SPB consists of 2 components in this git repo, in top-level directories:\n  * spb_history - A Django web app to track build history.\n  Accessible at http://staging.bioconductor.org:8000\n  * workers - Python scripts that run on each build machine\n\nOptions to start a build\n=========================\n\n1. Submitting packages to [github](https://github.com/Bioconductor/Contributions)\n2. [Manually restart][] a build.\n\n[Manually restart]: https://github.com/Bioconductor/packagebuilder/blob/master/documentation/Troubleshooting.md#manually-restarting-a-build\n\nThese all send messages to an installation of RabbitMQ (a Java-based messaging framework)\nto listeners on each build machine ([server.py](workers/server.py)). The build machines\nstart building the package and send back progress messages.\n\nThere is another listener (called [archiver.py](workers/archiver.py) which\nwrites build events to a database, where they can\nthen be displayed by the spb_history web application\n(written in Python's Django framework).\n\nWhere possible, the code uses existing code from the Bioconductor Build\nSystem (BBS). In particular, it uses BBS scripts to set environment variables\nfor the build, though these are overridden in a couple of cases.\n\nOn the build machines, the listener that is always running is called\n[server.py](workers/server.py). When it receives a build request, it kicks off a script\nparticular to that build machine (called e.g. petty.sh or moscato2.bat)\nwhich sets environment variables, then runs [builder.py](workers/builder.py) to do the\nactual building.\n\nBuild Machines\n==============\n\nSee `active_devel_builders` in http://bioconductor.org/config.yaml\nto see which machines are used by the SPB.\n\nSystem Integration\n===========================\n\nA job is submitted to the Single Package Builder (SPB) when a package is submitted to the github contributions\n(https://github.com/Bioconductor/Contributions) or when the version of an existing package is bumped.\n\n* New Package Submission\n\nThe code that manages when issues (packages) are submitted can be found at\n[issue_tracker_github](https://github.com/Bioconductor/issue_tracker_github)\nand is implemented on the AWS cloud instance issues.bioconductor.org.\n\n* Version bump\n\nWhen a package submits a version bump, the above manager is run.\nIt sends a message to the listeners on each build node as described\nat the start of this document.\n\nIn both cases, the message sent to the  SPB contains a flag indicating that this build was\noriginated by the SPB.\n\nWhen the build is complete, another script on staging\n[track_build_completion.py](spb_history/track_build_completion.py)\nis listening, and it posts a message to the contribuctions issue\n(using an HTTPS request) including a link to the build report.\n\n* SPB Moving Parts\n\nThe SPB was designed to use a message broker (in this case RabbitMQ)\nso that components could be loosely coupled and so that they could\nsend information in an asynchronous, real-time fashion, allowing\nlive updates.\n\nIn practice there are some dependencies between the moving parts.\n\nHere are the moving parts, first briefly and then in more detail.\n\n* Broker - currently a machine in the cloud called\n  rabbitmq. All it does is run an RabbitMQ\n  message broker. Messages are passed using a text protocol\n  called STOMP.\n* Issue manager. Currently implemented on the machine\n  issues.bioconductor.org. It detects when a package is submitted,\n  or updated, and sends a message to the SPB telling it to start a build.\n* build node server; a python script called [server.py](workers/server.py) should\n  be running on each build node at all times. When an\n  instruction is received to start a build, this script\n  starts a new process (first sourcing some variables\n  specific to the build node, then running a python script\n  called [builder.py](workers/builder.py)).\n* Three scripts that run on the staging.bioconductor.org machine:\n  1. [track_build_completion.py](spb_history/track_build_completion.py): monitors progress\n     of builds and when a build is complete (all machines\n     have finished) it posts the build report to the web\n     and posts a message to the issue.\n  2. [archiver.py](workers/archiver.py): monitors progress of builds, writes all build\n     info to a database to be displayed by a Django web app\n     (what you see when you hit\n[http://staging.bioconductor.org:8000/](http://staging.bioconductor.org:8000/)).\n  3. [rerun_build.py](spb_history/rerun_build.py): for manually kicking off an SPB build\n     without having to version bump or resubmit.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbioconductor%2Fpackagebuilder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbioconductor%2Fpackagebuilder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbioconductor%2Fpackagebuilder/lists"}