{"id":16190010,"url":"https://github.com/brson/rust-infra","last_synced_at":"2025-04-07T14:19:08.453Z","repository":{"id":66303344,"uuid":"36249348","full_name":"brson/rust-infra","owner":"brson","description":"Information about Rust infrastructure","archived":false,"fork":false,"pushed_at":"2015-07-23T20:46:22.000Z","size":176,"stargazers_count":8,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-13T16:48:07.961Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brson.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-05-25T19:07:26.000Z","updated_at":"2024-03-14T18:32:37.000Z","dependencies_parsed_at":"2023-02-20T02:45:42.293Z","dependency_job_id":null,"html_url":"https://github.com/brson/rust-infra","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brson%2Frust-infra","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brson%2Frust-infra/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brson%2Frust-infra/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brson%2Frust-infra/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brson","download_url":"https://codeload.github.com/brson/rust-infra/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247666016,"owners_count":20975788,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-10T07:38:06.618Z","updated_at":"2025-04-07T14:19:08.431Z","avatar_url":"https://github.com/brson.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Links Ahoy\n\n* [rust-buildbot](https://github.com/rust-lang/rust-buildbot). Our configuration for [buildbot](http://buildbot.net/).\n* [Our buildbot deployment](http://buildbot.rust-lang.org/)\n* [Homu](https://github.com/barosl/homu). Homu is a script we use with buildbot to do ['pre-commit' testing](http://graydon2.dreamwidth.org/1597.html).\n* [Our homu deployment](http://buildbot.rust-lang.org/homu/)\n* [Homu Rust queue](http://buildbot.rust-lang.org/homu/queue/rust)\n* [Homu Cargo queue](http://buildbot.rust-lang.org/homu/queue/cargo)\n* [Travis dashboard](http://buildbot.rust-lang.org/travis/travis.html). We use Travis to test rust-lang crates that live outside of rust-lang/rust.\n* [Automation metabug](https://github.com/rust-lang/rust/issues/17356). The evergrowing list of things to do.\n* [rust-admin](https://github.com/brson/rust-admin). Old information, sometimes still worth looking at. This repo is private because it contains secrets.\n* [AWS console](https://console.aws.amazon.com/console/). Login page for AWS.\n* [play.rust-lang.org](https://play.rust-lang.org/). Our online Rust evaluator. It has a cross-site API so that it can be used by other projects, e.g. rust-by-example lets all examples be evaluated on play.rlo.\n* [rustup.sh](http://github.com/rust-lang/rustup). A script commonly used to download the compiler from release channels. It interprets the metadata uploaded by the distribution builders.\n* [rust-packaging](https://github.com/rust-lang/rust-packaging) - The script run by the 'dist-packaging' builders to produce the final Rust builds in a variety of packaging formats.\n* [playpen](https://github.com/rust-lang/rust-playpen) - The code that runs play.rust-lang.org.\n\n\n\n# SSH and security tips\n\nLearn to love ssh and ssh-agent. Any time you ssh into another machine I suggest using the `-A` flag. This will forward your ssh-agent keyring to the remote machine so you can further authenticate as yourself when ssh'ing into other machines or pushing git repos.\n\nYou'll inevitably need access to the Mozilla network from outside the office. You do this by registering your SSH key with LDAP, then ssh'ing to e.g. banderson@ssh.mozilla.com. From there you can get to other machines, including our AWS bastion server (a 'bastion' is a secure server you access the rest of the system through).\n\nI suggest turning on 2-factor authentication on GitHub.\n\nAs somebody responsible for Rust infrastructure you are going to have access to secrets which, if compromised, could result in significant damage to the project. Please take steps to secure them. Personally I keep my stuff on a secure [Wuala](https://www.wuala.com) drive then further encrypted with [encfs](https://vgough.github.io/encfs/) (though note that encfs is unmaintained and has known security issues - it is not sufficient on its own and there are probably better solutions now). If you can, don't permanently decrypt the GPG messages you receive containing secrets - just run gpg again every time you need access.\n\nPut a password on your GPG key.\n\n\n\n\n# AWS\n\nMost of our infrastructure is on AWS.\n\nThe bastion is at the public IP 54.215.17.149, and called 'bastion' in EC2. All access goes through it. It's used to get to either the production or development master or slaves. It's in the `rust-bastion` security group, which only allows connections from specific IP's (currently only the MV office), so to access the automation from outside of MV the security group must be augmented.\n\nWe use EC2, S3 and CloudFront. CloudFront is a content delivery network used soley for serving our s3 bucket over HTTPS via static.rust-lang.org.\n\n## Accessing AWS machines\n\nAccess to AWS machines requires the *shared* buildbot-west-slave-key.pem SSH key. Add it to your keyring with `ssh-add buildbot-west-slave-key.pem`, then `ssh -A ubuntu@54.215.17.49` to get into the bastion. From there you can ssh to other machines.\n\nWindows machines on AWS are accessed by tunnelling RDP through the bastion. Establish the tunnel from your local machine by running e.g. `ssh -L54.215.17.49:3389:$windows_slave_address:3389`, then rdp to localhost e.g. `rdesktop localhost`.\n\n## Machines\n\nDescriptions of some AWS machines, as labeled in EC2:\n\n* bastion - The machine you access other AWS machines through\n* rust-prod-master - The production buildbot build master, also runs other related scripts.\n* rust-dev-master - The development buildbot build master.\n* prod-slave-* - Production buildslaves. These start and stop on demand.\n* doc.rust-lang.org - Mislabeled. An nginx proxy for various purposes, mostly wrapping HTTP services in HTTPS.\n* play.rust-lang.org - The machine hosting play.rust-lang.org. Runs arbitrary Rust code in a sandbox.\n\n## Proxy setup\n\nThe machine labeled doc.rust-lang.org is an nginx proxy that we use for a variety of purposes. It's most important function is serving HTTP content over HTTPS.\n\nThis machine contains the rust-lang.org wildcard TLS cert.\n\n\n\n\n# Buildbot\n\nWe have a decently customized buildbot environment, the [source of which](https://github.com/rust-lang/rust-buildbot) is open. It has been developed incrementally and haphazardly over years. Due to that and the somewhat complex nature of buildbot the program logic can take a while to grasp.\n\nWe run buildbot in two environments: 'prod' and 'dev'. The dev environent tends to only get used for developing 'significant' changes to rust-buildbot and it often lags behind the prod environment.\n\nBuildbot is used for three purposes: continuous integration, distribution, try builds (a service for devs to test their changes on our bots).\n\nWe further customize buildbot by driving continuous integration through [Homu](https://github.com/rust-lang/rust-buildbot) which implements ['pre-commit' testing](http://graydon2.dreamwidth.org/1597.html), testing GitHub pull requests *before* merging them into master. This is an unorthodox setup - most CI systems merge then test. This setup though guarantees that our build is always green - there are no 'sheriffs' who have to go through the tree after the fact fixing breakage.\n\nThere are two accounts used to log into buildbot via the web interface: 'any-build' and 'rust'. The 'rust' user can start and stop builds, but not for any 'dist' builders (the ones that publish releases). 'any-build' has full access. The passwords to these are shared.\n\n## The build master\n\nThe buildbot architecture features a single master that coordinates the build and an arbitrary number of slaves.\n\nThe production build master is located at the public IP 54.241.248.193 and has the EC2 name, rust-prod-master.\n\nThe build master is mostly operated by the 'rustbuild' user. To access, ssh from the bastion to `rustbuild@10.190.147.69` (the private IP).\n\nDon't bother trying to `buildbot restart master` - the slave shutdown takes a long time and the restart will time out. When shutting down with `buildbot stop master`, make sure you wait until buildbot actually exits (it takes a long time to wait on the EC2 slaves) before starting again.\n\nBuildbot listens *locally* on port 8010 via HTTP, and Homu on port 7942. A logal nginx instance is configured via `/etc/nginx/available-sites/default` to proxy them both to port 80.\n\nThe buildbot source is installed at ~/rust-buildbot, homu at ~/homu.\n\nThe build master contains an s3cmd configured so that - when necessary - people can upload directly to s3. The automation does all of the interfacing to s3 under normal circumstances.\n\nThe rustbuild user runs a number of cronjobs.\n\nThe build master contains critical secrets, including the GPG subkey for signing releases and an s3 access token.\n\n## Important files\n\n* ~/rust-buildbot/master/master.cfg - The buildbot script. It's python - buildbot convention uses the .cfg extension though.\n* ~/rust-buildbot/master/master.cfx.txt - Environment specific configuration. Contains secrets.\n* ~/rust-buildbot/master/slave-list.txt - The list of slaves. Contains secrets.\n* ~/homu/cfg.toml - The homu config\n* ~/invalidate.sh - A fairly ineffective script for daily invalidating artifacts on Cloudfront, our CDN.\n* ~/travis - The script that creates http://buildbot.rust-lang.org/travis/travis.html periodically via cron job\n* ~/update-rust-dist-index.sh - Another script that generates index.html files for our s3 bucket.\n* ~/s3-directory-listing - Used by the above.\n\nThe other stuff in rustbuild's home directory is (probably) old and unused.\n\n## The build slaves\n\nThe build master starts EC2 slaves on demand, calling them `prod-slave-$name`. Buildslaves communicate to the master via stunnel.\n\nWhen an EC2 slave boots it runs ~/rust-buildbot/setup-slave.sh, which pulls the slave name and password, and the IP address of the build master from EC2 data, configures and runs stunnel and the buildslave.\n\nThe macs are configured manually, and should restart on reboot, but in practice they tend to need to be fiddled with after a reboot.\n\nThe macs live on brson's desk, at the following addresses:\n\n- rust-mac3.corp.mtv2.mozilla.com\n- rust-mac4.corp.mtv2.mozilla.com\n- rust-mac5.corp.mtv2.mozilla.com\n- rust-mac6.corp.mtv2.mozilla.com\n\nThese can only be accessed from within the Mozilla network. To get to them from outside first ssh to ssh.mozilla.com.\n\n## Continuous Integration\n\nContinuous integration is done by the 'auto-*' builders in buildbot, builds on which are started by homu pushing to the 'auto' branch.\n\nHomu is running in an instance of\n[screen](https://www.gnu.org/software/screen/) under the *root*\naccount, not *rustbuild*. To access it ssh from bastion to the build\nmaster as root, then run `screen -R` (reattach to existing session).\nTo detach press `Ctrl-A`, then `D` (`Ctrl-A` is the screen command key).\n\nCargo's continuous integration is done by the 'cargo-*' builders, also driven by homu.\n\nTODO. There's probably a lot more to say about Rust continuous integration.\n\n## Distribution\n\nThe 'stable-dist-*', 'beta-dist-*', and 'nightly-dist-*' builders are all used for publishing releases of Rust. Their interaction is somewhat complex, customized in master.cfg. These builders upload artifacts to the s3 'dist/' or 'cargo-dit/' which are then interpreted by various tools, like rustup.sh.\n\nTODO. There's probably a lot more to say about Rust distribution.\n\n# Travis\n\nTODO\n\n# crates.io\n\n[crates.io](https://github.com/rust-lang/crates.io) runs on Heroku.  It has 2 web dynos and 1 worker dyno, and is under the `mozillacorporation` organization. Heroku outages that might affect crates.io are listed [here](https://status.heroku.com/).\n\nDNS for crates.io is hoted by [Zerigo](https://www.zerigo.com/managed-dns) because it supports ANAME records. The ANAME record is necessary to make SSL work on the apex domain (crates.io) with Heroku hosting. Zerigo was chosen because it's bundled with Heroku automatically. \n\n\n# play.rust-lang.org\n\nTODO\n\n# S3\n\n# DNS addresses\n\nA list of DNS addresses owned by rust-lang and what they are.\n\n* www.rust-lang.org - The website, hosted via GitHub pages.\n* static.rust-lang.org - Mostly used for distributing binaries, hosted via CloudFront, maps directly to the static-rust-lang-org s3 bucket.\n* dev-static.rust-lang.org - The equivalent of the above in dev.\n* doc.rust-lang.org - The doc server, proxies over HTTPS via nginx.\n* users.rust-lang.org - A [Discourse](http://discourse.org) instance for Rust users, hosted by Discourse.\n* internals.rust-lang.org - A Discourse instance for Rust developers, hosted by Discourse.\n* blog.rust-lang.org - Hosted by GitHub pages.\n* play.rust-lang.org - Hosted on AWS.\n* playtest.rust-lang.org\n* {imap,pop,smtp}.rust-lang.org - IMAP acess to @rust-lang emails. Rarely used, though discourse may be using the smtp server.\n\nWe use easydns.com. Ask brson if you need access.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrson%2Frust-infra","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrson%2Frust-infra","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrson%2Frust-infra/lists"}