{"id":16866946,"url":"https://github.com/webern/webern","last_synced_at":"2026-03-18T22:43:35.624Z","repository":{"id":88144513,"uuid":"332927330","full_name":"webern/webern","owner":"webern","description":null,"archived":false,"fork":false,"pushed_at":"2024-09-15T09:18:10.000Z","size":3,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-18T17:59:05.420Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/webern.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-26T00:47:30.000Z","updated_at":"2024-09-15T09:18:13.000Z","dependencies_parsed_at":"2024-11-25T00:00:35.564Z","dependency_job_id":null,"html_url":"https://github.com/webern/webern","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/webern/webern","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webern%2Fwebern","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webern%2Fwebern/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webern%2Fwebern/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webern%2Fwebern/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/webern","download_url":"https://codeload.github.com/webern/webern/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webern%2Fwebern/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267709885,"owners_count":24131932,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-29T02:00:12.549Z","response_time":2574,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T14:52:13.459Z","updated_at":"2026-02-06T09:05:57.469Z","avatar_url":"https://github.com/webern.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Matthew James Briggs\n\n- Experienced systems software engineer\n- Working in **Rust** professionally since 2020.\n\n**Call me Matt**\n\n- Q: Why print the full name, including the middle name?\n- A: Because there are many people by the name of Matt Briggs in the world.\n\n*Or call me Matthew, or Mateo (I've been working on my Spanish)*\n\n**Webern**\n\n- Q: Why a weird GitHub username?\n- A: Because I made it a long time ago before I knew GitHub was important.\n\nAnton Webern was a 12-tone composer of the *Second Viennese School*, buddies with Schönberg and\nBerg.\nWebern DGAF and wrote tiny, short, sparse pieces,\nsometimes so small you can count the notes.\nBack when I was in music school, I was impressed and wrote some short, sparse pieces of my own.\n\n**Annoying Disclaimer**\n\nUnless I say otherwise, anything I say on GitHub or anywhere else is my own opinion and not that of\nmy employers, whether past or present.\n\n---\n\n# Portfolio\n\nThis is a guide to some of the PRs, design docs, issues and other artifacts that are useful to find,\nespecially when I am interviewing for a new position.\n\n## AWS Bottlerocket\n\nI worked on the Bottlerocket team at Amazon from 2020 to 2024 (4.5 years), where I was a systems\nengineer working primarily with Rust.\n\nI worked on various projects, including:\n\n- The Rust implementation of The Update Framework (TUF), named [tough].\n- The Bottlerocket update system.\n- A Kubernasties[^1]-based testing system for Bottlerocket.\n- A basic metrics collection system.\n- The Twoliter Bottlerocket image customization build tool.\n\n### Twoliter\n\nI led the design and implementation of developer tooling for creating custom variants of\nBottlerocket, called [Twoliter]. When [Fargate] wanted to use Bottlerocket to replace AL2 in its\nnext platform version, the problem we faced was that Bottlerocket variants were defined in-line in\nthe\nBottlerocket monorepo. The only way for a customer to customize Bottlerocket would be to fork the\nmonorepo, maintain difficult patches, and build everything from source. We needed to create a\nproject-based solution for **Out of Tree Builds**.\n\nTwoliter and Kits were the solution. Twoliter is a binary-released, command-line tool for managing\nyour Bottlerocket variants. It allows you to compose packages provided by the Bottlerocket team in\n*Kits* to select what pre-built, Bottlerocket-provided software you want in your image, and allowing\nyou to add any software of your own to the image.\n\n- [Twoliter Design Document]: I authored this design at the start of the project.\n- [Twoliter GitHub Issues]: I initiated the project with a series of GitHub issues that, when\n  completed, would realize the design.\n- [Twoliter Pull Requests]: My pull requests on the project sorted chronologically.\n- [Tokio and Async Stuff]: I often get asked in recruiter screens for Rust jobs if I know how to\n  write async code, use `tokio`, etc. Yes, I do.\n- [Buildsys Refactoring]: The `buildsys` tool needed a lot of knocking around before it could\n  start building and using kits. This was a big refactor to `builder.rs` and consolidation of the\n  environment variable interface.\n- [CLI Bugfix]: Here's a deceptively simple bugfix of the CLI argument interface. Check out the\n  regression testing.\n- [Change RPM Output Directory Structure]: It had to change before we could make kits a reality.\n- [First Kit Build]: Buildsys builds a toy test kit for the first time!\n- [Remove Agile Workaround]: Remove the workaround that we were using to support our first customer\n  before kits were finished.\n\nI have written a more extensive overview of the project for an deep-dive interview that I did with\none company.\n\n- [Twoliter project description document](./TWOLITER.md)\n\n### Signed Migrations\n\nThis was an early, Level 5-scoped project in which I altered [tough] and the Bottlerocket update\nsystem such that migration binaries would be verified by cryptographic signature before execution\nupon an update reboot.\n\n**Background**\n\nBottlerocket uses an immutable root filesystem. Most parts of it are read-only, while parts that\nneed to be writeable (such as `/etc`) are tempfs. In this way, no changes to the root filesystem can\npersist a reboot. This is accomplished with secure boot, SELinux and DM-verity.\n\nThe system is configured by updating settings through an HTTP service that runs locally on a domain\nsocket. These user-settings are persisted on the user data partition in a schema called the\n*datastore*. On boot, or when settings change, the `/etc` files are generated from templates and\naffected services are re-started.\n\nA Bottlerocket system is atomically updatable. That is, a completely new root filesystem image can\nbe downloaded into a spare partition. Grub is updated to make the other partition the live one, and\na reboot takes place.\n\nHere is a simplified picture of the partition layout.\n\n```text\n┌─────────────────────────────────────────────────────────────────────┐\n│                     Bottlerocket Disk Layout                        │\n├───────────────────┬───────────────────┬─────────────────────────────┤\n│                   │                   │                             │\n│         A         │         B         │         User Data           │\n│    (Active OS)    │   (Passive OS)    │   (Containers, Settings)    │\n│                   │                   │                             │\n└───────────────────┴───────────────────┴─────────────────────────────┘\n```\n\n**The Update Framework (TUF)**\n\nThe Update Framework is used to secure the authenticity of these updates on a Bottlerocket system.\nThe image is downloaded, and starting with `root.json`, which has been shipped with Bottlerocket as\nthe root of trust, TUF ensures that whatever is downloaded from the Bottlerocket update repository\nis authentic.\n\n**Migrations**\n\nBecause the schema of the datastore could change on upgrade or downgrade, Bottlerocket borrowed the\nidea of *migrations* from SQL ORM[^2]. In our case, these were little binaries that would run on the\nfirst boot into a new version and take the datastore up or down to the correct version.\n\nDuring the update process, these binaries are downloaded and stored on the persistent data\npartition ready to be executed, if needed, upon the reboot into a different version.\n\n**The Problem: Security**\n\nPrior to the release of Bottlerocket 1.0, these migration binaries were only being authenticated\nwhen they were being downloaded from the TUF repository. There was a TOCTOU vulnerability in which\nthese binaries could possibly be replaced by an attacker between the time of the download and when\nthey were needed during reboot. It was understood that this needed to be closed before Bottlerocket\nwas to be made Generally Available (v1.0).\n\n**The Solution**\n\nThe solution was to cache all the TUF repository metadata necessary to re-validate these binaries\nduring the boot. Then the migrator service needed to be changed to re-validate the binaries and run\nthem from memory.\n\n**Step 1: Design**\n\nStep 1 was to outline the process fully to the team and to early-adopters of Bottlerocket. I wrote\nan *RFC*-style GitHub issue in advance of any changes to describe what we were planning to do.\n\nSee: [RFC: Signed Migrations].\n\n**Step 2: Update the TUF Library**\n\nOur Rust TUF library, named [tough], needed to be altered such that it could be told to cache its\nmetadata and be run in an *unsafe* mode where it ignores expired metadata. This is because the\nnetwork is not available during the boot to enact the TUF specification for fetching updated\nmetadata files.\n\n- [tough: add method for caching a repo with its metadata]\n- [tough: support the loading of an expired repo in unsafe mode]\n- While were at it, here are all of my [tough PRs].\n\n**Step 3: Change Bottlerocket's Update Systems**\n\nChange the downloader and migrator systems in Bottlerocket to cache the metadata, to re-check the\nbinary signatures during reboot, and to run them from memory. Also add extensive testing to the\nmigration system which was a) difficult to test, and b) previously untested.\n\n- [updog: move migration searching code to update_metadata for reuse]\n- [updog and migrator: signed migrations]\n- [migrator: end-to-end test]\n\n**Step 4: Remove Compat Code**\n\nBecause this was a breaking change, we needed to support both modes of operation through a \"pivot\"\nversion of Bottlerocket which would point hosts to a new update repository. Once that was done, the\nold migration code could be removed.\n\n- [migrator: remove unsigned migration support]\n\n## Bottlerocket Test System\n\nI designed the [Bottlerocket test system]. I used the K-word technology for this. It included an\nOperator for driving resource and test state, and two Custom Resource Definitions, one for\nResources (such as EC2 instances, clusters, etc) and one for tests (such as the Sonobouy conformance\nsuite). Oh yeah, this was done in Rust, not Go, since we were a Rust-based team.\n\n- Here is my [test system Design Document].\n- Here are some [cool traits and objects] that I created.\n\nI was proud of my test system's generality when we needed to add testing for bare metal servers.\nThis was completely foreign from what we had in mind when I designed the framework. But we were able\nto add bare metal testing with little or no change to the framework!\n\n- [Bare metal agents]\n\n[^1]: I really don't want the K-word associated with my skill set 😅. I don't mind if we use it to\nrun our stuff, I just don't want to be a Kubernasties-focused developer.\n\n[^2]: The datastore is not a SQL database. It is just a schema of JSON files, but the idea of\nmigrations was inspired by SQL ORM solutions such as Ruby on Rails.\n\n\u003c!-- @formatter:off --\u003e\n\n[Bare metal agents]: https://github.com/bottlerocket-os/bottlerocket-test-system/pull/773\n[Bottlerocket test system]: https://github.com/bottlerocket-os/bottlerocket-test-system\n[Bottlerocket Test System Design Document]: https://github.com/bottlerocket-os/bottlerocket-test-system/blob/develop/docs/DESIGN.md\n[Buildsys Refactoring]: https://github.com/bottlerocket-os/twoliter/pull/134\n[CLI Bugfix]: https://github.com/bottlerocket-os/twoliter/pull/148\n[Change RPM Output Directory Structure]: https://github.com/bottlerocket-os/twoliter/pull/210\n[Fargate]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Fargate.html#:~:text=Bottlerocket\n[First Kit Build]: https://github.com/bottlerocket-os/twoliter/pull/227\n[RFC: Signed Migrations]: https://github.com/bottlerocket-os/bottlerocket/issues/905\n[Remove Agile Workaround]: https://github.com/bottlerocket-os/twoliter/pull/286\n[The Update Framework]: https://theupdateframework.io/\n[Tokio and Async Stuff]: https://github.com/bottlerocket-os/twoliter/pull/98\n[Twoliter Design Document]: https://github.com/bottlerocket-os/twoliter/tree/develop/docs/design\n[Twoliter GitHub Issues]: https://github.com/bottlerocket-os/twoliter/issues?q=is%3Aissue%20state%3Aclosed%20author%3Awebern%20sort%3Acreated-asc\n[Twoliter Pull Requests]: https://github.com/bottlerocket-os/twoliter/pulls?q=is%3Apr+author%3Awebern+sort%3Acreated-asc\n[Twoliter]: https://github.com/bottlerocket-os/twoliter\n[cool traits and objects]: https://github.com/bottlerocket-os/bottlerocket-test-system/tree/develop/agent/resource-agent/src\n[migrator: end-to-end test]: https://github.com/bottlerocket-os/bottlerocket/pull/956\n[migrator: remove unsigned migration support]: https://github.com/bottlerocket-os/bottlerocket/pull/966\n[test system Design Document]: https://github.com/bottlerocket-os/bottlerocket-test-system\n[tough PRs]: https://github.com/awslabs/tough/pulls?q=+is%3Apr+sort%3Acreated-asc+author%3Awebern+\n[tough: add method for caching a repo with its metadata]: https://github.com/awslabs/tough/pull/111\n[tough: support the loading of an expired repo in unsafe mode]: https://github.com/awslabs/tough/pull/121\n[tough]: https://github.com/awslabs/tough\n[updog and migrator: signed migrations]: https://github.com/bottlerocket-os/bottlerocket/pull/930\n[updog: move migration searching code to update_metadata for reuse]: https://github.com/bottlerocket-os/bottlerocket/pull/920\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwebern%2Fwebern","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwebern%2Fwebern","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwebern%2Fwebern/lists"}