{"id":29675717,"url":"https://github.com/oxidecomputer/rfd-api","last_synced_at":"2025-07-22T23:38:10.055Z","repository":{"id":227123719,"uuid":"670723754","full_name":"oxidecomputer/rfd-api","owner":"oxidecomputer","description":"Read, write, and process RFDs","archived":false,"fork":false,"pushed_at":"2025-07-21T03:05:16.000Z","size":2039,"stargazers_count":75,"open_issues_count":28,"forks_count":13,"subscribers_count":18,"default_branch":"main","last_synced_at":"2025-07-21T05:21:29.521Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxidecomputer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-07-25T17:28:04.000Z","updated_at":"2025-06-25T03:45:26.000Z","dependencies_parsed_at":"2025-04-19T04:35:33.558Z","dependency_job_id":"95c9f7d8-617d-4da3-b827-d9632f27d6fa","html_url":"https://github.com/oxidecomputer/rfd-api","commit_stats":null,"previous_names":["oxidecomputer/rfd-api"],"tags_count":26,"template":false,"template_full_name":null,"purl":"pkg:github/oxidecomputer/rfd-api","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxidecomputer%2Frfd-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxidecomputer%2Frfd-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxidecomputer%2Frfd-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxidecomputer%2Frfd-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxidecomputer","download_url":"https://codeload.github.com/oxidecomputer/rfd-api/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxidecomputer%2Frfd-api/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266245339,"owners_count":23898810,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-22T23:38:05.377Z","updated_at":"2025-07-22T23:38:10.030Z","avatar_url":"https://github.com/oxidecomputer.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# rfd-api\n\nBackend services and tools for processing and managing RFDs\n\n## RFD CLI\n### Getting Started\n1. Download the latest release of `rfd-cli` or run `cargo run -p rfd-cli`\n2. Configure the API host with `rfd-cli config set host https://rfd-api.shared.oxide.computer`\n3. Choose an authentication mode based on the kind of session you want, either a short-term session token (id) or a long-term api token (token).\n\n#### Authenticate with short lived session\nTo log in with a short lived session run:\n```sh\nrfd-cli auth login google\n```\n\n#### Authenticate with long lived token\nTo generate and log in with a long lived token run:\n```sh\nrfd-cli auth login google -m token\n```\n\n### Formatting\nResults can be output either as machine (JSON) or human (tab) readable formats. A format can be specified per call via the `--format` argument. To persist this setting and apply it to call calls, it can be set in your config file via:\n\n```sh\nrfd-cli config set format \u003cFORMAT\u003e\n```\n\n## Backend\n\nThe RFD API backend is made up of two services:\n* `rfd-api` - API for accessing RFDs and handling GitHub webhooks\n* `rfd-processor` - Scans for RFDs to update, handles RFD state transitions, manages RFD assets\n\nThe RFD API backend services expect to run against a Postgres database.\n\n### API\n\nRunning the API requires setting up a configuration file as outlined in `config.example.toml`.\n\n### Processor\n\nDependencies\n\n* asciidoctor\n* Node\n  * @mermaid-js/mermaid-cli\n* Ruby\n  * rouge\n  * asciidoctor-pdf\n  * asciidoctor-mermaid\n\n## Background\n\nObjects reference:\n```\n                                    ┌─────────────────┐ ┌─────────────────┐\n                                    │                 │ │                 │\n                ┌─────────────────┐ │ PDF cccccc Src1 │ │ PDF cccccc Src2 │\n                │                 │ │                 │ │                 │\n                │ PDF aaaaaa Src1 │ └────────┬────────┘ └────────┬────────┘\n                │                 │          │                   │\n                └────────┬────────┘          ├───────────────────┘\n                         │                   │\n                ┌────────┴────────┐ ┌────────┴────────┐\n                │                 │ │                 │\n                │ Revision aaaaaa │ │ Revision cccccc │\n                ├─────────────────┤ ├─────────────────┤\n              ┌─┤ commit_sha      │ │ commit_sha      │\n              │ └────────┬────────┘ └────────┬────────┘\n              │          │                   │\n              │          └───────────────────┤\n┌─────────┐   │                              │\n│         │   └─────────────────┐            │\n│ Scanner │                     │            |\n├─────────┤     ┌────────────┐  │   ┌────────┴────────┐\n│ sha     ├───┐ │            │  │   |                 |\n├─────────┤   │ │ Job 1      │  │   │  RFD 123        │\n│ branch* ├─┐ │ ├────────────┤  │   ├─────────────────┤\n└─────────┘ │ ├─┤ sha        ├──┘   │ id              │\n            │ │ ├────────────┤      ├─────────────────┤\n┌─────────┐ ├─┼─┤ rfd        ├──────┤ rfd_number      │\n│         │ │ │ └────────────┘      └─────────────────┘\n│ Webhook │ │ │\n├─────────┤ │ │\n│ sha     ├─┼─┘\n├─────────┤ │\n│ branch* ├─┘\n└─────────┘\n```\nNote: Scanner and Webhook operations that occur on the default branch do not use the branch name for\ndetermining the RFD number to update. Instead they use the numeric portion of the \n`rfd/{number}/README.adoc` path.\n\n### Revisions\n\nEvery revision is tied to a commit* against a RFD readme file. There is no guarantee though that\nthere exists a revision though for every commit. While the RFD API will attempt to create a revision\nfor every commit, outages, missing webhooks, or internal errors can result in missing revisions.\nCurrently the background periodic processor does not attempt to backfill missing revisions, it only\nensures that there is a revision for the latest commit it sees during its run.\n\nNote: Force pushes may result in the removal of the commit that triggered a revision.\n\n## RFD Processing\n\nThe RFD processors primary purpose is the implement and maintain the specifications defined in RFD 1.\nAs internal needs have grown though, so has the processor. Each step of the processor is implemented\nas a separate action. Currently the supported actions are:\n\n| Action                 | Purpose |\n|------------------------|------------\n| copy_images_to_storage | Copies images and static files associated with a RFD to cloud storage\n| create_pull_request    | Create a PR for the RFD if it does not have one and the RFD is in discussion\n| ensure_default_state   | Checks that RFDs on the default branch have appropriate states\n| ensure_pr_state        | Updates the state attribute for RFDs not on the default branch as needed\n| update_discussion_url  | Updates the discussion url attribute in the RFD contents\n| update_pdfs            | Create and upload a PDF version of the RFD revision\n| update_pull_request    | Update pull request titles and labels so they align with the RFD content\n| update_search_index    | Update the RFD search index with the new RFD contents\n\n### Content Updates\n\nRFD processing manipulates both internally stored state as well as the source content document of\nthe RFD it is processing. The two cases where the processor will update the contents of a RFD are:\n\n1. a RFD has an incorrect discussion url\n2. a RFD is in an incorrect state\n\nThe first update is the easier of the two. For any RFD that has an open discussion PR, the processor\nwill check that the `discussion` attribute in the RFD document matches the url of the discussion PR.\nNote though that there is a bug here currently related to the order in which revisions may be processed.\n\nState checking is a bit more complex. For a RFD that has an open discussion PR, the processor will\nensure that the RFD state is set to `discussion`. For RFDs that are merged to the default branch\nthough, there is not a good determination as to which of the final states to assign them. Instead\nthe processor will emit a warning when it encounters such a case.\n\n### Update Runners\n\nRFD updates occur via two mechanism. The first of which is in response to GitHub webhook calls for\npushes against the RFD repo. The RFDs that are updated in response to a webhook depend on the branch\nthat was updated and the contents of the commit. RFDs are also updated via a periodic processor so\nthat the processor can account for webhook calls that were either missed, dropped, or failed due to\nsome internal error.\n\n### Webhooks\n\nWebhook calls are accepted by the `rfd-api` server which validates the call and determines the RFDs\nto update. Pushes to the default branch will allow for updates to occur to any RFD number. So if a\ncommit contains an update to RFD 1, RFD 2, and RFD 3, then three update jobs will be scheduled. In\ncontrast to this, if the commit is made against a specific branch (i.e. 0123) then a job will only\nbe scheduled if a change is made to RFD 123.\n\nNote that the `rfd-api` server does not perform RFD updates. It is responsible only for validating\ncalls and scheduling update jobs. Once scheduled, the job will be processed by the `rfd-processor`.\n\n### Scanner\n\nThe scanner can be run at a configurable interval which is largely dependent on the size of the RFD\nrepo itself, and GitHub rate limits. Currently we run the scanner on a 15 minute interval.\n\n## Authentication\n\n### Accounts and Providers\n\n```\n              ┌───────────────────┐\n              │                   │\n              │ api_user_provider │\n              ├───────────────────┤\n              │ id                │\n┌────────┐    ├───────────────────┤\n│ Google ├────┤ remote_id         │   ┌─────────────────┐\n└────────┘    ├───────────────────┤   │                 │\n              │ api_user          ├─┐ │ api_user        │\n              └───────────────────┘ │ ├─────────────────┤\n                                    ├─┤ id              │\n              ┌───────────────────┐ │ └─────────────────┘\n              │                   │ │\n              │ api_user_provider │ │\n              ├───────────────────┤ │\n              │ id                │ │\n┌────────┐    ├───────────────────┤ │\n│ GitHub ├────┤ remote_id         │ │\n└────────┘    ├───────────────────┤ │\n              │ api_user          ├─┘\n              └───────────────────┘\n```\n\n### Account Provider Linking\n\n(Note: not yet implemented)\n\nThe RFD API does not perform any kind of automatic account linking. Every new remote provider id that\nis seen results in a new account being generated. This is problematic though if you want to be able to\nlog in to your account via multiple remote accounts. To support this, providers can be moved between\naccounts.\n\nTransfers are performed manually and require access to both the source account that currently owns the\nprovider and the target account. To initiate a transfer, the source account will make a call to\n`/api-user-provider/{identifier}/link-token` where `identifier` is the id of the provider to transfer.\nThis endpoint will return a token that can be used to move the provider to a new account.\n\nThe target account then needs to call `/api-user/{identifier}/link` with the generated token to link\nthe provider to their account, where `identifier` is the id of the account to link the provider to.\nNote that only the owner of an account can link new providers to it. As such it is only valid to call\nthis endpoint with an `identifier == caller_identifier`.\n\n### OAuth2\n\nUsers can authenticate to the RFD API via OAuth2. The RFD APIs OAuth implementation is backed by remote\nproviders. Currently two providers are supported (both of which are OAuth2 providers themselves): GitHub\nand Google. Authenticating against the RFD API will return an access token that is valid for TBD.\nRefresh tokens are not supported by the RFD API. For application contexts where longer term access is\nrequired, fine-grained API tokens should be used instead.\n\n### OAuth2 Scopes\n\n|Scope               | Description                                      |\n|--------------------|--------------------------------------------------|\n| user:info:r        | Read information about users and their providers |\n| user:info:w        | Update information about users                   |\n| user:provider:w    | Update user providers                            |\n| user:token:r       | Read API token information for users             |\n| user:token:w       | Create API tokens for users                      |\n| group:r            | Read group information                           |\n| group:w            | Create, update, and delete groups                |\n| group:membership:w | Add and remove users from groups                 |\n| rfd:content:r      | List and fetch RFDs                              |\n| rfd:discussion:r   | Fetch RFD discussions                            |\n| search             | Search for RFDs                                  |\n| oauth:client:r     | List OAuth clients                               |\n| oauth:client:w     | Create and update OAuth clients                  |\n\n### OAuth2 Authorization Code\n\nThe RFD API supports the OAuth2 authorization code flow via two remote providers: GitHub and Google.\nA caller can choose which remote provider to send a user to by using the corresponding endpoint:\n\n`/login/oauth/github/code/authorize` - Authenticate with GitHub\n\n`/login/oauth/google/code/authorize` - Authenticate with Google\n\n### OAuth2 Device Code\n\nTo an OAuth2 client, the device flow appears as a spec compliant device flow. Internally the RFD API\ndefers to device and user code creation and validation to the remote provider. It then exposes a\ncustom token endpoint that proxies requests from the client to the appropriate remote provider.\nOnce a remote provider responds successfully with an access token, the RFD API will perform its\ninternal account lookup / creation logic to find a matching RFD API user account. The remote access\ntoken is then thrown away and an access token for the RFD API is returned.\n\nAs with the authorization code flow, the RFD API does not provide refresh tokens.\n\nExample of device flow with the Google provider\n\n```\nBrowser                Client                    RFD API                        Google\n   │                    │                           │                             │\n   │                    │    Request oauth config   │                             │\n   │                    ├──────────────────────────►│                             │\n   │                    │◄──────────────────────────┤                             │\n   │                    │     Return with custom    │                             │\n   │                    │       token endpoint      │                             │\n   │                    │                           │                             │\n   │                    │    Device authz request   │                             │\n   │                    ├───────────────────────────┼────────────────────────────►│\n   │  Authenticate with │◄──────────────────────────┼─────────────────────────────┤\n   │  Google and enter  │    Return device_code,    │                             │\n   │      user_code     │      user_code, etc       │                             │\n   │◄───────────────────┤                           │                             │\n   │                    │                           │                             │\n   │                    │    Poll token endpoint    │                             │\n   │                    ├──────────────────────────►│                             │\n   │                    │        device_code        │      Proxied token call     │\n   │                    │             .             ├────────────────────────────►│\n   │                    │             .             │◄────────────────────────────┤\n   │                    │             .             │     Return access token     │\n   │                    │◄──────────────────────────┤                             │\n   │                    │     Failure response:     │                             │\n   │                    │     Authn not complete    │                             │\n   ├───────────────────►│                           │                             │\n   │   Complete authn   │    Poll token endpoint    │                             │\n   │                    ├──────────────────────────►│                             │\n   │                    │        device_code        │      Proxied token call     │\n   │                    │                           ├────────────────────────────►│\n   │                    │                           │◄────────────────────────────┤\n   │                    │                           │     Return access token     │\n   │                    │◄──────────────────────────┤                             │\n   │                    │    Use access token to    │                             │\n   │                    │    fetch user info and    │                             │\n   │                    │    perform authn based    │                             │\n   │                    │    on remote user id      │                             │\n   │                    │    into the RFD API.      │                             │\n   │                    │    Return RFD API token   │                             │\n   │                    │                           │                             │\n   │                    │                           │                             │\n```\n\n## Authorization\n\n### Permissions\n\nPermissions can be assigned to both users and groups (see below). Permissions are always additive,\nand a callers full permissions are the combined set of their directly assigned permissions and their\ngroup permissions.\n\n[Api Permissions](rfd-api/src/permissions.rs)\n\n### Groups\n\nGroups are a way to manage sets of permissions and assigned them to one or more users. Permissions\nfrom multiple groups are always additive. Users can be assigned to any number of groups, and group\nassignments are stored on user records. Sub-groups are not supported.\n\n```\n                  ┌──────────────┐\n                  │              │\n                  │ access_group │\n                  ├──────────────┤\n┌─────────────┐ ┌─┤ id           │\n│             │ │ ├──────────────┤\n│ api_user    │ │ │ permissions  │\n├─────────────┤ │ └──────────────┘\n│ permissions │ │\n├─────────────┤ │ ┌──────────────┐\n│ groups      ├─┤ │              │\n└─────────────┘ │ │ access_group │\n                │ ├──────────────┤\n                └─┤ id           │\n                  ├──────────────┤\n                  │ permissions  │\n                  └──────────────┘\n```\n\n### Mappers\n\nBy default, new accounts do not have any permissions. The only thing they can do is login. Mappers\ncan be used to assign default permissions to accounts immediately upon login. Mappers apply to both\nexisting and new accounts. Mappers are currently only additive. They can assign permissions and\ngroups, but they can not remove them. Mappers that remove assignments may be supported in the future.\n\nA mapper contains a `condition` and a set of values to apply. The `condition` is tested against the\nuser information returned from a remote provider, if it returns true then the values are applied to\nthe user account associated with the user provider.\n\nNotably this means that mappers explicitly only run when a user authenticates via a remote provider.\n\nA `mappers.toml` file can be used to configure mappers that should be installed during startup of\nthe RFD API.\n\n#### Supported Mappers\n\n**Email Address** - Maps from a fully specified email address to a list of permissions and/or list\nof groups. This mapper can be used with GitHub or Google.\n\n```toml\n[[mappers]]\nname = \"Initial admin\"\nrule = \"email_address\"\nemail = \"user@domain.com\"\ngroups = [\n  \"admin\"\n]\n```\n\n```bash\ncargo run -p rfd-cli mapper create --json-body /dev/stdin \u003c\u003cEOM\n{\n  \"name\": \"add_email_address\",\n  \"max_activations\": 1,\n  \"rule\": {\n    \"rule\": \"email_address\",\n    \"email\": \"user@domain.com\",\n    \"groups\": [\n      \"admin\"\n    ]\n  }\n}\nEOM\n```\n\n**Email Domain** - Maps from a email domain to a list of permissions and/or list of groups. This\nmapper can be\nused with GitHub or Google.\n\n```toml\n[[mappers]]\nname = \"Employees\"\nrule = \"email_domain\"\ndomain = \"domain.com\"\ngroups = [\n  \"company-employee\"\n]\n```\n\n```bash\ncargo run -p rfd-cli mapper create --json-body /dev/stdin \u003c\u003cEOM\n{\n  \"name\": \"add_email_domain\",\n  \"max_activations\": 5,\n  \"rule\": {\n    \"rule\": \"email_domain\",\n    \"domain\": \"domain.com\",\n    \"groups\": [\n      \"company-employee\"\n    ]\n  }\n}\nEOM\n```\n\n**GitHub Username** - Maps from a GitHub username to a list of permissions and/or list of groups.\nAs expected, this mapper can only succeed with a GitHub provider.\n\n```toml\n[[mappers]]\nname = \"Friend\"\nrule = \"github_username\"\ndomain = \"githubuser\"\ngroups = [\n  \"friend-of-company\"\n]\n```\n\n```bash\ncargo run -p rfd-cli mapper create --json-body /dev/stdin \u003c\u003cEOM\n{\n  \"name\": \"add_github_user\",\n  \"max_activations\": 1,\n  \"rule\": {\n    \"rule\": \"github_username\",\n    \"github_username\": \"githubuser\",\n    \"groups\": [\n      \"friend-of-company\"\n    ]\n  }\n}\nEOM\n```\n\n## Contributing\n\nThis repo is public because others are interested in the RFD process and the tooling we've\nbuilt around it. In its present state, it's the code we're using as the backend to\n[our RFD frontend](https://rfd.shared.oxide.computer/). We're open to PRs that\nimprove these services, especially if they make the repo easier for others to use and contribute\nto. However, we are a small company, and the primary goal of this repo is as an internal\ntool for Oxide, so we can't guarantee that PRs will be integrated.\n\n## License\n\nUnless otherwise noted, all components are licensed under the\n[Mozilla Public License Version 2.0](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxidecomputer%2Frfd-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxidecomputer%2Frfd-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxidecomputer%2Frfd-api/lists"}