{"id":13406747,"url":"https://github.com/microsoft/opensource-management-portal","last_synced_at":"2025-05-15T10:03:42.798Z","repository":{"id":36985908,"uuid":"46088926","full_name":"microsoft/opensource-management-portal","owner":"microsoft","description":"Microsoft's monolithic, opinionated Open Source Management Portal enabling enterprise scale self-service powered by the GitHub API 🏔🧑‍💻🧰","archived":false,"fork":false,"pushed_at":"2025-05-13T01:52:18.000Z","size":11758,"stargazers_count":507,"open_issues_count":29,"forks_count":130,"subscribers_count":34,"default_branch":"main","last_synced_at":"2025-05-15T00:08:30.842Z","etag":null,"topics":["github","github-app","management","microsoft","ospo","portal"],"latest_commit_sha":null,"homepage":"https://jeffwilcox.blog/2019/06/scaling-25k/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/microsoft.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":"SUPPORT.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2015-11-13T00:10:29.000Z","updated_at":"2025-05-05T14:08:02.000Z","dependencies_parsed_at":"2024-03-30T23:24:53.070Z","dependency_job_id":"64a41377-609c-40ad-884d-8e634369485e","html_url":"https://github.com/microsoft/opensource-management-portal","commit_stats":{"total_commits":906,"total_committers":28,"mean_commits":"32.357142857142854","dds":0.6346578366445916,"last_synced_commit":"f2390bc4ef00b8fbdd6a6247f2d6639888a3d5bf"},"previous_names":["microsoft/opensource-portal"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2Fopensource-management-portal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2Fopensource-management-portal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2Fopensource-management-portal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2Fopensource-management-portal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/microsoft","download_url":"https://codeload.github.com/microsoft/opensource-management-portal/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254319716,"owners_count":22051072,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["github","github-app","management","microsoft","ospo","portal"],"created_at":"2024-07-30T19:02:38.206Z","updated_at":"2025-05-15T10:03:37.684Z","avatar_url":"https://github.com/microsoft.png","language":"TypeScript","readme":"# Open Source Management Portal\n\nThis application represents the home for open source engineering experiences\nat Microsoft. As a backend application it manages source of truth for many\ntypes of corporate open source metadata, historical intent of repos\nand projects, hosts a rich front-end, and also a set of APIs used by partner\nteams.\n\nWhile we prefer native GitHub experiences, when it comes to displaying certain info\nand being more transparent about permissions and metadata, especially on\nGitHub, which has no extensible user interface, we end up using and driving\npeople to this Open Source Management Portal to get the information they\nneed.\n\nAt Microsoft, 50,000 engineers are using a version of this portal as part of\ntheir open source engineering experience. However, Microsoft does have a set\nof \"company-specific\" extensions, including a separate React frontend client,\nthat are not currently part of this repository. And... yup, if we were to\nstart over today, we'd probably make this a Next.js-or-similar project.\n\nCore capabilities and features of this application:\n\n- **Linking GitHub accounts ⛓️** for enterprise use\n- **Self-service GitHub organization joining 🙋** for engineers\n- **Creating and managing GitHub open source repositories 👩‍💻**\n- **Displaying transparent information, metrics, and company-specific data** about our GitHub open source presence around permissions, access, metadata, intent, and especially cross-organization views and search indexes\n- **People inventory 👨‍🦳🧑‍🚀🧒🏽** to help people connect GitHub public logins with corporate identities\n- **Intercepting forks and new repositories 🔐** to inject compliance and approval processes\n- **Disable and enable 🔑** experiences for GitHub repositories\n- **Just-in-time (JIT) access 🚪** for open source repositories, teams, and organizations, aligning with the principle of least privilege access\n- **Sudo ⚡️** capabilities for repos, teams, organizations to remove persistent broad ownership and admin permissions\n- **Hosting APIs 🍽️** to create repos, large-scale orgs to access link data, and reports\n- **Background jobs 👷‍♂️** to maintain eventual consistency, run tasks, gather metrics, and prepare OKRs\n- **Team join requests/approvals with context 🚪** building beyond the GitHub experience\n- **Automated offboarding 🛶** when people take on new opportunities\n\nAt Microsoft, additional capabilities include:\n\n- **Pre-release business and legal approvals to release projects 🧑‍⚖️**\n- **Requesting contribution reviews ✍🏾** within policy\n- **Service Tree and Direct Owners inventory 🌳** for showing accountable ownership information for repos when available\n- **Hosting internal docs 📚** at aka.ms/opensource\n- **Hosting a subset of opensource.microsoft.com's APIs 🌍** to bring to life the Microsoft open source presence\n\nThe management portal is designed to be fast, efficient, and get out of the way of engineers\nto get their important work done, with an emphasis on _relentless automation_ and _delegation_.\n\nMost of the experience is eventually consistent; however, operational actions\nsuch as joining teams, orgs, sudo operations, etc., are fully consistent at the time\nthey are requested.\n\n## Implementation Details and More Docs\n\nPlease see the `docs/` sub-folder, including [docs/index.md](docs/index.md).\n\n## API\n\nPlease see the [docs/api.md](docs/api.md) file for information about the current API.\n\n## Application stack for learning\n\nAs a TypeScript/Node.js backend application, with a React frontend, the\nmanagement portal also serves as a learning opportunity for Microsoft's\nengineering systems teams to understand the experience that non-.NET stack\napplications may have. The 1ES+OSPO teams partner to ship the application\nbased on essentially a fork of this open source repo.\n\nAs of 2022, the backend site is hosted by Azure App Service with\nLinux containers, while the background cronjobs and daemons run in\nAzure Kubernetes Service (AKS) clusters. All containers are built on top\nof the CBL Mariner distro.\n\nThe app started as a hackathon project in an ancient JavaScript era full\nof \"callback hell\", and has evolved through to third-party promise libraries\nto native ECMAScript promises and to TypeScript. So it both shows its age,\nand, is, interesting.\n\n### Web app authentication\n\nThe **primary** authentication for the site is **Azure Active Directory** for\ncorporate users.\n\nThe **secondary** authentication is **GitHub**. This allows users not using\nGitHub to fully explore the site, link, and otherwise be productive.\n\n_In theory, open source friends, this project could be made a bit more\nextensible. In the past, we prototyped Google authentication, as an example,\nfor the primary aspect. Contributions welcome!_\n\nAPIs can use either JWTs or an active web app session in some cases, used\nby the React frontend.\n\n### Configuration ⛳️\n\nMany feature flags exist.\n\nPlease see [docs/configuration.md](docs/configuration.md)\n\n### Jobs 💼\n\nPlease see [docs/jobs.md](docs/jobs.md)\n\n## Service dependencies\n\n- GitHub organization(s)\n- Hosting environment\n- Background job environment for eventual consistency work and maintenance cronjobs\n- Daemon hosting for near-real-time process\n- Queue system\n- A cache system or multi-tiered cache implementation\n- Azure Active Directory and the Microsoft Graph\n- An email service to send mail\n- Optional insights or telemetry system\n\n### Source of truth store 🧑‍⚖️\n\nThe backend maintains in a data store of your choice key metadata for\nrepositories, links, and general compliance info. The backend supports\nnatively Azure Storage, Azure Table, Azure CosmosDB, and Postgres.\n\nAt Microsoft we currently use **Postgres** for source of truth including:\n\n- GitHub organization configuration\n- corporate GitHub repository metadata\n- corporate identity-to-GitHub login links\n- compliance metadata (enable/disabled repos)\n\n### Respecting the GitHub API\n\nTo be friendly to GitHub, we strive to be very efficient and fair in\nour use of the GitHub API. We cache as much as we can, and have a\nnative concept of building on top of GitHub's **Conditional Request**\nbest practice for GitHub Apps: whenever possible, we send the `e-tag` for\na request, and we will use our cache for many types of operations.\n\nFor long multi-page GitHub REST API v3 responses, we will maintain a\ncache of those responses and rebuild them slowly in the background,\nas the site is eventually consistent for most views.\n\nFor operational work, a real-time API call is used to continue to be\naccurate and secure when working around granting access or managing\naccess to superuser features.\n\n### Cache\n\nThe primary cache layer is backed by **CosmosDB** documents, in a hybrid\napproach where larger documents fallback to **Azure Storage** (blob). Redis is\nalso supported for open source users of the site.\n\n### Background event processing firehose and cronjobs\n\nThere are at least 2 ongoing single-instance daemonsets and many cronjobs\nthat also keep the site efficient, up-to-date, and gather important info.\n\nThe daemons:\n\n- **Firehouse**: webhook event processing from a queue for eventual consistency and reacting to GitHub events around compliance/audit/scale/management\n- **Just-in-time**: JIT revocations, audit log event gathering, and analysis\n\nExample cronjobs:\n\n- Make sure caches are primed occasionally\n- Remind people to setup or delete repos\n- Automatically delete repos that are not setup in a time window\n- Disabling repos out of compliance\n- Collecting data and metrics for reports and user interface experiences\n- Backing up link data\n- Prepare stats for an OKR\n\n#### About the firehose in detail\n\nWhile the original portal works fine for very small GitHub presences, it\nwas designed around the idea that the cache would fill, while respecting the\nGitHub REST API by using [Conditional Requests](https://docs.github.com/en/rest/overview/resources-in-the-rest-api#conditional-requests), and being very\neventually consistent.\n\nHowever, the REST API v3 (non-GraphQL client) maximum size for a page of results\nis 100 entries, which ... is very painful if you have tens of thousands of anything.\n\nThe \"firehose\" is designed to be run either within the app itself, or as a secondary\napp processing results. At Microsoft, we use a service bus to process webhook events\nfrom GitHub, since we have a robust webhook ingestion mechanism elsewhere. The\nfirehose runs as a daemon that pulls off the queue and works to keep the \"query cache\" primed with newer information than the REST API may have in some cases.\n\nWhat this improves:\n\n- The user views of the orgs, repos, teams they are added to and have access to\n- Cross-organization views and querying\n\nThe firehose and query cache are _not_ used for important or auth-style scenarios:\n\n- Query cache is not used to make permission decisions\n- Query cache is not used to authorize access to administrative functions\n\nWe did at one point design the idea of having a `/webhook` endpoint and validating\nthe webhook signatures before processing hooks for simple app hosting, but it's\nslightly broken right now and disabled at Microsoft.\n\n## Dev prep, build, deploy\n\n### Install Node packages\n\nMake sure to include dev dependencies.\n\nThe default assets package is a _super ancient_ Bootstrap and jQuery app that\nin theory provides basic skin for the site, favicons, graphics, etc. However,\nit's ... really, really, really old. Microsoft discards the default-assets-package,\nusing a different set of assets, so you've been mildly warned.\n\nThe `main` module of the defined default-assets-package should resolve to the\npath to serve content from. Since the default version uses \\[ancient\\] Grunt to build the\nassets, it returns the `__dirname` + `/public`, which is the output/built location for Grunt.\n\n```bash\nnpm install\ncd default-assets-package\nnpm install\n```\n\n### Build\n\n```bash\nnpm run build\n```\n\nYou need to rebuild the default-assets-package if you change something. [see Static Site Assets](docs/staticSiteAssets.md)\n\n### Codespaces instructions\n\nYou will likely want to use a defined environment to save time spinning up many variables, follow one of the below paths:\n\n- GitHub Codespaces account-level secrets for your environment variables as well\n- use a `.env` file up a folder from the cloned repository in your Codespace environment\n- configure environment variables once the devcontainer boots\n- GitHub Codespaces repo-specific secrets\n\nWhether as a secret or in the `../env` from the root, set\n\n- `CONFIGURATION_ENVIRONMENT`: `development` (or similar)\n\nThen, you'll also need to make sure authentication will work when redirecting to the running\nCodespaces environment.\n\n### GitHub authentication\n\nYou'll want to bring your own GitHub App and use its client ID and client secret for\nauthentication. [Configure your account-specific Codespace secrets](https://github.com/settings/codespaces).\n\n- `CODESPACES_GITHUB_AUTHENTICATION_ENABLED`: set to `1` to enable\n- `CODESPACES_GITHUB_CLIENT_ID`: the client ID\n- `CODESPACES_GITHUB_CLIENT_SECRET`: the client secret\n\nConfigure the secrets for your fork and/or this repository as necessary. The redirect URL will\nbe dynamically generated and included in the startup debug output. Make sure that the hostname\nis an appropriate callback URL for the GitHub app.\n\n#### Enterprise Managed Users impersonation/override\n\nSince the underlying repository and the Codespace are likely hosted in GHEC EMU,\nyou will also need to use the debug-time impersonation features to override the EMU\nuser information after a GitHub callback with your GitHub.com account.\n\nFor ease of use, an initial impersonation override feature is available that\nonly will override a GitHub EMU response:\n\n- `CODESPACES_IMPERSONATE_OVERRIDE_EMU_ENABLED`: set to `1` to allow in your environment\n- `CODESPACES_IMPERSONATE_OVERRIDE_EMU_LOGIN`: set to the login to use _only_ when an EMU user authenticates. _The primary impersonation feature will still be used after this._\n\n### AAD authentication\n\nConfigure your AAD application in an appropriate tenant.\n\n- `CODESPACES_AAD_AUTHENTICATION_ENABLED`: set to `1` to enable\n- Set the other AAD variables for your environment as necessary:\n  - `AAD_CLIENT_ID`\n  - `AAD_CLIENT_SECRET`\n  - ...\n\n### Private artifacts\n\nThe Microsoft-internal fork of this project uses a private Azure Artifact feed\nto bring in additional components and libraries. These are not applicable to\nthe open source upstream and should be excluded currently.\n\n### Building the Docker image\n\n```bash\ndocker build -t opensource-management-portal .\n```\n\n#### Run (OSS instructions)\n\n\u003e This section is from the open source community\n\nThe most easy way to run is by using the docker-compose setup. This will bootup the postgres and redis components as well. The docker-compose setup depends on 2 environment files and 1 json file:\n\n- .docker.env\n- .secrets.env\n- env-orgs.json\n\nMake sure to copy the .secrets.env.example and env-orgs.json.example files and provide the configuration values.\n\n```bash\ncp .secrets.env.example .secrets.env\ncp env-orgs.json.example env-orgs.json\n# provide configuration values for .secrets.env and env-orgs.json\ndocker-compose up\n```\n\nIf you desire to run all on your local machine (redis, postgres) you might want to use following approach.\n\n```bash\n# ensure redis and postgres is running on localhost\nsource .secrets.env\nsource .local.env\nnpm run start\n```\n\n#### Troubleshooting\n\nIf the docker image doesn't start you can debug the image using an interactive shell session. This allows\nyou to browse the folders, update the files to test things and run the portal.\n\n```bash\n$ docker run --rm -it --env-file .secrets.env --env-file .docker.env --entrypoint /bin/sh opensource-management-portal\n/usr/src/repos $ ls\napp.js                   data                     lib                      package.json             tsconfig.tsbuildinfo     webhooks\napp.js.map               entities                 localEnvironment.js      routes                   user\nbin                      features                 localEnvironment.js.map  test                     utils.js\nbusiness                 github                   middleware               transitional.js          utils.js.map\nconfig                   jobs                     node_modules             transitional.js.map      views\n/usr/src/repos $ npm run start-in-container\n```\n\n### Test\n\nThis project basically has _very few tests_, and aspirations to start using Jest better. Oops. Bad debt as multiple hackathons combine, along with\nproduction dependencies on GitHub...\n\n### Bare minimum local development environment\n\nIf you place a JSON file `.env` above the directory of your cloned repo\n(to prevent committing secrets to your repo by accident or in your editor),\nyou can configure the following extreme minimum working set to use the app.\n\nThe central operations token is a personal access token that is a **org owner**\nof the GitHub org(s) being managed.\n\n```env\nDEBUG_ALLOW_HTTP=1\nGITHUB_CENTRAL_OPERATIONS_TOKEN=a github token for the app\nGITHUB_ORGANIZATIONS_FILE=../../env-orgs.json\nGITHUB_CLIENT_ID=your client id\nGITHUB_CLIENT_SECRET=your client secret\nGITHUB_CALLBACK_URL=http://localhost:3000/auth/github/callback\nAAD_CLIENT_ID=your corporate app id\nAAD_REDIRECT_URL=http://localhost:3000/auth/azure/callback\nAAD_CLIENT_SECRET=a secret for the corporate app\nAAD_TENANT_ID=your tenant id\nAAD_ISSUER=https://sts.windows.net/your tenant id/\n```\n\nIn this mode memory providers are used, including a mocked Redis client. Note\nthat this does mean that a large GitHub organization configured with memory\nproviders could become a token use nightmare, as each new execution of the app\nwithout a Redis Cache behind the scenes is going to have 100% cache misses for\nGitHub metadata. Consider configuring a development or local Redis server to\nkeep cached data around.\n\n\u003e The built-in Redis mock will likely be removed when we move to the next\n\u003e major semver of the Node Redis library.\n\n## Collaboration\n\nThis project began as a hackathon... so still has growing pains years later.\nSince this is technically a _backend web application_ and includes some\nserver-generated user interface, the project was not originally designed\nto be shared as something that runs out-of-the-box, but... it is possible.\n\nTo collaborate on extensibility and improvements, please sync in the issues\nfirst so we can come up with the best approach.\n\nAgain, since Microsoft strips most of the `routes/` and uses a React frontend\non this app, it's likely `routes/` and the Pug rendering is... old. Very old.\n\nHopefully this **monolith** can at least be an interesting learning\nopportunity in crufty old ancient apps evolving on the JavaScript front!\n\n### Work to be done (OSS project)\n\n- Support more interesting cloud and data providers\n- Support other authentication technologies\n- Any tests\n- More tests\n- Ship the front-end UI to the world as open source\n- Continuing to refactor out Microsoft-specific things when possible\n\n## Project origin\n\nAn introduction to this project is available in a [2015 post by JWilcox](https://jeffwilcox.blog/2015/11/azure-on-github/) and a\n[2019 follow-up post, \"Scaling from 2,000 to 25,000\"](https://jeffwilcox.blog/2019/06/scaling-25k/).\n\nAn Open Source Hub concept was prototyped by a Microsoft subsidiary and\nthe early Open Source Programs Office to make very clear the open source\nexperiences, docs, and guides for Microsoft's culture change to working\nmore in the open, releasing projects, and connecting everything together.\n\nAt the same time, GitHub was very basic, and it was necessary to automate and\nmake self-service the GitHub engineering system to work at an enterprise scale.\nWhen Azure became the first approved organization to use GitHub at Microsoft,\nthis portal scaled access and built guardrails around the GitHub environment.\n\n## LICENSE\n\n[MIT License](LICENSE)\n\n## Contributing\n\nThis project welcomes contributions and suggestions. Most contributions require you to agree to a\nContributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us\nthe rights to use your contribution. For details, visit \u003chttps://cla.opensource.microsoft.com\u003e.\n\nWhen you submit a pull request, a CLA bot will automatically determine whether you need to provide\na CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions\nprovided by the bot. You will only need to do this once across all repos using our CLA.\n\nThis project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\nFor more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or\ncontact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.\n\n## Trademarks\n\nThis project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft\ntrademarks or logos is subject to and must follow\n[Microsoft's Trademark \u0026 Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).\nUse of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.\nAny use of third-party trademarks or logos are subject to those third-party's policies.\n","funding_links":[],"categories":["TypeScript","github","GitHub Management"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrosoft%2Fopensource-management-portal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmicrosoft%2Fopensource-management-portal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrosoft%2Fopensource-management-portal/lists"}