{"id":52037840,"url":"https://github.com/adobe/oss-contributors","last_synced_at":"2026-08-01T21:03:36.655Z","repository":{"id":42356620,"uuid":"171754623","full_name":"adobe/oss-contributors","owner":"adobe","description":"How do tech companies rank amongst themselves when it comes to github.com activity?","archived":false,"fork":false,"pushed_at":"2024-04-08T12:48:55.000Z","size":293,"stargazers_count":35,"open_issues_count":14,"forks_count":8,"subscribers_count":3,"default_branch":"master","last_synced_at":"2026-04-21T04:52:45.370Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/adobe.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-02-20T21:49:21.000Z","updated_at":"2025-10-31T13:59:01.000Z","dependencies_parsed_at":"2022-09-14T09:42:39.722Z","dependency_job_id":null,"html_url":"https://github.com/adobe/oss-contributors","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/adobe/oss-contributors","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adobe%2Foss-contributors","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adobe%2Foss-contributors/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adobe%2Foss-contributors/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adobe%2Foss-contributors/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/adobe","download_url":"https://codeload.github.com/adobe/oss-contributors/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adobe%2Foss-contributors/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":36170392,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-08-01T02:00:05.789Z","response_time":100,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-08-01T21:03:35.923Z","updated_at":"2026-08-01T21:03:36.649Z","avatar_url":"https://github.com/adobe.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tracking Open Source Contributors [![Build Status](https://travis-ci.com/adobe/oss-contributors.svg?branch=master)](https://travis-ci.com/adobe/oss-contributors)\n\n\u003e Build a(n improved) ranking of companies-as-contributors-to-public-GitHub (based on [this blog post](https://medium.freecodecamp.org/the-top-contributors-to-github-2017-be98ab854e87)).\n\n## Too Long; Didn't Read\n\nPretty graphs over [here](https://docs.google.com/spreadsheets/d/1EosxNv67tC2IYFY_RFeCoSY3JipiVfYyky9VejKAS9k/edit#gid=566883526).\n\n## Why?\n\nThe user-to-company association in [the ranking blog post that inspired us](https://medium.freecodecamp.org/the-top-contributors-to-github-2017-be98ab854e87)\nis not ideal: it uses the email associated to a `git` config, and if the domain\nto the email is NOT of a public mail provider (gmail, yahoo, etc), it assumes it's\na company. That's not a great way of going about it because not many people use\ntheir company's e-mail in the git config they use with their public GitHub.com account.\n\nTo make that association better, this project cross-reference GitHub.com activity,\nwhich is tracked via [githubarchive.org](http://githubarchive.org) data (and is\n[freely available as a dataset](https://www.githubarchive.org/#bigquery) in\n[Google BigQuery](http://bigquery.cloud.google.com)) with\n[GitHub.com](http://github.com) user profiles. We pull the `company` field\nfrom user's profiles and store those in a periodically-updated (currently\nmonthly) database that we then copy over into BigQuery.\n\n## Features\n\n- Leverages [githubarchive.org](http://githubarchive.org)'s [freely available dataset on Google BigQuery](https://www.githubarchive.org/#bigquery)\n  to track public user activity on GitHub.\n- A GitHub.com REST API crawler that pulls users' company associations (based\n  on their public profile), that we then store in a database (and periodically\n  update).\n- Tracking and visualizing GitHub contributors from tech companies' activity\n  over time in a [spreadsheet](https://docs.google.com/spreadsheets/d/1EosxNv67tC2IYFY_RFeCoSY3JipiVfYyky9VejKAS9k/edit#gid=566883526).\n\n## Implementation\n\nWe have a [BigQuery project](https://bigquery.cloud.google.com/dataset/public-github-adobe)\nwith relevant supporting tables and queries. If you'd like access, contact @filmaj\n(via an issue in this repo or on twitter). This project contains:\n\n1. A database table tracking user-company associations (currently done in an Adobe IT managed MySQL DB).\n   Fields include GitHub username, company field, fingerprint (ETag value as\n   reported from GitHub, as a cache-buster). We synchronize the MySQL DB with\n   BigQuery every now and then using a command this program provides.\n2. Another [table tracks GitHub usernames active over a certain time period](https://bigquery.cloud.google.com/table/public-github-adobe:github_archive_query_views.users_pushes_2017?pli=1).\n    - We have one table, [`users_pushes_2017`](https://bigquery.cloud.google.com/table/public-github-adobe:github_archive_query_views.users_pushes_2017?pli=1),\n      that we used as a baseline. This table tracked all GitHub users that had\n      at least 1 commit in 2017.\n    - We will [make incremental tables containing activity, for each passing month](https://bigquery.cloud.google.com/table/public-github-adobe:github_archive_query_views.users_pushes_2018_01?pli=1),\n      and track how things progress.\n3. For each active user identified in (2), we pound the GitHub REST API to pull\n   user profile info, and drop the `company` field from that info into the DB\n   table described in (1).\n\n### How Are Companies Tracked?\n\nCheck out the [`src/util/companies.js`](src/util/companies.js) file. How it\nworks:\n\n1. There is a \"catch-all\" regular expression (🤡) that tries to match on known\n   tech company names.\n2. If a match is detected, then we try to map that back to a nicer label for a\n   company name. Note that multiple expressions from the company catch-all may\n   map to a single company (e.g. AWS, AMZN and Amazon all map back to Amazon).\n\n## TODO\n\n1. Describe how to use bigquery in conjunction with this repo.\n1. Real-time visualization of the data.\n2. Tests.\n\n## Requirements\n\n- Node.js 9+\n- a BigQuery account, and a `bigquery.json` file is needed in the root of the repo, which contains the\n  credentials for access to Google Cloud BigQuery. More info on how to set this\n  file up is available on [BigQuery\n  docs](https://cloud.google.com/bigquery/docs/authentication/service-account-file).\n- a `oauth.token` file is needed in the root of the repo, which contains GitHub.com\n  personal access tokens, one per line, which we will use to get data from\n  api.github.com. In my version of this file, I have several tokens (thanks to all\n  my nice friends who graciously granted me one) as there is a maximum of 5,000 calls\n  per hour to the GitHub REST API.\n- a MySQL database to store user-company associations. Currently using an Adobe-IT-managed\n  instance: hostname `leopardprdd`, database name, table name and username are all\n  `GHUSERCO`, running on port 3323. @filmaj has the password. The schema for this\n  table is under the [`usercompany.sql`](usercompany.sql) file.\n\n## Doing The Thing\n\n    $ npm install\n    $ npm link\n\nAt this point you should be able to run the CLI and provide it subcommands:\n\n### Updating MySQL DB of User-Company Affiliations\n\nThis command will pull the rows from a bigquery table containing github.com\nusernames, pull user profile information for each user from the GitHub.com REST\nAPI and store the result of the `company` field (and the `ETag`) in a MySQL DB\ntable.\n\n    $ node bin/oss.js update-db \u003cbigquery-table-of-user-activity\u003e\n\nRunning this command and pointing it to a bigquery table containing ~1.5 million\ngithub.com usernames, on last run (Feb 2018), took about 6 days.\n\n### Uploading Results Back to BigQuery\n\nThis command will push the MysQL DB up to BigQuery.\nThis command will delete the table you specify before pushing up the results.\n\n    $ node bin/oss.js db-to-bigquery \u003cbigquery-table-of-user-company-affiliations\u003e\n\nOn last run (Feb 2018), this command took a few minutes to complete.\n\n# Putting It All Together\n\nIf you're still with me here: wow, thanks for sticking it out. How all of this\nfits together:\n\n1. Run the [incremental user activity query on BigQuery](db/githubarchive_incremental_active_users_query.sql), and store the result in a new table. I usually run this on a monthly basis, but you are free to use whatever time interval you wish.\n2. Run this program's [`update-db`\n   command](#updating-mysql-db-of-user-company-affiliations), specifying the\n   bigquery table name you created in (1), to get the latest company\n   affiliations for the users identified in (1) stored in your MySQL DB. This\n   usually takes _days_. You have been warned.\n3. Run this program's [`db-to-bigquery`\n   command](#uploading-results-back-to-bigquery) to send these affiliations up\n   to bigquery. Note that the table you specify to store these affiliations in,\n   if it already exists, will be deleted. This should only take a few minutes.\n4. Run the [contributor-count, repo-count and stars-accrued query on\n   BigQuery](db/githubarchive_stars.sql), and store the result in a new table.\n   This query will look at all github activity over the time period you specify\n   (top of the query) and correlate it with the user-company affiliations table\n   we created in (3). Make sure you use the correct table name for the\n   user-company affiliations in the query (search for `JOIN`). BigQuery is\n   awesome so this should never take more than a minute, though do keep an eye\n   on your bill as, well, money goes fast ;)\n5. Bask in sweet, sweet data.\n\n# Contributing\n\nFirstly, check out our [contribution guidelines](.github/CONTRIBUTING.md).\nSecondly, there are probably way better ways of doing this! For example, I've\nnoticed that the company field info is somewhat available directly in BigQuery,\nso probably the whole \"use a MySQL DB\" thing is dumb. I'm grateful for any help\n🙏.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadobe%2Foss-contributors","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadobe%2Foss-contributors","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadobe%2Foss-contributors/lists"}