{"id":13630395,"url":"https://github.com/zalando-incubator/roadblock","last_synced_at":"2025-04-17T13:32:19.300Z","repository":{"id":57170268,"uuid":"134678777","full_name":"zalando-incubator/roadblock","owner":"zalando-incubator","description":"A node.js application for pulling github organisation statistics into a database.","archived":true,"fork":false,"pushed_at":"2019-09-27T14:24:39.000Z","size":201,"stargazers_count":16,"open_issues_count":4,"forks_count":5,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-17T21:33:14.984Z","etag":null,"topics":["github-api","monitoring","nodejs","sequelizejs"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zalando-incubator.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null}},"created_at":"2018-05-24T07:34:47.000Z","updated_at":"2023-12-05T15:04:08.000Z","dependencies_parsed_at":"2022-08-27T12:01:55.366Z","dependency_job_id":null,"html_url":"https://github.com/zalando-incubator/roadblock","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zalando-incubator%2Froadblock","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zalando-incubator%2Froadblock/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zalando-incubator%2Froadblock/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zalando-incubator%2Froadblock/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zalando-incubator","download_url":"https://codeload.github.com/zalando-incubator/roadblock/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249344845,"owners_count":21254748,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["github-api","monitoring","nodejs","sequelizejs"],"created_at":"2024-08-01T22:01:41.184Z","updated_at":"2025-04-17T13:32:19.034Z","avatar_url":"https://github.com/zalando-incubator.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"# Roadblock\n\nA Node.js application for collecting Github statistics into a _SQLite or PostgreSQL_ database.\n\nThis project was built with simplicity and ease of use in mind. We simply wanted GitHub data in a relational database which we could then create visualisations for using Metabase (https://www.metabase.com/).\n\n\n## Installing and using\nInstall from npm as a global command\n```\n  \u003e npm i @zalando/roadblock -g\n```\n\nRun the roadblock command in an empty folder\n```\n  \u003e roadblock\n```\n\nThis will generate a basic `roadblock.json` file which you can then modify:\n\n```\n{\n    \"github\": {\n        \"token\": \"xxx\",\n        \"url\" : \"https://internal.gith.ub/api/v3\" \n    },\n    \"tasks\": [\n        \"pre/*\", \"org/*\", \"repo/releases\"\n    ],\n    \"orgs\": [\n        \"My-Org\", \"Second-org\"\n    ]\n}\n```\n\n**Github.token** a [github token](https://github.com/settings/tokens/new) is required to access most data - ensure that the token have read access to `repo`, `repo:status`, `public_repo`, `read:org`, `read:user`, `read:discussion`.\n\n**url** is only required if you want to collect data from Github Enterprise.\n\n**Tasks** Specify what data you want to collect, by default it is set to * which means\nrun all possible tasks.\nEither use wildcards like `*` or `org/*` or set to \nspecific tasks like `repo/issues` or ignore specific tasks with `!repo/profiles`\n\n**Orgs** By default roadblock will attempt to collect from all orgs, which the token \nhave access to, to filter or to query additional orgs, set them here.\n\nUse either: `*` `orgname` or `!orgname`.\n\n\n#### Configuration as arguments\nConfiguration values can also be passed from the command line to avoid storing tokens in \nclear text:\n\n```\n\u003e roadblock github.token=YOURTOKEN\n\n\u003e roadblock orgs=[\"zalando\",\"custom\"]\n```\n\n### Output\nScript will run between 10 and 20 minutes and store collected data in a SQLite datase - you can also configure a postgres instance if needed.\n\nSqlite Database and json summaries will be stored in the folder where the `roadblock` is invoked.\n\n## Included tasks\nThe task system in roadblock divides the different data collection tasks into 4 seperate phases. \n\n#### [Pre](tasks/pre)\nTasks to collect initial data points, default is to collect configured organisations\n\n- `pre/organisations` - Collects available organisations from the user and configured orgs\n- `pre/calendar` - Creates a calendar table with years and months, helpful when querying data\n\n#### [Org](tasks/org)\nTasks run for each seperate organisation, each task is passed an organisation object to process data based on. \n\n- `org/members` - collect all members of the organisations\n- `org/repository` - collect all public, non-fork repositories on the organisation\n- `org/vulnerabilities` - collect all security alerts from all repositories on the organisation\n  \n#### [Repo](tasks/repo)\nTasks run for each collected repository. \n\n- `repo/collaborators` - Collect all collaborators on a repository\n- `repo/commits` - Collect all repository commits\n- `repo/contributions` - Collect all contributions (summarised changes)\n- `repo/issues` - Collect all issues on the repositories\n- `repo/profiles` - Repository health / community profile\n- `repo/pullrequests` - Repository pull requests\n- `repo/releases` - All releases on repository\n- `repo/topics` - Repository topics       \n\n#### [Post](tasks/post)\nTasks to run after all org and repo data collection is completed\n\n- `post/export` - Export organisation and repository stats to json files\n- `post/upstream` - Collect upstream contribution stats from external repositories.\n\n\n## Using the source\n```\n\u003e Clone this project to your local machine\n\u003e git clone https://github.com/zalando-incubator/roadblock.git\n\u003e cd roadblock\n\n\u003e Run npm install and start collecting data\n\u003e npm install\n\n```\n\n### Pre Requisites\n\nWhat software you need to install:\n\n- Node.js\n- [Metabase](https://www.metabase.com/) (optional) - to visualise the collected data\n\n## Built With\n\n* [Sequelize](http://docs.sequelizejs.com/) - Node.js ORM\n* [GhRequestor](https://github.com/Microsoft/ghrequestor) - Github client for fetching large amounts of data\n\n## Contributing\n\nPlease read [CONTRIBUTING.md](CONTRIBUTING.md) for details on our code of conduct, and the process for submitting pull requests to us.\n\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzalando-incubator%2Froadblock","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzalando-incubator%2Froadblock","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzalando-incubator%2Froadblock/lists"}