{"id":18684978,"url":"https://github.com/h1alexbel/ghminer","last_synced_at":"2025-04-12T04:32:43.554Z","repository":{"id":240229817,"uuid":"801971222","full_name":"h1alexbel/ghminer","owner":"h1alexbel","description":"Command-line GitHub repository miner that aggregates dataset for your researches","archived":false,"fork":false,"pushed_at":"2025-03-22T15:03:04.000Z","size":242,"stargazers_count":6,"open_issues_count":7,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-22T16:19:36.562Z","etag":null,"topics":["github-repository-search","javascript","research-dataset"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/h1alexbel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-17T09:08:50.000Z","updated_at":"2025-03-22T15:02:46.000Z","dependencies_parsed_at":"2024-05-22T09:30:39.020Z","dependency_job_id":"21c7f1b0-73e7-4a6a-9e48-45a293a3eee5","html_url":"https://github.com/h1alexbel/ghminer","commit_stats":null,"previous_names":["h1alexbel/ghminer"],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/h1alexbel%2Fghminer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/h1alexbel%2Fghminer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/h1alexbel%2Fghminer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/h1alexbel%2Fghminer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/h1alexbel","download_url":"https://codeload.github.com/h1alexbel/ghminer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248517346,"owners_count":21117435,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["github-repository-search","javascript","research-dataset"],"created_at":"2024-11-07T10:19:55.997Z","updated_at":"2025-04-12T04:32:38.540Z","avatar_url":"https://github.com/h1alexbel.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ghminer\n\n[![EO principles respected here](https://www.elegantobjects.org/badge.svg)](https://www.elegantobjects.org)\n[![DevOps By Rultor.com](http://www.rultor.com/b/h1alexbel/samples-filter)](http://www.rultor.com/p/h1alexbel/samples-filter)\n[![We recommend IntelliJ IDEA](https://www.elegantobjects.org/intellij-idea.svg)](https://www.jetbrains.com/idea/)\n\n[![test](https://github.com/h1alexbel/ghminer/actions/workflows/test.yml/badge.svg)](https://github.com/h1alexbel/ghminer/actions/workflows/test.yml)\n![NPM Version](https://img.shields.io/npm/v/ghminer)\n[![codecov](https://codecov.io/gh/h1alexbel/ghminer/graph/badge.svg?token=RraKKKENlR)](https://codecov.io/gh/h1alexbel/ghminer)\n[![PDD status](http://www.0pdd.com/svg?name=h1alexbel/ghminer)](http://www.0pdd.com/p?name=h1alexbel/ghminer)\n[![Hits-of-Code](https://hitsofcode.com/github/h1alexbel/ghminer)](https://hitsofcode.com/view/github/h1alexbel/ghminer)\n[![License](https://img.shields.io/badge/license-MIT-green.svg)](https://github.com/h1alexbel/ghminer/blob/master/LICENSE.txt)\n\nghminer is a command-line dataset miner, that aggregates set of public GitHub\nrepositories from [GitHub GraphQL API] and flushes the result into CSV and JSON\nfiles. This tool is based on [ksegla/GitHubMiner] prototype.\n\nRead [this][blogpost] blog post about `ghminer`, as a dataset miner from GitHub\nto your researches.\n\n**Motivation**. For our researches we require reasonably big datasets in order\nto properly analyze GitHub repositories and their metrics. To do so, we need\naggregate them somehow. Default [GitHub Search API] does not help much, since\nit has [limitation] of 1000 repositories per query. Our tool uses GitHub\nGraphQL API instead, and can offer to utilize multiple [GitHub PATs]\nin order to automate the build of the such huge dataset and increase research\nproductivity.\n\n## How to use\n\nFirst, install it from [npm](https://www.npmjs.com/package/ghminer) like that:\n\n```bash\nnpm install -g ghminer\n```\n\nthen, execute:\n\n```bash\nghminer --query \"stars:2..100\" --start \"2005-01-01\" --end \"2024-01-01\" --tokens pats.txt\n```\n\nAlso, you should have these files: `ghminer.graphql` for GraphQL query, and\n`ghminer.json` for parsing the response from GitHub API. In GraphQL query, you\ncan have all [GitHub supported fields][Gh Explorer] you want. However, to keep\nthis query running to collect all possible repositories, ghminer requires you to\nhave the following structure:\n\n* `search` with `$searchQuery`, `$first`, `$after` attributes.\n* `pageInfo` with `endCursor`, `hasNextPage` attributes.\n* `repositoryCount` field.\n\nHere is an example:\n\n```graphql\nquery ($searchQuery: String!, $first: Int, $after: String) {\n    search(query: $searchQuery, type: REPOSITORY, first: $first, after: $after) {\n        repositoryCount\n        nodes {\n            ... on Repository {\n                nameWithOwner\n                defaultBranchRef {\n                    name\n                }\n                licenseInfo {\n                    spdxId\n                }\n            }\n        }\n        pageInfo {\n            endCursor\n            hasNextPage\n        }\n    }\n}\n```\n\nand `ghminer.json`:\n\n```json\n{\n  \"repo\": \"nameWithOwner\",\n  \"branch\": \"defaultBranchRef.name\",\n  \"license\": \"licence.spdxId\"\n}\n```\n\nAfter it will be done, you should have `result.csv` file with all GitHub\nrepositories those were created in the provided date range.\n\n### Bigger example\n\nConsider this as more complicated example, demonstrating how to fetch various\nfields from GitHub repository:\n\n`ghminer.graphql`:\n\n```graphql\nquery ($searchQuery: String!, $first: Int, $after: String) {\n    search(query: $searchQuery, type: REPOSITORY, first: $first, after: $after) {\n        repositoryCount\n        nodes {\n            ... on Repository {\n                nameWithOwner\n                description\n                defaultBranchRef {\n                    name\n                }\n                defaultBranchRef {\n                    name\n                    target {\n                        repository {\n                            object(expression: \"HEAD:README.md\") {\n                                ... on Blob {\n                                    text\n                                }\n                            }\n                        }\n                        ... on Commit {\n                            history(first: 1) {\n                                totalCount\n                                edges {\n                                    node {\n                                        committedDate\n                                    }\n                                }\n                            }\n                        }\n                    }\n                }\n                repositoryTopics(first: 10) {\n                    edges {\n                        node {\n                            topic {\n                                name\n                            }\n                        }\n                    }\n                }\n                issues(states: [OPEN]) {\n                    totalCount\n                }\n                pullRequests {\n                    totalCount\n                }\n                object(expression: \"HEAD:.github/workflows/\") {\n                    ... on Tree {\n                        entries {\n                            name\n                            object {\n                                ... on Blob {\n                                    byteSize\n                                }\n                            }\n                        }\n                    }\n                }\n            }\n        }\n        pageInfo {\n            endCursor\n            hasNextPage\n        }\n    }\n}\n```\n\n`ghminer.json`:\n\n```json\n{\n  \"repo\": \"nameWithOwner\",\n  \"description\": \"description\",\n  \"branch\": \"defaultBranchRef.name\",\n  \"readme\": \"defaultBranchRef.target.repository.object.text\",\n  \"topics\": \"repositoryTopics.edges[].node.topic.name\",\n  \"issues\": \"issues.totalCount\",\n  \"pulls\": \"pullRequests.totalCount\",\n  \"commits\": \"defaultBranchRef.target.history.totalCount\",\n  \"lastCommitDate\": \"defaultBranchRef.target.history.edges[0].node.committedDate\",\n  \"workflows\": \"object.entries.length\"\n}\n```\n\nAlso, check [this repo][sr-detection], where ghminer is used to collect\nJava repositories from GitHub for research experiment.\n\n## CLI Options\n\n| Option        | Required |                                                                                                        Description                                                                                                         |\n|---------------|----------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|\n| `--query`     | ✅        |                                                                                                 [GitHub Search API query]                                                                                                  |\n| `--graphql`   | ✅        |                                                                              Path to GitHub API GraphQL query, default is `ghminer.graphql`.                                                                               |\n| `--schema`    | ✅        |                                                                                     Path to parsing schema, default is `ghminer.json`.                                                                                     |\n| `--start`     | ✅        |                                                                       The start date to search the repositories, in [ISO] format; e.g. `2024-01-01`.                                                                       |\n| `--end`       | ✅        |                                                                        The end date to search the repositories, in [ISO] format; e.g. `2024-01-01`.                                                                        |\n| `--tokens`    | ✅        | Text file name that contains a number of [GitHub PATs]. Those will be used in order to pass GitHub API rate limits. Add as many tokens as needed, considering the amount of data (they should be separated by line break). |\n| `--date`      | ❌        |                                               The type of the date field to search on, you can choose from `created`, `updated` and `pushed`, the default one is `created`.                                                |\n| `--batchsize` | ❌        |                                                                        Request batch-size value in the range `10..100`. The default value is `10`.                                                                         |\n| `--filename`  | ❌        |                                                                The name of the file for the found repos (CSV and JSON files). The default one is `result`.                                                                 |\n| `--json`      | ❌        |                                                                                             Save found repos as JSON file too.                                                                                             |\n\n## How to contribute\n\nFork repository, make changes, send us a [pull request](https://www.yegor256.com/2014/04/15/github-guidelines.html).\nWe will review your changes and apply them to the `master` branch shortly,\nprovided they don't violate our quality standards. To avoid frustration,\nbefore sending us your pull request please run full npm build:\n\n```bash\nnpm test\n```\n\nYou will need [Node 20+] installed.\n\n[ksegla/GitHubMiner]: https://github.com/ksegla/GitHubMiner\n[GitHub Search API]: https://api.github.com\n[GitHub Search API query]: https://docs.github.com/en/search-github/searching-on-github/searching-for-repositories\n[ISO]: https://en.wikipedia.org/wiki/ISO_8601\n[GitHub GraphQL API]: https://api.github.com/graphql\n[GitHub PAts]: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens\n[limitation]: https://stackoverflow.com/questions/37602893/github-search-limit-results\n[Node 20+]: https://nodejs.org/en/download/package-manager\n[blogpost]: https://h1alexbel.github.io/2024/05/24/ghminer.html\n[Gh Explorer]: https://docs.github.com/en/graphql/overview/explorer\n[sr-detection]: https://github.com/h1alexbel/sr-detection\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fh1alexbel%2Fghminer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fh1alexbel%2Fghminer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fh1alexbel%2Fghminer/lists"}