{"id":18684994,"url":"https://github.com/h1alexbel/srdataset","last_synced_at":"2025-04-12T04:32:49.360Z","repository":{"id":241357405,"uuid":"805286429","full_name":"h1alexbel/srdataset","owner":"h1alexbel","description":"GitHub repositories dataset that contains sample repositories (SRs), with their metrics and metadata","archived":false,"fork":false,"pushed_at":"2024-09-11T19:14:20.000Z","size":194,"stargazers_count":4,"open_issues_count":19,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-09-13T01:13:08.718Z","etag":null,"topics":["docker","github-repositories","research-dataset","shell"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/h1alexbel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-24T09:02:37.000Z","updated_at":"2024-06-28T08:59:15.000Z","dependencies_parsed_at":"2024-05-28T00:32:47.086Z","dependency_job_id":"87755782-20e2-421d-8d24-86b2bda73ff5","html_url":"https://github.com/h1alexbel/srdataset","commit_stats":null,"previous_names":["h1alexbel/srdataset"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/h1alexbel%2Fsrdataset","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/h1alexbel%2Fsrdataset/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/h1alexbel%2Fsrdataset/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/h1alexbel%2Fsrdataset/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/h1alexbel","download_url":"https://codeload.github.com/h1alexbel/srdataset/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223495116,"owners_count":17154651,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","github-repositories","research-dataset","shell"],"created_at":"2024-11-07T10:19:58.670Z","updated_at":"2024-11-07T10:19:58.741Z","avatar_url":"https://github.com/h1alexbel.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# srdataset\n\n**Attention!** This repository is out of date. Repository and research moved to [h1alexbel/sr-detection](github.com/h1alexbel/sr-detection).\n\n[![py](https://github.com/h1alexbel/srdataset/actions/workflows/make.yml/badge.svg)](https://github.com/h1alexbel/srdataset/actions/workflows/make.yml)\n[![PDD status](http://www.0pdd.com/svg?name=h1alexbel/srdataset)](http://www.0pdd.com/p?name=h1alexbel/srdataset)\n[![Hits-of-Code](https://hitsofcode.com/github/h1alexbel/srdataset)](https://hitsofcode.com/view/github/h1alexbel/srdataset)\n[![License](https://img.shields.io/badge/license-MIT-green.svg)](https://github.com/h1alexbel/srdataset/blob/master/LICENSE.txt)\n\nSRdataset is an [unlabeled] dataset of GitHub repositories containing SRs\n(sample repositories).\n\n**Motivation**. During work on models for [samples-filter] project, we\ndiscovered the need for the automation of the dataset building process\non remote servers, since we need to collect automatically a number of GitHub\nrepositories of data being productive in our research. In order to do this,\nwe integrated [ghminer] with a few scripts, and packaged all of that as Docker\ncontainer.\n\n## How to use\n\nTo build a new version of dataset run this:\n\n```bash\ndocker run --detach --name=srdataset --rm --volume \"$(pwd):/srdataset\" \\\n  -e \"CSV=repos\" \\\n  -e \"SEARCH_QUERY=\u003cquery\u003e\" \\\n  -e \"START_DATE=2019-01-01\" \\\n  -e \"END_DATE=2024-05-01\" \\\n  -e \"HF_TOKEN=xxx\" \\\n  -e \"INFERENCE_CHECKPOINT=sentence-transformers/all-MiniLM-L6-v2\" \\\n  -e \"PATS=pats.txt\" \\\n  --oom-kill-disable \\\n  abialiauski/srdataset:0.0.1\n```\n\nWhere `\u003cquery\u003e` is the [search query] to the GitHub API,\n`2019-01-01` is a start date to search the repositories those were created at\nthis date, `2024-05-01` is an end to search the repositories those were created\nat this date, `xxx` is [HuggingFace token], required for accessing\n[inference endpoint][HuggingFace Inference] in order to generate textual\nembeddings; `pats.txt` is file contains a number of [GitHub PATs].\n\nThe building process can take a while. After it completed, you should have\nthese files:\n\n* `results.csv` with all collected repositories.\n* `repos.csv` with all preprocessed and filtered repositories.\n* `texts.csv` repository textual metadata used for generating embeddings.\n* `text-embeddings.csv` with ready-to-cluster repositories with textual vectors\nonly.\n* `similar.csv` with input textual examples and their top-5 most similar\nanalogues from generated embeddings.\n* `numerical.csv` with ready-to-cluster repositories with numerical data only.\n* `mix.csv` with ready-to-cluster repositories that contain both: numerical and\ntextual vectors.\n\nIf you run container with `-e PUSH_TO_HF=true`, you should expect that after\npreprocess, we will push output CSV files to the files to the profile, passed\nto `-e HF_PROFILE`, within provided `HF_TOKEN`. All outputs will be pushed into\ndatasets with `sr-` prefix.\n\nIf you run container with `-e \"CLUSTER=true\"`, you should have one ZIP file\nnamed like `clusters-2024-06-21-18:22.zip` and containing these files:\n\n```text\nagglomerative\n  /mix\n    /members\n    ...\n  /numerical\n    /members\n    ...\n  /textual\n    /members\n    ...\ndbscan/... (the same structure)\ngmm/...\nkmeans/...\nsource/...\n```\n\nAll experiments are grouped by model name: `kmeans`, `dbscan`, `agglomerative`,\netc. In each that model directory you should have `members` directory and a set\nof plots. `members` contains a set of text files tagged with output cluster\nlabel e.g. `0.txt`. In `source` you should have all CSV files that were used to\ngenerate clusters.\n\n## How to contribute\n\nFork repository, make changes, send us a [pull request](https://www.yegor256.com/2014/04/15/github-guidelines.html).\nWe will review your changes and apply them to the `master` branch shortly,\nprovided they don't violate our quality standards. To avoid frustration,\nbefore sending us your pull request please run full make build:\n\n```bash\nmake env test\n```\n\n[unlabeled]: https://en.wikipedia.org/wiki/Unsupervised_learning\n[samples-filter]: https://github.com/h1alexbel/samples-filter\n[ghminer]: https://github.com/h1alexbel/ghminer\n[GitHub PAT]: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens\n[GitHub PATs]: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens\n[HuggingFace token]: https://huggingface.co/docs/hub/en/security-tokens\n[HuggingFace Inference]: https://huggingface.co/inference-endpoints/dedicated\n[search query]: https://docs.github.com/en/search-github/searching-on-github/searching-for-repositories\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fh1alexbel%2Fsrdataset","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fh1alexbel%2Fsrdataset","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fh1alexbel%2Fsrdataset/lists"}