{"id":17796429,"url":"https://github.com/rvantonder/cryptoss","last_synced_at":"2025-03-17T02:31:24.962Z","repository":{"id":141621719,"uuid":"169338876","full_name":"rvantonder/CryptOSS","owner":"rvantonder","description":"Collect and view OSS cryptocurrency development.","archived":false,"fork":false,"pushed_at":"2019-07-04T08:41:56.000Z","size":73153,"stargazers_count":7,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-15T00:42:37.635Z","etag":null,"topics":["cryptocurrency","github","mining-software-repositories","open-source-software","software-metrics"],"latest_commit_sha":null,"homepage":"https://rvantonder.github.io/CryptOSS","language":"OCaml","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rvantonder.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-02-06T01:16:04.000Z","updated_at":"2024-11-05T00:21:04.000Z","dependencies_parsed_at":"2024-01-22T19:33:04.505Z","dependency_job_id":"b851e890-95d1-4e9a-865a-c45ab8576bf6","html_url":"https://github.com/rvantonder/CryptOSS","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rvantonder%2FCryptOSS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rvantonder%2FCryptOSS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rvantonder%2FCryptOSS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rvantonder%2FCryptOSS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rvantonder","download_url":"https://codeload.github.com/rvantonder/CryptOSS/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243837011,"owners_count":20355813,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cryptocurrency","github","mining-software-repositories","open-source-software","software-metrics"],"created_at":"2024-10-27T11:45:42.026Z","updated_at":"2025-03-17T02:31:24.139Z","avatar_url":"https://github.com/rvantonder.png","language":"OCaml","readme":"# CryptOSS\n\nThis repository contains tooling for collecting and viewing Cryptocurrency Open Source Software (OSS) development.\n\n\u003cdetails\u003e\n  \u003csummary\u003eClick to expand Papers and Citations related to this project.\u003c/summary\u003e\n  \n```\n@inproceedings{trockman-striking-gold-2019, \n  title = {{Striking Gold in Software Repositories? An Econometric Study of Cryptocurrencies on GitHub}},\n  booktitle = \"International Conference on Mining Software Repositories\", author = \"Trockman, Asher and {van~Tonder}, Rijnard and Vasilescu, Bogdan\",\n  series = {MSR '19},\n  year = 2019\n}\n```\n[Paper Link](https://cmustrudel.github.io/papers/msr19crypto.pdf)\n\n```\n@inproceedings{van-tonder-crypto-oss-2019, \n  title = {{A Panel Data Set of Cryptocurrency Development Activity on GitHub}},\n  booktitle = \"International Conference on Mining Software Repositories\",\n  author = \"{van~Tonder}, Rijnard and Trockman, Asher and {Le~Goues}, Claire\",\n  series = {MSR '19},\n  year = 2019\n} \n```\n[Paper Link](https://www.cs.cmu.edu/~rvantond/pdfs/crypto-oss-msr-data-2019.pdf)\n\u003c/details\u003e\n\n## CSV and raw data\n\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.2595588.svg)](https://doi.org/10.5281/zenodo.2595588)\n\n[View the GitHub data online](https://rvantonder.github.io/CryptOSS). This does not include the full data available in the CSV above, which includes cryptocurrency prices, market cap, and trading volumes.\n\n## Building the tooling\n\n- Install [opam](https://opam.ocaml.org/doc/Install.html). Typically:\n\n```\nsh \u003c(curl -sL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)\n```\n\n- Then run \n```\nopam init\nopam switch create 4.05.0 4.05.0 \n```\n\n- Next:\n\n```\nopam install core opium yojson hmap tyxml tls\n```\n\nThen:\n\n```\nopam pin add github https://github.com/rvantonder/ocaml-github.git \n```\n\nThen type `make` in this repository. The scripts and command-line utilities should now work. Let's step through the possible uses. \n\n### Collecting your own data\n\nThe `cronjob` folder contains the `crontab` for actively polling and collecting GitHub data. It's a good place to look if you want to understand how to collect data.\n\n- `cronjob/crontab`: The crontab pulls data by invoking `cronjob/save.sh` and `cronjob/ranks.sh` at certain intervals (these can be customized).\n\n- `cronjob/save.sh`: Essentially runs the `crunch.exe save` command (with a user-supplied GitHub token), see [here](https://github.com/rvantonder/CryptOSS/blob/master/cronjob/save.sh#L14). This command takes a list of comma-separated names registered in the `db.ml` [file](https://github.com/rvantonder/CryptOSS/blob/master/lib/db.ml). You can see the invocation of the `save.sh` script in the `crontab` [file](https://github.com/rvantonder/CryptOSS/blob/master/cronjob/crontab). \n\n- `cronjob/ranks.sh`: Pulls cryptocurrency data from CoinMarketCap\n\n- `batches`: The crontab uses batches of cryptcurrencies (listed in files) [example](https://github.com/rvantonder/CryptOSS/blob/master/cronjob/batches/batch-0.txt)). Each batch corresponds to a list of cryptocurrencies that fit within the 5000 request rate limit for GitHub, so that batched requests can be spaced out over 24 hours. The interval and batch size can be changed depending on need (see `cronjob/batches/generate.sh`).\n\nBesides the cronjob, you can manually save data by running, say, `crunch.exe save Bitcoin -token \u003cyour-github-token\u003e`. This produces a `.dat` file, as processed by `./pipeline.sh`.\n\nThe list of supported cryptocurrencies are in the [database file](https://github.com/rvantonder/CryptOSS/blob/master/lib/db.ml). Modify it to include your own, and type `make` again to update the tooling. You can then run `crunch.exe save \u003cMy Crypto\u003e -token ...`. \n\n### Processing data\n\nIf you want more control over data processing besides `./pipeline.sh`, you can use `crunch.exe load`. You can generate a CSV file from a `.dat` with a command like:\n\n```\ncrunch.exe load -no-forks -csv -with-ranks \u003cranks.json file from CoinMarketCap\u003e -with-date \u003cDD-MM-YYYY\u003e \u003csome-dat-file\u003e.dat\n```\n\nA similar command is [used in the csv-of-dat.sh script](https://github.com/rvantonder/CryptOSS/blob/master/csv-of-dat.sh#L17) to generate the MSR data set.\n\nYou can generate aggregate values by running `./crunch.exe aggregate` on some directory containing `.dat` files. This will create `.agg` files. `.agg` files can be used to generate the web view.\n\n### Generating the web view\n\nThe `./deploy.sh` [script](https://github.com/rvantonder/CryptOSS/blob/master/deploy.sh#L13-L18) builds a static site. If you want to create the webview for a particular date, say Oct 10, 2018 (containing `.dat`s), simply run `./deploy.sh datastore/2018-10-10`. This will generate a web view in `docs`.\n\n### Recreating the MSR dataset from the raw data\n\nCreate a directory called `datastore`. Download and untar the [raw data file](https://zenodo.org/record/2595588/files/raw-data-2018-01-21-to-2019-02-04.tar.gz?download=1) in this directory.\nIn the toplevel of this repository, run `./pipeline.sh \u003cN\u003e`, where `N` is the number of parallel jobs (this speeds up processing). You can ignore any warnings/errors. Once finished, you'll have generated `.csv` files in the toplevel directory.\n\nFeel free to add your own data in the `datastore` (for some date), and rerun `./pipeline.sh`.\n\n-----\n\n[![DOI](https://zenodo.org/badge/169338876.svg)](https://zenodo.org/badge/latestdoi/169338876)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frvantonder%2Fcryptoss","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frvantonder%2Fcryptoss","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frvantonder%2Fcryptoss/lists"}