{"id":18458579,"url":"https://github.com/softwaresystemslaboratory/ptmtorrent","last_synced_at":"2026-03-05T20:03:19.333Z","repository":{"id":65521508,"uuid":"583056411","full_name":"SoftwareSystemsLaboratory/PTMTorrent","owner":"SoftwareSystemsLaboratory","description":"Code to generate the PTMTorrent dataset","archived":false,"fork":false,"pushed_at":"2023-07-25T21:08:41.000Z","size":764,"stargazers_count":8,"open_issues_count":3,"forks_count":11,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-04-13T03:55:15.467Z","etag":null,"topics":["model-hub","pre-trained-model","ptm","torrent"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SoftwareSystemsLaboratory.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":".zenodo.json"}},"created_at":"2022-12-28T16:24:44.000Z","updated_at":"2025-03-15T07:31:43.000Z","dependencies_parsed_at":"2025-04-13T03:55:17.501Z","dependency_job_id":"2c9282ae-cf84-42d8-a3f7-32f5f2fc0047","html_url":"https://github.com/SoftwareSystemsLaboratory/PTMTorrent","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/SoftwareSystemsLaboratory/PTMTorrent","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SoftwareSystemsLaboratory%2FPTMTorrent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SoftwareSystemsLaboratory%2FPTMTorrent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SoftwareSystemsLaboratory%2FPTMTorrent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SoftwareSystemsLaboratory%2FPTMTorrent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SoftwareSystemsLaboratory","download_url":"https://codeload.github.com/SoftwareSystemsLaboratory/PTMTorrent/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SoftwareSystemsLaboratory%2FPTMTorrent/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266520988,"owners_count":23942368,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["model-hub","pre-trained-model","ptm","torrent"],"created_at":"2024-11-06T08:19:08.939Z","updated_at":"2026-03-05T20:03:19.246Z","avatar_url":"https://github.com/SoftwareSystemsLaboratory.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PTMTorrent\n\n\u003e Code to generate the PTMTorrent dataset\n\n[![Python Version](https://img.shields.io/badge/Python-3.10.9-blue)](https://img.shields.io/badge/Python-3.10.9-blue)\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.7570357.svg)](https://doi.org/10.5281/zenodo.7570357)\n[![Release Project](https://github.com/SoftwareSystemsLaboratory/PTM-Torrent/actions/workflows/release.yml/badge.svg)](https://github.com/SoftwareSystemsLaboratory/PTM-Torrent/actions/workflows/release.yml)\n\n## Table of Contents\n\n- [PTMTorrent](#ptmtorrent)\n  - [Table of Contents](#table-of-contents)\n  - [About](#about)\n    - [Supported Model Hubs](#supported-model-hubs)\n  - [Dependencies](#dependencies)\n  - [How To Install](#how-to-install)\n    - [From GitHub Releases](#from-github-releases)\n    - [From Source](#from-source)\n  - [How to Run](#how-to-run)\n    - [As Individual Scripts](#as-individual-scripts)\n  - [Data Representation](#data-representation)\n  - [Pre-Packaged Dataset](#pre-packaged-dataset)\n  - [Example Usage of Dataset](#example-usage-of-dataset)\n  - [How to Cite](#how-to-cite)\n  - [References](#references)\n\n## About\n\nThis repository contains the scripts to generate the *PTMTorrent* dataset. The\ndataset contains sets of pre-trained machine learning models (PTM)\n[`git`](https://git-scm.com) repositories hosted on popular model hubs.\nSupporting metadata from each model hub as well as standardized metadata\nspecified by [this JSON Schema](ptm_torrent/utils/schemas/ptmtorrent.json) is\nalso included in.\n\n### Supported Model Hubs\n\nThe following model hubs are supported by our software:\n\n- [Hugging Face](https://huggingface.co/)\n- [Modelhub](https://modelhub.ai/)\n- [ModelZoo](https://modelzoo.co/)\n- [PyTorch Hub](https://pytorch.org/hub/)\n- [ONNX Model Zoo](https://github.com/onnx/models/)\n\n## Dependencies\n\nThis project is dependent upon the following software:\n\n- [`Python 3.10.9`](https://www.python.org/downloads/release/python-3109/)\n\n\u003e Package dependencies are given in [`pypoetry.toml`](pyproject.toml) and\n\u003e handled by [`poetry`](https://python-poetry.org/)\n\n- [`Git`](https://git-scm.com)\n- [`Git LFS`](https://git-lfs.com/)\n\n## How To Install\n\nTo run this project, it must be packaged and installed first.\n\nThe package can either be installed from our\n[GitHub Releases](#from-github-releases) or built and installed\n[From Source](#from-source).\n\n### From GitHub Releases\n\n1. Download the latest `.tar.gz` or `.whl` file from\n   [here](https://github.com/SoftwareSystemsLaboratory/PTM-Torrent/releases).\n1. Install via pip: `python3.10 -m pip install ptm_torrent-*`\n\n### From Source\n\n\u003e Instructions were written for Linux operating systems\n\n1. Clone the project locally:\n   `git clone https://github.com/SoftwareSystemsLaboratory/PTM-Torrent`\n1. `cd` into the project: `cd PTM-Torrent`\n1. Create a `Python 3.10` virtual environment: `python3.10 -m venv env`\n1. Activate virtual environment: `source env/bin/activate`\n1. Upgrade `pip`: `python -m pip install --upgrade pip`\n1. Install `poetry`: `python -m pip install poetry`\n1. Install `Python` dependencies through `poetry`: `python -m poetry install`\n1. Build with `poetry`: `python -m poetry build`\n1. Install with `pip`: `python -m pip install dist/ptm_torrent*.tar.gz`\n\n## How to Run\n\nAfter [installing the package](#how-to-install), this project can be ran as\nindividual scripts per model hub.\n\n### As Individual Scripts\n\nEach model hub's scripts are separated by directory in the\n[`ptm_torrent`](ptm_torrent/) directory. The directory for each specific model\nhub's scripts, the main runner script, and download size, is listed in the table\nbelow:\n\n| Model Hub      | Scripts Directory          | Script Name   | Download Size |\n| -------------- | -------------------------- | ------------- | ------------- |\n| Hugging Face   | `ptm_torrent/huggingface`  | `__main__.py` | 61 TB         |\n| Modelhub       | `ptm_torrent/modelhub`     | `__main__.py` | 721 MB        |\n| ModelZoo       | `ptm_torrent/modelzoo`     | `__main__.py` | 151 GB        |\n| ONNX Model Zoo | `pmt_torrent/onnxmodelzoo` | `__main__.py` | 441 MB        |\n| Pytorch Hub    | `ptm_torrent/pytorchhub`   | `__main__.py` | 1.5 GB        |\n\n\u003c!-- Table created with https://www.tablesgenerator.com/markdown_tables --\u003e\n\nThere are other supporting scripts within each model hub's scripts directory.\nThese scripts are ran in order by the model hub's `__main__.py` file. The order\nin which to run these scripts (should the `__main__.py` file be insufficient) is\ndescribed in each model hub's `README.md` file within the scripts directory.\n\n\u003e NOTE: Hugging Face's `__main__.py` can be parameritized to allow for a\n\u003e specific percentage of the model hub to be downloaded. By default, it is the\n\u003e first 0.1 (10%) of models sorted by downloads in descending order.\n\nTo run any of the scripts, execute the following command pattern:\n\n- `python 'Scripts Directory'/'Script Name'`\n\nFor example, to run Hugging Face's scripts:\n\n- `python ptm_torrent/huggingface/__main__.py`\n\n## Data Representation\n\nEach model hub script generates the following directory structure **per model\nhub**:\n\n```shell\n📦data\n ┗ 📂MODELHUB\n ┃ ┣ 📂html\n ┃ ┃ ┗ 📂metadata\n ┃ ┃ ┃ ┃ ┗ 📂models\n ┃ ┃ ┣ 📂json\n ┃ ┃ ┃ ┗ 📂metadata\n ┃ ┃ ┃ ┃ ┗ 📂models\n ┃ ┃ ┗ 📂repos\n ┃ ┃ ┃ ┗ 📂AUTHOR\n ┃ ┃ ┃ ┃ ┗ 📂MODEL\n```\n\n\u003e This directory structure is generated relative to where the script is ran\n\u003e from. Example: if the script was ran from the home directory (`~`), then the\n\u003e `data` directory would be stored at `~/data`.\n\nWhere:\n\n- data/`MODELHUB` is the same name as the `Python` module directory that\n  contained the script.\n- data/MODELHUB/repos/`AUTHOR` is the author name of the repository that was\n  cloned.\n- data/MODELHUB/repos/AUTHOR/`MODEL` is the name of the repository that was\n  cloned.\n\nModel hub scripts do not overwrite the directory. In other words, it is a safe\noperation to run multiple model hub scripts from the same directory sequentially\nor concurrently.\n\nSpecifics about the types of metadata files and content that are produced by the\nscripts can be found in each model hub's script directory's `README.md` file.\n\n## Pre-Packaged Dataset\n\nAn existing dataset is available on\n[this Purdue University Globus share](https://app.globus.org/file-manager?origin_id=55e17a6e-9d8f-11ed-a2a2-8383522b48d9\u0026origin_path=%2F%7E%2F).\n\nIf you are unfamiliar with Globus, we prepared a guide in the [globus-docs/](globus-docs/) directory.\n\n## Example Usage of Dataset\n\nAn example usage of the dataset is described within the\n[`example`](example/README.md) directory.\n\n## How to Cite\n\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.7570357.svg)](https://doi.org/10.5281/zenodo.7570357)\n\nThis project has a DOI on [Zenodo](https://doi.org/10.5281/zenodo.7570357).\nPlease visit our Zenodo page for the latest citation information.\n\n## References\n\n\u003e References are sorted by alphabetical order and not how they appear in this\n\u003e document.\n\n\\[1\\] “Git.” \u003chttps://git-scm.com/\u003e (accessed Jan. 25, 2023).\n\n\\[2\\] “Git Large File Storage,” Git Large File Storage. \u003chttps://git-lfs.com/\u003e\n(accessed Jan. 25, 2023).\n\n\\[3\\] “Hugging Face – The AI community building the future.,” Jan. 03, 2023.\n\u003chttps://huggingface.co/\u003e (accessed Jan. 25, 2023).\n\n\\[4\\] “Model Zoo - Deep learning code and pretrained models for transfer\nlearning, educational purposes, and more.” \u003chttps://modelzoo.co/\u003e (accessed Jan.\n25, 2023).\n\n\\[5\\] “Modelhub.” \u003chttps://modelhub.ai/\u003e (accessed Jan. 25, 2023).\n\n\\[6\\] “MSR 2023 - Data and Tool Showcase Track - MSR 2023.”\n\u003chttps://conf.researchr.org/track/msr-2023/msr-2023-data-showcase\u003e (accessed\nJan. 25, 2023).\n\n\\[7\\] “ONNX Model Zoo.” Open Neural Network Exchange, Jan. 25, 2023. Accessed:\nJan. 25, 2023. \\[Online\\]. Available: \u003chttps://github.com/onnx/models\u003e\n\n\\[8\\] “pip documentation v22.3.1.” \u003chttps://pip.pypa.io/en/stable/\u003e (accessed\nJan. 25, 2023).\n\n\\[9\\] “Poetry - Python dependency management and packaging made easy.”\n\u003chttps://python-poetry.org/\u003e (accessed Jan. 25, 2023).\n\n\\[10\\] “Python Release Python 3.10.9,” Python.org.\n\u003chttps://www.python.org/downloads/release/python-3109/\u003e (accessed Jan. 25,\n2023).\n\n\\[11\\] “PyTorch Hub.” \u003chttps://www.pytorch.org/hub\u003e (accessed Jan. 25, 2023).\n\n\\[12\\] W. Jiang et al., “SoftwareSystemsLaboratory/PTMTorrent.” Zenodo, Jan. 25,\n2023\\. doi: 10.5281/zenodo.7570357.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoftwaresystemslaboratory%2Fptmtorrent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsoftwaresystemslaboratory%2Fptmtorrent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoftwaresystemslaboratory%2Fptmtorrent/lists"}