{"id":13563764,"url":"https://github.com/jbenet/data","last_synced_at":"2026-03-07T03:03:13.751Z","repository":{"id":11903051,"uuid":"14466838","full_name":"jbenet/data","owner":"jbenet","description":"package manager for datasets","archived":false,"fork":false,"pushed_at":"2015-03-18T02:22:18.000Z","size":797,"stargazers_count":60,"open_issues_count":12,"forks_count":1,"subscribers_count":21,"default_branch":"master","last_synced_at":"2025-06-06T03:06:55.715Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jbenet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-11-17T13:02:46.000Z","updated_at":"2024-12-10T18:41:48.000Z","dependencies_parsed_at":"2022-09-07T02:52:41.925Z","dependency_job_id":null,"html_url":"https://github.com/jbenet/data","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/jbenet/data","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jbenet%2Fdata","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jbenet%2Fdata/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jbenet%2Fdata/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jbenet%2Fdata/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jbenet","download_url":"https://codeload.github.com/jbenet/data/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jbenet%2Fdata/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30206339,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-06T19:07:06.838Z","status":"online","status_checked_at":"2026-03-07T02:00:06.765Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T13:01:23.052Z","updated_at":"2026-03-07T03:03:13.732Z","avatar_url":"https://github.com/jbenet.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# data - package manager for datasets\n\nImagine installing datasets like this:\n\n    data get jbenet/norb\n\nIt's about time we used all we've learned making package managers to fix the\nawful data management problem. Read the [designdoc](dev/designdoc.md) and\nthe [roadmap](dev/roadmap.md).\n\n#### Table of Contents\n\n- [Install](#install)\n- [Usage](#usage)\n- [Datafile](#datafile)\n- [Development](#development)\n- [About](#about)\n- [Examples](#examples)\n\n## Install\n\nTwo ways to install:\n- [from pre-built binary distributions](http://datadex.io/doc/install) (the easy way)\n- [from source](http://datadex.io/doc/source-install) (the hard way)\n\n## Usage\n\nPlease see the [command reference](http://datadex.io/doc/ref).\n\n### Downloading datasets\n\nDownloading datasets is trivial:\n```\n\u003e data get jbenet/mnist\nInstalled jbenet/mnist@1.0 at datasets/jbenet/mnist@1.0\n```\n\n### Add datasets to projects\n\nOr, if you want to add datasets to a project, create a Datafile like this one:\n```\n\u003e cat Datafile\ndependencies:\n- jbenet/mnist@1.0\n- jbenet/cifar-10\n- jbenet/cifar-100\n```\n\nThen, run `data get` to install the dependencies (it works like `npm install`):\n```\n\u003e data get\nInstalled jbenet/mnist@1.0 at datasets/jbenet/mnist@1.0\nInstalled jbenet/cifar-10@1.0 at datasets/jbenet/cifar-10@1.0\nInstalled jbenet/cifar-100@1.0 at datasets/jbenet/cifar-100@1.0\n```\n\nYou can even commit the Datafile to version control, so your collaborators or users can easily get the data:\n```\n\u003e git clone github.com/jbenet/ml-vision-comparisons\n\u003e cd ml-vision-comparisons\n\u003e data get\nInstalled jbenet/mnist@1.0 at datasets/jbenet/mnist@1.0\nInstalled jbenet/cifar-10@1.0 at datasets/jbenet/cifar-10@1.0\nInstalled jbenet/cifar-100@1.0 at datasets/jbenet/cifar-100@1.0\n```\n\n### Publishing datasets\n\nPublishing datasets is simple:\n\n1. make a directory with all the files you want to publish.\n2. `cd` into it, and run `data publish` within the directory.\n3. `data` will guide you through creating a Datafile.\n4. Then, `data` will upload and publish the package.\n\n```\n\u003e data publish\n\u003clots of output\u003e\nPublished jbenet/mnist@1.0 (b5f84c2).\n```\n\nNote that uploading can take a long while, as we'll upload all the files to S3, ensuring others can always get them.\n\n\n## Datafile\n\ndata tracks the definition of dataset packages, and dependencies in a\n`Datafile` (in the style of `Makefile`, `Vagrantfile`, `Procfile`, and\nfriends). Both published dataset packages, and regular projects use it.\nIn a way, your project defines a dataset made up of other datasets, like\n`package.json` in `npm`.\n\n```\n# Datafile format\n# A YAML (inc json) doc with the following keys:\n\n# required\nhandle: \u003cauthor\u003e/\u003cname\u003e[.\u003cformat\u003e][@\u003ctag\u003e]\ntitle: Dataset Title\n\n# optional functionality\ndependencies: [\u003cother dataset handles\u003e]\nformats: {\u003cformat\u003e : \u003cformat url\u003e}\n\n# optional information\ndescription: Text describing dataset.\nrepository: \u003crepo url\u003e\nwebsite: \u003cdataset url\u003e\nlicense: \u003clicense url\u003e\ncontributors: [\"Author Name [\u003cemail\u003e] [(url)]\u003e\", ...]\nsources: [\u003csource urls\u003e]\n```\nMay be outdated. See [datafile.go](datafile.go).\n\n### why yaml?\n\nYAML is much more readable than json. One of `data`'s [design goals\n](https://github.com/jbenet/data/blob/master/dev/designdoc.md#design-goals)\nis an Intuitive UX. Since the target users are scientists in various domains,\nany extra syntax, parse errors, and other annoyances could cease to provide\nthe ease of use `data` aims for. I've always found this\n\n```\ndataset: feynman/spinning-plate-measurements\ntitle: Measurements of Plate Rotation\ncontributors:\n  - Richard Feynman \u003cfeynman@caltech.edu\u003e\nwebsite: http://caltech.edu/~feynman/not-girls/plate-stuff/trial3\n```\n\nmuch more friendly and approachable than this\n\n```\n{\n  \"dataset\": \"feynman/spinning-plate-measurements\",\n  \"title\": \"Measurements of Plate Rotation\",\n  \"contributors\": [\n    \"Richard Feynman \u003cfeynman@caltech.edu\u003e\"\n  ],\n  \"website\": \"http://caltech.edu/~feynman/not-girls/plate-stuff/trial3\"\n}\n```\n\nIt's already hard enough to get anyone to do anything. Don't add more hoops to\njump through than necessary. Each step will cause significant dropoff in\nconversion funnels. (Remember, [Apple pays Amazon for 1-click buy](https://www.apple.com/pr/library/2000/09/18Apple-Licenses-Amazon-com-1-Click-Patent-and-Trademark.html)...)\n\nAnd, since YAML is a superset of json, you can do whatever you want.\n\n\n## Development\n\nSetup:\n\n1. [install go](http://golang.org/doc/install)\n2. Run\n\n```\ngit clone https://github.com/jbenet/data\ncd data\nmake deps\nmake install\n```\n\nYou'll want to run [datadex](https://github.com/jbenet/datadex) too.\n\n## About\n\nThis project started because data management is a massive problem in science.\nIt should be **trivial** to (a) find, (b) download, (c) track, (d) manage,\n(e) re-format, (f) publish, (g) cite, and (h) collaborate on datasets. Data\nmanagement is a problem in other domains (engineering, civics, etc), and `data`\nseeks to be general enough to be used with any kind of dataset, but the target\nuse case is saving scientists' time.\n\nMany people agree we direly need the\n\"[GitHub for Science](http://static.benet.ai/t/github-for-science.md)\";\nscientific collaboration problems are large and numerous.\nIt is not entirely clear how, and in which order, to tackle these\nchallenges, or even how to drive adoption of solutions across fields. I think\nsimple and powerful tools can solve large problems neatly. Perhaps the best\nway to tackle scientific collaboration is by decoupling interconnected\nproblems, and building simple tools to solve them. Over time, reliable\ninfrastructure can be built with these. git, github, and arxiv are great\nexamples to follow.\n\n`data` is an attempt to solve the fairly self-contained issue of downloading,\npublishing, and managing datasets. Let's take what computer scientists have\nlearned about version control and distributed collaboration on source code,\nand apply it to the data management problem. Let's build new data tools and\ninfrastructure with the software engineering and systems design principles\nthat made git, apt, npm, and github successful.\n\n### Acknowledgements\n\n`data` is released under the MIT License.\n\nAuthored by [@jbenet](https://github.com/jbenet). Feel free to contact me\nat \u003cjuan@benet.ai\u003e, but please post\n[issues](https://github.com/jbenet/data/issues) on github first.\n\nSpecial thanks to\n[@colah](https://github.com/colah) (original idea and\n[data.py](https://github.com/colah/data)),\n[@damodei](https://github.com/damodei), and\n[@davidad](https://github.com/davidad),\nwho provided valuable thoughts + discussion on this problem.\n\n## Examples\n\n```\ndata - dataset package manager\n\nBasic commands:\n\n    get         Download and install dataset.\n    list        List installed datasets.\n    info        Show dataset information.\n    publish     Guided dataset publishing.\n\nTool commands:\n\n    version     Show data version information.\n    config      Manage data configuration.\n    user        Manage users and credentials.\n    commands    List all available commands.\n\nAdvanced Commands:\n\n    blob        Manage blobs in the blobstore.\n    manifest    Generate and manipulate dataset manifest.\n    pack        Dataset packaging, upload, and download.\n\nUse \"data help \u003ccommand\u003e\" for more information about a command.\n```\n\n### data get\n\n```\n# author/dataset\n\u003e data get jbenet/bar\nDownloading jbenet/foo from datadex.\nget blob b53ce99 Manifest\nget blob 2183ea8 Datafile\nget blob 63443e4 data.csv\ncopy blob 63443e4 data.txt\ncopy blob 63443e4 data.xsl\nget blob b53ce99 Manifest\n\nInstalled jbenet/foo@1.0 at datasets/jbenet/foo\n```\n\n### data list\n\n```\n\u003e data list\njbenet/bar@1.0\n```\n\n### data info\n\n```\n\u003e data info jbenet/foo\ndataset: jbenet/foo@1.0\ntitle: Foo Dataset\ndescription: The first dataset to use data.\nlicense: MIT\n\n# shows the Datafile\n\u003e cat datasets/jbenet/bar/Datafile\ndataset: foo/bar@1.1\n```\n\n### data publish\n\n```\n\u003e data publish\n==\u003e Guided Data Package Publishing.\n\n==\u003e Step 1/3: Creating the package.\nVerifying Datafile fields...\nGenerating manifest...\ndata manifest: added Datafile\ndata manifest: added data.csv\ndata manifest: added data.txt\ndata manifest: added data.xsl\ndata manifest: hashed 2183ea8 Datafile\ndata manifest: hashed 63443e4 data.csv\ndata manifest: hashed 63443e4 data.txt\ndata manifest: hashed 63443e4 data.xsl\n\n==\u003e Step 2/3: Uploading the package contents.\nput blob 2183ea8 Datafile - uploading\nput blob 63443e4 data.csv - exists\nput blob b53ce99 Manifest - uploading\n\n==\u003e Step 3/3: Publishing the package to the index.\ndata pack: published jbenet/foo@1.0 (b53ce99).\n```\n\nEt voila! You can now use `data get foo/bar` to retrieve it!\n\n### data config\n\n```\n\u003e data config index.datadex.url http://localhost:8080\n\u003e data config index.datadex.url\nhttp://localhost:8080\n```\n\n### data user\n\n```\n\u003e data user\ndata user - Manage users and credentials.\n\nCommands:\n\n    add         Register new user with index.\n    auth        Authenticate user account.\n    pass        Change user password.\n    info        Show (or edit) public user information.\n    url         Output user profile url.\n\nUse \"user help \u003ccommand\u003e\" for more information about a command.\n\n\u003e data user add\nUsername: juan\nPassword (6 char min):\nEmail (for security): juan@benet.ai\njuan registered.\n\n\u003e data user auth\nUsername: juan\nPassword:\nAuthenticated as juan.\n\n\u003e data user info\nname: \"\"\nemail: juan@benet.ai\n\n\u003e data user info jbenet\nname: Juan\nemail: juan@benet.ai\ngithub: jbenet\ntwitter: '@jbenet'\nwebsite: benet.ai\n\n\u003e data user info --edit\nEditing user profile. [Current value].\nFull Name: [] Juan Batiz-Benet\nWebsite Url: []\nGithub username: []\nTwitter username: []\nProfile saved.\n\n\u003e data user info\nname: Juan Batiz-Benet\nemail: juan@benet.ai\n\n\u003e data user pass\nUsername: juan\nCurrent Password:\nNew Password (6 char min):\nPassword changed. You will receive an email notification.\n\n\u003e data user url\nhttp://datadex.io:8080/juan\n```\n\n### data manifest (plumbing)\n\n```\n\u003e data manifest add filename\ndata manifest: added filename\n\n\u003e data manifest hash filename\ndata manifest: hashed 61a66fd filename\n\n\u003e cat .data-manifest\nfilename: 61a66fda64e397a82d9f0c8b7b3f7ba6bca79b12\n\n\u003e data manifest rm filename\ndata manifest: removed filename\n```\n\n### data blob (plumbing)\n```\ndata blob - Manage blobs in the blobstore.\n\nCommands:\n\n    put         Upload blobs to a remote blobstore.\n    get         Download blobs from a remote blobstore.\n    url         Output Url for blob named by \u003chash\u003e.\n\nUse \"blob help \u003ccommand\u003e\" for more information about a command.\n```\n\n```\n\u003e cat Manifest\nDatafile: 0d0c669b4c2b05402d9cc87298f3d7ce372a4c80\ndata.csv: 63443e4d74c3a170499fa9cfde5ae2224060b09e\ndata.txt: 63443e4d74c3a170499fa9cfde5ae2224060b09e\ndata.xsl: 63443e4d74c3a170499fa9cfde5ae2224060b09e\n\n\u003e data blob put --all\nput blob 0d0c669 Datafile\nput blob 63443e4 data.csv\n\n\u003e data blob get 63443e4d74c3a170499fa9cfde5ae2224060b09e\ndata blob get 63443e4d74c3a170499fa9cfde5ae2224060b09e\nget blob 63443e4 data.csv\ncopy blob 63443e4 data.txt\ncopy blob 63443e4 data.xsl\n\n\u003e data blob url\nhttp://datadex.archives.s3.amazonaws.com/blob/0d0c669b4c2b05402d9cc87298f3d7ce372a4c80\nhttp://datadex.archives.s3.amazonaws.com/blob/63443e4d74c3a170499fa9cfde5ae2224060b09e\n```\n\n### data pack (plumbing)\n\nThis is probably the most informative command to look at.\n\n```\ndata pack - Dataset packaging, upload, and download.\n\nCommands:\n\n    make        Create or update package description.\n    manifest    Show current package manifest.\n    upload      Upload package contents to remote storage.\n    download    Download package contents from remote storage.\n    publish     Publish package reference to dataset index.\n    check       Verify all file checksums match.\n\nUse \"pack help \u003ccommand\u003e\" for more information about a command.\n```\n\n\n```\n\u003e ls\ndata.csv  data.txt  data.xsl\n\n\u003e cat data.*\nBAR BAR BAR\nBAR BAR BAR\nBAR BAR BAR\n\n\u003e data pack make # interactive\nVerifying Datafile fields...\nEnter author name (required): foo\nEnter dataset id (required): bar\nEnter dataset version (required): 1.1\nEnter dataset title (optional): Barrr\nEnter description (optional): A bar dataset.\nEnter license name (optional): MIT\nGenerating manifest...\ndata manifest: hashed 0d0c669 Datafile\ndata manifest: hashed 63443e4 data.csv\ndata manifest: hashed 63443e4 data.txt\ndata manifest: hashed 63443e4 data.xsl\n\n\u003e ls\nDatafile  Manifest  data.csv  data.txt  data.xsl\n\n\u003e data pack manifest\nDatafile: 0d0c669b4c2b05402d9cc87298f3d7ce372a4c80\ndata.csv: 63443e4d74c3a170499fa9cfde5ae2224060b09e\ndata.txt: 63443e4d74c3a170499fa9cfde5ae2224060b09e\ndata.xsl: 63443e4d74c3a170499fa9cfde5ae2224060b09e\n\n\u003e data pack upload\nput blob 0d0c669 Datafile\nput blob 63443e4 data.csv\nput blob 8a2e6f6 Manifest\n\n\u003e rm data.*\n\n\u003e ls\nDatafile  Manifest\n\n\u003e data pack download\nget blob 63443e4 data.csv\ncopy blob 63443e4 data.txt\ncopy blob 63443e4 data.xsl\n\n\u003e ls\nDatafile  Manifest  data.csv  data.txt  data.xsl\n\n\u003e data pack check\ndata pack: 4 checksums pass\n\n\u003e echo \"FOO FOO FOO\" \u003e data.csv\n\n\u003e data pack check\ndata manifest: check 63443e4 data.csv FAIL\ndata pack: 1/4 checksums failed!\n\n\u003e data pack download\ncopy blob 63443e4 data.csv\n\n\u003e data pack check\ndata pack: 4 checksums pass\n\n\u003e data pack publish\ndata pack: published foo/bar@1.1 (8a2e6f6).\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjbenet%2Fdata","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjbenet%2Fdata","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjbenet%2Fdata/lists"}