{"id":18014793,"url":"https://github.com/zmedelis/hfds-clj","last_synced_at":"2026-03-15T08:04:58.317Z","repository":{"id":205640690,"uuid":"714701875","full_name":"zmedelis/hfds-clj","owner":"zmedelis","description":"Access to HuggingFace datasets via Clojure","archived":false,"fork":false,"pushed_at":"2025-12-30T08:57:11.000Z","size":44,"stargazers_count":13,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-01-02T21:21:50.410Z","etag":null,"topics":["clojure","datasets","huggingface"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zmedelis.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-11-05T16:21:47.000Z","updated_at":"2025-12-30T08:54:39.000Z","dependencies_parsed_at":null,"dependency_job_id":"17088193-a918-4139-8213-11457f0bbfc9","html_url":"https://github.com/zmedelis/hfds-clj","commit_stats":null,"previous_names":["zmedelis/hfds-clj"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/zmedelis/hfds-clj","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zmedelis%2Fhfds-clj","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zmedelis%2Fhfds-clj/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zmedelis%2Fhfds-clj/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zmedelis%2Fhfds-clj/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zmedelis","download_url":"https://codeload.github.com/zmedelis/hfds-clj/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zmedelis%2Fhfds-clj/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30538255,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-15T07:17:37.589Z","status":"ssl_error","status_checked_at":"2026-03-15T07:17:31.738Z","response_time":61,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure","datasets","huggingface"],"created_at":"2024-10-30T04:11:04.948Z","updated_at":"2026-03-15T08:04:58.311Z","avatar_url":"https://github.com/zmedelis.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"## 🪣 hfds-clj\n\n[![Clojars Project](https://img.shields.io/clojars/v/io.github.zmedelis/hfds-clj.svg)](https://clojars.org/io.github.zmedelis/hfds-clj)\n\n**hfds-clj** is a lib to help you get to the [HuggingFace datasets](https://huggingface.co/datasets) and [HuggingFace models](https://huggingface.co/models). \n\nThe lib provides seamless access to datasets via this process:\n* *downloading* HF dataset,\n* *caching* downloaded set locally, and\n* *serving* it from there for subsequent requests.\n\nIt does not aim to replicate the full range of functionality found in the [HuggingFace datasets library](https://huggingface.co/docs/datasets/v2.14.5/en/index). Though as an immediate extension, it would be great to support [Dataset Features](https://huggingface.co/docs/datasets/v2.14.5/en/about_dataset_features).\n\n## Download datasets\n\n\n### CLI\n\nData sets can be downloaded from the command line\n```\nclojure -X:download-dataset :dataset \"allenai/prosocial-dialog\"\n```\n\nSee next section for parameter description.\n\n### Code\n\n```clojure\n(require '[hfds-clj.datasets :refer [load-dataset]])\n```\n\nDownload HF datasets with this oneliner, where a single parameter is the dataset name as provided on the HF dataset page.\n\n```clojure\n(load-dataset \"Anthropic/hh-rlhf\")\n```\nThe second call with `Anthropic/hh-rlhf` parameter will load it from the cache and return a lazy sequence of all the dataset records.\n\nA more fine-grained data set request is supported via a parameterized call:\n\n```clojure\n(load-dataset  {:dataset \"allenai/prosocial-dialog\"\n                          :split   \"train\"\n                          :config  \"default\"\n                          :offset  0\n                          :length  100}\n               {:hfds/download-mode :reuse-dataset-if-exists\n                :hfds/cache-dir     \"/data\"\n                :hfds/limit         4000}))\n```\n\n## Downloads models\n\nModels can be downloaded and stored on disk via this CLI call:\n```\nclojure -X:download-model :model '\"nvidia/Gemma-2b-it-ONNX-INT4\"' :hf-token \"\u003chuggingface API token\u003e\" :models-base-dir '\"/tmp/models\"'\n```\n\n\n# Usage as Clojure tool\n'hfds-clj'  can be used as well as a Clojure tool.\n\nInstallation as tool (latest GIT version)\n\n```bash\nclojure -Ttools install io.github.zmedelis/hfds-clj '{:git/url \"https://github.com/zmedelis/hfds-clj\" :git/sha \"4a84254030fceca8bf3f5e8dce4226b4b8cdf48a\"}' :as hfds-clj\n```\nExample to to call it as tool, to download data \n```bash\nclojure -Thfds-clj hfds-clj.datasets/download-cli :dataset \"allenai/prosocial-dialog\"\n```\n\nand model:\n```bash\nclojure -Thfds-clj hfds-clj.models/download-cli \n     :models-base-dir '\"/tmp/models\"' \n     :model '\"nvidia/Gemma-2b-it-ONNX-INT4\"' \n     :revision '\"4fe167cca69847b5218e7cc37c7e1984056cf340\"'\n     :hf-token \"\u003chuggingface API token\u003e\" \n\n```\n`:revision` is optional. If not specified, \"main\" (= latest)  is used\n\n`:hf-token`  is optional. Most models are public, so no huggingface token is needed\n\n\n\n## Notes\n\n* This is extracted from [Bosquet](https://github.com/zmedelis/bosquet) where HuggingFace datasets are used for LLM related developments.\n* Thanks to [TrueGrit](https://github.com/KingMob/TrueGrit) helping to robustly fetch data from HF API\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzmedelis%2Fhfds-clj","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzmedelis%2Fhfds-clj","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzmedelis%2Fhfds-clj/lists"}