{"id":38670151,"url":"https://github.com/deepghs/hfutils","last_synced_at":"2026-01-17T09:54:33.529Z","repository":{"id":213903835,"uuid":"735220374","full_name":"deepghs/hfutils","owner":"deepghs","description":"Useful utilities for huggingface","archived":false,"fork":false,"pushed_at":"2025-12-17T08:07:52.000Z","size":479886,"stargazers_count":22,"open_issues_count":3,"forks_count":3,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-19T17:15:45.495Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://hfutils.readthedocs.io/en/latest/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deepghs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-12-24T05:20:20.000Z","updated_at":"2025-12-17T08:07:56.000Z","dependencies_parsed_at":"2024-05-19T13:42:59.222Z","dependency_job_id":"f2dd58ed-54d1-4027-b678-f77e6e9fcd1b","html_url":"https://github.com/deepghs/hfutils","commit_stats":null,"previous_names":["deepghs/hfutils"],"tags_count":50,"template":false,"template_full_name":null,"purl":"pkg:github/deepghs/hfutils","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepghs%2Fhfutils","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepghs%2Fhfutils/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepghs%2Fhfutils/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepghs%2Fhfutils/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deepghs","download_url":"https://codeload.github.com/deepghs/hfutils/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepghs%2Fhfutils/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28505565,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T06:57:29.758Z","status":"ssl_error","status_checked_at":"2026-01-17T06:56:03.931Z","response_time":85,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-17T09:54:33.471Z","updated_at":"2026-01-17T09:54:33.523Z","avatar_url":"https://github.com/deepghs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# hfutils\n\n[![PyPI](https://img.shields.io/pypi/v/hfutils)](https://pypi.org/project/hfutils/)\n![PyPI - Python Version](https://img.shields.io/pypi/pyversions/hfutils)\n![Loc](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/narugo1992/b98a0fd1468c4858abf2a355bc9b4039/raw/loc.json)\n![Comments](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/narugo1992/b98a0fd1468c4858abf2a355bc9b4039/raw/comments.json)\n\n[![Code Test](https://github.com/deepghs/hfutils/workflows/Code%20Test/badge.svg)](https://github.com/deepghs/hfutils/actions?query=workflow%3A%22Code+Test%22)\n[![Package Release](https://github.com/deepghs/hfutils/workflows/Package%20Release/badge.svg)](https://github.com/deepghs/hfutils/actions?query=workflow%3A%22Package+Release%22)\n[![codecov](https://codecov.io/gh/deepghs/hfutils/branch/main/graph/badge.svg?token=XJVDP4EFAT)](https://codecov.io/gh/deepghs/hfutils)\n\n[![Discord](https://img.shields.io/discord/1157587327879745558?style=social\u0026logo=discord\u0026link=https%3A%2F%2Fdiscord.gg%2FTwdHJ42N72)](https://discord.gg/TwdHJ42N72)\n![GitHub Org's stars](https://img.shields.io/github/stars/deepghs)\n[![GitHub stars](https://img.shields.io/github/stars/deepghs/hfutils)](https://github.com/deepghs/hfutils/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/deepghs/hfutils)](https://github.com/deepghs/hfutils/network)\n![GitHub commit activity](https://img.shields.io/github/commit-activity/m/deepghs/hfutils)\n[![GitHub issues](https://img.shields.io/github/issues/deepghs/hfutils)](https://github.com/deepghs/hfutils/issues)\n[![GitHub pulls](https://img.shields.io/github/issues-pr/deepghs/hfutils)](https://github.com/deepghs/hfutils/pulls)\n[![Contributors](https://img.shields.io/github/contributors/deepghs/hfutils)](https://github.com/deepghs/hfutils/graphs/contributors)\n[![GitHub license](https://img.shields.io/github/license/deepghs/hfutils)](https://github.com/deepghs/hfutils/blob/master/LICENSE)\n\nUseful utilities for huggingface\n\n## Quick Start\n\nTo get started with `hfutils`, you can install it using PyPI:\n\n```shell\npip install hfutils\n\n```\n\nAlternatively, you can install it from the source code:\n\n```shell\ngit clone https://github.com/deepghs/hfutils.git\ncd hfutils\npip install .\n\n```\n\nVerify the installation by checking the version:\n\n```shell\nhfutils -v\n\n```\n\nIf Python is not available in your local environment, we recommend downloading the pre-compiled executable version from\nthe [releases](https://github.com/deepghs/hfutils/releases).\n\n## Upload Data\n\nUpload data to repositories using the following commands:\n\n```shell\n# Upload a single file to the repository\nhfutils upload -r your/repository -i /your/local/file -f file/in/your/repo\n\n# Upload files in a directory as an archive file to the repository\n# More formats of archive files are supported\n# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html\nhfutils upload -r your/repository -i /your/local/directory -a archive/file/in/your/repo.zip\n\n# Upload files in a directory as a directory tree to the repository\nhfutils upload -r your/repository -i /your/local/directory -d dir/in/your/repo\n\n```\n\nYou can achieve the same using the Python API:\n\n```python\nfrom hfutils.operate import upload_file_to_file, upload_directory_as_archive, upload_directory_as_directory\n\n# Upload a single file to the repository\nupload_file_to_file(\n    local_file='/your/local/file',\n    repo_id='your/repository',\n    file_in_repo='file/in/your/repo'\n)\n\n# Upload files in a directory as an archive file to the repository\n# More formats of archive files are supported\n# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html\nupload_directory_as_archive(\n    local_directory='/your/local/directory',\n    repo_id='your/repository',\n    archive_in_repo='archive/file/in/your/repo.zip'\n)\n\n# Upload files in a directory as a directory tree to the repository\nupload_directory_as_directory(\n    local_directory='/your/local/directory',\n    repo_id='your/repository',\n    path_in_repo='dir/in/your/repo'\n)\n```\n\nExplore additional options for uploading:\n\n```shell\nhfutils upload -h\n\n```\n\n## Download Data\n\nDownload data from repositories using the following commands:\n\n```shell\n# Download a single file from the repository\nhfutils download -r your/repository -o /your/local/file -f file/in/your/repo\n\n# Download an archive file from the repository and extract it to the given directory\n# More formats of archive files are supported\n# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html\nhfutils download -r your/repository -o /your/local/directory -a archive/file/in/your/repo.zip\n\n# Download files from the repository as a directory tree\nhfutils download -r your/repository -o /your/local/directory -d dir/in/your/repo\n\n```\n\nUse the Python API for the same functionality:\n\n```python\nfrom hfutils.operate import download_file_to_file, download_archive_as_directory, download_directory_as_directory\n\n# Download a single file from the repository\ndownload_file_to_file(\n    local_file='/your/local/file',\n    repo_id='your/repository',\n    file_in_repo='file/in/your/repo'\n)\n\n# Download an archive file from the repository and extract it to the given directory\n# More formats of archive files are supported\n# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html\ndownload_archive_as_directory(\n    local_directory='/your/local/directory',\n    repo_id='your/repository',\n    file_in_repo='archive/file/in/your/repo.zip',\n)\n\n# Download files from the repository as a directory tree\ndownload_directory_as_directory(\n    local_directory='/your/local/directory',\n    repo_id='your/repository',\n    dir_in_repo='dir/in/your/repo'\n)\n```\n\nExplore additional options for downloading:\n\n```shell\nhfutils download -h\n\n```\n\n## List Files in Repository\n\nList files in repositories\n\n```shell\nhfutils ls -r your/repository -o /your/local/file -d subdir/in/repo\n```\n\n## Supported Formats\n\nBy default, we support the `zip` and `tar` formats, including `.zip`, `.tar`, `.tar.gz`, `.tar.bz2`, and `.tar.xz`.\n\nIf you require support for `rar` and `7z` files, install the extra dependencies using the following command:\n\n```shell\npip install hfutils[rar,7z]\n```\n\n**NOTE:** Creating RAR archive files is not supported. We use the [rarfile](https://github.com/markokr/rarfile) library,\nwhich lacks the functionality for creating RAR files.\n\n## How to Access Private Repositories\n\nSimply configure the `HF_TOKEN` environment variable by using your HuggingFace access token.\nNote that write permissions are required if you plan to upload any content.\n\n## How to Use Hf-Transfer for Acceleration\n\nIf you are using the PyPI CLI, you need to install `hfutils` with the following command:\n\n```shell\npip install hfutils[transfer]\n\n```\n\nIf you are using a precompiled executable file, the transfer module is integrated inside; simply use it.\n\nEnable Hf-Transfer acceleration by setting the environment variable `HF_HUB_ENABLE_HF_TRANSFER` to `1`.\n\n## How to Change the Temporary Directory\n\nThe temporary directory is utilized for storing partially downloaded files, consuming a considerable amount of disk\nspace.\n\nIf your disk, especially the C drive on Windows, does not have sufficient space, simply set the `TMPDIR` to designate\nanother directory as the temporary directory.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepghs%2Fhfutils","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeepghs%2Fhfutils","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepghs%2Fhfutils/lists"}