{"id":22582822,"url":"https://github.com/vtuber-plan/olah","last_synced_at":"2025-05-16T12:06:26.715Z","repository":{"id":203551115,"uuid":"708875037","full_name":"vtuber-plan/olah","owner":"vtuber-plan","description":"Self-hosted huggingface mirror service. 自建huggingface镜像服务。","archived":false,"fork":false,"pushed_at":"2025-03-12T04:14:17.000Z","size":199,"stargazers_count":149,"open_issues_count":9,"forks_count":20,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-03T17:09:51.005Z","etag":null,"topics":["datasets","huggingface","llm","mirror","mirror-server","self-hosting"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vtuber-plan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-23T15:01:52.000Z","updated_at":"2025-04-02T06:50:11.000Z","dependencies_parsed_at":"2023-12-20T18:27:21.768Z","dependency_job_id":"671591e1-9cb0-456a-90eb-6783f4b43e93","html_url":"https://github.com/vtuber-plan/olah","commit_stats":null,"previous_names":["vtuber-plan/olah"],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vtuber-plan%2Folah","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vtuber-plan%2Folah/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vtuber-plan%2Folah/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vtuber-plan%2Folah/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vtuber-plan","download_url":"https://codeload.github.com/vtuber-plan/olah/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248525757,"owners_count":21118747,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datasets","huggingface","llm","mirror","mirror-server","self-hosting"],"created_at":"2024-12-08T06:12:25.111Z","updated_at":"2025-04-12T06:20:54.111Z","avatar_url":"https://github.com/vtuber-plan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003eOlah\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n\u003cb\u003eSelf-hosted Lightweight Huggingface Mirror Service\u003c/b\u003e\n\nOlah is a self-hosted lightweight huggingface mirror service. `Olah` means `hello` in Hilichurlian.\nOlah implemented the `mirroring` feature for huggingface resources, rather than just a simple `reverse proxy`.\nOlah does not immediately mirror the entire huggingface website but mirrors the resources at the file block level when users download them (or we can say cache them).\n\nOther languages: [中文](README_zh.md)\n\n## Advantages of Olah\nOlah has the capability to cache files in chunks while users download them. Upon subsequent downloads, the files can be directly retrieved from the cache, greatly enhancing download speeds and saving bandwidth.\nAdditionally, Olah offers a range of cache control policies. Administrators can configure which repositories are accessible and which ones can be cached through a configuration file.\n\n## Features\n* Huggingface Data Cache\n* Models mirror\n* Datasets mirror\n* Spaces mirror\n\n## Install\n\n### Method 1: With pip\n\n```bash\npip install olah\n```\n\nor:\n\n```bash\npip install git+https://github.com/vtuber-plan/olah.git \n```\n\n### Method 2: From source\n\n1. Clone this repository\n```bash\ngit clone https://github.com/vtuber-plan/olah.git\ncd olah\n```\n\n2. Install the Package\n```bash\npip install --upgrade pip\npip install -e .\n```\n\n## Quick Start\nRun the command in the console: \n```bash\nolah-cli\n```\n\nThen set the Environment Variable `HF_ENDPOINT` to the mirror site (Here is http://localhost:8090).\n\nLinux: \n```bash\nexport HF_ENDPOINT=http://localhost:8090\n```\n\nWindows Powershell:\n```bash\n$env:HF_ENDPOINT = \"http://localhost:8090\"\n```\n\nStarting from now on, all download operations in the HuggingFace library will be proxied through this mirror site.\n```bash\npip install -U huggingface_hub\n```\n\n```python\nfrom huggingface_hub import snapshot_download\n\nsnapshot_download(repo_id='Qwen/Qwen-7B', repo_type='model',\n                  local_dir='./model_dir', resume_download=True,\n                  max_workers=8)\n```\n\nOr you can download models and datasets by using huggingface cli.\n\nDownload GPT2:\n```bash\nhuggingface-cli download --resume-download openai-community/gpt2 --local-dir gpt2\n```\n\nDownload WikiText:\n```bash\nhuggingface-cli download --repo-type dataset --resume-download Salesforce/wikitext --local-dir wikitext\n```\n\nYou can check the path `./repos`, in which olah stores all cached datasets and models.\n\n## Start the server\nRun the command in the console: \n```bash\nolah-cli\n```\n\nOr you can specify the host address and listening port:\n```bash\nolah-cli --host localhost --port 8090\n```\n**Note: Please change --mirror-netloc and --mirror-lfs-netloc to the actual URLs of the mirror sites when modifying the host and port.**\n```bash\nolah-cli --host 192.168.1.100 --port 8090 --mirror-netloc 192.168.1.100:8090\n```\n\nThe default mirror cache path is `./repos`, you can change it by `--repos-path` parameter:\n```bash\nolah-cli --host localhost --port 8090 --repos-path ./hf_mirrors\n```\n\n**Note that the cached data between different versions cannot be migrated. Please delete the cache folder before upgrading to the latest version of Olah.**\n\nIn deployment scenarios, there may be high concurrent downloads, leading to Timeout errors for new connections.\nYou can set the `WEB_CONCURRENCY` variable for uvicorn to increase the number of workers, thereby enhancing concurrency in production environments.\nFor example, on Linux:\n```bash\nexport WEB_CONCURRENCY=4\n```\n\n## More Configurations\n\nAdditional configurations can be controlled through a configuration file by passing the `configs.toml` file as a command parameter:\n```bash\nolah-cli -c configs.toml\n```\n\nThe complete content of the configuration file can be found at [assets/full_configs.toml](https://github.com/vtuber-plan/olah/blob/main/assets/full_configs.toml).\n\n### Configuration Details\nThe first section, `basic`, is used to set up basic configurations for the mirror site:\n```toml\n[basic]\nhost = \"localhost\"\nport = 8090\nssl-key = \"\"\nssl-cert = \"\"\nrepos-path = \"./repos\"\ncache-size-limit = \"\"\ncache-clean-strategy = \"LRU\"\nhf-scheme = \"https\"\nhf-netloc = \"huggingface.co\"\nhf-lfs-netloc = \"cdn-lfs.huggingface.co\"\nmirror-scheme = \"http\"\nmirror-netloc = \"localhost:8090\"\nmirror-lfs-netloc = \"localhost:8090\"\nmirrors-path = [\"./mirrors_dir\"]\n```\n- `host`: Sets the host address that Olah listens to.\n- `port`: Sets the port that Olah listens to.\n- `ssl-key` and `ssl-cert`: When enabling HTTPS, specify the file paths for the key and certificate.\n- `repos-path`: Specifies the directory for storing cached data.\n- `cache-size-limit`: Specifies cache size limit (For example, 100G, 500GB, 2TB). Olah will scan the size of the cache folder every hour. If it exceeds the limit, olah will delete some cache files.\n- `cache-clean-strategy`: Specifies cache cleaning strategy (Available strategies: LRU, FIFO, LARGE_FIRST).\n- `hf-scheme`: Network protocol for the Hugging Face official site (usually no need to modify).\n- `hf-netloc`: Network location of the Hugging Face official site (usually no need to modify).\n- `hf-lfs-netloc`: Network location for Hugging Face official site's LFS files (usually no need to modify).\n- `mirror-scheme`: Network protocol for the Olah mirror site (should match the above settings; change to HTTPS if providing `ssl-key` and `ssl-cert`).\n- `mirror-netloc`: Network location of the Olah mirror site (should match `host` and `port` settings).\n- `mirror-lfs-netloc`: Network location for Olah mirror site's LFS (should match `host` and `port` settings).\n- `mirrors-path`: Additional mirror file directories. If you have already cloned some Git repositories, you can place them in this directory for downloading. In this example, the directory is `./mirrors_dir`. To add a dataset like `Salesforce/wikitext`, you can place the Git repository in the directory `./mirrors_dir/datasets/Salesforce/wikitext`. Similarly, models can be placed under `./mirrors_dir/models/organization/repository`.\n\nThe second section allows for accessibility restrictions:\n```toml\n[accessibility]\noffline = false\n\n[[accessibility.proxy]]\nrepo = \"cais/mmlu\"\nallow = true\n\n[[accessibility.proxy]]\nrepo = \"adept/fuyu-8b\"\nallow = false\n\n[[accessibility.proxy]]\nrepo = \"mistralai/*\"\nallow = true\n\n[[accessibility.proxy]]\nrepo = \"mistralai/Mistral.*\"\nallow = false\nuse_re = true\n\n[[accessibility.cache]]\nrepo = \"cais/mmlu\"\nallow = true\n\n[[accessibility.cache]]\nrepo = \"adept/fuyu-8b\"\nallow = false\n```\n- `offline`: Sets whether the Olah mirror site enters offline mode, no longer making requests to the Hugging Face official site for data updates. However, cached repositories can still be downloaded.\n- `proxy`: Determines if the repository can be accessed through a proxy. By default, all repositories are allowed. The `repo` field is used to match the repository name. Regular expressions and wildcards can be used by setting `use_re` to control whether to use regular expressions (default is to use wildcards). The `allow` field controls whether the repository is allowed to be proxied.\n- `cache`: Determines if the repository will be cached. By default, all repositories are allowed. The `repo` field is used to match the repository name. Regular expressions and wildcards can be used by setting `use_re` to control whether to use regular expressions (default is to use wildcards). The `allow` field controls whether the repository is allowed to be cached.\n\n## Future Work\n\n* Administrator and user system\n* OOS backend support\n* Mirror Update Schedule Task\n\n## License\n\nolah is released under the MIT License.\n\n\n## See also\n\n- [olah-docs](https://github.com/vtuber-plan/olah/tree/main/docs)\n- [olah-source](https://github.com/vtuber-plan/olah)\n\n\n## Star History\n\n[![Star History Chart](https://api.star-history.com/svg?repos=vtuber-plan/olah\u0026type=Date)](https://star-history.com/#vtuber-plan/olah\u0026Date)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvtuber-plan%2Folah","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvtuber-plan%2Folah","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvtuber-plan%2Folah/lists"}