https://github.com/ibob/git-lfs-download
Download full or partial git-lfs repos without temporarily using 2x disk space
https://github.com/ibob/git-lfs-download
Last synced: 10 months ago
JSON representation
Download full or partial git-lfs repos without temporarily using 2x disk space
- Host: GitHub
- URL: https://github.com/ibob/git-lfs-download
- Owner: iboB
- License: mit
- Created: 2023-07-19T08:09:57.000Z (about 3 years ago)
- Default Branch: master
- Last Pushed: 2023-10-13T03:13:38.000Z (almost 3 years ago)
- Last Synced: 2025-01-01T01:50:23.132Z (over 1 year ago)
- Language: Ruby
- Homepage:
- Size: 17.6 KB
- Stars: 30
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# git-lfs-download
A CLI tool which downloads a git-lfs repo fully or partially without temporarily using 2x disk space and without using any disk space for lfs files which are not downloaded.
**Usage:**
*A Ruby interpreter is needed for the tool. Ruby 3.0.0 or later is required*
`$ ./git-lfs-download [--match ] [--without ]`
By default the full repository is downloaded. Adding `--match` patterns will only include files which match any of them. Adding `--without` patterns will exclude files which match any of them.
## Task
You want to download a git-lfs repo
## Problem
Using `$ git clone` will download the repo, but after the download it will copy all files from the `.git` directory to the clone directory. This uses at least double the disk space of the repo.
You may not have 2x disk space. It could be hundreds of gigabytes: game assets, weights for huge neural networks...
Time is wasted in copying these huge files. Note that this will likely be negligible compared to the download time, but still, unpleasant.
## Proper solution
Use [git-archive](https://git-scm.com/docs/git-archive) and pipe its output to an extracting program.
`$ git archive --format=tar --remote= HEAD | tar xf -`
This will also solve the sane problem if you encounter it with non git-lfs files.
Unfortunately git-archive can exert quite the cpu toll on the server and most git providers (like GitHub and Huggingface) have it disabled server-side rendering it unusable.
## Task 2
You want to download only some large files from a git lfs repo
## Proper solution
Manually `$ git lfs pull --include `. This is pretty cumbersome especially if there are many files, or files to filter-*out*.
Disk space x2 will be required.
## Hack which this project implements
* Clone a shallow repo skipping lfs smudge
* Parse `.gitattributes` to identify lfs files
* Filter lfs files according to provided options (if any)
* Fetch lfs files
* `mv` the files from `.git/lfs/objects` to their places in the git root
* `$ rm -rf .git`
## Copyright
[](https://opensource.org/licenses/MIT)
This software is distributed under the MIT Software License.
See accompanying file LICENSE or copy [here](https://opensource.org/licenses/MIT).
Copyright © 2023 [Borislav Stanimirov](http://github.com/iboB)