https://github.com/w0rmr1d3r/cloner
A tool to clone efficiently all the repos in an organization
https://github.com/w0rmr1d3r/cloner
clone git github multiprocessing multithreading organization python python3
Last synced: 7 months ago
JSON representation
A tool to clone efficiently all the repos in an organization
- Host: GitHub
- URL: https://github.com/w0rmr1d3r/cloner
- Owner: w0rmr1d3r
- License: mit
- Created: 2021-11-03T18:34:38.000Z (about 4 years ago)
- Default Branch: master
- Last Pushed: 2024-04-12T09:57:00.000Z (over 1 year ago)
- Last Synced: 2024-04-12T16:13:23.940Z (over 1 year ago)
- Topics: clone, git, github, multiprocessing, multithreading, organization, python, python3
- Language: Python
- Homepage: https://pypi.org/project/wr-cloner/
- Size: 404 KB
- Stars: 8
- Watchers: 4
- Forks: 0
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- Funding: .github/FUNDING.yml
- License: LICENSE
- Security: SECURITY.md
Awesome Lists containing this project
README
# cloner
A tool to clone efficiently all the repos in an organization
[](https://pypi.org/project/wr-cloner/)
[](https://github.com/w0rmr1d3r/cloner/releases)
[](https://github.com/w0rmr1d3r/cloner/actions/workflows/ci.yml)
[](https://github.com/w0rmr1d3r/cloner/actions/workflows/codeql.yml)
[](https://scorecard.dev/viewer/?uri=github.com/w0rmr1d3r/cloner)
[](https://www.bestpractices.dev/projects/9982)
[](https://pypistats.org/packages/wr-cloner)
[](https://pepy.tech/projects/wr-cloner)
## Install
_In PyPi it's called **wr-cloner**, cloner was already taken :sad:_
```bash
pip install wr-cloner
```
## Usage
_Note: If using cloner after cloning the project and not a PyPi package, add "python" at the start.
There's an example of that in the examples section._
```text
Usage: cloner [OPTIONS] GITHUB_ORGANIZATION
A tool to clone efficiently all the repos in an organization.
Options:
--version Show the version and exit.
--token TEXT GitHub token to read private repos. This
parameter is needed when cloning from a
GitHub Enterprise server.
--ghe TEXT GitHub Enterprise URL. It needs the
GITHUB_ORGANIZATION parameter to clone repos
from there and the TOKEN option as well.
--threads INTEGER Number of threads and processes to use. For
maximum threads and processes on the system,
use '--max-threads' [default: 4]
--max-threads If declared, uses the maximum available
threads and processes in the system. As per
physical cores on the system cpu.
--logging [ERROR|WARNING|INFO|DEBUG]
Logging level [default: INFO]
--path TEXT Sets a path where to clone the repositories
(eg: ./another/path/)
--git-options TEXT Add options to the clone command (eg: --git-
options "--depth 1"). By default, clones
quietly (--quiet).
--ignore-archived If declared, will ignore archived repos when
cloning.
--ignore-template If declared, will ignore template repos when
cloning.
--ignore-fork If declared, will ignore fork repos when
cloning.
--exclude-repos TEXT Comma separated list of repository names to
exclude from cloning. Example:
"repository1,repository2".
--help Show this message and exit.
```
### Examples
```bash
# For github.com with 8 threads
cloner --threads 8 GITHUB_ORGANIZATION
# For github.com with the maximum threads on the system running
cloner --max-threads GITHUB_ORGANIZATION
# For GHE, default threads
cloner --ghe GHE_URL --token SUPER_SECURE_TOKEN GITHUB_ORGANIZATION
# Cloning with options
cloner --git-options "--depth 1" GITHUB_ORGANIZATION
```
### Examples if cloning the repo
```bash
# Install dependencies with
make install
# Adjust PYTHONPATH
export PYTHONPATH=$PYTHONPATH:.
# Use it
python cloner GITHUB_ORGANIZATION
```
## Contributing
Check the [CONTRIBUTING.md](CONTRIBUTING.md) file.
## Security
Follow the instructions in the [SECURITY.md](SECURITY.md) file.
## License
[MIT](https://github.com/w0rmr1d3r/cloner/blob/master/LICENSE)
## Known Issues
- Multithreading doesn't work to clone repos, since the `os.system` call is 1 for each PID. The splitting is done with
multithreading, the cloning with multiprocessing. Same amount of threads and processes.
- [Windows usage and support](docs/WINDOWS.md)
- If seeing `RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8)` either adjust your git config,
as
per [this StackOverflow issue](https://stackoverflow.com/questions/59282476/error-rpc-failed-curl-92-http-2-stream-0-was-not-closed-cleanly-protocol-erro).
If still persists, either decrease the number of threads using the `--threads` option or adjust the depth of cloning
with `--git-options "--depth 10"` (10 or any other small number).
## Star History
[](https://star-history.com/#w0rmr1d3r/cloner&Date)