{"id":22795820,"url":"https://github.com/dynamist/subgit","last_synced_at":"2026-05-16T00:37:48.148Z","repository":{"id":43454986,"uuid":"225139243","full_name":"dynamist/subgit","owner":"dynamist","description":"Sub-git repo handler","archived":false,"fork":false,"pushed_at":"2024-01-11T10:09:50.000Z","size":164,"stargazers_count":0,"open_issues_count":10,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-09-25T10:52:29.475Z","etag":null,"topics":["git","git-subcommand","python-library","python3","repo"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dynamist.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-12-01T09:55:27.000Z","updated_at":"2024-05-21T13:10:54.000Z","dependencies_parsed_at":"2025-02-05T20:27:07.946Z","dependency_job_id":"8e3fee6b-1433-49ef-92c6-fc014c7150ff","html_url":"https://github.com/dynamist/subgit","commit_stats":null,"previous_names":["dynamist/sgit"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/dynamist/subgit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dynamist%2Fsubgit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dynamist%2Fsubgit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dynamist%2Fsubgit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dynamist%2Fsubgit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dynamist","download_url":"https://codeload.github.com/dynamist/subgit/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dynamist%2Fsubgit/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":281563757,"owners_count":26522696,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-29T02:00:06.901Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["git","git-subcommand","python-library","python3","repo"],"created_at":"2024-12-12T05:08:47.762Z","updated_at":"2025-10-29T04:38:19.138Z","avatar_url":"https://github.com/dynamist.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Subgit\n\nThe name came originally from *Submodule Git*, which became *Subgit*.\n\nThe main purpose of this tool is to be used as a sync for cases to pull together a set of different Git repos into a deterministic directory tree. It is used to collect various individual parts into a whole.\n\nThe main two advantages over other solutions is that when compared to the normal git submodules, you can more easily specify and avoid having to commit into your tree every single update that you want to have in your subgit checkout. In a submodule solution where you want to update and track the current and latest commit for the maste branch, you do not have to update your configuration file each time you make a new commit in the child repo. In the cases where you are pointing to tags and stable releases of a software, there is not much difference. When comparing to other tools that manipulates and pulls in each remote repos entire tree into your tree, this tool avoids that and dont care about that scenario. The only file you have to commit into a repo that you want sub repos to be exposed to is the `.subgit.yml` config file.\n\nThis tool has been primarly constructed to be a part inside a CI/CD solution where you want to in a build step, clone the repo that contains the `.subgit.yml` config file, clone out whatever branches or tags for all the child repos and then perform some action or build step or test or similar function while using all of the repos from one spot.\n\nIn the event of you having a configuration file with another name than `.subgit.yml`, you can easily add the `-c`, or `--conf`, flag to point to a an alternative path of the configuration file.\n\n```bash\nsubgit pull -c path/to/conf-file\n```\n\n## Usage\n\nSubgit installs a `subgit` command. For the Git afficinados there is a Git [subcommand](https://github.com/topics/git-subcommand) which allows accessing Subgit by running `git sub [...]`. \n\n\n### !!WARNING!!\n\nSubgit do not leave any guarantees that it will NOT MODIFY or THROW AWAY any local changes or modifications done to the git repo that it checks out and handles. Subgit is `NOT` in any way capable of commiting any changes or pushing any changes to your git remotes. This tool is only intended to be able to pull in and together a set of other git repos into one folder.\n\n\n### Quickstart\n\nInstall the tool with `pip install subgit`. The tool requires `python\u003e=3.8.0`, but we always recommend the latest major release as standard.\n\nCreate a temporary folder where you want your sub repos to be located\n\n```bash\nmkdir /tmp/subgit; cd /tmp/subgit\n```\n\nInitialize an empty `.subgit.yml` config file.\n\n```bash\nsubgit init\n\n# or optionally you can specify the initial repo and clone url you want to be added with\nsubgit init pykwalify git@github.com:Grokzen/pykwalify.git\n```\n\nInspect the content by looking inside the `.subgit.yml` config file\n\n```bash\ncat .subgit.yml\n\n# This will show the default config in an empty config file\nrepos: [ ]\n```\n\nTo add any number of git repos that you want to clone by manually editing the `.subgit.yml` configuration file.\n\nNext step is to make the initial git clone/pull of all repos in the config file and move the repo to the specified revision. Running `subgit pull` command without any arguments will update all repos defined in the configuration file. If your repo is not present on disk it will make a initial `git clone` before moving to your selected revision.\n\n```bash\n# Pull all repos in your config file\nsubgit pull\n\n# Or you can pull a specific repo from your config file\nsubgit pull pykwalify\n```\n\nSubgit relies on your OpenSSH config or other Git config is properly setup and configured in such a way that you can clone the Git repo without having to specifying any credentials.\n\nYou can view a summary of your current repo and config state by running `subgit status`.\n\n\n## Fetch changes in a repo\n\nIf you want to `git fetch` all or a subset of git repos in your config then you can use the `subgit fetch` command. The benefit of doing a fetch is that you can fetch home all changes to a set of git repos but you do not have to update and move each repo to a new commit. In general git operations, it is always more safe to run `git fetch` before you do a checkout or `git pull` to update your local cloned repos. This allows you to inspect the changes incomming before commiting to pulling them.\n\nThe fetch command supports the selection of either all repos or a subset of repos. The fetch command will never prompt the user asking if they want to do a update as fetch is considered a non-descrutive command.\n\n```bash\n# Fetch all repos in sequence\nsubgit fetch\n\n# Fetch one specified repo\nsubgit fetch pykwalify\n```\n\n## Delete pulled repos\n\nYou can delete local copies of your repos by using `subgit delete` command. This will only remove your repos locally, also only if they're considered *clean*. This means that there are no commited changes or untracked files. You will get no explicit warning about what changes makes the repos *dirty*, except the specific repo which contain the changes.\n\nThe `subgit delete` command supports the selection of either all repos or a subset of repos.\n\n```bash\n# Delete all repos in sequence\nsubgit delete\n\n# Delete one specified repo\nsubgit delete pykwalify\n```\n\n## Inspect repos from Github or Gitlab\n\nIf the user wants all repos from a group or account to be written to a file, subgit offers a way to do this by using `subgit inspect`. This prints all repos  subgit config file format (YAML) to stdout. Redirect the output to a file to get a correct subgit configuration file. By default `subgit inspect` excludes archived repos and those not owned by the specified user. By adding the `--archived` flag will filter only archived repos.\n\n**Authentication for Github CLI**\n\nDownload the Github CLI from https://github.com/cli/cli/releases/\n\nOnce installed you should have access to `gh` command in your terminal.\n\nLogin to your Github account with `gh auth login`. To verify you have correct access you can run `gh repo list` and it should list your personal Github repos.\n\n**Authentication for Gitlab CLI**\n\nInstall the Gitlab CLI from https://python-gitlab.readthedocs.io/en/stable/\n\nOnce installed you should have access to `gitlab` command in your terminal.\n\nNext generate a new private API token for your account in gitlab. Create one here https://gitlab.com/-/profile/personal_access_tokens and set it to *readonly permissions* for everything including API access. \n\nExport your token in your terminal with `export GITLAB_PRIVATE_TOKEN=\u003cYOUR_TOKEN\u003e` and it will allow for API access. Test this by running `gitlab project list`\n\n```bash\n# List all repos from either Github or Gitlab\nsubgit inspect github \u003cYOUR_USERNAME\u003e\nsubgit inspect gitlab \u003cYOUR_USERNAME\u003e\n\n# Optionally redirect the output to a subgit configuration file\nsubgit inspect github \u003cYOUR_USERNAME\u003e \u003e .my-gh-repos.yml\nsubgit inspect gitlab \u003cYOUR_USERNAME\u003e \u003e .my-gl-repos.yml\n```\n\n## Reset changed repos\n\nYou can reset repos in which changes have been made such as new files or new commits using `subgit reset` command. Using no particular option, `subgit reset` will reset the index to the origin pointer/reference. I will compare your working directory to that of the remote. If there are no changes, it will simply tell you that the current repo is clean. If there are any untracked files (files that are not added or commited) the '--hard' flag has to be appended to the command in order to force a reset.\n\n```bash\n# Reset all repos in sequence if they are dirty\nsubgit reset\n\n# Reset one specified repo if it's dirty\nsubgit reset pykwalify\n```\n\n## Clean changed repos\n\nWith `subgit clean` you can clean a repo from dirty files if `subgit reset` hasn't done the job. By using `subgit clean` you have a variety of options. Either use `--force` flag to only delete untracked files, without recursively removing untracked directories. Or append the `-d` flag in addition to `--force` to also remove directories. There is also a `--dry-run` flag that only shows you what would be removed, if the command was successfully executed.\n\n```bash\n# Clean all dirty repos from untracked files in sequence, skipping directories.\nsubgit clean --force\n\n# Clean all dirty repos from untracked files, including directories.\nsubgit clean --force -d\n\n# Cleans 'pykwalify' if it's dirty\nsubgit clean pykwalify --force -d\n\n# Shows only what would be done if run successfully\nsubgit clean pykwalify --force -d --dry-run\n```\n\nCreate a new git branch from the `master` branch and commit all changes you want to contribute in there. After your work is complete submit a Merge Request on gitlab back to the `master` branch. Ask a collegue for review and approval of the changes. Also if you are not he maintainer or your reviewer is not the maintainer, it is always good to ping and ask that person as well before mergin big changes. Smaller changes or fixes do not require this, but it is always encouraged to do this for all MR:s.\n\nRequirement for a merge request\n\n- Pytests should always pass and be green\n- Any gitlab actions or PR specific tests/validation shold be green\n- No merge conflicts with master branch, if you have then you either merge master into your branch and resolve conflicts, or you rebase your branch ontop of master branch\n- Always do basic useability tests with most common commands as tests do not always show errors with everything\n\n\n## Sparse checkout\n\nThis feature requires you to have git version `2.25.0` or later\n\nA in-depth blog about sparse checkout in general can be read [here by github.blog](https://github.blog/2020-01-17-bring-your-monorepo-down-to-size-with-sparse-checkout/)\n\nThe main idea with sparse checkout feature is to reduce the size of mono repos into more manageable chunks of both output files in your file tree, but also to enable less git refs to clone.\n\nBy adding the key `sparse.paths` to your repo config, you define which folders should be pulled into the file tree.\n\n```\n# Example config file\nrepos:\n  - name: phabfive\n    url: git@github.com:dynamist/phabfive.git\n    revision:\n      branch: \"master\"\n    sparse:\n      paths:\n        - \"phabfive/\"\n        - \"tests/\"\n ```\n\nThis example would clone the entire repo, all the git refs and utilize sparse checkout on the repo at that given revision (could be branch, commit or tag) and then configure the git clone to have the two folders `phabfive/` and `tests/` pulled into the file tree.\n\nThe paths you define works similar to how `.gitignore` works. What you define in reality is a filter that is matched against all files and folders in the checked out repo. This means that you can add paths like `*.py` or `*.md` or any other syntax that `.gitignore` syntax supports and it will be used as a filter for what files is pulled into the file tree. Remember that a subfolder that has a matching file within will be created even if that filename is not matching any provided path. Similarly, the other way around that if you specify `tests/` as a path it will include all sub folders \u0026 files even if they don't match the filter.\n\n\n### Run unitest suite \u0026 Tox\n\nTo run all unit tests run `pytest` from the root folder. It will use your default python environment you are in right now.\n\nWe use tox to run multi python python tests to validate our test suite works our supported python version range.\n\nTox is installed in the pip extras block `.[test]`. To run all tests/linters against all versions simply run `tox` from your cli. To run a specific python version run `tox -e py310`. Most linux distros do not have all python versions installed by default.\n\nThis guide https://linuxize.com/post/how-to-install-python-3-8-on-ubuntu-18-04/ can help you to install older python versions either from dead-snakes repos or from source code.\n\n\n### Extra development option flags\n\nYou can currently use the following envroinment flags `DEBUG=1` to get additional debugging and detailed information about any uncaught exceptions and detailed information about cli flags etc.\n\nTo get into a PDB debugger on any uncaught exceptions you can set the environment flag `PDB=1` and that will take you into a pdb debugger at the cli exit point so you can dig into any issues you might have.\n\n\n## Python support guidelines\n\nWe follow the method of always suporting the latest released major version of python and two major versions back. This gives us backwards compatibility for about 2-3 years depending on the release speed of new python versions.\n\nWhen a new major version is incorporated, tested and validated it works as expected and that dependencies is not broken, the update for new python version support should be released with the next major version of subgit. A major python version should never be dropped in a minor version update.\n\nPython 2.7 is EOL and that version will not be supported moving forward.\n\n\n## Build and publish a release to pypi\n\nEnsure that you have updated all version tags to the desired version number for the next release. Either update the minor release or update the major release depending on the changes introduced within this release. Note that all support documents like releasenotes and other help documents must be completed and updated before this these build steps should be done. Once the build is uploaded to pypi it can't be reuploaded with a new build without bumping the version number again.\n\nCreate a fresh virtualenv. Install required build dependencies. Build the pckages\n\n```bash\npython -m pip install --upgrade setuptools wheel twine\n\n# Remove the old dist/ folder to avoid collisions and reuploading existing releases\nrm -r dist/\n\n# Generate the source build and wheels build\npython setup.py sdist bdist_wheel\n```\n\nTest and verify package can be installed locally within the virtualenv and that basic usage test cases works as expected and that simple cli oppreations like `subgit --help` and `subgit pull` and `subgit status` etc works as expected.\n\nNext step is to test upload the release to the test-pypi server to ensure that the package will be accepted by pypi official server. Note that this step requires you to have a valid test-pypi account over at https://test.pypi.org/ and that you have the permissions to manage and upload the subgit shared project. If you do not have these things setup, please talk to the repo maintaner to setup this.\n\nTo upload the previous built dist packages to pypi run\n\n```bash\npython -m twine upload --repository testpypi dist/*\n```\n\nInput your username + password to the prompt. Note that username is case sensetive.\n\nValidate your release was uploaded correct by visiting https://test.pypi.org/manage/project/subgit/ and look for the version number you uploaded. If this looks good and works then continue to upload the final release to regular pypi.org.\n\n```bash\n# Upload to pypi.org\npython -m twine upload dist/*\n```\n\nInput your username + password but this time for your account on pypi.org and NOT test.pypi.org. These are two separate accounts. Same here, validate the build by going to https://pypi.org/manage/project/subgit/ and once these files is uploaded they are published and can't be rebuilt/replaced w/o making a post fix release. Read up onn python.org own release documentation for those steps.\n\nFinally you should create the git tag for the git commit that you built. Run `git tag \u003csemver-version\u003e` to make the tag and push them with `git push --tags` to publish the tag. Rembmer to only do tags on the master branch.\n\n\n## Project details\n\n|   |   |\n|---|---|\n| python support         | 3.8, 3.9, 3.10, 3.11 |\n| Source code            | https://github.com/dynamist/subgit |\n| Changelog              | https://github.com/dynamist/subgit/blob/master/CHANGELOG.md |\n| Issues                 | https://github.com/dynamist/subgit/issues |\n| Projects page          | https://github.com/dynamist/subgit/projects/1\n| pypi                   | https://pypi.python.org/pypi/subgit/ |\n| License                | `Apache-2.0` https://github.com/dynamist/subgit/blob/master/LICENSE |\n| Copyright              | `Copyright (c) 2019-2023 Dynamist AB` |\n| git repo               | `git@github.com:dynamist/subgit.git` |\n| install stable         | `pip install subgit` |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdynamist%2Fsubgit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdynamist%2Fsubgit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdynamist%2Fsubgit/lists"}