{"id":28496039,"url":"https://github.com/sonarsource/public-git-sync","last_synced_at":"2025-07-02T10:31:56.059Z","repository":{"id":30891851,"uuid":"126302620","full_name":"SonarSource/public-git-sync","owner":"SonarSource","description":"Private to public Git repository synchronization","archived":false,"fork":false,"pushed_at":"2025-04-10T14:44:07.000Z","size":56,"stargazers_count":6,"open_issues_count":5,"forks_count":3,"subscribers_count":22,"default_branch":"master","last_synced_at":"2025-06-08T11:51:24.312Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SonarSource.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-03-22T08:19:28.000Z","updated_at":"2025-02-19T08:23:37.000Z","dependencies_parsed_at":"2024-08-05T18:02:15.571Z","dependency_job_id":"9a18581f-94b9-4e8e-bb8d-898a2c201f81","html_url":"https://github.com/SonarSource/public-git-sync","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/SonarSource/public-git-sync","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SonarSource%2Fpublic-git-sync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SonarSource%2Fpublic-git-sync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SonarSource%2Fpublic-git-sync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SonarSource%2Fpublic-git-sync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SonarSource","download_url":"https://codeload.github.com/SonarSource/public-git-sync/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SonarSource%2Fpublic-git-sync/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263120750,"owners_count":23416880,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-08T11:38:29.847Z","updated_at":"2025-07-02T10:31:56.048Z","avatar_url":"https://github.com/SonarSource.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Public-Git-Sync\n\naka the \"Private to public Git repository synchronization tool\"\n\nThis Git repository contains the source of the Public-Git-Sync tool.\n\n## About\n\nThe purpose of Public-Git-Sync is to replicate the history of a branch with a given name from a private repository into a branch with the same name in a public repository.\n\nPublic-Git-Sync works one branch at a time.\n\nPublic-Git-Sync relies on the assumption that content of directory `private` in the root of the private repository must not be made public.\nPublic-Git-Sync strips any private content from commits, and removes any commit which contains only private content, before pushing them to the public repository.\n\nThe tool is made of 4 bash scripts to be called, **in order**:\n\n1. `initialize_branch_synchronization.sh` to be called only once per branch to initialize Public-Git-Sync for this branch (see below \"initalization\")\n2. `sync_public_branch.sh` to be called each time to synchronize a branch (see below \"synchronization\")\n3. `finish_sync_public_branch.sh` called automatically by `sync_public_branch.sh` unless a manual operation is needed to complete synchronization (see below \"cherry-pick conflicts\")\n4. `commit_sync_public_branch.sh` to be called after `sync_public_branch.sh` to commit synchronization to private and public remote repositories\n\n**All scripts must be run from the root directory of a clone of the private repository**\n\n### sonar-enterprise branch\nThe `sonar-enterprise` repository uses a modified version of this script, which excludes the GitHub Action workflows folder (`.github/workflows`).\nIt is available on the `sonar-enterprise` branch.\n\n# How to use\n\n## Prerequesites\n\n1. a private repository (eg. sonarsource/sonar-enterprise)\n2. a public repository (eg. sonarsource/sonarqube)\n3. a branch to synchronize in the private repository (eg. master, branch-7.9, ...)\n4. `bash` and `git` (configured to access both repositories)\n\n## Initialization\n\nPublic-Git-Sync must be initialized for each branch (later refered to as `[branch_name]`).\n\nThe `initialize_branch_synchronization.sh` script will create:\n\n1. a branch `[branch_name]` to synchronize to in the public repository\n2. a branch named `public_[branch_name]` in the private repository which is the public version of `[branch_name]` in the private repository\n3. a pair of initial synchronization Git references (these references are used to track previous synchronization points)\n\nThese elements are mandatory for `sync_public_branch.sh` to run.\n\nTo do the initialization, you will need:\n\n1. the name of the branch to synchronize\n2. a pair of commits which you know are synchronized with each other (aka. a pair of \"synchronized commits\")\n  * when synchronizing master, it will likely be the first commit ever in master (assuming it has no content in `private` directory already)\n  * when synchronizing another branch than master, assuming master is synchronized, it will be the synchronized commits of the fork point of that branch with master\n\nHere is an example of initialization with branch `branch-7.9` of the `sonar-enterprise` repository:\n\n```bash\ngit clone git@github.com:sonarsource/public-git-sync.git synchronization\ngit clone git@github.com:sonarsource/sonar-enterprise.git\ncd sonar-enterprise\n../synchronization/initialize_branch_synchronization.sh sq git@github.com:sonarsource/sonarqube.git branch-7.9 9d45cf3bd58bafc6acbfac447fc70a1b5fe2f050 0dc7f1ec3d08fd5cd39e23b35b236bbfa7ec8ae6\n```\n\n### Note on the synchronized commits\n\nThe synchronized commit of the private branch must (obviously) exist in the private repository. It has to belong to the synchronized branch in the private repository (this is not currently enforced) but it doesn't have to be the HEAD of it.\n\nThe synchronized commit of the public branch can, but doesn't have to, exist in the private repository.\n\nIf it exists in the private repository, `initialize_branch_synchronization.sh` will be able to create the synchronized branch in the public repository for you. If it doesn't, then synchronized branch in the public repository must exist and this commit must be the HEAD of it.\n\n### Note on branches\n\nCurrent implementation of `initialize_branch_synchronization.sh` will fail if `[branch_name]` or `public_[branch_name]` exist and can't be updated to the specified commits (ie. it performs a default `git push`).\n\n\n\n## Synchronization\n\n`sync_public_branch.sh` is the bash script you want to call on a regular basis, most likely through an automated task.\n\nTo run this script, you will need:\n\n1. the name of the branch to synchronize\n2. to have initialized Public-Git-Sync on this branch (see above \"Initialization\")\n\n`sync_public_branch.sh` does not make any change to the remote public and private repositories. All changes are local to the current clone of the private repository.\n\nTo \"commit\" these changes to the remote public and private repositories, you must call `commit_sync_public_branch.sh` (or do it manually, not advised).\n\nHere is an example of synchronization of branch `branch-7.9` of the `sonar-enterprise` repository:\n\n```bash\ngit clone git@github.com:sonarsource/public-git-sync.git synchronization\ngit clone git@github.com:sonarsource/sonar-enterprise.git\ncd sonar-enterprise\n../synchronization/sync_public_branch.sh sq git@github.com:sonarsource/sonarqube.git branch-7.9\n../synchronization/commit_sync_public_branch.sh sq branch-7.9\n```\n\n### cherry-pick conflicts\n\nIt may happen that `sync_public_branch.sh` fails to perform automatically all operations to create the public version of `[branch_name]`. Most specifically, these would be `cherry-pick` commands which can't be made automatically by Git.\n\nIn such case, `sync_public_branch.sh` will stop prematuraly with a message such as the following:\n\n```\nResolve the current cherry-pick, and then run bash cherry-pick.branch-7.9.8325.sh to continue.\nFinally, run 'finish_sync_public_branch.sh branch-7.9' to complete the recovery.\n```\n\nTo investigate and resolve the conflict:\n\n* use `git status`, `git diff`, edit files as necessary and then use `git cherry-pick --continue`\n* execute the indicated script: `./cherry-pick.branch-7.9.8325.sh` which will apply the following cherry-pick commands\n* execute `finish_sync_public_branch.sh branch-7.9` as indicated to finalize the sync\n* finally, as when `sync_public_branch.sh` succeeds, you will have to call `commit_sync_public_branch.sh` to push synchronization results to public and private remote repositories\n\n### Note about `finish_sync_public_branch.sh`\n\nUnder regular operation of Public-Git-Sync, you won't have to call the `finish_sync_public_branch.sh` script.\n\nThis script only exists appart from `sync_public_branch.sh` to provide an easy procedure to recover a cherry-pick conflicts (see above).\n\n\n# How it works\n\n## Replicating a tree of commits\n\nWhat Public-Git-Sync basically does is to replicate a tree of commits into another tree of commits (so, keeping the order) excluding all changes which apply to the `private` directory.\n\nTo illustrate, take the following trees of two synchronized private and public branches:\n\n\u003e `A`, `B`, `C`, ... are the sha1 of the commits and `change A`, `change B`, `change C`, ... are the title of the commits\n\n```\n  public branch               private branch\n       A change A                  A' change A\n       |                           |\n       B change B                  B' change B\n       |                           |\n       C change C                  |\n       |                           |\n       D change D                  D' change D\n       |                           |\n       E change E                  E' change E\n       |                           |\n       F change F                  F' change F\n```\n\nSeveral observations:\n\n1. the title and the order of the commits are preserved (so are the authors, emails, etc.)\n2. there is no commit `C'` because this commit contained only private content\n3. any of the commits `B'` to `F'` could have the same content as respectively, `B` to `F`, they will still have a different sha1 because their parent is not the same as in the public branch\n\n### Notes on merge commits\n\nIn the example above, each commit has a single parent. This is not the case for merge commits.\n\nPublic-Git-Sync handles merge commits by taking only one parent into account. This implies that merge commit will be there, unless it's empty/has only private content, and will lose the information of the other parent.\n\n\n### Technical details\n\nPublic-Git-Sync relies on Git commands `filter-branch` and `cherry-pick` and a pair of work branches (called `[branch_name]_work` and `public_[branch_name]_work`).\n\nThe `sync_public_branch.sh` script does the following (on top of some sanity checks):\n\n1. (re)create branch `[branch_name]_work` as a copy of `[branch_name]`\n2. use `git filter-branch` to remove any private content from the last synchronized commit to the HEAD of `[branch_name]_work`\n  * this creates commits without private content\n  * and also some empty commits which had only private content\n3. (re)create branch `public_[branch_name]_work` from `public_[branch_name]`\n4. apply in order, with `git cherry-pick`, the filtered commits from `[branch_name]_work` into `public_[branch_name]`\n\t* these are the operations which may require manual fix by user to complete\n\t* this branch can contain empty commits at this stage\n\nThe `finish_sync_public_branch.sh` is then called and does the following:\n\n1. remove empty commits from `public_[branch_name]_work` using `git filter-branch`\n\t* in the example above, `C'` did exist for a while but is deleted at this stage\n2. update local branch `public_[branch_name]` to now be the same as `public_[branch_name]_work`\n3. perform a sanity check to ensure **no private content are present in `public_[branch_name]`**\n4. create synchronization refs to the HEADs of `[branch_name]` and `public_[branch_name]`\n\nSo far, all changes made are local to the current clone of the private repository.\n\nThey can be reviewed or discarded by simply deleting the clone.\n\nTo push the changes to the remote public and private repositories, use `commit_sync_public_branch.sh` which does the following:\n\n1. push branch `public_[branch_name]` to private repository\n2. push branch `[branch_name]` to public repository with content of `public_[branch_name]`\n3. push the synchonization refs\n\n### Note on `commit_sync_public_branch.sh`\n\n`commit_sync_public_branch.sh` will have no effect if there is nothing to push. It can be safely called after a call of `sync_public_branch.sh` which found nothing to synchronize.\n\n`commit_sync_public_branch.sh` will push all synchronization refs not yet present in remote private repository, not only those created by the previous synchronization (ie. it's stateless). This implies it can be used to push at once multiple synchronizations (this is not advised, though).\n\n\n## Keeping track of synchronized commits\n\nThe second important feature of Public-Git-Sync is to keep track of synchronized commits.\n\nThis way, `sync_public_branch.sh` can:\n\n1. synchronize only the new commits from the private branch\n2. detect there is nothing new to synchronize\n3. prevent synchronisation time to increase (linearily or worse) with the number of commits in `[branch_name]`\n4. prevent possible rewriting (new sha1) of already synchronized commits and a lot of messy consequences that would have\n5. rely on safe and simple `git push --forward-only` to publish to the public repository\n\nPublic-Git-Sync achieves this by storing Git references (later referenced to as \"synchronization refs\") in the private repository. These references are stored in `refs/public_sync`.\n\nSynchronization refs always go by two: one pointing to a commit in the private branch, one pointing to a commit in the public branch.\n\nThese two commits are referred to as \"synchronized commits\". You can consider synchronized commits as one commit being the public version of the other.\n\nLet's have a look at a sequence of synchronization calls over time for the example above.\n\nInitialization\n\n```\n  public branch               private branch                synchronized commits\n       A change A                  A' change A                   A -\u003e A'\n```\n\nSynchronizing `B`\n\n```\n  public branch               private branch                synchronized commits\n       A change A                  A' change A                   A -\u003e A'\n       |                           |\n       B change B                  B' change B                   B -\u003e B'\n```\n\nSynchronizing `C`\n\n\u003e since `C` has no public content, no new commit will be pushed to the public branch. The synchronized commit for `C` is therefor the current HEAD of the public branch: `B'`\n\n```\n  public branch               private branch                synchronized commits\n       A change A                  A' change A                   A -\u003e A'\n       |                           |\n       B change B                  B' change B                   B -\u003e B'\n       |                           |\n       C change C                  |                             C -\u003e B'\n```\n\nSynchronizing `F`\n\n\u003e since the last synchronized commit is `C`, all commits since `C` will be synchronized: `D`, `E` and `F`. Public-Git-Sync will create syncronization refs only for the HEADs: `F` and `F'`.\n\n```\n  public branch               private branch                synchronized commits\n       A change A                  A' change A                   A -\u003e A'\n       |                           |\n       B change B                  B' change B                   B -\u003e B'\n       |                           |\n       C change C                  |                             C -\u003e B'\n       |                           |\n       D change D                  D' change D\n       |                           |\n       E change E                  E' change E\n       |                           |\n       F change F                  F' change F                   F -\u003e F'\n```\n\n### Technical details\n\nSynchronization refs are stored in the ref directory `refs/public_sync`.\n\nRefs to synchronized commits are stored in a sudirectory which name is a timestamp, eg.: `2019-10-16_13-41-03`. Timestamp allows to sort references and identify the most recent ones. The timestamp is basically the time when `sync_public_branch.sh` is called.\n\nIn this directory, each synchronized commit has its reference named after the branch it belongs to.\n\nCurrently, synchronization refs are stored forever. An improvement issue ([#6](https://github.com/sonarsource/public-git-sync/issues/6)) has been filled.\n\n### Note on clone and fork\n\nGit clone does not pull synchronization refs by default. They have to be pulled explicitly.\n\nExample:\n\n```\ngit fetch --no-tags origin \"+refs/public_sync/*:refs/public_sync/*\"\n```\n\nGithub does not copy synchronization refs when forking a repository. They will have to be copied manually.\n\n## Contributing\n\nCommit should be pushed at first on:\n- `master` if applicable\n- `sonar-enterprise` if specific\n\nThen, if `master` has new commits, a \"GitHub rebase and merge\" should be performed from `master` to `sonar-enterprise`.\n\n---\nStarting from this situation, when working with protected branches\n\n```\n* M3 - (master) New commits on master\n* M2\n* M1\n| * S3 - (tag: v0.2.0, sonar-enterprise) Custom commits on sonar-enterprise\n| * S2\n| * S1\n|/  \n* 00 - Common commits\n*\n```\n\nThen, perform a GitHub \"Rebase and Merge\", that is actually a **complete cherry-pick**\n```\n* M3' - (sonar-enterprise) New commits from master\n* M2'\n* M1'\n* S3 - (tag: v0.2.0) Custom commits on sonar-enterprise\n* S2\n* S1\n| * M3 - (master) New commits on master\n| * M2\n| * M1\n|/  \n* 00 - Common commits\n* \n```\n\n## License\n\nCopyright 2018-2020.\n\nLicensed under the [GNU Lesser General Public License, Version 3.0](http://www.gnu.org/licenses/lgpl.txt)\n\n## Authors\n\nSébastien Lesaint and Janos Gyerik\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsonarsource%2Fpublic-git-sync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsonarsource%2Fpublic-git-sync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsonarsource%2Fpublic-git-sync/lists"}