{"id":13533113,"url":"https://github.com/hraban/tomono","last_synced_at":"2025-04-14T08:53:27.911Z","repository":{"id":37666122,"uuid":"49424963","full_name":"hraban/tomono","owner":"hraban","description":"Multi- To Mono-repository merge","archived":false,"fork":false,"pushed_at":"2024-10-27T23:19:30.000Z","size":312,"stargazers_count":879,"open_issues_count":4,"forks_count":137,"subscribers_count":28,"default_branch":"master","last_synced_at":"2025-04-07T01:09:25.871Z","etag":null,"topics":["emacs","git","literate-programming","monorepo-tooling","monorepository","noweb","org-babel"],"latest_commit_sha":null,"homepage":"https://tomono.0brg.net","language":"CSS","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hraban.png","metadata":{"files":{"readme":"Readme.org","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"hraban"}},"created_at":"2016-01-11T12:28:17.000Z","updated_at":"2025-04-01T14:48:13.000Z","dependencies_parsed_at":"2022-08-08T21:15:36.411Z","dependency_job_id":"6d557995-853a-4042-af19-f7011ae2eb47","html_url":"https://github.com/hraban/tomono","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hraban%2Ftomono","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hraban%2Ftomono/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hraban%2Ftomono/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hraban%2Ftomono/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hraban","download_url":"https://codeload.github.com/hraban/tomono/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248852107,"owners_count":21171839,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["emacs","git","literate-programming","monorepo-tooling","monorepository","noweb","org-babel"],"created_at":"2024-08-01T07:01:16.678Z","updated_at":"2025-04-14T08:53:27.881Z","avatar_url":"https://github.com/hraban.png","language":"CSS","funding_links":["https://github.com/sponsors/hraban"],"categories":["迁移工具","CSS","Shell"],"sub_categories":["Code ownership"],"readme":"#+TITLE:        Multi- to Monorepo Migration\n#+DESCRIPTION:  Migrate your multirepo to a monorepo using a bash script\n#+AUTHOR:       Hraban Luyat\n#+EMAIL:        hraban@0brg.net\n#+PROPERTY:     header-args       :noweb no-export :eval never\n#+EXPORT_FILE_NAME: index.html\n#+html_head:    \u003clink rel=stylesheet href=./style.css\u003e\n#+options: html-link-use-abs-url:nil html-postamble:auto html-preamble:t ':t toc:nil\n#+options: html-scripts:t html-style:t html5-fancy:t tex:html creator:t date:t author:nil\n#+html_doctype: html5\n#+html_container: div\n#+html_head_extra: \u003cmeta name=color-scheme content=\"light dark\"\u003e\n#+BIND: org-html-validation-link: nil\n\nThis script merges multiple independent tiny repositories into a single \"monorepo\". Every original repo is moved into its own subdirectory, branches with the same name are all merged. See [[#example][Example]] for the details.\n\nDownload the =tomono= script on [[https://github.com/hraban/tomono][github.com/hraban/tomono]].\n\n#+TOC: headlines 1\n\n* Features\n\n- 🕙 Full history of all your prior repos is intact, no changes to checksums\n- #️⃣ Signatures of old repos stay valid\n- 🔁 Create the monorepo and keep pulling in changes from your minirepos later\n- 🔀 Pull in entire new repos as you go, no need to prepare the whole thing at once\n- 🏷 Tags are namespaced to avoid clashes, but tag signatures remain valid\n- 🉑 Branches with weird names (slashes, etc)\n- 👥 No conflicts between files with the same name\n- 📁 Every project gets its own subdirectory\n\n* Usage\n#+TOC: headlines 1 local\n\nRun the =tomono= script with your config on stdin, in the following format:\n\n#+begin_example\n$ cat my-repos.txt\ngit@github.com:mycompany/my-repo-abc.git  abc\ngit@github.com:mycompany/my-repo-def.git  def\ngit@github.com:mycompany/my-lib-uuu.git   uuu  lib/uuu\ngit@github.com:mycompany/my-lib-zzz.git   zzz  lib/zzz\nhttps://gitee.com/shijie/zhongguo.git     中国\n#+end_example\n\nConcrete example:\n\n#+begin_src shell :exports code\n$ cat my-repos.txt | /path/to/tomono\n#+end_src\n\nThat should be all ✅.\n\n#+begin_comment\nYes #uselessuseofcat but it is clearer than \u003c \u003e ! # $) \u0026\u0026*!\u0026♨±⌘︎ to newbies.\n#+end_comment\n\n** Custom name for monorepo directory\n\nDon’t like =core=? Set a different name through an envvar before running the script:\n\n#+begin_src shell\nexport MONOREPO_NAME=the-big-repo\n#+end_src\n\n** Custom “master” / “main” branch name\n\nNo need to do anything. This script does not handle any master / main branch in any special way. It just merges whatever branches exist. Don’t have a “master” branch? None will be created.\n\nMake sure your own computer has the right branch set up in its =init.defaultBranch= setting.\n\n** Continue existing migration\n\nLarge teams can’t afford to “stop the world” while a migration is in progress. You’ll be fixing stuff and pulling in new repositories as you go.\n\nHere’s how to pull in an entirely new set of repositories:\n\n#+begin_src shell :exports code\n/path/to/tomono --continue \u003c my-new-repos.txt\n#+end_src\n\nMake sure you have your environment set up exactly the same as above. Particularly, you must be in the parent dir of the monorepo.\n\n** Tags\n\nTags are namespaced per remote, to avoid clashes. If your remote =foo= and =bar= both have a tag =v1.0.0=, your monorepo ends up with =foo/v1.0.0= and =bar/v1.0.0= pointing at their relevant commits.\n\nIf you don’t like this rewriting, you can fetch all tags from a specific remote to the top-level of the monorepo:\n\n#+begin_src shell :export code :results none\n$ git fetch --tags foo\n#+end_src\n\nBe prepared to deal with any conflicts.\n\n*** Lightweight vs. Annotated Tags\n\nN.B.: This namespacing works for all tags: lightweight, annotated, signed. However, for the latter two, there is one snag: an annotated tag contains its own tag name as part of the commit. I have chosen not to modify the object itself, so the annotated tag object thinks it still has its old name. This is a mixed bag: it depends on your case whether that’s a feature or a bug. One major advantage of this approach is that signed tags remain valid. But you will occasionally get messages like:\n\n#+begin_example\n$ git describe linux/v5.9-rc4\nwarning: tag 'linux/v5.9-rc4' is externally known as 'v5.9-rc4'\nv5.9-rc4-0-gf4d51dffc6c0\n#+end_example\n\nIf you know what you’re doing, you can force update all signed and annotated tags to their (nested) ref tag name with the following snippet:\n\n#+begin_src shell :export code :results none\ngit for-each-ref --format '%(objecttype) %(refname:lstrip=2)' | \\\n    sed -ne 's/^tag //p' |\n    GIT_EDITOR=true xargs -I + -n 1 -- git tag -f -a + +^{}\n#+end_src\n\nN.B.: this will convert all signed tags to regular annotated tags (their signatures would fail anyway).\n\nSource: [[https://github.com/mwasilew2/tomono/commit/16aa7918aa9d912a30b563152bda62c77414cbe1][GitHub user mwasilew2]].\n\n* Example\n:PROPERTIES:\n:CUSTOM_ID: example\n:END:\n#+TOC: headlines 1 local\n\nRun these commands to set up a fresh directory with git monorepos that you can later merge:\n\n** Initial setup of fake repos\n\n#+begin_src shell :exports code :eval never-export :results none :noweb-ref test-setup\nd=\"$(mktemp -d)\"\necho \"Setting up fresh multi-repos in $d\"\ncd \"$d\"\n\nmkdir foo\n(\n    cd foo\n    git init\n    git commit -m \"foo’s empty root\" --allow-empty\n    echo \"This is foo\" \u003e i-am-foo.txt\n    git add -A\n    git commit -m \"foo’s master\"\n    git tag v1.0\n    git checkout -b branch-a\n    echo \"I am a new foo feature\" \u003e feature-a.txt\n    git add -A\n    git commit -m \"foo’s feature branch A\"\n)\n\nmkdir 中文\n(\n    cd 中文\n    git init\n    echo \"你好\" \u003e 你好.txt\n    git add -A\n    git commit -m \"中文的root\"\n    git tag v1.0\n    git checkout -b branch-a\n    echo \"你好 from feature-a\" \u003e feature-a.txt\n    git add -A\n    git commit -m \"new 中文 feature branch A\"\n    git branch branch-b master\n    git checkout branch-b\n    echo \"I am an entirely new 中文 feature: B\" \u003e feature-b.txt\n    git add -A\n    git commit -m \"中文’s feature branch B\"\n)\n#+end_src\n\nYou now have two directories:\n\n- =foo= (branches: =master=, =branch-a=)\n- =中文= (branches: =master=, =branch-a=, =branch-b=)\n\n** Combine into monorepo\n\nAssuming the =tomono= script is in your =$PATH=, you can invoke it like this, from that same directory:\n\n#+begin_src shell :exports code :eval never-export :results none :noweb-ref test-run\ntomono \u003c\u003cEOF\n$PWD/foo foo\n$PWD/中文 中文\nEOF\n#+end_src\n\nThis will create a new directory, =core=, where you can find a git tree which looks somewhat like this:\n\n#+begin_example\n,*   b742af2 Merge 中文/branch-a (branch-a)\n|\\\n| * c05c53c new 中文 feature branch A (中文/branch-a)\n,* |   a51d138 Merge foo/branch-a\n|\\ \\\n| * | ebb490a foo’s feature branch A (foo/branch-a)\n,* | | a08fa18 Root commit for monorepo branch branch-a\n / /\n| | *   c53bf94 Merge 中文/branch-b (branch-b)\n| | |\\\n| | | * 5e7f4f5 中文’s feature branch B (中文/branch-b)\n| | |/\n| |/|\n| | * 2738327 Root commit for monorepo branch branch-b\n| |\n| | *   9a4b33a Merge 中文/master (HEAD -\u003e master)\n| | |\\\n| | |/\n| |/|\n| * | a9841a8 中文的root (tag: 中文/v1.0, 中文/master)\n|  /\n| *   b75840e Merge foo/master\n| |\\\n| |/\n|/|\n,* | 1515265 foo’s master (tag: foo/v1.0, foo/master)\n,* | f71fcde foo’s empty root\n /\n,* 7803cf5 Root commit for monorepo branch master\n#+end_example\n\n** Pull in new changes from a remote\n\nIt’s possible that while you’re working on setting up your fresh monorepo, new changes have been pushed to the existing single repos:\n\n#+begin_src shell :exports code :eval never-export :results none\n(\n    cd foo\n    echo New changes \u003e\u003e i-am-foo.txt\n    git commit -va -m 'New changes to foo'\n)\n#+end_src\n\nBecause their history was imported verbatim and nothing has been rewritten, you can import those changes into the monorepo.\n\nFirst, fetch the changes from the remote:\n\n#+begin_src shell :exports code :results none\n$ cd core\n$ git fetch foo\n#+end_src\n\nNow merge your changes using subtree merge:\n\n#+begin_src shell\ngit checkout master\ngit merge -X subtree=foo/ foo/master\n#+end_src\n\nAnd the updates should be reflected in the monorepo:\n\n#+begin_src shell :exports code :results none\n$ cat foo/i-am-foo.txt\nThis is foo\nNew changes\n#+end_src\n\nI used the branch master in this example, but any branch works the same way.\n\n** Continue\n\nNow imagine you want to pull in a third repository into the monorepo:\n\n#+begin_src shell :exports code :eval never-export :results none :noweb-ref test-setup\nmkdir zimlib\n(\n    cd zimlib\n    git init\n    echo \"This is zim\" \u003e i-am-zim.txt\n    git add -A\n    git commit -m \"zim’s master\"\n    git checkout -b branch-a\n    echo \"I am a new zim feature\" \u003e feature-a.txt\n    git add -A\n    git commit -m \"zim’s feature branch A\"\n    # And some more weird stuff, to mess with you\n    git checkout master\n    git checkout -d\n    echo top secret \u003e james-bond.txt\n    git add -A\n    git commit -m \"I am unreachable\"\n    git tag leaking-you HEAD\n    git checkout --orphan empty-branch\n    git rm --cached -r .\n    git clean -dfx\n    git commit -m \"zim’s tricky empty orphan branch\" --allow-empty\n)\n#+end_src\n\nContinue importing it:\n\n#+begin_src shell :exports code :eval never-export :results none :noweb-ref test-run\necho \"$PWD/zimlib zim lib/zim\" | tomono --continue\n#+end_src\n\nNote that we used a different name for this subrepo, inside the =lib= dir.\n\nThe result is that it gets imported into the existing monorepo, alongside the existing two projects:\n\n#+begin_example\n$ cd core\n$ git checkout master\nSwitched to branch 'master'\n$ tree\n.\n├── foo\n│   └── i-am-foo.txt\n├── lib\n│   └── zim\n│       └── i-am-zim.txt\n└── 中文\n    └── 你好.txt\n\n4 directories, 3 files\n$ git checkout branch-a\nSwitched to branch 'branch-a'\n$ tree\n.\n├── foo\n│   ├── feature-a.txt\n│   └── i-am-foo.txt\n├── lib\n│   └── zim\n│       ├── feature-a.txt\n│       └── i-am-zim.txt\n└── 中文\n    ├── feature-a.txt\n    └── 你好.txt\n\n4 directories, 6 files\n$ head **/feature-a.txt\n==\u003e foo/feature-a.txt \u003c==\nI am a new foo feature\n\n==\u003e lib/zim/feature-a.txt \u003c==\nI am a new zim feature\n\n==\u003e 中文/feature-a.txt \u003c==\n你好 from feature-a\n#+end_example\n\n* Implementation\n:PROPERTIES:\n:CUSTOM_ID: implementation\n:END:\n\n#+begin_quote\n(This section is best viewed in [[https://tomono.0brg.net/#implementation][HTML form]]; the GitHub Readme viewer misses some info.)\n#+end_quote\n\n#+TOC: headlines 1 local\n\nThe outer program structure is a flat bash script which loops over every repo supplied over stdin:\n\n#+CAPTION: top-level\n#+NAME: top-level\n#+BEGIN_SRC shell :tangle tomono :shebang \"#!/usr/bin/env bash\" :references yes\n\u003c\u003cinit\u003e\u003e\n\n# Note this is top-level in the script so it’s reading from the script’s stdin\nwhile \u003c\u003cwindows-fix\u003e\u003e read -r repourl reponame repopath; do\n    if [[ -z \"$repopath\" ]]; then\n        repopath=\"$reponame\"\n    fi\n\n    \u003c\u003chandle-remote\u003e\u003e\ndone\n\n\u003c\u003cfinalize\u003e\u003e\n\n# \u003c\u003ccopyright\u003e\u003e\n#+END_SRC\n\n** Per repository\n\nEvery repository is fetched and fully handled individually, and sequentially:\n\n1. fetch all the data related to this repository,\n2. immediately check out and initialise every single branch which belongs to that repository.\n\n#+CAPTION: handle-remote\n#+NAME: handle-remote\n#+BEGIN_SRC shell :references yes\ngit remote add \"$reponame\" \"$repourl\"\ngit config --add \"remote.$reponame.fetch\" \"+refs/tags/*:refs/tags/$reponame/*\"\ngit config \"remote.$reponame.tagOpt\" --no-tags\ngit fetch --atomic \"$reponame\"\n\n\u003c\u003clist-branches\u003e\u003e | while read -r branch ; do\n    \u003c\u003chandle-branch\u003e\u003e\ndone\n#+END_SRC\n\nThe remotes are configured to make sure that a default fetch always fetch all tags, and also puts them in their own namespace. The default refspec for tags is =+refs/tags/*:refs/tags/*=, as you can see that puts everything from the remote at the same level in your monorepo. Obviously that will cause clashes, so we add the reponame as an extra namespace.\n\nThe =--no-tags= option is the complement to =--tags=, which has that default refspec we don’t want. That’s why we disable it and roll our own, entirely.\n\n** Per branch (this is where the magic happens)\n\nIn the context of /a single repository,/ every branch is independently read into a subdirectory for that repository, and merged into the monorepo.\n\nThis is the money shot.\n\n#+CAPTION: handle-branch\n#+NAME: handle-branch\n#+BEGIN_SRC shell\n\u003c\u003cmove-files-to-subdirectory\u003e\u003e\n\u003c\u003censure-on-target-branch-in-monorepo\u003e\u003e\n\ngit read-tree --prefix \"$repopath\" \"refs/remotes/$reponame/$branch\"\ntree=\"$(git write-tree)\"\nmerge_commit=\"$(git commit-tree \\\n    \"$tree\" \\\n    -p \"$branch\" \\\n    -p \"$move_commit\" \\\n    -m \"Merge $reponame/$branch\")\"\ngit reset -q \"$merge_commit\"\n#+END_SRC\n\nSource: [[https://git-scm.com/book/en/v2/Git-Internals-Git-Objects]]\n\n*** Move files to a subdirectory\n\nThe files are moved in a separate, isolated pre-merge step: this helps keep the merge commit a \"pure\" merge and helps =git log --follow= heuristics.\n\n#+name: move-files-to-subdirectory\n#+caption: move-files-to-subdirectory\n#+begin_src shell\ngit read-tree \"$empty_tree\"\ngit read-tree --prefix \"$repopath\" \"refs/remotes/$reponame/$branch\"\ntree=\"$(git write-tree)\"\nmove_commit=\"$(git commit-tree \\\n    \"$tree\" \\\n    -p \"refs/remotes/$reponame/$branch\" \\\n    -m \"Move all files to $repopath/\")\"\n#+end_src\n\nSource: https://stackoverflow.com/a/17440474/4359699\n\n\n*** Ensure we are on the right branch\n\nIn this snippet, we ensure that we are ready to merge fresh code from a subrepo into this branch: either we checkout an existing branch in the monorepo by this name, or we create a fresh one.\n\nWe are given the variable =$branch= which is the final name of the branch we want to operate on. It is the same as the name of the branch in each individual target repo.\n\n#+CAPTION: ensure-on-target-branch-in-monorepo\n#+NAME: ensure-on-target-branch-in-monorepo\n#+BEGIN_SRC shell\nif ! git show-ref --verify --quiet \"refs/heads/$branch\"; then\n    root_commit=\"$(git commit-tree \\\n        \"$empty_tree\" \\\n        -m \"Root commit for monorepo branch $branch\")\"\n    git branch -- \"$branch\" \"$root_commit\"\nfi\ngit symbolic-ref HEAD \"refs/heads/$branch\"\ngit reset -q\n#+END_SRC\n\nInstead of using =git checkout --orphan= and trying to create a new empty commit from the index, we create the empty commit directly and point the new branch to it. Then, we read the branch, new or existing, into the index. Now we have the current index representing the branch, and HEAD pointing at the branch. This allows us to stay in the index and avoid the worktree.\n\nWorking with HEAD feels odd, and it requires using =git reset= to update the branch, rather than =git branch -f ...=, because the branch is checked out. This is still more reliable than not pointing HEAD at the branch, because HEAD is always pointing at /some/ branch (e.g. “master”), so it is easier to just assume you’re /always/ pointing at the “current” branch.\n\nSources:\n- [[https://stackoverflow.com/q/9765453]]\n- [[https://stackoverflow.com/a/6070417]]\n\n*** Non-goal: merging into root\n\nGitHub user @woopla proposed in [[https://github.com/hraban/tomono/pull/42][#42]] the ability to merge a minirepo into the monorepo root, as if you used =.= as the subdirectory. We ended up not going for it, but it was interesting to investigate how to do this with =git read-tree=. The closest I got was:\n\n#+begin_src shell\nif [[ \"$repopath\" == \".\" ]]; then\n    # Experimental—is this how git read-tree works? I find it very confusing.\n    git read-tree \"$branch\" \"$reponame/$branch\"\nelse\n    git read-tree --prefix \"$repopath\" \"$reponame/$branch\"\nfi\n#+end_src\n\nI must to confess I find the [[https://git-scm.com/docs/git-read-tree][git read-tree]] man page too daunting to fully stand by this. I mostly figured it out by trial and error. It seems to work?\n\nIf anyone could explain to me exactly what this tool is supposed to do, what those separate stages are (it talks about “stage 0” to “stage 3” in its 3 way merge), and how you would cleanly do this, just for argument’s sake, I’d love to know.\n\nBut, as it turned out, this tool already has a way to merge a repo into the root: just make it the monorepo, and use it as a target for a =--continue= operation. That solves that.\n\n** Set up the monorepo directory\n\nWe create a fresh directory for this script to run in, or continue on an existing one if the =--continue= flag is passed.\n\n#+CAPTION: prep-dir\n#+NAME: prep-dir\n#+BEGIN_SRC shell\n# Poor man’s arg parse :/\narg=\"${1-}\"\n: \"${MONOREPO_NAME:=core}\"\n\ncase \"$arg\" in\n    \"\")\n        if [[ -d \"$MONOREPO_NAME\" ]]; then\n            \u003e\u00262 echo \"monorepo directory $MONOREPO_NAME already exists\"\n            exit 1\n        fi\n        mkdir \"$MONOREPO_NAME\"\n        cd \"$MONOREPO_NAME\"\n        git init\n        ;;\n\n    \"--continue\")\n        if [[ ! -d \"$MONOREPO_NAME\" ]]; then\n            \u003e\u00262 echo \"Asked to --continue, but monorepo directory $MONOREPO_NAME doesn’t exist\"\n            exit 1\n        fi\n        cd \"$MONOREPO_NAME\"\n        if git status --porcelain | grep . ; then\n            \u003e\u00262 echo \"Git status shows pending changes in the repo. Cannot --continue.\"\n            exit 1\n        fi\n        # There isn’t anything special about --continue, really.\n        ;;\n\n    \"--help\" | \"-h\" | \"help\")\n        cat \u003c\u003cEOF\nUsage: tomono [--continue]\n\nFor more information, see the documentation at \"https://tomono.0brg.net\".\nEOF\n        exit 0\n        ;;\n\n    ,*)\n        \u003e\u00262 echo \"Unexpected argument: $arg\"\n        \u003e\u00262 echo\n        \u003e\u00262 echo \"Usage: tomono [--continue]\"\n        exit 1\n        ;;\nesac\n#+END_SRC\n\nMost of this rigmarole is about UI, and preventing mistakes. As you can see, there is functionally no difference between continuing and starting fresh, beyond =mkdir= and =git init=. At the end of the day, every repo is read in greedily, and whether you do that on an existing monorepo, or a fresh one, doesn’t matter: every repo name you read in, is in fact itself like a =--continue= operation.\n\nIt’s horrible and kludgy but I just want to get something working out the door, for now.\n\n** List individual branches\n\nI want a single branch name per line on stdout, for a single specific remote:\n\n#+CAPTION: list-branches\n#+NAME: list-branches\n#+BEGIN_SRC shell\ngit branch -r --no-color --list \"$reponame/*\" --format \"%(refname:lstrip=3)\"\n#+END_SRC\n\n*** Implementations that didn’t make the cut\n\nSolutions I abandoned, due to one short-coming or another:\n\n**** =git branch -r= with grep\n\nThe most straight-forward way to list branch names:\n\n#+begin_src shell :exports code :results none\n$ git branch -r\n  bar/branch-a\n  bar/branch-b\n  bar/master\n  foo/branch-a\n  foo/master\n#+end_src\n\nThis could be combined with =grep= to filter all branches for a specific remote, and filter out the name. It’s very close, but how do you reliably remove an unknown string?\n\n**** =find .git/refs/hooks=\n\n#+begin_src shell\n( cd \".git/refs/remotes/$reponame\" \u0026\u0026 find . -type f -mindepth 1 | sed -e s/..// )\n#+end_src\n\nCloser, but ugly, and I got reports that it missed some branches (although I was never able to repro)\n\n**** =git ls-remote=\n\n#+begin_src shell\ngit ls-remote --heads --refs \"$reponame\" | sed 's_[^ ]* *refs/heads/__'\n#+end_src\n\nOriginally suggested in a [[https://github.com/hraban/tomono/pull/39][PR 39]], I’ve decided not to use this because =git-ls-remote= actively queries the remote to list its branches, rather than inspecting the local state of whatever we just fetched. That feels like a race condition at best, and becomes very annoying if you’re dealing with password protected remotes or otherwise inaccessible repos.\n\n** Init \u0026 finalize\n\nInitialization is what you’d expect from a shell script:\n\n#+caption: init\n#+name: init\n#+begin_src shell :references yes\n\u003c\u003cset-flags\u003e\u003e\n\n\u003c\u003cprep-dir\u003e\u003e\n\nempty_tree=\"$(git hash-object -t tree /dev/null)\"\n#+end_src\n\nOn the other side, when done, update the working tree to whatever the current branch is to avoid any confusion:\n\n#+caption: finalize\n#+name: finalize\n#+begin_src shell\ngit checkout .\n#+end_src\n\n*** Error flags, warnings, debug\n\nVarious sh flags allow us to control the behaviour of the shell: treat\nany unknown variable reference as an error, treat any non-zero exit\nstatus in a pipeline as an error (instead of only looking at the last\nprogram), and treat any error as fatal and quit. Additionally, if the\n=DEBUGSH= environment variable is set, enable \"debug\" mode by echoing\nevery command before it gets executed.\n\n#+CAPTION: set-flags\n#+NAME: set-flags\n#+BEGIN_SRC shell\nset -euo pipefail ${DEBUGSH+-x}\n\nif ((BASH_VERSINFO[0] \u003e 4 || (BASH_VERSINFO[0] == 4 \u0026\u0026 BASH_VERSINFO[1] \u003e= 4))); then\n\tshopt -s inherit_errexit\nfi\n#+END_SRC\n\nAlso contains a monstrosity which is essentially a version guard around the =inherit_errexit= option, which was only introduced in Bash 4.4. Notably Mac’s default bash doesn’t support it so the version guard is useful.\n\n*** Windows newline fix\n\nOn Windows the config file could contain windows newline endings (CRLF). Bash doesn’t handle those as proper field separators. Even on Windows...\n\nWe force it by adding CR as a field separator:\n\n#+caption: windows-fix\n#+name: windows-fix\n#+begin_src shell\nIFS=$'\\r'\"$IFS\"\n#+end_src\n\nIt can’t hurt to do this on other computers, because who has a carriage return in their repo name or path? Nobody does.\n\nThe real question is: why is this not standard in Bash for Windows? Who knows. I’d add it to my .bashrc if I were you 🤷‍♀️.\n\n* Building the code                                                :noexport:\n\nThis is for tomono development only—end users can directly use the =tomono= script from this repo without building anything.\n\n** Nix\n\nTo build a stand-alone executable:\n\n#+begin_src shell :results none :eval never-export\nnix build .#dist\n#+end_src\n\nFind the executable in =./result/bin/=, and the documentation in =./result/doc=.\n\nTo test the code\n\n#+begin_src shell :results none\nnix flake check .\n#+end_src\n\nTroubleshooting: If you don’t have flakes enabled, add this flag just after the =nix= command:\n\n#+begin_src shell :results none\nnix --extra-experimental-features \"nix-command flakes\" ...\n#+end_src\n\n** Manually using Emacs\n\nYou can use Emacs to build the code manually:\n\nMost of the code in this repository is generated from this readme file. This can be done in stock Emacs, by opening this file and calling =M-x org-babel-tangle=.\n\nThis file can also be exported to HTML. You can use the code below (and its exported command =literate-html-export=) to add some flourish to the HTML.\n\n#+BEGIN_SRC emacs-lisp :exports code :results none :tangle literate-html.el :eval never-export :noweb yes\n;;; literate-html.el --- Export org file to HTML -*- lexical-binding: t; -*-\n\n;; Author: Hraban Luyat \u003chraban@0brg.net\u003e\n;; Keywords: lisp\n;; Version: 0.0.1\n;; Package-Requires: ((emacs \"27.1\") (dash \"2.19.1\"))\n;; URL: https://tomono.0brg.net/\n\n;; \u003c\u003ccopyright\u003e\u003e\n\n;;; Commentary:\n\n;; Slightly more elaborate HTML export for literate programming in Org, aka\n;; babel + noweb. Adds references between listings.\n\n;;; Code:\n\n(require 'cl-lib)\n(require 'dash)\n(require 's)\n(require 'org)\n(require 'ox-html) ;; For the dynamic config vars\n\n(defun literate-html--org-info-name (info)\n  (nth 4 info))\n\n(defun literate-html--insert-ln (\u0026rest args)\n  (apply #'insert args)\n  (newline))\n\n(defun literate-html--should-reference (info)\n  \"Determine if this info block is a referencing code block\"\n  (not (memq (alist-get :noweb (nth 2 info))\n             '(nil \"no\"))))\n\n(defun literate-html--re-findall (re str \u0026optional offset)\n  \"Find all matches of a regex in the given string\"\n  (let ((start (string-match re str offset))\n        (end (match-end 0)))\n    (when (numberp start)\n      (cons (substring str start end) (literate-html--re-findall re str end)))))\n\n;; Match groups are the perfect tool to achieve this but EL's regex is\n;; inferior and it's not worth the hassle. Blag it manually.\n\n(defun literate-html--strip-delimiters (s prefix suffix)\n  \"Strip a PREFIX and SUFFIX delimiter from S.\n\n(literate-html--strip-delimiters \\\"\u003ca\u003e\\\" \\\"\u003c\\\" \\\"\u003e\\\")\n=\u003e \\\"a\\\"\n\nNote this function trusts the input string has those delimiters\"\n  (substring s (length prefix) (- (length suffix))))\n\n(defun literate-html--strip-noweb-delimiters (s)\n  \"Strip the org noweb link delimiters from S, usually \u003c\u003c and \u003e\u003e\"\n  (literate-html--strip-delimiters s\n                        org-babel-noweb-wrap-start\n                        org-babel-noweb-wrap-end))\n\n(defun literate-html--extract-refs (body)\n  (mapcar #'literate-html--strip-noweb-delimiters\n          (literate-html--re-findall (org-babel-noweb-wrap) body)))\n\n(defun literate-html--add-to-hash-list (k elem hash)\n  \"Assuming the HASH values are lists, add this ELEM to K’s list\"\n  (puthash k (cons elem (gethash k hash)) hash))\n\n(defvar literate-html--forward-refs)\n(defvar literate-html--back-refs)\n\n(defun literate-html--register-refs (name refs)\n  (puthash name refs literate-html--forward-refs)\n  ;; Add a backreference to every ref\n  (mapc (lambda (ref)\n          (literate-html--add-to-hash-list ref name literate-html--back-refs))\n        refs))\n\n(defun literate-html--parse-blocks ()\n  (let ((literate-html--forward-refs (make-hash-table :test 'equal))\n        (literate-html--back-refs (make-hash-table :test 'equal)))\n    (org-babel-map-src-blocks nil\n      ;; Probably not v efficient, but should be memoized anyway?\n      (let* ((info (org-babel-get-src-block-info full-block))\n             (name (literate-html--org-info-name info)))\n        (when (and name (literate-html--should-reference info))\n          (literate-html--register-refs name (literate-html--extract-refs body)))))\n    (list literate-html--forward-refs literate-html--back-refs)))\n\n(defun literate-html--format-ref (ref)\n  (format \"[[%s][%s]]\" ref ref))\n\n(defun literate-html--insert-references-block (info title refs)\n  (when refs\n    (insert title)\n    (-\u003e\u003e refs (mapcar 'literate-html--format-ref) (s-join \", \") literate-html--insert-ln)\n    (newline)))\n\n(defun literate-html--insert-references (info forward back)\n  (when (or forward back)\n    (newline)\n    (literate-html--insert-ln \":REFERENCES:\")\n    (literate-html--insert-references-block info \"References: \" forward)\n    (literate-html--insert-references-block info \"Used by: \" back)\n    (literate-html--insert-ln \":END:\")))\n\n(defun literate-html--fix-references (backend)\n  \"Append a references section to every noweb codeblock\"\n  (cl-destructuring-bind (forward-refs back-refs) (literate-html--parse-blocks)\n    (org-babel-map-src-blocks nil\n      (let ((info (org-babel-get-src-block-info full-block)))\n        (when (literate-html--should-reference info)\n          (let ((name (literate-html--org-info-name info)))\n            (goto-char end-block)\n            (literate-html--insert-references\n             info\n             (gethash name forward-refs)\n             (gethash name back-refs))))))))\n\n(defun literate-html-export ()\n  \"Export current org buffer to HTML\"\n  (interactive)\n  (add-hook 'org-export-before-parsing-hook 'literate-html--fix-references nil t)\n\n  ;; The HTML output\n  (let ((org-html-htmlize-output-type 'css))\n    (org-html-export-to-html)))\n\n(provide 'literate-html)\n#+END_SRC\n\n* Tests\n:PROPERTIES:\n:CUSTOM_ID: tests\n:END:\n\n#+begin_quote\n(This section is best viewed in [[https://tomono.0brg.net/#tests][HTML form]]; the GitHub Readme viewer misses some info.)\n#+end_quote\n\nThe examples from this document can be combined into a test script:\n\n#+name: test\n#+BEGIN_SRC shell :tangle test :shebang \"#!/usr/bin/env bash\" :noweb yes :references yes\n\u003c\u003cset-flags\u003e\u003e\n# In tests always echo the command:\nset -x\nexport DEBUGSH=true\n\n# The tomono script is tangled right next to the test script\nexport PATH=\"$PWD:$PATH\"\n\n# Ensure testing always works even on unconfigured CI etc\nexport GIT_AUTHOR_NAME=\"Test\"\nexport GIT_AUTHOR_EMAIL=\"test@test.com\"\nexport GIT_COMMITTER_NAME=\"Test\"\nexport GIT_COMMITTER_EMAIL=\"test@test.com\"\n\n\u003c\u003ctest-setup\u003e\u003e\n\u003c\u003ctest-run\u003e\u003e\n\u003c\u003ctest-evaluate\u003e\u003e\n\n\u003c\u003ctest-extra\u003e\u003e\n#+END_SRC\n\n#+begin_comment\nI’ve chosen to export the fully tangled script to HTML export and hide the separate test implementation below, because I think it makes more sense as a single large script.\n\nAnother note: I originally organized the code in this \"1. implementation, 2. example code (aka test setup), 3. test \u0026 checks\" order during the rewrite, but I now realise that was \"writer-first\" thinking, not \"reader-first\". The natural flow of this text, it is now becoming clear, is to organize all code by subject, aka by which problem it’s solving. When you find a new bug, you want both the explanation of the bug, the code that solves it, and the tests that check it, to all live in one single location. And again that top-level \"test setup, test run, test evaluate\": that’s more top-level writer-first organization. As a reader you want tiny independent chunks.\n#+end_comment\n\nAll we needed to write was the code that actually evaluates the tests and fixtures.\n\n#+name: test-evaluate\n#+begin_src shell :exports none :results none :eval never-export :references yes\n(\ncd core\n\necho \"Checking branch list\"\ndiff -u \u003c(git branch --no-color --list --format \"%(refname:lstrip=2)\" | sort) \u003c(cat \u003c\u003cEOF\nbranch-a\nbranch-b\nempty-branch\nmaster\nEOF\n)\n\necho \"Checking master\"\ngit checkout master\ndiff -u \u003c(find . -name '*.txt' | sort | xargs head) \u003c(cat \u003c\u003cEOF\n==\u003e ./foo/i-am-foo.txt \u003c==\nThis is foo\n\n==\u003e ./lib/zim/i-am-zim.txt \u003c==\nThis is zim\n\n==\u003e ./中文/你好.txt \u003c==\n你好\nEOF\n)\n\necho \"Checking branch-a\"\ngit checkout branch-a\ndiff -u \u003c(find . -name '*.txt' | sort | xargs head) \u003c(cat \u003c\u003cEOF\n==\u003e ./foo/feature-a.txt \u003c==\nI am a new foo feature\n\n==\u003e ./foo/i-am-foo.txt \u003c==\nThis is foo\n\n==\u003e ./lib/zim/feature-a.txt \u003c==\nI am a new zim feature\n\n==\u003e ./lib/zim/i-am-zim.txt \u003c==\nThis is zim\n\n==\u003e ./中文/feature-a.txt \u003c==\n你好 from feature-a\n\n==\u003e ./中文/你好.txt \u003c==\n你好\nEOF\n)\n)\n#+end_src\n\nI use that weird =diff -u \u003c(..)= trick instead of a string compare like ~[[ \"foo\" == \"...\" ]]~ , because the diff shows you where the problem is, instead of just failing the test without comment.\n\n** Edge case: same branch and tag name\n\nIf you have a branch and tag with the same name in a git repo, you will be familiar with this error:\n\n#+begin_quote\nwarning: refname 'foo' is ambiguous.\n#+end_quote\n\nSee [[https://github.com/hraban/tomono/issues/53][#53]]. This happens whenever you refer to the tag or branch by its bare name, without specifying whether it’s a tag or a branch. To fix this, the monorepo script must always use =refs/heads/...= to specify the branch name.\n\nExample:\n\n#+begin_src shell :exports code :eval never-export :results none :noweb-ref test-extra\nmkdir duplicates\n(\n  cd duplicates\n  git init -b check-dupes\n  echo a \u003e a\n  echo b \u003e b\n  git add -A\n  git commit -m commit1 a\n  git tag check-dupes\n  git commit -m commit2 b\n)\n#+end_src\n\nWe now have a =duplicates= repository with a branch /and/ tag =check-dupes=, pointing at different revisions. After including it in the monorepo:\n\n#+begin_src shell :exports code :eval never-export :results none :noweb-ref test-extra\necho \"$PWD/duplicates duplicates\" | tomono --continue\n#+end_src\n\nWe should get:\n\n#+begin_src shell :exports both :eval never-exports :results none :noweb-ref test-extra\n(\n  cd core\n  git checkout check-dupes\n  # This file must exist\n  diff -u duplicates/a \u003c(echo a)\n  # This file too\n  diff -u duplicates/b \u003c(echo b)\n)\n#+end_src\n\n* Copyright and license\n\nThis is a cleanroom reimplementation of the tomono.sh script, originally written with copyright assigned to Ravelin Ltd., a UK fraud detection company. There were some questions around licensing, and it was unclear how to go forward with maintenance of this project given its dispersed copyright, so I went ahead and rewrote the entire thing for a fresh start.\n\nThe license and copyright attribution of this entire document can now be set:\n\n#+CAPTION: copyright\n#+NAME: copyright\n#+BEGIN_SRC fundamental\nCopyright © 2020, 2022–2024 Hraban Luyat\n\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU Affero General Public License as\npublished by the Free Software Foundation, version 3 of the License.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\nGNU Affero General Public License for more details.\n\nYou should have received a copy of the GNU Affero General Public License\nalong with this program.  If not, see \u003chttps://www.gnu.org/licenses/\u003e.\n#+END_SRC\n\nI did not look at the original implementation at all while developing this.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhraban%2Ftomono","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhraban%2Ftomono","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhraban%2Ftomono/lists"}