{"id":22915708,"url":"https://github.com/andreas-kupries/mirror","last_synced_at":"2025-10-25T11:06:58.575Z","repository":{"id":49345284,"uuid":"158746668","full_name":"andreas-kupries/mirror","owner":"andreas-kupries","description":" Mirror of mirror @ core.tcl.tk/akupries -- Mirroring (Backup) of repositories","archived":false,"fork":false,"pushed_at":"2024-02-17T17:38:16.000Z","size":1877,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-01T12:14:01.912Z","etag":null,"topics":["backup-utility","fossil-scm","git","hg","mercurial","svn","tcl"],"latest_commit_sha":null,"homepage":"https://core.tcl.tk/akupries/mirror","language":"Tcl","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andreas-kupries.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-11-22T20:19:01.000Z","updated_at":"2024-07-23T10:58:16.000Z","dependencies_parsed_at":"2022-09-16T19:23:20.845Z","dependency_job_id":null,"html_url":"https://github.com/andreas-kupries/mirror","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/andreas-kupries/mirror","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreas-kupries%2Fmirror","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreas-kupries%2Fmirror/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreas-kupries%2Fmirror/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreas-kupries%2Fmirror/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andreas-kupries","download_url":"https://codeload.github.com/andreas-kupries/mirror/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreas-kupries%2Fmirror/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280943394,"owners_count":26417747,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-25T02:00:06.499Z","response_time":81,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup-utility","fossil-scm","git","hg","mercurial","svn","tcl"],"created_at":"2024-12-14T05:27:52.018Z","updated_at":"2025-10-25T11:06:58.544Z","avatar_url":"https://github.com/andreas-kupries.png","language":"Tcl","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Mirror Management\n\n## Intro\n\nMM is a tool to ease backing up a large number of repositories.\n\nIt currently supports `fossil`, `git(hub)`, `mercurial` (`hg`), and\n`subversion` (`svn`) repositories.\n\nIts operation is made easier by these DVCS all having replication and\nsynchronization protocols baked into them and their clients. Because\nof that MM did not have to invent anything new, it gets by just by\ninvoking the existing tools (`fossil`, `hg`, `git`, `git hub`, `svn`).\n\nAll management is done through a command line application with\nintegrated help, called `mirror`.\n\nBeyond management MM is also capable of exposing the pool of backups\nto the web. This is done by `mirror` generating a static website which\ncan then be served by a web engine of the user's choice. This\nfunctionality requires `TclSSG` to be installed, expecting its main\napplication to be accessible under the name `ssg`.\n\nIt should be noted MM does not automatically perform repository\nupdates on its own. It expects to be invoked by some external\nscheduler, for example `cron`, for this.\n\n## Examples\n\nCurrently the only example of MM in use can be found at\n[https://akupries.tclers.tk/r/tcl](https://akupries.tclers.tk/r/tcl).\nThis location collects and mirrors as many Tcl-related repositories as\nit can find.\n\n## Basics of operation\n\n### Adding repositories\n\nThe basic command to add a single __repository__ to the system is\n\n    mirror add \u003curl\u003e\n\nMM will try to figure out the type of __repository__ to mirror from\nthe specified url, and further derive a name for the __mirror set__ to\nhold the repository as well.\n\nIf it guesses wrong the options `--vcs` and `--name` can be used to\nexplicitly specify the correct values.\n\n    mirror add \u003curl\u003e --name \u003cname\u003e --vcs \u003cvcs\u003e\n\nThe set of version control systems supported by the installed `mirror`\ncan be queried with\n\n    mirror vcs\n\nGoing back to `add` and its auto-detection of vcs type, the currently\nemployed heuristics (.i.e url patterns), are, in order:\n\n|Pattern\t\t\t|VCS\t|Notes\t\t\t\t|\n|---\t\t\t\t|---\t|---\t\t\t\t|\n|`*github*`\t\t\t|github\t|Requires `git hub` \u0026 `git`\t|\n|`*git*`\t\t\t|git\t|Requires `git`\t       \t\t|\n|`*hg.code.sf.net*`\t\t|hg\t|Requires `hg`\t\t\t|\n|`*hg.code.sourceforge.net*`\t|hg\t|S.a.\t\t\t\t|\n|`*svn.code.sf.net*`\t\t|svn\t|Requires `svn`\t\t\t|\n|`*svn.code.sourceforge.net*`\t|svn\t|S.a.\t\t\t\t|\n|`*`\t\t\t\t|fossil\t|Requires `fossil`\t\t|\n\nAs can be seen, the order does matter, and `fossil` is the catch-all\nfallback.\n\nThe two main concepts here are the __repository__, identified by its\nurl, and the __mirror set__, identified by its name.\n\nWhile `mirror add` always places the specified repository into its own\n__mirror set__ the latter can contain more than one repository, while\neach repository always belongs to only one mirror set.\n\nMirror sets are there to group related repositories together. The\ncommand to coalesce mirror sets into one after adding repositories is\n`mirror merge`.\n\nThe action comes at a price, and with restrictions. All repositories\nin a mirror set for the same type VCS will share the local backing\nstore.\n\nFor fossil repositories MM can and does use the asociated\n`projectcode` to detect attempts at merging unrelated repositories,\nand rejects such. For git(hub) no such information exists, and the\nonly warning will be the message `no common ancestors found` when\nupdating such a mirror set. For mercurial the situation is similar.\n\nOn the positive side placing related repositories together reduces the\namount of disk space required.\n\n### Quick access to content\n\nAn important structure maintained by MM is the __rolodex__.\n\nIt is a stack which is updated whenever repositories are added or\nremoved, and mirror sets merged and split. This makes it easy to\nquickly reference repositories which were recently worked on.\n\nThe last and previously used repositories are accessible through the\n`@c` and `@p` short hands. The repositories further down the history\nare accessible via `@num`.\n\nThe new rolodex's contents are always shown after a command changing\nit completes, and can be explicitly queried with `mirror current`.\n\nSearch operations like `mirror list \u003csubstring\u003e` write their results\nto the rolodex as well.\n\nNote, the rolodex is of limited size. The initial default is __20__\nentries. This configuration can be queried and changed with the\n`mirror limit` command.\n\nThe same limit `L` also applies to the output of the `mirror list`\ncommand when not used to search for content. In that case it shows\nonly `L` entries per invokation, and a series of invokations cycles\nthrough the entire list of repositories.\n\n### Updating the mirror\n\nThe command to update the mirror is `mirror update` (sic!).\n\nTo prevent overloading both the local machine and the remote locations\neach invokation of this command will only update a subset of the known\nmirror sets. To this end MM manages an internal queue new mirror sets\nare added to, and mirror sets to update are taken from from the\nfront. When the queue runs empty it is simply refilled again with all\nthe mirror sets known at that time.\n\nThe current state of the queue is accessible via `mirror pending`,\nwith the mirror sets to be taken by the next invokation of `mirror\nupdate` at the top and marked.\n\nThe default is to update _5_ mirror sets per invokation. This\nconfiguration can be queried and changed with the `mirror take`\ncommand.\n\nTogether with being driven by the liks of `cron` this keeps the local\nload low, and distributes the remote load over a larger time interval\nas well, with cron interval and number of sets taken per cycle the\nmain knobs to regulate this.\n\n### Bulk operations\n\nWhile `add` and `merge` are the only operations needed to add new\nrepositories, and manage their mirror sets, using them still will be\ntedious when having to add a large batch.\n\nTo simply this case we have the command `mirror import`. It takes a\nsimple text specifying repositories and their mirror sets in simple\nmarkup and imports them all in one batch, performing all the necessary\n`add` and `merge` operations.\n\nThe file format is line-oriented, with each non-comment line\nspecifying either a repository, or a mirror set. Comments start with a\nhash-character (`#`, U+0023) and run to the end of their line. Empty\nlines are ignored.\n\nThe simplest possible import file looks like\n\n    R \u003cvcs\u003e \u003curl\u003e\n    M \u003cname\u003e\n\nand is equivalent to \n\n    mirror add \u003curl\u003e --name \u003cname\u003e --vcs \u003cvcs\u003e\n\nTo place more repositories into the mirror set \u003cname\u003e simply place\nmore repository specification before it, like\n\n    # A comment\n    R \u003cvcs1\u003e \u003curl1\u003e\n    R \u003cvcs2\u003e \u003curl2\u003e\n    ...\n    M \u003cname\u003e\n\nAny number of repositories and mirror sets can be specified.\n\nOn the converse side of the above is `mirror export`, which writes the\ncurrent state of repositories and mirror sets to `stdout`, in a format\ndirectly usable to `mirror import`.\n\n### 404 - No contact at this number\n\nGiven that MM is for the backup of remote repositories to protect\nagainst their loss, it is only right to handle the possibility of\nremote locations vanishing.\n\nSuch a sitation will actually not disturb the operation of `mirror\nupdate`, and if the loss is temporary the situation will resolve\nitself when update comes back to the repository in question, and\nsimply pull more data from the other side.\n\nHowever when the situation appears to be permanent then the manager\nmight not wish to spend cycles and bandwidth on querying a repository\nwhich is gone. Yet the local backup should not be deleted either.\n\nThus we have `mirror disable` and `mirror enable` with which we can\ntake a repository out of the update rotation, or put it back in. A\nrepository in the rotation is called __active__, and __inactive__\notherwise.\n\n### Web site\n\n... TODO ...\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandreas-kupries%2Fmirror","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandreas-kupries%2Fmirror","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandreas-kupries%2Fmirror/lists"}