{"id":16944458,"url":"https://github.com/jmkim/archvsync","last_synced_at":"2026-05-01T03:38:47.338Z","repository":{"id":82075490,"uuid":"62716652","full_name":"jmkim/archvsync","owner":"jmkim","description":"Fork of https://ftp-master.debian.org/git/archvsync.git","archived":false,"fork":false,"pushed_at":"2016-11-25T05:04:53.000Z","size":233,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-30T10:16:02.274Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Perl","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jmkim.png","metadata":{"files":{"readme":"README","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-07-06T11:41:28.000Z","updated_at":"2016-11-25T05:04:54.000Z","dependencies_parsed_at":null,"dependency_job_id":"8b6dbfd6-175e-42b4-a6e6-8e53a26cfee0","html_url":"https://github.com/jmkim/archvsync","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/jmkim/archvsync","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmkim%2Farchvsync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmkim%2Farchvsync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmkim%2Farchvsync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmkim%2Farchvsync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jmkim","download_url":"https://codeload.github.com/jmkim/archvsync/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmkim%2Farchvsync/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32484352,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-30T13:12:12.517Z","status":"online","status_checked_at":"2026-05-01T02:00:05.856Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T21:18:13.521Z","updated_at":"2026-05-01T03:38:47.323Z","avatar_url":"https://github.com/jmkim.png","language":"Perl","funding_links":[],"categories":[],"sub_categories":[],"readme":"Archvsync\n=========\n\nThis is the central repository for the Debian mirror scripts.  The scripts\nin this repository are written for the purposes of maintaining a Debian\narchive mirror (and shortly, a Debian bug mirror), but they should be\neasily generalizable.\n\n\nCurrently the following scripts are available:\n\n  * ftpsync     - Used to sync an archive using rsync\n  * runmirrors  - Used to notify leaf nodes of available updates\n  * dircombine  - Internal script to manage the mirror user's $HOME\n                  on debian.org machines\n  * typicalsync - Generates a typical Debian mirror\n  * udh         - We are lazy, just a shorthand to avoid typing the\n                  commands, ignore... :)\n\nUsage\n=====\nFor impatient people, short usage instruction:\n\n - Create a dedicated user for the whole mirror.\n - Create a seperate directory for the mirror, writeable by the new user.\n - Place the ftpsync script in the mirror user's $HOME/bin (or just $HOME)\n - Place the ftpsync.conf.sample into $HOME/etc as ftpsync.conf and edit\n   it to suit your system.  You should at the very least change the TO=\n   and RSYNC_HOST lines.\n - Create $HOME/log (or wherever you point $LOGDIR to)\n - If only you receive an update trigger,\n   Setup the .ssh/authorized_keys for the mirror user and place the public key of\n   your upstream mirror into it. Preface it with\nno-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command=\"~/bin/ftpsync\",from=\"IPADDRESS\"\n   and replace $IPADDRESS with that of your upstream mirror.\n - You are finished\n\nIn order to receive different pushes or syncs from different archives,\nname the config file ftpsync-$ARCHIVE.conf and call the ftpsync script\nwith the commandline \"sync:archive:$ARCHIVE\". Replace $ARCHIVE with a\nsensible value. If your upstream mirror pushes you using runmirrors\nbundled together with this sync script, you do not need to add the\n\"sync:archive\" parameter to the commandline, the scripts deal with it\nautomatically.\n\nCommon names used in the Debian mirror network are:\n* standard debian archive:\tdefault empty $ARCHIVE (ftpsync.conf)\n* debian-backports archive:\tARCHIVE=backports (ftpsync-backports.conf)\n* debian-ports archive:\t\tARCHIVE=ports (ftpsync-ports.conf)\n* debian-security archive:\tARCHIVE=security (ftpsync-security.conf)\n\n\nDebian mirror script minimum requirements\n=========================================\nAs always, you may use whatever scripts you want for your Debian mirror,\nbut we *STRONGLY* recommend you to not invent your own. However, if you\nwant to be listed as a mirror it *MUST* support the following minimal\nfunctionality:\n\n - Must perform a 2-stage sync\n   The archive mirroring must be done in 2 stages. The first rsync run\n   must ignore the index files.  The correct exclude options for the\n   first rsync run are:\n  --exclude Packages* --exclude Sources* --exclude Release* --exclude=InRelease --include=i18n/by-hash/** --exclude=i18n/* --exclude ls-lR*\n   The first stage must not delete any files.\n\n   The second stage should then transfer the above excluded files and\n   delete files that no longer belong on the mirror.\n\n   Rationale: If archive mirroring is done in a single stage, there will be\n   periods of time during which the index files will reference files not\n   yet mirrored.\n\n - Must not ignore pushes whil(e|st) running.\n   If a push is received during a run of the mirror sync, it MUST NOT\n   be ignored.  The whole synchronization process must be rerun.\n\n   Rationale: Most implementations of Debian mirror scripts will leave the\n   mirror in an inconsistent state in the event of a second push being\n   received while the first sync is still running.  It is likely that in\n   the near future, the frequency of pushes will increase.\n\n - Should understand multi-stage pushes.\n   The script should parse the arguments it gets via ssh, and if they\n   contain a hint to only sync stage1 or stage2, then ONLY those steps\n   SHOULD be performed.\n\n   Rationale: This enables us to coordinate the timing of the first\n   and second stage pushes and minimize the time during which the\n   archive is desynchronized.  This is especially important for mirrors\n   that are involved in a round robin or GeoDNS setup.\n\n   The minimum arguments the script has to understand are:\n   sync:stage1   Only sync stage1\n   sync:stage2   Only sync stage2\n   sync:all      Do everything. Default if none of stage1/2 are\n                 present.\n   There are more possible arguments, for a complete list see the\n   ftpsync script in our git repository.\n\n\n\nftpsync\n=======\n\nThis script is based on the old anonftpsync script.  It has been rewritten\nto add flexibilty and fix a number of outstanding issues.\n\nSome of the advantages of the new version are:\n - Nearly every aspect is configurable\n - Correct support for multiple pushes\n - Support for multi-stage archive synchronisations\n - Support for hook scripts at various points\n - Support for multiple archives, even if they are pushed using one ssh key\n - Support for multi-hop, multi-stage archive synchronisations\n\n  Correct support for multiple pushes\n  -----------------------------------\n  When the script receives a second push while it is running and syncing\n  the archive it won't ignore it. Instead it will rerun the\n  synchronisation step to ensure the archive is correctly synchronised.\n\n  Scripts that fail to do that risk ending up with an inconsistent archive.\n\n\n  Can do multi-stage archive synchronisations\n  -------------------------------------------\n  The script can be told to only perform the first or second stage of the\n  archive synchronisation.\n\n  This enables us to send all the binary packages and sources to a\n  number of mirrors, and then tell all of them to sync the\n  Packages/Release files at once. This will keep the timeframe in which\n  the mirrors are out of sync very small and will greatly help things like\n  DNS RR entries or even the planned GeoDNS setup.\n\n\n  Multi-hop, multi-stage archive synchronisations\n  -----------------------------------------------\n  The script can be told to perform a multi-hop multi-stage archive\n  synchronisation.\n\n  This is basically the same as the multi-stage synchronisation\n  explained above, but enables the downstream mirror to push his own\n  staged/multi-hop downstreams before returning. This has the same\n  advantage than the multi-stage synchronisation but allows us to do\n  this over multiple level of mirrors. (Imagine one push going from\n  Europe to Australia, where then locally 3 others get updated before\n  stage2 is sent out. Instead of 4times transferring data from Europe to\n  Australia, just to have them all updated near instantly).\n\n\n  Can run hook scripts\n  --------------------\n  ftpsync currently allows 5 hook scripts to run at various points of the\n  mirror sync run.\n\n  Hook1: After lock is acquired, before first rsync\n  Hook2: After first rsync, if successful\n  Hook3: After second rsync, if successful\n  Hook4: Right before leaf mirror triggering\n  Hook5: After leaf mirror trigger (only if we have slave mirrors; HUB=true)\n\n  Note that Hook3 and Hook4 are likely to be called directly after each other.\n  The difference is that Hook3 is called *every* time the second rsync\n  succeeds even if the mirroring needs to re-run due to a second push.\n  Hook4 is only executed if mirroring is completed.\n\n\n  Support for multiple archives, even if they are pushed using one ssh key\n  ------------------------------------------------------------------------\n  If you get multiple archives from your upstream mirror (say Debian,\n  Debian-Backports and Volatile), previously you had to use 3 different ssh\n  keys to be able to automagically synchronize them.  This script can do it\n  all with just one key, if your upstream mirror tells you which archive.\n  See \"Commandline/SSH options\" below for further details.\n\n\nFor details of all available options, please see the extensive documentation\nin the sample configuration file.\n\n\nCommandline/SSH options\n=======================\nScript options may be set either on the local command line, or passed by\nspecifying an ssh \"command\".  Local commandline options always have\nprecedence over the SSH_ORIGINAL_COMMAND ones.\n\nCurrently this script understands the options listed below. To make them\ntake effect they MUST be prepended by \"sync:\".\n\nOption         Behaviour\nstage1         Only do stage1 sync\nstage2         Only do stage2 sync\nall            Do a complete sync (default)\nmhop           Do a multi-hop sync\narchive:foo    Sync archive foo (if the file $HOME/etc/ftpsync-foo.conf\n               exists and is configured)\ncallback       Call back when done (needs proper ssh setup for this to\n               work). It will always use the \"command\" callback:$HOSTNAME\n               where $HOSTNAME is the one defined in config and\n               will happen before slave mirrors are triggered.\n\nSo, to get the script to sync all of the archive behind bpo and call back when\nit is complete, use an upstream trigger of\nssh $USER@$HOST sync:all sync:archive:bpo sync:callback\n\n\nMirror trace files\n==================\nEvery mirror needs to have a 'trace' file under project/trace.\nThe filename has to be the full hostname (eg. hostname -f), or in the\ncase of a mirror participating in RR DNS (where users will never use\nthe hostname) the name of the DNS RR entry, eg. security.debian.org\nfor the security rotation.). (Note: ftp.$COUNTRY.debian.org is always\nwrong, don't use that).\n\nThe contents are defined as:\n\nFirst line is always the output of \"date -u\", ideally run with LANG\nand LC_ALL set to POSIX.\n\nFor the following lines there are two possibilities. Old, legacy,\nstyle is described first, followed by the new standard, as written out\nby ftpsync since February 2016. The latter is now the preferred\nformat.\n\nLegacy format:\n Second line: Freeform text containing the program name and version\n Third line: Text \"Running on host: \" followed by hostname -f\n\nPreferred new format:\n\nLines two to the end of file follow a RFC822 style format, though the\nfield names can have spaces. Currently the following fields are\ndefined, listed in the order as output by ftpsync, though the order is\nnot mandantory:\n\nDate: Date in RFC822 Format when the tracefile got generated, ie. end\n        of mirror run.\n      Example: Sun, 28 Feb 2016 09:40:29 +0000\n\nDate-Started: As Date, but the time the mirror run started.\n              Example: Sun, 28 Feb 2016 09:33:55 +0000\n\nArchive Serial: Archive serial for mirror run, taken from main\n                archives tracefile.\n                Example: 2016022802\n\nUsed ftpsync version: Version string of ftpsync.\n                      Non-ftpsync mirrors are encouraged to switch to\n                      ftpsync, but could output their own name here too.\n                      Example: 20150425\n\nRunning on host: FQDN of mirror host. This may not match the actual\n                   mirror name, its the real hostname.\n                 Example: klecker.debian.org\n\nArchitectures:  List of architectures included in the mirror.\n                Example: GUESSED:{ source amd64 arm64 armel armhf hurd-i386 i386 ia64 kfreebsd-amd64 kfreebsd-i386 mips mips64el mipsel powerpc ppc64el s390 s390x sparc}\n\nUpstream-Mirror: From where does the mirror get its data.\n                 Example: ftp-master.debian.org\n\nSSL: Boolean, is ftpsync rsync over ssl feature used\n     Example: true\n\nTotal bytes received in rsync: rsync --stats output, bytes received\n                               Example: 1109846675\n\nTotal time spent in stage1 rsync: Seconds of runtime for stage1\n                                  Example: 347\n\nTotal time spent in stage2 rsync: Seconds of runtime for stage2\n                                  Example: 47\n\nTotal time spent in rsync: How long in total for rsync\n                           Example: 394\n\nAverage rate: How fast did the sync go\n              2816869 B/s\n\n\nThe third line in the legacy format and the hostname for the \"Running\non Host\" line for the new format MUST NOT be the DNS RR name, even if\nthe mirror is part of it. It MUST BE the hosts own name. This is in\ncontrast to the filename, which SHOULD be the DNS RR name.\n\n\nrunmirrors\n==========\nThis script is used to tell leaf mirrors that it is time to synchronize\ntheir copy of the archive. This is done by parsing a mirror list and\nusing ssh to \"push\" the leaf nodes. You can read much more about the\nprinciple behind the push at [1], essentially it tells the receiving\nend to run a pre-defined script. As the whole setup is extremely limited\nand the ssh key is not usable for anything else than the pre-defined\nscript this is the most secure method for such an action.\n\nThis script supports two types of pushes: The normal single stage push,\nas well as the newer multi-stage push.\n\nThe normal push, as described above, will simply push the leaf node and\nthen go on with the other nodes.\n\nThe multi-staged push first pushes a mirror and tells it to only do a\nstage1 sync run. Then it waits for the mirror (and all others being pushed\nin the same run) to finish that run, before it tells all of the staged\nmirrors to do the stage2 sync.\n\nThis way you can do a nearly-simultaneous update of multiple hosts.\nThis is useful in situations where periods of desynchronization should\nbe kept as small as possible. Examples of scenarios where this might be\nuseful include multiple hosts in a DNS Round Robin entry.\n\nFor details on the mirror list please see the documented\nrunmirrors.mirror.sample file.\n\n\n[1] http://blog.ganneff.de/blog/2007/12/29/ssh-triggers.html\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmkim%2Farchvsync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjmkim%2Farchvsync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmkim%2Farchvsync/lists"}