{"id":19388429,"url":"https://github.com/mk-fg/distfiles-convergence","last_synced_at":"2025-04-23T23:31:38.493Z","repository":{"id":3389894,"uuid":"4438607","full_name":"mk-fg/distfiles-convergence","owner":"mk-fg","description":"Tool to verify integrity of the local source tarballs (or distfiles) by mirror network consensus","archived":true,"fork":false,"pushed_at":"2014-05-15T13:48:30.000Z","size":200,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-24T19:58:19.299Z","etag":null,"topics":["checksum","integrity","manifest","python","tripwire","yaml"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"wtfpl","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mk-fg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-05-24T22:06:52.000Z","updated_at":"2023-11-05T19:44:42.000Z","dependencies_parsed_at":"2022-08-27T12:52:18.330Z","dependency_job_id":null,"html_url":"https://github.com/mk-fg/distfiles-convergence","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mk-fg%2Fdistfiles-convergence","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mk-fg%2Fdistfiles-convergence/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mk-fg%2Fdistfiles-convergence/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mk-fg%2Fdistfiles-convergence/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mk-fg","download_url":"https://codeload.github.com/mk-fg/distfiles-convergence/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250532055,"owners_count":21446107,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["checksum","integrity","manifest","python","tripwire","yaml"],"created_at":"2024-11-10T10:12:39.910Z","updated_at":"2025-04-23T23:31:38.231Z","avatar_url":"https://github.com/mk-fg.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"distfiles-convergence\n--------------------\n\n\"It's not just an academic issue.\nNow a days, every blackhat and their mother has owned a major distfile mirror,\nand it's something of a weekend sport to own the smaller more obscure distros\nand trojan their OpenSSH.\nIt's a real threat. It's happening now; it's part of the world we live in.\"\n -- [zx2c4](http://article.gmane.org/gmane.linux.distributions.exherbo.devel/1072)\n\nA simple tool to verify integrity of the local source tarballs or distfiles (as\nthey're called in source-based systems like gentoo or freebsd) by comparing\nthem to another mirrors and available manifests, seeking consensus.\n\n\nUsage\n--------------------\n\nAll the local paths and remote URLs are specified in the [configuration\nfile](https://github.com/mk-fg/distfiles-convergence/blob/master/distfiles_convergence/core.yaml).\n\nSince several configuration files can be specified (each later one overidding\ncorresponding values in the former), it's recommended never to touch the shipped\noriginal file (which gets read automatically) and just create a simplier config,\noverriding what's necessary, for example:\n\n\tlocal:\n\t\t- /srv/distfiles\n\n\tremote:\n\t\tgentoo_portage:\n\t\t\t- /usr/portage\n\t\trsync:\n\t\t\t- rsync://mirror.netcologne.de/gentoo/distfiles/\n\t\t\t- rsync://rsync.mirrorservice.org/www.ibiblio.org/gentoo/\n\t\t\t- rsync://gentoo.mirrors.tds.net/gentoo/distfiles/\n\t\t\t- rsync://trumpetti.atm.tut.fi/gentoo/distfiles/\n\t\t\t- rsync://gentoo.gossamerhost.com/gentoo-distfiles/distfiles/\n\n\tgoal:\n\t\tquery:\n\t\t\tratio: 0.7\n\t\t\thard_min: 2\n\t\t\thard_max: 5\n\n\texclude:\n\t\tfrom_files:\n\t\t\t- /var/lib/dc/exclude.txt\n\n\tmanifest:\n\t\ttype: dbm\n\t\tpath: /var/lib/dc/distfiles.db\n\n\tchecks:\n\t\tgentoo_portage:\n\t\t\tmeta_manifest: /var/lib/dc/portage.{hash}.db\n\nWhen that put as, say, `/etc/dc.yaml`, check can be ran as:\n\n\t% distfiles-convergence -c /etc/dc.yaml\n\nUse \"--debug\" option to see what it's actually doing there.\nSee [baseline configuration\nfile](https://github.com/mk-fg/distfiles-convergence/blob/master/distfiles_convergence/core.yaml)\nfor the full list of available options and their descriptions.\n\nUpon start, app will checksum distfiles in \"local\" path, then go over \"remote\"\nsources in the order in which they're specified, comparing checksums (or files,\nin case of rsync mirrors, since it's generally less traffic and load than full\ndownload and checksum), trying to meet specified \"goal\" (in this particular\nconfig - match against 2-5 or 70% of specified mirrors).\n\nThere are some more subtleties in the process, but basically it's just that.\n\n\nInstallation\n--------------------\n\nIt's a regular package for Python 2.7 (not 3.X), but not in pypi, so can be\ninstalled from a checkout with something like that:\n\n\t% python setup.py install\n\nBetter way would be to use [pip](http://pip-installer.org/) to install all the\nnecessary dependencies as well:\n\n\t% pip install 'git://github.com/mk-fg/distfiles-convergence.git#egg=distfiles-convergence'\n\nNote that to install stuff in system-wide PATH and site-packages, elevated\nprivileges are often required.\nUse \"install --user\",\n[~/.pydistutils.cfg](http://docs.python.org/install/index.html#distutils-configuration-files)\nor [virtualenv](http://pypi.python.org/pypi/virtualenv) to do unprivileged\ninstalls into custom paths.\n\nAlternatively, `./distfiles-convergence` can be run right from the checkout tree,\nwithout any installation.\n\n### Requirements\n\n* Python 2.7 (not 3.X)\n* [layered-yaml-attrdict-config](https://github.com/mk-fg/layered-yaml-attrdict-config)\n* [plumbum](http://plumbum.readthedocs.org/) to work with rsync and\n  gentoo_portage mirrors\n\n\nMirror types\n--------------------\n\n### Gentoo Portage\n\nGentoo portage tree contains \"Manifest\" files with several strong checksums for\neach distfile. Package managers (portage and paludis) use and check these, so\nthey also get quite a lot of review from different network perspectives.\n\nThese Manifests can be easily used, as well as the similar Manifest files from\nany gentoo overlay.\nKeeping local tree (or overlays) in sync with the upstream is outside the scope\nof this app though (but can be done with a simple cronjob).\n\nFor the list of available gentoo portage tree mirrors, see\nhttp://www.gentoo.org/main/en/mirrors-rsync.xml\n\n### Rsync mirrors\n\nThese are more efficient traffic-wise than regular http(s) or ftp mirrors,\nbecause rsync on the server can cooperate with local rsync and just\ncalculate/compare the local/remote checksums.\n\nAlso, requests for checks on rsync mirrors get batched into a single\nconnection/run, to reduce the load on the mirrors.\n\nList of rsync mirrors (among others) can be found here, for instance:\nhttp://www.gentoo.org/main/en/mirrors2.xml\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmk-fg%2Fdistfiles-convergence","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmk-fg%2Fdistfiles-convergence","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmk-fg%2Fdistfiles-convergence/lists"}