{"id":13483623,"url":"https://github.com/pypa/bandersnatch","last_synced_at":"2025-06-18T19:06:18.859Z","repository":{"id":37334487,"uuid":"133377409","full_name":"pypa/bandersnatch","owner":"pypa","description":"A PyPI mirror client according to PEP 381 http://www.python.org/dev/peps/pep-0381/","archived":false,"fork":false,"pushed_at":"2025-06-16T19:58:51.000Z","size":2798,"stargazers_count":491,"open_issues_count":55,"forks_count":149,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-06-16T20:28:43.436Z","etag":null,"topics":["bandersnatch","mirror","mirroring","pep381","pypi","pypi-mirror-client"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"afl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pypa.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":"docs/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-05-14T14:52:22.000Z","updated_at":"2025-06-16T19:57:19.000Z","dependencies_parsed_at":"2023-09-25T03:59:59.424Z","dependency_job_id":"6ee7c913-a091-4bc0-a3f1-107e28e22591","html_url":"https://github.com/pypa/bandersnatch","commit_stats":{"total_commits":1641,"total_committers":85,"mean_commits":"19.305882352941175","dds":0.687995124923827,"last_synced_commit":"01ce4dfbfdf52eab8640297ba14797f2669121e6"},"previous_names":[],"tags_count":59,"template":false,"template_full_name":null,"purl":"pkg:github/pypa/bandersnatch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pypa%2Fbandersnatch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pypa%2Fbandersnatch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pypa%2Fbandersnatch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pypa%2Fbandersnatch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pypa","download_url":"https://codeload.github.com/pypa/bandersnatch/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pypa%2Fbandersnatch/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260616537,"owners_count":23036867,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bandersnatch","mirror","mirroring","pep381","pypi","pypi-mirror-client"],"created_at":"2024-07-31T17:01:13.438Z","updated_at":"2025-06-18T19:06:13.844Z","avatar_url":"https://github.com/pypa.png","language":"Python","funding_links":[],"categories":["\u003ca id=\"tag-dev\" href=\"#tag-dev\"\u003eDev\u003c/a\u003e","Python","包仓库"],"sub_categories":["\u003ca id=\"tag-dev.pkg_repo\" href=\"#tag-dev.pkg_repo\"\u003ePackage Repositories\u003c/a\u003e"],"readme":"[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black)\n[![Actions Status](https://github.com/pypa/bandersnatch/workflows/bandersnatch_ci/badge.svg)](https://github.com/pypa/bandersnatch/actions)\n[![codecov.io](https://codecov.io/github/pypa/bandersnatch/coverage.svg?branch=master)](https://codecov.io/github/pypa/bandersnatch)\n[![Documentation Status](https://readthedocs.org/projects/bandersnatch/badge/?version=latest)](http://bandersnatch.readthedocs.io/en/latest/?badge=latest)\n[![Downloads](https://pepy.tech/badge/bandersnatch)](https://pepy.tech/project/bandersnatch)\n\n______________________________________________________________________\n\nThis is a PyPI mirror client according to `PEP 381` + `PEP 503` + `PEP 691`\n\u003chttp://www.python.org/dev/peps/pep-0381/\u003e.\n\n- bandersnatch \u003e=6.0 implements PEP691\n- bandersnatch \u003e=4.0 supports *Linux*, *MacOSX* + *Windows*\n- [Documentation](https://bandersnatch.readthedocs.io/en/latest/)\n\n**bandersnatch maintainers** are looking for more **help**! Please refer to our\n[MAINTAINER](https://github.com/pypa/bandersnatch/blob/master/MAINTAINERS.md)\ndocumentation to see the roles and responsibilities. We would also\nask you read our **Mission Statement** to ensure it aligns with your thoughts for\nthis project.\n\n- If interested contact @cooperlees\n\n## Installation\n\nThe following instructions will place the bandersnatch executable in a\nvirtualenv under `bandersnatch/bin/bandersnatch`.\n\n- bandersnatch **requires** `\u003e= Python 3.11.0`\n\n## Docker\n\nThis will pull latest build. Please use a specific tag if desired.\n\n- Docker image includes `/bandersnatch/src/runner.py` to periodically\n  run a `bandersnatch mirror`\n  - Please `/bandersnatch/src/runner.py --help` for usage\n- With docker, we recommend bind mounting in a read only `bandersnatch.conf`\n  - Defaults to `/conf/bandersnatch.conf`\n\n```shell\ndocker pull pypa/bandersnatch\ndocker run pypa/bandersnatch bandersnatch --help\n```\n\n### Docker Compose\n\nBandersnatch setup using docker-compose is available [here](https://github.com/pypa/bandersnatch/tree/main/src/bandersnatch_docker_compose)\n\n### pip\n\nThis installs the latest stable, released version.\n\n```shell\npython3 -m venv bandersnatch\nbandersnatch/bin/pip install bandersnatch\nbandersnatch/bin/bandersnatch --help\n```\n\n## Quickstart\n\n- Run `bandersnatch mirror` - it will create an empty configuration file\n  for you in `/etc/bandersnatch.conf`.\n- Review `/etc/bandersnatch.conf` and adapt to your needs.\n- Run `bandersnatch mirror` again. It will populate your mirror with the\n  current status of all PyPI packages.\n  Current mirror package size can be seen here: \u003chttps://pypi.org/stats/\u003e\n- A `blocklist` or `allowlist` can be created to cut down your mirror size.\n  You might want to [Analyze PyPI downloads](https://packaging.python.org/guides/analyzing-pypi-package-downloads/)\n  to determine which packages to add to your list.\n- Run `bandersnatch mirror` regularly to update your mirror with any\n  intermediate changes.\n\n### Webserver\n\nConfigure your webserver to serve the `web/` sub-directory of the mirror.\nFor PEP691 support we need to respect the format the client requests.\n\nFor an [nginx](https://www.nginx.com/) example, please look at our\n[banderx](https://github.com/pypa/bandersnatch/tree/main/src/banderx)\ndocker container and [nginx.conf](https://github.com/pypa/bandersnatch/blob/main/src/banderx/nginx.conf)\nexample configuration.\n\n- Note that it is a good idea to have your webserver publish the HTML index\n  files correctly with UTF-8 as the charset. The index pages will work without\n  it but if humans look at the pages the characters will end up looking funny.\n\n- Make sure that the webserver uses UTF-8 to look up unicode path names. nginx\n  gets this right by default - not sure about others.\n\nFor more information visit out [official documentation](https://bandersnatch.readthedocs.io/)\nfor instructions on how to use a NGINX example Docker Image.\n\nIf you are looking to an docker-compose example head over [here](https://github.com/pypa/bandersnatch/tree/main/src/bandersnatch_docker_compose)\n\n### Cron jobs\n\nYou need to set up one cron job to run the mirror itself.\n\nHere's a sample that you could place in `/etc/cron.d/bandersnatch`:\n\n```cron\n    LC_ALL=en_US.utf8\n    */2 * * * * root bandersnatch mirror |\u0026 logger -t bandersnatch[mirror]\n```\n\nThis assumes that you have a `logger` utility installed that will convert the\noutput of the commands to syslog entries.\n\n[SystemD Timers](https://www.freedesktop.org/software/systemd/man/systemd.timer.html)\nare also another alternative in today's modern world.\n\n### Maintenance\n\nbandersnatch does not keep much local state in addition to the mirrored data.\nIn general you can just keep rerunning `bandersnatch mirror` to make it fix\nerrors.\n\nIf you want to force bandersnatch to check everything against the master PyPI:\n\n- run `bandersnatch mirror --force-check` to move status files if they exist in your mirror directory in order get a full sync.\n\nBe aware that full syncs likely take hours depending on PyPI's performance and your network latency and bandwidth.\n\n#### Other Commands\n\n- `bandersnatch delete --help` - Allows you to specify package(s) to be removed from your mirror (*dangerous*)\n- `bandersnatch verify --help` - Crawls your repo and fixes any missed files + deletes any unowned files found (*dangerous*)\n\n### Operational notes\n\n#### Case-sensitive filesystem needed\n\nYou need to run bandersnatch on a case-sensitive filesystem.\n\nOS X natively does this OK even though the filesystem is not strictly\ncase-sensitive and bandersnatch will work fine when running on OS X. However,\ntarring a bandersnatch data directory and moving it to, e.g. Linux with a\ncase-sensitive filesystem will lead to inconsistencies. You can fix those by\ndeleting the status files and have bandersnatch run a full check on your data.\n\n#### Windows requires elevated prompt\n\nBandersnatch makes use of symbolic links. On Windows, this permission is turned off by default for non-admin users. In order to run bandersnatch on Windows either call it from an elevated command prompt (i.e. right-click, run-as Administrator) or give yourself symlink permissions in the group policy editor.\n\n#### Many sub-directories needed\n\nThe PyPI has a quite extensive list of packages that we need to maintain in a\nflat directory. Filesystems with small limits on the number of sub-directories\nper directory can run into a problem like this:\n\n```console\n    2013-07-09 16:11:33,331 ERROR: Error syncing package: zweb@802449\n    OSError: [Errno 31] Too many links: '../pypi/web/simple/zweb'\n```\n\nSpecifically we recommend to avoid using ext3. Ext4 and newer does not have the\nlimitation of 32k sub-directories.\n\n#### Client Compatibility\n\nA bandersnatch static mirror is compatible only to the \"static\", cacheable\nparts of PyPI that are needed to support package installation. It does not\nsupport more dynamic APIs of PyPI that maybe be used by various clients for\nother purposes.\n\nAn example of an unsupported API is [PyPI's XML-RPC interface](https://warehouse.readthedocs.io/api-reference/xml-rpc/), which is used when running `pip search`.\n\n### Bandersnatch Mission\n\nThe bandersnatch project strives to:\n\n- Mirror all static objects of the Python Package Index (\u003chttps://pypi.org/\u003e)\n- bandersnatch's main goal is to support the main global index to local syncing **only**\n- This will allow organizations to have lower latency access to PyPI and\n  save bandwidth on their WAN connections and more importantly the PyPI CDN\n- Custom features and requests may be accepted if they can be of a *plugin* form\n  - e.g. refer to the `blocklist` and `allowlist` plugins\n\n### Contact\n\nIf you have questions or comments, please submit a bug report to\n\u003chttps://github.com/pypa/bandersnatch/issues/new\u003e\n\n- Discord: #bandersnatch now sit in the *PyPA Discord* server. To join visit \u003chttps://discord.com/invite/pypa\u003e\n\n### Code of Conduct\n\nEveryone interacting in the bandersnatch project's codebases, issue trackers,\nchat rooms, and mailing lists is expected to follow the\n[PSF Code of Conduct](https://github.com/pypa/.github/blob/main/CODE_OF_CONDUCT.md).\n\n### Kudos\n\nThis client is based on the original pep381client by *Martin v. Loewis*.\n\n*Richard Jones* was very patient answering questions at PyCon 2013 and made the\nprotocol more reliable by implementing some PyPI enhancements.\n\n*Christian Theune* for creating and maintaining `bandersnatch` for many years!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpypa%2Fbandersnatch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpypa%2Fbandersnatch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpypa%2Fbandersnatch/lists"}